Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at Idelic (now part of Descartes), about how his team standardized DAG creation and logging, built a Jenkins-driven testing workflow with per-PR Airflow environments, and is opening up DAG configuration and triggering to their own customers through the Safety Suite UI.
Key Takeaways:
- 00:00 Introduction.
- 01:40 Idelic's mission: collecting data from North American trucking fleets and using it to identify risky drivers so fleets can intervene before accidents happen.
- 02:59 Where Airflow fits: Idelic runs on Astro, with each customer/integration combination as its own DAG.
- 03:56 The reality of vendor integrations: APIs, SFTP, email, screen scraping, and even executables installed on customer servers to push data out.
- 06:11 How the DAG count grew and where things started to hurt as a small team tried to keep up.
- 08:49 The standardized framework: abstract methods that wrap common Airflow functions and generate task groups for each ETL phase, plus standardized logging across vendors.
- 11:15 CI/CD with Jenkins: per-PR Airflow deployments that QA can spin up on demand through a Jira hook and tear back down when done.
- 13:31 Letting customers create their own DAGs through a wizard in Safety Suite, with plans for customer-triggered runs, filtered result review, and human-in-the-loop approval.
- 15:37 Using the Airflow API to trigger customer DAGs, and why Idelic pushes run data into its own database instead of relying on the Airflow logs API.
- 18:58 What Matt wants next from Airflow: task and DAG level trend tracking, and more from Otto on self-healing DAGs.
Resources Mentioned:
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
