At Lyft, a small data engineering team supports dozens of scientists and analysts who tune driver pay configurations on demand, all under SOX controls. In this episode Ken Obata, Senior Software Engineer at Lyft, joins Kenten Danas to walk through how his team built self-service Dags with an event-driven integration between GitHub Actions and Airflow, and how they solved the data modeling and consistency problems that came with it.
Key Takeaways:
- 00:00 Introduction.
- 02:49 How Airflow responsibility is split at Lyft between the infrastructure org (scaling, version control, Astronomer partnership) and product teams that write Dags, maintain metadata, and orchestrate ETL and MLOps.
- 04:00 Contrasting data problems at large ownership-first orgs (data reconciliation) versus mid-size lean-data orgs like Lyft (data engineers as the bottleneck).
- 07:04 Why Lyft treats Airflow as a service layer rather than a scheduled ETL system, including productionizing Jupyter notebooks from science teams.
- 10:14 The experience running Airflow on Astro, including the 2 to 3 upgrade and consolidated observability in one place.
- 12:13 The two problems driving self-service Dags: supporting 30+ users with a small data engineering team while meeting SOX compliance on driver pay changes.
- 14:38 How the GitHub Actions and Airflow integration works bidirectionally, blocking PRs until validation completes.
- 18:28 Why Lyft chose event-driven over polling: easier upstream-to-downstream context passing and avoiding wasted compute on on-demand work.
- 20:23 The two hardest production challenges: GitHub's tight coupling of check IDs to commit SHAs, and multi-hour simulations racing user rebases. How Lyft built an external context store to solve both.
- 25:15 Why Airflow 3.3's pluggable retry policy and task state store would unlock efficiency gains for transient GitHub failures and long-running Spark simulations.
- 26:39 Looking ahead to task loops and agentic operators for optimization problems with multiple correct answers.
Resources Mentioned:
Lyft Apache Airflow Astro by Astronomer GitHub Actions
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
