Manufacturing data pipelines can't afford silent failures. When quality decisions and shop-floor visibility depend on Airflow, a broken DAG at midnight can cost real money. In this episode, Kenten Danas talks with Kumuda Sreenivasa, Founder of Receitly and Senior Data Architect at ATC Drivetrain, about how her team built self-healing Airflow pipelines, where AI fits into the recovery loop, and how they orchestrate AI agents as governed workflow components.
Key Takeaways:
- 00:00 Introduction.
- 02:29 How ATC Drivetrain uses Airflow across thousands of DAGs to orchestrate ETL/ELT jobs, data quality checks, and production reporting for a complex automotive remanufacturing environment.
- 04:08 Defining self-healing: pipelines that identify a known failure, decide whether they can recover safely, execute an approved action, and validate the result, all without paging an engineer.
- 07:03 The five-layer self-healing architecture: observe, classify, policy, recover, and validate.
- 10:00 Where AI fits in the recovery loop: classification, context gathering, and recommendations, but never bypassing operational policies or approval steps.
- 13:00 A concrete before/after: a currency-exchange failure caught overnight by AI-assisted recovery that saved four hours of downtime and roughly 120K.
- 14:27 Confidence levels and success rates: about 95% of small failure modes recover on their own.
- 15:34 AI-assisted troubleshooting at scale: how contextual log analysis and recommended actions save engineers from digging through thousands of log lines.
- 20:13 Orchestrating AI agents through Airflow: treating agents as bounded, governed workflow components with human approval and confidence-based stop conditions.
- 23:45 What Kumuda wants next from Airflow: stronger AI agent governance, standardized tracking of prompts and tool calls, and more flexible event-driven execution.
Resources Mentioned:
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
