When a real-time loan eligibility scoring pipeline is built on cron jobs, midnight pages are inevitable. In this episode, Chandan Gowda, Data Engineer at Accion Labs, joins Kenten to discuss how his team uses Airflow across client projects, including a financial services scoring use case and a POC applying production-grade orchestration to RAG and GenAI data pipelines.
Key Takeaways:
- 00:00 Introduction.
- 01:00 What Accion Labs does as a technology consulting and services firm working across BFSI, healthcare, and retail.
- 02:00 Chandan's role at the intersection of data engineering and GenAI, building pipelines one week and RAG-based agents the next.
- 04:20 Why Airflow tends to win client evaluations: infrastructure agnostic, no cloud lock-in, fine-grained control over pipeline logic.
- 06:00 Containerizing Airflow on Kubernetes or VMs so migrations between clouds don't require a rewrite.
- 08:14 The loan eligibility scoring use case for a financial services client, and replacing fragile cron jobs with a single Airflow DAG end to end, cutting effort by about 25%.
- 10:42 Triggering strategy: S3 file sensors as the primary trigger handling 90% of runs, plus a scheduled fallback as a safety net.
- 12:53 End-to-end flow inside Airflow: ingestion into the data lake, validation and transformation with credit bureau joins, containerized model inference, and writeback to the loan management system.
- 15:56 The AI orchestration POC and why the data feeding GenAI models needs the same rigor as any production pipeline.
- 18:09 Using Airflow to detect document changes and re-chunk and re-embed only what changed, with quality thresholds and rollback before promoting to live.
- 20:23 The roadmap: model evaluation pipelines, multi-agent orchestration, and provider packages for LangChain, OpenAI, and Hugging Face.
- 22:32 Wishlist for Airflow: native event-driven triggers beyond polling sensors, first-class observability for AI workloads, and better dynamic DAG generation at scale.
Resources Mentioned:
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
