Orchestrating data across more than 30 companies means most Airflow users on the platform aren't data engineers. In this episode, Kenten Danas talks with Lucas Trubiano, Data Engineer at itti, the technology company within Grupo Vázquez in Paraguay. Lucas walks through the custom YAML framework his Center of Excellence built on top of Airflow, how they baked data quality and custom operators into it, and how a spec-driven AI workflow now lets product and business users contribute to templates without knowing Python.
Key Takeaways:
- 00:00 Introduction.
- 01:47 What itti and Grupo Vázquez do, and the Data Engineering Center of Excellence's mandate to build a 360-degree view of the customer across more than 30 companies.
- 02:56 How Airflow fits in as a central task orchestrator (not a processing engine) across around 300 production DAGs.
- 04:25 Managing enterprise-scale Airflow: preferring Airflow-as-a-service, plus enabling self-service for non-technical users through YAML.
- 05:44 Why itti built a second, more opinionated YAML framework after DAG Factory-style customization created a code review bottleneck.
- 07:14 More than 80% of new DAGs are now created with the new framework because it's simply faster.
- 07:54 How the framework works end to end: Python DAGs, Jinja templates, YAML configs, and CI/CD compilation.
- 09:10 A Google Sheets ingestion example that shows how prevalidation, download, and processing tasks are hidden behind a simple YAML config to preserve reliability.
- 10:14 Building an in-house data quality tool that tests per partition instead of full-scanning tables, triggered via custom Airflow operators.
- 11:17 Custom operators for dbt, the in-house data quality tool, and AWS services like QuickSight dashboard refreshes, and how OSS Airflow makes them portable across instances.
- 15:35 Spec-driven development with a fork of GitHub Spec Kit so business users can describe what they want and let agents generate DAGs against certified templates.
- 17:24 Slack-native error routing: every DAG has an owner team, common errors ship with explanations, and only deep issues escalate to the central team.
- 19:29 Where they're heading next: agent-triggered pull requests for self-healing pipelines.
- 20:22 Airflow 3 wishlist: backfill improvements, event-driven orchestration for streaming pipelines, and Human in the Loop for generative AI DAGs.
Resources Mentioned:
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
