Strict sequential execution across DAGs sounds simple until you have scheduled pipelines and event-driven pipelines writing to the same MongoDB collections. Ivana Isailovic, Senior Big Data Engineer at Synechron, joins Marc Lamberti to walk through the three-layer DAG architecture her team built to solve exactly that, plus how they generate 200-task DAGs and how they rebuilt subdag-style group retries in Airflow 3.
Key Takeaways:
- (00:00) Introduction.
- (02:16) The stack: Snowflake source, MongoDB with a medallion (bronze, silver, gold) layout, Spark for processing, Airflow for orchestration, Elasticsearch for reports.
- (05:29) Why standard Airflow options (max active runs, pools, dependency setups) each solved only part of the problem.
- (06:40) Data consistency across bronze, silver, and gold layers is what forced strict sequential execution.
- (09:11) Scheduled DAGs versus event-driven DAGs triggered at any moment from the application side.
- (10:04) The three-layer architecture: trigger DAGs, a single proxy DAG that controls the queue, and main ETL DAGs.
- (12:00) The queue is literally another DAG. The proxy DAG allows only one active run and serializes everything behind it.
- (15:13) 200-task DAGs generated from nested task groups and YAML configuration files, with DAG versions tied to release numbers.
- (17:38) How the layers talk to each other: sensors and TriggerDagRunOperator.
- (19:44) Migrating from subdags to task groups without losing the ability to retry a whole group.
- (21:49) Airflow 2.9 approach: reset task instance state via the metadata DB, keyed off the task group identifier.
- (23:13) Airflow 3 approach: move the retry logic onto the official REST API for stability, security, and maintainability.
Resources Mentioned:
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
