Automating Airflow Backfills with Marquez

Summary

<blockquote> <em> Open Source Summit is a collection of events covering the most important technologies, topics, and issues affecting open source today. </em> </blockquote> <br>

As a data engineer, backfilling data is an important part of your day-to-day work. But backfilling interdependent DAGs is a time-consuming and often unpleasant experience.

Willy Lulciuc, Senior Software Engineer at Astronomer, will briefly introduce you to how backfills are handled in Airflow, then discuss how DAG lineage metadata stored in Marquez can be used to automate backfilling DAGs with complex upstream and downstream dependencies.