Dynamic DAGs

WATCH ON DEMAND

Summary:

The simplest way of creating an Airflow DAG is to write it as a static Python file. However, sometimes manually writing DAGs isn't practical. Maybe you have hundreds or thousands of DAGs that do similar things, with just a parameter changing between them. Or maybe you need a set of DAGs to load tables, but don't want to manually update DAGs every time those tables change. In these cases, and others, it can make more sense to dynamically generate DAGs. Because everything in Airflow is code, you can dynamically generate DAGs using Python alone. In this webinar, we'll talk about when you might want to dynamically generate your DAGs, show a couple of methods for doing so, and discuss problems that can arise when implementing dynamic generation at scale.

In this webinar we covered:

  • How Airflow identifies a DAG
  • Use cases for dynamically generating DAGs
  • Commonly used methods for dynamic generation
  • Pitfalls and common issues with dynamic generation

Missed the Webinar? Sign up for the Recap

Recap Preview

The simplest way of creating an Airflow DAG is to write it as a static Python file. However, sometimes manually writing DAGs isn't practical.

Maybe you have hundreds or thousands of DAGs that do similar things, with just a parameter changing between them. Or maybe you need a set of DAGs to load tables, but don't want to manually update DAGs every time those tables change.

In these cases, and others, it can make more sense to dynamically generate DAGs. Because everything in Airflow is code, you can dynamically generate DAGs using Python alone.

In this webinar, we'll talk about when you might want to dynamically generate your DAGs, show a couple of methods for doing so, and discuss problems that can arise when implementing dynamic generation at scale.

Generating DAGs - The Static Way

Most people who have used Airflow are familiar with defining DAGs statically. You create a Python file, instantiate your DAG, and define your tasks.

dynamicDags1

Hosted By

Kenten Danas

Kenten Danas

Field Engineer

Viraj Parekh

Viraj Parekh

Field CTO