WEBINAR

Dynamic Tasks in Airflow

Recorded On October 18, 2022

  • Kenten Danas
  • Tamara Fingerlin

In this webinar, we provide an overview of the dynamic task mapping feature that was first introduced in Airflow 2.3 and expanded further in Airflow 2.4. Previously, tasks could only be generated dynamically at the time the DAG was parsed. Now, with dynamic task mapping, you can create tasks at runtime based on external criteria, without knowing in advance how many tasks you need.

The webinar, which reviews the basic functions of dynamic task mapping through pertinent use cases, answers questions like:

  • What is the difference between static and dynamic tasks in Airflow?
  • How do you implement dynamic task mapping?
  • What are the benefits of dynamic task mapping?
  • What is a dynamic DAG and how does it compare to a dynamic task?

All of the sample code shown in this webinar can be found in this repo.

See More Resources

Astro 101 Webinar: An Introduction to fully-managed Airflow

Modern Infrastructure for World Class AI Applications

Astronomer Providers

Reusable DAG Patterns with TaskGroups

Try Astro for Free for 14 Days

Sign up with your business email and get up to $500 in free credits.

Get Started

Build, run, & observe your data workflows. All in one place.

Build, run, & observe
your data workflows.
All in one place.

Try Astro today and get up to $500 in free credits during your 14-day trial.