globe
Webinars

Dynamic Tasks in Airflow

Watch On Demand

Missed the Webinar? Watch Now.

By proceeding you agree to our Privacy Policy , our Website Terms and to receive emails from Astronomer.

Summary

With the releases of Airflow 2.3 and 2.4, users can write DAGs that dynamically generate parallel tasks at runtime. This feature, known as dynamic task mapping, is a paradigm shift for DAG design in Airflow. Prior to Airflow 2.3, tasks could only be generated dynamically at the time that the DAG was parsed, meaning you had to change your DAG code if you needed to adjust tasks based on some external factor. With dynamic task mapping, you can easily write DAGs that create tasks based on your current runtime environment.

In this webinar, we’ll cover everything you need to know to implement dynamic tasks in your DAGs, including:

  • How to apply basic dynamic task mapping concepts
  • How to map over multiple parameters and different combinations of parameters
  • How to use dynamic tasks to implement common use cases
  • When you should use dynamic tasks versus generating dynamic DAGs

Kenten Danas - Lead Developer Advocate at Astronomer

Kenten is a Lead Developer Advocate at Astronomer, with a background in field engineering, data engineering, and consulting. She has first-hand experience adopting and running Airflow as a consultant, and is passionate about helping other data engineers scale and get the most out of their Airflow experience.


Tamara Fingerlin - Developer Advocate at Astronomer

Tamara is a Developer Advocate at Astronomer, the commercial developer behind Apache Airflow. She discovered her passion for data and optimizing workflows in her previous career and now focuses on creating educational resources that help beginners to get started with Airflow. Being a career changer herself, she cares deeply about making exciting Airflow features accessible for users of all backgrounds. In her spare time she helps to organize data challenges as a Data Scientist at the World Data League, a competition tackling social impact challenges.


Astronomer Webinars are biweekly, real-time online sessions for data pipeline authors hosted by Astronomer’s Apache Airflow experts. During an hour-long meeting, participants have a chance to dive into the most important features and practices related to Apache Airflow and data orchestration — from Airflow 2+ feature highlights to DAG writing best practices. At the end of each webinar, we open the floor for a Q&A to ensure that participants leave the event confident about their newly acquired knowledge.

Hosted By

  • Kenten Danas Kenten Danas Lead Developer Advocate
    Astronomer
  • Tamara Fingerlin Tamara Fingerlin Developer Advocate
    Astronomer