WEBINAR

How to expertly organize your DAGs with task groups

Recorded On July 18, 2023

  • Tamara Fingerlin
  • Kenten Danas

This webinar covers task groups, a great Airflow feature that allow you to to organize complex DAG structures. While they were originally designed as a UI feature, task group functionality has been expanded to help with everything from task organization, to task state management, to modularizing common patterns in your pipelines. Using task groups is a beginner-friendly way to take your Airflow DAGs to the next level.

Watch the recording to learn everything you need to know to use task groups both in simple and advanced ways, including:

  • How to create and manage task groups with classes and decorators.
  • How to parameterize task groups.
  • How to dynamically map task groups.
  • How to modularize common patterns into task groups.
  • How to use task groups included in provider packages like Cosmos and the Astro Databricks provider.

All code covered in this webinar can be found in this repo. You can also learn more about task groups in our Academy course and Airflow concept guide.

See More Resources

Optimizing your Airflow Developer Experience: Exploring DAG Versioning and Backfills

The Astro Python SDK Load File Function

How to orchestrate your dbt workflows with Airflow

Getting Started With the Official Airflow Helm Chart

Try Astro for Free for 14 Days

Sign up with your business email and get up to $500 in free credits.

Get Started

Build, run, & observe your data workflows. All in one place.

Build, run, & observe
your data workflows.
All in one place.

Try Astro today and get up to $500 in free credits during your 14-day trial.