WEBINAR

Data Driven Scheduling

Recorded On October 11, 2022

  • Kenten Danas
  • Viraj Parekh

In this session, Live with Astronomer dives into the new datasets feature introduced in Airflow 2.4. We’ll show how DAGs that access the same data now have explicit, visible relationships, and how DAGs can be scheduled based on updates to these datasets. This feature is a big step forward in making Airflow data-aware, and vastly expands Airflow’s scheduling capabilities beyond time-based methods like Cron. And it’s widely applicable to many common use cases.

All of the sample code shown in this webinar can be found in this repo.

Live with Astronomer is a biweekly series of short, real-time online sessions for data pipeline authors, hosted by Astronomer’s resident Airflow experts. Topics range from Airflow feature highlights to DAG authoring how-tos to interviews with the developers of Airflow, and every session includes an audience Q&A. It’s all designed to make sure you get the most out of your Airflow experience.

See More Resources

A Deep Dive into the Airflow UI

TaskFlow API in Airflow 2.0

How to pass data between your Airflow tasks

What's new in Airflow 2.10

Try Astro for Free for 14 Days

Sign up with your business email and get up to $500 in free credits.

Get Started

Build, run, & observe your data workflows. All in one place.

Build, run, & observe
your data workflows.
All in one place.

Try Astro today and get up to $500 in free credits during your 14-day trial.