Watch the recording

By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.

Data-Aware Scheduling with the Astro Python SDK

Recorded on

  • Tamara FingerlinSenior Developer AdvocateAstronomer
  • Benji LampelEcosystem EngineerAstronomer

In Airflow 2.4 the Datasets feature was introduced.

This allows data-aware scheduling:

  • The DAG author (you) can tell Airflow that a task is updating a Dataset: outlets=[Dataset(“s3://my_bucket”)]
  • DAGs can be scheduled to run on these updates to Datasets: schedule=[Dataset(“s3://my_bucket”)]

You can find all the needed resources in this Github repository.

Watch Now