Data-Aware Scheduling with the Astro Python SDK

In Airflow 2.4 the Datasets feature was introduced.

This allows data-aware scheduling:

• The DAG author (you) can tell Airflow that a task is updating a Dataset: outlets=[Dataset(“s3://my_bucket”)]
• DAGs can be scheduled to run on these updates to Datasets: schedule=[Dataset(“s3://my_bucket”)]

You can find all the needed resources in this Github repository.