Watch the recording
By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.
Datasets and Data-Aware Scheduling in Airflow
Recorded on
Tamara FingerlinSenior Developer AdvocateAstronomer
Constance MartineauStaff Product ManagerAstronomer
Datasets in Airflow are a powerful feature that allow you to define explicit dependencies between DAGs and the data they rely on. By leveraging datasets, you can establish clearer relationships between workflows, optimize scheduling for DAGs accessing the same data, and improve coordination across teams. This approach enhances pipeline efficiency, transparency, and overall data orchestration.
In this webinar, we’ll explore how to take full advantage of datasets and data-aware scheduling, including:
- Implementing datasets effectively, from pipeline design to advanced scheduling techniques
- Monitoring dataset-driven pipelines using the Airflow UI and Astro Observe
- Upcoming enhancements in Airflow 3, including updates to datasets and additional functionality
Resources:
