Data quality is key to the health of an organization’s data ecosystem. With in-DAG quality checks, you can halt pipelines before bad data makes its way into production.
Implementing data quality checks efficiently can be challenging, but has recently gotten a lot easier with the addition of automatic setup/teardown tasks in Airflow 2.7. Setup/teardown tasks support the pattern of creating resources to run data quality checks and tearing them down once the checks are completed.
This webinar covers everything you need to know about setup/teardown tasks and how to use them to support your data quality checks, including:
- Basic setup/teardown task concepts and usage.
- Efficient DAG design for data quality checks using setup/teardown tasks.
- A real world example showing how Astronomer uses this pattern in our own DAGs.
