Join us for Astro Days: NYC on Sept 27!
Webinars

Implementing Data Quality Checks in Airflow

Watch On Demand

Hosted By

  • Kenten Danas Kenten Danas Lead Developer Advocate
  • Benji Lampel Benji Lampel Ecosystem Engineer

Summary

Data quality is key to the success of an organization’s data systems. In Apache Airflow, implementing data quality checks in DAGs is both easy and effective. With in-DAG quality checks, you can halt pipelines and alert stakeholders before bad data makes its way to a production lake or warehouse.

Executing SQL queries — one of the most common use cases for data pipelines — is a simple way to implement data quality checks. In this webinar, we’ll cover everything you need to know about using SQL for data quality checks, including:

  • How to use the new Common SQL provider
  • How to implement column-level checks
  • How to implement aggregated table-level checks
  • How data quality metrics can be consumed via OpenLineage
  • How to design your pipeline to run, notify, and branch based on the results of data quality checks

Kenten Danas - Lead Developer Advocate at Astronomer

Kenten is a Lead Developer Advocate at Astronomer with a background in field engineering, data engineering, and consulting. She has first-hand experience adopting and running Airflow as a consultant, and is passionate about helping other data engineers scale and get the most out of their Airflow experience.

Benji Lampel - Ecosystem Engineer at Astronomer

Benji is an ecosystem engineer at Astronomer, based in Brooklyn, New York. His work involves understanding the data ecosystem and crafting production grade architectures focused on data quality use cases.

Astronomer Webinars are biweekly, real-time online sessions for data pipeline authors hosted by Astronomer’s Apache Airflow experts. During an hour-long meeting, participants have a chance to dive into the most important features and practices related to Apache Airflow and data orchestration — from Airflow 2+ feature highlights to DAG writing best practices. At the end of each webinar, we open the floor to a Q&A to ensure the participants leave the event confident about their newly acquired knowledge.

Missed the Webinar? Sign up for the recap.

By proceeding you agree to our Privacy Policy and to receive emails from Astronomer.