Building an End-to-End Data Observability System at Netflix with Joseph Machado

Building reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy.

Joseph Machado, Senior Data Engineer at Netflix, joins us to share practical insights gleaned from supporting Netflix’s Ads business as well as over a decade of experience in the data engineering space. He discusses implementing audit publish patterns, building observability dashboards, defining in-band and separate data quality checks, and optimizing data validation across large-scale systems.

Key Takeaways:

(03:14) Supporting data privacy and engineering efficiency within data systems.

(10:41) Validating outputs with reconciliation checks to catch transformation issues.

(16:06) Applying standardized patterns for auditing, validating and publishing data.

(19:28) Capturing historical check results to monitor system health and improvements.

(21:29) Treating data quality and availability as separate monitoring concerns.

(26:26) Using containerization strategies to streamline pipeline executions.

(29:47) Leveraging orchestration platforms for better visibility and retry capability.

(31:59) Managing business pressure without sacrificing data quality practices.

(35:46) Starting simple with quality checks and evolving toward more complex frameworks.

Resources Mentioned:

Netflix website

Start Data Engineering

Apache Airflow

dbt Labs

Great Expectations

https://www.astronomer.io/events/roadshow/london/

https://www.astronomer.io/events/roadshow/new-york/

https://www.astronomer.io/events/roadshow/sydney/

https://www.astronomer.io/events/roadshow/san-francisco/

https://www.astronomer.io/events/roadshow/chicago/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning