Hosted By
-
Julien Le Dem
Chief Architect
Astronomer
Summary
<blockquote> <em> Berlin Buzzwords is a conference on storing, processing, streaming and searching large amounts of digital data, with a focus on open source software projects. </em> </blockquote> <br>
OpenLineage provides a standard for lineage collection that spans multiple platforms, including Apache Airflow, Apache Spark, Flink, and dbt. This empowers teams to diagnose and address widespread data quality and efficiency issues in real time.
In this session, Julien Le Dem, our Chief Architect, will show how to trace data lineage across Apache Spark and Apache Airflow. He will walk through the OpenLineage architecture and provide a live demo of a running pipeline with real-time data lineage.