As data ecosystems become increasingly complex, it is essential for organizations to have complete visibility into the performance and health of their environments. Today, I am thrilled to announce that Astronomer has acquired Datakin, the real-time, operational data lineage tool from founders of the OpenLineage and Marquez open-source projects. Datakin traces and visualizes detailed data lineage through pipelines, allowing teams to identify, resolve, and prevent issues before they become a liability.
Astronomer and Datakin — Better Together
We share Datakin’s mission to help organizations build and manage reliable data ecosystems that deliver trusted data, which in turn drives business-critical decisions. Joining forces will allow us to accelerate this objective considerably. As we integrate Datakin’s capabilities into the Astronomer platform, customers will benefit from the observability and context lineage provides, along with the power to build, run, and observe pipelines as code with our modern data orchestrator. Datakin will become an integral part of the Astronomer ecosystem and our customers’ data teams will be able to collect lineage metadata natively from their Airflow environments, visualize dependencies across pipelines, and explore lineage and history of datasets over time.
Datakin — Complete Operational Awareness
Datakin delivers an end-to-end solution for engineers and data scientists looking to increase understanding across distributed data systems. It automatically traces data lineage and tracks it as it changes over time, so teams have complete operational visibility and the tools they need to quickly resolve issues. Built around OpenLineage, an open-source framework for operational data lineage, and Marquez, a lineage metadata server and user interface, Datakin allows you to see your entire data ecosystem in one easy-to-manage view, identify and address data failures, and anticipate risks. The teams behind Datakin include founders of and principal contributors to OpenLineage and Marquez, incubation-level projects at the LF AI & Data Foundation. The company was founded by CEO Laurent Paris and CTO Julien Le Dem. A maker at heart, Laurent has built and led world-class technology organizations at Yahoo, Spotify, and WeWork. Julien is a renowned community builder and problem solver who — as the co-creator of Apache Parquet and a contributor to Apache Pig, Apache Arrow, and Apache Iceberg — has dedicated his life to open source and data.
Astronomer — Building the Future of Workflow Orchestration
Astronomer delivers a modern data orchestration platform, powered by Apache Airflow, that empowers data teams to build, run, and observe data pipelines. Founded in 2018, Astronomer is the steward of the Airflow project, at the center of the Airflow community and ecosystem, and the driving force behind releases and the Airflow roadmap. Airflow’s comprehensive orchestration capabilities and flexible, Python-based pipelines-as-code model rapidly made it the most popular open-source orchestrator available. Astronomer’s global customers range from innovative startups to the world’s largest big-box retailer. We work with hundreds of companies, including Sonos, Condé Nast, Herman Miller, Rappi, and Electronic Arts, delivering thousands of successful production Airflow environments. With a focus on developer productivity, observability, and flexibility for any organization at any scale, Astronomer helps to quickly and easily deliver results with Airflow.
Please join us in welcoming the Datakin team to Astronomer. We’ll be announcing exciting new product developments in the coming months as we work together to take data orchestration to the next level.
Sign up here for an early look at how Astronomer and Datakin will work together.
See coverage of this and other Astronomer news in Christine Hall’s TechCrunch story.
