Astro + Databricks
Run pipelines using Databricks with Astro, the modern data orchestration platform powered by Apache Airflow. Astro’s Databricks provider offers seamless integration with the Databricks environment and asynchronous operators provide a cost-effective way to schedule your Astro pipelines in relation to Databricks events.
About Databricks
Databricks is a popular unified data science, engineering, and analytics platform built around combining features of data warehouses and data lakes into a lakehouse architecture. Use Astro as an orchestrator, and use an execution framework like Databricks, to do the heavy lifting of your data processing.
Use Case
Databricks offers extensive computing power and a way to define tasks using languages commonly used for Data Science such as R, SQL, Scala, and Python. Combine these assets with Astro’s extensive data ecosystem integrations and data lineage capabilities to write robust Machine Learning pipelines in an interdisciplinary and interconnected way.
