WEBINAR

How to Orchestrate Databricks Jobs Using Airflow

Recorded On December 5, 2024

  • Kenten Danas
  • Tamara Fingerlin

Using Airflow with Databricks is common in the data ecosystem. Many data teams leverage Databricks to run heavy workloads like training machine learning models, data transformations, and data analysis. Using Airflow as a tool-agnostic orchestrator in combination with Databricks provides several advantages, such as easy integration with other tools in your data stack, and managing your pipelines as code.

Airflow’s open-source functionality makes it easy to orchestrate Databricks Jobs, allowing you to take advantage of Databricks’ Job clusters while maintaining full visibility from Airflow. In this webinar, we show how to use Airflow to orchestrate your Databricks Jobs by using the Airflow Databricks Provider. You can find the code shown in the demo in this repo.

See More Resources

How to orchestrate your dbt workflows with Airflow

Astro 101 Webinar: An Introduction to fully-managed Airflow

How to pass data between your Airflow tasks

How to expertly organize your DAGs with task groups

Try Astro for Free for 14 Days

Sign up with your business email and get up to $500 in free credits.

Get Started

Build, run, & observe your data workflows. All in one place.

Build, run, & observe
your data workflows.
All in one place.

Try Astro today and get up to $500 in free credits during your 14-day trial.