How to Orchestrate Databricks Jobs Using Airflow

Using Airflow with Databricks is common in the data ecosystem. Many data teams leverage Databricks to run heavy workloads like data transformations for ETL or ELT pipelines, and data analysis. Using Airflow as a tool-agnostic orchestrator in combination with Databricks provides several advantages, such as easy integration with other tools in your data stack, and managing your pipelines as code.

Airflow’s open-source functionality makes it easy to orchestrate Databricks Jobs, allowing you to take advantage of Databricks’ Job clusters while maintaining full visibility from Airflow.

In this webinar, you’ll learn how to:

• Use Airflow to orchestrate your Databricks Jobs by using the Airflow Databricks Provider.
• Monitor your Databricks Jobs from Airflow
• Write tasks that run Databricks Jobs using familiar DAG code