Databricks is a SaaS product for data processing using Apache Spark. Integrating Databricks with Airflow lets you manage Databricks clusters, as well as execute and monitor Databricks jobs from an Airflow DAG.
This guide provides the basic setup for creating a Databricks connection. For a complete integration tutorial, see Orchestrate Databricks jobs with Airflow.
A connection from Airflow to Databricks requires the following information:
Complete the following steps to retrieve these values:
https://dbc-75fc7ab7-96a6.cloud.databricks.com/ or https://your-org.cloud.databricks.com/.Astro users can also create connections using the Astro Environment Manager, which stores connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See Create Airflow connections in the Astro UI.
Open your Astro project and add the following line to your requirements.txt file:
This will install the Databricks provider package, which makes the Databricks connection type available in Airflow.
Run astro dev restart to restart your local Airflow environment and apply your changes in requirements.txt.
In the Airflow UI for your local Airflow environment, go to Admin > Connections. Click + to add a new connection, then select the connection type as Databricks.
Fill out the following connection fields using the information you retrieved from Get connection details:
Click Test. After the connection test succeeds, click Save.
Airflow uses Python’s requests library to connect to Databricks through the BaseDatabricksHook.