Azure Data Factory (ADF) is a cloud-based data integration and transformation service used to build data pipelines. Integrating ADF with Airflow allows you to run ADF pipelines and check their status from an Airflow DAG.
This guide provides the basic setup for creating an ADF connection. For a complete integration tutorial, see Run Azure Data Factory pipelines in Airflow.
A connection from Airflow to Azure Data Factory requires the following information:
Complete the following steps to retrieve all of these values:
Astro users can also create connections using the Astro Environment Manager, which stores connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See Create Airflow connections in the Astro UI.
Open your Astro project and add the following line to your requirements.txt file:
This will install the Azure provider package, which makes the Azure Data Factory connection type available in Airflow.
Run astro dev restart to restart your local Airflow environment and apply your changes in requirements.txt.
In the Airflow UI for your local Airflow environment, go to Admin > Connections. Click + to add a new connection, then choose Azure Data Factory as the connection type.
Fill out the following connection fields using the information you retrieved from Get connection details:
Click Test. After the connection test succeeds, click Save.
To use the same connection for multiple data factories or multiple resource groups, skip the Resource Group and Factory Name fields in the connection configuration. Instead, you can pass these values to the default_args of a DAG or as parameters to the AzureDataFactoryOperator. For example:
Airflow uses the azure-mgmt-datafactory library from Azure SDK for Python to connect to Azure Data Factory using AzureDataFactoryHook.