Skip to main content

Create an Azure Blob Storage connection in Airflow

Azure Blob Storage provides the storage for all of your Azure Storage data objects, including blobs, files, queues, and tables. Integrating your Azure storage account with Airflow lets you perform different kind of operations on blob objects stored in the cloud. For example, you can create or delete a container, upload or read a blob, or download blobs using Airflow.

This guide explains how to set up an Azure Blob Storage connection using the Azure Blob Storage connection type. Astronomer recommends using this connection type because it utilizes the wasb protocol, which means you can connect with any Azure Storage account including Azure Data Lake Gen 1 and Azure Data Lake Gen 2.

Prerequisites

Get connection details

To create an Azure Blob Storage connection in Airflow, you can use any of the following methods:

Microsoft generates two shared access keys by default for every storage account. You can use them to give Airflow access to the data in your storage account.

An Azure Blob Storage connection using a shared access key requires the following information:

  • Name of the storage account
  • Shared access key

Complete the following steps to retrieve these values:

  1. In your Azure portal, open your storage account.
  2. Copy the name of your storage account.
  3. Follow Microsoft documentation to copy the storage account Key.

Create your connection

  1. Open your Astro project and add the following line to your requirements.txt file:

    apache-airflow-providers-microsoft-azure

    This installs the Microsoft Azure provider package, which makes the Azure Blob Storage connection type available in Airflow.

  2. Run astro dev restart to restart your local Airflow environment and apply your changes in requirements.txt.

  3. In the Airflow UI for your local Airflow environment, go to Admin > Connections. Click + to add a new connection, then choose the Azure Blob Storage connection type.

  4. Fill out the following connection fields using the information you retrieved from Get connection details:

    • Connection Id: Enter a name for the connection.
    • Blob Storage Login: Enter your storage account name.
    • Blog Storage Key: Enter your storage account Key.
  5. Click Test. After the connection test succeeds, click Save.

    azure-connection-storage-access-key

How it works

Airflow uses the Azure SDK for Python to connect to Azure services through the WasbHook.

See also

Was this page helpful?

Sign up for Developer Updates

Get a summary of new Astro features once a month.

You can unsubscribe at any time.
By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.