Skip to main content

Set up Azure Key Vault as your secrets backend

This topic provides setup steps for configuring Azure Key Vault as a secrets backend on Astro.

If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see Configure a Secrets Backend.

Prerequisites

  • A Deployment.
  • The Astro CLI.
  • An Astro project.
  • An existing Azure Key Vault linked to a resource group.
  • Your Key Vault URL. To find this, go to your Key Vault overview page > Vault URI.

If you do not already have Key Vault configured, read Microsoft Azure documentation.

Step 1: Register Astro as an app on Azure

Follow the Microsoft Azure documentation to register a new application for Astro.

At a minimum, you need to add a secret that Astro can use to authenticate to Key Vault.

Note the value of the application's client ID and secret for Step 3.

Step 2: Create an access policy

Follow the Microsoft documentation to create a new access policy for the application that you just registered. The settings you need to configure for your policy are:

  • Configure from template: Select Key, Secret, & Certificate Management.
  • Select principal: Select the name of the application that you registered in Step 1.

Step 3: Set up Key Vault locally

In your Astro project, add the following line to your requirements.txt file:

apache-airflow-providers-microsoft-azure

Add the following environment variables to your .env file:

AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "tenant_id": "<your-tenant-id>", "client_id": "<your-client-id>", "client_secret": "<your-client-secret>"}

To find your your client ID in Azure Portal, go to App Registration page > Application (Client) ID. To find your tenant ID, go to App Registration page > Directory (tenant) ID. To find your client secret, go to App Registration Page > Certificates and Secrets > Client Secrets > Value.

This configuration tells Airflow to look for variable information at the airflow/variables/* path in Azure Key Vault and connection information at the airflow/connections/* path. You can now run a DAG locally to check that your variables are accessible using Variable.get("<your-variable-key>").

By default, this setup requires that you prefix any secret names in Key Vault with airflow-connections or airflow-variables. If you don't want to use prefixes in your Key Vault secret names, set the values for sep, "connections_prefix", and "variables_prefix" to "" within AIRFLOW__SECRETS__BACKEND_KWARGS.

Step 4: Deploy to Astro

  1. Run the following commands to export your environment variables to Astro.

    astro deployment variable create --deployment-id <your-deployment-id> --load --env .env

    In the Astro UI, mark AIRFLOW__SECRETS__BACKEND_KWARGS as Secret. See Set environment variables in the Astro UI.

  2. Run the following command to push your updated requirements.txt file to Astro:

    astro deploy --deployment-id <your-deployment-id> 
  3. (Optional) Remove the environment variables from your .env file, or store your .env file so that your credentials are hidden, for example with GitHub secrets.

Was this page helpful?

Sign up for Developer Updates

Get a summary of new Astro features once a month.

You can unsubscribe at any time.
By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.