Set up Azure Key Vault as your secrets backend

This topic provides setup steps for configuring Azure Key Vault as a secrets backend on Astro.

If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see Configure a Secrets Backend.

Prerequisites

A Deployment.
The Astro CLI.
An Astro project.
An existing Azure Key Vault linked to a resource group.
Your Key Vault URL. To find this, go to your Key Vault overview page > Vault URI.
(Remote Execution Only) Helm installed
(Remote Execution Only) The values.yaml file from the Register Agents modal in your Deployments>Agents page.

If you do not already have Key Vault configured, read Microsoft Azure documentation.

Step 1: Register Astro as an app on Azure

If you use a managed identity to authenticate to Key Vault, skip to Step 3.

Follow the Microsoft Azure documentation to register a new application for Astro.

At a minimum, you need to add a secret that Astro can use to authenticate to Key Vault.

Note the value of the application’s client ID and secret for Step 3.

Step 2: Create an access policy

If you use a managed identity to authenticate to Key Vault, skip to Step 3. Ensure your managed identity has an access policy or Azure RBAC role that grants it access to your Key Vault secrets.

Follow the Microsoft documentation to create a new access policy for the application that you just registered. The settings you need to configure for your policy are:

Configure from template: Select Key, Secret, & Certificate Management.
Select principal: Select the name of the application that you registered in Step 1.

Step 3: Set up Key Vault locally

Astro

Remote Execution

In your Astro project, add the following line to your requirements.txt file:

apache-airflow-providers-microsoft-azure

Add the following environment variables to your .env file. Choose the option that matches your authentication method:

Client secret authentication:

AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "tenant_id": "<your-tenant-id>", "client_id": "<your-client-id>", "client_secret": "<your-client-secret>"}

For client secret authentication, find your client ID in Azure Portal at App Registration page > Application (Client) ID. To find your tenant ID, go to App Registration page > Directory (tenant) ID. To find your client secret, go to App Registration Page > Certificates and Secrets > Client Secrets > Value.

Managed identity authentication:

Before using managed identity authentication, you must configure your Deployment with a workload identity. See the Azure tab in Authorize a Deployment to cloud resources using workload identity to set up your managed identity and authorize it to your Deployment.

AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "managed_identity_client_id": "<your-managed-identity-client-id>", "workload_identity_tenant_id": "<your-tenant-id>"}

For managed identity authentication, find your managed identity client ID in Azure Portal at Managed Identities > your identity > Client ID. To find your tenant ID, go to Microsoft Entra ID > Overview > Tenant ID.

This configuration tells Airflow to look for variable information at the airflow/variables/* path in Azure Key Vault and connection information at the airflow/connections/* path. You can now run a dag locally to check that your variables are accessible using Variable.get("<your-variable-key>").

By default, this setup requires that you prefix any secret names in Key Vault with airflow-connections or airflow-variables. If you don’t want to use prefixes in your Key Vault secret names, set the values for sep, "connections_prefix", and "variables_prefix" to "" within AIRFLOW__SECRETS__BACKEND_KWARGS.

Step 4: Deploy configuration

Astro

Remote Execution

Run the following commands to export your environment variables to Astro.

1 astro deployment variable create --deployment-id <your-deployment-id> --load --env .env

In the Astro UI, mark AIRFLOW__SECRETS__BACKEND_KWARGS as Secret. See Set environment variables in the Astro UI.

Run the following command to push your updated requirements.txt file to Astro:

1 astro deploy --deployment-id <your-deployment-id>

(Optional) Remove the environment variables from your .env file, or store your .env file so that your credentials are hidden, for example with GitHub secrets.