Configure Azure Key Vault as a secrets backend on Astro Private Cloud

In this section, you’ll learn how to use Azure Key Vault as a secrets backend on Astro Private Cloud.

Prerequisites

• A Deployment.
• The Astro CLI.
• An Astro project initialized with astro dev init.
• An existing Azure Key Vault linked to a resource group.
• Your Key Vault URL. To find this, go to your Key Vault overview page > Vault URI.

If you do not already have Key Vault configured, see the Microsoft Azure documentation.

Step 1: Register Astro Private Cloud as an app on Azure

Follow the Microsoft Azure documentation to register a new application for Astro Private Cloud.

At a minimum, you need to add a secret that Astro Private Cloud can use to authenticate to Key Vault.

Note the value of the application’s client ID and secret for Step 3.

Step 2: Create an access policy

Follow the Microsoft documentation to create a new access policy for the application that you just registered. The settings you need to configure for your policy are:

• Configure from template: Select Key, Secret, & Certificate Management.
• Select principal: Select the name of the application that you registered in Step 1.

Step 3: Set up Key Vault locally

In your Astro project, add the following line to your requirements.txt file:

apache-airflow-providers-microsoft-azure

In your Dockerfile, add the following environment variables with your own values:

1
ENV AZURE_CLIENT_ID="<your-client-id>" # Found on App Registration page > 'Application (Client) ID'
2
ENV AZURE_TENANT_ID="<your-tenant-id>" # Found on App Registration page > 'Directory (tenant) ID'
3
ENV AZURE_CLIENT_SECRET="<your-client-secret>" # Found on App Registration Page > Certificates and Secrets > Client Secrets > 'Value'
4
ENV AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
5
ENV AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>"}

This tells Airflow to look for variable information at the airflow-variables-* path in Azure Key Vault and connection information at the airflow-connections-* path. In the next step, you’ll run an example dag to test this configuration locally.

Step 4: Test Key Vault locally

To test your Key Vault setup on Astro Private Cloud locally, create a new secret in Key Vault containing either a variable or a connection.

Once you create a test secret, write a simple dag which calls the secret and add this dag to your project’s dags directory. For example, you can use the following dag to print the value of a variable to your task logs:

1
from datetime import datetime
2
3
from airflow import DAG
4
from airflow.models import Variable
5
from airflow.operators.python import PythonOperator
6
7
def print_var():
8
    my_var = Variable.get("<your-variable-key>")
9
    print(f'My variable is: {my_var}')
10
11
with DAG('example_secrets_dags', start_date=datetime(2022, 1, 1), schedule=None) as dag:
12
13
  test_task = PythonOperator(
14
      task_id='test-task',
15
      python_callable=print_var,
16
)

To test your changes:

1. Run astro dev stop followed by astro dev start to push your changes to your local Airflow environment.

2. In the Airflow UI (http://localhost:8080/admin/), trigger your new dag.

3. Click on test-task > View Logs. If you ran the example dag above, you should see the contents of your secret in the task logs:

   {logging_mixin.py:109} INFO - My variable is: my-test-variable

Once you confirm that the setup was successful, you can delete this dag.

Step 5: Push changes to Astro Private Cloud

Once you’ve confirmed that your secrets are being imported correctly to your local environment, you’re ready to configure the same feature in a Deployment on Astro Private Cloud.

1. In the Astro Private Cloud UI, add the same environment variables found in your Dockerfile to your Deployment environment variables. Specify the AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_CLIENT_SECRET variables as Secret to ensure that your credentials are stored securely.
2. In your Astro project, delete the environment variables from your Dockerfile.
3. Deploy your changes to Astro Private Cloud.

From here, you can store any Airflow variables or connections as secrets on Key Vault and use them in your project.