Manage DeploymentsSecrets backend

Set up Hashicorp Vault as your secrets backend

This topic provides steps for using Hashicorp Vault as a secrets backend for both local development and on Astro.

To do this, you will:

  • Create an AppRole in Vault which grants Astro minimal required permissions.
  • Write a test Airflow variable or connection as a secret to your Vault server.
  • Configure your Astro project to pull the secret from Vault.
  • Test the backend in a local environment.
  • Deploy your changes to Astro.

If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see Configure a Secrets Backend.

Prerequisites

If you do not already have a Vault server deployed but would like to test this feature, Astronomer recommends that you either:

Step 1: Create a Policy and AppRole in Vault

To use Vault as a secrets backend, Astronomer recommends configuring a Vault AppRole with a policy that grants only the minimum necessary permissions for Astro. For Remote Execution Deployments, you can use any Vault authentication method you prefer, for example Kubernetes auth if your agents and Vault are running on Kubernetes.

To do this:

  1. Run the following command to create a Vault policy that Astro can use to access a Vault server:

    1vault auth enable approle
    2vault policy write astro_policy - <<EOF
    3path "secret/*" {
    4  capabilities = ["create", "read", "update", "patch", "delete", "list"]
    5}
    6EOF

  2. Run the following command to create a Vault AppRole:

    1vault auth enable approle
    2vault write auth/approle/role/astro_role \
    3    role_id=astro_role \
    4    secret_id_ttl=0 \
    5    secret_id_num_uses=0 \
    6    token_num_uses=0 \
    7    token_ttl=24h \
    8    token_max_ttl=24h \
    9    token_policies=astro_policy

  3. Run the following command to retrieve the secret-id for your AppRole:

    $vault write -f auth/approle/role/<your-approle>/secret-id

    Save this value. You’ll use this later to complete the setup.

Step 2: Create an Airflow variable or connection in Vault

To start, create an Airflow variable or connection in Vault that you want to store as a secret. It can be either a real or test value. You will use this secret to test your backend’s functionality.

You can use an existing mount point or create a new one to store your Airflow connections and variables. For example, to create a new mount point called airflow, run the following Vault CLI command:

$vault secrets enable -path=airflow -version=2 kv

To store an Airflow variable in Vault as a secret at the path variables, run the following Vault CLI command with your own values:

$vault kv put -mount=airflow variables/<your-variable-name> value=<your-value-value>

To store an Airflow connection in Vault as a secret at the path connections, first format the connection as a URI. Then, run the following Vault CLI command with your own values:

$vault kv put -mount=airflow connections/<your-connection-name> conn_uri=<connection-type>://<connection-login>:<connection-password>@<connection-host>:<connection-port>

To format existing connections in URI format, see Import and export connections.

Do not use custom key names for your secrets. Airflow requires the key name value for all Airflow variables and the key name conn_uri for all Airflow connections as shown in the previous commands.

To confirm that your secret was written to Vault successfully, run:

$# For variables
>$ vault kv get -mount=airflow variables/<your-variable-name>
>
># For connections
>$ vault kv get -mount=airflow connections/<your-connection-name>

Step 3: Set up Vault locally

In your Astro project, add the Hashicorp Airflow provider to your project by adding the following to your requirements.txt file:

$apache-airflow-providers-hashicorp

Then, add the following environment variables to your .env file:

$AIRFLOW__SECRETS__BACKEND=airflow.providers.hashicorp.secrets.vault.VaultBackend
>AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_path": "connections", "variables_path": "variables",  "mount_point": "airflow", "url": "http://host.docker.internal:8200", "auth_type": "approle", "role_id":"astro_role", "secret_id":"<your-approle-secret>"}

If you run Vault on Hashicorp Cloud Platform (HCP):

  • Replace http://host.docker.internal:8200 with https://<your-cluster>.hashicorp.cloud:8200.
  • Add "namespace": "admin" as an argument after url.

This tells Airflow to look for variable and connection information at the airflow/variables/* and airflow/connections/* paths in your Vault server. You can now run a dag locally to check that your variables are accessible using Variable.get("<your-variable-key>").

For more information on the Airflow provider for Hashicorp Vault and how to further customize your integration, see the Apache Airflow documentation.

Step 4: Deploy configuration

  1. Run the following commands to export your environment variables to Astro:

    $astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND=airflow.providers.hashicorp.secrets.vault.VaultBackend
    >
    >astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_path": "connections", "variables_path": "variables", "mount_point": "airflow", "url": "<your-hashicorpvault-url>", "auth_type": "approle", "role_id":"astro_role", "secret_id":"<your-approle-secret>"}' --secret

  2. Run the following command to push your updated requirements.txt file to Astro:

    $astro deploy --deployment-id <your-deployment-id>

  3. (Optional) Remove the environment variables from your .env file or store your .env file in a safe location to protect your credentials in AIRFLOW__SECRETS__BACKEND_KWARGS.

Now, any Airflow variable or connection that you write to your Vault server can be successfully accessed and pulled by any dag in your Deployment on Astro.