Authenticate to cloud services with user credentials

Prerequisites​

• A user account on AWS with access to AWS cloud resources.
• The AWS CLI.
• The Astro CLI.
• An Astro project.

Retrieve AWS user credentials locally​

Run the following command to obtain your user credentials locally:

aws configure

This command prompts you for your Access Key Id, Secret Access Key, Region, and output format. If you log into AWS using single sign-on (SSO), run aws configure sso instead.

The AWS CLI then stores your credentials in two separate files:

• .aws/config
• .aws/credentials

The location of these files depends on your operating system:

• Linux: /home/<username>/.aws
• Mac: /Users/<username>/.aws
• Windows: %UserProfile%/.aws

Configure your Astro project​

The Astro CLI runs Airflow in a Docker-based environment. To give Airflow access to your credential files, you'll mount the .aws folder as a volume in Docker.

1. In your Astro project, create a file named docker-compose.override.yml with the following configuration:

   • Mac
   • Linux
   • Windows

   version: "3.1"
   services:
       scheduler:
           volumes:
           - /Users/<username>/.aws:/home/astro/.aws:rw
       webserver:
           volumes:
           - /Users/<username>/.aws:/home/astro/.aws:rw
       triggerer:
           volumes:
           - /Users/<username>/.aws:/home/astro/.aws:rw
   

2. In your Astro project's .env file, add the following environment variables. Make sure that the volume path is the same as the one you configured in the docker-compose.override.yml.

   AWS_CONFIG_FILE=/home/astro/.aws/config
   AWS_SHARED_CREDENTIALS_FILE=/home/astro/.aws/credentials
   

When you run Airflow locally, all AWS connections without defined credentials automatically fall back to your user credentials when connecting to AWS. Airflow applies and overrides user credentials for AWS connections in the following order:

• Mounted user credentials in the ~/.aws/config file.
• Configurations in aws_access_key_id, aws_secret_access_key, and aws_session_token.
• An explicit username & password provided in the connection.

For example, if you completed the configuration in this document and then created a new AWS connection with its own username and password, Airflow would use those credentials instead of the credentials in ~/.aws/config.

Prerequisites​

• A user account on GCP with access to GCP cloud resources.
• The Google Cloud SDK.
• The Astro CLI.
• An Astro project.
• Optional. Access to a secrets backend hosted on GCP, such as GCP Secret Manager.

Retrieve GCP user credentials locally​

Run the following command to obtain your user credentials locally:

gcloud auth application-default login

The SDK provides a link to a webpage where you can log in to your Google Cloud account. After you complete your login, the SDK stores your user credentials in a file named application_default_credentials.json.

The location of this file depends on your operating system:

• Linux: $HOME/.config/gcloud/application_default_credentials.json
• Mac: /Users/<username>/.config/gcloud/application_default_credentials.json
• Windows: %APPDATA%/gcloud/application_default_credentials.json

Configure your Astro project​

The Astro CLI runs Airflow in a Docker-based environment. To give Airflow access to your credential file, mount it as a Docker volume.

1. In your Astro project, create a file named docker-compose.override.yml to your project with the following configuration:

   • Mac
   • Linux
   • Windows

   version: "3.1"
   services:
       scheduler:
           volumes:
           - /Users/<username>/.config/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
       webserver:
           volumes:
           - /Users/<username>/.config/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
       triggerer:
           volumes:
           - /Users/<username>/.config/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
   

2. In your Astro project's .env file, add the following environment variable. Ensure that this volume path is the same as the one you configured in docker-compose.override.yml.

   GOOGLE_APPLICATION_CREDENTIALS=/usr/local/airflow/gcloud/application_default_credentials.json
   

When you run Airflow locally, all GCP connections without defined credentials automatically fall back to your user credentials when connecting to GCP. Airflow applies and overrides user credentials for GCP connections in the following order:

• Mounted user credentials in the /~/gcloud/ folder
• Configurations in gcp_keyfile_dict
• An explicit username & password provided in the connection

For example, if you completed the configuration in this document and then created a new GCP connection with its own username and password, Airflow would use those credentials instead of the credentials in ~/gcloud/application_default_credentials.json.

Prerequisites​

• A user account on Azure with access to Azure cloud resources.
• The Azure CLI.
• The Astro CLI.
• An Astro project
• If you're using Windows, Windows Subsystem Linux.

Retrieve Azure user credentials locally​

Run the following command to obtain your user credentials locally:

az login

The CLI provides you with a link to a webpage where you authenticate to your Azure account. Once you complete the login, the CLI stores your user credentials in your local Azure configuration folder. The developer account credentials are used in place of the credentials associated with the Registered Application (Service Principal) in Microsoft Entra ID.

The default location of the Azure configuration folder depends on your operating system:

• Linux: $HOME/.azure/
• Mac: /Users/<username>/.azure
• Windows: %USERPROFILE%/.azure/

Configure your Astro project​

The Astro CLI runs Airflow in a Docker-based environment. To give Airflow access to your credential files, mount the .azure folder as a volume in Docker.

1. In your Astro project, create a file named docker-compose.override.yml with the following configuration:

   • Mac
   • Windows and Linux

   version: "3.1"
   services:
       scheduler:
           volumes:
           - /Users/<username>/.azure:/usr/local/airflow/.azure:rw
       webserver:
           volumes:
           - /Users/<username>/.azure:/usr/local/airflow/.azure:rw
       triggerer:
           volumes:
           - /Users/<username>/.azure:/usr/local/airflow/.azure:rw
   

2. Add the following lines after the FROM line in your Dockerfile to install the Azure CLI inside your Astro Runtime image:

   # FROM ...
   USER root
   RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash
   USER ASTRO
   

3. Add the following environment variable to your .env file. Make sure the file path is the same volume location you configured in docker-compose.override.yml

   AZURE_CONFIG_DIR=/usr/local/airflow/.azure
   

When you run Airflow locally, all Azure connections without defined credentials automatically fall back to your user credentials when connecting to Azure. Airflow applies and overrides user credentials for Azure connections in the following order:

• Mounted user credentials in /~/.azure.
• Configurations in azure_client_id, azure_tenant_id, and azure_client_secret.
• An explicit username & password provided in the connection.

For example, if you completed the configuration in this document and then created a new Azure connection with its own username and password, Airflow would use those credentials instead of the credentials in ~/.azure/config.

1. Create a secret for an Airflow variable or connection in AWS Secrets Manager. All Airflow variables and connection keys must be prefixed with the following strings respectively:

   • airflow/variables/<my_variable_name>
   • airflow/connections/<my_connection_id>

   For example when adding the secret variable my_secret_var you will need to give the secret the name airflow/variables/my_secret_var.

   When setting the secret type, choose Other type of secret and select the Plaintext option. If you're creating a connection URI or a non-dict variable as a secret, remove the brackets and quotations that are pre-populated in the plaintext field.

2. Add the following environment variables to your Astro project .env file. For additional configuration options, see the Apache Airflow documentation. Make sure to specify your region_name.

   AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
   AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "region_name": "<your-aws-region>"}
   

3. Run the following command to start Airflow locally:

   astro dev start
   

4. Access the Airflow UI at localhost:8080 and create an Airflow AWS connection named aws_standard with no credentials. See Connections.

   When you use this connection in your DAG, it will fall back to using your configured user credentials.

5. Add a DAG which uses the secrets backend to your Astro project dags directory. You can use the following example DAG to retrieve <my_variable_name> and <my_connection_id> from the secrets backend and print it to the terminal:

   from airflow.models.dag import DAG
   from airflow.hooks.base import BaseHook
   from airflow.models import Variable
   from airflow.decorators import task
   from datetime import datetime
   
   with DAG(
       'example_secrets_dag',
       start_date=datetime(2022, 1, 1),
       schedule=None
   ):
   
       @task
       def print_var():
           my_var = Variable.get("<my_variable_name>")
           print(f"My secret variable is: {my_var}") # secrets will be masked in the logs!
   
           conn = BaseHook.get_connection(conn_id="<my_connection_id>")
           print(f"My secret connection is: {conn.get_uri()}") # secrets will be masked in the logs!
           
       print_var()
   

6. In the Airflow UI, unpause your DAG and click Play to trigger a DAG run.

7. View logs for your DAG run. If the connection was successful, your masked secrets appear in your logs. See Airflow logging.

1. Create a secret for an Airflow variable or connection in GCP Secret Manager. You can do this using the Google Cloud Console or the gcloud CLI. All Airflow variables and connection keys must be prefixed with the following strings respectively:

   • airflow-variables-<my_variable_name>
   • airflow-connections-<my_connection_name>

   For example when adding the secret variable my_secret_var you will need to give the secret the name airflow-variables-my_secret_var.

2. Add the following environment variables to your Astro project .env file. For additional configuration options, see the Apache Airflow documentation. Make sure to specify your project_id.

   AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
   AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "project_id": "<my-project-id>"}
   

3. Run the following command to start Airflow locally:

   astro dev start
   

4. Access the Airflow UI at localhost:8080 and create an Airflow GCP connection named gcp_standard with no credentials. See Connections.

   When you use this connection in your DAG, it will fall back to using your configured user credentials.

5. Add a DAG which uses the secrets backend to your Astro project dags directory. You can use the following example DAG to retrieve <my_variable_name> and <my_connection_id> from the secrets backend and print it to the terminal:

   from airflow.models.dag import DAG
   from airflow.hooks.base import BaseHook
   from airflow.models import Variable
   from airflow.decorators import task
   from datetime import datetime
   
   with DAG(
       'example_secrets_dag',
       start_date=datetime(2022, 1, 1),
       schedule=None
   ):
   
       @task
       def print_var():
           my_var = Variable.get("<my_variable_name>")
           print(f"My secret variable is: {my_var}")
   
           conn = BaseHook.get_connection(conn_id="<my_connection_name>")
           print(f"My secret connection is: {conn.get_uri()}")
   
       print_var()
   

6. In the Airflow UI, unpause your DAG and click Play to trigger a DAG run.

7. View logs for your DAG run. If the connection was successful, your masked secrets appear in your logs. See Airflow logging.

1. Create a secret for an Airflow variable or connection in Azure Key Vault. All Airflow variables and connection keys must be prefixed with the following strings respectively:

   • airflow-variables-<my_variable_name>
   • airflow-connections-<my_connection_name>

   For example, to use a secret named mysecretvar in your DAG, you must name the secret airflow-variables-mysecretvar.

   You will need to store your connection in URI format.

2. In your Astro project, add the following line to Astro project requirements.txt file:

   apache-airflow-providers-microsoft-azure
   

3. Add the following environment variables to your Astro project .env file. For additional configuration options, see the Apache Airflow documentation. Make sure to specify your vault_url.

   AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
   AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>"}
   

   By default, this setup requires that you prefix any secret names in Key Vault with airflow-connections or airflow-variables. If you don't want to use prefixes in your Key Vault secret names, set the values for "connections_prefix" and "variables_prefix" to "" within AIRFLOW__SECRETS__BACKEND_KWARGS. The vault_url can be found on the overview page of your Key vault under Vault URI.

4. Run the following command to start Airflow locally:

   astro dev start
   

5. Access the Airflow UI at localhost:8080 and create an Airflow Azure connection named azure_standard with no credentials. See Connections.

   When you use this connection in your DAG, it will fall back to using your configured user credentials.

6. Add a DAG which uses the secrets backend to your Astro project dags directory. You can use the following example DAG to retrieve a value from airflow/variables and print it to the terminal:

   from airflow.models.dag import DAG
   from airflow.hooks.base import BaseHook
   from airflow.models import Variable
   from airflow.decorators import task
   from datetime import datetime
   
   with DAG(
       'example_secrets_dag',
       start_date=datetime(2022, 1, 1),
       schedule=None
   ):
   
       @task
       def print_var():
           my_var = Variable.get("mysecretvar")
           print(f"My secret variable is: {my_var}")
   
           conn = BaseHook.get_connection(conn_id="mysecretconnection")
           print(f"My secret connection is: {conn.get_uri()}")
   
       print_var()
   

7. In the Airflow UI, unpause your DAG and click Play to trigger a DAG run.

8. View logs for your DAG run. If the connection was successful, your masked secrets appear in your logs. See Airflow logging.