Manage environment variables on Astro
On Astro, you can create, update, or delete environment variables for a Deployment in the following ways:
- Using the Deployment's Environment Variables tab in your Deployment's Environment settings.
- Using your Astro project
Dockerfile
.
The way you manage environment variables can affect security and access for your variable data. See Choose a strategy to determine which management strategy is right for your use case.
Additionally, you can test environment variables from your local Astro environment and export them to the Astro UI. See Manage environment variables locally.
Using the Astro UI
Setting environment variables using the Astro UI is the quickest and easiest way to manage environment variables on Astro.
-
In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.
-
Click the Environment tab.
-
Click Environment variables.
-
Click Edit Variables.
-
Enter an environment variable key and value. For sensitive credentials that should be treated with an additional layer of security, select the Secret checkbox. This permanently hides the variable's value from all users in your Workspace.
-
Click Update Environment Variables to save your changes. Your Airflow scheduler, webserver, and workers restart. After saving, it can take up to two minutes for new variables to be applied to your Deployment.
Edit or delete existing values
After you set an environment variable key, only the environment variable value can be modified. You can modify environment variables that are set as Secret, but the existing secret variable value is never shown. When you modify a secret environment variable, the existing value is erased and you are prompted to enter a new value.
-
In the Astro UI, select a Workspace, click Deployments, and then select a Deployment.
-
Click the Environment tab.
-
Click the Environment Variables tab.
-
Click Edit Variables.
-
Modify the value of the variable you want to edit.
-
Click Update Environment Variables to save your changes. Your Airflow scheduler, webserver, and workers restart. After saving, it can take up to two minutes for updated variables to be applied to your Deployment.
Using your Dockerfile
If you want to store environment variables with an external version control tool, Astronomer recommends setting them in your Dockerfile
. This file is automatically created when you first initialize an Astro project using astro dev init
.
Environment variables set in your Dockerfile
are stored in plain text. For this reason, Astronomer recommends storing sensitive environment variables using the Astro UI or a third-party secrets backend. For more information, see Configure a secrets backend.
-
Open your Astro project
Dockerfile
. -
To add the environment variables, declare an ENV command with the environment variable key and value. For example, the following
Dockerfile
sets two environment variables:FROM quay.io/astronomer/astro-runtime:12.0.0
ENV AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG=1
ENV AIRFLOW_VAR_MY_VAR=25 -
Save your Dockerfile and run
astro deploy
to deploy your variables to an Astro Deployment. To apply your changes locally, useastro dev restart
to rebuild your image. -
(Optional) To verify if the environment variables are applied correctly to Astro Deployment or your local Airflow environment, you can use
os.getenv("AIRFLOW_VAR_MY_VAR")
inside of Airflow DAGs and tasks.To view a list of all the environment variables set in your local Airflow environment, refer to the Step 4 of Using Astro CLI in local Airflow environment
To delete an environment variable from your Astro Runtime image, remove or comment the line in your Dockerfile
that defines it.
Environment variables set in your Dockerfile are not visible in the Astro UI.
Manage environment variables locally
You can use the Astro CLI to set environment variables on Astro and your local Airflow environment. If you're developing locally, the best way to manage environment variables is using your Astro project .env
file.
-
Open your Astro project
.env
file. -
Use the following format to set your environment variables in the
.env
file:KEY=VALUE
Environment variables should be in all-caps and not include spaces.
Alternatively, you can run
astro deployment variable list --save
to copy environment variables from an existing Deployment to a file. -
Restart your local environment using
astro dev restart
.
To confirm that your environment variables were applied:
- Run
astro dev bash --scheduler
to log in to the scheduler container. - Run
printenv | grep <your-env-variable>
in the container to print all environment variables that are applied to your environment. - Run
exit
to exit the container.
To export the contents of your .env
file to an Astro Deployment, run the following command:
astro deployment variable update --deployment-id <your-deployment-id> --load .env
When you use the .env
file to add or update environment variables on Astro, it will overwrite all existing variables in your Astro Deployment. To update only select environment variables, run astro deployment variable create
without the --load
option. For example, the following command creates two new environment variables without affecting existing Deployment environment variables:
astro deployment variable create AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT=60 ENVIRONMENT_TYPE=dev --deployment-id cl03oiq7d80402nwn7fsl3dmv
After you deploy environment variables, your Deployment automatically restarts to and applies the variables. To verify if the environment variables were applied correctly, go to Environment > Environment Variables of your Deployment settings in the Astro UI.
Use multiple .env files
The Astro CLI looks for .env
by default, but if you want to specify multiple files, make .env
a top-level directory and create sub-files within that folder.
A project with multiple .env
files might look like the following:
my_project
├── Dockerfile
├── dags
│ └── my_dag
├── include
│ └── my_operators
├── airflow_settings.yaml
└── .env
├── dev.env
└── prod.env
See also
- Set Airflow connections using environment variables.
- Set Airflow variables using environment variables.
- Import and export environment variables