Environment variables

On Astro, an Environment Variable is a key-value configuration that can be set at the Deployment or Workspace level. You can use environment variables to configure custom environment variables for your Deployments, customize core settings of Airflow and its pre-installed providers, or store Airflow connections and variables.

Some scenarios where you might use environment variables in your Deployment include:

• Adding a token or a URL that is required by your Airflow Dags or tasks.
• Integrating with Datadog or other third-party tooling to export Deployment metrics.
• Specifying a tag that’s added to all resources created by the Deployment and indicates whether a resource is for development or production.

Some examples of customizing core settings of Airflow or any of its pre-installed providers include:

• Changing the import timeout of DAGBag using AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT.
• Setting up an SMTP service to receive Airflow alerts by email.

You can also use environment variables to store Airflow connections and variables.

Some environment variables on Astro are set globally and can’t be overridden for individual Deployments, while others are used by Astro Runtime to enhance your Airflow experience. For more information on these, see Global environment variables.

Choose a strategy

Environment variables can be used in many different contexts on Airflow. To choose the right management and implementation strategy for your specific use case, it’s helpful to know how Astro prioritizes and stores environment variables for each available management option.

Management options

On Astro, you can manage environment variables from three different locations:

• Workspace Environment Manager: Create environment variables at the Workspace level and link them to multiple Deployments. This allows you to define common environment variables once and share them across Deployments, with the ability to override values per Deployment. See Create environment variables in Astro for setup steps.
• Deployment Environment Variables tab: Set environment variables specific to a single Deployment through your Deployment’s Environment Variables tab in your Deployment’s Environment settings in the Astro UI. This is the fastest way to set a Deployment-specific environment variable. See Using the Astro UI for setup steps.
• Astro project Dockerfile: Store environment variables in your Dockerfile to manage them as code in a version control tool like GitHub. However, environment variables stored in the Dockerfile don’t appear in the Astro UI and might be harder to reference from Dag code. Using the Dockerfile is recommended for more complex production use cases, such as implementing a secrets backend. See Using your Dockerfile for setup steps.

At the local development level, you can use your Astro project .env file to set and test environment variables. When you’re ready to push these environment variables to a Deployment, you can use the Astro CLI to export and store them in the Astro UI for your Deployment.

How environment variables are stored in the Astro UI

Deployment-level environment variables

When you set a non-secret environment variable in the Deployment UI, Astronomer stores the variable in a database that is hosted and managed by Astronomer.

When you set a secret environment variable in the Deployment UI, the following happens:

1. Astro generates a manifest that defines a Kubernetes secret, named env-secrets, that contains your variable’s key and value.
2. Astro applies this manifest to your Deployment’s namespace.
3. After the manifest is applied, the key and value of your environment variable are stored in a managed etcd cluster at rest within Astro.

This process occurs every time you update the environment variable’s key or value. The Deployment restarts when Deployment-level environment variables are saved.

Workspace-level environment variables

When you create an environment variable in the Workspace Environment Manager, Astro stores the environment variable in an Astronomer-hosted secrets manager and applies it to Deployments as Kubernetes Secrets. See How environment variables are stored for details.

Environment variable priority

On Astro, environment variables are applied in the following order of precedence, from highest to lowest:

1. Deployment-level environment variables (set in the Deployment’s Environment Variables tab in the Astro UI)
2. Workspace-level environment variables (set in the Workspace Environment Manager)
3. Environment variables in your Dockerfile

For example, if you set AIRFLOW__CORE__PARALLELISM with one value in the Deployment UI, another value in the Workspace Environment Manager, and a third value in your Dockerfile, the value set in the Deployment UI takes precedence.

When you view environment variables in a Deployment’s Environment Variables page, you’ll see a unified view of both Deployment and Workspace environment variables, with clear indicators showing the source and any overrides.

Similarly, in local development, environment variables set in your .env file take precedence over environment variables set in your Dockerfile.

Example use cases

For most use cases, Astronomer recommends using the Astro UI to store your environment variables for the following reasons:

• It’s easy to use.
• It has built-in security for secret environment variables.
• You can import and export environment variables using the Astro CLI.

There are some scenarios when you might want to use a mix of methods or strategies other than the Astro UI. The following table prescribes specific methods for various common use cases.