Develop DAGsConnections and variables

Create Airflow variables in the Astro UI

You can create and manage Airflow variables for Deployments with the Astro Environment Manager in the Astro UI. The Environment Manager uses an Astro-managed secrets backend to store Airflow variable key-value pairs as Kubernetes Secrets.

Using the Environment Manager, you can quickly and securely create Airflow variables once and share them to multiple Deployments without having to set up your own secrets backend. You can also create a variable once and use it across multiple Airflow Deployments.

For example, you can configure a variable with the access token credentials for a sandbox or development environment. Then, you can later configure your variable to be applied to all Deployments in the workspace by default. This means that when you create new Deployments, they automatically have access to your development environment. Later, you can override the variable to use your production credentials in production Deployments by using a field override or edit.

Compared to creating a variable in the Airflow UI, when you create a variable in the Astro UI, you can:

  • Share the variable with multiple Deployments within the Workspace.
  • Share the variable with Astro IDE projects so that their ephemeral test Deployments can use it.
  • Override the variable value for individual Deployments or Astro IDE projects.
  • Use variables in branch-based deploys and PR previews.

Workspace Owners and Operators can create and assign variables, while Workspace Authors can view configured variables and use them in Deployments.

How variables are stored

When you create an Airflow variable in the Environment Manager, Astro stores Airflow variable details in an Astronomer-hosted secrets manager, and then applies variables to Deployments as Kubernetes Secrets. Specifically the following steps occur:

  • Astro stores the variable value in a secure secrets manager hosted by Astronomer.
  • When a variable is assigned to a Deployment, Astro uses Airflow’s provided local filesystem secrets backend to mount your Airflow variables as Kubernetes Secrets.
  • When your Dags use your variables, Airflow reads the details from the filesystem using the Airflow local filesystem secrets backend.

This process occurs every time you create or update a variable.

Prerequisites

Create an Airflow Variable

You can create Airflow variables both at the Deployment and Workspace level. When you create a variable at the Deployment level, the variable details are available only to that specific Deployment. When you create a variable at the Workspace level, you can apply the variable to several Deployments and override its value as needed for each Deployment.

To create a variable at the Workspace level:

  1. In the Astro UI, click Environment in the left menu to open the Airflow Variables page.
  2. Click + Airflow Variable to add a new variable.
  3. Choose whether you want your variable to be Automatically link to all deployments by setting the toggle to On. See Link variables to Deployments for more information.
  4. Enter your information in the required fields.
  5. Choose whether you want your variable to be Not Secret or Secret, when working with it in the future. This allows you to store API keys, tokens, or other secrets in an Airflow Variable without those credentials being available to future editors.
  6. Click Create Airflow Variable.
  7. If you haven’t linked your variable to all deployments, make it accessible to individual Deployments. See Link variables to Deployments.

To create a variable at the Deployment level:

  1. In the Astro UI, select a Deployment, then click the Environment tab within the Deployment menu.
  2. Click + Airflow Variable to add a new variable.
  3. Enter your information in the required fields.
  4. Choose whether you want your variable to be Not Secret or Secret, when working with it in the future. This allows you to store API keys, tokens, or other secrets in an Airflow Variable without those credentials being available to future editors.
  5. Click Create Airflow Variable to make your new variable.

After you create a variable, you can reference its Airflow Variable Key from Dag code like you would with any Airflow variable created through the Airflow UI.

After you create a variable at the Workspace level, you can link it to multiple Deployments. Linking variables is useful for standardizing external resource usage across your entire team.

For the most flexibility, you can set default variables and override the variable details per-Deployment based on details like the Deployment’s usage and environment type (production or development).

  1. In the Astro UI, click Environment in the left menu to open the Airflow Variables page.
  2. Click the variable you want to link to a Deployment.
  3. Click + Link Deployment.
  4. Choose a Deployment from the list that appears.
  5. (Optional) Click More options and then add any value overrides for this Deployment. For example, if your variable provides a development token by default, you can override its details to instead provide a production token.
  6. Click Link Airflow Variable.

Configure variable sharing for a Workspace

You can configure Astro to link Workspace-level variables to all Deployments in the Workspace by default.

This is useful, for example, when you need to configure a variable for development environments that all Deployments in a Workspace should start with. Then, when you create new Deployments, they automatically have a default variable to your development resources.

When you’re ready to connect your Deployments to production resources, you can either replace the variable or override the variable field values with your production resource information.

If you toggle the Automatically link to all deployments setting from Off to On, Astro respects any variable value overrides that you might have configured for existing linked Deployments.

  1. Click Environment in the main menu to open the Airflow Variables page.
  2. Click the variable that you want to add per-Deployment field overrides to.
  3. Toggle the Automatically link to all deployments setting to choose either:
    • Off: Only share the variable individually to Deployments.
    • On: Link to all current and future Deployments in this Workspace.
  4. (Optional) Change the default variable value.
  5. Click Update Airflow Variable to save.

Override variable values

If you create a variable at the Workspace level and link it to a Deployment, you can later edit the variable within the Deployment to specify a value override. When you override the variable’s value, you specify the value that you want to use for one Deployment, but not for others. This way, you can configure the variable a single time, but still have the flexibility to customize variables at the Deployment level.

For example, you might have created a variable that points to a dev token, and then add a value override to specify a staging or production token for your staging and production Deployments to use.

  1. Click Environment in the main menu to open the Airflow Variables page.
  2. Click the variable that you want to add per-Deployment field overrides to.
  3. Click Edit to open the variable configurations for a specific linked Deployment.
  4. Switch the Override value toggle from No Override to Override.
  5. Add the override value.
  6. Click Update Airflow Variable link.

In addition to Deployments, you can link Workspace-level Airflow variables to Astro IDE projects. When you link a variable to an Astro IDE project, the project’s ephemeral test Deployments start with that variable available, without affecting Deployments that aren’t started from the IDE.

For the most flexibility, you can set a default value at the Workspace level and override it per project, similar to how you override values per Deployment.

  1. In the Astro UI, click Environment in the left menu to open the Airflow Variables page.
  2. Click the variable you want to link to an Astro IDE project.
  3. Open the Linked Projects tab.
  4. Click Link Project.
  5. Choose an Astro IDE project from the list.
  6. (Optional) Toggle Override Value and add a value override for this project.
  7. Click Link Airflow Variable.

Configure project sharing for a Workspace

You can configure Astro to link a Workspace-level variable to all Astro IDE projects in the Workspace by default. This is useful when you want every project’s test Deployments to start with the same set of development values.

  1. In the Astro UI, click Environment in the left menu, then click Airflow Variables.
  2. Click the variable that you want to share with all Astro IDE projects.
  3. Click the Edit icon next to Auto-linking in the variable details.
  4. In the Edit Airflow Variable dialog, under Auto-linking, toggle ALL ASTRO IDE PROJECTS to On.
  5. Click Update Airflow Variable.

Variable precedence in Astro IDE test Deployments

When an Astro IDE project starts a test Deployment, the Deployment receives the union of:

  • Variables linked to the project, including Workspace-level variables linked to the project and variables scoped directly to the project.
  • Workspace-level variables linked to that Deployment.

If both the project and a linked Deployment define an Airflow variable with the same key, the project-level value takes precedence in the test Deployment.

Migrate existing Airflow variables to the Environment Manager

If you have Airflow variables defined in a Deployment’s Airflow metadata database, you can move them into the Environment Manager so that they can be reused across Deployments and Astro IDE projects. See Migrate existing objects to the Environment Manager.

Promote a Deployment-scoped Airflow variable to the Workspace

If you created an Airflow variable on a single Deployment and later want to reuse it across other Deployments or Astro IDE projects, you can promote it to the Workspace level from the Deployment’s Environment tab. See Promote a Deployment environment object to the Workspace.