Develop DAGsConnections and variables

Manage Airflow connections, variables, and environment variables

Airflow connections are used for storing credentials and other information necessary for connecting to external services. Airflow variables are a generic way to store and retrieve arbitrary content or settings as a simple key value store within Airflow. Environment variables are key-value configurations that can be used to configure Airflow settings, store credentials, or pass configuration to your DAGs.

Use this document to select the right Airflow connection, variable, and environment variable management strategies for your team.

Airflow supports several different methods for managing connections, variables, and environment variables. Each of these strategies has benefits and limitations related to their security and ease of use. The strategies you choose should be compatible with both your local environments and Astro Deployments, allowing you to import and export objects between the two contexts.

For in-depth information on managing connections and variables, see Connection Basics and Variable Basics.

Prerequisites

Choose a connection, variable, and environment variable management strategy

The following table suggests possible management strategies for specific use cases.

ScenarioStrategy
I’m getting started and want to quickly create Airflow objectsAstro Environment Manager
I prefer to manage my Airflow variables in a Git repository and to upload directly to AirflowAirflow UI
I need to keep my connections, variables, and environment variables stored in a centralized and secure location.Secrets backend or Astro Environment Manager
I want to create Connections, Airflow Variables, or environment variables once, and then apply them to multiple Deployments in a Workspace.Astro Environment Manager
I don’t have a secrets backend, but I still want some security and permissions attached to Airflow objects.Astro Environment Manager or Environment variables

How Airflow finds connections

Because connections serve different purposes in Airflow, you might want to use a different strategy for each object type. For example, you can use a secrets backend for connections and use combination of a json files and the Airflow UI for variables.

If you use a mix of strategies for managing connections, it’s important to know which connection value that Airflow gives precedence to in the case of a conflict. The order of precedence for connections is:

  1. Secrets Backend
  2. Astro Environment Manager
  3. Environment Variables
  4. Airflow’s metadata database (Airflow UI)

Airflow checks for connections in order from highest (secrets backend) to lowest (Airflow UI) precedence. Airflow uses the first configuration for a given connection ID that it finds.

For example, if you store a connection with the same connection ID in both a secrets backend and the Airflow UI, Airflow reads and uses the connection configured in the secrets backend first, instead of using a connection configuration with the same connection ID stored in the Airflow UI.

If you only want to test connections or export connections in a JSON or URI format, use the Airflow UI to manage your connection. You can then use the Astro CLI commands to export the connections in a URI or JSON format. See Import and export connections and variables.

Compare strategies

The following sections explain the benefits, limitations, and implementations of each strategy in more detail.

Astro Environment Manager

Astro includes a connection, Airflow variable, and environment variable management system that behaves like you are using an Astro-managed secrets backend. After you create a connection, variable, or environment variable in the Astro UI, you can share it with multiple Deployments in a Workspace, override values on a per-Deployment basis, or hide secret values after creating them. See Create connections in Astro, Create variables in Astro, or Create environment variables in Astro for setup steps.

For Remote Execution Deployments, you can only manage environment variables and Metrics export with the Environment Manager. You cannot create, update, or assign Connections or Airflow Variables.

Benefits

  • It securely stores your connection, variable, and environment variable configuration and authentication information on Astro, without a secrets backend.
  • You can create a configuration once and then add it to multiple Deployments in a Workspace.
  • Authenticate connections using credentials other than what the Airflow UI supports.
  • Hide the values for Airflow Variable keys and environment variables in the Astro UI, after you finish creating them.
  • After you configure a connection, variable, or environment variable, the authentication information is securely stored in an Astro-managed secrets backend.
  • Control whether connections, Airflow variables, or environment variables can be used in a single Deployment or shared across Deployments in a Workspace.
  • Override certain fields in connections, variable values, or environment variable values per Deployment, so you can create a general configuration and then customize its behavior.

Limitations

  • If you create a connection in the Astro UI, you also need to add its related provider package to the requirements.txt file in your Astro project.
  • Only available with Astro Runtime 9.3.0 and greater.
  • You can’t see connections, Airflow variables, or environment variables defined in the Astro UI in the Airflow UI.
  • You need WORKSPACE_OPERATOR or WORKSPACE_OWNER user permissions.
  • You can’t programmatically import connections, Airflow variables, or environment variables to the Environment Manager from your local environment.
To see how you can use connections set in the Astro Environment Manager in a best practice, branch-based Deployment setup, see the Manage Astro connections in branch-based deploy workflows use case.

Airflow UI

You can create Airflow connections and variables is through the Airflow UI. This experience is the same for both local Airflow environments and Astro Deployments. Astronomer recommends this method if you’re just getting started with Airflow, you want to get your dags running quickly, or if you want to export connections in a URI/JSON format.

Benefits

  • The UI has features for correctly formatting and testing your connections.
  • It’s easy to change variables or connections to test different use cases on the fly.
  • You can export and import your variables from the Airflow UI using JSON files and the Astro CLI. See Import and export connections and variables.
  • Connections and variables are encrypted and stored in the Airflow metadata database.

Limitations

  • You cannot export or import connections from the UI for security reasons.
  • Managing many connections or variables can become unwieldy.
  • In a local environment, you lose your connections and variables if you delete your metadata database with astro dev kill.

Secrets backend

A secrets backend is the most secure way to store connections and variables. You can access a secrets backend both locally and on Astro by configuring the appropriate credentials in your Airflow environment. Astronomer recommends this method for all staging and production deployments. See the following documentation for setup steps:

Benefits

  • Store objects in a centralized location alongside other secrets used by your organization.
  • Comply with internal security postures and policies that protect your organization.
  • Recover objects if your Airflow environments go down.
  • Share secrets across different Airflow environments.
  • Allow selective access to connections and variables by using connections_prefix and variables_prefix.
  • Limit the number of open connections to your metadata database, especially if you are using your connections and variables outside of task definitions.

Limitations

  • A third-party secrets manager is required.
  • Separate configurations might be required for using a secrets backend locally and on Astro.
  • You cannot use the Airflow UI to view connections and variables.
  • You are responsible for ensuring that secrets are encrypted.

Environment variables

You can use Airflow’s system-level environment variables to store connections and variables, or to configure general Airflow settings and pass configuration to your DAGs. There are two ways to manage environment variables on Astro:

  • Workspace-level environment variables (Astro Environment Manager): Create environment variables at the Workspace level and share them across multiple Deployments. This is the recommended approach when you want to standardize environment variables across Deployments. See Create environment variables in Astro.
  • Deployment-level environment variables: Set environment variables specific to a single Deployment through the Deployment UI or your Dockerfile. This is recommended when you don’t have a secrets backend, but you still want to take advantage of security and RBAC features. See Manage environment variables on Astro.

Airflow connections and variables are stored in the Airflow metadata database. Calling them outside of task definitions and operators requires an additional connection to the Airflow metadata database which is used every time the scheduler parses a dag. By adding connections and variables as environment variables, you can lower the amount of open connections and improve the performance of your database and resources.

Benefits

  • If you use an .env file for your local Airflow environment and your local metadata database is corrupted or accidentally deleted, you still have access to all of your connections and variables.
  • You can export environment variables from a local Airflow environment to Astro using the Astro CLI. See Import and export connections and variables.
  • You can override Airflow variables set in the Airflow UI. See Environment variable priority
  • You can create your environment variables from the Astro UI at both the Workspace and Deployment level. See Create environment variables in Astro or Manage environment variables on Astro.
  • Environment variables marked as Secret are encrypted in the Astronomer control plane. See How environment variables are stored on Astro for details.
  • This approach limits the number of open connections to your metadata database, especially if you are using your connections and variables outside of task definitions.
  • Workspace-level environment variables can be shared across multiple Deployments with override capabilities per Deployment.

Limitations

  • You can’t view connections and variables from the Airflow UI when they are stored as environment variables.
  • You must restart your local environment using astro dev restart whenever you make changes to your .env file.
  • When stored in your .env file or Dockerfile, environment variables are defined in plain text.
  • Connections must be formatted as either a URI or serialized JSON.
  • Deployment-level environment variables are not as secure or centralized compared to a secrets backend or the Workspace Environment Manager.
  • You cannot directly export Deployment-level environment variables from the Astro UI to a local Airflow environment. See Import and export Airflow objects.

Other strategies

While it’s possible to manage Airflow connections and variables with these strategies, Astronomer doesn’t recommend them at scale:

  • You can use the Airflow REST API to programmatically create Airflow connections and variables for a Deployment. Airflow objects created with the API are stored in the Airflow metadata database and visible in the Airflow UI.
  • For local Astro projects, you can use airflow_settings.yaml for defining your connections and variables. See Configure airflow_settings.yaml for more details.

See also