Create Airflow connections in the Astro UI

You can create and manage Airflow connections for Deployments with the Astro Environment Manager in the Astro UI. The Environment Manager uses an Astro-managed secrets backend to store connection configurations as Kubernetes Secrets.

Using the Environment Manager, you can quickly and securely create connections once and share them to multiple Deployments without having to set up your own secrets backend. For example, you can configure a connection with the credentials for a sandbox or development environment. Then, you can later configure your connection to be applied to all Deployments in the workspace by default. This means that when you create new Deployments, they automatically have access to your development environment. Later, you can edit the connection to point to your production resources by using field overrides.

Compared to creating a connection in the Airflow UI, when you create a connection in the Astro UI, you can:

• Share the connection with multiple Deployments within the Workspace.
• Share the connection with Astro IDE projects so that their ephemeral test Deployments can use it.
• Override fields in the connection for individual Deployments or Astro IDE projects.
• Use configured connections in local Airflow environments. See Import and export connections and variables.
• Use connections in branch-based deploys and PR previews.

Workspace Owners and Operators can create and assign connections, while Workspace Authors can view configured connections and use them in Deployments. If your Organization has Environment Secrets Fetching enabled, you can additionally use configured connections, including ones that contain secrets, in local development environments. See Import and export connections and variables.

How connections are stored

When you create an Airflow connection in the Environment Manager, Astro stores Airflow connection details in an Astronomer-hosted secrets manager, and then applies connections to Deployments as Kubernetes Secrets. Specifically the following steps occur:

• Astro stores the connection details in a secure secrets manager hosted by Astronomer.
• When a connection is assigned to a Deployment, Astro uses Airflow’s provided local filesystem secrets backend to mount your connections as Kubernetes Secrets.
• When your Dags use your connections, Airflow reads the connection details from the filesystem using the Airflow local filesystem secrets backend.

This process occurs every time you create or update a connection.

When you use connections for local development, the Astro CLI reads the connections from the Astro API and injects them into the local Airflow instance’s metadata database.

Fetch environment secrets

The Astro CLI can automatically retrieve connections from the Astro UI when you start your local airflow instance with astro dev start --deployment-id=<your-deployment-id>, which means you can use your connection details without needing to manage credentials between local and deployed environments. Local environments fetch connection information the same way as for Deployments, so they require an active internet connection and for you to be logged in with the Astro CLI. You can only fetch environment secrets from Deployments that belong to Workspaces where you are at least a Workspace Member.

Prerequisites

• Workspace Operator or Workspace Owner user permissions
• A Deployment on Astro. See Create a Deployment
• Astro Runtime 9.3.0 or greater

Create a connection

You can create connections both at the Deployment and Workspace level. When you create a connection at the Deployment level, the connection details are available only to that specific Deployment. When you create a connection at the Workspace level, you can apply the connection to several Deployments and override specific fields as needed for each Deployment.

To create a connection at the Workspace level:

1. In the Astro UI, click Environment in the left menu to open the Connections page.

2. Click + Connection to add a new connection.

3. Find the service you want to connect from the list of available options.

   If you don’t see your service in the list, or you don’t see fields that you need in a service that is in the list, you can select the Generic option to specify a connection using Airflow’s standard connection fields.

4. Enter the information for your connection in the listed fields.

5. Click Create Connection.

6. Make your connection accessible to Deployments. See Link connections to Deployments.

To create a connection at the Deployment level:

1. In the Astro UI, select a Deployment, then click the Environment tab within the Deployment menu.

2. Click + Connection to add a new connection.

3. Find the service you want to connect from the list of available options.

   If you don’t see your service in the list, or you don’t see fields that you need in a service that is in the list, you can select the Generic option to specify a connection using Airflow’s standard connection fields.

4. Enter your information in the required fields.

5. Click Create Connection to make your new connection.

After you create a connection, you can reference its Connection ID from Dag code like you would with any Airflow connection created through the Airflow UI.

Link connections to Deployments

After you create a connection at the Workspace level, you can link it to multiple Deployments. Linking connections is useful for standardizing external resource usage across your entire team.

For the most flexibility, you can set default connections and override the connection details per-Deployment based on details like the Deployment’s usage and environment type (production or development).

Step 1: Link the connection

1. In the Astro UI, click Environment in the left menu to open the Connections page.
2. Click the connection you want to link to a Deployment.
3. Click + Link Deployment.
4. Choose a Deployment from the list that appears.
5. (Optional) Click More options and then add any field overrides for this Deployment. For example, if your connection requests access to a development database by default, you can override its details to instead request access to a production database.
6. Click Link connection.

Step 2: (Optional) Add provider packages to your Deployment

Some connection types require installing dependencies on your Deployment through provider packages. If your connection type requires a provider package and the provider package is neither included in Astro Runtime nor included in the requirements.txt file of your Astro project, Airflow won’t be able to use your connection.

If you are uncertain what provider package the connection needs, you can check in the Airflow Registry.

1. Open the local Astro project for your Deployment.
2. Add the required provider package name to your project’s requirements.txt and save your changes.
3. Deploy your project with the new provider to your Deployment.

Configure connection sharing for a Workspace

You can configure Astro to link Workspace-level connections to all Deployments in the Workspace by default.

This is useful, for example, when you need to configure a connection for development environments that all Deployments in a Workspace should start with. Then, when you create new Deployments, they automatically have a default connection to your development resources.

When you’re ready to connect your Deployments to production resources, you can either replace the connection or override the connection field values with your production resource information.

If you change the setting from Restricted to Linked to all Deployments, Astro respects any connection field overrides that you might have configured for existing linked Deployments.

1. Click Environment in the main menu to open the Connections page.
2. Click the connection that you want to add per-Deployment field overrides to.
3. Click Deployment Sharing and toggle the setting to choose either:
   • Restricted: Only share the connection individually to Deployments.
   • Linked to all Deployments: Link to all current and future Deployments in this Workspace.
4. (Optional) Change the default connection field values.
5. Click Update connection to save.

Override connection fields

If you create a connection at the Workspace level and link it to a Deployment, you can later edit the connection within the Deployment to specify field overrides. When you override a field, you specify values that you want to use for a one Deployment, but not for others. This way, you can configure the connection and authentication a single time, but still have the flexibility to customize connection at the Deployment level.

For example, you might have created a connection to a Snowflake account, and then add field overrides to specify the default schemas or databases you want each Deployment to use.

1. Click Environment in the main menu to open the Connections page.
2. Click the connection that you want to add per-Deployment field overrides to.
3. (Optional) Click Deployment Sharing and choose if you want to Restrict or Link to all Deployments. You can also change the default connection field values. Click Update connection to save.
4. Click Edit to open the connection configurations for a specific linked Deployment.
5. Add the override values to the fields you want to edit. You might need to open More options to find the full list of available fields.
6. Click Update connection link.

Link connections to Astro IDE projects

In addition to Deployments, you can link Workspace-level connections to Astro IDE projects. When you link a connection to an Astro IDE project, the project’s ephemeral test Deployments start with that connection available, without affecting Deployments that aren’t started from the IDE.

For the most flexibility, you can set a default connection at the Workspace level and override its fields per project, similar to how you override fields per Deployment.

Link a connection to an Astro IDE project

1. In the Astro UI, click Environment in the left menu, then click Connections.
2. Click the connection you want to link to an Astro IDE project.
3. Click the Linked Projects tab.
4. Click Link Project.
5. Choose an Astro IDE project from the list. Optionally, add field overrides for this project. For example, point the connection to a dedicated test database for the project.
6. Confirm the link.

Configure project sharing for a Workspace

You can configure Astro to link a Workspace-level connection to all Astro IDE projects in the Workspace by default. This is useful when you want every project’s test Deployments to start with the same set of development credentials.

1. In the Astro UI, click Environment in the left menu, then click Connections.
2. Click the connection that you want to share with all Astro IDE projects.
3. Click the Edit icon next to Auto-linking in the connection details.
4. In the Edit Connection dialog, under Auto-linking, toggle ALL ASTRO IDE PROJECTS to On.
5. Click Update Connection.

Connection precedence in Astro IDE test Deployments

When an Astro IDE project starts a test Deployment, the Deployment receives the union of:

• Connections linked to the project, including Workspace-level connections linked to the project and connections scoped directly to the project.
• Workspace-level connections linked to that Deployment.

If both the project and a linked Deployment define a connection with the same connection ID, the project-level value takes precedence in the test Deployment.

Migrate existing connections to the Environment Manager

If you have connections defined in a Deployment’s Airflow metadata database, you can move them into the Environment Manager so that they can be reused across Deployments and Astro IDE projects. See Migrate existing objects to the Environment Manager.

Promote a Deployment-scoped connection to the Workspace

If you created a connection on a single Deployment and later want to reuse it across other Deployments or Astro IDE projects, you can promote it to the Workspace level from the Deployment’s Environment tab. See Promote a Deployment environment object to the Workspace.