# Using the Airflow REST API with Astro
Source: https://astronomer.io/docs/astro/airflow-api

Use the Airflow REST API in Astro to trigger Dag runs, list Dags, manage task instances, and automate workflows programmatically. Includes authentication setup and code examples.

For Deployments on Astro, you can use the Airflow [REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) to automate Airflow workflows. For example, you can externally trigger a dag run without accessing your Deployment directly by making an HTTP request in Python or cURL to the [dagRuns endpoint](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/post_dag_run) in the Airflow REST API.

To test Airflow API calls in a local Airflow environment running with the Astro CLI, see [Troubleshoot your local Airflow environment](/docs/cli/v1.43/run-airflow-locally).

<Info>Updates to the Airflow REST API are released in new Airflow versions and new releases don’t have a separate release cycle or versioning scheme. To take advantage of specific Airflow REST API functionality, you might need to upgrade Astro Runtime. See [Upgrade Runtime](/docs/runtime/upgrade-astro-runtime) and the [Airflow release notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html).</Info>

<Info>
  **Airflow REST API v2 (Airflow 3.0+)**

  The Airflow REST API is available at `/api/v2` for Airflow 3.0 and above. Some endpoints and parameters have changed. See the [Airflow API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) documentation for more information.
</Info>

## Prerequisites

* A Deployment on Astro.
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or an [Organization API token](/docs/astro/organization-api-tokens).
* [cURL](https://curl.se/) or, if using Python, the [Requests library](https://docs.python-requests.org/en/latest/index.html).
* The [Astro CLI](/docs/cli/v1.43/overview).

## Step 1: Retrieve your access token

<Tabs>
  <Tab title="Workspace token">
    Follow the steps in [Create a Workspace API token](/docs/astro/workspace-api-tokens#create-a-workspace-api-token) to create your token. Make sure to save the token on creation in order to use it later in this setup.
  </Tab>

  <Tab title="Organization token">
    Follow the steps in [Create a Organization API token](/docs/astro/organization-api-tokens#create-an-organization-api-token) to create your token. Make sure to save the token on creation in order to use it later in this setup.
  </Tab>

  <Tab title="Deployment API token">
    Follow the steps in [Create a Deployment API token](/docs/astro/deployment-api-tokens#create-a-deployment-api-token) to create your token. Make sure to save the token on creation in order to use it later in this setup.
  </Tab>
</Tabs>

## Step 2: Retrieve the Deployment URL

Your Deployment URL is the [host](https://swagger.io/docs/specification/2-0/api-host-and-base-path/) you use to call the Airflow API.

1. Run the following command to retrieve the URL for your Deployment Airflow UI:

   ```sh wrap theme={null}
   astro deployment inspect -n <deployment-name> --key metadata.airflow_api_url
   ```

Alternatively, you can retrieve your Deployment URL by opening the Airflow UI for your Deployment on Astro and copying the URL of the page up to `/home`. For example, if the home page of your Deployment Airflow UI is hosted at `clq52c95r000208i8c7wahwxt.astronomer.run/dz3uu847/home`, your Deployment URL is `clq52c95r000208i8c7wahwxt.astronomer.run/dz3uu847`.

## Step 3: Make an Airflow API request

You can execute requests against any endpoint that is listed in the [Airflow REST API reference](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html).

To make a request based on Airflow documentation, make sure to:

* Use the Astro access token from Step 1 for authentication.
* Replace `airflow.apache.org` with your Deployment URL from Step 1.

<Info>The Airflow REST API doesn't have rate-limiting.</Info>

## Example API Requests

The following are common examples of Airflow REST API requests that you can run against a Deployment on Astro.

### List Dags

To retrieve a list of all Dags in a Deployment, you can run a `GET` request to the [`dags` endpoint](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_dags)

#### cURL

```sh wrap theme={null}
curl -X GET https://<your-deployment-url>/api/v2/dags \
   -H 'Authorization: Bearer <your-access-token>'
```

#### Python

```python wrap theme={null}
import requests
token = "<your-access-token>"
deployment_url = "<your-deployment-url>"
response = requests.get(
   url=f"https://{deployment_url}/api/v2/dags",
   headers={"Authorization": f"Bearer {token}"}
)
print(response.json())
# Prints data about all dags in your Deployment
```

### Trigger a Dag run

You can trigger a Dag run by executing a `POST` request to Airflow's [`dagRuns` endpoint](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/post_dag_run).

This will trigger a Dag run for the Dag you specify, which is equivalent to clicking the **Play** button in the main **Dags** view of the Airflow UI. The request body must include the `logical_date` key — it can be `null`, which runs the Dag immediately, but it can't be omitted. An empty body `{}` returns a `422 Unprocessable Entity` error.

#### cURL

```sh wrap theme={null}
curl -X POST https://<your-deployment-url>/api/v2/dags/<your-dag-id>/dagRuns \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer <your-access-token>' \
   -d '{"logical_date": null}'
```

#### Python

```python wrap theme={null}
import requests
token = "<your-access-token>"
deployment_url = "<your-deployment-url>"
dag_id = "<your-dag-id>"
response = requests.post(
    url=f"https://{deployment_url}/api/v2/dags/{dag_id}/dagRuns",
    headers={
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json"
    },
    data='{"logical_date": null}'
)
print(response.json())
# Prints metadata of the dag run that was just triggered
```

### Trigger a Dag run by date

You can also specify a `logical_date` at the time in which you wish to trigger the Dag run by passing the `logical_date` with the desired timestamp with the request's `data` field. The timestamp string is expressed in UTC and must be specified in the format `"YYYY-MM-DDTHH:MM:SSZ"`, where:

* `YYYY` represents the year.
* `MM` represents the month.
* `DD` represents the day.
* `HH` represents the hour.
* `MM` represents the minute.
* `SS` represents the second.
* `Z` stands for "Zulu" time, which represents UTC.

#### cURL

```sh wrap theme={null}

curl -v -X POST https://<your-deployment-url>/api/v2/dags/<your-dag-id>/dagRuns \
   -H 'Authorization: Bearer <your-access-token>' \
   -H 'content-type: application/json' \
   -d '{"logical_date":"2022-11-16T11:34:00Z"}'
```

#### Python

Using Python:

```python wrap theme={null}
import requests
token = "<your-access-token>"
deployment_url = "<your-deployment-url>"
dag_id = "<your-dag-id>"
response = requests.post(
    url=f"https://{deployment_url}/api/v2/dags/{dag_id}/dagRuns",
    headers={
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json"
    },
    data='{"logical_date": "2021-11-16T11:34:01Z"}'
)
print(response.json())
# Prints metadata of the dag run that was just triggered
```

### Pause a Dag

You can pause a Dag by executing a `PATCH` command against the [`dag` endpoint](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/patch_dag).

Replace `<your-dag-id>` with your own value.

#### cURL

```sh wrap theme={null}
curl -X PATCH https://<your-deployment-url>/api/v2/dags/<your-dag-id> \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer <your-access-token>' \
   -d '{"is_paused": true}'
```

#### Python

```python wrap theme={null}
import requests
token = "<your-access-token>"
deployment_url = "<your-deployment-url>"
dag_id = "<your-dag-id>"
response = requests.patch(
    url=f"https://{deployment_url}/api/v2/dags/{dag_id}",
    headers={
        "Authorization": f"Bearer {token}",
        "Content-Type": "application/json"
    },
    data='{"is_paused": true}'
)
print(response.json())
# Prints data about the dag with id <dag-id>
```

### Trigger Dag runs across Deployments

You can use the Airflow REST API to make a request in one Deployment that triggers a Dag run in a different Deployment. This is sometimes necessary when you have interdependent workflows across multiple Deployments. On Astro, you can do this for any Deployment in any Workspace or cluster.

This topic has guidelines on how to trigger a Dag run, but you can modify the example Dag provided to trigger any request that's supported in the Airflow REST API.

1. Create a [Deployment API token](/docs/astro/deployment-api-tokens) for the Deployment that contains the Dag you want to trigger.

2. In the Deployment that contains the triggering Dag, create an [Airflow HTTP connection](https://airflow.apache.org/docs/apache-airflow-providers-http/stable/connections/http.html) with the following values:

   * **Connection Id**: `http_conn`
   * **Connection Type**: HTTP
   * **Host**: `<your-deployment-url>`
   * **Schema**: `https`
   * **Extra**:

     ```json wrap theme={null}
     {
        "Content-Type": "application/json",
        "Authorization": "Bearer <your-deployment-api-token>"
     }
     ```

     See [Manage connections in Apache Airflow](/docs/learn/connections).

<Info>If the `HTTP` connection type is not available, double check that the [HTTP provider](https://airflow.apache.org/registry/providers/http/) is installed in your Airflow environment. If it's not, add `apache-airflow-providers-http` to the `requirements.txt` file of our Astro project and redeploy it to Astro.</Info>

3. In your triggering Dag, add the following task. It uses the [`HttpOperator`](https://airflow.apache.org/registry/providers/http#http-http-HttpOperator) to make a request to the `dagRuns` endpoint of the Deployment that contains the Dag to trigger.

   ```python wrap theme={null}
   from datetime import datetime

   from airflow.models.dag import DAG
   from airflow.providers.http.operators.http import HttpOperator

   with DAG(
       dag_id="triggering_dag",
       start_date=datetime(2024, 1, 1),
       schedule=None,
   ):
       HttpOperator(
           task_id="trigger_external_dag",
           log_response=True,
           method="POST",
           endpoint="api/v2/dags/<triggered-dag>/dagRuns",
           http_conn_id="http_conn",
           data={
               "logical_date": "{{ logical_date }}",
               # To pass parameters, add: "params": {"foo": "bar"}
           },
       )
   ```


# Configure API server autoscaling
Source: https://astronomer.io/docs/astro/api-server-autoscaling

Enable horizontal autoscaling for the Airflow API server to handle high task concurrency and bursty UI or REST API traffic.

<Note>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Note>

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

The Airflow API server serves the Airflow UI, the Airflow REST API, and the task execution API that Astro and Celery workers use to fetch and report on tasks. By default, every Airflow 3 Deployment runs with two API server replicas. For workloads with high task concurrency, large Dags, or spikes in UI and REST API traffic, you can enable horizontal autoscaling so that Astro adds API server replicas when load is high and removes them when load decreases.

Use API server autoscaling to:

* Sustain workloads with thousands of concurrent tasks without saturating a fixed pair of API server replicas.
* Support smooth Airflow 2 to Airflow 3 upgrades for environments that pushed the previous webserver to its limits.
* Cap your spend by setting an explicit maximum replica count.

## How autoscaling works

When you enable autoscaling, Astro provisions a [Kubernetes Horizontal Pod Autoscaler (HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/) that tracks CPU utilization across the running API server replicas. The HPA adds or removes replicas to keep the average CPU utilization at 80 percent of the per-replica CPU request.

* When average CPU utilization is consistently above the 80 percent target, Astro adds replicas, up to your configured **API Server Max Replicas** value.
* When average CPU utilization is below the target, Astro removes replicas, down to the minimum of two.
* The minimum replica count is fixed at two. This preserves API availability during a Pod restart and is the same as the default replica count when autoscaling is disabled.
* The maximum replica count is configurable from `2` to `10` in the Astro UI. Setting **API Server Max Replicas** to `2` runs a fixed two-replica configuration and effectively prevents scaling.

The HPA evaluates utilization continuously, and replica changes typically take effect within one to two minutes. The same high-availability and anti-affinity rules that apply to other Deployment components apply to API server replicas. See [Enable high availability](/docs/astro/deployment-resources#enable-high-availability).

## Expected replica counts

The following table shows the approximate API server replica counts you can expect at different levels of task concurrency. Actual counts can vary based on Airflow UI and REST API traffic, Dag complexity, and other workload characteristics.

| Concurrent tasks | API server replicas |
| ---------------- | ------------------- |
| \~500            | 3                   |
| \~1,000          | 4                   |
| \~2,000          | 6                   |
| \~3,000          | 7                   |

## Prerequisites

API server autoscaling is supported on Airflow 3 Deployments running on Astro:

* Standard or dedicated clusters.
* Remote Execution Deployments.

API server autoscaling works with all executors: Astro, Celery, and Kubernetes.

API server autoscaling is not available for Airflow 2 Deployments. Airflow 2 uses the Airflow webserver, which doesn't support horizontal autoscaling on Astro.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Enable or disable API server autoscaling

<Tabs>
  <Tab title="New Astro UI">
    <Steps>
      <Step title="Open the Deployment">
        In the Astro UI, click **Deployments**, then select a Deployment.
      </Step>

      <Step title="Edit the Deployment">
        Click the Deployment's **More actions** menu (⋯), then select **Edit Deployment**.
      </Step>

      <Step title="Toggle API server autoscaling">
        In the **Advanced** section, set the **API Server Autoscaling** toggle to on or off.

        When enabling autoscaling, the **API Server Max Replicas** list becomes editable. Select a value from `2` to `10`. The default is `10`, which gives Astro the largest possible scaling headroom. Choose a lower value to cap your replica count at a known maximum.

        When disabling autoscaling, Astro preserves the previous **API Server Max Replicas** value, so you can re-enable autoscaling later with the same configuration.
      </Step>

      <Step title="Save the configuration">
        Click **Update Deployment**. When you enable autoscaling, Astro applies the new HPA without restarting the running API server replicas. When you disable autoscaling, Astro removes the HPA and the Deployment runs with two API server replicas.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Legacy UI">
    <Steps>
      <Step title="Open the Deployment">
        In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
      </Step>

      <Step title="Edit the Deployment">
        Click the **Options** menu of the Deployment, and then select **Edit Deployment**.
      </Step>

      <Step title="Toggle API server autoscaling">
        In the **Advanced** section, set the **API Server Autoscaling** toggle to on or off.

        When enabling autoscaling, the **API Server Max Replicas** list becomes editable. Select a value from `2` to `10`. The default is `10`, which gives Astro the largest possible scaling headroom. Choose a lower value to cap your replica count at a known maximum.

        When disabling autoscaling, Astro preserves the previous **API Server Max Replicas** value, so you can re-enable autoscaling later with the same configuration.
      </Step>

      <Step title="Save the configuration">
        Click **Update Deployment**. When you enable autoscaling, Astro applies the new HPA without restarting the running API server replicas. When you disable autoscaling, Astro removes the HPA and the Deployment runs with two API server replicas.
      </Step>
    </Steps>
  </Tab>
</Tabs>

After you enable autoscaling, the Deployment **Details** page shows the configured range under **API Server Autoscaling**, for example `Enabled (2-10 replicas)`.

## Billing

API server replicas are sized the same as an A5 worker. Each Deployment includes two API server replicas at no additional cost. When autoscaling adds replicas above the included two, Astro charges for the additional replicas based on their uptime duration at the A5 unit rate, similar to [KE/KPO chargeback](https://www.astronomer.io/pricing/#:~:text=How%20will%20I%20be%20charged%20for%20the%20Kubernetes%20Executor%20and%20Kubernetes%20Pod%20Operator%3F). The charges appear on your invoice under **Runtime Compute** as **API Server**.

If autoscaling is enabled but the Deployment never exceeds two replicas, no additional API server charges apply.

## Limitations

* The minimum replica count is fixed at two. You can't scale the API server below two replicas.
* The maximum replica count is capped at ten in the Astro UI. If your workload requires more than ten replicas, Astronomer recommends moving some workloads to another Deployment.


# About Astro
Source: https://astronomer.io/docs/astro/astro-architecture

Learn about how Astro is structured to maximize the power of Apache Airflow.

Astro is a fully-managed SaaS application for data orchestration that helps teams write and run data pipelines with Apache Airflow at any level of scale. The infrastructure to run Airflow is managed entirely by Astronomer, enabling you to shift your focus from infrastructure to using data to grow your business.

This document provides an overview of the architecture and key concepts in Astro that enable your team to make the most of your data pipelines. See [Astro Features](/docs/astro/features) to learn more about Astro functionality. To get started with Astro, see [Start a trial](/docs/astro/trial).

## Architecture

Astro, also known as Astro Hosted, both simplifies and optimizes Airflow so you can customize the parts of your environment that matter most to you, with Astro taking care of the rest. Whether you use Astro to power internal analytics or to train machine learning models, learning how Astro works can help you decide which deployment and connectivity model works best for you and enables your team to start running Dags.

You can work with Astro using the [Astro UI](#astro-ui) in your web browser, the [Astro CLI](#astro-cli), or the Astro API. Astro makes it easy for Airflow environments, called [Deployments](#deployment), to securely connect to external data services.

The *Astro Hypervisor* is an Astronomer-managed component of the Astro platform that scales and optimizes your Deployments and the [clusters](#cluster) they're hosted on.

The following diagram shows how these components work together to help you manage Airflow on Astro.

<Frame>
  <img alt="Astro hypervisor architecture overview" />
</Frame>

## Key concepts

The following sections give a detailed explanation of key concepts that you need to understand in order to work with Astro. You can find definitions for additional Astro-specific terminology in the [Astro glossary](/docs/astro/astro-glossary).

### Astro CLI

The [Astro CLI](/docs/cli/v1.43/overview) is an open source interface you can use to test Airflow Dags locally, deploy code to Astro, and automate key Astro actions as part of a CI/CD process. Using the Astro CLI to run Airflow locally requires Docker or an alternative container management tool, like Podman.

An Airflow project created with the Astro CLI is also known as an *Astro project*. It contains the set of files necessary to run Airflow, including dedicated folders for your Dag files, Python packages, utility files, and more. Astronomer recommends that you create a dedicated Git repository for each Astro project. To run a Dag, you add the Dag to your Astro project and deploy your Astro project to Astro.

See [Run your first Dag with the Astro CLI](/docs/astro/first-dag-cli) to create your first Astro project.

### Astro UI

The Astro UI, hosted at `https://cloud.astronomer.io`, is the primary interface for accessing and managing Astro from your web browser. You can use the Astro UI to:

* Manage users, teams, and permissions.
* Create and configure Deployments, including infrastructure resources and compute.
* View all of your organization's Deployments, Dags, and tasks in a single place.
* Monitor the health of your Airflow environments with a variety of alerts, logs, and analytics interfaces.
* Stay up to date with the latest Astro features.

When you open the Astro UI, you land on the **Recently Failed Dags** homepage, which lists Dags with failed runs in the selected time window. Use the **1D**, **1W**, and **1M** filters to set the time window, and the Workspace filter to scope the list to a specific Workspace. The list includes any Dag that started up to seven days before the start of the selected time window, not only Dags that started within the window. For example, with the **1M** filter, the list includes Dags that started as early as one month and seven days ago.

For each Dag, the table shows its run history, failure rate, and last run time. Click **Investigate** to start an [Otto investigation](/docs/astro/otto-investigate) of the most recent failed run, or click a Dag row to open its [Asset Catalog](/docs/astro/assets-overview) details page.

### Deployment

An Astro *Deployment* is an Airflow environment hosted on Astro. It encompasses all core Airflow components, including the Airflow webserver, scheduler, and workers, along with additional tools for reliability and observability. It runs in an isolated Kubernetes namespace in an [Astro cluster](#cluster) and has a set of attached resources to run your Airflow tasks.

Compared to an open source Airflow environment, an Astro Deployment is easy to create, delete, and modify through either the Astro UI or with the Astro CLI. You can [fine-tune resources and settings](/docs/astro/deployment-settings) directly from the Astro UI, see metrics and analytics for your Dags, review your deploy history, and more. The infrastructure required to run a Deployment is managed by Astronomer.

To run Dags in a Deployment, you must either deploy an Astro project manually from your local machine or configure an automated deploy process using a third-party CI/CD tool with the Astro CLI. Then, you can open the Airflow UI from the Astro UI and view your Dags. See [Run your first Dag](/docs/astro/run-first-dag) to get started with examples of either workflow.

### Astro Runtime

*Astro Runtime* is a [debian-based Docker image](https://quay.io/repository/astronomer/astro-runtime) that bundles Apache Airflow with optimized configurations and add-ons that make your Airflow experience reliable, fast, and scalable. Astronomer releases an Astro Runtime distribution for each version of Apache airflow.

Every Deployment and Astro project uses Astro Runtime at its core. Astronomer provides [extended support and bug fixes](/docs/runtime/runtime-version-lifecycle-policy) to Astro Runtime versions, so that you can keep your Dags running for longer without disruption.

See [Astro Runtime Architecture and features](/docs/runtime/runtime-image-architecture) for a complete feature list.

### Workspace

A *Workspace* is a collection of Deployments that can be accessed by a specific group of users. You can use a Workspace to group Deployments that share a business use case or environment trait. For example, your data science team might have a dedicated Workspace with two Deployments within it. Workspaces don't require any resources to run and are only an abstraction for grouping Deployments and configuring user access to them. All Deployments must belong to a Workspace.

You can assign new users [Workspace roles](/docs/astro/user-permissions#workspace-roles) that include varying levels of access to your Deployments.

### Cluster

A *cluster* in Astro is a Kubernetes cluster that hosts the infrastructure required to run your Airflow environments, also known as [Deployments](#deployment) in Astro. There are two types of clusters in Astro:

* A *standard cluster* is a multi-tenant cluster that's pre-configured by Astronomer. It's the default cluster type and the quickest way to get an Airflow environment up and running on Astro. Each Deployment in a standard cluster exists in its own isolated Kubernetes namespace. To run a Deployment in a standard cluster, you select a cloud provider and region when you create the Deployment. Then, Astro automatically creates your Deployment in an existing standard cluster based on your configuration. See [Standard cluster configurations](/docs/astro/resource-reference-hosted#standard-cluster-regions) for a list of all cloud providers and regions where you can use standard clusters.

* A *dedicated cluster* is a single-tenant cluster that's used exclusively by Organizations with additional security and networking requirements. Compared to standard clusters, dedicated clusters provide more configuration options for [cloud providers and regions](/docs/astro/resource-reference-hosted#dedicated-cluster-regions), as well as private network connectivity and [security](/docs/astro/authorize-workspaces-to-a-cluster). Note that due to expanded resource usage, dedicated clusters cost more than standard clusters. See [Create a Dedicated cluster](/docs/astro/create-dedicated-cluster) to get started with dedicated clusters.

For both cluster types, Astro manages all underlying infrastructure and provides secure connectivity options to all data services in your ecosystem.

### Organization

An Astro *Organization* is the highest level entity in Astro and represents a shared space for your company on Astro. An Organization is automatically created when you first sign up for Astronomer. At the Organization level, you can manage all of your users, Deployments, Workspaces, and clusters from a single place in the Astro UI.

To securely manage user access, you can [integrate your Organization with an identity provider (IdP)](/docs/astro/configure-idp) and [set up SCIM provisioning](/docs/astro/set-up-scim-provisioning) to have new users automatically join Astro with the correct permissions.

### Execution mode

Astro supports two [execution modes](/docs/astro/execution-mode) that give you flexibility in how and where your Airflow tasks run: **Hosted** and **Remote** execution. Both modes use the same orchestration layer, so you can choose the right execution model based on your infrastructure, security requirements, and performance needs.

* *Hosted execution* is the default mode for Deployments on Astro. In this mode, Astronomer manages all infrastructure for task execution, including worker autoscaling, Dag versioning, and compute provisioning. Hosted execution is well suited for getting started quickly without managing your own execution environment.

* *Remote execution* mode separates orchestration from execution, allowing Airflow tasks to run entirely in your own infrastructure whether it's on-premises or in the cloud, while keeping data, code, and secrets local. The Astro Orchestration Plane handles scheduling, coordination, and observability, while Remote Execution Agents in your environment poll for tasks through a secure Remote Execution API and run them locally.

Each Remote Execution Agent includes a Dag processor, Triggerer, and Worker, which enables Dag parsing, task execution, secret management, and logging within your environment. This model supports custom infrastructure and security requirements while maintaining centralized control and visibility through Astro.

### Astro Observe

Astro Observe provides a fully integrated observability layer in Astro that gives you complete visibility into the health, performance, and reliability of your data pipelines and the data products they power. Unlike standalone tools, Astro Observe is purpose-built for Apache Airflow and deeply connected to the orchestration layer, making it easy to monitor your pipelines with full execution context and minimal setup.

With Astro Observe, you can:

* Monitor Dag execution with real-time task-level insights and pipeline metadata.

* Visualize upstream and downstream dependencies using automatic lineage powered by OpenLineage.

* Set SLAs and alerts on key data products to catch issues before they cause downstream failures.

* Investigate and debug problems quickly with AI-generated log summaries and consolidated pipeline views.

* Track pipeline and asset-level cost metrics (coming soon) to optimize your data operations.

Astro Observe is natively integrated into the Astro platform and can be enabled to provide immediate visibility. As soon as your Dags run, you get access to task logs, lineage graphs, and pipeline health insights with no extra configuration or infrastructure required. This helps your team move quickly, resolve issues efficiently, and deliver trusted data products with confidence.

## Access control architecture

Astro uses role-based access control (RBAC) to define which users are permitted to take certain actions or access certain resources. For example, you can assign a user or automation tool permission to deploy Dags, but not to delete a Deployment. Roles in Astro are defined at the Workspace and Organization levels.

Each Astro user has a Workspace role in every Workspace they belong to, plus a single Organization role. Users can also belong to [Teams](/docs/astro/manage-teams), which apply the same role across a group of users. To automate managing user roles or deploying code, you can create API tokens with specific roles to automate most actions on Astro.

Use the following diagram as a reference for how these components interact with each other in Astro.

<Frame>
  <img alt="A diagram showing how all Astro RBAC components fit together" />
</Frame>


# MCP servers
Source: https://astronomer.io/docs/astro/astro-mcp-server

Connect AI tools to Astro and Airflow using Model Context Protocol (MCP) servers

<Note>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Note>

Astro provides experimental Model Context Protocol (MCP) servers that allow AI models and agents to securely access your Astro and Airflow resources. Connect to MCP servers natively or by using the `mcp-remote` module in compatible AI clients.

For enhanced agent capabilities such as Dag authoring, testing, and debugging, see the [Astronomer AI Agent tooling repository](https://github.com/astronomer/agents).

## MCP servers

Astro provides the following experimental MCP servers:

* **Astro Registry MCP server** (`astro-registry`) is a remote MCP server for accessing the Astro Registry.
* **Astro Cloud MCP server** (`astro-cloud`) is an authenticated remote MCP server for discovering and managing your Astro resources, including Deployments, Workspaces, and environment variables.

<Warning>
  **Astro Cloud MCP server deprecation**

  The current Astro Cloud MCP server (`astro-cloud`) is deprecated and shouldn't be used. An updated Astro MCP server is in progress and is slated for early access in July 2026. For more information, contact your Astronomer account executive.
</Warning>

## Set up the Astro Registry MCP server

<Tip>If you experience any issues, try restarting your client or disabling and re-enabling the Astro MCP server connection.</Tip>

<Tabs>
  <Tab title="Windsurf">
    1. Open Windsurf settings with `CTRL/CMD + ,`.

    2. Navigate to `Cascade` > `Manage plugins`.

    3. Select `View raw config`.

    4. Add the following configuration. Only include the `astro-cloud` section if you're using the authenticated Astro Cloud MCP server.

       ```json wrap theme={null}
       {
         "mcpServers": {
           "astro-registry": {
             "command": "npx",
             "args": [
               "-y",
               "mcp-remote",
               "https://api.astronomer.io/registryV2/v1alpha1/mcp"
             ]
           },
           "astro-cloud": {
             "command": "npx",
             "args": [
               "-y",
               "mcp-remote",
               "https://api.astronomer.io/v1alpha1/mcp?organizationId=<my-organization-id>",
               "--header",
               "Authorization: Bearer ${AUTH_TOKEN}"
             ],
             "env": {
               "AUTH_TOKEN": "<my-auth-token>"
             }
           }
         }
       }
       ```

    5. Click `Save` to save your configuration and check the `Plugins` section to verify that your server is connected.
  </Tab>

  <Tab title="Cursor">
    1. Open Cursor.

    2. Navigate to `Tools & Integrations` > `Add Custom MCP`.

    3. Add the following configuration. Only include the `astro-cloud` section if you're using the authenticated Astro Cloud MCP server.

       ```json wrap theme={null}
       {
         "mcpServers": {
           "astro-registry": {
             "url": "https://api.astronomer.io/registryV2/v1alpha1/mcp"
           },
           "astro-cloud": {
             "url": "https://api.astronomer.io/v1alpha1/mcp?organizationId=<my-organization-id>",
             "headers": {
               "Authorization":"Bearer <my-auth-token>"
             }
           }
         }
       }
       ```

    4. Save the configuration and check the `Tools & Integrations` > `MCP Servers` section to verify that your server is connected.
  </Tab>

  <Tab title="Claude Code">
    Claude Code connects to remote MCP servers natively without the `mcp-remote` module. You can configure the server at the project level (`.mcp.json`) or at the user level (`~/.claude.json`).

    The Astro Registry MCP server is public and requires no authentication. Run the following command to add it to your project:

    ```sh wrap theme={null}
    claude mcp add --transport http --scope project astro-registry \
      https://api.astronomer.io/registryV2/v1alpha1/mcp
    ```

    #### Add the Astro Cloud MCP server

    The Astro Cloud MCP server requires your Organization ID and an API token.

    <Steps>
      <Step title="Retrieve your Organization ID">
        If you have the Astro CLI installed and authenticated, run `astro organization list` to find your Organization ID. Otherwise, open the [Astro UI](https://cloud.astronomer.io/settings/general) and copy your Organization ID from **Settings** > **General**.
      </Step>

      <Step title="Create an API token">
        Create a [Workspace or Organization API token](/docs/astro/workspace-api-tokens) in the Astro UI.
      </Step>

      <Step title="Set the token as an environment variable">
        Add the following to your shell profile (`.zshrc`, `.bashrc`, or equivalent) so it persists across sessions:

        ```bash wrap theme={null}
        export ASTRO_AUTH_TOKEN=<your-token>
        ```
      </Step>

      <Step title="Add the server">
        Run the following command, replacing `<your-organization-id>` with the Organization ID you retrieved earlier:

        ```bash wrap theme={null}
        claude mcp add --transport http --scope project astro-cloud \
          "https://api.astronomer.io/v1alpha1/mcp?organizationId=<your-organization-id>" \
          --header "Authorization: Bearer ${ASTRO_AUTH_TOKEN}"
        ```
      </Step>

      <Step title="Verify the connection">
        Run `claude mcp list` and confirm both `astro-registry` and `astro-cloud` appear.
      </Step>
    </Steps>

    The resulting `.mcp.json` file should look similar to the following:

    ```json wrap theme={null}
    {
      "mcpServers": {
        "astro-registry": {
          "type": "http",
          "url": "https://api.astronomer.io/registryV2/v1alpha1/mcp"
        }
      }
    }
    ```
  </Tab>
</Tabs>

## Airflow MCP Plugin

The **Airflow MCP Plugin** ([`astro-airflow-mcp`](https://github.com/astronomer/agents/tree/main/astro-airflow-mcp)) is an open-source package that installs directly into an Airflow Deployment and exposes an MCP endpoint on the webserver. This gives AI tools direct access to Airflow's REST API — Dags, task logs, connections, variables, and more — without running a separate server.

Unlike the Astro Cloud MCP server (which manages Astro-level resources like Deployments and Workspaces), the Airflow MCP Plugin provides Airflow-level access for a single Deployment: listing Dags, viewing task logs, inspecting connections, diagnosing failures, and more.

The plugin auto-detects the installed Airflow version and registers the appropriate integration:

* **Airflow 3.x**: Mounts as a FastAPI app on the API server.
* **Airflow 2.x** (2.4 or later): Registers as a Flask blueprint on the webserver.

### Prerequisites

* An Airflow Deployment on Astro. The plugin supports Airflow 3.x (Runtime 3.1 or later) and Airflow 2.x (2.4 or later).
* A [Deployment API token](/docs/astro/deployment-api-tokens) with a role that allows POST requests (required by the MCP protocol). See [Configure authentication](#configure-authentication) for role options.

### Install the plugin

Add `astro-airflow-mcp` to your Astro project's `requirements.txt`:

```text wrap theme={null}
astro-airflow-mcp
```

Deploy the change. The package auto-registers as an Airflow plugin — no Dockerfile changes or additional configuration needed.

### Set environment variables

After deploying, set the following environment variable on your Deployment to block write operations:

```sh wrap theme={null}
astro deployment variable create \
  --deployment-id <deployment-id> \
  AF_READ_ONLY=true
```

| Variable       | Required    | Description                                                                                                                                                          |
| -------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AF_READ_ONLY` | Recommended | Blocks all write operations (trigger, pause, clear, delete) at the MCP server level, regardless of the token's permissions. Set to `true` for safe read-only access. |

<Note>In plugin mode, the MCP server always runs in stateless HTTP mode, so Claude Code and other MCP clients work without additional configuration. The `FASTMCP_STATELESS_HTTP` environment variable applies only when running the MCP server as a standalone process.</Note>

### Configure authentication

The MCP protocol uses POST requests for all operations, including read-only ones. Your Deployment API token does **not** need write permissions for read-only MCP access: authorization is still based on the underlying Airflow permissions, so a custom role with read permissions such as `*.get` is sufficient. The important exception is `WORKSPACE_MEMBER`, which Astro's auth proxy blocks from making the POST requests the MCP protocol requires.

#### Option A: Use a built-in role

Create a [Deployment API token](/docs/astro/deployment-api-tokens) with one of the following roles:

* **DEPLOYMENT\_ADMIN** — Full access (works but overprivileged)
* **WORKSPACE\_OPERATOR** or **WORKSPACE\_AUTHOR** — Moderate privilege

<Warning>
  **WORKSPACE\_MEMBER** does not work with the MCP plugin. Astro's auth proxy blocks POST requests for this role, which prevents the MCP protocol handshake from completing.
</Warning>

#### Option B: Create a custom `MCP_VIEWER` role (recommended)

For least-privilege access, create a [custom Deployment role](/docs/astro/customize-deployment-roles) that includes only read permissions. This ensures the token can use all MCP read tools but cannot modify any Airflow resources.

1. In the Astro UI, go to **Settings**, then in the **Access Management** section, click **Roles & Permissions**, click **Custom**, then click **+ New Custom Role** (in the legacy UI, go to **Organization Settings** > **Access Management** > **Roles**, then click **+ Add Role**).

2. Set the **Scope** to **Deployment**.

3. Name the role `MCP_VIEWER` with a description like "Read-only access for Airflow MCP plugin."

4. Select all `deployment.airflow.*.get` permissions. See [Custom role permissions reference](/docs/astro/deployment-role-reference) for the full list. The role should include these 28 permissions:

   | Permission                               | Purpose                                    |
   | ---------------------------------------- | ------------------------------------------ |
   | `deployment.get`                         | Get Deployment information                 |
   | `deployment.airflow.adminMenu.get`       | View Admin menu (gates Airflow config API) |
   | `deployment.airflow.astronomer.get`      | View Astronomer menu                       |
   | `deployment.airflow.auditLog.get`        | View audit logs                            |
   | `deployment.airflow.browseMenu.get`      | View Browse menu                           |
   | `deployment.airflow.clusterActivity.get` | View cluster activity and DAG stats        |
   | `deployment.airflow.config.get`          | View Config menu                           |
   | `deployment.airflow.connection.get`      | View connections                           |
   | `deployment.airflow.customMenu.get`      | View custom plugin menus                   |
   | `deployment.airflow.dag.get`             | View Dags                                  |
   | `deployment.airflow.dagCode.get`         | View DAG source code                       |
   | `deployment.airflow.dagDependencies.get` | View DAG dependencies                      |
   | `deployment.airflow.dagRun.get`          | View DAG runs                              |
   | `deployment.airflow.datasets.get`        | View assets and datasets                   |
   | `deployment.airflow.docs.get`            | View documentation links                   |
   | `deployment.airflow.importError.get`     | View import errors                         |
   | `deployment.airflow.job.get`             | View scheduler jobs                        |
   | `deployment.airflow.plugin.get`          | View plugins                               |
   | `deployment.airflow.pool.get`            | View pools                                 |
   | `deployment.airflow.provider.get`        | View providers                             |
   | `deployment.airflow.slaMiss.get`         | View SLA misses                            |
   | `deployment.airflow.taskInstance.get`    | View task instances                        |
   | `deployment.airflow.taskLog.get`         | View task logs                             |
   | `deployment.airflow.taskReschedule.get`  | View task reschedules                      |
   | `deployment.airflow.trigger.get`         | View triggers                              |
   | `deployment.airflow.variable.get`        | View variables                             |
   | `deployment.airflow.website.get`         | Access the Airflow UI                      |
   | `deployment.airflow.xcom.get`            | View XCom data                             |

5. Click **Create role**.

Then create a Deployment API token with the new role:

```sh wrap theme={null}
astro deployment token create \
  --deployment-id <deployment-id> \
  --name "mcp-viewer" \
  --role MCP_VIEWER
```

### Connect your MCP client

After the plugin is deployed and a token is created, the MCP endpoint is available at:

```text wrap theme={null}
https://<deployment-webserver-url>/mcp/v1/
```

You can find your Deployment's webserver URL in the Astro UI on the Deployment's overview page.

<Tabs>
  <Tab title="Claude Code">
    ```sh wrap theme={null}
    claude mcp add -t http -s user \
      -H "Authorization: Bearer <token>" \
      -- airflow \
      "https://<deployment-webserver-url>/mcp/v1/"
    ```

    <Warning>
      Use `-t http`, not `-t sse`. The MCP endpoint returns SSE-formatted responses, but the correct Claude Code transport type is `http`. Using `-t sse` causes the connection to fail.
    </Warning>
  </Tab>

  <Tab title="Cursor">
    Add the following to your MCP configuration in Cursor (**Tools & Integrations** > **Add Custom MCP**):

    ```json wrap theme={null}
    {
      "mcpServers": {
        "airflow": {
          "url": "https://<deployment-webserver-url>/mcp/v1/",
          "headers": {
            "Authorization": "Bearer <token>"
          }
        }
      }
    }
    ```
  </Tab>

  <Tab title="VS Code / Other Clients">
    Add the following to your MCP client configuration file:

    ```json wrap theme={null}
    {
      "mcpServers": {
        "airflow": {
          "url": "https://<deployment-webserver-url>/mcp/v1/",
          "headers": {
            "Authorization": "Bearer <token>"
          }
        }
      }
    }
    ```
  </Tab>
</Tabs>

### Troubleshooting

| Symptom                                            | Cause                                                | Fix                                                                                                                                               |
| -------------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Failed to connect` with `-t sse`                  | Wrong transport type                                 | Use `-t http` instead of `-t sse` in Claude Code                                                                                                  |
| 401 Unauthorized on MCP endpoint                   | Missing or expired token                             | Regenerate the Deployment API token                                                                                                               |
| 403 Forbidden on some read tools                   | Incomplete role permissions                          | Ensure the role includes all `deployment.airflow.*.get` permissions listed above                                                                  |
| 403 on `get_airflow_config` or `list_dag_warnings` | Airflow Admin RBAC required                          | A small number of Airflow API endpoints require the Admin role regardless of Astro permissions. Most MCP tools work without Admin access.         |
| `WORKSPACE_MEMBER` token fails                     | POST blocked by Astro auth proxy                     | Use a higher-privilege role or create a custom `MCP_VIEWER` role                                                                                  |
| 404 on MCP endpoint right after a deploy           | Plugin loads per-worker, rolling restart in progress | Wait for the rollout to settle, then retry. The first MCP call per worker also adds a small one-time latency while the FastMCP lifespan warms up. |


# Create Airflow connections in the Astro UI
Source: https://astronomer.io/docs/astro/create-and-link-connections

Create Airflow connections and link them to multiple Deployments in the Astro Environment Manager.

You can create and manage Airflow connections for Deployments with the Astro Environment Manager in the Astro UI. The Environment Manager uses an Astro-managed secrets backend to store connection configurations as Kubernetes Secrets.

Using the Environment Manager, you can quickly and securely create connections once and share them to multiple Deployments without having to set up your own secrets backend. For example, you can configure a connection with the credentials for a sandbox or development environment. Then, you can later configure your connection to be applied to all Deployments in the workspace by default. This means that when you create new Deployments, they automatically have access to your development environment. Later, you can edit the connection to point to your production resources by using [field overrides](#override-connection-fields).

Compared to creating a connection in the Airflow UI, when you create a connection in the Astro UI, you can:

* Share the connection with multiple Deployments within the Workspace.
* Share the connection with [Astro IDE](/docs/astro/ide-overview) projects so that their ephemeral test Deployments can use it.
* Override fields in the connection for individual Deployments or Astro IDE projects.
* Use configured connections in local Airflow environments. See [Import and export connections and variables](/docs/cli/v1.43/local-connections).
* Use connections in branch-based deploys and PR previews.

Workspace Owners and Operators can create and assign connections, while Workspace Authors can view configured connections and use them in Deployments. If your Organization has [**Environment Secrets Fetching**](/docs/astro/organization-settings#configure-environment-secrets-fetching-for-the-astro-environment-manager) enabled, you can additionally use configured connections, including ones that contain secrets, in local development environments. See [Import and export connections and variables](/docs/cli/v1.43/local-connections).

<Frame>
  <img alt="Example of the Connections tab in the Astro Environment Manager page" />
</Frame>

## How connections are stored

When you create an Airflow connection in the Environment Manager, Astro stores Airflow connection details in an Astronomer-hosted secrets manager, and then applies connections to Deployments as Kubernetes Secrets. Specifically the following steps occur:

* Astro stores the connection details in a secure secrets manager hosted by Astronomer.
* When a connection is assigned to a Deployment, Astro uses Airflow's provided [local filesystem secrets backend](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/local-filesystem-secrets-backend.html) to mount your connections as Kubernetes Secrets.
* When your Dags use your connections, Airflow reads the connection details from the filesystem using the Airflow local filesystem secrets backend.

This process occurs every time you create or update a connection.

When you use connections for local development, the Astro CLI reads the connections from the Astro API and injects them into the local Airflow instance's metadata database.

### Fetch environment secrets

The Astro CLI can automatically retrieve connections from the Astro UI when you start your local airflow instance with `astro dev start --deployment-id=<your-deployment-id>`, which means you can use your connection details without needing to manage credentials between local and deployed environments. Local environments fetch connection information the same way as for Deployments, so they require an active internet connection and for you to be logged in with the Astro CLI. You can only fetch environment secrets from Deployments that belong to Workspaces where you are at least a Workspace Member.

<Tip>By default, connections can't be exported locally. However, if you want to work with connections locally, the Organization Owner can enable [**Environment Secrets Fetching**](/docs/astro/organization-settings#configure-environment-secrets-fetching-for-the-astro-environment-manager) in the Astro UI.</Tip>

## Prerequisites

* Workspace Operator or Workspace Owner [user permissions](/docs/astro/user-permissions)
* A Deployment on Astro. See [Create a Deployment](/docs/astro/create-deployment)
* Astro Runtime 9.3.0 or greater

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Create a connection

You can create connections both at the Deployment and Workspace level. When you create a connection at the Deployment level, the connection details are available only to that specific Deployment. When you create a connection at the Workspace level, you can apply the connection to several Deployments and override specific fields as needed for each Deployment.

To create a connection at the Workspace level:

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Environment** > **Connections**.

    2. Click **+ New Connection** to add a new connection.

    3. Find the service you want to connect from the list of available options.

       <Tip>If you don't see your service in the list, or you don't see fields that you need in a service that is in the list, you can select the **Generic** option to specify a connection using [Airflow's standard connection fields](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html).</Tip>

    4. Enter the information for your connection in the listed fields.

    5. Click **Create Connection**.

    6. Make your connection accessible to Deployments. See [Link connections to Deployments](#link-connections-to-deployments).
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Environment** in the left menu to open the **Connections** page.

    2. Click **+ Connection** to add a new connection.

    3. Find the service you want to connect from the list of available options.

       <Tip>If you don't see your service in the list, or you don't see fields that you need in a service that is in the list, you can select the **Generic** option to specify a connection using [Airflow's standard connection fields](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html).</Tip>

    4. Enter the information for your connection in the listed fields.

    5. Click **Create Connection**.

    6. Make your connection accessible to Deployments. See [Link connections to Deployments](#link-connections-to-deployments).
  </Tab>
</Tabs>

To create a connection at the Deployment level:

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, select a Deployment, then click the **Environment** tab.

    2. Click **Connections**, then click **+ New Connection**.

    3. Find the service you want to connect from the list of available options.

       <Tip>If you don't see your service in the list, or you don't see fields that you need in a service that is in the list, you can select the **Generic** option to specify a connection using [Airflow's standard connection fields](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html).</Tip>

    4. Enter your information in the required fields.

    5. Click **Create Connection** to make your new connection.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, select a Deployment, then click the **Environment** tab.

    2. Click **+ Connection** to add a new connection.

    3. Find the service you want to connect from the list of available options.

       <Tip>If you don't see your service in the list, or you don't see fields that you need in a service that is in the list, you can select the **Generic** option to specify a connection using [Airflow's standard connection fields](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html).</Tip>

    4. Enter your information in the required fields.

    5. Click **Create Connection** to make your new connection.
  </Tab>
</Tabs>

After you create a connection, you can reference its **Connection ID** from Dag code like you would with any Airflow connection created through the Airflow UI.

## Link connections to Deployments

After you create a connection at the Workspace level, you can link it to multiple Deployments. Linking connections is useful for standardizing external resource usage across your entire team.

For the most flexibility, you can set default connections and override the connection details per-Deployment based on details like the Deployment's usage and environment type (production or development).

### Step 1: Link the connection

1. In the Astro UI, go to **Environment** > **Connections**.
2. Click the connection you want to link to a Deployment.
3. Click **+ Link Deployment**.
4. Choose a Deployment from the list that appears.
5. (Optional) Click **More options** and then add any field overrides for this Deployment. For example, if your connection requests access to a development database by default, you can override its details to instead request access to a production database.
6. Click **Link connection**.

### Step 2: (Optional) Add provider packages to your Deployment

Some connection types require installing dependencies on your Deployment through provider packages. If your connection type requires a provider package and the provider package is neither [included in Astro Runtime](/docs/runtime/runtime-image-architecture#provider-packages) nor included in the `requirements.txt` file of your Astro project, Airflow won't be able to use your connection.

If you are uncertain what provider package the connection needs, you can check in the [Airflow Registry](https://airflow.apache.org/registry/).

1. Open the local Astro project for your Deployment.
2. Add the required provider package name to your project's `requirements.txt` and save your changes.
3. Deploy your project with the new provider to your Deployment.

## Configure connection sharing for a Workspace

You can configure Astro to link Workspace-level connections to all Deployments in the Workspace by default.

This is useful, for example, when you need to configure a connection for development environments that all Deployments in a Workspace should start with. Then, when you create new Deployments, they automatically have a default connection to your development resources.

When you're ready to connect your Deployments to production resources, you can either replace the connection or [override the connection field](#override-connection-fields) values with your production resource information.

If you change the setting from **Restricted** to **Linked to all Deployments**, Astro respects any connection field overrides that you might have configured for existing linked Deployments.

<Frame>
  <img alt="Edit Deployment Sharing settings in the Environment Manager view" />
</Frame>

1. In the Astro UI, go to **Environment** > **Connections**.
2. Click the connection that you want to add per-Deployment field overrides to.
3. Click **Deployment Sharing** and toggle the setting to choose either:
   * **Restricted**: Only share the connection individually to Deployments.
   * **Linked to all Deployments**: Link to all current and future Deployments in this Workspace.
4. (Optional) Change the default connection field values.
5. Click **Update connection** to save.

## Override connection fields

If you create a connection at the Workspace level and link it to a Deployment, you can later edit the connection within the Deployment to specify field overrides. When you override a field, you specify values that you want to use for a one Deployment, but not for others. This way, you can configure the connection and authentication a single time, but still have the flexibility to customize connection at the Deployment level.

For example, you might have created a connection to a Snowflake account, and then add field overrides to specify the default schemas or databases you want each Deployment to use.

1. In the Astro UI, go to **Environment** > **Connections**.
2. Click the connection that you want to add per-Deployment field overrides to.
3. (Optional) Click **Deployment Sharing** and choose if you want to **Restrict** or **Link to all Deployments**. You can also change the default connection field values. Click **Update connection** to save.
4. Click **Edit** to open the connection configurations for a specific linked Deployment.
5. Add the override values to the fields you want to edit. You might need to open **More options** to find the full list of available fields.
6. Click **Update connection link**.

## Link connections to Astro IDE projects

In addition to Deployments, you can link Workspace-level connections to [Astro IDE](/docs/astro/ide-overview) projects. When you link a connection to an Astro IDE project, the project's ephemeral test Deployments start with that connection available, without affecting Deployments that aren't started from the IDE.

For the most flexibility, you can set a default connection at the Workspace level and override its fields per project, similar to how you override fields per Deployment.

### Link a connection to an Astro IDE project

1. In the Astro UI, go to **Environment** > **Connections**.
2. Click the connection you want to link to an Astro IDE project.
3. Click the **Linked Projects** tab.
4. Click **Link Project**.
5. Choose an Astro IDE project from the list. Optionally, add field overrides for this project. For example, point the connection to a dedicated test database for the project.
6. Confirm the link.

### Configure project sharing for a Workspace

You can configure Astro to link a Workspace-level connection to all Astro IDE projects in the Workspace by default. This is useful when you want every project's test Deployments to start with the same set of development credentials.

1. In the Astro UI, go to **Environment** > **Connections**.
2. Click the connection that you want to share with all Astro IDE projects.
3. Click the **Edit** icon next to **Auto-linking** in the connection details.
4. In the **Edit Connection** dialog, under **Auto-linking**, toggle **ALL ASTRO IDE PROJECTS** to **On**.
5. Click **Update Connection**.

### Connection precedence in Astro IDE test Deployments

When an Astro IDE project starts a test Deployment, the Deployment receives the union of:

* Connections linked to the project, including Workspace-level connections linked to the project and connections scoped directly to the project.
* Workspace-level connections linked to that Deployment.

If both the project and a linked Deployment define a connection with the same connection ID, the project-level value takes precedence in the test Deployment.

## Migrate existing connections to the Environment Manager

If you have connections defined in a Deployment's Airflow metadata database, you can move them into the Environment Manager so that they can be reused across Deployments and Astro IDE projects. See [Migrate existing objects to the Environment Manager](/docs/astro/migrate-metadata-db-to-environment-manager).

## Promote a Deployment-scoped connection to the Workspace

If you created a connection on a single Deployment and later want to reuse it across other Deployments or Astro IDE projects, you can promote it to the Workspace level from the Deployment's Environment tab. See [Promote a Deployment environment object to the Workspace](/docs/astro/migrate-metadata-db-to-environment-manager#promote-a-deployment-environment-object-to-the-workspace).


# Create environment variables in the Astro UI
Source: https://astronomer.io/docs/astro/create-and-link-environment-variables

Create environment variables and link them to multiple Deployments in the Astro Environment Manager.

You can create and manage environment variables for Deployments with the Astro Environment Manager in the Astro UI. The Environment Manager uses an Astro-managed secrets backend to store environment variable key-value pairs as Kubernetes Secrets.

Using the Environment Manager, you can quickly and securely create environment variables once and share them to multiple Deployments without having to set up your own secrets backend.

For example, you can configure an environment variable with credentials for a sandbox or development environment. Then, you can later configure your environment variable to be applied to all Deployments in the workspace by default. This means that when you create new Deployments, they automatically have access to your development environment. Later, you can edit the environment variable to point to your production resources by using [value overrides](#override-environment-variable-values).

When you create an environment variable in the Environment Manager instead of the Deployment UI, you can:

* Share the environment variable with multiple Deployments within the Workspace.
* Share the environment variable with [Astro IDE](/docs/astro/ide-overview) projects so that their ephemeral test Deployments can use it.
* Override the environment variable value for individual Deployments or Astro IDE projects.
* Use environment variables in branch-based deploys and PR previews.

<Info>
  Workspace Owners and Operators can create and assign environment variables, while Workspace Authors can view configured environment variables and use them in Deployments. Learn more about [user permissions](/docs/astro/user-permissions).
</Info>

## How environment variables are stored

When you create an environment variable in the Environment Manager, Astro stores environment variable details in an Astronomer-hosted secrets manager, and then applies environment variables to Deployments as Kubernetes Secrets. Specifically the following steps occur:

* Astro stores the environment variable value in a secure secrets manager hosted by Astronomer.
* When an environment variable is assigned to a Deployment, Astro applies your environment variable as a Kubernetes Secret to your Deployment's namespace.
* When your Deployment starts, the environment variable is loaded into the Airflow environment.

This process occurs every time you create or update an environment variable.

<Warning>
  Environment variables marked as secret are stored securely by Astronomer and are not shown in the Astro UI. However, it's possible for a user in your organization to create or configure a dag that exposes secret values in Airflow task logs. Airflow task logs are visible to all Workspace members in the Airflow UI and accessible in your Astro cluster's storage.

  To avoid exposing secret values in task logs, instruct users to not log environment variables in Dag code.
</Warning>

## Prerequisites

* Workspace Operator or Workspace Owner [user permissions](/docs/astro/user-permissions)
* An [Astro Deployment](/docs/astro/create-deployment)
* Astro Runtime 9.3.0 or greater

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Create an environment variable

You can create environment variables both at the Deployment and Workspace level. When you create an environment variable at the Deployment level, the environment variable is available only to that specific Deployment. When you create an environment variable at the Workspace level, you can apply the environment variable to several Deployments and override its value as needed for each Deployment.

### Create an environment variable at the Workspace level:

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Environment** > **Environment Variables**.
    2. Click **+ New Environment Variable** to add a new environment variable.
    3. Choose whether you want your environment variable to be **Automatically link to all deployments** by setting the toggle to **On**. See [Link environment variables to Deployments](#link-environment-variables-to-deployments) for more information.
    4. Enter your information in the required fields.
    5. Choose whether you want your environment variable to be **Not Secret** or **Secret**. This allows you to store API keys, tokens, or other secrets in an environment variable without those credentials being available to future editors.
    6. Click **Create environment variable**.
    7. If you haven't linked your environment variable to all deployments, make it accessible to individual Deployments. See [Link environment variables to Deployments](#link-environment-variables-to-deployments).
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Environment** in the left menu to open the **Environment Variables** page.
    2. Click **+ Environment Variable** to add a new environment variable.
    3. Choose whether you want your environment variable to be **Automatically link to all deployments** by setting the toggle to **On**. See [Link environment variables to Deployments](#link-environment-variables-to-deployments) for more information.
    4. Enter your information in the required fields.
    5. Choose whether you want your environment variable to be **Not Secret** or **Secret**. This allows you to store API keys, tokens, or other secrets in an environment variable without those credentials being available to future editors.
    6. Click **Create environment variable**.
    7. If you haven't linked your environment variable to all deployments, make it accessible to individual Deployments. See [Link environment variables to Deployments](#link-environment-variables-to-deployments).
  </Tab>
</Tabs>

### Create an environment variable at the Deployment level:

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, select a Deployment, then click the **Environment** tab.
    2. Click **Environment variables**.
    3. Click **Edit Deployment Variables** (or **+ New Environment Variable** if you have no variables configured yet).
    4. Enter an environment variable key and value. For sensitive credentials that should be treated with an additional layer of security, select the **Secret** checkbox. This permanently hides the variable's value from all users in your Workspace.
    5. Click **Update Environment Variables** to save your changes. Your Airflow scheduler, webserver, and workers restart. After saving, it can take up to two minutes for new variables to be applied to your Deployment.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, select a Deployment, then click the **Environment** tab.
    2. Click **Environment variables**.
    3. Click **Edit Deployment Variables** (or **+ New Environment Variable** if you have no variables configured yet).
    4. Enter an environment variable key and value. For sensitive credentials that should be treated with an additional layer of security, select the **Secret** checkbox. This permanently hides the variable's value from all users in your Workspace.
    5. Click **Update Environment Variables** to save your changes. Your Airflow scheduler, webserver, and workers restart. After saving, it can take up to two minutes for new variables to be applied to your Deployment.
  </Tab>
</Tabs>

After you create an environment variable, you can reference it from Dag code using standard Python methods like `os.getenv()`. For more information and examples of using environment variables in your Dag code, see [Using environment variable](/docs/learn/airflow-variables#using-environment-variables).

## Link environment variables to Deployments

After you create an environment variable at the Workspace level, you can link it to multiple Deployments. Linking environment variables is useful for standardizing configuration across your entire team.

For the most flexibility, you can set default environment variables and override the environment variable values per-Deployment based on details like the Deployment's usage and environment type (production or development).

1. In the Astro UI, go to **Environment** > **Environment Variables**.
2. Click the environment variable you want to link to a Deployment.
3. Click **+ Link Deployment**.
4. Choose a Deployment from the list that appears.
5. (Optional) Click **More options** and then add any value overrides for this Deployment. For example, if your environment variable provides a development API key by default, you can override its value to instead provide a production API key.
6. Click **Link environment variable**.

## Configure environment variable sharing for a Workspace

You can configure Astro to link Workspace-level environment variables to all Deployments in the Workspace by default.

This is useful, for example, when you need to configure an environment variable for development environments that all Deployments in a Workspace should start with. Then, when you create new Deployments, they automatically have a default environment variable to your development resources.

When you're ready to connect your Deployments to production resources, you can either replace the environment variable or [override the environment variable values](#override-environment-variable-values) with your production resource information.

If you toggle the **Automatically link to all deployments** setting from **Off** to **On**, Astro respects any environment variable value overrides that you might have configured for existing linked Deployments.

1. In the Astro UI, go to **Environment** > **Environment Variables**.
2. Click the environment variable that you want to add per-Deployment overrides to.
3. Toggle the **Automatically link to all deployments** setting to choose either:
   * **Off**: Only share the environment variable individually to Deployments.
   * **On**: Link to all current and future Deployments in this Workspace.
4. (Optional) Change the default environment variable value.
5. Click **Update environment variable** to save.

## Override environment variable values

If you create an environment variable at the Workspace level and link it to a Deployment, you can later edit the environment variable within the Deployment to specify a value override. When you override the value, you specify the value that you want to use for one Deployment, but not for others. This way, you can configure the environment variable a single time, but still have the flexibility to customize environment variables at the Deployment level.

For example, you might have created an environment variable that points to a development API endpoint, and then add a value override to specify a staging or production API endpoint for your staging and production Deployments to use.

### Environment variable precedence

When an environment variable is defined at both the Workspace and Deployment levels with the same key, the Deployment-level value always takes precedence. The Deployment environment variables page shows a unified view of both workspace and deployment environment variables, with clear indicators showing:

* The source of each environment variable (Workspace or Deployment)
* Override badges when a deployment environment variable overrides a workspace one
* Tooltips explaining the relationship between environment variables

To override a workspace environment variable value for a specific Deployment:

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, select a Deployment, then click the **Environment** tab.
    2. Click **Environment variables**.
    3. Click **Edit Deployment Variables**.
    4. To override a workspace environment variable, add a new environment variable with the same key as the workspace environment variable but with a different value. The deployment value will take precedence.
    5. Click **Update Environment Variables** to save your changes.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, select a Deployment, then click the **Environment** tab.
    2. Click **Environment variables**.
    3. Click **Edit Deployment Variables**.
    4. To override a workspace environment variable, add a new environment variable with the same key as the workspace environment variable but with a different value. The deployment value will take precedence.
    5. Click **Update Environment Variables** to save your changes.
  </Tab>
</Tabs>

Alternatively, you can override from the Workspace level:

1. In the Astro UI, go to **Environment** > **Environment Variables**.
2. Click the environment variable that you want to add per-Deployment overrides to.
3. Click **Edit** to open the environment variable configurations for a specific linked Deployment.
4. Switch the **Override value** toggle from **No Override** to **Override**.
5. Add the override value.
6. Click **Update environment variable link**.

## Link environment variables to Astro IDE projects

In addition to Deployments, you can link Workspace-level environment variables to [Astro IDE](/docs/astro/ide-overview) projects. When you link an environment variable to an Astro IDE project, the project's ephemeral test Deployments start with that variable available, without affecting Deployments that aren't started from the IDE.

For the most flexibility, you can set a default value at the Workspace level and override it per project, similar to how you override values per Deployment.

### Link an environment variable to an Astro IDE project

1. In the Astro UI, go to **Environment** > **Environment Variables**.
2. Click the environment variable you want to link to an Astro IDE project.
3. Click the **Linked Projects** tab.
4. Click **Link Project**.
5. Choose an Astro IDE project from the list. Optionally, add a value override for this project.
6. Confirm the link.

### Configure project sharing for a Workspace

You can configure Astro to link a Workspace-level environment variable to all Astro IDE projects in the Workspace by default. This is useful when you want every project's test Deployments to start with the same configuration values.

1. In the Astro UI, go to **Environment** > **Environment Variables**.
2. Click the environment variable that you want to share with all Astro IDE projects.
3. Click the **Edit** icon next to **Auto-linking** in the variable details.
4. In the **Edit Environment Variable** dialog, under **Auto-linking**, toggle **ALL ASTRO IDE PROJECTS** to **On**.
5. Click **Update Environment Variable**.

### Environment variable precedence in Astro IDE test Deployments

When an Astro IDE project starts a test Deployment, the Deployment receives the union of:

* Environment variables linked to the project, including Workspace-level variables linked to the project and variables scoped directly to the project.
* Workspace-level environment variables linked to that Deployment.

If both the project and a linked Deployment define an environment variable with the same key, the project-level value takes precedence in the test Deployment.

## Migrate existing environment variables to the Environment Manager

If you have environment variables set directly on a Deployment, you can move them into the Environment Manager so that they can be reused across Deployments and Astro IDE projects. See [Migrate existing objects to the Environment Manager](/docs/astro/migrate-metadata-db-to-environment-manager).

## Promote a Deployment-scoped environment variable to the Workspace

If you created an environment variable on a single Deployment and later want to reuse it across other Deployments or Astro IDE projects, you can promote it to the Workspace level from the Deployment's Environment tab. See [Promote a Deployment environment object to the Workspace](/docs/astro/migrate-metadata-db-to-environment-manager#promote-a-deployment-environment-object-to-the-workspace).

## See also

* [Manage environment variables on Astro](/docs/astro/manage-env-vars)
* [Environment variables overview](/docs/astro/environment-variables)
* [Manage connections and variables](/docs/astro/manage-connections-variables)


# Create Airflow variables in the Astro UI
Source: https://astronomer.io/docs/astro/create-and-link-variables

Create Airflow variables and link them to multiple Deployments in the Astro Environment Manager.

You can create and manage Airflow variables for Deployments with the Astro Environment Manager in the Astro UI. The Environment Manager uses an Astro-managed secrets backend to store Airflow variable key-value pairs as Kubernetes Secrets.

Using the Environment Manager, you can quickly and securely create Airflow variables once and share them to multiple Deployments without having to set up your own secrets backend. You can also create a variable once and use it across multiple Airflow Deployments.

For example, you can configure a variable with the access token credentials for a sandbox or development environment. Then, you can later configure your variable to be applied to all Deployments in the workspace by default. This means that when you create new Deployments, they automatically have access to your development environment. Later, you can override the variable to use your production credentials in production Deployments by using a [field override or edit](#override-variable-values).

Compared to creating a variable in the Airflow UI, when you create a variable in the Astro UI, you can:

* Share the variable with multiple Deployments within the Workspace.
* Share the variable with [Astro IDE](/docs/astro/ide-overview) projects so that their ephemeral test Deployments can use it.
* Override the variable value for individual Deployments or Astro IDE projects.
* Use variables in branch-based deploys and PR previews.

Workspace Owners and Operators can create and assign variables, while Workspace Authors can view configured variables and use them in Deployments.

## How variables are stored

When you create an Airflow variable in the Environment Manager, Astro stores Airflow variable details in an Astronomer-hosted secrets manager, and then applies variables to Deployments as Kubernetes Secrets. Specifically the following steps occur:

* Astro stores the variable value in a secure secrets manager hosted by Astronomer.
* When a variable is assigned to a Deployment, Astro uses Airflow's provided [local filesystem secrets backend](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/local-filesystem-secrets-backend.html) to mount your Airflow variables as Kubernetes Secrets.
* When your Dags use your variables, Airflow reads the details from the filesystem using the Airflow local filesystem secrets backend.

This process occurs every time you create or update a variable.

## Prerequisites

* Workspace Operator or Workspace Owner [user permissions](/docs/astro/user-permissions)
* A Deployment on Astro. See [Create a Deployment](/docs/astro/create-deployment)
* Astro Runtime 9.3.0 or greater

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Create an Airflow Variable

You can create Airflow variables both at the Deployment and Workspace level. When you create a variable at the Deployment level, the variable details are available only to that specific Deployment. When you create a variable at the Workspace level, you can apply the variable to several Deployments and override its value as needed for each Deployment.

To create a variable at the Workspace level:

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Environment** > **Airflow Variables**.
    2. Click **+ New Airflow Variable** to add a new variable.
    3. Choose whether you want your variable to be **Automatically link to all deployments** by setting the toggle to **On**. See [Link variables to Deployments](#link-variables-to-deployments) for more information.
    4. Enter your information in the required fields.
    5. Choose whether you want your variable to be **Not Secret** or **Secret**, when working with it in the future. This allows you to store API keys, tokens, or other secrets in an Airflow Variable without those credentials being available to future editors.
    6. Click **Create Airflow Variable**.
    7. If you haven't linked your variable to all deployments, make it accessible to individual Deployments. See [Link variables to Deployments](#link-variables-to-deployments).
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Environment** in the left menu to open the **Airflow Variables** page.
    2. Click **+ Airflow Variable** to add a new variable.
    3. Choose whether you want your variable to be **Automatically link to all deployments** by setting the toggle to **On**. See [Link variables to Deployments](#link-variables-to-deployments) for more information.
    4. Enter your information in the required fields.
    5. Choose whether you want your variable to be **Not Secret** or **Secret**, when working with it in the future. This allows you to store API keys, tokens, or other secrets in an Airflow Variable without those credentials being available to future editors.
    6. Click **Create Airflow Variable**.
    7. If you haven't linked your variable to all deployments, make it accessible to individual Deployments. See [Link variables to Deployments](#link-variables-to-deployments).
  </Tab>
</Tabs>

To create a variable at the Deployment level:

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, select a Deployment, then click the **Environment** tab.
    2. Click **Airflow Variables**, then click **+ New Airflow Variable**.
    3. Enter your information in the required fields.
    4. Choose whether you want your variable to be **Not Secret** or **Secret**, when working with it in the future. This allows you to store API keys, tokens, or other secrets in an Airflow Variable without those credentials being available to future editors.
    5. Click **Create Airflow Variable** to make your new variable.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, select a Deployment, then click the **Environment** tab.
    2. Click **+ Airflow Variable** to add a new variable.
    3. Enter your information in the required fields.
    4. Choose whether you want your variable to be **Not Secret** or **Secret**, when working with it in the future. This allows you to store API keys, tokens, or other secrets in an Airflow Variable without those credentials being available to future editors.
    5. Click **Create Airflow Variable** to make your new variable.
  </Tab>
</Tabs>

After you create a variable, you can reference its **Airflow Variable Key** from Dag code like you would with any Airflow variable created through the Airflow UI.

## Link variables to Deployments

After you create a variable at the Workspace level, you can link it to multiple Deployments. Linking variables is useful for standardizing external resource usage across your entire team.

For the most flexibility, you can set default variables and override the variable details per-Deployment based on details like the Deployment's usage and environment type (production or development).

1. In the Astro UI, go to **Environment** > **Airflow Variables**.
2. Click the variable you want to link to a Deployment.
3. Click **+ Link Deployment**.
4. Choose a Deployment from the list that appears.
5. (Optional) Click **More options** and then add any value overrides for this Deployment. For example, if your variable provides a development token by default, you can override its details to instead provide a production token.
6. Click **Link Airflow Variable**.

## Configure variable sharing for a Workspace

You can configure Astro to link Workspace-level variables to all Deployments in the Workspace by default.

This is useful, for example, when you need to configure a variable for development environments that all Deployments in a Workspace should start with. Then, when you create new Deployments, they automatically have a default variable to your development resources.

When you're ready to connect your Deployments to production resources, you can either replace the variable or [override the variable field](#override-variable-values) values with your production resource information.

If you toggle the **Automatically link to all deployments** setting from **Off** to **On**, Astro respects any variable value overrides that you might have configured for existing linked Deployments.

1. In the Astro UI, go to **Environment** > **Airflow Variables**.
2. Click the variable that you want to add per-Deployment field overrides to.
3. Toggle the **Automatically link to all deployments** setting to choose either:
   * **Off**: Only share the variable individually to Deployments.
   * **On**: Link to all current and future Deployments in this Workspace.
4. (Optional) Change the default variable value.
5. Click **Update Airflow Variable** to save.

## Override variable values

If you create a variable at the Workspace level and link it to a Deployment, you can later edit the variable within the Deployment to specify a value override. When you override the variable's value, you specify the value that you want to use for one Deployment, but not for others. This way, you can configure the variable a single time, but still have the flexibility to customize variables at the Deployment level.

For example, you might have created a variable that points to a dev token, and then add a value override to specify a staging or production token for your staging and production Deployments to use.

1. In the Astro UI, go to **Environment** > **Airflow Variables**.
2. Click the variable that you want to add per-Deployment field overrides to.
3. Click **Edit** to open the variable configurations for a specific linked Deployment.
4. Switch the **Override value** toggle from **No Override** to **Override**.
5. Add the override value.
6. Click **Update Airflow Variable link**.

## Link Airflow variables to Astro IDE projects

In addition to Deployments, you can link Workspace-level Airflow variables to [Astro IDE](/docs/astro/ide-overview) projects. When you link a variable to an Astro IDE project, the project's ephemeral test Deployments start with that variable available, without affecting Deployments that aren't started from the IDE.

For the most flexibility, you can set a default value at the Workspace level and override it per project, similar to how you override values per Deployment.

### Link a variable to an Astro IDE project

1. In the Astro UI, go to **Environment** > **Airflow Variables**.
2. Click the variable you want to link to an Astro IDE project.
3. Open the **Linked Projects** tab.
4. Click **Link Project**.
5. Choose an Astro IDE project from the list.
6. (Optional) Toggle  **Override Value** and add a value override for this project.
7. Click **Link Airflow Variable**.

### Configure project sharing for a Workspace

You can configure Astro to link a Workspace-level variable to all Astro IDE projects in the Workspace by default. This is useful when you want every project's test Deployments to start with the same set of development values.

1. In the Astro UI, go to **Environment** > **Airflow Variables**.
2. Click the variable that you want to share with all Astro IDE projects.
3. Click the **Edit** icon next to **Auto-linking** in the variable details.
4. In the **Edit Airflow Variable** dialog, under **Auto-linking**, toggle **ALL ASTRO IDE PROJECTS** to **On**.
5. Click **Update Airflow Variable**.

### Variable precedence in Astro IDE test Deployments

When an Astro IDE project starts a test Deployment, the Deployment receives the union of:

* Variables linked to the project, including Workspace-level variables linked to the project and variables scoped directly to the project.
* Workspace-level variables linked to that Deployment.

If both the project and a linked Deployment define an Airflow variable with the same key, the project-level value takes precedence in the test Deployment.

## Migrate existing Airflow variables to the Environment Manager

If you have Airflow variables defined in a Deployment's Airflow metadata database, you can move them into the Environment Manager so that they can be reused across Deployments and Astro IDE projects. See [Migrate existing objects to the Environment Manager](/docs/astro/migrate-metadata-db-to-environment-manager).

## Promote a Deployment-scoped Airflow variable to the Workspace

If you created an Airflow variable on a single Deployment and later want to reuse it across other Deployments or Astro IDE projects, you can promote it to the Workspace level from the Deployment's Environment tab. See [Promote a Deployment environment object to the Workspace](/docs/astro/migrate-metadata-db-to-environment-manager#promote-a-deployment-environment-object-to-the-workspace).


# Create a Deployment
Source: https://astronomer.io/docs/astro/create-deployment

Learn how to create an Astro Deployment. After you’ve created a Deployment, you can deploy Dags to it from the Astro command-line interface (CLI), or from a continuous integration and continuous delivery (CI/CD) pipeline.

An Astro Deployment is an Airflow environment that is powered by [Astro Runtime](/docs/runtime/runtime-image-architecture). It runs all core Airflow components, including the Airflow webserver, scheduler, and workers, plus additional tooling for reliability and observability.

There are multiple ways to create an Astro Deployment:

* Manually, using the Astro UI. This is the most basic way to create a Deployment and is the focus of this document. You can create a Deployment in the UI using either:
  * Quick Start: (Default) Rapid creation with predefined templates for the most common Deployment types.
  * Advanced configuration: Full control over Deployment details.
* Programmatically, using [`astro deployment create`](/docs/cli/v1.43/astro-deployment-create).
* Programmatically, using a Deployment template file. See [Manage Deployments as code](/docs/astro/manage-deployments-as-code).
* Programmatically, using [Astro Terraform Provider](https://registry.terraform.io/providers/astronomer/astro/latest/docs/resources/deployment).

<Tip>
  When using the Astro UI, you can select either a quick start template for simplicity or switch to advanced configuration for advanced configuration.
</Tip>

## Execution mode

To create a Deployment, you can specify **Hosted** or **Remote** execution mode in the advanced configuration of the Deployment. Read below for a summary of the two modes and see [Execution mode](/docs/astro/execution-mode) for more information.

### Hosted execution

Hosted execution mode is Astronomer-Hosted execution and orchestration for hands-off infrastructure management. Hosted is the default execution that offers a convenient, secure, and stable way to run Airflow workloads without having to manage workers in your environment.

### Remote Execution

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

Remote Execution mode is available on dedicated clusters to run tasks in your hardware or private clouds with only outbound connections to Astro’s Orchestration Plane. Sensitive data stays local, ideal for regulated or multi-regional deployments. Choose this option if:

* You need to run Airflow tasks in on-prem or edge environments without exposing inbound connections.
* Data locality and sovereignty are critical, ensuring sensitive or compliance-bound data stays in your environment.
* You want to avoid inbound firewall changes, simplifying network security by using only outbound connections.
* Your workloads require runtime flexibility, allowing execution on Kubernetes, bare metal, or other infrastructure.
* Strong isolation of secrets, data, and logs is a priority.
* You need scalability for high-throughput workflows, large-memory tasks, or GPU-accelerated AI workloads.

### Cluster type

To create a Deployment, you must choose a cluster type to host the Deployment:

* A **standard cluster** is the default cluster type and the quickest way to get an Airflow environment up and running on Astro. A standard cluster is a multi-tenant cluster managed by Astronomer where each Deployment exists in its own dedicated Kubernetes namespace. To run a Deployment in a standard cluster, you select a cloud provider and region when you create the Deployment. Then, Astro automatically creates your Deployment in an existing standard cluster based on your configuration.

* A **dedicated cluster** is a single-tenant Kubernetes cluster that's used exclusively by your team. Choose this option if:

  * You need private networking support between Astro and your cloud or on-premise data services.
  * You want to use a specific cloud provider or region that is not supported on standard clusters.
  * You need to run Airflow environments in separate clusters for business or security reasons.

    Note that due to expanded resource usage, dedicated clusters cost more than standard clusters. If no dedicated clusters are available to select, see [Create a Dedicated cluster](/docs/astro/create-dedicated-cluster) to create a new one.

<Note>If your Organization has the **Enforce Dedicated Clusters** policy enabled, the Standard Cluster option isn't available when creating a Deployment. See [Enforce dedicated clusters for new Deployments](/docs/astro/organization-settings#enforce-dedicated-clusters-for-new-deployments).</Note>

After you create a Deployment, you can deploy Dags to it using the Astro CLI on your local machine or a continuous integration/continuous delivery (CI/CD) tool. All Dags and tasks on Astro are executed within a Deployment.

Every Deployment is hosted on an Astro cluster with its own dedicated resources that you can [customize](/docs/astro/deployment-resources) to fine-tune your resource usage. To restrict communication between Deployments, resources for each Deployment are isolated within a corresponding Kubernetes namespace. See [Deployment network isolation](/docs/astro/data-protection#deployment-network-isolation).

<Info>
  If you're migrating to Astro from OSS Airflow or another Astronomer product, and you currently use an older version of Airflow, you can still create Deployments with the corresponding version of Astro Runtime even if it is deprecated according to the [Astro Runtime maintenance policy](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-maintenance-policy). This allows you to migrate your Dags to Astro without needing to make any code changes and then immediately upgrade to a new version of Airflow. Note that after you migrate your Dags, Astronomer recommends upgrading to a supported version of Astro Runtime as soon as you can.

  See [Run a deprecated Astro Runtime version](/docs/runtime/upgrade-astro-runtime#run-a-deprecated-astro-runtime-version).
</Info>

## Prerequisites

* A [Workspace](/docs/astro/manage-workspaces)

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Create a Deployment

<Tabs>
  <Tab title="New Astro UI">
    <Steps>
      <Step title="Go to Deployments">
        In the Astro UI, go to **Deployments** and click **+ New Deployment**.
      </Step>

      <Step title="Select a Workspace">
        Select a **Workspace** from the dropdown.
      </Step>

      <Step title="Choose a template or switch to custom configuration">
        In the **Quick Start** panel, enter a **Name** and choose a Deployment template:

        * **Development**: Low-cost and secure environment for experimentation and rapid iteration.
        * **Pre-Production**: Scalable and reliable validation environment for production readiness.
        * **Production**: Highly available and reliable environment for business-critical workloads.

        For more options, click **Switch to custom configuration**. You can also change these settings after the Deployment is created. See [Deployment resources](/docs/astro/deployment-resources).
      </Step>

      <Step title="Choose a cluster type">
        Under **Cluster**, select a cluster type:

        * For **Standard Cluster**, choose a cloud provider and region.
        * For **Dedicated Cluster**, select a cluster from the dropdown, or click **Create New Cluster** if you need a new one.

        The cluster can't be modified after the Deployment is created.
      </Step>

      <Step title="Create the Deployment">
        Click **Create Deployment**.
        The status displays as **Creating** until healthy and the Airflow UI becomes available for use.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Legacy UI">
    <Steps>
      <Step title="In the Astro UI, go to Deployments">
        Open the **Deployments** section of the Astro UI and click **+ Deployment**. Enter a unique **Deployment Name**.
      </Step>

      <Step title="Select a Deployment Template or switch to advanced configuration">
        The **Quick Start** panel appears by default and requires you to select from three templated options:

        * **Development**: Low-cost and secure environment for experimentation and rapid iteration.
        * **Pre-Production**: Scalable and reliable validation environment for production readiness.
        * **Production**: Highly available and reliable environment for business-critical workloads.

        To use a template, select the most appropriate card and complete the required fields. For advanced resource or execution configuration, select **Switch to advanced configuration**.

        <Warning>
          Switching from advanced configuration back to quick start resets all custom settings after confirmation.
        </Warning>
      </Step>

      <Step title="(Optional) Configure Wake Schedules for Development Mode Deployments">
        When you select the Development template or enable Development Mode, you can define schedules to control when the Deployment is available for task execution. Set timezone, days of the week, and start/end times.
      </Step>

      <Step title="Choose cluster type">
        Select a **Cluster Type**.

        * For **Standard Cluster**, choose a cloud provider and region.
        * For **Dedicated Cluster**, select a cluster from the dropdown or create a new one if required.
      </Step>

      <Step title="(Optional) Configure advanced resources and execution settings">
        For advanced use cases, you can further customize details such as execution mode, executor type, and worker resources by switching to advanced configuration.
        If you don't have specific requirements, the default options are suitable for most workflows.

        For a complete list of available options and resource settings, see [Deployment resources](/docs/astro/deployment-resources).
      </Step>

      <Step title="Create the Deployment">
        Click **Create Deployment**.
        The status displays as **Creating** until healthy and the Airflow UI becomes available for use.
      </Step>
    </Steps>
  </Tab>
</Tabs>

For more information about possible Deployment health statuses, see [Deployment health](/docs/astro/deployment-health-incidents).

## Next steps

* [Deployment settings](/docs/astro/deployment-settings)
* [Set environment variables on Astro](/docs/astro/environment-variables)
* [Authenticate an automation tool to Astro](/docs/astro/automation-authentication)


# Customize Otto
Source: https://astronomer.io/docs/astro/customize-otto

Tailor Otto to your team by adding team conventions, reusable skills, and permission rules that Otto applies to every session.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

Otto becomes more useful the more it knows about how your team works. This page describes the ways you can customize Otto so that every engineer on the team benefits from shared conventions, reusable workflows, and team-wide permission rules.

When Otto is customized, new engineers get your conventions from day one, corrections one engineer makes compound across the team, and operational knowledge stops living only in senior engineers' heads.

## What you can customize

Otto draws on four categories of team-specific customization. Each lives in files you commit to your repository, so updates propagate through your standard Git workflow.

* **Memory** — team conventions, naming patterns, connection configs, retry policies, approved operators, and other knowledge you want Otto to apply automatically. Memory files live in `.astro/memory/`. See [Memory](/docs/astro/otto-memory).
* **Skills** — reusable, structured workflows the team runs often, such as deploy patterns, testing patterns, and migration playbooks. Skills live in `.astro/otto/skills/`. See [Skills](/docs/astro/otto-skills).
* **Permissions** — allow, ask, and deny rules that control what Otto can do, including file rules, command patterns, and permission modes. Rules live in `.astro/otto/permissions.json`. See [Permissions](/docs/astro/otto-permissions).
* **Extensions** — toggles for bundled Otto features such as Dag validation, memory commands, and hosted skills. Extension settings live in `.astro/otto/extensions.json`. See [Extensions](/docs/astro/otto-extensions).

<Note>
  In addition to the files in `.astro/`, Otto reads `AGENTS.md` and `CLAUDE.md` files for context. Otto loads them from `~/.astro/otto/AGENTS.md` or `~/.astro/otto/CLAUDE.md`, then walks up from the current working directory to `/`. When both files exist in the same folder, Otto prioritizes `AGENTS.md` over `CLAUDE.md`.
</Note>

## Recommended project layout

A customized Astro project typically looks like:

```text wrap theme={null}
my-astro-project/
  dags/
  .astro/
    memory/
      MEMORY.md
      conventions.md
      connections.md
      operators.md
    otto/
      skills/
        deploy-to-staging/
          SKILL.md
      permissions.json
      extensions.json
```

Commit these files to your repository. When team members pull the latest main branch, their next Otto session loads the updated customizations automatically.

## Get started

Customize Otto by working through the agent. Describe what you want Otto to remember, write, or enforce, and Otto produces the corresponding files in `.astro/`. Each file is plain Markdown or JSON, so you can review, edit, and commit changes in your Git diff like any other code change.

### Seed memory from an established project

Best for teams whose conventions live in code and conversations but aren't documented yet.

Run Otto in your project folder:

```sh wrap theme={null}
astro otto
```

Then run `/bootstrap` in the Otto interface. Otto reviews your git history and available GitHub context to extract durable learnings, then writes memory files for the durable patterns it finds. Every read and write happens as a visible tool call, so you can see what Otto pulls from history and what it persists.

<Note>
  For `/bootstrap` to include GitHub PR history, install and authenticate the [GitHub CLI](https://cli.github.com/). Without it, Otto bootstraps from local git history alone.
</Note>

### Add memory from existing standards

Best for teams that already have conventions documented elsewhere, such as a wiki page, a `STANDARDS.md` file, or an onboarding guide.

Paste the content into an Otto session and ask Otto to extract memory files from it. For example:

```text wrap theme={null}
> Here is our team's convention doc. Read it and propose memory files for the parts that should apply to every Dag we write.
```

Otto produces draft memory files for review, following the same review pattern as `/bootstrap`.

### Capture learnings with `/remember`

During an active session, run `/remember` to have Otto review the conversation and generate memory files from durable points. Use `/remember` to grow your team's memory organically as you work.

### Ask Otto to write skills and permission rules

For workflows you run often, describe the workflow to Otto. For example:

```text wrap theme={null}
> Write a skill for our staging deploy that runs make test, deploys to the staging Deployment, and tails the scheduler logs.
```

Otto generates a `SKILL.md` under `.astro/otto/skills/` for you to review. The same pattern applies to permission rules. Describe what Otto should and shouldn't do, and Otto produces or updates `.astro/otto/permissions.json`.

### Commit and roll out

After Otto writes a customization, the changes appear in your Git diff. Review, edit if needed, and commit them. On the next pull, every engineer's Otto sessions load the new customizations automatically.

## Choose a model

You can pick your own model for an Otto session. Otto supports models from OpenAI, Anthropic, and Google through the Astronomer Gateway. Run `/model` in an interactive session to browse what's available to you and switch the active model mid-session. To pin a model at launch, pass `--model <id>`.

See [`astro otto` model selection](/docs/cli/v1.43/astro-otto#model-selection) for the full reference.

## Next steps

* [Memory](/docs/astro/otto-memory) — Memory tiers, file format, and how Otto loads context.
* [Skills](/docs/astro/otto-skills) — How to author local skills for reusable workflows.
* [Permissions](/docs/astro/otto-permissions) — Configure allow, ask, and deny rules for tool calls.
* [Settings](/docs/astro/otto-settings) — Configuration files, environment variables, and settings precedence.


# Dag versioning
Source: https://astronomer.io/docs/astro/dag-versioning

Track dag versions across runs using Airflow 3 dag versioning and dag bundles on Astro.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Dag versioning is a feature introduced in Airflow 3. Previously, in Airflow 2, Airflow automatically used the most recent dag code. This meant that changing your dag could affect observability into previous task runs or affect the success of in-progress dags. Airflow 3 addresses these issues with *Dag bundles* and *Dag versioning*.

Airflow automatically versions your dag runs in the Airflow UI. When you make significant changes to your Dags, like changing task behavior, Airflow creates a new version. Each dag run is also associated with a specific version visible in the Airflow UI. This allows you to quickly identify the outcome of particular code changes.

Dag bundles include the dags files and their supporting files and can also be versioned depending on the bundle type you choose. These versions are separate from dag versions, and reflect the version of the source. The default, `LocalDagBundle` is not versioned, but some dag bundles are. The `GitDagBundle` is versioned by the underlying backend, Git. Every commit made to the configured Git repository creates a new version, even if a commit does not change your dag code.

Read more about [Dag versioning](/docs/learn/airflow-dag-versioning), including details about how to configure a dag bundle for OSS Airflow.

## Dag bundles on Astro Hosted

On Astro Hosted, you don't need to configure a dag bundle backend. As a fully managed service, Astro Hosted configures a versioned dag bundle for you that contains the dags pushed to the Deployment.

Astro Hosted does not support configuring custom dag bundles.

## Dag bundles on Astro with Remote Execution

For Remote Execution mode Deployments, you must configure a dag bundle backend for the Remote Execution Agents in your environment. See [Configure dag bundles for Remote Execution](/docs/astro/remote-execution-configure-dag-sources) for more information on how to configure the dag bundle backend as well as differences between bundle types.


# Write and run dags on Astro
Source: https://astronomer.io/docs/astro/dags-overview

Learn how to use code-level features specific to Astro in your Apache Airflow dags.

Astro includes several features that enhance the Apache Airflow development experience, from dag writing to testing. To use these features, you might need to modify how you write your dags and manage the rest of your code.

Use this documentation to learn about the key differences between managing dags on Astro versus on other platforms.

## Project structure

To develop and run dags on Astronomer products, your dags must belong to an Astro project. An *Astro project* contains all files required to run your dags both locally and on Astronomer. In addition to dag files, an Astro project includes Python and OS-level packages, your Astro Runtime version, and any other files your workers need to access when you run tasks.

See [Create an Astro project](/docs/cli/v1.43/get-started-cli) to learn more about how to create and run Astro projects.

## Airflow and Astro Runtime versioning

When you migrate to Astro from another Apache Airflow service, there are a few differences to note with regards to how Astro handles versioning, upgrades, and runtime builds.

* **Astro Runtime versioning scheme**: Each Astro project uses a specific version of Astro Runtime, which is Astronomer's version of Apache Airflow that includes additional observability and performance features. Each Astro Runtime version corresponds to one Apache Airflow version, but the versioning scheme is different. For example, Astro Runtime 11.3.0 corresponds to Airflow 2.9.1. See [Astro Runtime maintenance policy](/docs/runtime/runtime-version-lifecycle-policy).
* **Upgrading Airflow**: As you continue to develop within an Astro project, you'll need to upgrade your Astro Runtime version to take advantage of new Astro and Apache Airflow features and fixes. Your Astro Runtime version is defined in the `Dockerfile` of your Astro project. Unlike with open source Airflow, upgrading Astro Runtime does not require you to manually migrate your metadata database. To upgrade your version of Airflow, you only have to change the Astro Runtime version listed in your project's Dockerfile and rebuild your project. See [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime) for instructions on how to upgrade.
* **Runtime Arguments**. Your Dockerfile is also where you can define additional runtime arguments that trigger whenever your project builds. You can use these arguments to mount resources, such as API tokens, to your Airflow environment without including the specific resources in your project files. See [Customize your Dockerfile](/docs/cli/v1.43/customize-dockerfile) for more details.

## Testing environments

There are two ways to run dags within the Astro ecosystem: Either in a local Airflow environment or in an Astro Deployment. Each has its own purpose in the development lifecycle.

* **Local Airflow environment**: You can run dags on your local machine using the Astro CLI. This development method is most useful if you need to quickly iterate and test changes, such as when fixing a bug, or you're just getting started with Airflow. Testing locally is free and open source.
* **Deployment**: When you deploy your dags to Astro, they run on a managed Deployment. Use Deployments to run production code, or create a development Deployment to test changes over a longer period of time than you could in a local Airflow environment. To test code on a Deployment, you must have an [Astro account](/docs/astro/log-in-to-astro) and an administrator on your team must grant you access to the Deployment. For more information about how to structure Deployments for specific development workflows, see [Connections and branch-based deploys](/docs/astro/best-practices/connections-branch-deploys).

### Unit tests

Whether you're building your project locally or deploying to Astro, you can run unit tests with the Astro CLI to ensure that your code meets basic standards before you run your dags. The Astro CLI includes a default set of unit tests which you can run alongside your own tests in a single sequence. See [Test your dags](/docs/cli/v1.43/test-your-astro-project-locally) for more information.

## Airflow feature enhancements

Astro includes several features that enhance open source Apache Airflow functionality.

* On Astro, the Astro UI renders [Airflow tags](https://airflow.apache.org/docs/apache-airflow/stable/howto/add-dag-tags.html) defined in your dags. Use tags to filter dags across all Deployments from a single screen.
* The [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager) allows you to create and manage Airflow connections directly from the Astro UI. Instead of being limited to defining connections in the Airflow UI or with a secrets manager, you can create connections from the Environment Manager on Astro and use the connections in your local Airflow environment or across multiple Deployments and Workspaces.
* Astro has built-in infrastructure to run the `KubernetesPodOperator` and Kubernetes executor, such as default Pod limits and requests. Task-level resource limits and requests are set by default on Astro Deployments, which means that tasks running in Kubernetes Pods never request more resources than expected. See [Run the Kubernetes executor](/docs/astro/kubernetes-executor) and [Run the `KubernetesPodOperator`](/docs/astro/kubernetespodoperator) for more specific instructions and examples.

## Dag observability

In local Airflow environments, you can use the Airflow UI to check your dag runs, task logs, and component logs just as you would in any other Airflow environment.

In the Astro UI, you have access to the **DAGs** page in addition to the Airflow UI. From here, you can manage dag and task runs for any Deployment in your Workspace.

Astronomer recommends using the UI that best fits the need your team. If you prefer managing your dags from a single place and find the way that the way the Astro page is designed helpful, you don't need to use the Airflow UI. If you're a longtime user of Apache Airflow, you might feel more comfortable in the Airflow UI and don't need to use the Astro UI.

See [View logs](/docs/astro/view-logs) and [Manage dag runs](/docs/astro/manage-dags) for more information.

## Airflow alerts

Astro supports a set of alerting features that in many cases replace Apache Airflow SLAs or failure notification. There are some circumstances where Astronomer recommends configuring Astro alerts instead of Airflow SLAs or failure notifications because it can simplify your dag code and make it easier to manage alerts across multiple dags. See [When to use Airflow or Astro alerts for your pipelines on Astro](/docs/astro/best-practices/airflow-vs-astro-alerts).

## Astronomer Cosmos

In addition to its commercial products, Astronomer maintains [Cosmos](https://www.astronomer.io/cosmos/) an open source tool for orchestrating dbt Core projects from a dag. Cosmos gives you more visibility into every step of your dbt project and lets you use Airflow's data awareness features with your dbt models. See the [Cosmos documentation](https://astronomer.github.io/astronomer-cosmos/) for more information.


# Deploy code to Astro
Source: https://astronomer.io/docs/astro/deploy-code

Learn about the different ways you can deploy code to Astro.

To run your code on Astro, you need to deploy it to a Deployment. You can deploy part or all of an Astro project to an Astro Deployment.

There are several options for deploying code to a Deployment:

* **Project deploys**: Run `astro deploy` to build every non-Dag file in your Astro project as a Docker image and deploy the image to all Airflow components in a Deployment. This includes your `Dockerfile`, plugins, and all Python and OS-level packages. Dags are deployed separately to each Airflow component through a sidecar container. See [Deploy a project image](/docs/astro/deploy-project-image).
* **Dag-only deploys**: Run `astro deploy --dags` to deploy only your Dag files to Astro. If you only need to deploy Dag changes, running this command is faster than running `astro deploy` since it doesn't require installing dependencies. See [Deploy Dags](/docs/astro/deploy-dags).
* **Image-only deploys**: Run `astro deploy --image` to build and deploy your Astro project configurations as a Docker image without deploying your Dags. This is useful if you have a multi-repo CI/CD strategy, and you want to deploy your Dags and project configurations from different repositories or storage buckets. See [Image-only deploys](/docs/astro/deploy-dags#trigger-an-image-only-deploy).
* **dbt deploys**: Run `astro deploy --dbt` to deploy only a dbt project to Astro. See [Deploy dbt projects to Astro](/docs/astro/deploy-dbt-project). You must use this command to deploy dbt projects to Astro.
* **Remote Execution project deploys**: For Deployments configured for [Remote Execution](/docs/astro/execution-mode#remote-execution), use `astro deploy` to build and deploy the Runtime (orchestration plane) image, and use `astro remote deploy` to build and push client images for your Remote Execution Agents. See [Initialize and deploy Remote Execution projects](/docs/astro/deploy-project-remote-execution) for full instructions.
* **GitHub integration deploys**: The Astronomer GitHub integration allows you to map a branch from your GitHub repository to directly deploy to Astro after you merge pull requests to the mapped branch. See [Deploy code with the Astro GitHub integration](/docs/astro/deploy-github-integration).

For each deploy option, you can either trigger the deploy manually or through CI/CD. CI/CD pipelines can include both image deploys and Dag-only deploys, and they can deploy to multiple different Deployments based on different branches in your git repository. See [CI/CD overview](/docs/astro/set-up-ci-cd).

<Note>For Dags running on Astro using Hosted [execution mode](/docs/astro/execution-mode), Astro configures a specialized versioned Dag bundle automatically, without any need for additional setup. Configuring custom Dag bundles and using several Dag bundles in the same Astro Deployment is only supported when using [Remote Execution mode](/docs/astro/execution-mode#remote-execution). For more information, see [Configure Dag bundles for Remote Execution](/docs/astro/remote-execution-configure-dag-sources).</Note>

If multiple deploys are triggered simultaneously or additional deploys are triggered while a deploy is still processing, whether manually or through CI/CD, the first deploy is processed and then the subsequent deploys are completed. This behavior is different from how Astro processes simultaneous code deploys with the [GitHub integration](/docs/astro/deploy-github-integration#deploys), which cancels the first deploy, and applies the most recent.

## See also

* [Create an Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project)
* [Develop your Astro project](/docs/cli/v1.43/develop-project)


# Deploy Dags to Astro
Source: https://astronomer.io/docs/astro/deploy-dags

Learn about the different ways you can deploy code to Astro.

Dag-only deploys are the fastest way to deploy code to Astro. They are recommended if you only need to deploy changes made to the `dags` directory of your Astro project.

Dag-only deploys are enabled by default on all Deployments on Astro Hosted. When they are enabled, you must still [deploy your project image](/docs/astro/deploy-project-image) when you make a change to any file in your Astro project that is not in the `dags` directory, or when you [upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime).

Dag-only deploys have the following benefits:

* Dag-only deploys are significantly faster than project deploys.
* Deployments pick up Dag-only deploys without restarting. This results in a more efficient use of workers and no downtime for your Deployments.
* If you have a CI/CD process that includes both Dag and image-based deploys, you can use your repository's permissions to control which users can perform which kinds of deploys. See [Dag deploy templates](/docs/astro/ci-cd-templates/template-overview#dag-deploy-templates) for how you can set this up in your CI/CD pipelines.
* You can use Dag deploys to update your Dags when you have slow upload speeds on your internet connection.

## Trigger a Dag-only deploy

Triggering a Dag-only deploy pushes Dags to Astro and mounts them to the workers and schedulers in your Deployment. Dag-only deploys do not disrupt running tasks and do not cause any components to restart when you push code. If you deploy changes to a Dag that is currently running, active task runs finish executing according to the code from before you triggered a deploy. New task runs are scheduled using the code from your latest deploy.

Run the following command to deploy only your `dags` directory to a Deployment:

```sh wrap theme={null}
astro deploy --dags
```

<Note>
  When you run `astro deploy --dags`, all existing Dags in the Deployment are replaced by the new Dags that you are deploying. The deployment process replaces the entire Dag bundle with the new set of Dags, rather than incrementally adding or updating individual Dag files.
</Note>

<Warning>
  Unlike image deploys, Dag deploys do not include pytest as part of the deploy process by default. This means that Dag deploys might include faulty code that would otherwise be caught in an image deploy.

  To have the same testing process in Dag deploys, run `astro deploy --dags --pytest`. Note that this can increase the time your deploys take. See [Test your Astro project](/docs/cli/v1.43/test-your-astro-project-locally) for more information about how to use the Astro CLI's pytest features.
</Warning>

## Trigger an image-only deploy

Even if you primarily use Dag-only deploys, you still need to occasionally make image deploys to update your Astro Runtime version or install dependencies. However, depending on your CI/CD strategy, triggering a full project deploy with `astro deploy` might affect your existing Dags.

When you trigger an image-only deploy, it builds every non-Dag file in your Astro project as a Docker image and deploys the image to all Airflow components in a Deployment. This includes your `Dockerfile`, plugins, and all Python and OS-level packages. Dags are not deployed, and your Deployment Dag folder and Dag bundle version are not affected. Because an image-only deploy ignores files in the `dags` directory during the Docker image build, it's important to place any files needed for the build in an appropriate directory such as the `include` directory. See [Airflow best practices](/docs/learn/dag-best-practices#use-a-consistent-file-structure) for information on recommended file structure and other Dag writing best practices. Running the `astro deploy --image` command will also restart the Kubernetes Pods. Image-only deploys are only available when you have Dag-only deploys enabled.

Run the following command to build and deploy only your non-dag files to a Deployment:

```sh wrap theme={null}
astro deploy --image
```

<Tip>If you use [prebuilt Docker images](/docs/astro/deploy-project-image#deploy-a-prebuilt-docker-image) for your image deploys, you can use both the `--image-name` and `--image` flags to update your image without updating your `dags` folder.</Tip>

## Enable or disable Dag-only deploys on a Deployment

If you have Workspace Owner permissions, you can enable or disable Dag-only deploys for a Deployment at any time. After you disable Dag-only deploys and trigger a code deploy:

* Any changes to your Dag code are deployed as part of your Astro project Docker image.
* In the Astro UI, your Deployment **DAG Bundle Version** doesn't update when you deploy code.

To determine if turning off Dag-only deploy functionality is the right choice for your organization, contact [Astronomer support](https://cloud.astronomer.io/open-support-request).

Before you enable or disable Dag-only deploys on a Deployment, ensure the following:

* You have access to the latest version of your Deployment's Astro project.
* You can update your Deployment using the Astro CLI.

<Danger>Carefully read and complete all of the following steps to ensure that disabling or enabling Dag-only deploys doesn't disrupt your Deployment. **Immediately after you update the setting, you must trigger an image deploy to your Astro Deployment using `astro deploy`**. If you don't complete this step, you can't access your Dags in the Airflow UI.</Danger>

### Prerequisites

Before you enable or disable Dag-only deploys on a Deployment, ensure the following:

* You have Workspace Owner permissions for the Workspace that hosts the Deployment.
* You have access to the latest version of your Deployment's Astro project.
* You can update your Deployment using the Astro CLI.

### Enable Dag-only deploys

Dag-only deploys are enabled by default on Astro Hosted. While enabled:

* You can run `astro deploy --dags` to deploy only Dags to your Deployment.
* In the Astro UI, your Deployment **DAG Bundle Version** updates when you trigger an image deploy or a Dag-only deploy.
* When you only deploy Dags, it doesn't automatically upgrade your Runtime version. You must periodically complete a full image deploy to [upgrade the Runtime version](/docs/runtime/upgrade-astro-runtime).
* Your Deployment includes infrastructure for deploying your Dags separately from your project image. See [What happens during a code deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy).

1. Run the following command to enable dag-only deploys:

   ```sh wrap theme={null}
   astro deployment update --dag-deploy enable
   ```

2. Run the following command to deploy all of the files in your Astro project as a Docker image:

   ```sh wrap theme={null}
   astro deploy
   ```

### Disable Dag-only deploys

After you disable Dag-only deploys:

* You can't run `astro deploy --dags` to trigger a Dag-only deploy to your Deployment.
* Any changes to your Dag code are deployed as part of your Astro project Docker image.
* In the Astro UI, your Deployment **DAG Bundle Version** doesn't update when you deploy code.
* Your Deployment doesn't include infrastructure for deploying your Dags separately from your project image. See [What happens during a code deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy).

1. Run the following command to disable dag-only deploys:

   ```sh wrap theme={null}
   astro deployment update --dag-deploy disable
   ```

2. Run the following command to deploy all of the files in your Astro project as a Docker image:

   ```sh wrap theme={null}
   astro deploy
   ```


# Deploy dbt projects to Astro
Source: https://astronomer.io/docs/astro/deploy-dbt-project

Learn how to deploy and run dbt projects with Apache Airflow on Astro.

Astro supports a range of options when it comes to adding your dbt project to your Deployment. To orchestrate dbt jobs with Apache Airflow, you first need to deploy your dbt project to Astro along with your dags and the rest of your Airflow code.

This guide includes information on the different options your team has for deploying dbt code to Apache Airflow and Astro. Your team can choose the option that best fits your team's software development lifecycle.

For more information and recommendations on using dbt with Apache Airflow and Astro, see [Orchestrate dbt Core jobs with Airflow and Cosmos](/docs/learn/airflow-dbt).

## Repository strategy

Depending on your organization's software development lifecycle, there are three ways you can organize your dbt project relative to your Astro project:

* In the same Git repository and directory as your Astro project.
* In the same Git repository but in a separate directory.
* In a separate Git repository.

Astro supports all three methods, but Astronomer recommends having your dbt project in the same Git repository as your Astro project, but in a different directory. Then, you can use dbt deploys to independently deploy dbt code to Astro from your dbt directory without needing to deploy either a full Astro project image or your dags. This strategy allows your team maintaining dbt to work independently from your team managing Airflow dags, but team members can all see shared code in a single Git repository.

You can see additional recommendations for Astro repository strategy, in the [Repo strategy best practices](/docs/astro/best-practices/repo-structure) guide.

## Feature overview

Astro supports two basic dbt code deploy strategies. You can:

* **Include dbt code in your Astro project**: Make dbt code directly available in the Docker image powered by Astro Runtime that contains your Astro project. This approach works best for small teams just starting to integrate dbt code in Astro.
* **Use dbt Deploys**: Independently deploy bundles of dbt code directly to Astro, outside the context of your Astro project and Docker image. Astronomer recommends this approach for teams who have dbt code in a dedicated directory or Git repository.

<a />

## Option 1: Include dbt code in your Astro project

The most direct way to set up dbt on Astro is by including dbt code in your full image. To accomplish this, Astronomer recommends adding the directory `/dbt` to the top level of your Astro project and including individual dbt projects inside this directory.

<Tip> To see an example of this pattern, check out the demo repository of [Astronomer Cosmos](https://github.com/astronomer/cosmos-demo). </Tip>

Finally, to push your new dbt code to your Deployment, perform a deploy with the Astro CLI by running:

```sh wrap theme={null}
astro deploy
```

While this is the quickest way to get dbt code in your Deployment, there can be drawbacks if your team makes more frequent changes to dbt code or uses more advanced CI/CD. These include:

1. Slower build times when changing only dbt code.
2. dbt code must live in the same repository and directory as your Astro project.
3. Multiple teams developing Airflow and dbt separately must re-deploy the entire Astro project for each iteration.

If these downsides become applicable to your team, try Option 2.

<a />

## Option 2: Use dbt Deploys to independently ship dbt code

<Note>There is a hard limit of 10 dbt bundles per Astro Deployment</Note>

dbt Deploys allow you to easily deploy your dbt project to Astro without needing complex processes to incorporate your two sets of code. When you use a dbt Deploy, Astro bundles all files in your dbt project and pushes them to Astro, where they are mounted on your Airflow containers so that your dags can access them. This allows you to deploy dbt code without requiring you to use a full Astro image deploy.

### Prerequisites

* An Astro Deployment
* An Astro project. Astronomer supports both dbt Cloud and dbt Core.
* A dbt project
* The [Astro CLI v1.28 or greater](/docs/cli/v1.43/install-cli)

### Step 1: (Optional) Deploy your full Astro image

In order to first deploy a dbt project to Astro, Astronomer recommends that you have an Astro project already running on your Deployment with dags that need to read from dbt. That way, your dbt project will be read and used when you deploy it.

If you are using a new Deployment, first deploy your Astro project with the Astro CLI by running:

```sh wrap theme={null}
astro deploy
```

### Step 2: Deploy your dbt project

Next, navigate to the directory of your dbt project. This directory should include the `dbt_project.yml` file, as in the case of the [Classic dbt Jaffle Shop](https://github.com/dbt-labs/jaffle-shop-classic?tab=readme-ov-file) in the following example:

```text wrap theme={null}
.
├── etc
├── models
├── seeds
├── .gitignore
├── LICENSE
├── README.MD
└── dbt_project.yml
```

From the CLI, run the following command to deploy your dbt project. The command prompts you to choose the Deployment that you want to deploy your dbt project to.

```sh wrap theme={null}
astro dbt deploy
```

By default, `astro dbt deploy` attaches the dbt code to the default path of `/usr/local/airflow/dbt/<dbt-project-name>`.

See [`astro dbt deploy`](/docs/cli/v1.43/astro-dbt-deploy) for more information about this command.

<Tip>
  If your dbt code is accessed at a different path or folder than the default path, specify a custom mount path with the following flag:

  ```sh wrap theme={null}
  astro dbt deploy --mount-path /usr/local/airflow/dbt/example-dbt-project
  ```
</Tip>

Congratulations, you've added your dbt code to your Deployment! Check the [Astro UI](https://cloud.astronomer.io/) to see more information about your dbt Deploy.

<Frame>
  <img alt="Astro UI with dbt Deploys" />
</Frame>

### Delete your dbt project

If you want to remove dbt code from your deployment, you can also delete the dbt project from the Airflow environments where you deployed it. This command does not delete your dbt project source files. It only removes your project from the Airflow containers where it was mounted. When you run this command, you will be prompted to choose the Deployment from which you want to remove the project.

```sh wrap theme={null}
astro dbt delete
```

See [`astro dbt delete`](/docs/cli/v1.43/astro-dbt-delete) for more information about this command.


# Deploy code with the Astro GitHub integration
Source: https://astronomer.io/docs/astro/deploy-github-integration

Learn how to automatically deploy Apache Airflow code to Astro from GitHub with a built-in integration.

Astronomer's built-in GitHub integration is the fastest way to implement CI/CD for Apache Airflow and deploy code to Astro. Astro’s automatic deploy system both eliminates the need to implement GitHub Actions and gives you greater visibility into the code you’re running on Astro.

To deploy code through an integrated GitHub repository, you first connect a GitHub repository with your Astro project to an Astro Workspace. Then, you map a Git branch in that repository to an Astro Deployment. When a pull request is merged into your mapped branch, your code is automatically deployed to Astro.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

<Frame>
  <img alt="A diagram showing how branches in a GitHub repository directly correlate to specific Deployments in an Astro Workspace." />
</Frame>

Compared to deploying code manually using the Astro CLI or through a custom CI/CD process, using Astro’s GitHub integration:

* Allows you to enforce software development best practices without maintaining custom CI/CD scripts.
* Enables developers to iterate on Dag code quickly.
* Shows Git metadata directly in the Astro UI, including Git commit descriptions.
* Gives you greater visibility into the status and detailed logs of an individual deploy.

<Frame>
  <img alt="A diagram showing how when an engineer commits code to a branch, then after merging the branch to GitHub, the Astro project updates." />
</Frame>

## Best practices

To make the most out of the Astro GitHub integration, Astronomer recommends the following:

* Consider mapping long-lasting `main` and `dev` branches in GitHub to Deployments. For example, you would map a `main` branch to a production Deployment and your `dev` branch to your development Deployment. This approach is described in [Develop a CI/CD strategy](/docs/astro/set-up-ci-cd#multiple-environments).
* Run CI checks on your Dags and project code as part of your pull request review process. There is currently no testing included in the default GitHub deploy process, so ensure that your Dag changes pass unit tests before they’re merged and deployed.
* Write descriptive commit messages or pull request descriptions for any project changes. These messages appear in the Astro UI under **Overview** to give your team more context about each deploy.

## Limitations

The Astro GitHub integration isn't supported if any of the following are true:

* You build a custom Astro Runtime image using `docker build` as part of your deploy process, for example to [pull Python packages from a private repository](https://www.astronomer.io/docs/astro). However, using the `-base` distribution of Astro Runtime is supported.
* You need to deploy images from an external or private Docker registry.
* You use the `--image-only` flag in the Astro CLI, or you need to exclude Dags from specific deploys. This is usually the case if you have a multiple-repository deploy strategy as described in [Develop a CI/CD strategy](/docs/astro/set-up-ci-cd#multiple-repositories).
* Your repository is hosted in an on-premises GitHub Enterprise server. To deploy code from an on-premises repository, see [Private network CI/CD templates](/docs/astro/ci-cd-templates/github-actions-private-network).
* If you trigger a deploy that includes a downgrade of Astro Runtime, the deploy will silently fail. Astronomer recommends only downgrading Deployments using [rollbacks](/docs/astro/deploy-history#roll-back-to-a-past-deploy).

There is currently no limit or additional charge for deploying code to Astro with the GitHub integration, but Astronomer might contact you to discuss your usage if you trigger an unusually large number of deploys.

## Prerequisites

* Workspace Admin permissions on an Astro Workspace.
* At least one Astro Deployment.

**To connect to an existing repository**:

* A GitHub user account with read permissions to the repository containing your Astro project. If you don't have these permissions, send a request to your GitHub repository administrator when prompted by Astro. The read permissions are required so that you can map Deployments to repositories in the Astro UI.
* A GitHub repository that contains an [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project). The Astro project can exist at the root directory or any subdirectory in your repository.
* At least one branch, such as a `dev` or `production` branch, that you want to deploy to Astro.

**To create a new repository**:

* A GitHub user account with permission to create repositories. If you belong to a [GitHub Organization](https://docs.github.com/en/organizations/managing-peoples-access-to-your-organization-with-roles/roles-in-an-organization#permissions-for-organization-roles) and don't have these permissions, send a request to your GitHub repository administrator when prompted by Astro.
* After you create and connect to your repository from Astro, you need to add an [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project). The Astro project can exist at the root directory or any subdirectory in your repository.
* After you create and connect to your repository from Astro, you need to make at least one branch, such as a `dev` or `production` branch, that you want to deploy to Astro.

## Connect a GitHub repository to a Workspace

Before you begin, ensure that you’re logged in to GitHub with permissions to read code from the repository where you want to deploy code.

Open the **Git Deploys** page for your Workspace:

<Tabs>
  <Tab title="New Astro UI">
    In the Astro UI, go to **Settings** > **Workspaces**, select your Workspace, then click **Git Deploys**.
  </Tab>

  <Tab title="Legacy UI">
    In your Workspace, click **Workspace Settings** > **Git Deploys**.
  </Tab>
</Tabs>

Then, connect your repository:

1. Click **Authorize GitHub Application** to authorize Astro to a new GitHub account. A window appears instructing you to authorize the **Astro App** on your personal GitHub account. Follow the prompts to authorize the application.

<Frame>
  <img alt="The GitHub authorization screen for connecting a repository to GitHub. GitHub requests for the Astro App to have some of the permissions of your GitHub account" />
</Frame>

<Info>You only need to authorize the GitHub application once. Afterwards, you can connect to additional repositories directly from the Astro UI.</Info>

2. Return to the Astro UI. From the **Git Deploys** screen, click **Connect Repository**.
3. Select the organization that contains the repository you want to connect to your Deployment. If your GitHub organization isn't available, click **Add Organization on GitHub**. Select the organization that contains the repository you want to integrate with Astro, then click **Continue**. A new window prompts you to allow **Astro App** to access either all repositories or specific repositories within your GitHub Organization. Astronomer recommends **Only select repositories**. After you return to the Astro UI, refresh the page to make the GitHub organization appear as a selectable option.
4. Choose which repositories you want to enable the app for. You can either select an existing repository, or create a new repository. Then, click **Create and Connect Repository.**

<Frame>
  <img alt="&#x22;The GitHub installation screen for connecting a repository to Astro. GitHub requests for the Astro App to be installed in at least of your repositories." />
</Frame>

5. Configure the following fields:

   * **Repository:** Select the repository you want to integrate with Astro.
   * **Astro Project Path:** Specify the path to your Astro project relative to your GitHub repository directory, up to and including the Astro project folder. For example, for a project in `github-repository/myorg/myprojects/my-astro-project`, enter `myorg/myprojects/my-astro-project`. If the path is configured incorrectly, code doesn't deploy to Astro from GitHub. You can change this path after you create the integration by editing the **Branch Configuration** setting in the Astro UI.

6. Click **Connect Repository**.

7. Map specific branches in your repository to Deployments in your Workspace. For example, you can map a development branch and your production branch to separate Deployments, so that bugs in development don’t affect your production data pipelines. When you map a branch to a Deployment, any future commits to the Astro project in that branch trigger a code deploy to Astro.

<Info>You can add and edit branch mappings to connected GitHub repositories at any time from the **Git Deploy** page in the Astro UI.</Info>

Any commits to your mapped branches will now trigger a code deploy to the corresponding Deployment.

## Deploy from GitHub to Astro

To deploy code from your GitHub repository to Astro, you can:

* Make a direct commit to one of your mapped branches.
* Merge a pull request against one of your mapped branches.
* Select **Trigger Git Deploy** from the **More actions** menu in your Deployment settings. This deploys the latest commit from your branch. Astronomer recommends this action when your Git branch and Astro fall out of sync and you don't want to create a new commit to get them back in sync.
* Select **Trigger Git Deploy** from the **More actions** menu for a previously failed deploy. This redeploys that particular Git commit even if it's not necessarily the latest commit in your repository or branch. Astronomer recommends this action when a deploy fails and you want to try it again without creating a new commit.

Any of these actions triggers the Astro App to deploy your Astro project to the mapped Astro Deployment. When Dag-only deploys are enabled, your GitHub repository triggers:

* A Dag-only deploy if only your Dags are changed.
* A full project image deploy if you change a configuration in your project.

If Dag-only deploys are disabled, all code changes will trigger a full project image deploy. To learn more about Dag-only deploys, see [Deploy Dags to Astro](/docs/astro/deploy-dags).

You can check the status of your deploy in the Astro UI in the Deploy History listing on your Deployment's **Overview** tab.

## Review code deploys from the Astro UI

When you trigger a code deploy by committing a change to one of your mapped branches, details about the code deploy appear in the Astro UI. For past and currently running deploys, you can review:

* Whether a deploy was triggered by the integration.
* At what time the deploy was triggered.
* Which pull request or commit triggered the deploy. This includes the Git commit description and a link to your pull request if applicable.

<Frame>
  <img alt="The deploy history screen in the Astro UI with a GitHub deploy listed. The entry includes the commit that triggered the deploy and shows it was triggered by &#x22;GitHub App&#x22;" />
</Frame>

To review code deploys:

1. In the Astro UI, open your Deployment.
2. Click **Overview**.
3. Deploys triggered by the Astro App include a commit ID in their **Description** and are **Deployed By** the **GitHub App.** Click on a deploy triggered by the Astro App to see the logs for the deploy.

<Frame>
  <img alt="The status of a specific deploy in the Astro UI. The deploy is currently running and generating logs." />
</Frame>

## How it works

The Astro GitHub integration works through a [GitHub app](https://docs.github.com/en/apps/overview) that you install in your GitHub repository. The app is also authorized to act on behalf of your personal account so that it can see the GitHub organizations and repositories you have access to.

### Deploys

When you make a commit or merge a pull request to a mapped branch in your repository, the GitHub app sends a push event to the Astro API. After the Astro API confirms that the push event should result in a deploy, your project code is sent to Deployment-specific workers on Astro that deploy the code to your Deployment.

If an additional deploy is triggered while a deploy is currently processing for the same Deployment, Astro will terminate the first deploy and begin processing the second deploy. The first deploy will appear with a **Failed** state in your deploy history.

### Rollbacks

When you roll back a deploy created by the GitHub integration, Astro doesn't roll back the code in your GitHub repository. Therefore, rollbacks should only be used as a last resort due to the potential for your repository code to become out of sync with your deploy code.

If you do roll back a deploy, manually revert the code in your GitHub repository as soon as possible so that it matches the code running in your rolled back deploy. For more information, see [What happens during a rollback](/docs/astro/deploy-history#what-happens-during-a-deploy-rollback).


# Roll back to previous deploys using deploy histories
Source: https://astronomer.io/docs/astro/deploy-history

View a historical record of code deploys to an Astro Deployment and roll back to specific deploys when something goes wrong.

The **Overview** tab in the Astro UI shows you a record of all code deploys to your Deployment. Use this page to track the development of a Deployment and to pinpoint when your team made key changes to code.

Astronomer stores the image and Dags for all deploys made in the last 90 days. You can trigger a rollback to any of these deploys so that your Deployment starts running a previous version of your code.

Deploy rollbacks are an emergency option if a Deployment unexpectedly stops working after a recent deploy. For example, if one of your Dags worked in development but suddenly fails in a mission-critical production Deployment, you can roll back to your previous deploy to quickly get your pipeline running again. This allows you to troubleshoot the issue more thoroughly in development before redeploying to production. You can roll back to any deploy in the last three months regardless of your Runtime version, Dag code, or Deployment settings.

<Info>Astro supports rolling back from Airflow 3 to Airflow 2 but not all Airflow 3.x Deployments can be reverted to all Airflow 2.x versions. Review the [Airflow 3 to Airflow 2 rollback requirements](/docs/astro/airflow3/upgrade-af3#airflow-3-to-airflow-2-rollback-support-and-requirements).</Info>

<Frame>
  <img alt="View of the Overview tab in the Astro UI, with one deploy entry" />
</Frame>

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## View deploy history

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click **Overview**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
    2. Click **Overview**.
  </Tab>
</Tabs>

For each deploy, the **Deploy History** table shows the user who made the deploy, when they made the deploy, what image they used, the type of deploy, and any descriptions they added to the deploy.

### Types of deploys

There are different types of deploys, which are shared in the Deploy History table. You can read more about the different deploy options in [Deploy code to Astro](/docs/astro/deploy-code). These deploy types include:

* Image-only deploy
* Dag-only deploy
* GitHub deploy
* dbt project deploy

## Add a description to a deploy

Adding a description to a deploy is a helpful way to let other users know why you made a deploy and what the deploy contains. Descriptions appear in your deploy's entry in the **Deploy History** table.

To add a description to a deploy, specify the `--description` flag when you run `astro deploy`. For example:

```sh wrap theme={null}
astro deploy --description "Added a new 'monitor_weather' dag"
```

<Tip>If you deploy to Astro through CI/CD, Astronomer recommends adding the Git commit ID or equivalent version ID as the description for your deploy. This serves as a reference if you need to roll back your Git repository when you roll back your Deployment.</Tip>

## Roll back to a past deploy

<Danger>Astronomer recommends triggering Deployment rollbacks only as a last resort for recent deploys that aren't working as expected. Deployment rollbacks can be disruptive, especially if you triggered multiple deploys between your current version and the rollback version. See [What happens during a deploy rollback](#what-happens-during-a-deploy-rollback) before you trigger a rollback to anticipate any unexpected effects.</Danger>

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click **Overview**.
    3. Locate the deploy you want to roll back to. In the **Rollback to** column for the deploy, click **Deploy**.
    4. Provide a description for your rollback, then complete the confirmation to trigger the rollback.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
    2. Click **Overview**.
    3. Locate the deploy you want to roll back to. In the **Rollback to** column for the deploy, click **Deploy**.
    4. Provide a description for your rollback, then complete the confirmation to trigger the rollback.
  </Tab>
</Tabs>

You can roll back to any deploy within the last three months. If your last deploy was more than three months ago, you can only roll back to that deploy. After the rollback completes, the **Deploy History** table shows your rollback deploy as the most recent deploy and includes both your rollback description and rollback deploy time. Your Docker image tag, Dag bundle name, and dbt bundles are the same as the previous version you rolled back to. The historic deploy that you rolled back to still appears in chronological order in the table.

For example, consider a user who, on November 8, 2023 at 13:00, rolled back to a deploy from November 6, 2023 at 14:00. At the top of the **Deploy History** table, an entry for the rollback deploy would have the following information:

* **Time**: **14:00 11/8/2023**
* **Docker image**: **deploy-2023-11-6T14-00** (Or the custom name of your historical image tag)
* **DAG Bundle Version**: **2023-11-16T14:00:00.0000000Z**
* **Deploy description**: Your rollback description.

### What happens during a deploy rollback

A deploy rollback is a new deploy of a previous version of your code. This means that the rollback deploy appears as a new deploy in **Deploy History**, and the records for any deploys between your current version and rollback version are still preserved. In Git terms, this is equivalent to `git revert`.

When you trigger a rollback, the following information is rolled back:

* All project code, including Dags.
* Your Astro Runtime version.
* Your Deployment's Dag deploy setting.

The following information isn't rolled back:

* Your Deployment's resource configurations, such as executor and scheduler configurations.
* Your Deployment's environment variable values.
* Any other Deployment settings that you configure through the Astro UI, such as your Deployment name and description.
* For Runtime version downgrades, any data related to features that are not available in the rollback version are erased from the metadata database and not recoverable.

A rollback's effect on running tasks depends on whether the rollback downgrades your Deployment:

* If a rollback downgrades a Deployment to a previous version of Astro Runtime, all currently running tasks fail immediately. If a task has any remaining retries, those retries will run after the rollback is complete.
* If a rollback doesn't include a downgrade, any currently running tasks from before the rollback continue to run your latest code, while new Pods for downstream tasks run the code from the rollback version. This is identical behavior to pushing new code as described in [What happens during a code deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy).

## Roll back a Remote Execution Deployment

Remote Execution Agent Deployments require special handling for rollbacks.

The standard Astro deploy rollback doesn't manage Agent versions. This means you must manually roll back Remote Execution Agents before rolling back your Deployment in Astro.

Failing to do so can result in incompatibility between your Remote Agent and Astro Deployment causing interrupted workflows.

<Steps>
  <Step title="Roll back your Remote Execution Agents">
    Update each Remote Execution Agent to the desired target Runtime/Agent version by reverting their containers to the appropriate image. Ensure all agents are successfully reverted before proceeding to the next step.
  </Step>

  <Step title="Roll back your Astro Deployment">
    Trigger a [Astro Deployment rollback](#roll-back-to-a-past-deploy) to a deployment that is compatible with your Remote Agent's updated version.
  </Step>
</Steps>


# Deploy an Astro project as an image
Source: https://astronomer.io/docs/astro/deploy-project-image

Deploy a complete Astro project to a Deployment as a Docker image.

In a full deploy, the Astro CLI takes every file in your Astro project to builds them into a Docker image. This includes your `Dockerfile`, dags, plugins, and all Python and OS-level packages. The CLI then deploys the image to all Airflow components in a Deployment.

Use this document to learn how full deploys work and how to manually push your Astro project to a Deployment. For production environments, Astronomer recommends automating all code deploys with CI/CD. See [Choose a CI/CD strategy](/docs/astro/set-up-ci-cd).

See [dags-only deploys](/docs/astro/deploy-dags) to learn more about how to deploy your dags and images separately.

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview) is installed in an empty directory. If you're using an Apple M1 system with Astro Runtime 6.0.4 or later for local development, you must install Astro CLI 1.4.0 or later to deploy to Astro.
* An Astro Workspace with at least one [Deployment](/docs/astro/create-deployment).
* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project).

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Step 1: Authenticate to Astro

Run the following command to authenticate to Astro:

```sh wrap theme={null}
astro login
```

After running this command, you are prompted to open your web browser and log in to the Astro UI. After you complete this login, you are automatically authenticated to the CLI.

<Tip>If you have API credentials set as OS-level environment variables on your local machine, you can deploy directly to Astro without needing to manually authenticate. This setup is required for automating code deploys with [CI/CD](/docs/astro/set-up-ci-cd).</Tip>

## Step 2: Push your Astro project to an Astro Deployment

To deploy your Astro project, run:

```sh wrap theme={null}
astro deploy
```

This command returns a list of Deployments available in your Workspace and prompts you to pick one.

After you select a Deployment, the CLI parses your dags and runs a suite of pytests to ensure that they don't contain basic errors. This testing process is equivalent to running `astro dev parse` and `astro dev pytest` in a local Airflow environment. If any of your dags fail this testing process, the deploy to Astro also fails. To force a deploy even if your project has errors, you can run `astro deploy --force`. For more information about using pytests, see [Troubleshoot your local Airflow environment](/docs/cli/v1.43/run-airflow-locally) and [Testing Airflow dags](/docs/learn/testing-airflow).

If your code passes the testing phase, the Astro CLI deploys your project in two separate, simultaneous processes:

* The Astro CLI uploads your `dags` directory to Astronomer-hosted blob storage. Your Deployment downloads the dags from the blob storage and applies the code to all of its running Airflow containers.
* The Astro CLI builds all other project files into a Docker image and deploys this to an Astronomer-hosted Docker registry. The Deployment then applies the image to all of its running Airflow containers.

See [What happens during a project deploy](#what-happens-during-a-project-deploy) for a more detailed description of how the Astro CLI and Astro work together to deploy your code.

<Info>
  If you use Docker Desktop, ensure that the [**Use containerd for pulling and storing images**](https://docs.docker.com/desktop/containerd/#turn-on-the-containerd-image-store-feature) setting is turned off. Otherwise, you might receive errors when you run `astro deploy` such as:

  ```text wrap theme={null}
  Push access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  # or
  Unable to find image 'barren-ionization-0185/airflow:latest' locally
  Error response from daemon: pull access denied for barren-ionization-0185/airflow, repository does not exist or may require 'docker login'
  ```
</Info>

## Step 3: Validate your changes

When you start a code deploy to Astro, the status of the Deployment is **DEPLOYING** until it is determined that the underlying Airflow components are running the latest version of your code. During this time, you can hover over the status indicator to determine whether your entire Astro project or only dags were deployed .

When the deploy completes, the **Docker Image** field in the Astro UI are updated depending on the type of deploy you completed.

* The **Docker Image** field displays a unique identifier generated by a Continuous Integration (CI) tool or a timestamp generated by the Astro CLI after you complete an image deploy.

To confirm a deploy was successful, verify that the running versions of your Docker image and dag bundle have been updated.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Review the information in the **Docker Image** and **DAG Bundle Version** field to determine the Deployment code version.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
    2. Review the information in the **Docker Image** and **DAG Bundle Version** field to determine the Deployment code version.
  </Tab>
</Tabs>

## What happens during a project deploy

### Hosted execution mode

Read this section for a more detailed description of the project deploy process to a [Hosted execution mode](/docs/astro/execution-mode) Airflow 3 Deployment.

Your Deployment uses the following components to process your code deploy:

* A proprietary operator for deploying Docker images to your Airflow containers
* An Astro bundle backend for retrieving dags and non-dag bundles from versioned storage
* A blob storage container hosted by Astronomer

When you run `astro deploy`, the Astro CLI deploys all non-dag files in your project as an image to an Astronomer-hosted Docker registry. The proprietary operator pulls the images from a Docker registry, then updates the running image for all Airflow containers in your Deployment. Dag changes are deployed through a separate and simultaneous process.

For Airflow 3 Deployments with dag deploys enabled, the Astro bundle backend ensures [dag versioning](/docs/astro/dag-versioning) and dag rollbacks work out of the box. If dag deploys are disabled, the dags are stored in the image and can only be updated by image deploys. The Astro bundle backend also supports bundle deploys, enabling deployment of non-dag bundles such as `astro dbt deploy`. The backend downloads these bundles and mounts them at the configured path in the Airflow containers.

<Info>This process is different if your Deployment has dag-only deploys disabled. See [Enable/disable dag-only deploys on a Deployment](/docs/astro/deploy-dags#enable-or-disable-dag-only-deploys-on-a-deployment) for how the process changes when dag-only deploys are disabled.</Info>

### Remote execution mode

For [Remote execution mode](/docs/astro/execution-mode) Deployments, dag deploys are always disabled, and you must configure either the `LocalDagBundle` or the `GitDagBundle` so that [Remote Execution Agents](/docs/astro/remote-execution-configure-agents) in your environment can access your dags whether they are locally stored or in a git repository.

`LocalDagBundle` is the default Dag bundle type for the `dagBundleConfigList` config option but you can alternatively configure a git connection with `GitDagBundle` for extended versioning capabilities. See [Dag bundles](/docs/astro/remote-execution-configure-dag-sources) for more information.

### How Deployments handle code deploys

After a Deployment receives an image deploy, Astro gracefully terminates all of it's Airflow component containers, which means the existing worker Pods are allowed to finish their running tasks but the new worker Pods run your new code. For dag-only deploys, the Airflow component containers, including any workers or Kubernetes worker Pods that are currently running tasks, are preserved.

If you deploy code to a Deployment that is running a previous version of your code, then the following happens:

* Tasks that are `running` continue to run on existing workers and are not interrupted unless the task does not complete within 24 hours of the code deploy.
* One or more new workers are created alongside your existing workers and immediately start executing scheduled tasks based on your latest code.

  These new workers execute downstream tasks of dag runs that are in progress. For example, if you deploy to Astronomer when `Task A` of your dag is running, `Task A` continues to run on an old Celery worker. If `Task B` and `Task C` are downstream of `Task A`, they are both scheduled on new Celery workers running your latest code.

  This means that dag runs could fail due to downstream tasks running code from a different source than their upstream tasks. Dag runs that fail this way need to be fully restarted from the Airflow UI so that all tasks are executed based on the same source code.

Astronomer sets a grace period of 24 hours for all workers to allow running tasks to continue executing. This grace period is not configurable. If a task does not complete within 24 hours, its worker is terminated. Airflow marks the task as a [zombie](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#zombie-undead-tasks) and it retries according to the task's retry policy. This is to ensure that our team can reliably upgrade and maintain Astro as a service.

<Tip>
  If you want to force long-running tasks to terminate sooner than 24 hours, specify an [`execution_timeout`](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#timeouts) in your dag's task definition.

  Alternatively, if you find you have many long-running tasks that exceed 24 hours, you might want to refactor your dags using [deferrable operators](/docs/learn/deferrable-operators), as Astro does not guarantee reliability for tasks with a duration that exceeds 24 hours.
</Tip>

## Deploy a prebuilt Docker image

By default, running `astro deploy` with the Astro CLI builds your Astro project into a Docker image and deploys it to Astro. In some cases, you might want to skip the build step and deploy a prebuilt Docker image instead.

Deploying a prebuilt Docker image allows you to:

* Test a single Docker image across Deployments instead of rebuilding it each time.
* Reduce the time it takes to deploy. If your Astro project has a number of packages that take a long time to install, it can be more efficient to build it separately.
* Enforce separation between build and deploy stages in your CI/CD pipeline.
* Specify additional mounts and arguments in your project, which is required for setups such as [installing Python packages from private sources](/docs/cli/v1.43/private-python-packages#install-python-packages-from-a-private-pypi-index).

### Build and deploy a local image

To build an image locally and deploy it to Astro:

1. Run `docker build` from an Astro project directory or specify the command in a CI/CD pipeline. This Docker image must be based on Astro Runtime and be available in a local Docker registry. If you run this command on an Apple M1 computer or on a computer with an ARM64 processor, you must specify `--platform=linux/amd64` or else the deploy will fail. Astro Deployments require an AMD64-based image and do not support ARM64 architecture.

2. (Optional) Test your Docker image in a local Airflow environment by adding the `--image-name <image-name>` flag to any of the following commands:

   * `astro dev start`
   * `astro dev restart`
   * `astro dev parse`
   * `astro dev pytest`

3. Run `astro deploy --image-name <image-name>` or specify the command in a CI/CD pipeline.

### Pull and deploy from an artifact registry

If your organization builds and stores Astro Runtime images in an external artifact registry, you can pull the image and deploy it to Astro without rebuilding. This is useful when your CI/CD pipeline separates build and deploy stages, or when a centralized build process publishes images that multiple teams consume.

1. Authenticate to your artifact registry. The following examples show authentication commands for common registries:

   **Google Artifact Registry:**

   ```sh wrap theme={null}
   gcloud auth configure-docker <your-region>-docker.pkg.dev
   ```

   **Amazon ECR:**

   ```sh wrap theme={null}
   aws ecr get-login-password --region <your-region> | docker login --username AWS --password-stdin <your-account-id>.dkr.ecr.<your-region>.amazonaws.com
   ```

   **Azure Container Registry:**

   ```sh wrap theme={null}
   az acr login --name <your-registry-name>
   ```

2. Pull the prebuilt image from the registry:

   ```sh wrap theme={null}
   docker pull <registry-url>/<image-name>:<tag>
   ```

3. Deploy the pulled image to Astro:

   ```sh wrap theme={null}
   astro deploy <deployment-id> --image-name <registry-url>/<image-name>:<tag>
   ```

The image must be based on Astro Runtime. No rebuild occurs during this process. The Astro CLI pushes the pulled image directly to Astro.

For a complete CI/CD example, see the [Prebuilt image GitHub Actions template](/docs/astro/ci-cd-templates/github-actions-private-network#setup).

### Additional options

If you have dag-only deploys enabled, you can also use the `--image` flag to deploy a prebuilt image without also deploying your dags folder. Use `astro deploy --image-name <image-name> --image`.

For more information about the `--image-name` option, see the [CLI command reference](/docs/cli/v1.43/astro-deploy).

<Info>
  If you build an AMD64-based image and run `astro deploy` from an Apple M1 computer, you might see a warning in your terminal. You can ignore the warning.

  ```text wrap theme={null}
  WARNING: The requested image's platform (linux/amd64) does not match the detected host platform
  (linux/arm64/v8) and no specific platform was requested
  ```
</Info>

## Related documentation

* [Choose a CI/CD Strategy for deploying code to Astro](/docs/astro/set-up-ci-cd)
* [Develop your project](/docs/cli/v1.43/develop-project)
* [Set environment variables](/docs/astro/environment-variables)


# Initialize and deploy Remote Execution projects
Source: https://astronomer.io/docs/astro/deploy-project-remote-execution

Initialize and deploy an Astro project for Remote Execution.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Remote Execution Deployments require two sets of container images:

* **Astro Runtime images**: Astronomer's managed distribution of Apache Airflow, powering all Deployments on Astro.
* **Remote Execution Agent images**: Specifically built for Remote Execution Agents.

When using Remote Execution mode, the server (Deployment on Astro) and client (Remote Execution Agents) are deployed separately. You must maintain dedicated client images for the Remote Execution Agents. These client images are based on the [Astro Remote Execution Agent](/docs/astro/agent-release-notes) image and must be pushed to a remote image registry accessible by your Agents.

Because these client images are distinct from the usual Astro Runtime images, you need to use dedicated commands to build and deploy them.

This guide explains how to initialize an Astro project for Remote Execution and how to build and push client images for Remote Execution Agents.

<Info>
  For details about Astro project image deploys with Remote Execution, see [Deploy an Astro project](/docs/astro/deploy-project-image#remote-execution-mode).

  For best practices and procedures to upgrade your Astro Runtime version, including for Remote Execution Deployments, see [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime).
</Info>

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* An Astro Workspace with at least one [Remote Execution Deployment](/docs/astro/execution-mode).
* [Remote Execution Agents](/docs/astro/remote-execution-configure-agents) registered with your Deployment.
* A remote Docker registry accessible from your environment where Remote Execution Agents run.

## Build and deploy your Remote Execution project

<Steps>
  <Step title="Create a Remote Execution project">
    Run:

    ```sh wrap theme={null}
    astro dev init my-remote-project --remote-execution-enabled --remote-image-repository <remote-image-repository-url>
    ```

    This generates the required files for building client images.
    If you omit `--remote-image-repository`, you can configure it later with:

    ```sh wrap theme={null}
    astro config set remote.client_registry <remote-image-repository-url>
    ```
  </Step>

  <Step title="Review generated files">
    * `Dockerfile.client` – Dockerfile for client images
    * `requirements-client.txt` – Python dependencies for client images
    * `packages-client.txt` – OS-level packages for client images
  </Step>

  <Step title="Configure dag access">
    Set up dag access for your Remote Execution Agents using either `LocalDagBundle` or `GitDagBundle`.
    Update your Agent's Helm chart `values.yaml` accordingly. See [Dag bundles](/docs/astro/remote-execution-configure-dag-sources).
  </Step>

  <Step title="Log in to your remote Docker registry">
    Before you can deploy a client image, you must be logged in to your remote Docker registry.
    The Astro CLI does not manage authentication for your registry.

    Log in using:

    ```sh wrap theme={null}
    docker login <remote-registry> --username <registry-username> --password-stdin
    ```

    Replace `<remote-registry>` and `<registry-username>` with the details for your registry.
  </Step>

  <Step title="Build and deploy the Astro Deployment (server) image">
    From your project directory, deploy the Astro Runtime image to your Remote Execution Deployment:

    ```sh wrap theme={null}
    astro deploy
    ```

    This command builds your project and pushes the updated Astro Runtime image to Astro, updating the Airflow components in your managed Astro Deployment.

    Monitor the Deployment to confirm that the new version is running in the Astro UI.

    <Info>
      The [`astro deploy`](/docs/cli/v1.43/astro-deploy) command updates the orchestration plane only. To update the execution plane, follow the Remote Execution Agent image steps below.
    </Info>
  </Step>

  <Step title="Build and deploy Remote Execution Agent (client) images">
    From your project directory, run:

    ```sh wrap theme={null}
    astro remote deploy
    ```

    You can use additional flags as needed, such as:

    ```sh wrap theme={null}
    astro remote deploy --platform linux/amd64
    astro remote deploy --platform linux/amd64,linux/arm64
    astro remote deploy --image-name my-custom-image:tag
    astro remote deploy --build-secret id=mysecret,src=secrets.txt
    ```

    <Warning>
      **Platform mismatch can cause build failures**

      By default, `astro remote deploy` builds an image for your computer’s platform. For example, if you are on a Mac with Apple silicon, it will build for `linux/arm64`.
      If your Agents need `linux/amd64`, the image may not work.

      To avoid issues, always set the `--platform` flag:
      `astro remote deploy --platform linux/amd64,linux/arm64`
    </Warning>

    This command only pushes the client image to your remote registry. See [`astro remote`](/docs/cli/v1.43/astro-remote) for more information.
  </Step>

  <Step title="Update Remote Execution Agents to use the new image">
    After pushing the image, you must update your Agent configuration to actually use it:

    1. Edit your Agent Helm chart `values.yaml`.
    2. Update the `image` field for each of the `worker`, `dagProcessor`, and `triggerer` component sections to the new client image reference (as just pushed).
    3. Apply the changes by running:

    ```sh wrap theme={null}
    helm upgrade <release-name> <chart-path> -f values.yaml
    ```

    Agents will not use the new client image until this update is performed.

    <Info>
      **CI/CD pipeline**

      If you manage your Agent deployments via a CI/CD pipeline, update your pipeline to set the new client image in the Helm `values.yaml` file, and trigger the deployment process to roll out the change automatically.
      This ensures that updates to the client image are reliably applied across all agents without manual intervention.
    </Info>
  </Step>
</Steps>

## Related documentation

* [Remote Execution mode overview](/docs/astro/execution-mode)
* [Remote Execution Agents](/docs/astro/remote-execution-configure-agents)
* [Dag bundles](/docs/astro/remote-execution-configure-dag-sources)
* [`astro remote` CLI reference](/docs/cli/v1.43/astro-remote)
* [`astro dev init` CLI reference](/docs/cli/v1.43/astro-dev-init)


# Deployment details
Source: https://astronomer.io/docs/astro/deployment-details

Edit information about your Deployment, like metadata settings, observability settings, and user access settings.

Deployment details define how users can view and interact with your Deployment. They include metadata settings, observability settings, and user access settings.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Update a Deployment name and description

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click the Deployment's **More actions** menu (⋯), then select **Edit Deployment**.
    3. In the **Basic** section, update the Deployment **Name** or **Description**.
    4. Click **Update Deployment**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
    2. Click the **More Actions** menu of the Deployment you want to update, and select **Edit Deployment**.

       <Frame>
         <img alt="Edit Deployment in options menu" />
       </Frame>
    3. In the **Basic** section, update the Deployment **Name** or **Description**.
    4. Click **Update Deployment**.
  </Tab>
</Tabs>

## Configure Deployment contact emails

Configure a contact email to get proactive alerts directly from Astronomer support. Astronomer support uses contact emails to notify recipients in case there's an issue with the infrastructure for your Deployment, such as a problem with your scheduler or worker components.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click the **Details** tab. In the **Advanced** section, click **Edit**.
    3. To add an alert email:
       * In the **Contact Emails** section, click **Add Email**.
       * Enter an email address and then click **Add**.
    4. To delete an alert email address:
       * In the **Contact Emails** section, click **Delete** next to the email you want to delete.
       * Click **Yes, Continue**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
    2. Click the **Details** tab. In the **Advanced** section, click **Edit**.
    3. To add an alert email:
       * In the **Contact Emails** section, click **Add Email**.
       * Enter an email address and then click **Add**.
    4. To delete an alert email address:
       * In the **Contact Emails** section, click **Delete** next to the email you want to delete.
       * Click **Yes, Continue**.
  </Tab>
</Tabs>

In addition to alert emails for your Deployments, Astronomer recommends configuring [Astro alerts](/docs/astro/alerts) and subscribing to the [Astro status page](https://status.astronomer.io). When you subscribe to the status page, you'll receive email notifications about system-wide incidents as they happen.

### Fallback emails

For Deployments without a contact email set, Astronomer sets a fallback value to ensure critical issues don’t get missed. To see the fallback values for a Deployment without a contact email, go to the **Details** tab and find **Contact Emails** in the **Advanced** section. To override the fallback values displayed, set a contact email. We recommend that every Astro Deployment have a contact email set to ensure timely receipt of notifications.

For Deployments without a contact email, Astronomer uses the Deployment Creator as the fallback. For Deployments without a valid Deployment Creator, Astro uses Workspace Owners. In Workspaces without an Owner, Astro uses Organization Owners. Check that your Deployments, Workspaces, and Organizations each have a valid owner to ensure that key support communications don’t get missed.

<Frame>
  <img alt="See default fallback emails" />
</Frame>

## Enforce CI/CD deploys

<Note> This is feature is only available if you are on the **Business** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

By default, any user can deploy code either directly from the Astro CLI or from a CI/CD process that is authenticated with an API token. To help your team protect production environments from manual code deploys that circumvent your organization's CI/CD processes and checks, you can configure a Deployment so that users can't deploy code manually using the Astro CLI.

After you enable CI/CD enforcement on a Deployment, the Deployment accepts a deploy only if the deploy is authenticated using a Deployment API token, Workspace API token, or Organization API token. Astronomer recommends enabling this setting for all production environments.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click the Deployment's **More actions** menu (⋯), then select **Edit Deployment**.
    3. In the **Advanced** section, find **CI/CD Enforcement** and click the toggle to **On**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
    2. Click the **More Actions** menu of the Deployment you want to update, and select **Edit Deployment**.

       <Frame>
         <img alt="Edit Deployment in options menu" />
       </Frame>
    3. In the **Advanced** section, find **CI/CD Enforcement** and click the toggle to **On**.
  </Tab>
</Tabs>

You can also update your Workspace so that any new Deployments in the Workspace enforce CI/CD deploys by default. See [Update general Workspace settings](/docs/astro/manage-workspaces#update-general-workspace-settings).

<Info>
  When CI/CD enforcement is enabled for a Deployment, you can't enable or disable [Dag-only deploys](/docs/astro/deploy-dags) for the Deployment.

  To enable or disable dag-only deploys when CI/CD enforcement is turned on:

  1. Turn **CI/CD Enforcement** to **Off**.
  2. Enable or disable the dag-only deploy feature. See [Enable or disable dag-only deploys](/docs/astro/deploy-dags#enable-or-disable-dag-only-deploys-on-a-deployment).
  3. Turn **CI/CD Enforcement** back to **On**.

  You have to only complete these steps once. Once the dag-only deploy feature is enabled or disabled, you can turn CI/CD enforcement on or off at any time.
</Info>

## Delete a Deployment

When you delete a Deployment, all infrastructure resources assigned to the Deployment are immediately deleted.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click the Deployment's **More actions** menu (⋯), then select **Delete Deployment**.
    3. Enter the Deployment name shown in the dialog, then click **Yes, Continue**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
    2. Click the **More Actions** menu of the Deployment you want to delete, and select **Delete Deployment**.

       <Frame>
         <img alt="Delete Deployment in options menu" />
       </Frame>
    3. Enter `Delete` and click **Yes, Continue**.
  </Tab>
</Tabs>

## Find Deployment external IP addresses

Each Astro Deployment has its own external IP addresses. Allowlist these addresses on any external service as a first step to create a connection between the Deployment and the service.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Select the **Details** tab.
    3. In the **Other** section, find the **External IPs** associated with the Deployment.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
    2. Select the **Details** tab.
    3. In the **Other** section, you can find the **External IPs** associated with the Deployment.
  </Tab>
</Tabs>


# Configure Deployment resources
Source: https://astronomer.io/docs/astro/deployment-resources

Configure your Deployment resource settings to optimize Deployment performance.

Your Deployment resources are the computational resources Astro uses to run Airflow in the cloud. Update Deployment resource settings to optimize performance and reduce the cost of running Airflow in the cloud.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Update Airflow configurations

To update a Deployment's [Airflow configurations](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html), you set the configurations as environment variables on Astro. See [Set Airflow configurations using environment variables](/docs/astro/manage-env-vars).

## Update environment objects

You can add, update, and delete Airflow connections that were added through the Astro UI from your Deployment page. To edit a Deployment's [linked connections](/docs/astro/manage-connections-variables#astro-environment-manager), click the **Environment** tab, and then select the connection you want to **Edit**. See [Create connections with the Astro UI](/docs/astro/create-and-link-connections) for more options.

## Deployment executor

Astro supports three executors: the Astro, Celery, and Kubernetes executors. Only Celery and Kubernetes executors are available in the Apache Airflow open source project.

* [Celery executor](/docs/astro/celery-executor)
* [Kubernetes executor](/docs/astro/kubernetes-executor) (*Requires Astro Runtime version 8.1.0 or later*)
* [Astro executor](/docs/astro/astro-executor) (*Only available for Airflow 3.x Deployments*)

  See [Choose an executor](/docs/astro/executors-overview#choose-an-executor) to understand the benefits and limitations of each executor. When you've determined the right executor type for your Deployment, complete the steps in the following topic to update your Deployment's executor type.

### Update the Deployment executor

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.

    2. Click the Deployment's **More actions** menu (⋯), then select **Edit Deployment**.

    3. In the **Execution** section, select **Astro**, **Celery**, or **Kubernetes** in the **Executor** list.

       If you're moving from the Astro or Celery to the Kubernetes executor, all existing worker queues are deleted. Running tasks stop gracefully and all new tasks start with the selected executor.

    4. Click **Update Deployment**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.

    2. Click the **Details** tab and then click **Edit** in the **Execution** section.

       <Frame>
         <img alt="Edit Deployment in options menu" />
       </Frame>

    3. In the **Execution** section, select **Astro**, **Celery** or **Kubernetes** in the **Executor** list.

       If you're moving from the Astro or Celery to the Kubernetes executor, all existing worker queues are deleted. Running tasks stop gracefully and all new tasks start with the selected executor.

    4. Click **Update Deployment**.
  </Tab>
</Tabs>

See [Configure an executor](/docs/astro/executors-overview) for more information about each available executor type, including how to optimize executor usage.

## Configure Kubernetes Pod resources

The [Kubernetes executor](/docs/astro/kubernetes-executor) and [`KubernetesPodOperator`](/docs/astro/kubernetespodoperator) both use Kubernetes Pods to execute tasks. While you still need to configure Pods in your Dag code to define individual task environments, you can set some safeguards on Astro so that tasks in your Deployment don't request more CPU or memory than expected.

Set safeguards by configuring default Pod limits and requests from the Astro UI. If a task requests more CPU or memory than is currently allowed in your configuration, the task fails.

By default, Astro supports a maximum `KubernetesExecutor` Pod size of 43 vCPU and 86 GiB of memory. Astro can support Pod sizes up to 86 vCPU and 172 GiB of memory on request. If your workloads require more resources, contact [Astronomer support](/docs/astro/astro-support) to request larger Pod sizes for your Deployment.

To customize the resources for individual task Pods beyond the defaults, see [Customize a task's Kubernetes Pod](/docs/astro/kubernetes-executor#customize-a-task’s-kubernetes-pod).

<Info>To manage Kubernetes resources programmatically, you can set default Pod limits and resources with the [`astro deployment create`](/docs/cli/v1.43/astro-deployment-create) and [`astro deployment update`](/docs/cli/v1.43/astro-deployment-update) Astro CLI commands, or by adding the configurations to a [Deployment file](/docs/astro/deployment-file-reference).</Info>

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.

    2. Click the Deployment's **More actions** menu (⋯), then select **Edit Deployment**.

    3. In the **Execution** section, configure the following values:

       * **CPU Quota**: The maximum combined CPU usage across all running Pods on your Deployment.
       * **Memory Quota**: The maximum combined memory usage across all running Pods on your Deployment.
       * **Default Pod Size**:
         * **CPU**: The amount of CPUs that your tasks run with if no CPU usage is specified in their Pod configuration.
         * **Memory**: The amount of memory that your tasks run with if no memory usage is specified in their Pod configuration.
         * **Storage**: Choose the amount of ephemeral storage in GiB assigned to each pod in the Astro UI. This storage volume is transient and allows for the temporary storage and processing of data. The pod is assigned the minimum 0.25 GiB by default. The maximum possible quota is 100 GiB. Only ephemeral storage requests that are greater than the default minimum of 0.25 GiB are chargeable. Note that this feature is in [Preview](/docs/astro/feature-previews).

       For a Deployment running in a Hosted dedicated or shared cluster, the maximum possible **CPU** quota is 6400 vCPU and maximum **Memory** quota is 12800 GiB.

       <Warning>
         **Astro Hosted**

         On Astro Hosted, Astro automatically sets resource requests equal to limits for task Pods. This ensures Pods receive a Kubernetes [Guaranteed Quality of Service (QoS) class](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#guaranteed), which prevents resource contention and eviction. Because Astro uses the limit values as both requests and limits, your Pods are billed based on the **CPU** and **Memory** limits you configure, even if actual usage is lower. To avoid unexpected charges, set limits close to the resources your tasks require.

         Check your [Billing and usage](/docs/astro/manage-billing) to view your resource use and associated charges.
       </Warning>

    4. Click **Update Deployment**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.

    2. Click the **Options** menu and select **Edit Deployment**.

       <Frame>
         <img alt="Edit Deployment in options menu" />
       </Frame>

    3. In the **Execution** section, configure the following values:

       * **CPU Quota**: The maximum combined CPU usage across all running Pods on your Deployment.
       * **Memory Quota**: The maximum combined memory usage across all running Pods on your Deployment.
       * **Default Pod Size**:
         * **CPU**: The amount of CPUs that your tasks run with if no CPU usage is specified in their Pod configuration.
         * **Memory**: The amount of memory that your tasks run with if no memory usage is specified in their Pod configuration.
         * **Storage**: Choose the amount of ephemeral storage in GiB assigned to each pod in the Astro UI. This storage volume is transient and allows for the temporary storage and processing of data. The pod is assigned the minimum 0.25 GiB by default. The maximum possible quota is 100 GiB. Only ephemeral storage requests that are greater than the default minimum of 0.25 GiB are chargeable. Note that this feature is in [Preview](/docs/astro/feature-previews).

       For a Deployment running in a Hosted dedicated or shared cluster, the maximum possible **CPU** quota is 6400 vCPU and maximum **Memory** quota is 12800 GiB.

    <Warning>
      **Astro Hosted**

      On Astro Hosted, Astro automatically sets resource requests equal to limits for task Pods. This ensures Pods receive a Kubernetes [Guaranteed Quality of Service (QoS) class](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#guaranteed), which prevents resource contention and eviction. Because Astro uses the limit values as both requests and limits, your Pods are billed based on the **CPU** and **Memory** limits you configure, even if actual usage is lower. To avoid unexpected charges, set limits close to the resources your tasks require.

      Check your [Billing and usage](/docs/astro/manage-billing) to view your resource use and associated charges.
    </Warning>

    4. Click **Update Deployment**.
  </Tab>
</Tabs>

## Workers and worker queues

Workers are the Airflow components that execute your tasks. How workers are configured depends on the executor your Deployment uses:

* **Astro and Celery executors**: Workers run in configurable *worker queues*. Each worker queue defines the worker type (size), concurrency, autoscaling limits, and ephemeral storage for its workers. You can create multiple worker queues to optimize execution environments for different types of tasks. For example, you can assign resource-intensive machine learning tasks to a queue with larger workers, while running lightweight SQL tasks on smaller workers. See [Configure worker queues](/docs/astro/configure-worker-queues) for setup instructions.

* **Kubernetes executor**: Each task runs in its own dedicated Kubernetes Pod rather than in a shared worker. You configure Pod resources through default Pod size settings and per-task `pod_override` configurations. See [Configure Kubernetes Pod resources](#configure-kubernetes-pod-resources) and [Customize a task's Kubernetes Pod](/docs/astro/kubernetes-executor#customize-a-task’s-kubernetes-pod).

For details on worker autoscaling behavior, see [Configure the Celery executor](/docs/astro/celery-executor) or [Configure the Astro executor](/docs/astro/astro-executor).

## Scheduler

The [Airflow scheduler](https://airflow.apache.org/docs/apache-airflow/stable/concepts/scheduler.html) is responsible for monitoring task execution and triggering downstream tasks when the dependencies are met.

Scheduler resources must be set for each Deployment and are managed separately from cluster-level infrastructure. To ensure that your tasks have the CPU and memory required to complete successfully on Astro, you can provision the scheduler with varying amounts of CPU and memory.

<Info>
  **Remote Execution Deployments**

  The scheduler always receives a fixed allocation of **1 vCPU and 2 GiB memory** in [remote execution](/docs/astro/execution-mode) mode. You can't change scheduler resources for remote execution Deployments.
  You can enable [High Availability](#enable-high-availability) to run two schedulers in this mode, each with the same fixed resources.
</Info>

Unlike workers, schedulers don't autoscale. The resources you set for them are the resources you have regardless of usage. For more information about how scheduler configuration affects resources usage, see [Pricing](https://astronomer.io/pricing).

Astronomer Deployments run a single scheduler by default. You can configure your scheduler to have different amounts of resources based on how many tasks you need to schedule for hosted execution Deployments. For [remote execution](/docs/astro/execution-mode) Deployments, scheduler resources are fixed and not configurable.

You can also enable [High Availability](#enable-high-availability) to run two instances of PgBouncer and the Airflow Scheduler.

### Size options

Astro separates the scheduler and Dag processor for some Deployment sizes, which improves security and reliability because the `SchedulerJob` and `DAGProcessorJob` can run on separate Pods. Consider the following details when choosing your scheduler and Dag processor sizes:

* To use the separate scheduler and Dag processor, you must use at least version 9.7.0 of Astro Runtime. If your Deployment uses a lower Runtime version, then the scheduler and Dag processor run on the same Pod, and the Extra Large Deployment size isn't available.
* For **Small** Deployments, the scheduler and Dag processor run on the same Pod.
* **Extra Large** Deployments have two Dag processors allocated per Deployment.

<Tip>Astronomer recommends using **Medium** or larger for production Deployments for improved reliability and performance. This aligns with Apache Airflow best practices of running the scheduler and Dag processor as separate processes. For Airflow 3.x Deployments, **Medium** or larger is highly recommended, as open source Airflow 3 has fully separated the scheduler and Dag processor into distinct processes.</Tip>

The following table lists all possible scheduler and Dag processor sizes for Astro Hosted:

| Size                            | vCPU                                                | Memory                                                    | Ephemeral Storage                                         |
| ------------------------------- | --------------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------- |
| Small (Up to \~50 Dags)         | 1                                                   | 2GiB                                                      | 5 GiB                                                     |
| Medium (Up to \~250 Dags)       | **Scheduler**: 1 <br /> **DAG Processor**: 1        | **Scheduler**: 2 GiB <br /> **DAG Processor**: 2 GiB      | **Scheduler**: 1 GiB <br /> **DAG Processor**: 4 GiB      |
| Large (Up to \~1000 Dags)       | **Scheduler**: 1 <br /> **DAG Processor**: 3        | **Scheduler**: 2 GiB <br /> **DAG Processor**: 6 GiB      | **Scheduler**: 1 GiB <br /> **DAG Processor**: 4 GiB      |
| Extra Large (Up to \~2000 Dags) | **Scheduler**: 1 <br /> **DAG Processor (x2)**: 3.5 | **Scheduler**: 4 GiB <br /> **DAG Processor (x2)**: 6 GiB | **Scheduler**: 1 GiB <br /> **DAG Processor (x2)**: 4 GiB |

### Update scheduler size

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click the Deployment's **More actions** menu (⋯), then select **Edit Deployment**.
    3. In the **Advanced** section, choose a scheduler size. See [Scheduler](#scheduler).
    4. Click **Update Deployment**.

       The Airflow components of your Deployment automatically restart to apply the updated resource allocations. This action is equivalent to deploying code and triggers a rebuild of your Deployment image. If you're using the Celery executor, currently running tasks have 24 hours to complete before their running workers are terminated. See [What happens during a code deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy).
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.

    2. Click the **Options** menu of the Deployment you want to update, and select **Edit Deployment**.

       <Frame>
         <img alt="Edit Deployment in options menu" />
       </Frame>

    3. In the **Advanced** section, choose a scheduler size. See [Scheduler](#scheduler).

    4. Click **Update Deployment**.

       The Airflow components of your Deployment automatically restart to apply the updated resource allocations. This action is equivalent to deploying code and triggers a rebuild of your Deployment image. If you're using the Celery executor, currently running tasks have 24 hours to complete before their running workers are terminated. See [What happens during a code deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy).
  </Tab>
</Tabs>

## Triggerer

The Airflow triggerer executes deferred tasks in an [`asyncio` event loop](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio-event-loop). Running a triggerer is essential for using [deferrable operators](/docs/learn/deferrable-operators).

One replica of the Airflow triggerer with 0.5 vCPU and 1.875 GiB is automatically included in your Deployment, and is the same size for all Deployments. The included triggerer can typically support up to 1000 concurrent running triggers with low to medium memory requirements. If your use case requires over 1000 concurrent running triggers and/or medium to heavy memory requirements, Astro supports customizing the count and size of the triggerer. Leverage the triggerer metrics available in Deployment Analytics to right-size the triggerer. Additional requested compute beyond the default specs will be charged at the A5 worker unit rate, similar to [KE/KPO chargeback](https://www.astronomer.io/pricing/#:~:text=How%20will%20I%20be%20charged%20for%20the%20Kubernetes%20Executor%20and%20Kubernetes%20Pod%20Operator%3F).

The following table lists all available environment variables for customizing the count and size of the triggerer. Set each variable to the value shown in the **Default** or **Max** column, or to any value in between.

| Environment Variable               | Default  | Max       | Equivalent range |
| ---------------------------------- | -------- | --------- | ---------------- |
| `ASTRO_TRIGGERER_REPLICAS`         | `1`      | `8`       | 1–8 replicas     |
| `ASTRO_TRIGGERER_RESOURCES_CPU`    | `500m`   | `5000m`   | 0.5–5 vCPU       |
| `ASTRO_TRIGGERER_RESOURCES_MEMORY` | `1920Mi` | `15360Mi` | 1.875–15 GiB     |

The CPU and memory variables use Kubernetes resource units: `m` is millicores, where `1000m` equals 1 vCPU, and `Mi` is mebibytes, where `1024Mi` equals 1 GiB.

## API server

<Note>
  **Airflow 3**

  This component applies only to Airflow 3.x Deployments. Airflow 2.x Deployments use the Airflow webserver, which doesn't support horizontal autoscaling on Astro.
</Note>

The Airflow API server serves the Airflow UI, the Airflow REST API, and the task execution API used by Astro and Celery workers. Every Airflow 3 Deployment runs with two API server replicas by default.

For workloads with high task concurrency or spikes in UI and REST API traffic, you can enable horizontal autoscaling so that Astro adds API server replicas when load is high and removes them when load decreases. Autoscaling is implemented as a Kubernetes Horizontal Pod Autoscaler (HPA) that targets 80 percent average CPU utilization across the running replicas.

When you enable autoscaling, Astro maintains a minimum of two replicas and scales up to a maximum that you configure from `2` to `10`. Astro charges for replicas above the included two based on their CPU usage, and the charges appear on your invoice as **API Server** under **Runtime Compute**.

For configuration steps, supported deployment types, and the API payload, see [Configure API server autoscaling](/docs/astro/api-server-autoscaling).

## Enable high availability

<Note> This is feature is only available if you are on the **Team** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

By default, the Pods running your Deployment's Airflow components are distributed across multiple nodes. When you enable high availability, Astro re-configures the Deployment to be more resilient and avoid a single point of failure by running replicas of key infrastructure components across different availability zones. Changes in high availability mode include:

* Running two schedulers so that at least one is always available
* Running two Dag processors in [Deployments with a standalone Dag processor](#size-options)
* Running two webservers
* Running the following components on different nodes with [preferred zone-level anti-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#types-of-inter-pod-affinity-and-anti-affinity):
  * Scheduler
  * Dag processor
  * PG Bouncer
  * Webserver

This ensures that your Dags can continue to run if there's an issue with one of your Airflow components in a specific node or availability zone.

<Tip>High availability mode is recommended for production Deployments to increase reliability and availability of scheduling, Dag processing and the UI/API, while also providing protection against zonal outages.</Tip>

Because this setting results in more resource usage, it increases the cost of your Deployment. See [Pricing](https://astronomer.io/pricing).

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click the Deployment's **More actions** menu (⋯), then select **Edit Deployment**.
    3. In the **Basic** section, click the toggle to **On** for **High Availability**.
    4. Select **Update Deployment** to save your changes.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.

    2. Click the **Options** menu of the Deployment you want to update, and select **Edit Deployment**.

       <Frame>
         <img alt="Edit Deployment in options menu" />
       </Frame>

    3. In the **Basic** section, click the toggle to **On** for **High Availability**.

    4. Select **Update Deployment** to save your changes.
  </Tab>
</Tabs>

## Enable development mode

Enabling **Development Mode** on a Deployment allows you to safely separate development and production environments on Astro. Deployments in **Development Mode** can be hibernated to control costs. In addition to marking a Deployment as **Development Mode**, you can also control development and testing environments by creating ephemeral preview Deployments. To learn more about which approach is right for you, see [Manage development Deployments on Astro](/docs/astro/best-practices/manage-dev-deployments).

**Development Mode** is a prerequisite to [hibernate a Deployment](#hibernate-a-development-deployment) and must be enabled at Deployment creation. It can't be turned on after a Deployment has been created, but it can be turned off after a Deployment is created.

Development Deployments have the following constraints:

* The **Small Scheduler** (1 vCPU, 2 GiB RAM) is the only scheduler size supported.
* Development Deployments can't have a [P1 ticket](/docs/astro/astro-support#p1-critical-impact) raised against them.
* Development Deployments won't be treated as [“production”](/docs/astro/astro-support#ticket-priorities) by support.

## Hibernate a development Deployment

<Note>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Note>

When you create a Deployment on Astro, you pay for the infrastructure resources that are required to run the Deployment for the duration that it's active. In development environments when you aren't always running tasks, you can *hibernate*, or scale down, all Deployment resources on a specified schedule. When you hibernate a Deployment, all Deployment configurations are preserved, but computing resources are scaled to zero.

For example, if you only need to test a Dag during working hours, you can set a hibernation schedule for 5:00 PM until 9:00 AM on Monday through Friday. During this time, your Deployment settings are preserved and your cost on Astro for the Deployment is \$0. When the hibernation schedule ends, you can resume using the Deployment. Waking up a Deployment from hibernation is faster than creating a new Deployment and preserves all of your configurations.

<Frame>
  <img alt="Example of setting a hibernation schedule in the Astro UI. A schedule is being set so that the Deployment hibernates outside of work hours." />
</Frame>

Use Deployment hibernation to ensure that:

* You only pay for the resources that you need, when you need them.
* You don't have to delete a Deployment in order to avoid the cost of the Deployment.
* You don't have to recreate development environments and re-enter Deployment configurations.

### Prerequisites

You can hibernate a Deployment only if you enabled **Development Mode** when you [created the Deployment](/docs/astro/create-deployment). Deployments without this setting enabled can't be hibernated.

### Create a wake schedule

Before you create or edit a wake schedule for a Deployment, consider the following constraints:

* The Deployment must have the **Development Mode** setting turned on. This setting can be turned on only when you create a Deployment.
* The **High Availability** feature isn't supported. A Deployment with a hibernation schedule can't be highly available.
* Deployments with hibernation schedules aren't required to meet the uptime SLAs of standard production Deployments.

<Info>
  The following instructions show you how to configure wake schedules in the Astro UI. You can also configure wake schedules programmatically through the [Astro API](https://www.astronomer.io/docs/astro/api/v-1/deployment/create-a-deployment#request.body.CreateStandardDeploymentRequest.scalingSpec).
</Info>

To manage wake schedules:

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.

    2. Click the Deployment's **More actions** menu (⋯), then select **Edit Deployment**.

    3. In the **Advanced** section, find **Wake Schedules** and set one or more schedules using either the calendar-based interface or cron expressions. You can switch between the two interfaces by clicking the **Switch to cron scheduling** or **Switch to simple scheduling** buttons.

       <Info>To keep the Deployment hibernated indefinitely, select the **Never Wake** schedule type in the simple scheduling interface. You can also configure **Never Wake** in the cron scheduling interface by setting `@always` as the hibernate expression and `@never` as the wake expression. You can only enable one **Never Wake** schedule at a time. You can wake up the Deployment by clicking **Wake Up Deployment** in the Deployment's **More actions** menu (⋯), or programmatically through the [Astro API](https://www.astronomer.io/docs/astro/api/v-1/deployment/configure-a-hibernation-override-for-a-deployment).</Info>

    4. Click **Update Deployment**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click Deployments, then select a Deployment.
    2. Click Details. In the Advanced section of your Deployment configuration, click Edit.
    3. In the **Wake Schedules** section, set one or more schedules using either the calendar-based interface or cron expressions. You can switch between the two interfaces by clicking the **Switch to cron scheduling** or **Switch to simple scheduling** buttons.
       <Info>To keep the Deployment hibernated indefinitely, select the **Never Wake** schedule type in the simple scheduling interface. You can also configure **Never Wake** in the cron scheduling interface by setting `@always` as the hibernate expression and `@never` as the wake expression. You can only enable one **Never Wake** schedule at a time. You can wake up the Deployment by clicking **Wake Up Deployment** in the Deployment's **More actions** menu, or programmatically through the [Astro API](https://www.astronomer.io/docs/astro/api/v-1/deployment/configure-a-hibernation-override-for-a-deployment).</Info>
    4. Click **Update Deployment**.
  </Tab>
</Tabs>

<Warning>Astro sets all cron schedules for hibernation in UTC. If you're running a Deployment in another time zone, you must convert the cron expression for your time zone to UTC. For example, if you want your Deployment to hibernate at 17:00 EST, you use `0 22 * * *` (22:00 UTC) as the cron expression.</Warning>

When your hibernation schedule starts:

* Your Deployment shows a **Hibernating** status in the Astro UI:

  <Frame>
    <img alt="A Deployment with a Hibernating status on the Deployments page of the Astro UI" />
  </Frame>
* Any task that was previously running will be killed and marked as failed.
* Tasks and Dags don't run. Task instances that were already running or scheduled at the time of hibernation will fail and trigger any related notifications.
* No Deployment resources are available. This includes the scheduler, webserver, and all workers.
* You can't access the Airflow UI for the Deployment.
* You can't deploy project images or Dags to the Deployment.

When your hibernation schedule ends, the Deployment will start any Dag runs for data intervals that were missed during hibernation for Dags with `catchup=True`. To avoid incurring additional resource costs, Astronomer recommends disabling catchup on Dags in hibernating Deployments.

### Manually hibernate a Deployment

Instead of creating a regular hibernation schedule, you can manually hibernate a development Deployment from the Astro UI. This is recommended if you're not sure when you'll need to use the Deployment again after hibernating it.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click the Deployment's **More actions** menu (⋯), then select **Hibernate Deployment**.
    3. Configure the manual hibernation period, then click **Confirm**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
    2. Click the **More Actions** menu of the Deployment you want to update, then select **Hibernate Deployment**.
    3. Configure the manual hibernation period, then click **Confirm**.
  </Tab>
</Tabs>

If you need to run a task or Dag on a Deployment that is currently in hibernation, you can manually wake up a Deployment from hibernation before the end of its schedule.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.

    2. Click the Deployment's **More actions** menu (⋯), then select **Wake Up Deployment**.

    3. Select one of the following options for how you want your Deployment to wake up:

       * **Wake until further notice**: Your Deployment wakes up immediately for an indefinite period and ignores any configured hibernation schedules.
       * **Wake until set time and date**: Specify a time that the Deployment should go back into hibernation after waking up.
       * **Remove override and return to normal schedule**: The Deployment returns to following your configured hibernation schedules.

    4. Click **Confirm**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.

    2. Click the **More Actions** menu of the Deployment you want to update, then select **Wake Up Deployment**.

    3. Select one of the following options for how you want your Deployment to wake up:

       * **Wake until further notice**: Your Deployment wakes up immediately for an indefinite period and ignores any configured hibernation schedules.
       * **Wake until set time and date**: Specify a time that the Deployment should go back into hibernation after waking up.
       * **Remove override and return to normal schedule**: The Deployment returns to following your configured hibernation schedules.

    4. Click **Confirm**.
  </Tab>
</Tabs>


# Overview
Source: https://astronomer.io/docs/astro/deployment-settings

View details and options for your Deployment.

After you create an Astro Deployment, you can modify its settings using the Astro UI and Astro CLI to tailor its performance. There are two categories of Deployment configurations: Deployment *details* and *resources*.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

You can also view more details about your Deployment including:

<Tabs>
  <Tab title="New Astro UI">
    * **Overview**: View information about your Deployment including recent Dag usage and the [code deploy history](/docs/astro/deploy-history).
    * **Analytics**: Find [detailed metrics](/docs/astro/deployment-metrics) about the historical resource consumption when you run Dags.
    * **Logs**: See the [Airflow and task logs](/docs/astro/view-logs) for a Deployment.
    * **Environment**: Manage [environment variables](/docs/astro/environment-variables), [connections](/docs/astro/create-and-link-connections), [metrics exports](/docs/astro/export-metrics), and [Airflow variables](/docs/astro/create-and-link-variables).
    * **Access**: Manage user, team, and API token access to the Deployment.
    * **Alerts**: Create, manage, and delete [notification channels and alerts](/docs/astro/alerts).
    * **Incident History**: Review [Deployment health incidents](/docs/astro/deployment-health-incidents).
    * **AI Agents**: Configure [Otto investigation guidance](/docs/astro/otto-investigate) for the Deployment.
    * **Details**: Make changes or reference your Deployment configurations.
  </Tab>

  <Tab title="Legacy UI">
    * **Overview**: View information about your Deployment including recent dag usage and the [code deploy history](/docs/astro/deploy-history).
    * **Analytics**: Find [detailed metrics](/docs/astro/deployment-metrics) about the historical resource consumption when you run dags.
    * **Logs**: See the [Airflow and task logs](/docs/astro/view-logs) for a Deployment.
    * **Environment**: Manage [environment variables](/docs/astro/environment-variables), [connections](/docs/astro/create-and-link-connections), [metrics exports](/docs/astro/export-metrics), and [Airflow variables](/docs/astro/create-and-link-variables).
    * **Access**: Manage user, team, and API token access to the Deployment.
    * **Alerts**: Create, manage, and delete [notification channels and alerts](/docs/astro/alerts).
    * **Details**: Make changes or reference your Deployment configurations.
  </Tab>
</Tabs>

## Deployment details

[Deployment details](/docs/astro/deployment-details) are high level settings that define how users can interact with the Deployment. These settings include:

* Deployment names and descriptions.
* Deployment contact emails.
* Rules for what types of deploys are allowed.
* Your Deployment's current Workspace.

## Deployment resources

[Deployment resources](/docs/astro/deployment-resources) let you customize the resource use, infrastructure, and performance of your Deployment. Use these settings to optimize Deployment processing, optimize compute resources and cost, and enable advanced use cases with intensive workloads. The following configuration options are available for customizing your resource use:

* Change the Deployment executor.
* Configure Kubernetes Pod resources.
* Change Scheduler and Dag processor resources.
* Enforce CI/CD Deploys.
* Enable High Availability.
* Customize your Airflow environment using environment variables.

For advanced Deployment resource configurations, see [Manage Airflow executors on Astro](/docs/astro/executors-overview) and [Configure worker queues](/docs/astro/configure-worker-queues).

## Configure Otto investigation guidance

Otto investigation guidance is custom text you provide to tailor [Otto investigations](/docs/astro/otto-investigate) for a Deployment. By default, a Deployment inherits the Otto investigation guidance set at the [Workspace level](/docs/astro/manage-workspaces#configure-otto-investigation-guidance). You can override the inherited guidance with Deployment-specific instructions.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, select a Deployment.
    2. Click the **AI Agents** tab, then click **Edit**.
    3. Enter up to 10,000 characters of markdown guidance, then click **Save**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Deployment.
    2. Click the **AI Agents** tab.
    3. Enter up to 10,000 characters of markdown guidance, then save your changes.
  </Tab>
</Tabs>

Deployment-level guidance takes precedence over Workspace-level guidance for investigations on this Deployment.

<Info>
  **CLI**

  These documents focuses on configuring Deployments through the Astro UI. To configure Deployments as code using the Astro CLI, see [Manage Deployments as code](/docs/astro/manage-deployments-as-code).
</Info>

## See also

* [Set environment variables on Astro](/docs/astro/environment-variables)
* [Authenticate an automation tool to Astro](/docs/astro/automation-authentication)
* [Manage Deployments as Code](/docs/astro/manage-deployments-as-code)


# Toggle AI features for your Organization
Source: https://astronomer.io/docs/astro/enable-disable-astro-ai

Configure Organization-wide access to AI features, including Otto and AI-driven capabilities in the Astro IDE.

Organization Owners can enable or disable AI features throughout the Astro platform. Disabling AI features removes access to all AI-driven enhancements for all users in your Organization, including:

* [Otto](/docs/astro/otto-overview), Astronomer's data engineering agent in the Astro CLI.
* AI-powered authoring in the [Astro IDE](/docs/astro/ide-overview).
* [Investigations and log summaries](/docs/astro/root-cause-analysis) in Astro Observe.

Use this setting if your Organization has security, privacy, or compliance requirements regarding AI use.

<Steps>
  <Step title="Open Organization Settings">
    In the Astro UI, click **Organization Settings**.
    This opens the **General** Organization page.
  </Step>

  <Step title="Edit Organization Details">
    In the **Organization Detail** section, click **Edit Details**.
  </Step>

  <Step title="Toggle AI Features">
    Find the **AI Features** option. Set the toggle to **Enabled** to turn on AI features or **Disabled** to turn them off.
  </Step>

  <Step title="Save changes">
    Click **Update Organization** to apply your changes.
  </Step>
</Steps>

<Info>
  When AI features are disabled, the Astronomer LLM gateway stops issuing model tokens for the Organization, so Otto and other Astronomer-managed AI features stop working.
</Info>


# Astro Environment Manager overview
Source: https://astronomer.io/docs/astro/environment-manager-overview

Learn how the Astro Environment Manager stores Airflow connections, Airflow variables, and environment variables and shares them across Deployments and Astro IDE projects.

The Astro Environment Manager is a built-in secrets backend that you use in the Astro UI to create Airflow connections, Airflow variables, environment variables, and metrics exports, then share them across multiple Deployments and [Astro IDE](/docs/astro/ide-overview) projects. The Environment Manager stores each object in an Astro-managed secrets backend, so you can create objects once and reuse them without setting up your own secrets backend.

Using the Environment Manager, you can configure an object with the credentials for a sandbox or development environment, apply it to all Deployments in a Workspace by default, and later override its values to point to production resources on a per-Deployment basis. When you create new Deployments, they automatically have access to any objects that are linked to all Deployments in the Workspace.

To open the Environment Manager, click **Environment** in the left menu of the Astro UI.

## What you can manage

The Environment Manager supports the following object types. See the corresponding page for each type's full capabilities, prerequisites, and permissions:

* **Airflow connections**: Store credentials and configuration for connecting to external services. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).
* **Airflow variables**: Store and retrieve arbitrary content or settings as key-value pairs within Airflow. See [Create Airflow variables in the Astro UI](/docs/astro/create-and-link-variables).
* **Environment variables**: Store key-value configurations that configure Airflow settings, store credentials, or pass configuration to your Dags. See [Create environment variables in the Astro UI](/docs/astro/create-and-link-environment-variables).
* **Metrics exports**: Send Deployment metrics to an external monitoring tool. See [Export metrics from Astro](/docs/astro/export-metrics).

Creating these objects in the Astro UI, rather than the Airflow UI, lets you share them across Deployments and Astro IDE projects, override their values per Deployment or project, and use them in local development and branch-based deploys.

## How objects are stored

Astro stores each object in an Astronomer-hosted secrets manager and applies it to Deployments as a Kubernetes Secret. See [How connections are stored](/docs/astro/create-and-link-connections#how-connections-are-stored), [How variables are stored](/docs/astro/create-and-link-variables#how-variables-are-stored), and [How environment variables are stored](/docs/astro/create-and-link-environment-variables#how-environment-variables-are-stored) for the mechanics of each object type.

## Use objects in local development

The Astro CLI can retrieve connections and variables from the Astro UI for use in local Airflow environments. By default, connections can't be exported locally. To work with connections locally, an Organization Owner can enable [**Environment Secrets Fetching**](/docs/astro/organization-settings#configure-environment-secrets-fetching-for-the-astro-environment-manager) in the Astro UI. See [Import and export connections and variables](/docs/cli/v1.43/local-connections).

## Link objects to Astro IDE projects

In addition to Deployments, you can link Workspace-level objects to [Astro IDE](/docs/astro/ide-overview) projects. When you link an object to an Astro IDE project, the project's ephemeral test Deployments start with that object available, without affecting Deployments that aren't started from the IDE. See [Link connections to Astro IDE projects](/docs/astro/create-and-link-connections#link-connections-to-astro-ide-projects).

<Warning>For Remote Execution Deployments, you can only manage environment variables and Metrics export with the Environment Manager. You can't create, update, or assign Airflow connections or Airflow variables.</Warning>

## Migrate and promote existing objects

If you already define connections, Airflow variables, or environment variables in a Deployment, you can move them into the Environment Manager so that they can be reused across Deployments and Astro IDE projects. You can also promote a Deployment-scoped object to the Workspace level to reuse it elsewhere. See [Migrate existing objects to the Environment Manager](/docs/astro/migrate-metadata-db-to-environment-manager).

## See also

* [Manage Airflow connections, variables, and environment variables](/docs/astro/manage-connections-variables)
* [Import and export connections and variables](/docs/astro/import-export-connections-variables)


# Execution mode
Source: https://astronomer.io/docs/astro/execution-mode

Overview of Remote and Hosted execution

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

## Overview

Starting with Airflow 3.x Deployments on Astro, you can choose Remote Execution mode to securely run your tasks in your own environments, whether it is on-prem or edge, without opening inbound connections. Only essential scheduling and health data leave the execution plane.

You can still use the default Hosted execution mode for Astronomer-Hosted execution and orchestration for hands-off infrastructure management. Hosted execution mode on Astro is the quickest and easiest way to run data pipelines on Airflow.

## Hosted execution

Hosted execution mode is Astronomer-Hosted execution and orchestration for hands-off infrastructure management. Hosted is the default execution mode and supports dag versioning for Airflow 3 Deployments and autoscaling workers out of the box.

Hosted execution offers a convenient, secure, and stable way to run Airflow workloads without having to manage workers in your environment.

<Frame>
  <img alt="Astro Hosted architecture overview" />
</Frame>

## Remote Execution

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

[Remote Execution](/docs/astro/remote-execution-overview) on Astro is built on a decoupled architecture that separates the orchestration layer from the execution layer so sensitive data never leaves local infrastructure, and your code remains in secure repositories.

<Frame>
  <img alt="Astro Remote architecture overview" />
</Frame>

* **Orchestration Plane** (Managed by Astronomer): The centralized control layer that contains the Scheduler, Web/API Server, metadata DB, and at its core, the Remote Execution API. The Remote Execution API securely manages all communication with your Remote Execution Agents—assigning tasks, tracking agent health, and maintaining observability without accessing your data or code.

* **Execution Plane** (Customer-Controlled): Your infrastructure—on-prem or in the cloud—where tasks actually run. This is where your code, data, task logs, and secrets stay. Inside this environment are [Remote Execution Agents](/docs/astro/remote-execution-configure-agents), lightweight services that pull tasks from the Remote Execution API, run them locally, and report status and metrics back to Astro. The Remote Execution Agent includes a Dag processor, Triggerer, and Worker(s). This means that dag parsing, task execution (deferrable and non-deferrable), sourcing of secrets at runtime, and logging all happen in the execution plane. Remote Execution Agent can be configured for different workloads, offering full flexibility without compromising security, and you can deploy them on platforms and hardware of your choosing.

<Note>Add `https://<clusterId>.external.astronomer.run/` to your organization's outbound network allowlist so the Remote Execution Agents in your environment are able to heartbeat to the API server in the Astro orchestration plane.</Note>


# Astro features
Source: https://astronomer.io/docs/astro/features

Set environment variables on Astro to specify Airflow configurations and custom logic.

Astro offers a suite of first-class features that make it easy to author, run, and monitor Airflow dags.

### Enhanced Developer Productivity

* **Otto:** Accelerate dag authoring, Airflow upgrades, and pipeline investigation with Otto, a data engineering AI agent that understands your environment and your team's conventions. See [Otto overview](/docs/astro/otto-overview).

* **Astro API:** Standardize and simplify data exchange, accelerate development, and better leverage third-party services. See [Astro API](/docs/astro/api/v-1/overview).

* **GitHub Integration:** Enforce DevOps best practices directly from the Astro UI, eliminating manual setup. See [Astro GitHub integration](/docs/astro/deploy-github-integration).

* **Astro CLI:** Install, run, and test Airflow from your command line in under five minutes. See [Astro CLI](/docs/cli/v1.43/overview).

* **Branch-Based Deploys:** Create isolated development environments to run, test, and deploy your dags. See [Preview Deployment templates](/docs/astro/ci-cd-templates/github-actions-deployment-preview).

* **Registry:** Discover over 1,500 integrations and dag templates to accelerate workflow development. See [Airflow Registry](https://airflow.apache.org/registry/).

* **dbt Deploys:** Unify the management and deployment of dbt and Airflow on Astro. See [dbt Deploys](/docs/astro/deploy-dbt-project).

### Elastic Infrastructure at Scale and Increased Operational Efficiency

* **Connection Management:** Set up Airflow connections in one central location with our out-of-the-box, native solution. See [Create connections in Astro](/docs/astro/create-and-link-connections).

* **Dynamic Workers:** Automatically adjust the number of worker nodes based on workload fluctuations.

* **Scale-to-Zero Deployment:** Schedule zero-cost downtime for development Deployment savings. See [Hibernate a development Deployment](/docs/astro/deployment-resources#hibernate-a-development-deployment).

* **High Availability:** Ensure the dependable delivery of mission-critical data pipelines with high availability schedulers. See [Enable high availability](/docs/astro/deployment-resources#enable-high-availability).

* **In-Place Upgrades:** Update to the latest Airflow version without costly downtime or a lengthy migration process.

* **Deployment Rollbacks:** Roll back your Airflow Deployments on Astro to any prior code deploy. See [Deploy history and rollbacks](/docs/astro/deploy-history).

* **CI/CD:** Automate reliable code deployment, enforce reviews, promote code across environments, and ensure testing. Seamlessly integrated with your favorite CI/CD tools. See [Enforce CI/CD deploys](/docs/astro/deployment-details#enforce-ci/cd-deploys).

* **Astro Terraform Provider:** Automate the management and scaling of Airflow environments with Terraform. See [Astro Terraform Provider](https://registry.terraform.io/providers/astronomer/astro/latest/docs).

* **Data-Centric Alerting:** Configure alerts based on dag run and task states, receiving real-time notifications for proactive monitoring. See [Astro Alerts](/docs/astro/alerts).

* **Organizational Dashboards:** Get actionable insight into real-time platform data for streamlined decision-making. See [Organization dashboards](/docs/astro/organization-dashboard).

* **Cross Deployment Visibility + Health:** Gain visibility of your dags and metadata across Deployments. See [Astro Observe](/docs/astro/astro-observe)

* **Lineage:** Visualize task relationships and track data movement within workflows, including data movement in external systems. See [Data lineage](/docs/astro/create-data-products).

* **Universal Metrics Export:** Export detailed metrics from your Airflow Deployments to Prometheus. See [Export metrics](/docs/astro/export-metrics).

### Robust Enterprise-Grade Security

* **Role-Based Access Control (RBAC):** Granular role-based access control spanning Workspaces, Teams, and Deployments. See [Astro user permissions reference](/docs/astro/user-permissions).

* **Customer Managed Workload Identity:** Securely authorize data services from Airflow on Astro, mitigating data exposure. See [Authorize Deployments to cloud resources](/docs/astro/authorize-deployments-to-your-cloud).

* **Custom Deployment Roles:** Ensure your data remains secure with enterprise-level security features. See [Customize Deployment roles](/docs/astro/customize-deployment-roles).

* **Private Networking:** Establish secure connections by VPC peering directly with your private networks. See [Create network connections between Astro and external resources](/docs/astro/networking-overview).

* **Secrets Management:** Automate secret management across Airflow deployments, ensuring a unified interface for seamless handling of secrets. See [Secrets management](/docs/astro/secrets-management).

* **Encryption:** Automatically encrypts all data, both at rest and in transit, including databases, file systems, and network transmissions. See [Data protection](/docs/astro/data-protection).

* **Compliance Readiness:** Meet SOC 2, GDPR, HIPAA, and PCI DSS security standards compliance.

### Committer-led Support

* **24x7x365 Support:** Access to the world's leading Airflow experts and committers. See [Submit a support request](/docs/astro/astro-support).

* **Education, Enablement, and Certification:** Build Airflow expertise across your organization with diverse training and certification options. See [Astronomer Academy](https://academy.astronomer.io/).


# Run your first Dag with the Astro CLI
Source: https://astronomer.io/docs/astro/first-dag-cli

Learn how to run your first Apache Airflow® Dag on Astro with the Astro CLI.

Astro is the industry's leading managed service for [Apache Airflow®](https://airflow.apache.org/). To quickly learn how Astro works, follow the steps in this quickstart to create an Airflow environment and run your first Dag with the Astro CLI.

Specifically, you will:

* Install the CLI.
* Authenticate and sign in to Astro.
* Create a Deployment.
* Create an Astro project.
* Deploy Dags to Astro with the Astro CLI.
* Trigger a run of an example Dag in the Airflow UI.

This tutorial takes about 15 minutes. If you're new to Airflow and want a more in-depth tutorial, see [Airflow 101 Learning Path](https://academy.astronomer.io/path/airflow-101).

If you want to deploy your first Dag without installing any software to your local machine, see [Run your first Dag with GitHub Actions](/docs/astro/first-dag-github-actions).

## Prerequisites

* An Astro account. To start an Astro trial and create your free trial account, see [Start a trial](/docs/astro/trial).

<Tip>You don't need Docker or Podman to run Airflow locally. Standalone mode runs Airflow directly on your machine in a virtual environment. See [Step 4](#step-4-test-your-project-locally) for details.</Tip>

<Info>
  If you're on your organization's network and can't access Astro, make a request to add the following domains to the allowlist on your network:

  * `https://cloud.astronomer.io/`
  * `https://api.astronomer.io/`
  * `https://images.astronomer.cloud/`
  * `https://auth.astronomer.io/`
  * `https://updates.astronomer.io/`
  * `https://install.astronomer.io/`
  * `https://astro-<organization-id>.datakin.com/`
  * `https://<organization-id>.astronomer.run/`
</Info>

## Step 1: Install the Astro CLI

<Tabs>
  <Tab title="Mac">
    <Tip>If you're encountering problems with installing the CLI or don't want to install software locally, see [Run your first Dag with GitHub Actions](/docs/astro/first-dag-github-actions).</Tip>

    Use [Homebrew](https://brew.sh/) to install the latest version of the [Astro CLI](/docs/cli/v1.43/overview).

    ```sh wrap theme={null}
    brew install astro
    ```

    For more information about Astro CLI install options and troubleshooting, see [Install the Astro CLI](/docs/cli/v1.43/install-cli).
  </Tab>

  <Tab title="Windows with winget">
    <Tip>If you're encountering problems with installing the CLI or don't want to install software locally, see [Run your first Dag with GitHub Actions](/docs/astro/first-dag-github-actions).</Tip>

    The winget command line tool is supported on Windows 10 1709 (build 16299) or later, and is bundled with Windows 11 and modern versions of Windows 10 by default as the App Installer. If you don't have winget, you can [Install the CLI on Windows manually](/docs/cli/v1.43/install-cli) instead.

    1. Make sure you have the following:
       * Windows Subsystem for Linux (WSL) 2 enabled: See [Enable the Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install), [WSL basic commands](https://learn.microsoft.com/en-us/windows/wsl/basic-commands), and [Troubleshooting WSL 2](https://learn.microsoft.com/en-us/windows/wsl/troubleshooting#error-0x80370102-the-virtual-machine-could-not-be-started-because-a-required-feature-is-not-installed).
         * After you enable WSL, run:

           ```sh wrap theme={null}
           wsl --update
           wsl --install --no-distribution
           ```

       * The latest version of the Windows [App Installer](https://apps.microsoft.com/store/detail/app-installer/9NBLGGH4NNS1?hl=en-ca\&gl=ca).

       * Windows 10 1709 (build 16299) or later or Windows 11.

    2. Open Windows PowerShell as an administrator and then run the following command:

       ```sh wrap theme={null}
       winget install -e --id Astronomer.Astro
       ```

    3. Run the following command to access the location of the CLI executable:

       ```sh wrap theme={null}
       $env:path.split(";")
       ```

       From the text that appears, copy the path for the Astro CLI executable. It should be similar to `C:\Users\myname\AppData\Local\Microsoft\WinGet\Packages\Astronomer.Astro_Microsoft.Winget.Source_8wekyb3d8bbwe`.

    4. Paste the path into File Explorer or open the file path in terminal, then rename the Astro executable to `astro.exe`.

    For more information about Astro CLI install options and troubleshooting, see [Install the Astro CLI](/docs/cli/v1.43/install-cli).
  </Tab>

  <Tab title="Linux">
    <Tip>If you're encountering problems with installing the CLI or don't want to install software locally, see [Run your first Dag with GitHub Actions](/docs/astro/first-dag-github-actions).</Tip>

    Run the following command to install the latest version of the Astro CLI directly to `PATH`:

    ```sh wrap theme={null}
    curl -sSL install.astronomer.io | sudo bash -s
    ```

    For more information about Astro CLI install options and troubleshooting, see [Install the Astro CLI](/docs/cli/v1.43/install-cli).
  </Tab>
</Tabs>

## Step 2: Create a Deployment

An Astro *Deployment* is an instance of Apache Airflow that is powered by all core Airflow components, including a webserver, scheduler, and one or more workers. You deploy Dags to a Deployment, and you can have one or more Deployments within a Workspace.

1. Log in to the [Astro UI](https://cloud.astronomer.io), then on the **Deployments** page click **+ Deployment**.

2. In the **Name** field, enter a name for your Deployment. You can leave the other fields at their default values. This creates a basic Deployment on a standard Astronomer-hosted cluster. You can delete the Deployment after you finish testing your example Dag runs.

3. Click **Create Deployment**.

   A confirmation message appears indicating that the Deployment status is **Creating** until all underlying components in the Deployment are healthy. During this time, the Airflow UI is unavailable and you can't deploy code or modify Deployment settings. When the Deployment is ready, the status changes to **Healthy**.

   For more information about possible Deployment health statuses, see [Deployment health](/docs/astro/deployment-health-incidents). Or, to learn more about how to customize your Deployment settings, see [Deployment settings](/docs/astro/deployment-settings).

## Step 3: Create an Astro project

An *Astro project* contains the set of files necessary to run Airflow, including dedicated folders for your Dag files, plugins, and dependencies. All new Astro projects contain two example Dags. In this tutorial, you'll be deploying these example Dags to your Deployment on Astro.

1. Open your terminal or IDE.

2. Create a new folder for your Astro project:

   ```sh wrap theme={null}
   mkdir <your-astro-project-name>
   ```

3. Open the folder:

   ```sh wrap theme={null}
   cd <your-astro-project-name>
   ```

4. Run the following Astro CLI command to initialize an Astro project in the folder:

   ```sh wrap theme={null}
   astro dev init
   ```

   The command generates the following files in your folder:

   ```text wrap theme={null}
   .
   ├── .env # Local environment variables
   ├── dags # Where your dags go
   │   ├── example-dag-basic.py # Example dag that showcases a simple ETL data pipeline
   │   └── example-dag-advanced.py # Example dag that showcases more advanced Airflow features, such as the TaskFlow API
   ├── Dockerfile # For the Astro Runtime Docker image, environment variables, and overrides
   ├── include # For any other files you'd like to include
   ├── plugins # For any custom or community Airflow plugins
   │   └── example-plugin.py
   ├── tests # For any dag unit test files to be run with pytest
   │   └── test_dag_example.py # Example test that checks for basic errors in your dags
   ├── airflow_settings.yaml # For your Airflow connections, variables and pools (local only)
   ├── packages.txt # For OS-level packages
   └── requirements.txt # For Python packages
   ```

## Step 4: Test your project locally

Before deploying to Astro, you can test your Dags locally. This step is optional, but Astronomer recommends testing locally to catch issues before deploying.

<Tabs>
  <Tab title="Docker / Podman">
    Run the following command from your project folder:

    ```sh wrap theme={null}
    astro dev start
    ```

    This command builds your project and spins up 4 containers on your machine, one each for the Airflow webserver, scheduler, triggerer, and metadata database.
  </Tab>

  <Tab title="Standalone (no Docker)">
    Standalone mode runs on macOS, Linux, and Windows Subsystem for Linux (WSL) for both Airflow 2 and Airflow 3. It doesn't support native Windows because of Apache Airflow limitations. For more information, see [Apache Airflow prerequisites](https://airflow.apache.org/docs/apache-airflow/stable/installation/prerequisites.html).

    Run the following command from your project folder:

    ```sh wrap theme={null}
    astro dev start --standalone
    ```

    This command runs Airflow directly on your machine in a virtual environment, without Docker or Podman. To make standalone mode the default for your project, run:

    ```sh wrap theme={null}
    astro config set dev.mode standalone
    ```
  </Tab>
</Tabs>

After your project starts, open the Airflow UI at `http://localhost:8080/` and verify that the example Dags appear. When you're done testing, run `astro dev stop` to stop the local environment.

## Step 5: Deploy example Dags to your Astro Deployment

Dag-only deploys are an Astro feature that you can use to quickly update your Astro Deployment by only deploying the `dags` folder of your Astro project. You'll now trigger a Dag-only deploy to push your example Dags to Astro.

1. Run the following command to authenticate to Astro on the CLI:

   ```sh wrap theme={null}
   astro login astronomer.io
   ```

   After running this command, you are prompted to open your web browser and enter your credentials to the Astro UI. The Astro UI then automatically authenticates you to the CLI. The next time you sign in, you can run `astro login` without specifying a domain. If you run into issues signing in, check to make sure that you have the latest version of the Astro CLI. See [Upgrade the CLI](/docs/cli/v1.43/upgrade-cli).

2. Run the following command to deploy your Dags to Astro:

   ```sh wrap theme={null}
   astro deploy --dags
   ```

   If you have modified any files other than those in the `dags` directory, make sure to run `astro deploy` instead.

   This command returns a list of Deployments available in your Workspace and prompts you to confirm where you want to deploy your Dag code. After you select a Deployment, the CLI parses your Dags to ensure that they don't contain basic syntax and import errors. If your code passes the parse, the Astro CLI deploys your Dags to Astro. If you run into issues deploying your Dags, check to make sure that you have the latest version of the Astro CLI. See [Upgrade the CLI](/docs/cli/v1.43/upgrade-cli).

## Step 6: Trigger your Dag on Astro

Newly deployed Dags are paused by default and won't start running automatically. To run one of the example Dags in your Astro project according to its schedule, you must unpause it from the Airflow UI hosted on your Deployment.

1. In the Deployment page of the Astro UI, click the **Open Airflow** button.

2. In the main Dags view of the Airflow UI, click the slider button next to `example-dag-basic` to unpause it. If you hover over the Dag, it says `dag is Active`. When you do this, the Dag starts to run on the schedule that is defined in its code.

   <Frame>
     <img alt="Pause Dag slider in the Airflow UI" />
   </Frame>

3. Manually trigger a Dag run of `example-dag-basic` by clicking the play button in the **Actions** column. When you develop Dags on Astro, triggering a Dag run instead of waiting for the Dag schedule can help you quickly identify and resolve issues.

   After you press **Play**, the **Runs** and **Recent Tasks** sections for the Dag start to populate with data.

   <Frame>
     <img alt="Dag running in the Airflow UI" />
   </Frame>

   These circles represent different [states](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#task-instances) that your Dag and task runs can be in.

4. Click the name of the Dag, `example-dag-basic`, to open the **Grid** view for the Dag. To see if your Dag ran successfully, the most recent entry in the grid should have green squares for all of your tasks.

5. Pause your Dag by clicking the slider button next to `example-dag-basic`. This prevents your example Dag from running automatically and consuming your Deployment resources.

## Step 7: View your Dag status in the Astro UI

The Astro UI shows you information about the health of your Deployment, including analytics and logs for your Dag runs.

Go back to your Deployment page in the Astro UI. Because you ran your example Dag, your Deployment information page now has data about your Deployment and Dag runs. The following example shows what you might find in the **Overview** page for your Deployment.

<Frame>
  <img alt="Summary information about your Dag runs in the Analytics tab of a Quickstart Deployment." />
</Frame>

When you're done exploring, you can delete your Deployment from your **Deployments** page.

## Next Steps

Now that you've created and run your first Dag on Astro, the next step is to add your own Dags, build out the rest of your Astro project, and start testing real data. See:

* [Develop a project](/docs/cli/v1.43/develop-project).
* [Write your first Dag](/docs/learn/get-started-with-airflow).
* [Deploy code to Astro](/docs/astro/deploy-code).


# Run your first Dag with GitHub Actions
Source: https://astronomer.io/docs/astro/first-dag-github-actions

Learn how to run your first Apache Airflow® Dag on Astro with GitHub Actions.

<Tip>The Astro GitHub integration can automatically deploy code from a GitHub repository to Astro without you needing to configure a GitHub action. In addition, the Astro UI shows Git metadata for each deploy on your Deployment information screen. See [Deploy code with the Astro GitHub integration](/docs/astro/deploy-github-integration) for setup steps.</Tip>

Astro is the industry's leading managed service for [Apache Airflow®](https://airflow.apache.org/). To quickly learn how Astro works, follow the steps in this quickstart to create an Airflow environment and run your first Dag with GitHub Actions.

Specifically, you will:

* Authenticate and sign in to Astro.
* Create a Deployment.
* Fork an example GitHub repository with a new Astro project.
* Configure GitHub Actions.
* Trigger the GitHub Action to deploy an example Dag to Astro.
* Trigger a run of the example Dag in the Airflow UI.

The steps take about 15 minutes. If you prefer to use a CLI, you can alternatively create and run your first Dag [using the Astro CLI](/docs/astro/first-dag-cli) in the same amount of time.

This tutorial assumes that you're familiar with basic Apache Airflow concepts. If you're new to Airflow and want a more general introduction, see the [Airflow 101 Learning Path](https://academy.astronomer.io/path/airflow-101).

## Prerequisites

* An Astro account. To start an Astro trial and create your free trial account, see [Start a trial](/docs/astro/trial).
* A [GitHub account](https://docs.github.com/en/get-started/signing-up-for-github).

<Info>
  If you're on your organization's network and can't access Astro, make a request to add the following domains to the allowlist on your network:

  * `https://cloud.astronomer.io/`
  * `https://api.astronomer.io/`
  * `https://images.astronomer.cloud/`
  * `https://auth.astronomer.io/`
  * `https://updates.astronomer.io/`
  * `https://install.astronomer.io/`
  * `https://install.astronomer.io/`
  * `https://astro-<organization-id>.datakin.com/`
  * `https://<organization-id>.astronomer.run/`
</Info>

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Step 1: Create a Deployment

An Astro *Deployment* is an instance of Apache Airflow that is powered by all core Airflow components, including a webserver, scheduler, and one or more workers. You deploy Dags to a Deployment, and you can have one or more Deployments within your Workspace.

1. Log in to the [Astro UI](https://cloud.astronomer.io), then on the **Deployments** page click **+ Deployment**.

2. In the **Name** field, enter a name for your Deployment. You can leave the other fields at their default values. This creates a basic Deployment on a standard Astronomer-hosted cluster. You can delete the Deployment after you finish testing your example Dag runs.

3. Click **Create Deployment**.

   A confirmation message appears indicating that the Deployment status is **Creating** until all underlying components in the Deployment are healthy. During this time, the Airflow UI is unavailable and you can't deploy code or modify Deployment settings. When the Deployment is ready, the status changes to **Healthy**.

   For more information about possible Deployment health statuses, see [Deployment health](/docs/astro/deployment-health-incidents). Or, to learn more about how to customize your Deployment settings, see [Deployment settings](/docs/astro/deployment-settings).

<Tip>
  Astro contains an in-product tutorial that guides you through Steps 2-4 of this document and includes shortcut buttons for some key Astro actions. If you prefer to finish the quickstart this way, open your **Deployments** page in the Astro UI and choose your Deployment. In the **Deploy your first dag** section, click **With GitHub Actions** and follow the steps in the window that appears.

  If you don't see the **Deploy your first dag** option your Deployment page, click **Deploy dags?** to open it.
</Tip>

## Step 2: Fork the example project repository

This repository contains an *Astro project*, which is a collection of files required for running Airflow on Astro. An Astro project includes folders for Dag files, plugins, dependencies, and more. Specifically, this Astro project includes an example Dag which is a simple ETL pipeline example that queries the list of astronauts currently in space from the Open Notify API and prints a statement for each astronaut. The Dag uses the TaskFlow API to define tasks in Python, and dynamic task mapping to dynamically print a statement for each astronaut.

1. Open [the example project repository](https://github.com/astronomer/astro-example-dags/fork) in a new tab or browser window.

2. **Choose an owner** from your available options.

3. Keep the selection to **Copy the `main` branch only**.

4. Click **Create fork**.

## Step 3: Set up the GitHub Actions Workflow

This example repository also includes a pre-configured [Astronomer deploy action](https://github.com/astronomer/deploy-action), which you can use to set up a CI/CD deployment pipeline. In this step, you'll configure the GitHub action to deploy code from your forked repository to Astro and run the workflow.

<Tabs>
  <Tab title="New Astro UI">
    1. Open two browser windows: one with the [Astro UI](https://cloud.astronomer.io), and one with your forked GitHub repository.

    2. In the Astro UI, choose the Deployment where you want to deploy your Astro project.

    3. In GitHub, open your forked repository and click **Actions**.

    4. Click **I understand my workflows, go ahead and enable them.**

    The [workflow](https://github.com/astronomer/astro-example-dags/blob/main/.github/workflows/deploy-to-astro.yaml) is a script that uses API tokens to deploy Dags from a GitHub repository to your Deployment, without requiring any local development.

    5. Choose the **Astronomer CI - Deploy Code** workflow.

    6. Click **Run workflow**. This opens a window to enter information about your Astro Deployment.

    7. In the Astro UI, copy your **Deployment ID** from the Deployment information.

    8. In GitHub, paste your **Deployment ID**.

    9. In the Astro UI, click the **Access** tab on the Deployment screen.

    10. Click **API Tokens**.

    11. Click **+ Add API Token** and select **New Deployment API Token** to create a new API token, and give the token a **Name** and an **Expiration**.

    12. Click **Create API Token**, then copy the token that appears.

        <Warning>For security reasons, this is the only opportunity you have to copy your API token. After you exit the modal window, you can't copy it again. Be sure to save your token in a safe place or paste it immediately.</Warning>

    13. In GitHub, paste the API Token in the **API Token** field on your GitHub Actions workflow page.

    14. Click **Run workflow**.
  </Tab>

  <Tab title="Legacy UI">
    1. Open two browser windows: one with the [Astro UI](https://cloud.astronomer.io), and one with your forked GitHub repository.

    2. In the Astro UI, choose the Deployment where you want to deploy your Astro project.

    3. In GitHub, open your forked repository and click **Actions**.

    4. Click **I understand my workflows, go ahead and enable them.**

    The [workflow](https://github.com/astronomer/astro-example-dags/blob/main/.github/workflows/deploy-to-astro.yaml) is a script that uses API tokens to deploy Dags from a GitHub repository to your Deployment, without requiring any local development.

    5. Choose the **Astronomer CI - Deploy Code** workflow.

    6. Click **Run workflow**. This opens a window to enter information about your Astro Deployment.

    7. In the Astro UI, copy your **Deployment ID** from the Deployment information.

    8. In GitHub, paste your **Deployment ID**.

    9. In the Astro UI, click the **Access** tab on the Deployment screen.

    10. Click **API Tokens**.

    11. Click **+ API Token** and select **Add Deployment API Token** to create a new API token, and give the token a **Name** and an **Expiration**.

    12. Click **Create API Token**, then copy the token that appears.

        <Warning>For security reasons, this is the only opportunity you have to copy your API token. After you exit the modal window, you can't copy it again. Be sure to save your token in a safe place or paste it immediately.</Warning>

    13. In GitHub, paste the API Token in the **API Token** field on your GitHub Actions workflow page.

    14. Click **Run workflow**.
  </Tab>
</Tabs>

This automatically deploys the example Dags in your Astro project to your Deployment.

## Step 4: View your Dag run results

Open your Deployment in the Astro UI and click **DAGs** in the left sidebar, then click **Open in Airflow** for **example\_astronauts**. This opens the Dag details page for **example\_astronauts** in the Airflow UI. Each column in the grid represents a complete Dag run, and each block in the column represents a specific task instance.

<Frame>
  <img alt="Detailed view of the Dag run outcome." />
</Frame>

Congratulations! You deployed and ran your first Dag on Astro with GitHub Actions.

## Next Steps

* Develop your [Astro project](/docs/cli/v1.43/run-airflow-locally).
* Read more about [Developing CI/CD workflows](/docs/astro/set-up-ci-cd).
* Install [the CLI](/docs/cli/v1.43/install-cli) to test Dags or run Airflow locally.
* [Write your first Dag](/docs/learn/get-started-with-airflow).


# Run your first dag on Astro
Source: https://astronomer.io/docs/astro/first-dag-onboarding

Learn how to run your first Apache Airflow® dag on Astro when you sign up for Astro.

Astro is the industry's leading managed service for [Apache Airflow®](https://airflow.apache.org/). To quickly learn how Astro works, follow the steps in this quickstart to create an Airflow environment and run your first dag.

When you first sign up for Astro, you can choose to tailor your experience based on how you plan to use Astro. The onboarding process includes:

* Creating an Organization, Workspace, and Deployment
* Choosing an example Airflow project template to clone into your own sample GitHub Repository
* Connecting your Astro Deployment to your sample GitHub Repository, so you can deploy code from GitHub
* Running your first dag on Astro

This tutorial takes about 15 minutes. If you're new to Airflow and want a more in-depth tutorial, see [Airflow 101 Learning Path](https://academy.astronomer.io/path/airflow-101).

## Prerequisites

* An Astro account. To start an Astro trial and create your free trial account, see [Start a trial](/docs/astro/trial).
* (Recommended) A [GitHub account](https://docs.github.com/en/get-started/signing-up-for-github).

<Info>
  If you're on your organization's network and can't access Astro, make a request to add the following domains to the allowlist on your network:

  * `https://cloud.astronomer.io/`
  * `https://api.astronomer.io/`
  * `https://images.astronomer.cloud/`
  * `https://auth.astronomer.io/`
  * `https://updates.astronomer.io/`
  * `https://install.astronomer.io/`
  * `https://astro-<organization-id>.datakin.com/`
  * `https://<organization-id>.astronomer.run/`
</Info>

## Step 1: Tailor your Astro experience

Based on how you plan to use Astro, you can find more resources for particular topics relevant to your needs.

* **Business** or **Personal** use
* Familiarity with Apache Airflow
* Use cases, including:
  * AI/ML
  * Business Operations
  * Reporting and Analytics
  * Extract, transform, and loading operations (ETL)
  * Other

## Step 2: Create your Organization and Workspace

Enter a name for your Organization and Workspace. These can be changed later in the Astro UI.

An Astro *Organization* is the highest level entity in Astro and represents a shared space for your company on Astro. An Organization is automatically created when you first sign up for Astronomer. At the Organization level, you can manage all of your users, Deployments, Workspaces, and clusters from a single place in the Astro UI. A *Workspace* is a collection of Deployments that can be accessed by a specific group of users.

## Step 3: Select a template

An *Astro project* contains the set of files necessary to run Airflow, including dedicated folders for your dag files, plugins, and dependencies. Select a template Astro project to immediately get a demonstration of Airflow's orchestration capabilities.

* **Generative AI**: Airflow is a common orchestration engine for AI/Machine Learning jobs, especially for retrieval-augmented generation (RAG). This [generative AI project](https://github.com/astronomer/templates/blob/main/generative-ai/README.md) shows a simple example of building vector embeddings for text and then performing a semantic search on the embeddings.
* **ETL**: Use a template to make an example [ETL pipeline](https://github.com/astronomer/templates/blob/main/etl/README.md). This template shows an example pattern for defining an ETL workload using DuckDB as the data warehouse of choice.
* **Learning Airflow**: This example project is generated when you run `astro dev init` using the Astro CLI. It shows a basic Astro project with Airflow components and dags.
* **dbt on Astro**: This template showcases using [dbt and Airflow together with Cosmos](https://github.com/astronomer/templates/blob/main/dbt-on-astro/README.md), allowing you to deploy dbt in production according to Airflow best practices.
* **None**: This option allows you to manually deploy a project after making a Deployment. See [Deploy code with GitHub](/docs/astro/deploy-github-integration) to use a GitHub connection with an existing project in a repo. Or, you can choose to manually build and Deploy an example dag by following the steps in [Run your first dag with the Astro CLI](/docs/astro/first-dag-cli) or [Run your first dag with GitHub Actions](/docs/astro/first-dag-github-actions).

## Step 4: Set up your Astro Deployment

You can now confirm your selected template and finish configuring your Deployment resources and whether you want to use a GitHub connection for code deploys.

An Astro *Deployment* is an instance of Apache Airflow that is powered entirely by core Airflow components, including a webserver, scheduler, and one or more workers. You deploy dags to a Deployment, and you can have one or more Deployments within a Workspace.

* **Selected template**: When you finalize your Deployment, you can confirm or change the template option you selected in Step 3.
* **Git connection**: Enter your **GitHub Owner** and **Repository Owner** information to deploy code directly to an Astro Deployment when you merge changes to a specific branch. If you choose to skip connecting to GitHub, later, you can return and manually build and Deploy an example dag by following the steps in [Run your first dag with the Astro CLI](/docs/astro/first-dag-cli) or [Run your first dag with GitHub Actions](/docs/astro/first-dag-github-actions).
* **Review Deployment**: Enter a **Name** for your Deployment and choose the **Provider** that you want to use and **Region** where you'd like to host your resources.

After you finish configuring your Deployment, click **Create your Project Repository and Deploy to Astro**.

## Step 5: Run your first dag on Astro

When you create your project and deploy it to Astro, this automatically creates a new Deployment and makes a private GitHub repository in your account with the template that you selected. Then, you can open Airflow and run your dag.

Newly-deployed dags are paused by default and will not start running automatically. To run one of the example dags in your Astro project according to its schedule, you must unpause it from the Airflow UI hosted on your Deployment.

If you did not connect to GitHub or choose a sample template to use, you can instead follow the steps in [Run your first dag with the Astro CLI](/docs/astro/first-dag-cli) or [Run your first dag with GitHub Actions](/docs/astro/first-dag-github-actions) to start working in Astro.

1. Click **Go to your Deployment**.

2. Click **Deployments** and then choose your active Deployment to view details about it.

3. Select **Deploy History** to see when your code successfully deploys from your GitHub repository.

4. After your deploy is successful, click **Open Airflow** to access the Airflow UI.

5. Manually trigger a dag run by clicking the play button in the **Actions** column. When you develop dags on Astro, triggering a dag run instead of waiting for the dag schedule can help you quickly identify and resolve issues.

6. (Optional) Depending on the template you use, you might be prompted to configure some parameters for your dag to use. You can choose to keep the default parameters, or make changes from the default. When you finish making changes, click **Trigger** to run your dag.

   After you press **Play**, the **Runs** and **Recent Tasks** sections for the dag start to populate with data.

   <Frame>
     <img alt="Dag running in the Airflow UI" />
   </Frame>

   These circles represent different [states](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#task-instances) that your dag and task runs can be in.

7. Click the name of the dag to view more details about its execution.

<Tip>To learn more about the Airflow UI and running dags, see [An introduction to the Airflow UI](/docs/learn/airflow-ui)</Tip>

8. After your dag finishes executing, click **Logs** to view any printed output of your dag's results.

Congratulations! You deployed and ran your first dag on Astro!

## Next Steps

* Develop your [Astro project](/docs/cli/v1.43/run-airflow-locally).
* Read more about [Developing CI/CD workflows](/docs/astro/set-up-ci-cd).
* Install [the CLI](/docs/cli/v1.43/install-cli) to test dags or run Airflow locally.
* [Write your First dag](/docs/learn/get-started-with-airflow).


# Otto quickstart
Source: https://astronomer.io/docs/astro/get-started-otto

Run your first Otto session to author, debug, and manage Airflow Dags with AI assistance.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

This quickstart walks you through your first Otto session. By the end, Otto is running in your project and ready to help you author, debug, and manage Dags. You can also optionally start local Airflow so Otto can query Dag runs, task logs, connections, and variables.

<Note>
  Otto requires an Astro account. [Start a free trial](/docs/astro/trial) if you don't have one.
</Note>

## Prerequisites

* [Astro CLI v1.42+](/docs/cli/v1.43/install-cli) installed.
* An [Astro account](/docs/astro/trial). Run `astro login`.
* (Optional) An Astro project folder. Otto works without an Astro project, but project context makes it more useful.

## Step 1: Launch Otto

From your Astro project folder, run:

```sh wrap theme={null}
astro otto
```

Otto opens an interactive terminal interface. The CLI handles authentication automatically.

<Tip>
  **Have Otto start local Airflow**

  You can also prompt Otto to start a local Airflow environment:

  > "Configure this project for standalone mode and start local Airflow."

  With Airflow running, Otto can use the [`af` CLI](/docs/astro/otto-tools#af-cli) to query Dag runs, task logs, connections, and variables on the live instance, and [automatic Dag validation](/docs/astro/otto-tools#automatic-dag-validation) runs after each edit. See [Local Airflow overview](/docs/cli/v1.43/local-airflow-overview) for the standalone vs Docker comparison.
</Tip>

## Step 2: Run your first prompt

Enter a prompt in the input area. To explore your project, try:

> "Look at my Dags and summarize each one."

Otto reads your Dag files, checks the running Airflow instance for metadata, and provides a summary of each Dag with its schedule, tasks, and dependencies.

### Other prompts to try

> "Check my Dags for deprecated patterns."

Otto validates your code against your Airflow version.

> "Why did my last Dag run fail?"

Otto checks task logs and run history to diagnose failures.

> "Create a new Dag that pulls data from Snowflake and loads it into Postgres."

Otto authors a Dag using your project's conventions and connection IDs.

## Step 3: Resume a session

Otto persists session history automatically. To continue your most recent session:

```sh wrap theme={null}
astro otto --continue
```

To pick from a list of previous sessions in an interactive picker:

```sh wrap theme={null}
astro otto --resume
```

To open a specific session file:

```sh wrap theme={null}
astro otto --session <path-to-session-file>
```

Session files are stored as JSONL at `~/.astro/otto/sessions/`.

## What Otto knows about your project

When you launch Otto in an Astro project folder, it automatically detects:

* **Project root**: The folder containing `.astro/` configuration. If Otto doesn't find one, it falls back to the current working directory.
* **Airflow URL**: The local Airflow instance started by `astro dev start`.
* **Memory**: Existing memory files in `.astro/memory/` (shared project memory), `~/.astro/memory/<project-slug>/` (local project memory), or `~/.astro/memory/` (local user memory). See [Memory](/docs/astro/otto-memory).

## Next steps

* [Skills](/docs/astro/otto-skills) — Browse Otto's bundled skills and author your own for team-specific workflows.
* [Tools](/docs/astro/otto-tools) — Explore the full set of tools Otto uses, including the af CLI for Airflow.
* [Memory](/docs/astro/otto-memory) — Learn how Otto accumulates and uses team-specific knowledge.


# Author dags with the Astro IDE
Source: https://astronomer.io/docs/astro/ide-author-dags

Author Airflow dags in the Astro IDE with AI-assisted code generation and project rules.

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

The Astro IDE supports building, updating, understanding, and customizing Airflow dags with or without the Astronomer AI assistant. The AI features in the IDE are powered by [Otto](/docs/astro/otto-overview), Astronomer's data engineering agent. Otto brings Airflow-specific context, team memory, and specialized skills to code generation. For more details on Otto's capabilities, including memory, skills, and Airflow tools, see the [Otto overview](/docs/astro/otto-overview).

<Note>
  AI features can be enabled or disabled Organization-wide by an Organization Owner. When disabled, all AI-driven capabilities in the IDE are unavailable for all users. To learn how to configure this setting, see [Enable or disable AI features for your Organization](/docs/astro/enable-disable-astro-ai).
</Note>

## Author dag code with or without AI

The Astro IDE allows you to write or modify dag code directly. Or, you can use AI-powered features to generate or refactor code, including:

* Use context-aware AI tools to suggest, generate, or improve dag code, operator logic, and project structure.
* Ask the Astro IDE questions about the project to better understand existing dag code structure, logic, and dependencies.
* View the Astro IDE's reasoning, streaming live in the AI panel as it proposes changes.
* Review all proposed code changes before choosing to accept or reject them.
* Manually edit Python scripts, dag files, and supporting resources in the built-in editor.

<Warning>
  When you author dags in the Astro IDE, this consumes some of your Workspace AI credits. See the [Preview FAQs](https://www.astronomer.io/product/ide/#faq) for more details.
</Warning>

## Add custom project rules for code generation

You can define custom instructions or rules that guide the AI for all code generation tasks within your Airflow project.

<Steps>
  <Step title="Open Project Settings">
    Open the **Project Settings** from the **More Actions** menu in the toolbar.
  </Step>

  <Step title="Add custom instructions">
    Enter your linting rules, conventions, or any special instructions for the Astro IDE in the **Custom Instructions** field.
  </Step>

  <Step title="Save your changes">
    Click **Update Project** to apply your custom rules. These instructions will be used for all AI-generated code in this project.
  </Step>
</Steps>


# Blueprint
Source: https://astronomer.io/docs/astro/ide-blueprint

Use Blueprint to create Airflow Dags from platform-defined templates in the Astro IDE, without writing Airflow code.

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

Blueprint is a template-based Dag authoring mode in the Astro IDE. Platform teams define reusable building blocks in Python. Analysts, analytics engineers, and data scientists assemble those blocks into production Dags through a visual builder and form-driven configuration, without writing Airflow code.

The open-source [astronomer/blueprint](https://github.com/astronomer/blueprint) library powers Blueprint in the Astro IDE, handling blueprint definition, YAML parsing, validation, and Dag generation. See the GitHub repository for the full API reference, blueprint authoring guide, and examples.

## How Blueprint works

Blueprint has two sides: platform teams define blueprints; other team members use them.

| Concept         | What it is                                                                                                                                              |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Blueprint**   | A reusable building block defined in Python by the platform team. Each blueprint has a configuration schema that controls what users see and configure. |
| **Step**        | An instance of a blueprint placed on the canvas and configured with specific values.                                                                    |
| **Workflow**    | A collection of steps with dependencies and a schedule, stored as a Dag YAML file in your project repository.                                           |
| **Airflow Dag** | The production Dag generated automatically from the workflow Dag YAML.                                                                                  |

## Prerequisites

* An Astro Workspace with Astro IDE access.
* [Workspace Author](/docs/astro/user-permissions) permissions to use the Astro IDE. You need [Workspace Operator](/docs/astro/user-permissions) permissions to start ephemeral test Deployments.
* The [`airflow-blueprint`](https://github.com/astronomer/blueprint) library (version 0.2.0 or later) added to your project's `requirements.txt`.
* At least one blueprint defined in your project with its JSON schema generated. See [Set up blueprints](#set-up-blueprints).

## Set up blueprints

Platform teams complete this setup so blueprints appear in the Astro IDE library for other team members to use.

<Steps>
  <Step title="Install the library">
    Add the following line to your project's `requirements.txt`:

    ```text title="requirements.txt" wrap theme={null}
    airflow-blueprint>=0.2.0
    ```
  </Step>

  <Step title="Define blueprints in Python">
    Create blueprint files in your project under `dags/templates/`. Each blueprint is a Python class that defines a configuration schema and a `render()` method. The field descriptions you write become the form labels users see in the IDE.

    The Astro IDE renders supported Pydantic field types as interactive form fields. The following types are supported:

    | Pydantic type                     | UI field             | Notes                                                              |
    | --------------------------------- | -------------------- | ------------------------------------------------------------------ |
    | `str`                             | StringField          |                                                                    |
    | `int`                             | NumberField          | step=1                                                             |
    | `float`                           | NumberField          |                                                                    |
    | `bool`                            | BooleanField         |                                                                    |
    | `Literal["a","b"]`                | EnumField            |                                                                    |
    | `list[str]`                       | ArrayField           |                                                                    |
    | `list[int]`                       | ArrayField           | Validates whole numbers on entry                                   |
    | `list[float]`                     | ArrayField           | Validates numeric input on entry                                   |
    | `list` (untyped)                  | ArrayField           | Treated as string array                                            |
    | `Field(ge=, le=, pattern=, etc.)` | Constraint props     | Includes `exclusiveMin`/`exclusiveMax`, converted to ±1            |
    | `Field(multiple_of=)`             | NumberField step     | Only for `float`; `int` always uses step=1                         |
    | `Optional[T]` / `T \| None`       | Unwraps to T's field | Constraints from inner type preserved                              |
    | `Optional[Literal["a","b"]]`      | EnumField            |                                                                    |
    | Multi-version blueprint           | Matched variant      | Matches by discriminator and `selectedVersion`; defaults to latest |

    The following types are not supported and render as an unsupported field with a warning in the IDE:

    | Pydantic type              | Notes                                    |
    | -------------------------- | ---------------------------------------- |
    | `str \| int` (true unions) | Shows "unsupported schema type" warning  |
    | Nested `BaseModel`         |                                          |
    | `list[Model]`              |                                          |
    | `list[bool]`               | Text input doesn't fit true/false values |
    | `dict[str, T]`             |                                          |

    For full details on defining blueprints, including configuration models, field validation, and versioning, see the [Blueprint GitHub repository](https://github.com/astronomer/blueprint).
  </Step>

  <Step title="Create the loader file">
    Create a file at `dags/loader.py` with the following content:

    ```python title="dags/loader.py" wrap theme={null}
    from blueprint import build_all

    build_all()
    ```

    Airflow scans the `dags/` folder for Python files that contain Dag objects. This file tells Blueprint to discover all `.dag.yaml` files in your project, parse them, and register the resulting Dags with Airflow automatically.
  </Step>

  <Step title="Generate JSON schemas">
    For the Astro IDE to discover your blueprints, generate a JSON schema for each blueprint and output it to `blueprint/generated-schemas/` in your project:

    ```sh wrap theme={null}
    blueprint schema <blueprint-name> > blueprint/generated-schemas/<blueprint-name>.schema.json
    ```

    Repeat this command for each blueprint. The IDE reads this folder to determine which blueprints are available and uses the schemas to render configuration forms.

    <Note>
      After you add or update a blueprint, regenerate its schema so the Astro IDE picks up the changes.
    </Note>
  </Step>

  <Step title="Commit the schemas">
    Commit the generated schema files in `blueprint/generated-schemas/` to your repository so the IDE can read them when you or your team members open the project.
  </Step>
</Steps>

## Build a workflow with Blueprint

After your platform team has set up blueprints, other team members can build workflows in the Astro IDE without writing Airflow code.

<Steps>
  <Step title="Open Blueprint mode">
    1. Open your project in the Astro IDE.
    2. Click the **Blueprint** toggle in the top navigation bar. The interface switches to Blueprint mode.
  </Step>

  <Step title="Create a workflow">
    Click **New DAG** to create a new workflow. Enter a Dag ID and click **Generate DAG**. Blueprint creates a new Dag YAML file in your project's `dags/` folder.

    To open an existing workflow, select it from the workflow list.
  </Step>

  <Step title="Add steps from the library">
    1. Browse the library panel on the left side of the canvas. The library shows all blueprints your platform team has defined and registered.
    2. Drag a blueprint from the library onto the canvas. Each item you add to the canvas is a *step*.
    3. Repeat to add more steps.
  </Step>

  <Step title="Configure the Dag">
    To configure Dag properties such as a description, Dag ID, or schedule, click **DAG Properties**.
  </Step>

  <Step title="Configure each step">
    1. Click a step on the canvas to open its configuration form in the right panel.
    2. Fill in the fields your platform team has exposed. The IDE marks required fields. Field descriptions explain what each value controls.

    <Note>
      The configuration form only shows fields your platform team has explicitly exposed. Operational settings like retry logic, connection credentials, and default values are baked into the blueprint.
    </Note>
  </Step>

  <Step title="Set dependencies between steps">
    To define the order in which steps run, draw a dependency line between them:

    1. Hover over the step that should run first. A connection point appears at the edge.
    2. Click and drag from that connection point to the next step.
    3. Release to create the dependency. An arrow appears between the two steps.

    Steps with no dependencies run in parallel.
  </Step>

  <Step title="Set the schedule, description, or Dag ID">
    1. Open the **DAG Properties** panel.
    2. Enter a cron expression in the **Schedule** field.
    3. Configure any additional pipeline-level settings such as **Description**.
  </Step>
</Steps>

## Test a workflow

After building your workflow, test it in an ephemeral Deployment. See [Test and run code with the Astro IDE](/docs/astro/ide-test-run) for instructions.

### Fix issues

If a task fails or produces unexpected output:

* **Edit the configuration directly**: Click the step on the canvas to reopen its configuration form and adjust the values.
* **Use Ask AI to Fix**: Click **Ask AI to Fix** next to an error message for AI-generated suggestions on how to resolve the issue.

After making changes, click **Sync to Test** to push the updated configuration to the running Deployment without restarting it.

## Commit and deploy

After testing, commit your workflow and deploy it. See [Deploy code from the Astro IDE](/docs/astro/ide-deploy) or [Deploy code from the Astro IDE to Git](/docs/astro/ide-deploy-git).


# Deploy code from the Astro IDE
Source: https://astronomer.io/docs/astro/ide-deploy

Deploy Airflow dag changes from the Astro IDE directly to an Astro Deployment.

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

The Astro IDE offers an integrated workflow for deploying Airflow dag changes. This allows you to deploy your latest changes from the Astro IDE to your Astro environment.

## Deploy to an Astro Deployment

Follow these steps to deploy your latest changes from the Astro IDE to your Astro environment.

### Prerequisites

* An [Astro Deployment](/docs/astro/create-deployment).
* The project must have been:
  * Imported using the [Astro GitHub integration](/docs/astro/deploy-github-integration), or
  * Imported with the Astro CLI, or
  * Created directly in the Astro IDE.
    Projects imported via manual Git provider configuration cannot be deployed directly from the IDE. Commit and push your changes to your Git provider, then use your organization's standard CI/CD pipeline to deploy.

### Deploy your changes

<Steps>
  <Step title="Open the Deploy menu">
    In the Astro IDE, click the **More Actions** menu toolbar and select **Deploy Project...**.
  </Step>

  <Step title="Choose version and Deployment target">
    In the **Deploy Project** dialog:

    * Review or edit the **Version** you want to deploy.
    * Select the **Deployment** target from your available Deployments.
  </Step>

  <Step title="Deploy">
    Click **Deploy Project** to start the code deploy.
  </Step>

  <Step title="Confirm deployment">
    In the Astro UI, open the target Deployment and check **Deploy History** in the **Overview** tab.
    Confirm that the latest deployment entry shows **Astro IDE** in the **Deployed By** column to verify your code was successfully deployed from the IDE.
  </Step>
</Steps>


# Deploy code from the Astro IDE to Git
Source: https://astronomer.io/docs/astro/ide-deploy-git

Review, commit, and push Airflow dag changes from the Astro IDE to a connected Git repository.

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

The Astro IDE provides an integrated workflow for reviewing, committing, and deploying Airflow dag changes directly to a connected Git repository.

## Review, commit, and push changes with Git

The Astro IDE allows you to review, accept, and commit code updates directly to a connected Git repository.

### Prerequisites

* An Astro IDE project connected to a Git provider:
  * [Imported from GitHub](/docs/astro/ide-import-github-project) using the Astro GitHub integration, or
  * [Manually imported from any Git provider](/docs/astro/ide-import-git-project).

### Commit and push your changes

<Steps>
  <Step title="Review and accept changes">
    1. Review the suggested code changes directly in the code view and evaluate the AI’s suggestions and reasoning, visible in the left panel.
    2. Accept or reject each proposed change using the **Accept** or **Reject** button.
  </Step>

  <Step title="Prepare your commit">
    1. After accepting changes, click **Commit changes...** in the toolbar.
    2. In the commit dialog, you can enter a commit message manually, or click the wand icon to generate an AI summary. You can edit the AI-generated message before saving.
    3. Review the list of staged changes and confirm or switch the branch you are committing to.

    <Tip>
      There are two ways to create a new branch for your commits in Astro IDE:

      * Click the branch selector dropdown in the toolbar. In the input field that appears, type a new branch name and select the option to create a new branch from your current branch. Click **Create Branch**.
      * In the commit dialog, click the **Edit branch** button next to the branch name. Enter your new branch name, then click **Commit and push**.

      Both methods allow you to isolate your changes in a new branch.
    </Tip>
  </Step>

  <Step title="Commit and push to your Git branch">
    1. Click **Commit and push** to sync your changes with the selected branch in your remote Git repository.
    2. Confirm the commit in Git. Your message and all accepted changes appear in the repository history.
  </Step>
</Steps>


# Import CLI projects to the Astro IDE
Source: https://astronomer.io/docs/astro/ide-import-cli-project

Import an existing local Astro project into the Astro IDE using the CLI.

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

Import an existing project into the Astro IDE by uploading a local CLI project.

## Connect a CLI project

### Prerequisites

* A [local Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) directory that you want to import.
* The [Astro CLI](/docs/cli/v1.43/install-cli) installed and authenticated locally.
* [Workspace Author](/docs/astro/user-permissions) permissions.

### Import your project

<Steps>
  <Step title="Open Astro IDE">
    From the Astro UI main menu, select **Astro IDE**.
  </Step>

  <Step title="Open the Projects tab">
    Click **Projects** to open the Projects tab.
  </Step>

  <Step title="Click &#x22;+ Add Project&#x22;">
    Click **+ Add Project**, then select **Import CLI Project** from the dropdown menu.
  </Step>

  <Step title="Run the export command in your terminal">
    Copy the command from the dialog and run it in your terminal from the root of your existing Astro CLI project directory. This command uploads your local project files to the Astro IDE:

    ```sh wrap theme={null}
    astro ide project export
    ```
  </Step>

  <Step title="Refresh to complete import">
    Return to the Astro IDE and refresh the page to see your imported project.
  </Step>
</Steps>

## Start a new session with your imported project

After importing your project, you can start a new session with it.

### Prerequisites

You need to have an imported project in the Astro IDE. Follow the steps in [Connect a CLI project](#connect-a-cli-project) to import a CLI project if you haven't already.

### Open your project in a new session

<Steps>
  <Step title="Open the Astro IDE">
    From the Astro UI main menu, select **Astro IDE**.
  </Step>

  <Step title="Open the Projects tab">
    Click **Projects** to open the Projects tab.
  </Step>

  <Step title="Select your imported project">
    Click **New Session** next to the project you imported to start a new session with that project.
  </Step>
</Steps>

<Tip>
  **Resume a previous session**

  Once you have created a session with your imported project, you can close it, and later resume it. In the Astro IDE, go to the **Sessions** tab, and click the session you want to resume.
</Tip>


# Import a Git project to the Astro IDE
Source: https://astronomer.io/docs/astro/ide-import-git-project

Import an existing Git-based project from any provider into the Astro IDE.

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

Import an existing Git-based project from your preferred provider into the Astro IDE.

<Warning>
  You can't deploy code directly to an Astro Deployment from projects imported with a manual Git provider configuration.
  To deploy, commit and push your changes back to your Git provider, then trigger your usual CI/CD deployment process.
  Direct deploy from the IDE is only supported for projects imported with the Astro GitHub integration, the Astro CLI, or projects created in the IDE. See [Deploy code from the Astro IDE](/docs/astro/ide-deploy).
</Warning>

## Import a Git project

<Info>
  To import a GitHub project using the Astro GitHub integration, see the [GitHub import guide](/docs/astro/ide-import-github-project).
</Info>

### Prerequisites

* [Workspace Author](/docs/astro/user-permissions) permissions.
* A remote Git repository containing an Astro project.
* Access permissions to the repository.
* Project size under 200 MB and individual file sizes under 1 MB. See [Project requirements](/docs/astro/ide-overview#project-requirements).

### Import your project

<Steps>
  <Step title="Open the Astro IDE">
    From the Astro UI, select **Astro IDE**.
  </Step>

  <Step title="Open the Projects tab">
    Click **Projects** to open the **Projects** tab.
  </Step>

  <Step title="Add a project">
    Click **+ Add Project**, then select **Import Git Project** from the dropdown menu.
  </Step>

  <Step title="Configure manual Git provider connection">
    * Select your Git provider: **GitHub**, **GitLab**, **Bitbucket**, **Azure Repos**, or **Generic**.

    <Info>
      If you use self-hosted GitHub Enterprise Server, select **Generic** instead of **GitHub**. The **GitHub** option is only for GitHub.com (cloud-hosted) repositories.
    </Info>

    * Fill in the following fields:
      * **Account**: The organization or user as shown in the provider URL.
      * **Repository/Project**: The repository or project name.
      * **Astro Project Path**: The subdirectory where your Astro project is located, if not at repository root.
      * **Branch**: The branch to use. Defaults to the provider's default branch.

    * Select authentication type. Available authentication types may differ slightly for each Git provider:
      * **Basic**
      * **Personal access token**
      * **Deploy key or Personal SSH key**
      * **None (public repository)**: Select if you are connecting to a public repository that doesn't require authentication. You can't push changes to the remote repository without authentication.

    Click **Connect** to begin importing.
  </Step>

  <Step title="Complete import">
    After connecting, the Astro IDE clones your code and prepares the environment.
    Your imported project appears in the Astro IDE.
  </Step>
</Steps>

## Start a new session with your imported project

After importing your project, you can start a new session with it.

### Prerequisites

* An imported project in the Astro IDE. See [Import a Git project](#import-a-git-project) to import one if you haven't already.

### Open your project in a new session

<Steps>
  <Step title="Reopen the Astro IDE">
    From the Astro UI main menu, select **Astro IDE**.
  </Step>

  <Step title="Reopen the Projects tab">
    Click **Projects** to open the Projects tab.
  </Step>

  <Step title="Select your imported project">
    Click **New Session** next to the project you imported to start a new session with that project.
  </Step>
</Steps>

<Tip>
  **Resume a previous session**

  Once you have created a session with your imported project, you can close it, and later resume it. In the Astro IDE, go to the **Sessions** tab, and click the session you want to resume.
</Tip>


# Import GitHub projects to the Astro IDE
Source: https://astronomer.io/docs/astro/ide-import-github-project

Import an existing project into the Astro IDE by connecting your GitHub repository.

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

Import an existing project into the Astro IDE by connecting your GitHub repository.

## Connect a GitHub project

### Prerequisites

* A configured [Astro GitHub integration](/docs/astro/deploy-github-integration).
* [Workspace Author](/docs/astro/user-permissions) permissions.
* Project size under 200 MB and individual file sizes under 1 MB. See [Project requirements](/docs/astro/ide-overview#project-requirements).

### Connect your GitHub repository

<Steps>
  <Step title="Open the Astro IDE">
    From the Astro UI main menu, select **Astro IDE**.
  </Step>

  <Step title="Open the Projects tab">
    Click **Projects** to open the Projects tab.
  </Step>

  <Step title="Click &#x22;+ Add Project&#x22;">
    Click **+ Add Project**, then select **Import Git Project** from the dropdown menu.
  </Step>

  <Step title="Connect to a GitHub project">
    Click **Connect to GitHub project...**, then select a GitHub organization:

    * Choose your organization or personal GitHub account from the list.
    * Click **Add an organization from GitHub** if your organization isn't listed, and follow the prompts.
  </Step>

  <Step title="Select repository and configure import">
    Select a repository.

    * Choose from the **Repository** list of options. The Astro IDE only allows you to select from repositories that the GitHub app is authorized to access.
    * (Optional) Enter the **Astro Project Path** if your Astro project isn't in the repository root.
    * (Optional) Set a **Git Branch**. If not specified, the Astro IDE uses the default branch of the repository.
  </Step>

  <Step title="Connect">
    Click **Connect** to import your project. The Astro IDE can now load your repository and set up your environment.
  </Step>
</Steps>

## Start a new session with your imported project

After importing your project, you can start a new session with it.

### Prerequisites

You need to have an imported project in the Astro IDE. Follow the steps in [Connect a GitHub project](#connect-a-github-project) to import a GitHub project if you haven't already.

### Open your project in a new session

<Steps>
  <Step title="Open the Astro IDE">
    From the Astro UI main menu, select **Astro IDE**.
  </Step>

  <Step title="Open the Projects tab">
    Click **Projects** to open the Projects tab.
  </Step>

  <Step title="Select your imported project">
    Click **New Session** next to the project you imported to start a new session with that project.
  </Step>
</Steps>

<Tip>
  **Resume a previous session**

  Once you have created a session with your imported project, you can close it, and later resume it. In the Astro IDE, go to the **Sessions** tab, and click the session you want to resume.
</Tip>


# Astro IDE overview
Source: https://astronomer.io/docs/astro/ide-overview

Learn about the Astro IDE, a browser-based workspace for writing, testing, and deploying Airflow Dags.

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

The Astro IDE provides a browser-based workspace for Apache Airflow. In the IDE, you can write code, validate and test changes, and deploy or push your work directly, without a local development environment.

With Blueprint, the IDE also supports template-based Dag creation. Platform teams define reusable step templates with built-in guardrails, and team members such as analysts, analytics engineers, and data scientists can assemble them into production Dags through forms and a visual builder, without writing Airflow code or learning Airflow concepts.

## What is the Astro IDE?

The Astro IDE is an integrated development environment designed specifically for Airflow Dag development. The AI features are powered by [Otto](/docs/astro/otto-overview), Astronomer's data engineering agent, and help write and edit Dag code with awareness of your Airflow version, existing repository patterns, and Deployment environment. The Astro IDE has additional features that help you build, test, and ship Dags directly in the browser.

## Why use the Astro IDE?

General-purpose AI tools generate Dag code without understanding the specifics of your project, often resulting in output that requires significant modification. Traditional Dag development also demands manual local configuration and container management, which slows iteration. The Astro IDE removes these barriers, enabling you to:

* Write and update Airflow Dags more efficiently, with context-aware code generation.
* Test and deploy code directly from the browser with no extra steps or tools required.
* Apply project rules to help maintain code quality.
* Scale Dag creation beyond the platform team by letting analysts and other team members build workflows from governed templates using Blueprint.

## Core features

* **AI powered by Otto**: Generate Dag code that follows Airflow best practices, tailored to your project history, connections, and Astro Workspace. Otto brings Astronomer's proprietary knowledge about Airflow compatibility, team-specific memory, and specialized skills. [Learn more about authoring Dags](/docs/astro/ide-author-dags). For details on Otto's capabilities, see [Otto overview](/docs/astro/otto-overview).
* **No local setup required**: Run everything in the browser.
* **Project checks and guardrails**: Maintain project standards and code quality through linting and project rules. [Add custom project rules](/docs/astro/ide-author-dags#add-custom-project-rules-for-code-generation).
* **Integrated deployment and version control**: Deploy to Astro and push changes to branches. [See Deployment workflows](/docs/astro/ide-deploy).
* **Blueprint (Preview)**: Create Dags by assembling platform-defined templates through a visual builder. Platform teams control defaults, guardrails, and operational settings while other team members focus on their SQL or Python logic. [Learn more about Blueprint](/docs/astro/ide-blueprint).

## What can you do with the Astro IDE?

* **Create new Dags**: Generate Airflow-compliant code, including common operators and projects, with AI assistance. [Author Dags](/docs/astro/ide-author-dags).
* **Create Dags from templates with Blueprint**: Browse a curated library of templates defined by your platform team, assemble them into workflows on a visual canvas, configure each step through simple forms, and test the result in an ephemeral deployment — all without writing Airflow code. [Use Blueprint](/docs/astro/ide-blueprint). *(Preview)*
* **Update and refactor**: Edit existing Dags, including performing Airflow version upgrades.
* **Import projects**: Bring code into the IDE from any supported Git provider:
  * [Import from GitHub](/docs/astro/ide-import-github-project) using the Astro GitHub integration.
  * [Import from any Git provider manually](/docs/astro/ide-import-git-project).
  * [Import a local project with the Astro CLI](/docs/astro/ide-import-cli-project).
* **Customize and extend**: Build custom operators or orchestrate dbt projects using [Cosmos](https://astronomer.github.io/astronomer-cosmos/).
* **Isolate test configuration per project**: Link connections, Airflow variables, and environment variables to specific Astro IDE projects so that ephemeral test Deployments start with the values you need, without affecting production Deployments. See [Link connections to Astro IDE projects](/docs/astro/create-and-link-connections#link-connections-to-astro-ide-projects). You can also [migrate existing connections and variables](/docs/astro/migrate-metadata-db-to-environment-manager) from a Deployment's metadata database into the Environment Manager.
* **Test and deploy**: Start an ephemeral test Deployment for quick feedback or test against an existing Astro Deployment, then deploy changes to Astro or GitHub. See [Test and run code](/docs/astro/ide-test-run), [Deploy code to Astro](/docs/astro/ide-deploy), and [Deploy code to Git](/docs/astro/ide-deploy-git).

## Project requirements

The following size limits apply to Astro IDE projects:

* **Maximum project size**: 200MB
* **Maximum file size**: 1MB

You cannot import or sync projects that exceed these limits. If your project exceeds these limits, consider excluding large files or folders that are not required for Dag development.

To get started, see the [IDE quickstart](/docs/astro/ide-quickstart).


# Astro IDE Quickstart
Source: https://astronomer.io/docs/astro/ide-quickstart

Create your first project in the Astro IDE and run it on an ephemeral test Deployment.

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

Get started with Astro IDE by entering a prompt, deploying to an ephemeral test Deployment, and testing your Airflow Dag.

## Prerequisites

* Workspace Operator permissions in an [Astro Workspace](/docs/astro/manage-workspaces).

<Note>
  Workspace Author permissions let you create and edit projects with the Astro IDE, but you need Workspace Operator permissions to start an ephemeral test Deployment.
</Note>

## Build and deploy your first Dag

<Steps>
  <Step title="Log in and create a project">
    1. Log in to the Astro UI.
    2. Click **Astro IDE** in the menu. If you are the first person in your Workspace to use the Astro IDE, the **Projects** page is empty.
    3. You have several options to get started in the Astro IDE:
       * [**Connect GitHub Project**](/docs/astro/ide-import-github-project) – Use the Astro GitHub integration to directly import a repository.
       * [**Connect a project from any Git provider**](/docs/astro/ide-import-git-project) – Configure a manual connection for GitLab, Bitbucket, Azure Repos, or any generic Git provider.
       * [**Connect CLI Project**](/docs/astro/ide-import-cli-project) – Upload a local Astro project using the CLI.
       * Enter a prompt in the text box describing what kind of pipeline you want to build. The Astro IDE will generate a new project from your prompt.
    4. This quickstart guides you through creating a new project from a prompt.

    <Note>
      Projects in the Astro IDE are visible to all members of your Workspace, but sessions are private. Only you can see and access your own sessions and conversation history.
    </Note>
  </Step>

  <Step title="Prompt, review, and accept your first Dag">
    1. In the Astro IDE, enter the following prompt in the text box describing the Hello World Dag you want to build. The more specific the prompt is, the better the agent performs. For this quickstart, enter:

    ```text wrap theme={null}
    Generate a dag called hello_world_dag that prints 'Hello World' to the logs. The dag should:
    - Not require the scheduler to be running.
    - Contain a single PythonOperator task named print_hello.
    ```

    <Note>
      The Astro IDE ephemeral test environment sets `AIRFLOW__SCHEDULER__USE_JOB_SCHEDULE=false` by default. This means scheduled or downstream tasks do not run automatically. This setting enables the **Test** tab to give you direct, manual control over when tasks run, instead of relying on Airflow’s scheduler.
    </Note>

    2. The Astro IDE generates the code and display the proposed changes in the suggested changes panel.
    3. Review the changes, and iterate on them with the AI panel. Once you are satisfied with the changes, click the **Keep changes** (✔️) button in the panel to approve the new Dag.
  </Step>

  <Step title="Deploy and test">
    1. After you accept a code change, click **Start Test Deployment** to test your changes. This deploys the generated code to an ephemeral test Deployment that allows you to run tasks and view logs.

    <Note>
      Instead of starting an ephemeral test Deployment, you can open the **Test** tab and click **Use an existing Deployment** to test against an Astro Deployment your team already runs. See [Test against an existing Deployment](/docs/astro/ide-test-run#test-against-an-existing-deployment).
    </Note>

    <Info>
      AI can make mistakes, and the generated code can sometimes throw an import error. If you have an import error, the `hello_world_dag` doesn't appear in the **Test** tab, and the IDE lists the error in the **Import Errors** of the **Test Output** pane. To fix the error, paste the error message into the prompt box, so the IDE can suggest edits. Review the suggested changes, accept them, and click **Sync to Test** to update the Deployment.
    </Info>

    2. Open the **Test** tab and select **hello\_world\_dag** from the Dag options.
    3. Click **Run Task** to execute the Dag.
    4. Confirm the task ran successfully by checking the **Task Logs**.
  </Step>
</Steps>

<Info>
  You can also view your ephemeral test Deployment in the Airflow UI by clicking **Open Airflow** from the toolbar menu.
</Info>

<Tip>
  Your session persists even if you close your browser. The AI agent continues processing prompts in the background, and your full conversation history is preserved. Return to the Astro IDE at any time to resume where you left off.
</Tip>


# Test and run code with the Astro IDE
Source: https://astronomer.io/docs/astro/ide-test-run

Test and run Airflow Dags in the Astro IDE using ephemeral test Deployments or an existing Astro Deployment.

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

The Astro IDE enables you to quickly test, validate, and iterate on Airflow Dags without needing a local Airflow or Docker environment. You can test your code in two ways:

* Start an ephemeral test Deployment to run your code in a temporary, isolated environment based on your project configuration.
* Test against an existing Astro Deployment in your Workspace to reuse the configuration, connections, and resources it already runs.

In both cases, you can see how your code runs in a real environment, adjust Deployment settings, and troubleshoot using built-in interfaces.

## How ephemeral test Deployments work

Ephemeral test Deployments offer a temporary environment to run and validate your Airflow Dag changes before merging or deploying to production.

* Start an ephemeral test Deployment to launch an isolated environment based on your project configuration.
* Run your Dag and validate behavior using real dependencies, environment variables, and integrations.
* Detect import errors, syntax issues, or runtime problems as they occur in a realistic environment.
* Tear down the environment automatically when testing is complete, with no impact on existing Deployments.

<Warning>
  Using ephemeral Deployments in the Astro IDE consumes some of your Workspace AI credits. See the [Preview FAQs](https://www.astronomer.io/product/ide/#faq) for more details.
</Warning>

## Test against an existing Deployment

Instead of starting an ephemeral test Deployment, you can point your session at an existing Astro Deployment in your Workspace and test your Dags against the configuration, connections, and resources it already runs. This skips provisioning and lets you validate changes against a Deployment your team owns.

When you attach a session to an existing Deployment, the Astro IDE deploys your session's image and Dags to that Deployment, overwriting the image and Dags currently running there. The Deployment keeps its own connections, Airflow variables, environment variables, and resource configuration, so your Dags run against the values already configured on that Deployment rather than the objects linked to your Astro IDE project.

Unlike ephemeral test Deployments, an existing Deployment runs with its own configuration, so the presets optimized for ephemeral testing don't apply. For example, the Astro IDE doesn't set `AIRFLOW__SCHEDULER__USE_JOB_SCHEDULE=false`, so the Airflow scheduler runs as configured on that Deployment.

<Note>
  You need the `workspace.deployments.create` permission to test against an existing Deployment. Workspace Operator and higher roles have this permission. If you don't have it, the option appears dimmed.
</Note>

### Select a target Deployment

1. On the **Test** tab, with no active test Deployment, click **Use an existing Deployment** beneath **Start Test Deployment**.
2. In the **Use an existing Deployment** dialog, select a Deployment from your Workspace. Type to search by name, and scroll to load more results.
3. If the Deployment is already in use by another session, selecting it opens a confirmation dialog with a warning.
4. Click **Use Deployment** to confirm. If another session was using the Deployment, this takes over that session. The Astro IDE deploys your session's image and Dags to the selected Deployment.

The picker excludes Astro IDE test Deployments. A Deployment appears in the list but is dimmed and you can't select it if any of the following apply:

* It isn't healthy.
* CI/CD is enforced on it.
* Dag deploys are disabled on it.
* Remote execution is enabled on it.

### Sync changes to an existing Deployment

While your session is attached to an existing Deployment, click **Sync to Test** to deploy your latest changes to that Deployment.

### Stop using an existing Deployment

When your session is attached to an existing Deployment, the toolbar shows **Deployment** instead of **Test Deployment**. To detach your session:

1. Click the **Deployment** dropdown arrow.
2. Select **Stop using Deployment**.

Detaching leaves the Deployment running with the code you deployed to it. This differs from stopping an ephemeral test Deployment, which deletes the Deployment.

## Configure ephemeral test Deployment settings

Ephemeral test Deployments in the Astro IDE inherit default settings from your Workspace-level Astro IDE test Deployment configuration. You can adjust session-specific settings per Deployment, such as environment variables and connections, as needed within the IDE. To make connections, Airflow variables, or environment variables available to a project's ephemeral test Deployments, link them to the Astro IDE project from the Workspace Environment Manager. See [Make connections, Airflow variables, and environment variables available](#make-connections-airflow-variables-and-environment-variables-available).

#### Workspace-level Astro IDE test Deployment settings

You can control default resource behavior for ephemeral test Deployments in your Workspace under **Workspace Settings** > **General** > **Astro IDE**:

* **Test Deployment Timeout:** The amount of idle time before an ephemeral test Deployment is automatically stopped.
* **Test Deployment Cluster:** The cluster on which all new ephemeral test Deployments are provisioned.
* **Auto-Start Test Deployments:** When enabled, the Astro IDE automatically starts an ephemeral test Deployment for every session.

These workspace-level settings apply to all ephemeral test Deployments in the Astro IDE and you can tailor them to your Organization's needs.

<Tip>
  Ephemeral test Deployments have some preset configurations optimized for testing. For example, `AIRFLOW__SCHEDULER__USE_JOB_SCHEDULE=false` disables automatic task scheduling, so tasks run only when manually triggered in the **Test** tab. This is by design for better test control.
</Tip>

## Make connections, Airflow variables, and environment variables available

Ephemeral test Deployments need access to the same connections, Airflow variables, and environment variables your Dags use against external systems, databases, and cloud services. Linking these objects to your Astro IDE project from the Workspace Environment Manager lets you validate code, integrations, and data dependencies in a realistic environment without sharing those values with your production Deployments.

Astronomer recommends linking environment objects to the specific Astro IDE project that needs them. Project-linked objects only affect that project's ephemeral test Deployments. See:

* [Link a connection to an Astro IDE project](/docs/astro/create-and-link-connections#link-a-connection-to-an-astro-ide-project)
* [Link an Airflow variable to an Astro IDE project](/docs/astro/create-and-link-variables#link-a-variable-to-an-astro-ide-project)
* [Link an environment variable to an Astro IDE project](/docs/astro/create-and-link-environment-variables#link-an-environment-variable-to-an-astro-ide-project)

If you want a value to be available to every Astro IDE project in the Workspace, configure project sharing on the object so that it auto-links to all projects. See [Configure project sharing for a Workspace](/docs/astro/create-and-link-connections#configure-project-sharing-for-a-workspace).

If you already define connections or Airflow variables in a Deployment's Airflow metadata database, you can move them into the Environment Manager and then link them to your Astro IDE projects. See [Migrate existing objects to the Environment Manager](/docs/astro/migrate-metadata-db-to-environment-manager).

## Use a `.env` file for ephemeral test Deployments

When you start an ephemeral test Deployment, the Astro IDE automatically imports environment variables from a `.env` file in your project root. This lets you configure environment variables for testing without adding them through the Environment Manager.

### `.env` file format

Add a `.env` file to your project root with one variable per line:

```text wrap theme={null}
MY_API_KEY=your-api-key
DATABASE_URL=postgresql://user:password@host:5432/db
FEATURE_FLAG=true
```

### Variable precedence

[System environment](/docs/astro/platform-variables) variables always take precedence. If a variable in your `.env` file has the same name as a system variable, such as an Airflow or scheduler setting, the system variable overrides the `.env` value.

<Warning>
  The Deployment does not store variables from your `.env` file as secrets. Use the [Environment Manager](/docs/astro/create-and-link-environment-variables) for sensitive credentials.
</Warning>

<Info>
  If the `.env` file is missing or invalid, the Deployment starts normally without importing variables.
</Info>

## Access the Airflow UI

While testing, open the Airflow UI for your ephemeral Deployment to inspect Dags, tasks, and metadata using Airflow-native tools.

1. Click **Start Test Deployment** from your Dag in Astro IDE. After the Deployment starts, a dropdown arrow appears.
2. Click the dropdown arrow and select **Open Airflow**. The Airflow UI for your ephemeral test Deployment will open in a new browser tab, where you can inspect Dags, tasks, logs, and environment details.


# Import and export Airflow connections and variables
Source: https://astronomer.io/docs/astro/import-export-connections-variables

Learn how to import and export Apache Airflow objects between Airflow environments on Astro

After you create connections and variables in an Airflow environment, you might want to move them between environments for any of the following reasons:

* You are launching a production Airflow environment on Astro based on a locally running Airflow environment.
* You need to replicate a production Airflow environment on your local machine.
* Your team is migrating old Airflow environments to a new location.

Based on the [management strategy for your connections and variables](/docs/astro/manage-connections-variables), their storage location will vary. Use this document to learn how to export and import them from one environment to another.

<Tip>If you use the Astro Environment Manager to [create connections](/docs/astro/create-and-link-connections), instead of importing and exporting connections from Airflow, you can configure the CLI to automatically retrieve connection details from Astro when you're working locally. See [Work locally with Airflow connections hosted on Astro](/docs/cli/v1.43/local-connections) to set up this configuration and learn more about how Astro stores and syncs connection information.</Tip>

## From the Airflow UI and metadata database

When you use the Airflow UI to store your Airflow connections and variables, they are stored in Airflow's metadata database. If your variables are stored in Airflow's metadata database, you can use the Airflow UI to import and export them in bulk.

### Using the Airflow UI

To export variables from a local Airflow environment or Astro Deployment, complete the following steps:

1. In the Airflow UI, go to **Admin** > **Variables**.

2. Select the variables you want to export, then click **Export** in the **Actions** dropdown menu.

   The selected variables are exported to your local machine in a file named `variables.json`.

   <Frame>
     <img alt="Export Variables" />
   </Frame>

To import variables to a local Airflow environment or Astro Deployment from a `json` file, complete the following steps:

1. Either create a new `json` file with environment variables or follow the previous steps to export a `json` file from an existing Airflow environment. Your `json` file should look similar to the following:

   ```json title="variables.json" wrap theme={null}
   {
       "my_string_var": "test",
       "my_int_var": 1234,
       "my_secret_var": "my_secret",
       "my_json_var": {
           "key1": "val1",
           "key2": 123
       }
   }
   ```

2. In the Airflow UI, go to **Admin** > **Variables**.

3. Click **Choose file** and select the file containing your environment variables. Then, click **Import Variables**.

   After the variables are updated, the UI will show a confirmation message.

   <Frame>
     <img alt="Import Variables" />
   </Frame>

For security reasons, you can't bulk import or export connections from the Airflow UI.

### Using the Astro CLI (Local environments only)

Use [`astro dev object export`](/docs/cli/v1.43/astro-dev-object-export), [`astro dev object import`](/docs/cli/v1.43/astro-dev-object-import), and [`astro dev run`](/docs/cli/v1.43/astro-dev-run) to import and export connections and variables from a local Airflow environment to various formats. For example:

* To export all Airflow objects including connections, variables, and pools to your Astro project `.env` file in a URI format, run:

  ```sh wrap theme={null}
  astro dev object export --env-export
  ```

* To print only your connections in JSON format to STDOUT, run:

  ```sh wrap theme={null}
  astro dev run connections export - --file-format=env --serialization-format=json
  ```

* To import all Airflow objects from a file named `myairflowobjects.yaml` to a locally running Airflow environment, run:

  ```sh wrap theme={null}
  astro dev object import --settings-file="myairflowobjects.yaml"
  ```

## From a secrets backend

If you use a secrets backend to store connections and variables, you need to use your secrets backend's API to manage connections and variables between environments. Note that importing/exporting from a secrets backend is necessary only in rare circumstances, such as when:

* You want to migrate your secrets backend to a different account.
* You want to use your secrets as a reference for another Deployment and then customize it. Astronomer recommends maintaining separate `development` and `production` secrets backend.
* You want to migrate to a different cloud provider.

Refer to your secrets backend provider's documentation to learn how to manage resources using API calls:

* Google secret manager's [Python SDK](https://cloud.google.com/secret-manager/docs/reference/libraries#client-libraries-install-python) and [REST API](https://cloud.google.com/secret-manager/docs/reference/rest) reference.
* AWS secret manager's [Python SDK](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html) and [other SDKs](https://docs.aws.amazon.com/secretsmanager/latest/apireference/Welcome.html).
* Azure key vault's [SDK and API](https://learn.microsoft.com/en-us/azure/key-vault/general/developers-guide#apis-and-sdks-for-key-vault-management) reference.
* Hashicorp Vault's [Python SDK](https://developer.hashicorp.com/vault/docs/get-started/developer-qs#step-2-install-a-client-library)

To set up a secrets backend on Astro, see [configure secrets backend](/docs/astro/secrets-backend).

## From environment variables

This section covers **Deployment-level environment variables**. For information about **Workspace-level environment variables** managed through the Environment Manager, see [Create environment variables in Astro](/docs/astro/create-and-link-environment-variables). Workspace-level environment variables cannot be bulk exported, similar to connections and Airflow variables in the Environment Manager.

If you use an `.env` file to manage your connections and variables as Deployment-level environment variables in a local Astro project, you can import your Airflow objects to another local Airflow Astro project by copying the `.env` file into the destination project.

The `.env` file is not pushed to your Deployment when you run `astro deploy`. However, you can use Astro CLI commands to import the contents of your `.env` file to an Astro Deployment using `astro deployment` commands.

If your connections and variables are stored in a local Airflow metadata database, you can also export these to a `.env` file and then import them to an Astro Deployment as environment variables.

* To export all Airflow objects from your local Airflow environment, including connections and variables, to an `.env` file in URI format, run:

  ```sh wrap theme={null}
  astro dev object export --env-export
  ```

* To import all Airflow objects from an `.env` file to an Astro Deployment, run:

  ```sh wrap theme={null}
  astro deployment variable create -d <deployment-id> --load --env .env
  ```

If your connections and variables are defined as environment variables on an Astro Deployment, you can export them to a local Airflow environment using the Astro CLI.

* To export variables from your Deployment, run:

  ```sh wrap theme={null}
  astro deployment variable list --deployment-id <deployment-id> --save
  ```

  This command exports your variables to a local file named `.env`. The values of secret environment variables will be redacted.

See [Set environment variables on Astro](/docs/astro/manage-env-vars).

## From the Airflow REST API

You can use the [Airflow REST API](/docs/astro/airflow-api) to import and export connections and variables from a Deployment or local Airflow environment. Note that the Airflow REST API can only access connections and variables that are stored in the Airflow metadata database.

To export connections from any Airflow environment, you can use the [List Connections API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_connections) and [Get Connection API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_connection).

To export variables from any Airflow environment, you can use the [List Variables API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_variables) and [Get Variable API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_variable).

To import connections or variables to any Airflow environment, you can use the [Create Connection API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/post_connection) and [Create Variable API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/post_variables) respectively.


# Run images from Amazon Elastic Container Registry (ECR)
Source: https://astronomer.io/docs/astro/kpo-ecr

Configure KPO to pull container images from Amazon ECR.

<Info>Policy-based setup is available only on Astro dedicated clusters. To run images from a private registry on Astro standard clusters, follow the steps in [Private Registry](/docs/astro/kpo-private-registry).</Info>

By default, the `KubernetesPodOperator` expects to pull container images that are hosted publicly. If your images are hosted on the container registry native to your cloud provider, you can grant access to the images directly.

## Prerequisites

* An [Astro project](/docs/cli/v1.43/get-started-cli).
* An [Astro Deployment](/docs/astro/deployment-settings).
* Access to an Amazon ECR registry.

## Setup

<Steps>
  <Step title="Add Amazon ECR repository permissions">
    If your Docker image is hosted in an Amazon ECR repository, add a permissions policy to the repository to allow the `KubernetesPodOperator` to pull the Docker image. You don't need to create a Kubernetes secret, or specify the Kubernetes secret in your dag. Docker images hosted in Amazon ECR repositories can only be pulled from AWS clusters.

    1. Log in to the Amazon ECR Dashboard and then select **Menu** > **Repositories**.

    2. Click the **Private** tab and then click the name of the repository that hosts the Docker image.

    3. Click **Permissions** in the left menu.

    4. Click **Edit policy JSON**.

    5. Copy and paste the following policy into the **Edit JSON** pane:

       ```json wrap theme={null}
       {
           "Version": "2008-10-17",
           "Statement": [
               {
                   "Sid": "AllowImagePullAstro",
                   "Effect": "Allow",
                   "Principal": {
                       "AWS": "arn:aws:iam::<AstroAccountID>:role/EKS-NodeInstanceRole-<ClusterID>"
                   },
                   "Action": [
                       "ecr:GetDownloadUrlForLayer",
                       "ecr:BatchGetImage"
                   ]
               }
           ]
       }
       ```

       * Replace `<AstroAccountID>` with your Astro AWS account ID.
       * Replace `<ClusterID>` with your Cluster ID. To find the Cluster ID, in the Astro UI go to **Settings** > **Clusters** (in the legacy UI, go to **Organization Settings** > **Clusters**), then select the cluster. The **ID** is displayed at the top, along with other information about the Astro Cluster.

    6. Click **Save** to create a new permissions policy named **AllowImagePullAstro**.
  </Step>

  <Step title="Set up the KubernetesPodOperator">
    The following snippet is the minimum configuration you'll need to create a `KubernetesPodOperator` task on Astro:

    ```python wrap theme={null}
    from airflow.configuration import conf
    from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

    namespace = conf.get("kubernetes", "NAMESPACE")

    KubernetesPodOperator(
        namespace=namespace,
        image="<your-docker-image>",
        cmds=["<commands-for-image>"],
        arguments=["<arguments-for-image>"],
        labels={"<pod-label>": "<label-name>"},
        name="<pod-name>",
        task_id="<task-name>",
        get_logs=True,
        in_cluster=True,
    )
    ```

    For each instantiation of the `KubernetesPodOperator`, you must specify the following values:

    * `namespace = conf.get("kubernetes", "NAMESPACE")`: Every Deployment runs on its own Kubernetes namespace within a cluster. Information about this namespace can be programmatically imported as long as you set this variable.
    * `image`: This is the Docker image that the operator will use to run its defined task, commands, and arguments. Astro assumes that this value is an image tag that's publicly available on [Docker Hub](https://hub.docker.com/). To pull an image from a private registry, see [Pull images from a Private Registry](/docs/astro/kpo-private-registry).
    * `in_cluster`: If a Connection object is not passed to the `KubernetesPodOperator`'s `kubernetes_conn_id` parameter, specify `in_cluster=True` to run the task in the Deployment's Astro cluster.
  </Step>

  <Step title="Add Amazon ECR repository URI">
    Replace `<your-docker-image>` in the instantiation of the `KubernetesPodOperator` with the Amazon ECR repository URI that hosts the Docker image. To locate the URI:

    * In the Amazon ECR Dashboard, click **Repositories** in the left menu.
    * Open the **Private** tab and then copy the URI of the repository that hosts the Docker image.
  </Step>
</Steps>


# Run images from Google Artifact Registry
Source: https://astronomer.io/docs/astro/kpo-google-artifact-registry

Configure KPO to pull images from Google Artifact Registry.

<Info>Passwordless setup is available only on Astro dedicated clusters. For Astro standard clusters, follow the steps in [Private Registry](/docs/astro/kpo-private-registry) to create a Kubernetes secret containing your registry credentials.</Info>

By default, the `KubernetesPodOperator` expects to pull container images that are hosted publicly. If your images are hosted on the container registry native to your cloud provider, you can grant access to the images directly.

## Prerequisites

* An [Astro project](/docs/cli/v1.43/get-started-cli).
* An [Astro Deployment](/docs/astro/deployment-settings).
* Access to your Google Artifact Registry repository.

## Setup

If your container image is hosted in Google Artifact Registry repository, add a permissions policy to the repository to allow the `KubernetesPodOperator` to pull the Docker image. You don't need to create a Kubernetes secret or specify the Kubernetes secret in your dag. Docker images hosted in Google Artifact Registry repositories can be pulled only to Deployments hosted on GCP clusters.

<Steps>
  <Step title="Request the Compute Engine default service account ID">
    Contact [Astronomer support](https://support.astronomer.io) to request the Compute Engine default service account ID for your cluster.
  </Step>

  <Step title="Add Google Artifact Registry repository permissions">
    1. Log in to Google Artifact Registry.
    2. Click the checkbox next to the repository that you want to use.
    3. In the **Properties** pane that appears, click **ADD PRINCIPAL** in the **PERMISSIONS** tab.
    4. In the **Add Principals** text box, paste the Compute Engine default service account ID that was provided to you by Astronomer Support.
    5. In the **Assign Roles** selector, search for `Artifact Registry Reader` and select the role that appears.
    6. Click **Save** to grant read access for the registry  to Astro.
  </Step>

  <Step title="Set up the KubernetesPodOperator">
    The following snippet is the minimum configuration you'll need to create a `KubernetesPodOperator` task on Astro:

    ```python wrap theme={null}
    from airflow.configuration import conf
    from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

    namespace = conf.get("kubernetes", "NAMESPACE")

    KubernetesPodOperator(
        namespace=namespace,
        image="<your-docker-image>",
        cmds=["<commands-for-image>"],
        arguments=["<arguments-for-image>"],
        labels={"<pod-label>": "<label-name>"},
        name="<pod-name>",
        task_id="<task-name>",
        get_logs=True,
        in_cluster=True,
    )
    ```

    For each instantiation of the `KubernetesPodOperator`, you must specify the following values:

    * `namespace = conf.get("kubernetes", "NAMESPACE")`: Every Deployment runs on its own Kubernetes namespace within a cluster. Information about this namespace can be programmatically imported as long as you set this variable.
    * `image`: This is the Docker image that the operator will use to run its defined task, commands, and arguments. Astro assumes that this value is an image tag that's publicly available on [Docker Hub](https://hub.docker.com/). To pull an image from a private registry, see [Pull images from a Private Registry](/docs/astro/kpo-private-registry).
    * `in_cluster`: If a Connection object is not passed to the `KubernetesPodOperator`'s `kubernetes_conn_id` parameter, specify `in_cluster=True` to run the task in the Deployment's Astro cluster.
  </Step>

  <Step title="Add the Google Artifact Registry URI">
    When you configure an instantiation of the `KubernetesPodOperator`, replace `<your-docker-image>` with the Google Artifact Registry image URI. To retrieve the URI:

    * In the Google Artifact Registry, click the registry containing the image.
    * Click the image you want to use.
    * Click the copy icon next to the image in the top corner. The string you copy should be in the format `<GCP Region>-docker.pkg.dev/<Project Name>/<Registry Name>/<Image Name>`.
  </Step>
</Steps>


# Mount temporary directory
Source: https://astronomer.io/docs/astro/kpo-mount-temporary-directory

Learn how to mount a temporary directory for the KubernetesPodOperator on Astro.

You can run a task run the `KubernetesPodOperator` that uses your [Deployment's ephemeral storage](/docs/astro/deployment-resources#configure-kubernetes-pod-resources), mount an [emptyDir volume](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir-configuration-example) to the `KubernetesPodOperator`.

## Prerequisites

* Set up the [`KubernetesPodOperator` on Astro](/docs/astro/kubernetespodoperator).

## Setup

<Steps>
  <Step title="Mount an emptyDir volume">
    Add the following code example to your `KubernetesPodOperator` configuration.

    ```python {5-14,26-27} wrap theme={null}
    from airflow.configuration import conf
    from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
    from kubernetes.client import models as k8s

    volume = k8s.V1Volume(
        name="cache-volume",
        emptyDir={},
    )

    volume_mounts = [
        k8s.V1VolumeMount(
            mount_path="/cache", name="cache-volume"
        )
    ]

    example_volume_test = KubernetesPodOperator(
        namespace=namespace,
        image="<your-docker-image>",
        cmds=["<commands-for-image>"],
        arguments=["<arguments-for-image>"],
        labels={"<pod-label>": "<label-name>"},
        name="<pod-name>",
        task_id="<task-name>",
        get_logs=True,
        in_cluster=True,
        volume_mounts=volume_mounts,
        volumes=[volume],
    )
    ```
  </Step>
</Steps>


# Run images from a private registry
Source: https://astronomer.io/docs/astro/kpo-private-registry

Configure KPO to pull privately hosted container images.

By default, the `KubernetesPodOperator` expects to pull container images that are hosted publicly. If your images are hosted on the container registry native to your cloud provider, you can grant access to the images directly. Otherwise, if you are using any other private registry, you need to create a Kubernetes Secret containing credentials to the registry, then specify the Kubernetes Secret in your dag.

## Prerequisites

* An [Astro project](/docs/cli/v1.43/get-started-cli).
* An [Astro Deployment](/docs/astro/deployment-settings).
* Access to a private Docker registry.

## Setup

<Steps>
  <Step title="Create a Kubernetes Secret">
    To run Docker images from a private registry on Astro, a Kubernetes Secret that contains credentials to your registry must be created. Injecting this secret into your Deployment's namespace will give your tasks access to Docker images within your private registry.

    By default, the `KubernetesPodOperator` looks for publicly hosted images. However, you can pull images from a private registry.

    1. Retrieve a `config.json` file that contains your Docker credentials by following the [Docker documentation](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials). The generated file looks similar to the following:

       ```json title="config.json" wrap theme={null}
       {
           "auths": {
               "https://index.docker.io/v1/": {
                   "auth": "c3R...zE2"
               }
           }
       }
       ```

    2. Submit a request to [Astronomer support](https://cloud.astronomer.io/open-support-request) for creating a Kubernetes Secret to enable pulling images from private registries. Astronomer Support can provide you the necessary instructions on how to generate and securely send the credentials.
  </Step>

  <Step title="Specify the Kubernetes Secret in your dag">
    1. Astronomer adds the Kubernetes secret to your Deployment, Astronomer notifies you and provides you with the name of the secret.

    2. After you receive the name of your Kubernetes secret from Astronomer, you can run images from your private registry by importing `models` from `kubernetes.client` and configuring `image_pull_secrets` in your `KubernetesPodOperator` instantiation:

    ```python {1,5} wrap theme={null}
    from kubernetes.client import models as k8s

    KubernetesPodOperator(
        namespace=namespace,
        image_pull_secrets=[k8s.V1LocalObjectReference("<your-secret-name>")],
        image="<your-docker-image>",
        cmds=["<commands-for-image>"],
        arguments=["<arguments-for-image>"],
        labels={"<pod-label>": "<label-name>"},
        name="<pod-name>",
        task_id="<task-name>",
        get_logs=True,
        in_cluster=True,
    )
    ```
  </Step>
</Steps>


# Run your Deployment's current Airflow image
Source: https://astronomer.io/docs/astro/kpo-run-current-image

Configure KPO to run your Deployment's current Airflow image.

## Run the current Airflow image

You can run your Deployment's current Airflow image with the `KubernetesPodOperator` by using the environment variable, `ASTRONOMER_AIRFLOW_IMAGE`. This environment variable allows you to run KPO tasks using the same Runtime image that your Deployment uses, in a way that won't be affected by underlying cluster infrastructure changes that might change the base image repository URL. The `ASTRONOMER_AIRFLOW_IMAGE` environment variable allows you to ensure that your Deployment retrieves the correct image URL.

## Setup

<Steps>
  <Step title="Add current Airflow image configuration">
    Add the `ASTRONOMER_AIRFLOW_IMAGE` environment variable to your KPO configuration. For example:

    ```python {10-14} wrap theme={null}
    import os
    from airflow.configuration import conf
    from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
    from kubernetes.client import models as k8s
    KubernetesPodOperator(
        namespace=conf.get("kubernetes", "NAMESPACE"),
        image=os.getenv("ASTRONOMER_AIRFLOW_IMAGE"),
        image_pull_secrets=[
            k8s.V1LocalObjectReference("image-pull-secret"),
            k8s.V1LocalObjectReference("per-dp-registry-pull-secret"),
        ],
        cmds=["<commands-for-image>"],
        arguments=["<arguments-for-image>"],
        labels={"<pod-label>": "<label-name>"},
        name="<pod-name>",
        task_id="<task-name>",
        get_logs=True,
        in_cluster=True,
    )
    ```
  </Step>
</Steps>


# Use secret environment variables
Source: https://astronomer.io/docs/astro/kpo-secret-env-var

Use secret environment variables in tasks running on the Kubernetes executor.

Astro [environment variables](/docs/astro/environment-variables) marked as secrets are stored in a Kubernetes secret called `env-secrets`. To use a secret value in a task running on the Kubernetes executor, you pull the value from `env-secrets` and mount it to the Pod running your task as a new Kubernetes Secret.

## Setup

<Steps>
  <Step title="Add import to your dag file">
    Add the `Secret` import to your dag file:

    ```python wrap theme={null}
    from airflow.kubernetes.secret import Secret
    ```
  </Step>

  <Step title="Define a Kubernetes secret">
    Define a Kubernetes `Secret` in your dag instantiation using the following format:

    ```python wrap theme={null}
    secret_env = Secret(deploy_type="env", deploy_target="<VARIABLE_KEY>", secret="env-secrets", key="<VARIABLE_KEY>")
    namespace = conf.get("kubernetes", "NAMESPACE")
    ```
  </Step>

  <Step title="Reference the environment variable key">
    Reference the key for the environment variable, formatted as `$VARIABLE_KEY` in the task using the `KubernetesPodOperator`.
  </Step>
</Steps>

## Example

In the following example, a secret named `MY_SECRET` is pulled from `env-secrets` and printed to logs.

```python wrap theme={null}
import pendulum
from airflow.kubernetes.secret import Secret

from airflow.models import DAG
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
from airflow.configuration import conf

with DAG(
        dag_id="test-kube-pod-secret",
        start_date=pendulum.datetime(2022, 1, 1, tz="UTC"),
        end_date=pendulum.datetime(2022, 1, 5, tz="UTC"),
        schedule="@once",
        catchup=False,
) as dag:

    secret_env = Secret(deploy_type="env", deploy_target="MY_SECRET", secret="env-secrets", key="MY_SECRET")

    namespace = conf.get("kubernetes", "NAMESPACE")

    k = KubernetesPodOperator(
        namespace=namespace,
        image="ubuntu:16.04",
        cmds=["bash", "-cx"],
        arguments=["echo $MY_SECRET && sleep 150"],
        name="test-name",
        task_id="test-task",
        get_logs=True,
        in_cluster=True,
        secrets=[secret_env],
    )
```


# Use the @task.kubernetes decorator
Source: https://astronomer.io/docs/astro/kpo-task-decorator

Learn how to run the task.kubernetes decorator on Astro.

The `@task.kubernetes` decorator provides a TaskFlow alternative to the traditional `KubernetesPodOperator`, which allows you to run a specified task in its own Kubernetes pod. Note that the Docker image provided to the `@task.kubernetes` decorator's  `image` parameter must support executing Python scripts in order to leverage the `KubernetesPodOperator` decorator.

Like regular `@task` decorated functions, XComs can be passed to the Python script running in the dedicated Kubernetes pod. If `do_xcom_push` is set to `True` in the decorator parameters, the value returned by the decorated function is pushed to XCom.

Astronomer recommends using the `@task.kubernetes` decorator instead of the `KubernetesPodOperator` when using XCom with Python scripts in a dedicated Kubernetes pod.

```python expandable wrap theme={null}
from pendulum import datetime
from airflow.configuration import conf
from airflow.decorators import dag, task
import random

# get the current Kubernetes namespace Airflow is running in
namespace = conf.get("kubernetes", "NAMESPACE")

@dag(
    start_date=datetime(2023, 1, 1),
    catchup=False,
    schedule="@daily",
)
def kubernetes_decorator_example_dag():
    @task
    def extract_data():
        # simulating querying from a database
        data_point = random.randint(0, 100)
        return data_point

    @task.kubernetes(
        # specify the Docker image to launch, it needs to be able to run a Python script
        image="python",
        # launch the Pod on the same cluster as Airflow is running on
        in_cluster=True,
        # launch the Pod in the same namespace as Airflow is running in
        namespace=namespace,
        # Pod configuration
        # naming the Pod
        name="my_pod",
        # log stdout of the container as task logs
        get_logs=True,
        # log events in case of Pod failure
        log_events_on_failure=True,
        # enable pushing to XCom
        do_xcom_push=True,
    )
    def transform(data_point):
        multiplied_data_point = 23 * int(data_point)
        return multiplied_data_point

    @task
    def load_data(**context):
        # pull the XCom value that has been pushed by the KubernetesPodOperator
        transformed_data_point = context["ti"].xcom_pull(
            task_ids="transform", key="return_value"
        )
        print(transformed_data_point)

    load_data(transform(extract_data()))


kubernetes_decorator_example_dag()
```


# Configure task-level Pod resources
Source: https://astronomer.io/docs/astro/kpo-task-level-resources

Learn how to configure task-level Pod resources for the KubernetesPodOperator on Astro.

Astro automatically allocates resources to Pods created by the `KubernetesPodOperator`. Unless otherwise specified in your task-level configuration, the amount of resources your task Pod can use is defined by your [default Pod resource configuration](/docs/astro/deployment-resources#configure-kubernetes-pod-resources). To optimize your resource usage, Astronomer recommends specifying [compute resource requests and limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for each task.

## Setup

<Steps>
  <Step title="Define container resources">
    Define a `kubernetes.client.models.V1ResourceRequirements` object and provide that to the `container_resources` argument of the `KubernetesPodOperator`. For example:

    The following code example ensures that when this dag runs, it launches a Kubernetes Pod with exactly 800m of CPU and 3Gi of memory as long as that infrastructure is available in your Deployment. After the task finishes, the Pod terminates gracefully.

    ```python {19} wrap theme={null}
    from airflow.configuration import conf
    from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
    from kubernetes.client import models as k8s

    compute_resources = k8s.V1ResourceRequirements(
        limits={"cpu": "800m", "memory": "3Gi"},
        requests={"cpu": "800m", "memory": "3Gi"}
    )

    namespace = conf.get("kubernetes", "NAMESPACE")

    KubernetesPodOperator(
        namespace=namespace,
        image="<your-docker-image>",
        cmds=["<commands-for-image>"],
        arguments=["<arguments-for-image>"],
        labels={"<pod-label>": "<label-name>"},
        name="<pod-name>",
        container_resources=compute_resources,
        task_id="<task-name>",
        get_logs=True,
        in_cluster=True,
    )
    ```
  </Step>
</Steps>

<Warning>
  When you add custom labels, do not remove or modify the default Airflow labels that Astro applies to KPO Pods. See [Known limitations](/docs/astro/kubernetespodoperator#known-limitations).
</Warning>

<Warning>
  On Astro Hosted, Astro automatically sets resource requests equal to limits for `KubernetesPodOperator` task Pods. This ensures Pods receive a Kubernetes [Guaranteed Quality of Service (QoS) class](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#guaranteed), which prevents resource contention and eviction. Because Astro uses the limit values as both requests and limits, your Pods are billed based on the limits you configure, even if actual usage is lower. To avoid unexpected charges, set limits close to the resources your task requires. Check your [Billing and usage](/docs/astro/manage-billing) to view your resource use and associated charges.
</Warning>


# Run the KubernetesPodOperator on Astro
Source: https://astronomer.io/docs/astro/kubernetespodoperator

Learn how to run the KubernetesPodOperator on Astro. This operator dynamically launches a Pod in Kubernetes for each task and terminates each Pod when the task is complete.

The [`KubernetesPodOperator`](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html) is one of the most customizable Apache Airflow operators. A task using the `KubernetesPodOperator` runs in a dedicated, isolated Kubernetes Pod that terminates after the task completes. To learn more about the benefits and usage of the `KubernetesPodOperator`, see the [`KubernetesPodOperator` Learn guide](/docs/learn/kubepod-operator).

On Astro, the infrastructure required to run the `KubernetesPodOperator` is built into every Deployment and is managed by Astronomer. Astro supports setting a default Pod configuration so that any task Pods without specific resource requests and limits cannot exceed your expected resource usage for the Deployment.

Some task-level configurations will differ on Astro compared to other Airflow environments. Use this document to learn how to configure individual task Pods for different use cases on Astro. To configure the default Pod resources for all `KubernetesPodOperator` Pods, see [Configure Kubernetes Pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources).

## Known limitations

<Tip>By default, Astro supports a maximum `KubernetesPodOperator` Pod size of 43 vCPU and 86 GiB of memory. Astro can support Pod sizes up to 86 vCPU and 172 GiB of memory on request. Contact [Astro support](/docs/astro/astro-support) if you need to run larger jobs on `KubernetesPodOperator`.</Tip>

* Cross-account service accounts are not supported on Pods launched in an Astro cluster. To allow access to external data sources, you can provide credentials and secrets to tasks.
* Astro uses the following default Airflow labels on KPO Pods for internal health management:

  * `kubernetes_pod_operator`
  * `dag_id`
  * `task_id`
  * `run_id`
  * `map_index`
  * `try_number`

  Do not remove or modify these labels. Changing them prevents Astro from reconciling Pods with running task instances, which can cause the KPO healer to incorrectly handle your Pods. You can add your own custom labels to KPO Pods without affecting this behavior. See [Configure task-level Pod resources](/docs/astro/kpo-task-level-resources) for an example.
* PersistentVolumes (PVs) are not supported on Pods launched in an Astro cluster.
* You can't use an image built for an ARM architecture in the `KubernetesPodOperator`. To build images using the x86 architecture on a Mac with an Apple chip, include the `--platform` flag in the `FROM` command of the `Dockerfile` that constructs your custom image. For example:

  ```dockerfile wrap theme={null}
  FROM --platform=linux/amd64 postgres:latest
  ```

  If you use an ARM image, your KPO task will fail with the error: `base] exec /usr/bin/psql: exec format error`.

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project).
* An Astro [Deployment](/docs/astro/create-deployment).

## Set up the `KubernetesPodOperator` on Astro

The following snippet is the minimum configuration you'll need to create a `KubernetesPodOperator` task on Astro:

```python wrap theme={null}
from airflow.configuration import conf
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

namespace = conf.get("kubernetes", "NAMESPACE")

KubernetesPodOperator(
    namespace=namespace,
    image="<your-docker-image>",
    cmds=["<commands-for-image>"],
    arguments=["<arguments-for-image>"],
    labels={"<pod-label>": "<label-name>"},
    name="<pod-name>",
    task_id="<task-name>",
    get_logs=True,
    in_cluster=True,
)
```

For each instantiation of the `KubernetesPodOperator`, you must specify the following values:

* `namespace = conf.get("kubernetes", "NAMESPACE")`: Every Deployment runs on its own Kubernetes namespace within a cluster. Information about this namespace can be programmatically imported as long as you set this variable.
* `image`: This is the Docker image that the operator will use to run its defined task, commands, and arguments. Astro assumes that this value is an image tag that's publicly available on [Docker Hub](https://hub.docker.com/). To pull an image from a private registry, see [Pull images from a Private Registry](/docs/astro/kpo-private-registry).
* `in_cluster`: If a Connection object is not passed to the `KubernetesPodOperator`'s `kubernetes_conn_id` parameter, specify `in_cluster=True` to run the task in the Deployment's Astro cluster.

## Related documentation

* [How to use cluster ConfigMaps, Secrets, and Volumes with Pods](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html#how-to-use-cluster-configmaps-secrets-and-volumes-with-pod)
* [`KubernetesPodOperator` Airflow Guide](/docs/learn/kubepod-operator)


# Launch a Pod in an EKS cluster on AWS
Source: https://astronomer.io/docs/astro/launch-pod-external-cluster-aws

Use the KubernetesPodOperator to launch Pods in an external EKS cluster on AWS from your Airflow instance.

If some of your tasks require specific resources such as a GPU, you might want to run them in a different cluster than your Airflow instance. In setups where both clusters are used by the same AWS, Azure, or GCP account, you can manage separate clusters with roles and permissions.

<Info>To launch Pods in external clusters from a local Airflow environment, you must have valid authentication for the external cluster so that your local Airflow environment has permissions to launch a Pod in the external cluster. For managed Kubernetes services from public cloud providers, authentication is federated through the native IAM service. To grant the Astro role permissions to launch pods on your cluster, you can either include static credentials or use workload identity to authorize the Astro role to your cluster.</Info>

This example shows how to set up an EKS cluster on AWS and run a Pod on it from an Airflow instance where cross-account access is not available.

## Prerequisites

* Network connectivity between your Airflow execution environment and the external Kubernetes cluster:
  * **Hosted execution mode**: A [network connection](/docs/astro/networking-overview) between your Astro Deployment and the external cluster.
  * **Remote execution mode**: Network connectivity between the environment where your [Remote Execution Agent](/docs/astro/remote-execution-overview) runs and the external cluster. You are responsible for managing this connectivity. A direct network connection between Astro and the external cluster is not required.

## Setup

<Steps>
  <Step title="Set up your external cluster">
    1. [Create an EKS cluster IAM role](https://docs.aws.amazon.com/eks/latest/userguide/service_IAM_role.html#create-service-role) with a unique name and add the following permission policies:

       * `AmazonEKSWorkerNodePolicy`
       * `AmazonEKS_CNI_Policy`
       * `AmazonEC2ContainerRegistryReadOnly`

       Record the ARN of the new role, as it will be needed below.

    2. [Update the trust policy](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/edit_trust.html) of this new role to include the [workload identity](/docs/astro/authorize-deployments-to-your-cloud) of your Deployment. This step ensures that the role can be assumed by your Deployment.

       ```json wrap theme={null}
       {
       "Version": "2012-10-17",
       "Statement": [
           {
           "Effect": "Allow",
           "Principal": {
               "AWS": "arn:aws:iam::<aws account id>:<your user>",
               "Service": [
                   "ec2.amazonaws.com",
                   "eks.amazonaws.com"
               ]
           },
           "Action": "sts:AssumeRole"
           }
       ]
       }
       ```

    3. If you don't already have a cluster, [create a new EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/create-cluster.html) and assign the new role to it.
  </Step>

  <Step title="Retrieve the KubeConfig file from the EKS cluster">
    1. Use a `KubeConfig` file to remotely connect to your new cluster. On AWS, you can run the following command to retrieve it:

       ```sh wrap theme={null}
       aws eks --region <your-region> update-kubeconfig --name <cluster-name>  --kubeconfig my_kubeconfig.yaml
       ```

       This command creates a new `KubeConfig` file called `my_kubeconfig.yaml`.

    2. Ensure that the file below matches your generated KubeConfig. The newly generated KubeConfig must be edited to instruct the [AWS IAM Authenticator for Kubernetes](https://github.com/kubernetes-sigs/aws-iam-authenticator) to assume your new IAM Role created in Step 1.  Replace `<your assume role arn>` with the IAM Role ARN from Step 1.

       ```yaml title="my_kubeconfig.yaml" expandable wrap theme={null}
       apiVersion: v1
       clusters:
       - cluster:
           certificate-authority-data: <base 64 public certificate>
           server: <Kubernetes API Endpoint>
         name: <arn of your cluster>
       contexts:
       - context:
           cluster: <arn of your cluster>
           user: <arn of your cluster>
         name: <arn of your cluster>
       current-context: <arn of your cluster>
       kind: Config
       preferences: {}
       users:
       - name: <arn of your cluster>
         user:
           exec:
             apiVersion: client.authentication.k8s.io/v1alpha1
             args:
             - --region
             - <your cluster's AWS region>
             - eks
             - get-token
             - --cluster-name
             - <name of your cluster>
             - --role
             - <your assume role arn>
             command: aws
             interactiveMode: IfAvailable
             provideClusterInfo: false
       ```
  </Step>

  <Step title="Create a Kubernetes cluster connection">
    Astronomer recommends creating a Kubernetes cluster connection because it's more secure than adding an unencrypted `kubeconfig` file directly to your Astro project.

    1. Convert the `kubeconfig` configuration you retrieved from your cluster to JSON format.
    2. In either the Airflow UI or the Astro environment manager, create a new **Kubernetes Cluster Connection** connection. In the **Kube config (JSON format)** field, paste the `kubeconfig` configuration you retrieved from your cluster after converting it from `yaml` to `json` format.
    3. Click **Save**.

    You can now specify this connection in the configuration of any `KubernetesPodOperator` task that needs to access your external cluster.
  </Step>

  <Step title="Install the AWS CLI in your Astro environment">
    To connect to your external EKS cluster, you need to install the AWS CLI in your Astro project.

    1. Add the following to your `Dockerfile` to install the AWS CLI:

       ```dockerfile title="Dockerfile" wrap theme={null}
       USER root

       RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
       # Note: if you are testing your pipeline locally you may need to adjust the zip version to your dev local environment
       RUN unzip awscliv2.zip
       RUN ./aws/install

       USER astro
       ```

    2. Add the `unzip` package to your `packages.txt` file to make the `unzip` command available in your Docker container:

       ```text title="packages.txt" wrap theme={null}
       unzip
       ```

    If you are working locally, you need to restart your Astro project to apply the changes.
  </Step>

  <Step title="Configure your task">
    In your `KubernetesPodOperator` task configuration, ensure that you set `cluster-context` and `namespace` for your remote cluster. In the following example, the task launches a Pod in an external cluster based on the configuration defined in the `k8s` connection.

    ```python wrap theme={null}
    run_on_EKS = KubernetesPodOperator(
        task_id="run_on_EKS",
        kubernetes_conn_id="k8s",
        cluster_context="<your-cluster-id>",
        namespace="<your-namespace>",
        name="example_pod",
        image="ubuntu",
        cmds=["bash", "-cx"],
        arguments=["echo hello"],
        get_logs=True,
        startup_timeout_seconds=240,
    )
    ```
  </Step>
</Steps>

## Example dag

The following dag uses several classes from the [Amazon provider package](https://airflow.apache.org/registry/providers/amazon/) to dynamically spin up and delete Pods for each task in a newly created node group. If your remote Kubernetes cluster already has a node group available, you only need to define your task in the `KubernetesPodOperator` itself.

The example dag contains 5 consecutive tasks:

* Create a node group according to the user's specifications (For the example that uses GPU resources).
* Use a sensor to check that the cluster is running correctly.
* Use the `KubernetesPodOperator` to run any valid Docker image in a Pod on the newly created node group on the remote cluster. The example dag uses the standard `Ubuntu` image to print "hello" to the console using a `bash` command.
* Delete the node group.
* Verify that the node group has been deleted.

```python expandable wrap theme={null}
# import DAG object and utility packages
from airflow import DAG
from pendulum import datetime
from airflow.configuration import conf

# import the KubernetesPodOperator
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
    KubernetesPodOperator,
)

# import EKS related packages from the Amazon Provider
from airflow.providers.amazon.aws.hooks.eks import EksHook, NodegroupStates
from airflow.providers.amazon.aws.operators.eks import (
    EksCreateNodegroupOperator,
    EksDeleteNodegroupOperator,
)
from airflow.providers.amazon.aws.sensors.eks import EksNodegroupStateSensor

# custom class to create a node group with Nodes on EKS
class EksCreateNodegroupWithNodesOperator(EksCreateNodegroupOperator):
    def execute(self, context):
        # instantiating an EKSHook on the basis of the AWS connection (Step 5)
        eks_hook = EksHook(
            aws_conn_id=self.aws_conn_id,
            region_name=self.region,
        )

        # define the Node group to create
        eks_hook.create_nodegroup(
            clusterName=self.cluster_name,
            nodegroupName=self.nodegroup_name,
            subnets=self.nodegroup_subnets,
            nodeRole=self.nodegroup_role_arn,
            scalingConfig={"minSize": 1, "maxSize": 1, "desiredSize": 1},
            diskSize=20,
            instanceTypes=["g4dn.xlarge"],
            amiType="AL2_x86_64_GPU",  # get GPU resources
            updateConfig={"maxUnavailable": 1},
        )


# instantiate the DAG
with DAG(
    start_date=datetime(2022, 6, 1),
    catchup=False,
    schedule="@daily",
    dag_id="KPO_remote_EKS_cluster_example_dag",
) as dag:
    # task 1 creates the node group
    create_gpu_nodegroup = EksCreateNodegroupWithNodesOperator(
        task_id="create_gpu_nodegroup",
        cluster_name="<your cluster name>",
        nodegroup_name="gpu-nodes",
        nodegroup_subnets=["<your subnet>", "<your subnet>"],
        nodegroup_role_arn="<arn of your EKS role>",
        aws_conn_id="<your aws conn id>",
        region="<your region>",
    )

    # task 2 check for node group status, if it is up and running
    check_nodegroup_status = EKSNodegroupStateSensor(
        task_id="check_nodegroup_status",
        cluster_name="<your cluster name>",
        nodegroup_name="gpu-nodes",
        mode="reschedule",
        timeout=60 * 30,
        exponential_backoff=True,
        aws_conn_id="<your aws conn id>",
        region="<your region>",
    )

    # task 3 the KubernetesPodOperator running a task
    # here, cluster_context and the kubernetes_conn_id are defined at the task level.
    run_on_EKS = KubernetesPodOperator(
        task_id="run_on_EKS",
        cluster_context="<arn of your cluster>",
        namespace="airflow-kpo-default",
        name="example_pod",
        image="ubuntu",
        cmds=["bash", "-cx"],
        arguments=["echo hello"],
        get_logs=True,
        in_cluster=False,
        kubernetes_conn_id="k8s",
        startup_timeout_seconds=240,
    )

    # task 4 deleting the node group
    delete_gpu_nodegroup = EksDeleteNodegroupOperator(
        task_id="delete_gpu_nodegroup",
        cluster_name="<your cluster name>",
        nodegroup_name="gpu-nodes",
        aws_conn_id="<your aws conn id>",
        region="<your region>",
    )

    # task 5 checking that the node group was deleted successfully
    check_nodegroup_termination = EksNodegroupStateSensor(
        task_id="check_nodegroup_termination",
        cluster_name="<your cluster name>",
        nodegroup_name="gpu-nodes",
        aws_conn_id="<your aws conn id>",
        region="<your region>",
        mode="reschedule",
        timeout=60 * 30,
        target_state=NodegroupStates.NONEXISTENT,
    )

    # setting the dependencies
    create_gpu_nodegroup >> check_nodegroup_status >> run_on_EKS
    run_on_EKS >> delete_gpu_nodegroup >> check_nodegroup_termination
```


# Launch a Pod in an AKS cluster on Azure
Source: https://astronomer.io/docs/astro/launch-pod-external-cluster-azure

Use the KubernetesPodOperator to launch Pods in an external AKS cluster on Azure from your Airflow instance.

If some of your tasks require specific resources such as a GPU, you might want to run them in a different cluster than your Airflow instance. In setups where both clusters are used by the same AWS, Azure or GCP account, you can manage separate clusters with roles and permissions.

<Info>To launch Pods in external clusters from a local Airflow environment, you must have valid authentication for the external cluster so that your local Airflow environment has permissions to launch a Pod in the external cluster. For managed Kubernetes services from public cloud providers, authentication is federated through the native IAM service. To grant the Astro role permissions to launch pods on your cluster, you can either include static credentials or use workload identity to authorize the Astro role to your cluster.</Info>

This example shows how to configure an Azure Managed Identity (MI) to run a Pod on an AKS cluster from an Airflow instance where cross-account access is not available.

## Prerequisites

* Network connectivity between your Airflow execution environment and the external Kubernetes cluster:
  * **Hosted execution mode**: A [network connection](/docs/astro/networking-overview) between your Astro Deployment and the external cluster.
  * **Remote execution mode**: Network connectivity between the environment where your [Remote Execution Agent](/docs/astro/remote-execution-overview) runs and the external cluster. You are responsible for managing this connectivity. A direct network connection between Astro and the external cluster is not required.

## Setup

<Steps>
  <Step title="Set Up Azure Managed Identity">
    1. Create a [Microsoft Entra ID tenant](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-create-new-tenant) with Global Administrator or Application Administrator privileges.
    2. Create a [user-assigned managed identity on Azure](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?source=recommendations\&pivots=identity-mi-methods-azp#create-a-user-assigned-managed-identity).
    3. Authorize your Astro Deployment to Azure using Azure Managed Identity (MI) by following steps 1 and 2 described in the [Deployment Workload identity](/docs/astro/authorize-deployments-to-your-cloud?tab=azure#setup) set up.
    4. Confirm that the OIDC credentials appear in the Managed Identity's Federated credentials tab.
    5. From the Managed Identity's Properties tab, note the **Client ID**.
       From your Azure Portal, go to Azure Active Directory (Microsoft Entra ID) and note the **Tenant ID**.
       Both the **Client ID** and **Tenant ID** will be needed in Step 3 to configure your `kubeconfig` file.
  </Step>

  <Step title="Install dependencies in your Astro Runtime Docker Image">
    To trigger remote Pods on an Azure AKS Cluster, the following packages and dependencies need to be added to your Docker image.

    * Azure CLI
    * Kubectl
    * Kubelogin

    To do so, add the following commands to your Dockerfile:

    ```dockerfile title="Dockerfile" wrap theme={null}
    FROM quay.io/astronomer/astro-runtimeX.Y.Z

    USER root
    RUN curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
    RUN az aks install-cli

    USER astro
    ```
  </Step>

  <Step title="Configure your `kubeconfig` file">
    The following configuration file below is a sample Kubernetes `kubeconfig` file that allows the Kubernetes command-line tool, `kubectl`, or other clients to connect to a remote Kubernetes cluster, `remote-kpo`, using Azure Workload Identity for authentication.

    ```yaml title="kubeconfig" expandable wrap theme={null}
    # Specifies the version of the Kubernetes API for this configuration file.
    # v1 is the standard version used for kubeconfig files.
    apiVersion: v1
    # List of Kubernetes clusters that the configuration can connect to.
    clusters:
    - cluster:
        # base64-encoded certificate for the Kubernetes API server to verify SSL communication.
        certificate-authority-data: <certificate>
        # URL of the Kubernetes API server.
        # This is the endpoint of the remote cluster you want to interact with.
        server: <Azure server address>
        # Name of the cluster, which is referenced in the contexts section.
      name: <AKS cluster>
    # List of contexts that define which cluster and user combination to use when interacting with Kubernetes.
    contexts:
    # Describes the context for connecting to the cluster.
    - context:
        # References the cluster from the clusters section.
        cluster: <AKS cluster>
        # Associates the user configuration to be used for authentication with the cluster.
        user: <user>
      # The name of the context, which is referenced by current-context.
      name: <AKS cluster>
    # Specifies the active context that will be used by default when running kubectl commands.
    current-context: <AKS cluster>
    # Identifies the file type as a Kubernetes Config.
    kind: Config
    preferences: {}
    # List of users and the method they use for authentication.
    users:
    # Defines the user that is being used in the context.
    # This user is responsible for authenticating with the Kubernetes cluster.
    - name: <your user>
      user:
        exec:
          apiVersion: client.authentication.k8s.io/v1beta1
          args:
          - get-token
          - --login
          - workloadidentity
          - --tenant-id
          - <Tenant ID from Step 1>
          - --client-id
          - <Client ID from Step 1>
          # The server ID for Azure Kubernetes Service (AKS). This is a static ID representing AKS.
          - --server-id
          - 6dae42f8-4368-4678-94ff-3960e28e3630
          # Specifies the path to the federated token that the managed identity uses to authenticate.
          - --federated-token-file
          - /var/run/secrets/azure/tokens/azure-identity-token
          - --environment
          - AzurePublicCloud
          command: kubelogin
          # Specifies if the kubelogin command should not attempt to provide additional cluster information beyond the authentication token.
          provideClusterInfo: false
    ```
  </Step>

  <Step title="Create an Airflow Connection to use the `kubeconfig` file">
    To use the `kubeconfig` file, you will need to create a new Kubernetes Airflow Connection.

    There are multiple ways to pass the `kubeconfig` file to your Airflow Connection. If your `kubeconfig` file contains any sensitive information, we recommend storing it as JSON inside the connection, described in option 3.

    1. **External File in the default location**
       If the `kubeconfig` file resides in the default location on the machine (\~/.kube/config), you can leave all fields empty in the connection configuration. Airflow will automatically use the `kubeconfig` from the default location.
       Add the following `COPY` command at the end of your Dockerfile to add your `kubeconfig` file inside your Astro Runtime Docker Image.

    ```dockerfile wrap theme={null}
    COPY kubeconfig ~/.kube/config/airflow/kubeconfig
    ```

    2. **External file with a Custom Path**:
       You can specify a custom path to the `kubeconfig` file by inserting the path into the ***Kube config path*** field of your Airflow Connection.
       Add the following `COPY` command at the end of your Dockerfile to add your `kubeconfig` file inside your Astro Runtime Docker Image.

    ```dockerfile wrap theme={null}
    COPY kubeconfig /usr/local/airflow/kubeconfig
    ```

    3. **JSON Format**
       You can convert the `kubeconfig` file to JSON format and paste it into the Kube config (JSON format) field in the connection configuration.
       Use an online converter like [https://jsonformatter.org/yaml-to-json](https://jsonformatter.org/yaml-to-json) to convert YAML to JSON. Remove any sensitive information first.
  </Step>

  <Step title="Configure your task">
    Run a Kubernetes Pod with Airflow `KubernetesPodOperator`.

    ```python expandable wrap theme={null}
    from datetime import datetime

    # import the DAG object
    from airflow.models import DAG

    # import the KubernetesPodOperator
    from airflow.providers.cncf.kubernetes.operators.pod import (
        KubernetesPodOperator,
    )

    default_args = {
        "owner": "Astronomer",
        "depends_on_past": False,
    }

    # instantiate the dag
    with DAG(
        dag_id="remote_kpo",
        default_args=default_args,
        schedule=None,
        start_date=datetime(2024, 1, 1),
        tags=["KPO"],
    ):
        # launch a pod in the Kubernetes cluster
        remote_kpo = KubernetesPodOperator(
            task_id="az_remote_kpo",
            kubernetes_conn_id="<my-az-connection>",
            namespace="<my-aks-namespace>",
            image="debian",
            cmds=["bash", "-cx"],
            arguments=["echo", "hello world!"],
            name="hello-world",
            get_logs=True,
            in_cluster=False,
        )
    ```
  </Step>
</Steps>


# Launch a Pod in a GKE cluster on GCP
Source: https://astronomer.io/docs/astro/launch-pod-external-cluster-gcp

Use the KubernetesPodOperator to launch Pods in an external GKE cluster on Google Cloud from your Airflow instance.

If some of your tasks require specific resources such as a GPU, you might want to run them in a different cluster than your Airflow instance. In setups where both clusters belong to the same Google Cloud project, you can manage separate clusters with roles and permissions.

This document shows how to configure a Google Kubernetes Engine (GKE) cluster on Google Cloud and run a Pod on it from an Airflow instance where cross-project access isn't available.

<Info>
  To launch Pods in external clusters from a local Airflow environment, you must have valid authentication for the external cluster. For managed Kubernetes services from public cloud providers, authentication is federated through the native IAM service. To grant the Astro role permissions to launch Pods on your cluster, you can either include static credentials or use workload identity to authorize the Astro role to your cluster.
</Info>

## Prerequisites

* Network connectivity between your Airflow execution environment and the external Kubernetes cluster:
  * **Hosted execution mode**: A [network connection](/docs/astro/networking-overview) between your Astro Deployment and the external cluster.
  * **Remote execution mode**: Network connectivity between the environment where your [Remote Execution Agent](/docs/astro/remote-execution-overview) runs and the external cluster. You are responsible for managing this connectivity. A direct network connection between Astro and the external cluster isn't required.

## Setup

<Steps>
  <Step title="Set up your external GKE cluster">
    Follow Google Cloud's documentation to prepare a GKE cluster that your Astro Deployment can authenticate to:

    1. [Create a GKE cluster](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster) if you don't already have one.
    2. Authorize your Astro Deployment to Google Cloud by following the [Deployment workload identity setup](/docs/astro/authorize-deployments-to-your-cloud?tab=gcp).
    3. Grant the service account [IAM and Kubernetes RBAC permissions](https://cloud.google.com/kubernetes-engine/docs/how-to/role-based-access-control) in the namespace where your `KubernetesPodOperator` tasks run.

       At a minimum, provision the following permissions for your service account in your specified namespace:

       * `container.clusters.get`
       * `container.events.list`
       * `container.pods.get`
       * `container.pods.getLogs`
       * `container.pods.list`
       * `container.pods.create`
       * `container.pods.delete`
       * `container.pods.update`

       If your Dag uses `do_xcom_push=True`, also grant the `container.pods.exec` permission.
  </Step>

  <Step title="Install dependencies in your Astro Runtime Docker image">
    To connect to your external GKE cluster, the [`gcloud` CLI](https://cloud.google.com/sdk/docs/install) and the [`gke-gcloud-auth-plugin`](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin) must be available inside your Astro Runtime image.

    Add the following to your `Dockerfile`:

    ```dockerfile title="Dockerfile" wrap theme={null}
    USER root

    RUN apt-get update && apt-get install -y apt-transport-https ca-certificates gnupg curl \
        && curl -sL https://packages.cloud.google.com/apt/doc/apt-key.gpg \
            | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg \
        && echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" \
            > /etc/apt/sources.list.d/google-cloud-sdk.list \
        && apt-get update \
        && apt-get install -y google-cloud-cli google-cloud-cli-gke-gcloud-auth-plugin \
        && rm -rf /var/lib/apt/lists/*

    USER astro
    ```

    For production deployments, consider pinning the `google-cloud-cli-gke-gcloud-auth-plugin` version for build reproducibility, or using a multi-stage build with the `google/cloud-sdk:slim` image to copy only the plugin binary into your final image and reduce its size.

    Add the following line to your `requirements.txt` to include the CNCF Kubernetes provider:

    ```text title="requirements.txt" wrap theme={null}
    apache-airflow-providers-cncf-kubernetes
    ```
  </Step>

  <Step title="Configure your `kubeconfig` file">
    The following sample Kubernetes `kubeconfig` file allows the Kubernetes command-line tool, `kubectl`, or other clients to connect to a remote Kubernetes cluster using Google Cloud workload identity for authentication.

    ```yaml title="kubeconfig" expandable wrap theme={null}
    # Specifies the version of the Kubernetes API for this configuration file.
    # v1 is the standard version used for kubeconfig files.
    apiVersion: v1
    # List of Kubernetes clusters that the configuration can connect to.
    clusters:
    - cluster:
        # base64-encoded certificate for the Kubernetes API server to verify SSL communication.
        certificate-authority-data: <base64-public-certificate>
        # Endpoint of the remote cluster you want to interact with.
        server: https://<cluster-endpoint>
      # Name of the cluster, which is referenced in the contexts section.
      name: <GKE cluster>
    # List of contexts that define which cluster and user combination to use when interacting with Kubernetes.
    contexts:
    # Describes the context for connecting to the cluster.
    - context:
        # References the cluster from the clusters section.
        cluster: <GKE cluster>
        # Associates the user configuration to be used for authentication with the cluster.
        user: <user>
      # The name of the context, which is referenced by current-context.
      name: <GKE cluster>
    # Specifies the active context that will be used by default when running kubectl commands.
    current-context: <GKE cluster>
    # Identifies the file type as a Kubernetes Config.
    kind: Config
    preferences: {}
    # List of users and the method they use for authentication.
    users:
    # Defines the user that is being used in the context.
    # This user is responsible for authenticating with the Kubernetes cluster.
    - name: <user>
      user:
        exec:
          apiVersion: client.authentication.k8s.io/v1beta1
          command: gke-gcloud-auth-plugin
          provideClusterInfo: true
    ```

    Fetch the `certificate-authority-data` and `cluster-endpoint` fields from the GKE cluster details page or using the Google Cloud SDK.

    <Note>
      **Scaling considerations**

      If you run a high number of concurrently deferred tasks against this connection, consider a static bearer token instead of the exec-based plugin shown in the preceding `kubeconfig`. At high triggerer concurrency, the plugin's subprocess invocation on each credential refresh can block the triggerer's shared event loop.
    </Note>
  </Step>

  <Step title="Create an Airflow connection to use the `kubeconfig` file">
    To use the `kubeconfig` file, create a new Kubernetes Airflow connection.

    There are multiple ways to pass the `kubeconfig` file to your Airflow connection. If your `kubeconfig` file contains any sensitive information, Astronomer recommends storing it as JSON inside the connection, as described in the JSON format tab.

    <Tabs>
      <Tab title="JSON format (Recommended)">
        Convert the `kubeconfig` file to JSON format and paste it into the **Kube config (JSON format)** field in the connection configuration.
      </Tab>

      <Tab title="External file in the default location">
        If the `kubeconfig` file resides in the default location on the machine (`~/.kube/config`), you can leave all fields empty in the connection configuration. Airflow automatically uses the `kubeconfig` from the default location. Add the following `COPY` command at the end of your Dockerfile to add your `kubeconfig` file inside your Astro Runtime Docker image:

        ```dockerfile wrap theme={null}
        COPY kubeconfig ~/.kube/config/airflow/kubeconfig
        ```
      </Tab>

      <Tab title="External file with a custom path">
        You can specify a custom path to the `kubeconfig` file by inserting the path into the **Kube config path** field of your Airflow connection. Add the following `COPY` command at the end of your Dockerfile to add your `kubeconfig` file inside your Astro Runtime Docker image:

        ```dockerfile wrap theme={null}
        COPY kubeconfig /usr/local/airflow/kubeconfig
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Configure your task">
    In your `KubernetesPodOperator` task, set `kubernetes_conn_id` to the connection you created, `namespace` to the namespace in your GKE cluster where the Pod should run, and `in_cluster=False` so that the operator uses the connection's `kubeconfig` instead of looking for an in-cluster service account.

    ```python wrap theme={null}
    from airflow.decorators import dag
    from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
    from pendulum import datetime


    @dag(
        dag_id="remote_kpo",
        start_date=datetime(2024, 1, 1),
        schedule=None,
        tags=["kubernetes", "gke"],
    )
    def remote_kpo():
        KubernetesPodOperator(
            task_id="run_on_gke",
            kubernetes_conn_id="<my-gke-connection>",
            namespace="<my-namespace>",
            image="ubuntu:latest",
            cmds=["echo"],
            arguments=["External KPO is working!"],
            name="example-pod",
            get_logs=True,
            in_cluster=False,
        )


    remote_kpo()
    ```
  </Step>
</Steps>


# Log in to Astro
Source: https://astronomer.io/docs/astro/log-in-to-astro

Learn how you can use the Cloud user interface (UI) and the Astro command-line interface (CLI) to log in to Astro.

You can use the Astro UI and the Astronomer CLI to view and modify your Workspaces, Deployments, environment variables, tasks, and users. You need to authenticate your user credentials when you're using the Astro UI or the Astro CLI for development on Astro. See the Astronomer CLI documentation for more details, including available commands and flags.

## Prerequisites

* An Astronomer account.
* The [Astro CLI](/docs/cli/v1.43/overview).
* An email address with a domain that matches the domain configured for your Organization.

## Log in to the Astro UI

Go to `https://cloud.astronomer.io`, and select one of the following options to access the Astro UI:

* To authenticate with single sign-on (SSO), enter your email and click **Continue**. If your Organization has SSO enabled, you'll be redirected to your IdP authentication screen.
* To authenticate with your GitHub account, click **Continue with GitHub**, enter your username or email address, enter your password, and then click **Sign in**. If your Organization selects this log in option, you’ll receive an email invitation from your Organization Owner. You can't access the Organization without an invitation.
* To authenticate with your Google account, click **Continue with Google**, choose an account, enter your username and password, and then click **Sign In**. If your Organization selects this log in option, you’ll receive an email invitation from your Organization Owner. You can't access the Organization without an invitation.

## Log in to the Astro CLI

Developing locally with the Astro CLI does not require an Astro account. This includes commands such as `astro dev start` and `astro dev pytest`. If you want to use functionality specific to Astro, including managing users and [deploying code](/docs/astro/deploy-code), you must first log in to Astro with the Astro CLI.

Astronomer uses refresh tokens to make sure that you don’t need to log in to the Astro CLI every time you run a command.

1. In the Astro CLI, run the following command:

   ```sh wrap theme={null}
   astro login
   ```

2. Enter your email address and press **Enter**.

3. Press **Enter** to connect your account to Astronomer.

   If this is your first time logging in, the Astronomer Authorize App dialog appears. Click **Accept** to allow Astronomer to access your profile and email and allow offline access.

4. Select one of the following options to access the Astro UI:

   * Enter your email and click **Continue**.
   * To authenticate with your GitHub account, click **Continue with GitHub**, enter your username or email address, enter your password, and then click **Sign in**.
   * To authenticate with your Google account, click **Continue with Google**, choose an account, enter your username and password, and then click **Sign In**.

   Confirmation messages appear in the Astro UI and in the Astro CLI indicating that your login was successful and that your computer is now connected. The name of your default Workspace in the Astro CLI also appears. To switch Workspace contexts after you log in, run [astro workspace switch](/docs/cli/v1.43/astro-workspace-switch).

## Log in from a browserless system

The following options are available if you're unable to use a browser for authentication:

* Run `astro login -t` to log in with an authentication token. To obtain an authentication token on a separate machine, go to `https://cloud.astronomer.io/token`.
* Run `astro login -l` to retrieve an Astro UI log in URL and then copy the URL. In a separate terminal session, run `curl -u <user-email>:<password> <returned-url>`. This option doesn't work if you're using an identity provider (IdP) for account authentication.

## Access a different base domain

When you need to access Astro and Astro Private Cloud with the Astro CLI at the same time, you need to authenticate to each product individually by specifying a base domain for each Astronomer installation.

A base domain or URL is the static element of a website address. For example, when you visit the Astronomer website, the address bar always displays `https://www.astronomer.io` no matter what page you access on the Astronomer website.

For Astro users, the base domain is `cloud.astronomer.io`. For Astro Private Cloud, every cluster has a base domain that you must authenticate to in order to access it. If your organization has multiple clusters, you can run Astro CLI commands to quickly move from one base domain to another. This can be useful when you need to move from an Astro Private Cloud installation to Astro and are using the Astro CLI to perform actions on both accounts.

1. Run the following command to view a list of Astronomer base domains that you can access. Your current base domain is highlighted.

   ```sh wrap theme={null}
   astro context list
   ```

2. In the Astro CLI, run the following command to re-authenticate to the target base domain:

   ```sh wrap theme={null}
   astro login <basedomain>
   ```

3. Run the following command to switch to a different base domain:

   ```sh wrap theme={null}
   astro context switch <basedomain>
   ```

## Switch Organizations

You can belong to more than one Astro Organization. Having a role in an Organization does not guarantee access to the Organization through the Astro UI. To access another Organization, you need to be able to authenticate with one of the enabled authentication methods.

1. Log in to the Astro UI. By default, the Astro UI opens the first Organization that you joined.
2. Open your profile menu, then click **Switch Organization** (in the legacy UI, click the name of your current Organization in the top navigation bar, then click **Switch Organization**).
3. Select the Organization that you want to switch to.

## Reset your Astro password

If you log in to Astro with a text password, you can reset your password from the Astro UI login screen.

1. Go to `cloud.astronomer.io`. If you are logged in, log out.

2. On the login screen, enter your email address, then click **Continue**.

3. Click **Forgot password?**

   <Frame>
     <img alt="Button to reset password on the Astro UI login page" />
   </Frame>

4. Confirm your user email address is correct, then click **Continue**. Astronomer sends an email with a password reset option to the user email address.

5. Follow the instructions in the email sent by Astronomer to reset your password.


# Manage Airflow connections, variables, and environment variables
Source: https://astronomer.io/docs/astro/manage-connections-variables

Learn about different strategies for managing Airflow connections, variables, and environment variables in local environments and on Astro

*Airflow connections* are used for storing credentials and other information necessary for connecting to external services. *Airflow variables* are a generic way to store and retrieve arbitrary content or settings as a simple key value store within Airflow. *Environment variables* are key-value configurations that can be used to configure Airflow settings, store credentials, or pass configuration to your DAGs.

Use this document to select the right Airflow connection, variable, and environment variable management strategies for your team.

Airflow supports several different methods for managing connections, variables, and environment variables. Each of these strategies has benefits and limitations related to their security and ease of use. The strategies you choose should be compatible with both your local environments and Astro Deployments, allowing you to [import and export objects](/docs/astro/import-export-connections-variables) between the two contexts.

For in-depth information on managing connections and variables, see [Connection Basics](/docs/learn/connections) and [Variable Basics](/docs/learn/airflow-variables).

## Prerequisites

* A locally hosted Astro project created with the Astro CLI. See [Create a project](/docs/cli/v1.43/get-started-cli).
* A Deployment on Astro. See [Create a Deployment](/docs/astro/create-deployment).

## Choose a connection, variable, and environment variable management strategy

The following table suggests possible management strategies for specific use cases.

| Scenario                                                                                                                                        | Strategy                                                                                                   |
| ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| I'm getting started and want to quickly create Airflow objects                                                                                  | [Astro Environment Manager](#astro-environment-manager)                                                    |
| I prefer to manage my Airflow variables in a Git repository and to upload directly to Airflow                                                   | [Airflow UI](#airflow-ui)                                                                                  |
| I need to keep my connections, variables, and environment variables stored in a centralized and secure location.                                | [Secrets backend](#secrets-backend) or [Astro Environment Manager](#astro-environment-manager)             |
| I want to create Connections, Airflow Variables, or environment variables once, and then apply them to multiple Deployments in a Workspace.     | [Astro Environment Manager](#astro-environment-manager)                                                    |
| I want to share connections or variables with Astro IDE projects and their ephemeral test Deployments without affecting production Deployments. | [Astro Environment Manager](#astro-environment-manager)                                                    |
| I have existing connections or variables in a Deployment's Airflow metadata database and want to manage them centrally.                         | [Migrate existing objects to the Environment Manager](/docs/astro/migrate-metadata-db-to-environment-manager)   |
| I don't have a secrets backend, but I still want some security and permissions attached to Airflow objects.                                     | [Astro Environment Manager](#astro-environment-manager) or [Environment variables](#environment-variables) |

### How Airflow finds connections

Because connections serve different purposes in Airflow, you might want to use a different strategy for each object type. For example, you can use a secrets backend for connections and use combination of a `json` files and the Airflow UI for variables.

If you use a mix of strategies for managing connections, it's important to know which connection value that Airflow gives precedence to in the case of a conflict. The order of precedence for connections is:

1. Secrets Backend
2. Astro Environment Manager
3. Environment Variables
4. Airflow's metadata database (Airflow UI)

Airflow checks for connections in order from highest (secrets backend) to lowest (Airflow UI) precedence. Airflow uses the first configuration for a given connection ID that it finds.

For example, if you store a connection with the same connection ID in both a secrets backend and the Airflow UI, Airflow reads and uses the connection configured in the secrets backend first, instead of using a connection configuration with the same connection ID stored in the Airflow UI.

<Tip>If you only want to test connections or export connections in  a JSON or URI format, use the Airflow UI to [manage your connection](/docs/learn/connections#defining-connections-in-the-airflow-ui).  You can then use the Astro CLI commands to export the connections in a URI or JSON format. See [Import and export connections and variables](/docs/astro/import-export-connections-variables#from-the-airflow-ui-and-metadata-database).</Tip>

## Compare strategies

The following sections explain the benefits, limitations, and implementations of each strategy in more detail.

### Astro Environment Manager

Astro includes a built-in secrets backend for managing connections, Airflow variables, and environment variables. After you create a connection, variable, or environment variable in the Astro UI, you can share it with multiple Deployments in a Workspace, override values on a per-Deployment basis, or hide secret values after creating them. See [Create connections in Astro](/docs/astro/create-and-link-connections), [Create variables in Astro](/docs/astro/create-and-link-variables), or [Create environment variables in Astro](/docs/astro/create-and-link-environment-variables) for setup steps.

You can also link environment objects to [Astro IDE](/docs/astro/ide-overview) projects so that the project's ephemeral test Deployments start with the connections and variables they need. Project-scoped objects don't affect Deployments that aren't started from the Astro IDE. See [Link connections to Astro IDE projects](/docs/astro/create-and-link-connections#link-connections-to-astro-ide-projects).

If you already define connections, Airflow variables, or environment variables in a Deployment, you can move them into the Environment Manager so that they can be reused across Deployments and Astro IDE projects. See [Migrate existing objects to the Environment Manager](/docs/astro/migrate-metadata-db-to-environment-manager).

<Warning>For Remote Execution Deployments, you can only manage environment variables and Metrics export with the Environment Manager. You cannot create, update, or assign Connections or Airflow Variables.</Warning>

#### Benefits

* It securely stores your connection, variable, and environment variable configuration and authentication information on Astro, without a secrets backend.
* You can create a configuration once and then add it to multiple Deployments in a Workspace.
* Authenticate connections using credentials other than what the Airflow UI supports.
* Hide the values for Airflow Variable keys and environment variables in the Astro UI, after you finish creating them.
* After you configure a connection, variable, or environment variable, the authentication information is securely stored in an Astro-managed secrets backend.
* Control whether connections, Airflow variables, or environment variables can be used in a single Deployment or shared across Deployments in a Workspace.
* Override certain fields in connections, variable values, or environment variable values per Deployment, so you can create a general configuration and then customize its behavior.

#### Limitations

* If you create a connection in the Astro UI, you also need to add its related provider package to the `requirements.txt` file in your Astro project.
* Only available with Astro Runtime 9.3.0 and greater.
* You can't see connections, Airflow variables, or environment variables defined in the Astro UI in the Airflow UI.
* You need `WORKSPACE_OPERATOR` or `WORKSPACE_OWNER` user permissions.
* You can't programmatically import connections, Airflow variables, or environment variables to the Environment Manager from your local environment.

<Info>To see how you can use connections set in the Astro Environment Manager in a best practice, branch-based Deployment setup, see the [Manage Astro connections in branch-based deploy workflows](/docs/astro/best-practices/connections-branch-deploys) use case.</Info>

### Airflow UI

You can create Airflow connections and variables is through the Airflow UI. This experience is the same for both local Airflow environments and Astro Deployments. Astronomer recommends this method if you're just getting started with Airflow, you want to get your dags running quickly, or if you want to export connections in a URI/JSON format.

#### Benefits

* The UI has features for correctly formatting and testing your connections.
* It's easy to change variables or connections to test different use cases on the fly.
* You can export and import your variables from the Airflow UI using JSON files and the Astro CLI. See [Import and export connections and variables](/docs/astro/import-export-connections-variables#from-the-airflow-ui-and-metadata-database).
* Connections and variables are encrypted and stored in the Airflow metadata database.

#### Limitations

* You cannot export or import connections from the UI for security reasons.
* Managing many connections or variables can become unwieldy.
* In a local environment, you lose your connections and variables if you delete your metadata database with `astro dev kill`.

### Secrets backend

A secrets backend is the most secure way to store connections and variables. You can access a secrets backend both locally and on Astro by configuring the appropriate credentials in your Airflow environment. Astronomer recommends this method for all staging and production deployments. See the following documentation for setup steps:

* [Authenticate to clouds locally](/docs/cli/v1.43/authenticate-to-clouds)
* [Configure a secrets backend](/docs/astro/secrets-backend)

#### Benefits

* Store objects in a centralized location alongside other secrets used by your organization.
* Comply with internal security postures and policies that protect your organization.
* Recover objects if your Airflow environments go down.
* Share secrets across different Airflow environments.
* Allow selective access to connections and variables by using `connections_prefix` and `variables_prefix`.
* Limit the number of open connections to your metadata database, especially if you are using your connections and variables outside of task definitions.

#### Limitations

* A third-party secrets manager is required.
* Separate configurations might be required for using a secrets backend locally and on Astro.
* You cannot use the Airflow UI to view connections and variables.
* You are responsible for ensuring that secrets are encrypted.

### Environment variables

You can use Airflow's system-level environment variables to store connections and variables, or to configure general Airflow settings and pass configuration to your DAGs. There are two ways to manage environment variables on Astro:

* **Workspace-level environment variables** (Astro Environment Manager): Create environment variables at the Workspace level and share them across multiple Deployments. This is the recommended approach when you want to standardize environment variables across Deployments. See [Create environment variables in Astro](/docs/astro/create-and-link-environment-variables).
* **Deployment-level environment variables**: Set environment variables specific to a single Deployment through the Deployment UI or your Dockerfile. This is recommended when you don't have a secrets backend, but you still want to take advantage of security and RBAC features. See [Manage environment variables on Astro](/docs/astro/manage-env-vars).

Airflow connections and variables are stored in the Airflow metadata database. Calling them outside of task definitions and operators requires an additional connection to the Airflow metadata database which is used every time the scheduler parses a dag. By adding connections and variables as environment variables, you can lower the amount of open connections and improve the performance of your database and resources.

#### Benefits

* If you use an `.env` file for your local Airflow environment and your local metadata database is corrupted or accidentally deleted, you still have access to all of your connections and variables.
* You can export environment variables from a local Airflow environment to Astro using the Astro CLI. See [Import and export connections and variables](/docs/astro/import-export-connections-variables#from-environment-variables).
* You can override Airflow variables set in the Airflow UI. See [Environment variable priority](/docs/astro/environment-variables#environment-variable-priority)
* You can create your environment variables from the Astro UI at both the Workspace and Deployment level. See [Create environment variables in Astro](/docs/astro/create-and-link-environment-variables) or [Manage environment variables on Astro](/docs/astro/manage-env-vars).
* Environment variables marked as **Secret** are encrypted in the Astronomer control plane. See [How environment variables are stored on Astro](/docs/astro/environment-variables#how-environment-variables-are-stored-in-the-astro-ui) for details.
* This approach limits the number of open connections to your metadata database, especially if you are using your connections and variables outside of task definitions.
* Workspace-level environment variables can be shared across multiple Deployments with override capabilities per Deployment.

#### Limitations

* You can't view connections and variables from the Airflow UI when they are stored as environment variables.
* You must restart your local environment using `astro dev restart` whenever you make changes to your `.env` file.
* When stored in your `.env` file or Dockerfile, environment variables are defined in plain text.
* Connections must be formatted as either a URI or serialized JSON.
* Deployment-level environment variables are not as secure or centralized compared to a [secrets backend](/docs/astro/secrets-backend) or the Workspace Environment Manager.
* You cannot directly export Deployment-level environment variables from the Astro UI to a local Airflow environment. See [Import and export Airflow objects](/docs/astro/import-export-connections-variables#from-environment-variables).

## Other strategies

While it's possible to manage Airflow connections and variables with these strategies, Astronomer doesn't recommend them at scale:

* You can use the Airflow REST API to programmatically create Airflow connections and variables for a Deployment. Airflow objects created with the API are stored in the Airflow metadata database and visible in the Airflow UI.
* For local Astro projects, you can use `airflow_settings.yaml` for defining your connections and variables. See [Configure `airflow_settings.yaml`](/docs/cli/v1.43/develop-project#configure-airflow_settings-yaml-local-development-only) for more details.

## See also

* [Import and export Airflow objects](/docs/astro/import-export-connections-variables)
* [Authenticate to cloud services with user credentials](/docs/cli/v1.43/authenticate-to-clouds)


# Manage Dag runs
Source: https://astronomer.io/docs/astro/manage-dags

View, retry, and troubleshoot Dag runs across your Astro Workspace from the Dags page.

As a data engineer or data scientist, you might need to view details about your Dags' performance, including task logs, run status, and retries. You might additionally have to manually retry Dag runs or mark them as a specific status when troubleshooting any issues.

<Info>
  **Dag-level access control**

  For Deployments running Astro Runtime 3.1-12 or later, you can control who has access to individual Dags using Dag-level roles. See [Dag-level access control](/docs/astro/dag-level-access-control).
</Info>

## Dags overview

The **DAGs** page in the Astro UI lets you view all Dags in your Workspace from a single place. You can view high level metrics about each Dag in a summary table.

To access the **DAGs** page, either click **DAGs** on the left sidebar or click **DAGs** on a Deployment's information page.

<Frame>
  <img alt="The Dags page in the Astro UI, showing summary information for two Dags" />
</Frame>

The **DAGs** page shows the following summary information about the Dag runs for all Deployments in your Workspace. You can filter through these Dags using the left menu:

* Total Dag runs over the last 14 days, expressed as a bar chart.

  Each bar in the chart represents an individual Dag run. A bar's color represents whether the Dag run was a success or a failure, while its length represents the total duration of the Dag run. If there are more than 14 Dag runs in the last 14 days, then the chart shows only the 14 most recent Dag runs.
* **State**: Indicates whether the Dag is **Active** or **Paused**. If a Dag has a purple lightning symbol next to its name, that Dag is **Active**.
* **Last Run**: The duration of the last Dag run and the ending time of the Dag's most recent Dag run, expressed relative to the current time.
* **Schedule**: The frequency that the Dag runs and the starting time of the next Dag run, expressed relative to the current time.
* **Deployment**: The Deployment ID of the Deployment for the current Dag Run.
* **Owner(s)**: The Airflow Dag owner attribute. You can change the owner attribute when you write or update your Dag.
* **Tags**: The custom tags that you marked your Dag with. To add custom tags to a Dag, see [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/howto/add-Dag-tags.html).

## Access the Airflow UI for a Deployment

Every Astro Deployment runs its own webserver and instance of the Airflow UI. If you want to manage your Dag and task runs through the Airflow UI, you can access it either through the Astro UI or the Astro CLI.

To access the Airflow UI for your Deployment in the Astro UI:

1. In the Astro UI, open your Deployment.
2. Click **Open Airflow**.

To access a Deployment's Airflow UI from the Astro CLI:

1. Run the following command to retrieve the URL for the Deployment's Airflow UI:

   ```sh wrap theme={null}
   astro deployment inspect -n <deployment-name> -k metadata.webserver_url
   ```

2. Copy the URL into a web browser and hit enter to directly open the Airflow UI for your Deployment.

### Access the Dags page in the Airflow UI

1. In the Astro UI, Click **DAGs**.
2. Click **Open in Airflow** for the Dag that you want to manage.


# Migrate to Astro from Google Cloud Composer
Source: https://astronomer.io/docs/astro/migrate-gcc

Get started on Astro by migrating your Airflow code from Google Cloud Composer (GCC).

This is where you'll find instructions for migrating an Airflow environment from [Google Cloud Composer (GCC)](https://cloud.google.com/composer/docs/concepts/overview) to Astro.

To complete the migration process, you will:

* Prepare your source Airflow and set up your Astro Airflow environment.
* Migrate metadata from your source Airflow environment.
* Migrate dags and additional Airflow components from your source Airflow environment.
* Complete the cutover process to Astro.

## Prerequisites

Before starting the migration, ensure that the following are true:

* You have an Astro user account and can [log in to Astro](/docs/astro/log-in-to-astro).
* (Optional) You created a network connection from Astro to your external cloud resources..
* (Optional) You [configured your identity provider (IdP)](/docs/astro/configure-idp).

On your local machine, make sure you have:

* An [Astro account](/docs/astro/log-in-to-astro).
* The [Astro CLI](/docs/cli/v1.43/install-cli).

On the cloud service from which you're migrating, ensure that you have:

* A source Airflow environment on Airflow 2 or later.
* Read access to the source Airflow environment.
* Read access to any cloud storage buckets that store your DAGs.
* Read access to any source control tool that hosts your current Airflow code, such as GitHub.
* Permission to create new repositories on your source control tool.
* (Optional) Access to your secrets backend.
* (Optional) Permission to create new CI/CD pipelines.

All source Airflow environments on 1.x need to be upgraded to at least Airflow 2.0 before you can migrate them. Astronomer professional services can help you with the upgrade process.

<Info>
  If you're migrating to Astro from OSS Airflow or another Astronomer product, and you currently use an older version of Airflow, you can still create Deployments with the corresponding version of Astro Runtime even if it is deprecated according to the [Astro Runtime maintenance policy](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-maintenance-policy). This allows you to migrate your DAGs to Astro without needing to make any code changes and then immediately upgrade to a new version of Airflow. Note that after you migrate your DAGs, Astronomer recommends upgrading to a supported version of Astro Runtime as soon as you can.

  See [Run a deprecated Astro Runtime version](https://www.astronomer.io/docs/astro/upgrade-runtime#run-a-deprecated-astro-runtime-version).
</Info>

You can additionally use the `gcloud` CLI to expedite some steps in this guide.

## Step 1: Install Astronomer Starship

The [Astronomer Starship](https://astronomer.github.io/starship/) migration utility connects your source Airflow environment to your Astro Deployment and migrates your Airflow connections, Airflow variables, environment variables, and dags.

The Starship migration utility works as a [plugin](https://github.com/astronomer/starship/blob/main/README.md) with a user interface, or as an [Airflow operator](https://astronomer.github.io/starship/operator/) if you are migrating from a more restricted Airflow environment.

See the following table for information on which versions of Starship are available depending on your source Airflow environment:

| Source Airflow environment     | Starship plugin | Starship operator |
| ------------------------------ | --------------- | ----------------- |
| Airflow 1.x                    | ❌               | ❌                 |
| Cloud Composer 1 - Airflow 2.x |                 | ✔️️               |
| Cloud Composer 2 - Airflow 2.x | ✔️️             |                   |

To install the Starship plugin on your Cloud Composer 1 or Cloud Composer 2 instance, install the `astronomer-starship` package in your source Airflow environment. See [Install packages from PyPI](https://cloud.google.com/composer/docs/composer-2/install-python-dependencies#install-pypi)

<Tip>
  You can alternatively complete this installation with the `gcloud` CLI by running the following command:

  ```sh wrap theme={null}
  gcloud composer environments update [GCC_ENVIRONMENT_NAME] \
      --location [LOCATION] \
      --update-pypi-package=astronomer-starship
  ```
</Tip>

## Step 2: Create an Astro Workspace

In your Astro Organization, you can create *Workspaces*, which are a collection of users that have access to the same Deployments. Workspaces are typically owned by a single team.

You can choose to use an existing Workspace, or create a new one. However, you must have at least one Workspace to complete your migration.

1. Follow the steps in [Manage Workspaces](/docs/astro/manage-workspaces) to create a Workspace in the Astro UI for your migrated Airflow environments. Astronomer recommends naming your first Workspace after your data team or initial business use case with Airflow. You can update these names in the Astro UI after you finish the migration.

2. Follow the steps in [Manage Astro users](/docs/astro/manage-workspace-users#add-a-user-to-a-workspace) to add users from your team to the Workspace. See [Astro user permissions](/docs/astro/user-permissions#workspace-roles) for details about each available Workspace user role.

<Info>
  **CLI**

  You can add users to a Workspace an Organization using the Astro CLI. See:

  * [`astro workspace user add`](/docs/cli/v1.43/astro-workspace-user-add)
  * [`astro organization user invite`](/docs/cli/v1.43/astro-organization-user-invite)
</Info>

You can also automate adding batches of users to Astro with shell scripts. See [Add a group of users to Astro using the Astro CLI](/docs/astro/manage-workspace-users#add-a-group-of-users-to-a-workspace-using-the-astro-cli).

## Step 3: Create an Astro Deployment

A *Deployment* is an Astro Runtime environment that is powered by the core components of Apache Airflow. In a Deployment, you can deploy and run DAGs, configure worker resources, and view metrics.

You can choose to use an existing Deployment, or create a new one. However, you must have at least one Deployment to complete your migration.

Before you create your Deployment, copy the following information from your source Airflow environment:

* Environment name
* Airflow version
* Environment class or size
* Number of schedulers
* Minimum number of workers
* Maximum number of workers
* Execution role permissions
* Airflow configurations
* Environment variables

<Tabs>
  <Tab title="Option 1: Astro UI">
    <Info>
      **Alternative setup for Astro Hybrid**

      This setup varies slightly for Astro Hybrid users. See [Deployment settings](/docs/astro/deployment-settings) for all configurations related to Astro Hybrid Deployments.
    </Info>

    1. In the Astro UI, select a Workspace.

    2. On the **Deployments** page, click **Deployment**.

    3. Complete the following fields:

       * **Name**: Enter the name of your source Airflow environment.
       * **Astro Runtime**: Select the Runtime version that's based on the Airflow version in your source Airflow environment. See the following table to determine which version of Runtime to use. Where exact version matches are not available, the nearest Runtime version is provided with its supported Airflow version in parentheses.

       | Airflow Version | Runtime Version        |
       | --------------- | ---------------------- |
       | 2.0             | 3.0.4 (Airflow 2.1.1)¹ |
       | 2.2             | 4.2.9 (Airflow 2.2.5)  |
       | 2.4             | 6.3.0 (Airflow 2.4.3)  |

    <Info>¹The earliest available Airflow version on Astro Runtime is 2.1.1. There are no known risks for upgrading directly from Airflow 2.0 to Airflow 2.1.1 during migration. For a complete list of supported Airflow versions, see [Astro Runtime release and lifecycle schedule](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-lifecycle-schedule).</Info>

    * **Description**: (Optional) Enter a description for your Deployment.
    * **Cluster**: Choose whether you want to run your Deployment in a **Standard cluster** or **Dedicated cluster**. If you don't have specific networking or cloud requirements, Astronomer recommends using the default **Standard cluster** configurations.

    To configure and use dedicated clusters, see [Create a dedicated cluster](/docs/astro/create-dedicated-cluster). If you don't have the option of choosing between standard or dedicated, that means you are an Astro Hybrid user and must choose a cluster that has been configured for your Organization.

    * **Executor**: Choose the same executor as in your source Airflow environment.
    * **Scheduler**: Use the following table to determine the Deployment size you need based on the size of your source Airflow environment.

      | Environment size                | Scheduler size | vCPU                                                | Memory                                                     | Ephemeral Storage |
      | ------------------------------- | -------------- | --------------------------------------------------- | ---------------------------------------------------------- | ----------------- |
      | Small (Up to \~50 DAGs)         | Small          | 1                                                   | 2Gi                                                        | 5 GiB             |
      | Medium (Up to \~250 DAGs)       | Medium         | **Scheduler**: 1 <br /> **DAG Processor**: 1        | **Scheduler**: 2 GiB <br /> **DAG Processor**: 2 GiB²      | 5 GiB             |
      | Large (Up to \~1000 DAGs)       | Large          | **Scheduler**: 1 <br /> **DAG Processor**: 3        | **Scheduler**: 2 GiB <br /> **DAG Processor**: 6 GiB²      | 5 GiB             |
      | Extra Large (Up to \~2000 DAGs) | Extra-large    | **Scheduler**: 1 <br /> **DAG Processor (x2)**: 3.5 | **Scheduler**: 4 GiB <br /> **DAG Processor (x2)**: 6 GiB² | 5 GiB             |

    <Info>²Some of the following recommendations for CPU and memory might be less than what you currently allocate to Airflow components in your source environment. If you notice significant performance differences or your Deployment on Astro parses DAGs more slowly than your source Airflow environment, adjust your resource use on Astro. See [Configure Deployment resources](/docs/astro/deployment-resources)</Info>

    * **Worker Type**: Select the worker type for your default worker queue. See [Worker queues](/docs/astro/configure-worker-queues).
    * **Min / Max # Workers**: Set the same minimum and maximum worker count as in source Airflow environment.
    * **KPO Pods**: (Optional) If you use the `KubernetesPodOperator` or Kubernetes Executor, set limits on how many resources your tasks can request.

    4. Click **Create Deployment**.
    5. Specify any system-level environment variables as Astro environment variables. See [Environment variables](/docs/astro/manage-env-vars#use-the-astro-ui).
    6. Set an email to receive alerts from Astronomer support about your Deployments. See [Configure Deployment contact emails](/docs/astro/deployment-details#configure-deployment-contact-emails).
  </Tab>

  <Tab title="Option 2: Astro CLI & Deployments-as-Code">
    <Info>This option is available only on Astro Hybrid.</Info>

    1. On your local machine, create a directory with the name of the source Airflow environment. In this directory, create a file called `config.yaml`.

    2. Open `config.yaml` and add the following:

       ```yaml wrap theme={null}
       deployment:
         environment_variables:
           - is_secret: <true-or-false>
             key: <variable-name>
             value: <variable-value>
         configuration:
           name: <deployment-name>
           description: <deployment-description>
           runtime_version: <runtime-version>
           dag_deploy_enabled: false
           scheduler_au: <scheduler-au>
           scheduler_count: <scheduler-count>
           cluster_name: <cluster-name>
           workspace_name: <workspace-name>
         worker_queues:
           - name: default
             max_worker_count: <max-worker-count>
             min_worker_count: <min-worker-count>
             worker_concurrency: 16
             worker_type: <worker-type>
         alert_emails:
           - <alert-email>
       ```

    3. Replace the placeholder values in the configuration:

       * `<true-or-false>` / `<variable-name>` / `<variable-value>`: Set system-level environment variables for your Deployment and specify whether they should be secret. Repeat this configuration in the file for any additional variables you need to set.
       * `<deployment-name>`: Enter the same name as your source Airflow environment.
       * `<deployment-description>`: (Optional) Enter a description for your Deployment.
       * `<runtime-version>`: Select the Runtime version that's based on the Airflow version in your source Airflow environment. See the following table to determine which version of Runtime to use. Where exact version matches are not available, the nearest Runtime version is provided with its supported Airflow version in parentheses.

       | Airflow Version | Runtime Version        |
       | --------------- | ---------------------- |
       | 2.0             | 3.0.4 (Airflow 2.1.1)¹ |
       | 2.2             | 4.2.9 (Airflow 2.2.5)  |
       | 2.4             | 6.3.0 (Airflow 2.4.3)  |

    <Info>¹The earliest available Airflow version on Astro Runtime is 2.1.1. There are no known risks for upgrading directly from Airflow 2.0 to Airflow 2.1.1 during migration. For a complete list of supported Airflow versions, see [Astro Runtime release and lifecycle schedule](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-lifecycle-schedule).</Info>

    * `<scheduler-au>`: Set your scheduler size in Astronomer Units (AU). An AU is a unit of CPU and memory allocated to each scheduler in a Deployment. Use the following table to determine how many AUs you need based on the size of your source Airflow environment.

    | Environment size          | AUs | CPU / memory      |
    | ------------------------- | --- | ----------------- |
    | Small (Up to \~50 DAGs)   | 5   | .5vCPU, 1.88GiB   |
    | Medium (Up to \~250 DAGs) | 10  | 1vCPU, 3.75GiB²   |
    | Large (Up to \~1000 DAGs) | 15  | 1.5vCPU, 5.64GiB² |

    <Info>² Some of the following recommendations for CPU and memory are smaller than what you have in the equivalent source environment. Although you might not need more CPU or memory than what's recommended, some environments might parse DAGs slower than in your source Airflow environment to start. Use these recommendations as a starting point, then adjust your resource usage after tracking your performance on Astro.</Info>

    * `<scheduler-count>`: Specify the same number of schedulers as in your source Airflow environment.
    * `<cluster-name>`: Specify name of the Astro cluster in which you want to create this Deployment.
    * `<workspace-name>`: The name of the Workspace you created.
    * `<max-worker-count>/ <min-worker-count>`: Specify the same minimum and maximum worker count as in source Airflow environment.
    * `<worker-type>`: Specify the worker type for your default worker queue. You can see which worker types are available for your cluster in the **Clusters** menu of the Astro UI. If you did not customize the available worker types for your cluster, the default available worker types are:

    | AWS       | GCP           | Azure             |
    | --------- | ------------- | ----------------- |
    | M5.XLARGE | E2-STANDARD-4 | `STANDARD_D4D_V5` |

    * `<alert-email>`: Set an email to receive alerts from Astronomer support about your Deployments. See [Set up Astro alerts](/docs/astro/alerts).

    After you finish entering these values, your `config.yaml` file should look something like the following:

    ```yaml wrap theme={null}
    deployment:
      environment_variables:
        - is_secret: true
          key: MY_VARIABLE_KEY
          value: MY_VARIABLE_VALUE
        - is_secret: false
          key: MY_VARIABLE_KEY_2
          value: MY_VARIABLE_VALUE_2
      configuration:
        name: My Deployment
        description: The Deployment I'm using for migration.
        runtime_version: 6.3.0
        dag_deploy_enabled: false
        scheduler_au: 5
        scheduler_count: 2
        cluster_name: My Cluster
        workspace_name: My Workspace
      worker_queues:
        - name: default
          max_worker_count: 10
          min_worker_count: 1
          worker_concurrency: 16
          worker_type: E2-STANDARD-4
      alert_emails:
        - myalertemail@cosmicenergy.org
    ```

    4. Run the following command to push your configuration to Astro and create your Deployment:

       ```sh wrap theme={null}
       astro deployment create --deployment-file config.yaml
       ```
  </Tab>
</Tabs>

## Step 4: Use Starship to Migrate Airflow Connections and Variables

You might have defined Airflow connections and variables in the following places on your source Airflow environment:

* The Airflow UI (stored in the Airflow metadata database).
* Environment variables
* A secrets backend.

If you defined your Airflow variables and connections in the Airflow UI, you can migrate those to Astro with [Starship](https://astronomer.github.io/starship/). You can check which resources will be migrated by going to **Admin** > **Variables** and **Admin** > **Connections** in the Airflow UI to find your source Airflow environment information.

<Warning>Some environment variables or Airflow Settings, like global environment variable values, can't be migrated to Astro. See [Global environment variables](/docs/astro/platform-variables) for a list of variables that you can't migrate to Astro.</Warning>

<Tabs>
  <Tab title="Starship Plugin">
    1. Log in to Astro. In the Astro UI, open the Deployment you're migrating to.
    2. Click **Open Airflow** to open the Airflow UI for the Deployment. Copy the URL for the home page. It should look similar to `https://<your-organization>.astronomer.run/<id>/home`.
    3. Create a Deployment API token for the Deployment. The token should minimally have permissions to update the Deployment and deploy code. Copy this token. See [Create and manage Deployment API tokens](/docs/astro/deployment-api-tokens) for additional setup steps.
    4. Open the Airflow UI for your source Airflow environment, then go to **Astronomer** > **Migration Tool 🚀**.
    5. Ensure that the **Astronomer Product** toggle is set to **Astro**.
    6. In the **Airflow URL** section, fill in the fields so that the complete URL on the page matches the URL of the Airflow UI for the Deployment you're migrating to.
    7. Specify your API token in the **Token** field. Starship will confirm that it has access to your Deployment.
    8. Click **Connections**. In the table that appears, click **Migrate** for each connection that you want to migrate to Astro. After the migration is complete, the status **Migrated ✅** appears.
    9. Click **Pools**. In the table that appears, click **Migrate** for each connection that you want to migrate to Astro. After the migration is complete, the status **Migrated ✅** appears.
    10. Click **Variables**. In the table that appears, click **Migrate** for each variable that you want to migrate to Astro. After the migration is complete, the status **Migrated ✅** appears.
    11. Click **Environment variables**. In the table that appears, check the box for each environment variable that you want to migrate to Astro, then click **Migrate**. After the migration is complete, the status **Migrated ✅** appears.
    12. Click **DAG History**. In the table that appears, check the box for each DAG whose history you want to migrate to Astro, then click **Migrate**. After the migration is complete, the status **Migrated ✅** appears.
  </Tab>

  <Tab title="Starship operator">
    Refer to the [Configuration](https://astronomer.github.io/starship/operator/#usage) detailed instructions on using the operator.

    1. Log in to Astro. In the Astro UI, open the Deployment you're migrating to.

    2. Click **Open Airflow** to open the Airflow UI for the Deployment. Copy the URL for the home page. It should look similar to `https://<your-organization>.astronomer.run/<id>/home`.

    3. Create a Deployment API token for the Deployment. The token should minimally have permissions to update the Deployment and deploy code. Copy this token. See [Create and manage Deployment API tokens](/docs/astro/deployment-api-tokens) for additional setup steps.

    4. Add the following DAG to your source Airflow environment:

       ```python wrap theme={null}
       from airflow.models.dag import DAG
       from astronomer.starship.operators import AstroMigrationOperator
       from datetime import datetime

       with DAG(
         dag_id="astronomer_migration_dag",
         start_date=datetime(1970, 1, 1),
         schedule_interval=None,
       ) as dag:
         AstroMigrationOperator(
             task_id='export_meta',
             deployment_url='{{ dag_run.conf["deployment_url"] }}',
             token='{{ dag_run.conf["astro_token"] }}',
         )
       ```

    5. Deploy this DAG to your source Airflow environment.

    6. Once the DAG is available in the Airflow UI, click **Trigger DAG**, then click **Trigger DAG w/ config**.

    7. In **Configuration JSON**, add the following configuration:

       ```json wrap theme={null}
       {
           "deployment_url": "<your-deployment-url>",
           "astro_token": "<your-astro-token>"
       }
       ```

    8. Replace the following placeholder values:

       * `<your-deployment-url>`: The Deployment URL you copied in Step 2.
       * `<your-astro-token>`: The token you copied in Step 3.

    9. Click **Trigger**. After the DAG successfully runs, all connections, variables, and environment variables that are available from the Airflow UI are migrated to Astronomer.
  </Tab>
</Tabs>

## Step 5: Create an Astro project

1. Create a new directory for your Astro project:

   ```sh wrap theme={null}
   mkdir <your-astro-project-name>
   ```

2. Open the directory:

   ```sh wrap theme={null}
   cd <your-astro-project-name>
   ```

3. Run the following Astro CLI command to initialize an Astro project in the directory:

   ```sh wrap theme={null}
   astro dev init
   ```

   This command generates a set of files that will build into a Docker image that you can both run on your local machine and deploy to Astro.

4. Add the following line to your Astro project `requirements.txt` file:

   ```sh wrap theme={null}
   astronomer-starship
   ```

   When you deploy your code, this line installs the Starship migration tool on your Deployment so that you can migrate Airflow resources from your source environment to Astro.

5. (Optional) Run the following command to initialize a new git repository for your Astro project:

   ```sh wrap theme={null}
   git init
   ```

## Step 6: Migrate project code and dependencies to your Astro project

<Tabs>
  <Tab title="Manual">
    1. Open your Astro project Dockerfile. Update the Runtime version in first line to the version you selected for your Deployment in [Step 3](#step-3-create-an-astro-deployment). For example, if your Runtime version was 6.3.0, your Dockerfile would look like the following:

       ```dockerfile title="Dockerfile" wrap theme={null}
       FROM quay.io/astronomer/astro-runtime:6.3.0
       ```

       The `Dockerfile` defines the environment where all your Airflow components run. You can modify it to include build-time arguments for your Airflow environment, such as environment variables or credentials. For this migration, you only need to modify the Dockerfile to update your Astro Runtime version.

    2. Open your Astro project `requirements.txt` file and add all Python packages that you installed in your source Airflow environment. See [Google documentation](https://cloud.google.com/composer/docs/composer-2/install-python-dependencies#view-custom) for how to view a list of Python packages in your source Airflow environment.

    <Warning>To avoid breaking dependency upgrades, Astronomer recommends pinning your packages to the versions running in your source Airflow environment. For example, if you're running `apache-airflow-providers-snowflake` version 3.3.0 on Cloud composer, you would add `apache-airflow-providers-snowflake==3.3.0` to your Astro `requirements.txt` file.</Warning>

    3. Open your Astro project `dags` folder. Copy your dag files to `dags` from either your source control platform or GCS Bucket.

    4. If you used the [`plugins` folder](https://cloud.google.com/composer/docs/concepts/cloud-storage#folders_in_the_bucket) in your Cloud Composer storage bucket, copy the contents of this folder from your source control platform or GCS Bucket to your Astro project `/plugins` folder.

    5. If you used the [`data` folder](https://cloud.google.com/composer/docs/concepts/cloud-storage#data) in Cloud Composer, copy the contents of that folder from your source control platform or GCS Bucket to your Astro project `include` folder.

    After you confirm that your Astro project has all necessary dependencies, deploy the project to your Astro Deployment.

    1. Run the following command to authenticate to Astro:

       ```sh wrap theme={null}
       astro login
       ```

    2. Run the following command to deploy your project

       ```sh wrap theme={null}
       astro deploy
       ```

       This command returns a list of Deployments available in your Workspace and prompts you to pick one.
  </Tab>

  <Tab title="gcloud CLI">
    1. Open your Astro project Dockerfile. Update the Runtime version in first line to the version you selected for your Deployment in [Step 3](#step-3-create-an-astro-deployment). For example, if your Runtime version was 6.3.0, your Dockerfile would look like the following:

       ```dockerfile title="Dockerfile" wrap theme={null}
       FROM quay.io/astronomer/astro-runtime:6.3.0
       ```

       The `Dockerfile` defines the environment where all your Airflow components run. You can modify it to include build-time arguments for your Airflow environment, such as environment variables or credentials. For this migration, you only need to modify the Dockerfile to update your Astro Runtime version.

    2. Open your Astro project in your terminal. Run the following command to copy your PyPI packages from GCC to your `requirements.txt` file:

       ```sh wrap theme={null}
       gcloud composer environments describe <your-gcc-project-name> --format="value(config.softwareConfig.pypiPackages)" > requirements.txt
       ```

       Review the output in `requirements.txt` after running this command to ensure that all packages were imported on their own line of text.

       <Warning>
         To avoid breaking dependency upgrades, Astronomer recommends pinning your packages to the versions running in your source Airflow environment. For example, if you're running `apache-airflow-providers-snowflake` version 3.3.0 on Cloud composer, you would add `apache-airflow-providers-snowflake==3.3.0` to your Astro `requirements.txt` file.
       </Warning>

    3. Run the following command to copy your dags from Cloud Composer to your `dags` folder:

       ```sh wrap theme={null}
       gcloud composer environments storage dags export --destination=dags
       ```

       Review your dag files in `dags` after running this command to ensure that all dags were successfully exported.

    4. If you utilized the [`plugins` folder](https://cloud.google.com/composer/docs/concepts/cloud-storage#folders_in_the_bucket) in your Cloud Composer storage bucket, run the following command to copy your `plugins` folder contents to your Astro project:

       ```sh wrap theme={null}
       gcloud composer environments storage plugins export --destination=plugins
       ```

       Review the contents of your Astro project `plugins` folder to ensure that all files were successfully exported.

    5. If you utilized the [`data` folder](https://cloud.google.com/composer/docs/concepts/cloud-storage#data) in Cloud Composer, run the following command to copy your `data` folder contents to your Astro project:

       ```sh wrap theme={null}
       gcloud composer environments storage data export --destination=include
       ```

    After you confirm that your Astro project has all necessary dependencies, deploy the project to your Astro Deployment.

    1. Run the following command to authenticate to Astro:

       ```sh wrap theme={null}
       astro login
       ```

    2. Run the following command to deploy your project

       ```sh wrap theme={null}
       astro deploy
       ```

       This command returns a list of Deployments available in your Workspace and prompts you to pick one.
  </Tab>
</Tabs>

## Step 7: Configure additional data pipeline infrastructure

The core migration of your project is now complete. Read the following topics to see whether you need to set up any additional infrastructure on Astro before cutting over your dags.

### Set up CI/CD

If you used CI/CD to deploy code to your source Airflow environment, read the following documentation to learn about setting up a similar CI/CD pipeline for your Astro project:

* [Set-Up CI/CD](/docs/astro/set-up-ci-cd)
* [CI/CD templates](/docs/astro/ci-cd-templates/template-overview)

Similarly to GCC, you can deploy dags to Astro directly from a Google Cloud Storage (GCS) bucket. See [Deploy dags to from Google Cloud Storage to Astro](/docs/astro/ci-cd-templates/gcs).

### Set up a secrets backend

If you currently store Airflow variables or connections in a secrets backend, you need to integrate your secrets backend with Astro to access those objects from your migrated dags. See [Configure a Secrets Backend](/docs/astro/secrets-backend) for setup steps.

#### Instance permissions and trust policies

You can utilize Workload Identity or Service Account Keys to grant your Astro Deployment the same level of access to Google Services as your source Airflow environment. See [Connect GCP - Authorization options](/docs/astro/connect-gcp).

## Step 8: Test locally and check for import errors

Depending on how thoroughly you want to test your Airflow environment, you have a few options for testing your project locally before deploying to Astro.

* In your Astro project directory, run `astro dev parse` to check for any parsing errors in your dags.
* Run `astro run <dag-id>` to test a specific dag. This command compiles your dag and runs it in a single Airflow worker container based on your Astro project configurations.
* Run `astro dev start` to start a complete Airflow environment on your local machine. After your project starts up, you can access the Airflow UI at `localhost:8080`. See [Troubleshoot your local Airflow environment](/docs/cli/v1.43/run-airflow-locally).

Note that your migrated Airflow variables and connections are not available locally. You must deploy your project to Astro to test these resources.

## Step 9: Deploy to Astro

1. Run the following command to authenticate to Astro:

   ```sh wrap theme={null}
   astro login
   ```

2. Run the following command to deploy your project

   ```sh wrap theme={null}
   astro deploy
   ```

   This command returns a list of Deployments available in your Workspace and prompts you to pick one.

3. In the Astro UI, open your Deployment and click **Open Airflow**. Confirm that you can see your deployed DAGs in the Airflow UI.

## Step 10: Cut over from your source Airflow environment to Astro

After you successfully deploy your code to Astro, you need to migrate your workloads from your source Airflow environment to Astro on a DAG-by-DAG basis. Depending on how your workloads are set up, Astronomer recommends letting DAG owners determine the order to migrate and test DAGs.

You can complete the following steps in the few days or weeks following your migration set up. Provide updates to your Astronomer Data Engineer as they continue to assist you through the process and any solve any difficulties that arise.

Continue to validate and move your DAGs until you have fully cut over your source Airflow instance. After you finish migrating from your source Airflow environment, repeat the complete migration process for any other Airflow instances in your source Airflow environment.

#### Confirm connections and variables

In the Airflow UI for your Deployment, [test all connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#testing-connections) that you migrated from your source Airflow environment.

Additionally, check Airflow variable values in **Admin** > **Variables**.

#### Test and validate DAGs in Astro

To create a strategy for testing DAGs, determine which DAGs need the most care when running and testing them.

If your DAG workflow is idempotent and can run twice or more without negative effects, you can run and test these DAGs with minimal risk. If your DAG workflow is non-idempotent and can become invalid when you rerun it, you should test the DAG with more caution and downtime.

#### Cut over DAGs to Astro using Starship

Starship includes features for simultaneously pausing DAGs in your source Airflow environment and starting them on Astro. This allows you to cut over your production workflows without downtime.

<Tabs>
  <Tab title="Starship Plugin">
    For each DAG in your Astro Deployment:

    1. Confirm that the DAG ID in your Deployment is the same as the DAG ID in your source Airflow environment.

    2. In the Airflow UI for your source Airflow environment, go to **Astronomer** > **Migration Tool 🚀**.

    3. Click **DAGs cutover**. In the table that appears, click the Pause icon in the **Local** column for the DAG you're cutting over.

    4. Click the Start icon in the **Remote** column for the DAG you're cutting over.

    5. After completing this cutover, the Start and Pause icons switch. If there's an issue after cutting over, click the **Remote** pause button and then the **Local** start button to move your workflow back to your source Airflow environment.
  </Tab>

  <Tab title="Starship Operator">
    The Starship operator does not contain cut-over functionality.

    To cut over a DAG, pause the DAG in the source Airflow and unpause the DAG in Astro. Keep both Airflow environments open as you test and ensure that the cutover was successful.
  </Tab>
</Tabs>

### Optimize Deployment resource usage

#### Review DAG development features

Astro includes several features that enhance the Apache Airflow development experience, from DAG writing to testing. To make the most of these features, you might want to make adjustments to your existing DAG development workflows.

As you get started on Astro, review the list of features and changes that Astro brings to the Airflow development experience and consider how you want to implement these details in your development experience. See [Write and run DAGs on Astro](/docs/astro/dags-overview).

#### Monitor analytics

As you cut over DAGs, view [Deployment metrics](/docs/astro/deployment-metrics) to get a sense of how many resources your Deployment is using. Use this information to adjust your worker queues and resource usage accordingly, or to tell when a DAG isn't running as expected.

#### Modify instance types or use worker queues

If your current worker type doesn't have the right amount of resources for your workflows, see [Deployment settings](/docs/astro/deployment-settings) to learn about configuring worker types on your Deployments.

You can additionally configure [worker queues](/docs/astro/configure-worker-queues) to assign each of your tasks to different worker instance types. View your [Deployment metrics](/docs/astro/deployment-metrics) to help you determine what changes are required.

#### Enable DAG-only deploys

Deploying to Astro with DAG-only deploys enabled can make deploys faster in cases where you've only modified your `dags` directory. To enable the DAG-only deploy feature, see [Deploy DAGs only](/docs/astro/deploy-dags).


# Migrate existing objects to the Environment Manager
Source: https://astronomer.io/docs/astro/migrate-metadata-db-to-environment-manager

Move existing Airflow connections, Airflow variables, and Deployment environment variables into the Astro Environment Manager.

<Info>
  **Preview**

  The migration and promotion flows on this page are in [Preview](/docs/astro/feature-previews).
</Info>

You can bring existing Airflow connections, Airflow variables, environment variables, and metrics exports into the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager). After an object is in the Environment Manager, you can manage it centrally in the Workspace and link it to other Deployments or to [Astro IDE](/docs/astro/ide-overview) projects.

This page covers two flows:

* **Migrate from Airflow.** Bulk migrate connections and Airflow variables from a Deployment's Airflow metadata database into the Environment Manager.
* **Promote to Workspace Environment.** Promote a single Deployment-scoped connection, Airflow variable, environment variable, or metrics export to the Workspace level.

Each flow keeps the source object available on the Deployment by linking the new Workspace environment object back to that Deployment.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Migrate Airflow connections and variables from a Deployment

The **Migrate from Airflow** flow reads connections and Airflow variables from a Deployment's Airflow metadata database, copies each one into a Workspace environment object linked to the source Deployment, and removes it from the metadata database.

<Warning>
  Migration deletes the source connection or Airflow variable from the Deployment's metadata database. This action can't be undone. After migration, there is a short delay, around one minute, before the migrated objects are available to Dags running on the Deployment.
</Warning>

### Prerequisites

* Workspace Owner [user permissions](/docs/astro/user-permissions).
* A standard Astro Deployment with one or more connections or Airflow variables stored in its metadata database. Remote Execution Deployments aren't supported.

### Migrate a connection or Airflow variable

The migration page provides separate entry points for connections and Airflow variables. The steps are equivalent for both object types.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, select the Deployment that contains the objects you want to migrate, then click the **Environment** tab.
    2. Click **Connections** or **Airflow Variables** in the Deployment Environment menu.
    3. Click **Migrate from Airflow**. The action appears dimmed when the Deployment's metadata database doesn't contain any objects of that type.
    4. In the migration page, select the connections or Airflow variables you want to migrate. Each row shows whether the object is new to the Workspace or whether a Workspace object with the same key already exists. If a matching Workspace object already exists, Astro creates a Deployment-level override on that Workspace object rather than a duplicate.
    5. Click **Migrate**, then confirm in the dialog. The confirmation warns that Astro removes the source objects from the Deployment's metadata database.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, select the Deployment that contains the objects you want to migrate, then click the **Environment** tab.
    2. Click **Connections** or **Airflow Variables** in the Deployment Environment menu.
    3. Click **Migrate from Airflow**. The action appears dimmed when the Deployment's metadata database doesn't contain any objects of that type.
    4. In the migration page, select the connections or Airflow variables you want to migrate. Each row shows whether the object is new to the Workspace or whether a Workspace object with the same key already exists. If a matching Workspace object already exists, Astro creates a Deployment-level override on that Workspace object rather than a duplicate.
    5. Click **Migrate**, then confirm in the dialog. The confirmation warns that Astro removes the source objects from the Deployment's metadata database.
  </Tab>
</Tabs>

After the migration completes, the new objects appear in the Workspace **Environment** page. Each migrated object is linked to the source Deployment, so the Deployment continues to use the same values. The objects aren't automatically available to other Deployments or Astro IDE projects. To use a migrated object elsewhere, link it from the Workspace **Environment** page. See [Link connections to Deployments](/docs/astro/create-and-link-connections#link-connections-to-deployments) or [Link connections to Astro IDE projects](/docs/astro/create-and-link-connections#link-connections-to-astro-ide-projects).

## Promote a Deployment environment object to the Workspace

You can promote a Deployment-scoped connection, Airflow variable, environment variable, or metrics export to the Workspace level. After promotion, the object is managed in the Workspace and remains linked to the source Deployment, so the Deployment continues to use the same values.

### Prerequisites

* `workspace.envObjects.create` and `deployment.update` permissions, included with Workspace Operator or Workspace Owner [user permissions](/docs/astro/user-permissions).
* A Deployment-scoped object that isn't already an override of a Workspace-level object.

### Promote an object

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select the Deployment that contains the object.
    2. Click the **Environment** tab in the Deployment menu, then open **Connections**, **Airflow Variables**, **Environment Variables**, or **Metrics Exports**.
    3. Open the row-level more actions menu for the object you want to promote.
    4. Select **Promote to Workspace Environment**.
    5. Confirm the promotion in the dialog. For environment variables, enter the environment variable key to confirm.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, then select the Deployment that contains the object.
    2. Click the **Environment** tab in the Deployment menu, then open **Connections**, **Airflow Variables**, **Environment Variables**, or **Metrics Exports**.
    3. Open the row-level more actions menu for the object you want to promote.
    4. Select **Promote to Workspace Environment**.
    5. Confirm the promotion in the dialog. For environment variables, enter the environment variable key to confirm.
  </Tab>
</Tabs>

After promotion, the object appears in the Workspace **Environment** page. You can link it to additional Deployments and Astro IDE projects from there.

If a Workspace object with the same type and key already exists, Astro adds a Deployment-level override on the existing Workspace object instead of creating a duplicate.

## See also

* [Manage Airflow connections, variables, and environment variables](/docs/astro/manage-connections-variables)
* [Create connections in Astro](/docs/astro/create-and-link-connections)
* [Create Airflow variables in Astro](/docs/astro/create-and-link-variables)
* [Create environment variables in Astro](/docs/astro/create-and-link-environment-variables)


# Migrate to Astro from Amazon MWAA
Source: https://astronomer.io/docs/astro/migrate-mwaa

Get started on Astro Hosted by migrating your Airflow code from Amazon Managed Workflows for Apache Airflow (MWAA).

Migrate your Airflow environment from [Amazon Managed Workflows for Apache Airflow (MWAA)](https://aws.amazon.com/managed-workflows-for-apache-airflow/resources/) to Astro.

To complete the migration process, you will:

* Prepare your source Airflow and create a Deployment on Astro.
* Migrate metadata from your source Airflow environment.
* Migrate dags and additional Airflow components from your source Airflow environment.
* Complete the cutover process to Astro.

## Prerequisites

Before starting the migration, ensure that the following are true:

* You have an Astro user account and can [log in to Astro](/docs/astro/log-in-to-astro).
* (Optional) You created a network connection from Astro to your external cloud resources..
* (Optional) You [configured your identity provider (IdP)](/docs/astro/configure-idp).

On your local machine, make sure you have:

* An [Astro account](/docs/astro/log-in-to-astro).
* The [Astro CLI](/docs/cli/v1.43/install-cli).

On the cloud service from which you're migrating, ensure that you have:

* A source Airflow environment on Airflow 2 or later.
* Read access to the source Airflow environment.
* Read access to any cloud storage buckets that store your DAGs.
* Read access to any source control tool that hosts your current Airflow code, such as GitHub.
* Permission to create new repositories on your source control tool.
* (Optional) Access to your secrets backend.
* (Optional) Permission to create new CI/CD pipelines.

All source Airflow environments on 1.x need to be upgraded to at least Airflow 2.0 before you can migrate them. Astronomer professional services can help you with the upgrade process.

<Info>
  If you're migrating to Astro from OSS Airflow or another Astronomer product, and you currently use an older version of Airflow, you can still create Deployments with the corresponding version of Astro Runtime even if it is deprecated according to the [Astro Runtime maintenance policy](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-maintenance-policy). This allows you to migrate your DAGs to Astro without needing to make any code changes and then immediately upgrade to a new version of Airflow. Note that after you migrate your DAGs, Astronomer recommends upgrading to a supported version of Astro Runtime as soon as you can.

  See [Run a deprecated Astro Runtime version](https://www.astronomer.io/docs/astro/upgrade-runtime#run-a-deprecated-astro-runtime-version).
</Info>

(Optional) You can use the [AWS CLI](https://aws.amazon.com/cli/) to expedite some of the steps in this guide.

## Step 1: Install Astronomer Starship

The [Astronomer Starship](https://astronomer.github.io/starship/) migration utility connects your source Airflow environment to your Astro Deployment and migrates your Airflow connections, Airflow variables, environment variables, and dags.

The Starship migration utility works as a [plugin](https://github.com/astronomer/starship/blob/main/README.md) with a user interface, or as an [Airflow operator](https://astronomer.github.io/starship/operator/) if you are migrating from a more restricted Airflow environment.

<Warning>
  If you are migrating from an MWAA instance with a private webserver, you will need to use the `StarshipOperator` pattern.
</Warning>

See the following table for information on which versions of Starship are available, depending on your source Airflow environment:

| Source Airflow environment | Starship plugin | Starship operator |
| -------------------------- | --------------- | ----------------- |
| Airflow 1.x                | ❌               | ❌                 |
| MWAA v2.0.2                |                 | ✔️️               |
| MWAA v2.2.2                | ✔️️             |                   |
| MWAA v2.4.3                | ✔️️             |                   |

1. Download the `requirements.txt` file for your source Airflow environment from S3. See [AWS documentation](https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html#best-practices-dependencies-different-ways).
2. Add `astronomer-starship` on a new line to your `requirements.txt` file.
3. Reupload the file to your S3 bucket.
4. Update your Airflow environment to use the new version of this file.

<Tip>
  To complete this setup from the command line:

  1. Run the following commands to set environment variables on your local machine:

     ```sh wrap theme={null}
     export MWAA_NAME="MWAA"
     export MWAA_BUCKET="MWAA BUCKET"
     ```

  2. Run the following AWS CLI commands to install Starship:

     ```sh wrap theme={null}
     aws s3 cp "s3://$MWAA_BUCKET/requirements.txt" requirements.txt
     echo 'astronomer-starship' >> requirements.txt
     aws s3 cp requirements.txt "s3://$MWAA_BUCKET/requirements.txt"
     aws mwaa update-environment "$MWAA_NAME" --requirements-s3-object-version="$(aws s3api head-object --bucket=$MWAA_BUCKET --key=requirements.txt --query="VersionId")"
     ```
</Tip>

## Step 2: Create an Astro Workspace

In your Astro Organization, you can create *Workspaces*, which are a collection of users that have access to the same Deployments. Workspaces are typically owned by a single team.

You can choose to use an existing Workspace, or create a new one. However, you must have at least one Workspace to complete your migration.

1. Follow the steps in [Manage Workspaces](/docs/astro/manage-workspaces) to create a Workspace in the Astro UI for your migrated Airflow environments. Astronomer recommends naming your first Workspace after your data team or initial business use case with Airflow. You can update these names in the Astro UI after you finish the migration.

2. Follow the steps in [Manage Astro users](/docs/astro/manage-workspace-users#add-a-user-to-a-workspace) to add users from your team to the Workspace. See [Astro user permissions](/docs/astro/user-permissions#workspace-roles) for details about each available Workspace user role.

<Info>
  **CLI**

  You can add users to a Workspace an Organization using the Astro CLI. See:

  * [`astro workspace user add`](/docs/cli/v1.43/astro-workspace-user-add)
  * [`astro organization user invite`](/docs/cli/v1.43/astro-organization-user-invite)
</Info>

You can also automate adding batches of users to Astro with shell scripts. See [Add a group of users to Astro using the Astro CLI](/docs/astro/manage-workspace-users#add-a-group-of-users-to-a-workspace-using-the-astro-cli).

## Step 3: Create an Astro Deployment

A *Deployment* is an Astro Runtime environment that is powered by the core components of Apache Airflow. In a Deployment, you can deploy and run DAGs, configure worker resources, and view metrics.

You can choose to use an existing Deployment, or create a new one. However, you must have at least one Deployment to complete your migration.

Before you create your Deployment, copy the following information from your source Airflow environment:

* Environment name
* Airflow version
* Environment class or size
* Number of schedulers
* Minimum number of workers
* Maximum number of workers
* Execution role permissions
* Airflow configurations
* Environment variables

<Tabs>
  <Tab title="Option 1: Astro UI">
    <Info>
      **Alternative setup for Astro Hybrid**

      This setup varies slightly for Astro Hybrid users. See [Deployment settings](/docs/astro/deployment-settings) for all configurations related to Astro Hybrid Deployments.
    </Info>

    1. In the Astro UI, select a Workspace.

    2. On the **Deployments** page, click **Deployment**.

    3. Complete the following fields:

       * **Name**: Enter the name of your source Airflow environment.
       * **Astro Runtime**: Select the Runtime version that's based on the Airflow version in your source Airflow environment. See the following table to determine which version of Runtime to use. Where exact version matches are not available, the nearest Runtime version is provided with its supported Airflow version in parentheses.

       | Airflow Version | Runtime Version        |
       | --------------- | ---------------------- |
       | 2.0             | 3.0.4 (Airflow 2.1.1)¹ |
       | 2.2             | 4.2.9 (Airflow 2.2.5)  |
       | 2.4             | 6.3.0 (Airflow 2.4.3)  |

    <Info>¹The earliest available Airflow version on Astro Runtime is 2.1.1. There are no known risks for upgrading directly from Airflow 2.0 to Airflow 2.1.1 during migration. For a complete list of supported Airflow versions, see [Astro Runtime release and lifecycle schedule](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-lifecycle-schedule).</Info>

    * **Description**: (Optional) Enter a description for your Deployment.
    * **Cluster**: Choose whether you want to run your Deployment in a **Standard cluster** or **Dedicated cluster**. If you don't have specific networking or cloud requirements, Astronomer recommends using the default **Standard cluster** configurations.

    To configure and use dedicated clusters, see [Create a dedicated cluster](/docs/astro/create-dedicated-cluster). If you don't have the option of choosing between standard or dedicated, that means you are an Astro Hybrid user and must choose a cluster that has been configured for your Organization.

    * **Executor**: Choose the same executor as in your source Airflow environment.
    * **Scheduler**: Use the following table to determine the Deployment size you need based on the size of your source Airflow environment.

      | Environment size                | Scheduler size | vCPU                                                | Memory                                                     | Ephemeral Storage |
      | ------------------------------- | -------------- | --------------------------------------------------- | ---------------------------------------------------------- | ----------------- |
      | Small (Up to \~50 DAGs)         | Small          | 1                                                   | 2Gi                                                        | 5 GiB             |
      | Medium (Up to \~250 DAGs)       | Medium         | **Scheduler**: 1 <br /> **DAG Processor**: 1        | **Scheduler**: 2 GiB <br /> **DAG Processor**: 2 GiB²      | 5 GiB             |
      | Large (Up to \~1000 DAGs)       | Large          | **Scheduler**: 1 <br /> **DAG Processor**: 3        | **Scheduler**: 2 GiB <br /> **DAG Processor**: 6 GiB²      | 5 GiB             |
      | Extra Large (Up to \~2000 DAGs) | Extra-large    | **Scheduler**: 1 <br /> **DAG Processor (x2)**: 3.5 | **Scheduler**: 4 GiB <br /> **DAG Processor (x2)**: 6 GiB² | 5 GiB             |

    <Info>²Some of the following recommendations for CPU and memory might be less than what you currently allocate to Airflow components in your source environment. If you notice significant performance differences or your Deployment on Astro parses DAGs more slowly than your source Airflow environment, adjust your resource use on Astro. See [Configure Deployment resources](/docs/astro/deployment-resources)</Info>

    * **Worker Type**: Select the worker type for your default worker queue. See [Worker queues](/docs/astro/configure-worker-queues).
    * **Min / Max # Workers**: Set the same minimum and maximum worker count as in source Airflow environment.
    * **KPO Pods**: (Optional) If you use the `KubernetesPodOperator` or Kubernetes Executor, set limits on how many resources your tasks can request.

    4. Click **Create Deployment**.
    5. Specify any system-level environment variables as Astro environment variables. See [Environment variables](/docs/astro/manage-env-vars#use-the-astro-ui).
    6. Set an email to receive alerts from Astronomer support about your Deployments. See [Configure Deployment contact emails](/docs/astro/deployment-details#configure-deployment-contact-emails).
  </Tab>

  <Tab title="Option 2: Astro CLI & Deployments-as-Code">
    <Info>This option is available only on Astro Hybrid.</Info>

    1. On your local machine, create a directory with the name of the source Airflow environment. In this directory, create a file called `config.yaml`.

    2. Open `config.yaml` and add the following:

       ```yaml wrap theme={null}
       deployment:
         environment_variables:
           - is_secret: <true-or-false>
             key: <variable-name>
             value: <variable-value>
         configuration:
           name: <deployment-name>
           description: <deployment-description>
           runtime_version: <runtime-version>
           dag_deploy_enabled: false
           scheduler_au: <scheduler-au>
           scheduler_count: <scheduler-count>
           cluster_name: <cluster-name>
           workspace_name: <workspace-name>
         worker_queues:
           - name: default
             max_worker_count: <max-worker-count>
             min_worker_count: <min-worker-count>
             worker_concurrency: 16
             worker_type: <worker-type>
         alert_emails:
           - <alert-email>
       ```

    3. Replace the placeholder values in the configuration:

       * `<true-or-false>` / `<variable-name>` / `<variable-value>`: Set system-level environment variables for your Deployment and specify whether they should be secret. Repeat this configuration in the file for any additional variables you need to set.
       * `<deployment-name>`: Enter the same name as your source Airflow environment.
       * `<deployment-description>`: (Optional) Enter a description for your Deployment.
       * `<runtime-version>`: Select the Runtime version that's based on the Airflow version in your source Airflow environment. See the following table to determine which version of Runtime to use. Where exact version matches are not available, the nearest Runtime version is provided with its supported Airflow version in parentheses.

       | Airflow Version | Runtime Version        |
       | --------------- | ---------------------- |
       | 2.0             | 3.0.4 (Airflow 2.1.1)¹ |
       | 2.2             | 4.2.9 (Airflow 2.2.5)  |
       | 2.4             | 6.3.0 (Airflow 2.4.3)  |

    <Info>¹The earliest available Airflow version on Astro Runtime is 2.1.1. There are no known risks for upgrading directly from Airflow 2.0 to Airflow 2.1.1 during migration. For a complete list of supported Airflow versions, see [Astro Runtime release and lifecycle schedule](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-lifecycle-schedule).</Info>

    * `<scheduler-au>`: Set your scheduler size in Astronomer Units (AU). An AU is a unit of CPU and memory allocated to each scheduler in a Deployment. Use the following table to determine how many AUs you need based on the size of your source Airflow environment.

    | Environment size          | AUs | CPU / memory      |
    | ------------------------- | --- | ----------------- |
    | Small (Up to \~50 DAGs)   | 5   | .5vCPU, 1.88GiB   |
    | Medium (Up to \~250 DAGs) | 10  | 1vCPU, 3.75GiB²   |
    | Large (Up to \~1000 DAGs) | 15  | 1.5vCPU, 5.64GiB² |

    <Info>² Some of the following recommendations for CPU and memory are smaller than what you have in the equivalent source environment. Although you might not need more CPU or memory than what's recommended, some environments might parse DAGs slower than in your source Airflow environment to start. Use these recommendations as a starting point, then adjust your resource usage after tracking your performance on Astro.</Info>

    * `<scheduler-count>`: Specify the same number of schedulers as in your source Airflow environment.
    * `<cluster-name>`: Specify name of the Astro cluster in which you want to create this Deployment.
    * `<workspace-name>`: The name of the Workspace you created.
    * `<max-worker-count>/ <min-worker-count>`: Specify the same minimum and maximum worker count as in source Airflow environment.
    * `<worker-type>`: Specify the worker type for your default worker queue. You can see which worker types are available for your cluster in the **Clusters** menu of the Astro UI. If you did not customize the available worker types for your cluster, the default available worker types are:

    | AWS       | GCP           | Azure             |
    | --------- | ------------- | ----------------- |
    | M5.XLARGE | E2-STANDARD-4 | `STANDARD_D4D_V5` |

    * `<alert-email>`: Set an email to receive alerts from Astronomer support about your Deployments. See [Set up Astro alerts](/docs/astro/alerts).

    After you finish entering these values, your `config.yaml` file should look something like the following:

    ```yaml wrap theme={null}
    deployment:
      environment_variables:
        - is_secret: true
          key: MY_VARIABLE_KEY
          value: MY_VARIABLE_VALUE
        - is_secret: false
          key: MY_VARIABLE_KEY_2
          value: MY_VARIABLE_VALUE_2
      configuration:
        name: My Deployment
        description: The Deployment I'm using for migration.
        runtime_version: 6.3.0
        dag_deploy_enabled: false
        scheduler_au: 5
        scheduler_count: 2
        cluster_name: My Cluster
        workspace_name: My Workspace
      worker_queues:
        - name: default
          max_worker_count: 10
          min_worker_count: 1
          worker_concurrency: 16
          worker_type: E2-STANDARD-4
      alert_emails:
        - myalertemail@cosmicenergy.org
    ```

    4. Run the following command to push your configuration to Astro and create your Deployment:

       ```sh wrap theme={null}
       astro deployment create --deployment-file config.yaml
       ```
  </Tab>
</Tabs>

## Step 4: Use Starship to Migrate Airflow Connections and Variables

You might have defined Airflow connections and variables in the following places on your source Airflow environment:

* The Airflow UI (stored in the Airflow metadata database).
* Environment variables
* A secrets backend.

If you defined your Airflow variables and connections in the Airflow UI, you can migrate those to Astro with [Starship](https://astronomer.github.io/starship/). You can check which resources will be migrated by going to **Admin** > **Variables** and **Admin** > **Connections** in the Airflow UI to find your source Airflow environment information.

<Warning>Some environment variables or Airflow Settings, like global environment variable values, can't be migrated to Astro. See [Global environment variables](/docs/astro/platform-variables) for a list of variables that you can't migrate to Astro.</Warning>

<Tabs>
  <Tab title="Starship Plugin">
    1. Log in to Astro. In the Astro UI, open the Deployment you're migrating to.
    2. Click **Open Airflow** to open the Airflow UI for the Deployment. Copy the URL for the home page. It should look similar to `https://<your-organization>.astronomer.run/<id>/home`.
    3. Create a Deployment API token for the Deployment. The token should minimally have permissions to update the Deployment and deploy code. Copy this token. See [Create and manage Deployment API tokens](/docs/astro/deployment-api-tokens) for additional setup steps.
    4. Open the Airflow UI for your source Airflow environment, then go to **Astronomer** > **Migration Tool 🚀**.
    5. Ensure that the **Astronomer Product** toggle is set to **Astro**.
    6. In the **Airflow URL** section, fill in the fields so that the complete URL on the page matches the URL of the Airflow UI for the Deployment you're migrating to.
    7. Specify your API token in the **Token** field. Starship will confirm that it has access to your Deployment.
    8. Click **Connections**. In the table that appears, click **Migrate** for each connection that you want to migrate to Astro. After the migration is complete, the status **Migrated ✅** appears.
    9. Click **Pools**. In the table that appears, click **Migrate** for each connection that you want to migrate to Astro. After the migration is complete, the status **Migrated ✅** appears.
    10. Click **Variables**. In the table that appears, click **Migrate** for each variable that you want to migrate to Astro. After the migration is complete, the status **Migrated ✅** appears.
    11. Click **Environment variables**. In the table that appears, check the box for each environment variable that you want to migrate to Astro, then click **Migrate**. After the migration is complete, the status **Migrated ✅** appears.
    12. Click **DAG History**. In the table that appears, check the box for each DAG whose history you want to migrate to Astro, then click **Migrate**. After the migration is complete, the status **Migrated ✅** appears.
  </Tab>

  <Tab title="Starship operator">
    Refer to the [Configuration](https://astronomer.github.io/starship/operator/#usage) detailed instructions on using the operator.

    1. Log in to Astro. In the Astro UI, open the Deployment you're migrating to.

    2. Click **Open Airflow** to open the Airflow UI for the Deployment. Copy the URL for the home page. It should look similar to `https://<your-organization>.astronomer.run/<id>/home`.

    3. Create a Deployment API token for the Deployment. The token should minimally have permissions to update the Deployment and deploy code. Copy this token. See [Create and manage Deployment API tokens](/docs/astro/deployment-api-tokens) for additional setup steps.

    4. Add the following DAG to your source Airflow environment:

       ```python wrap theme={null}
       from airflow.models.dag import DAG
       from astronomer.starship.operators import AstroMigrationOperator
       from datetime import datetime

       with DAG(
         dag_id="astronomer_migration_dag",
         start_date=datetime(1970, 1, 1),
         schedule_interval=None,
       ) as dag:
         AstroMigrationOperator(
             task_id='export_meta',
             deployment_url='{{ dag_run.conf["deployment_url"] }}',
             token='{{ dag_run.conf["astro_token"] }}',
         )
       ```

    5. Deploy this DAG to your source Airflow environment.

    6. Once the DAG is available in the Airflow UI, click **Trigger DAG**, then click **Trigger DAG w/ config**.

    7. In **Configuration JSON**, add the following configuration:

       ```json wrap theme={null}
       {
           "deployment_url": "<your-deployment-url>",
           "astro_token": "<your-astro-token>"
       }
       ```

    8. Replace the following placeholder values:

       * `<your-deployment-url>`: The Deployment URL you copied in Step 2.
       * `<your-astro-token>`: The token you copied in Step 3.

    9. Click **Trigger**. After the DAG successfully runs, all connections, variables, and environment variables that are available from the Airflow UI are migrated to Astronomer.
  </Tab>
</Tabs>

## Step 5: Create an Astro project

1. Create a new directory for your Astro project:

   ```sh wrap theme={null}
   mkdir <your-astro-project-name>
   ```

2. Open the directory:

   ```sh wrap theme={null}
   cd <your-astro-project-name>
   ```

3. Run the following Astro CLI command to initialize an Astro project in the directory:

   ```sh wrap theme={null}
   astro dev init
   ```

   This command generates a set of files that will build into a Docker image that you can both run on your local machine and deploy to Astro.

4. Add the following line to your Astro project `requirements.txt` file:

   ```sh wrap theme={null}
   astronomer-starship
   ```

   When you deploy your code, this line installs the Starship migration tool on your Deployment so that you can migrate Airflow resources from your source environment to Astro.

5. (Optional) Run the following command to initialize a new git repository for your Astro project:

   ```sh wrap theme={null}
   git init
   ```

## Step 6: Migrate project code and dependencies to your Astro project

<Tabs>
  <Tab title="Manual">
    1. Open your Astro project Dockerfile. Update the Runtime version in first line to the version you selected for your Deployment in [Step 3](#step-3-create-an-astro-deployment). For example, if your Runtime version was 6.3.0, your Dockerfile would look like the following:

       ```dockerfile title="Dockerfile" wrap theme={null}
       FROM quay.io/astronomer/astro-runtime:6.3.0
       ```

       The `Dockerfile` defines the environment that all your Airflow components run in. You can modify it to make certain resources available to your Airflow environment like certificates or keys. For this migration, you only need to update your Runtime version.

    2. Open your Astro project `requirements.txt` file and add all Python packages from your source Airflow environment's `requirements.txt` file. See [AWS documentation](https://docs.aws.amazon.com/mwaa/latest/userguide/working-dags-dependencies.html) to find this file in your S3 bucket.

    <Warning>To avoid breaking dependency upgrades, Astronomer recommends pinning your packages to the versions running in your source Airflow environment. For example, if you're running `apache-airflow-providers-snowflake` version 3.3.0 on MWAA, you would add `apache-airflow-providers-snowflake==3.3.0` to your Astro `requirements.txt` file.</Warning>

    3. Open your Astro project `dags` folder. Add your dag files from either your source control platform or S3.

    4. If you used the [`plugins` folder](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html) in your MWAA project, copy the contents of this folder from your source control platform or S3 to the `/plugins` folder of your Astro project.

    After you confirm that your Astro project has all necessary dependencies, deploy the project to your Astro Deployment.

    1. Run the following command to authenticate to Astro:

       ```sh wrap theme={null}
       astro login
       ```

    2. Run the following command to deploy your project

       ```sh wrap theme={null}
       astro deploy
       ```

       This command returns a list of Deployments available in your Workspace and prompts you to pick one.
  </Tab>

  <Tab title="AWS CLI">
    1. Open your Astro project Dockerfile. Update the Runtime version in first line to the version you selected for your Deployment in [Step 3](#step-3-create-an-astro-deployment). For example, if your Runtime version was 6.3.0, your Dockerfile would look like the following:

       ```dockerfile title="Dockerfile" wrap theme={null}
       FROM quay.io/astronomer/astro-runtime:6.3.0
       ```

       The `Dockerfile` defines the environment that all your Airflow components run in. You can modify it to make certain resources available to your Airflow environment like certificates or keys. For this migration, you only need to update your Runtime version.

    2. Run the following command to copy your PyPI packages from AWS to your Astro project `requirements.txt` file:

       ```sh wrap theme={null}
       aws s3 cp s3://[BUCKET]/requirements.txt requirements.txt
       ```

       See [AWS documentation](https://docs.aws.amazon.com/mwaa/latest/userguide/working-dags-dependencies.html) to find this file in your S3 bucket. Review the output in your Astro project `requirements.txt` file after running this command to ensure that all packages were imported on their own line of text.

    3. Run the following command to copy your PyPI packages from AWS to your Astro project `requirements.txt` file:

       ```sh wrap theme={null}
       aws s3 cp --recursive s3://[BUCKET]/dags dags
       ```

       Review your dag files in `dags` after running this command to ensure that all dags were successfully exported.

       <Warning>
         To avoid breaking dependency upgrades, Astronomer recommends pinning your packages to the versions running in your source Airflow environment. For example, if you're running `apache-airflow-providers-snowflake` version 3.3.0 on MWAA, you would add `apache-airflow-providers-snowflake==3.3.0` to your Astro `requirements.txt` file.
       </Warning>

    4. If you used the [`plugins` folder](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html) in your MWAA project, run the following command to directly copy your PyPI packages from AWS to your Astro project `requirements.txt` file:

       ```sh wrap theme={null}
       aws s3 cp --recursive s3://[BUCKET]/plugins.zip plugins.zip
       unzip plugins.zip
       ```

       Review the contents of your Astro project `plugins` folder after running this command to ensure that all files were successfully exported.

    After you confirm that your Astro project has all necessary dependencies, deploy the project to your Astro Deployment.

    1. Run the following command to authenticate to Astro:

       ```sh wrap theme={null}
       astro login
       ```

    2. Run the following command to deploy your project

       ```sh wrap theme={null}
       astro deploy
       ```

       This command returns a list of Deployments available in your Workspace and prompts you to pick one.
  </Tab>
</Tabs>

## Step 7: Configure additional data pipeline infrastructure

The core migration of your project is now complete. Read the following to decide whether you need to set up any additional infrastructure on Astro before you cut over your dags.

### Set up CI/CD

If you used CI/CD to deploy code to your source Airflow environment, read the following documentation to learn about setting up a similar CI/CD pipeline for your Astro project:

* [Set-Up CI/CD](/docs/astro/set-up-ci-cd)
* [CI/CD templates](/docs/astro/ci-cd-templates/template-overview)

Similarly to MWAA, you can deploy dags to Astro directly from an S3 bucket. See [Deploy dags from an AWS S3 bucket to Astro using AWS Lambda](/docs/astro/ci-cd-templates/aws-s3).

### Set up a secrets backend

If you currently store Airflow variables or connections in a secrets backend, you also need to integrate your secrets backend with Astro to access those objects from your migrated dags. See [Configure a Secrets Backend](/docs/astro/secrets-backend) for setup steps.

## Step 8: Test locally and check for import errors

Depending on how thoroughly you want to test your Airflow environment, you can test your project locally before deploying to Astro.

* In your Astro project directory, run `astro dev parse` to check for any parsing errors in your dags.
* Run `astro run <dag-id>` to test a specific dag. This command compiles your dag and runs it in a single Airflow worker container based on your Astro project configurations.
* Run `astro dev start` to start a complete Airflow environment on your local machine. After your project starts up, you can access the Airflow UI at `localhost:8080`. See [Troubleshoot your local Airflow environment](/docs/cli/v1.43/run-airflow-locally).

<Info>
  Your migrated Airflow variables and connections are not available locally. You must deploy your project to Astro to test these Airflow objects.
</Info>

## Step 9: Deploy to Astro

1. Run the following command to authenticate to Astro:

   ```sh wrap theme={null}
   astro login
   ```

2. Run the following command to deploy your project

   ```sh wrap theme={null}
   astro deploy
   ```

   This command returns a list of Deployments available in your Workspace and prompts you to pick one.

3. In the Astro UI, open your Deployment and click **Open Airflow**. Confirm that you can see your deployed DAGs in the Airflow UI.

## Step 10: Cut over from your source Airflow environment to Astro

After you successfully deploy your code to Astro, you need to migrate your workloads from your source Airflow environment to Astro on a DAG-by-DAG basis. Depending on how your workloads are set up, Astronomer recommends letting DAG owners determine the order to migrate and test DAGs.

You can complete the following steps in the few days or weeks following your migration set up. Provide updates to your Astronomer Data Engineer as they continue to assist you through the process and any solve any difficulties that arise.

Continue to validate and move your DAGs until you have fully cut over your source Airflow instance. After you finish migrating from your source Airflow environment, repeat the complete migration process for any other Airflow instances in your source Airflow environment.

#### Confirm connections and variables

In the Airflow UI for your Deployment, [test all connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#testing-connections) that you migrated from your source Airflow environment.

Additionally, check Airflow variable values in **Admin** > **Variables**.

#### Test and validate DAGs in Astro

To create a strategy for testing DAGs, determine which DAGs need the most care when running and testing them.

If your DAG workflow is idempotent and can run twice or more without negative effects, you can run and test these DAGs with minimal risk. If your DAG workflow is non-idempotent and can become invalid when you rerun it, you should test the DAG with more caution and downtime.

#### Cut over DAGs to Astro using Starship

Starship includes features for simultaneously pausing DAGs in your source Airflow environment and starting them on Astro. This allows you to cut over your production workflows without downtime.

<Tabs>
  <Tab title="Starship Plugin">
    For each DAG in your Astro Deployment:

    1. Confirm that the DAG ID in your Deployment is the same as the DAG ID in your source Airflow environment.

    2. In the Airflow UI for your source Airflow environment, go to **Astronomer** > **Migration Tool 🚀**.

    3. Click **DAGs cutover**. In the table that appears, click the Pause icon in the **Local** column for the DAG you're cutting over.

    4. Click the Start icon in the **Remote** column for the DAG you're cutting over.

    5. After completing this cutover, the Start and Pause icons switch. If there's an issue after cutting over, click the **Remote** pause button and then the **Local** start button to move your workflow back to your source Airflow environment.
  </Tab>

  <Tab title="Starship Operator">
    The Starship operator does not contain cut-over functionality.

    To cut over a DAG, pause the DAG in the source Airflow and unpause the DAG in Astro. Keep both Airflow environments open as you test and ensure that the cutover was successful.
  </Tab>
</Tabs>

### Optimize Deployment resource usage

#### Review DAG development features

Astro includes several features that enhance the Apache Airflow development experience, from DAG writing to testing. To make the most of these features, you might want to make adjustments to your existing DAG development workflows.

As you get started on Astro, review the list of features and changes that Astro brings to the Airflow development experience and consider how you want to implement these details in your development experience. See [Write and run DAGs on Astro](/docs/astro/dags-overview).

#### Monitor analytics

As you cut over DAGs, view [Deployment metrics](/docs/astro/deployment-metrics) to get a sense of how many resources your Deployment is using. Use this information to adjust your worker queues and resource usage accordingly, or to tell when a DAG isn't running as expected.

#### Modify instance types or use worker queues

If your current worker type doesn't have the right amount of resources for your workflows, see [Deployment settings](/docs/astro/deployment-settings) to learn about configuring worker types on your Deployments.

You can additionally configure [worker queues](/docs/astro/configure-worker-queues) to assign each of your tasks to different worker instance types. View your [Deployment metrics](/docs/astro/deployment-metrics) to help you determine what changes are required.

#### Enable DAG-only deploys

Deploying to Astro with DAG-only deploys enabled can make deploys faster in cases where you've only modified your `dags` directory. To enable the DAG-only deploy feature, see [Deploy DAGs only](/docs/astro/deploy-dags).


# Review code with Otto
Source: https://astronomer.io/docs/astro/otto-code-review

Use Otto to review pull requests and merge requests automatically with the Otto review action for GitHub and GitLab.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

Otto reviews your Airflow code automatically when you open a pull request or merge request. The Otto review action runs Otto with a dedicated reviewer persona, posts inline review comments where the code has issues, and offers commit suggestions where a concrete fix is available.

The review runs Otto with `--persona reviewer`. The persona bundles the Astro and Airflow review prompt, a read-only tool allowlist, plan-mode permissions, and a structured verdict schema, so Otto returns review results tuned for Airflow projects rather than generic code review. See [Personas](/docs/cli/v1.43/astro-otto#personas).

The action supports both GitHub and GitLab:

* **GitHub**: A composite GitHub Action that runs on `pull_request` and posts the review to the pull request.
* **GitLab**: A CI/CD template and Docker image that run on merge request events and post the review to the merge request.

## What Otto reviews

Otto reviews changes against three layers of knowledge, so it catches issues that generic code review misses:

* **Astronomer compatibility knowledge**: Version-specific operator behavior, provider compatibility, deprecated patterns, and failure signatures drawn from the Astronomer compatibility knowledge base. This is the same knowledge base that powers [Airflow upgrades with Otto](/docs/astro/otto-upgrades).
* **Airflow community standards**: Current Dag patterns, testing approaches, and structure recommendations from the Apache Airflow community.
* **Your team's conventions**: Standards, retry policies, approved operators, and patterns captured in [Otto Memory](/docs/astro/otto-memory). Otto surfaces deviations from your conventions before they reach a human reviewer.

## How it works

On each run, the action:

1. Checks out the changed code and gathers the pull request or merge request metadata, the diff, and the prior review conversation.
2. Runs `astro otto --mode json --persona reviewer` against the gathered context.
3. Posts the result back to the pull request or merge request.

The reviewer persona returns a structured verdict with these fields:

* **`verdict`**: `approve`, `comment`, or `request_changes`.
* **`summary`**: A one-sentence summary of the change.
* **`reasoning`**: One to three sentences on the change as a whole.
* **`comments`**: Line-anchored findings, each with a severity of `high`, `medium`, or `low` and an optional commit suggestion.

On re-runs, the action edits its existing comments in place rather than re-posting. It keeps one sticky summary comment, updates or leaves its prior inline comments instead of duplicating them, and only re-posts the merge-gating review when the verdict changes.

## Prerequisites

* An Astro [Organization API token](/docs/astro/organization-api-tokens). The token only needs the Organization member permission.
* A GitHub repository or GitLab project that contains an Airflow project.

## Set up the review action

<Tabs>
  <Tab title="GitHub">
    On GitHub, the review runs as the [Otto Review](https://github.com/marketplace/actions/otto-review) action from the GitHub Marketplace. Add it to a workflow that runs on pull requests. To get the latest version, click **Use latest version** on the Marketplace listing.

    <Steps>
      <Step title="Add the workflow">
        In your repository, create a workflow file at `.github/workflows/otto-review.yml` with the following content:

        ```yaml title=".github/workflows/otto-review.yml" wrap theme={null}
        name: Astronomer CI - Review Code
        on:
          pull_request:
            branches:
              - main

        permissions:
          contents: read
          pull-requests: write

        env:
          ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
          ASTRO_ORGANIZATION: ${{ secrets.ASTRO_ORGANIZATION }}

        jobs:
          review:
            runs-on: ubuntu-latest
            steps:
              - name: Otto Review
                uses: astronomer/otto-review-action@v0.2.0
        ```
      </Step>

      <Step title="Add the required secrets">
        In your repository, go to **Settings** > **Secrets and variables** > **Actions** and add the following repository secrets:

        * `ASTRO_API_TOKEN`: Your Astro Organization API token.
        * `ASTRO_ORGANIZATION`: Your Astronomer Organization ID.
      </Step>

      <Step title="Open a pull request">
        Open a pull request against the `main` branch. The action runs Otto, posts a sticky summary comment, and adds inline comments where it finds issues.
      </Step>
    </Steps>
  </Tab>

  <Tab title="GitLab">
    GitLab has no marketplace for the action. Copy the CI/CD template into your project, or point your agent to the [otto-review-action repository](https://github.com/astronomer/otto-review-action).

    <Steps>
      <Step title="Add the CI/CD job">
        In your project's `.gitlab-ci.yml`, add the `otto_review` job:

        ```yaml title=".gitlab-ci.yml" wrap theme={null}
        otto_review:
          image: ghcr.io/astronomer/otto-review:v0
          rules:
            - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
          variables:
            OTTO_MAX_DIFF_LINES: "50000"
            OTTO_DRY_RUN: "false"
          script:
            - bash /opt/otto-review/gitlab/run-review.sh
        ```

        The image bakes the Astro CLI and the review scripts, so the job needs no repository clone and no per-run install. Pin the image tag to `:v0` to track the latest `v0.x` release, or use `:vX.Y.Z` to pin an exact version.
      </Step>

      <Step title="Add the required variables">
        In your project, go to **Settings** > **CI/CD** > **Variables** and add the following masked variables:

        * `ASTRO_API_TOKEN`: Your Astro Organization API token.
        * `ASTRO_ORGANIZATION`: Your Astronomer Organization ID.
        * `GITLAB_TOKEN`: A personal access token or project access token with the `api` scope. The `CI_JOB_TOKEN` isn't sufficient for the notes and discussions API.
      </Step>

      <Step title="Open a merge request">
        Open a merge request. The job runs Otto, posts a sticky summary note, and adds inline comments where it finds issues.
      </Step>
    </Steps>

    <Note>
      On GitLab, the action posts the verdict through the sticky summary and inline notes. It never approves or blocks the merge request, because GitLab has no native request-changes review event.
    </Note>
  </Tab>
</Tabs>

## Run a review on demand

Code review also works outside a CI/CD pipeline. Run the reviewer persona from your terminal to review changes before you push, or to audit existing pipelines that haven't been through a structured review:

```sh wrap theme={null}
astro otto --persona reviewer "review the Dags in this project"
```

See [`astro otto`](/docs/cli/v1.43/astro-otto#personas) for the full persona reference.

## Configure the review

Both platforms accept settings that change how the review runs. On GitHub, pass them as action inputs under `with:`. On GitLab, set them as CI/CD variables. The most common settings are:

| GitHub input     | GitLab variable       | Description                                                                                        |
| ---------------- | --------------------- | -------------------------------------------------------------------------------------------------- |
| `model`          | `OTTO_MODEL`          | Override the model Otto uses. The default uses the model that the reviewer persona's tier maps to. |
| `allowed-tools`  | `OTTO_ALLOWED_TOOLS`  | Override the reviewer persona's read-only tool allowlist (`read`, `grep`, `find`, `ls`, `bash`).   |
| `max-diff-lines` | `OTTO_MAX_DIFF_LINES` | Truncate diffs longer than this value. The default is `50000`.                                     |
| `dry-run`        | `OTTO_DRY_RUN`        | When `true`, post the summary and inline comments but no merge-gating review.                      |

For the complete list of inputs, variables, and outputs, see the [otto-review-action repository](https://github.com/astronomer/otto-review-action).

## Related

* [Otto overview](/docs/astro/otto-overview)
* [Investigate with Otto](/docs/astro/otto-investigate)
* [`astro otto`](/docs/cli/v1.43/astro-otto)
* [Deploy code with the Astro GitHub integration](/docs/astro/deploy-github-integration)
* [Skills](/docs/astro/otto-skills)


# Otto extensions
Source: https://astronomer.io/docs/astro/otto-extensions

Enable and disable Otto's bundled extensions, which add optional tools, slash commands, and behaviors to every session.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

Extensions are opt-in capabilities that ship inside the Otto binary and add tools, slash commands, or background behaviors to a session. Some are on by default because they catch common mistakes or add useful slash commands; others are off by default because they spawn subprocesses, consume credits, or materially change behavior.

## Bundled extensions

| Name             | Default | What it does                                                                                                                                                                                                                    |
| ---------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ask-user`       | on      | Registers the `ask_user` tool so Otto can ask 1-4 focused questions in a single batch rather than many one-at-a-time prompts.                                                                                                   |
| `dag-validation` | on      | After any edit or write to `dags/*.py`, runs `af dags errors` and surfaces import errors so Otto self-corrects in the same turn. Only fires when a local Airflow is reachable.                                                  |
| `memory`         | on      | Registers `/bootstrap` (seed memories from git and GitHub history) and `/remember` (save learnings from the current conversation).                                                                                              |
| `skills`         | on      | Provides the hosted-skill surface: the `skill` tool, the `/skills` picker, and `/skill:<name>` commands for hosted skills. Disabling removes hosted skills entirely; local skills still work. See [Skills](/docs/astro/otto-skills). |
| `spinner-verbs`  | on      | Replaces the default loader with space-themed verbs and an elapsed-time counter. Cosmetic only.                                                                                                                                 |
| `subagent`       | off     | Registers a `subagent` tool that delegates tasks to isolated `otto` subprocesses. Supports single (`task`) or parallel (`tasks: [...]`) delegation, with `fast` and `deep` model tiers.                                         |

## Enable or disable an extension

Four layers control whether an extension is on, merged in order (highest priority first):

| Layer                | Where                                                           | Example                                              |
| -------------------- | --------------------------------------------------------------- | ---------------------------------------------------- |
| CLI flag             | `--extension <name>`, `--no-extension <name>` (repeatable)      | `astro otto --no-extension dag-validation`           |
| Environment variable | `OTTO_EXTENSIONS`, `OTTO_DISABLED_EXTENSIONS` (comma-separated) | `OTTO_DISABLED_EXTENSIONS=dag-validation astro otto` |
| Project settings     | `.astro/otto/extensions.json`                                   | `{ "dag-validation": false }`                        |
| User settings        | `~/.astro/otto/extensions.json`                                 | `{ "memory": false }`                                |

Within a layer, an explicit disable beats an explicit enable. For example, `--extension foo --no-extension foo` disables `foo`.

## Manage extensions interactively

Run `/extensions` inside an Otto session to open an interactive editor.

The main picker shows each bundled extension with its current state and a `*` marker when a CLI flag or environment variable is overriding the settings files. Select an extension to open a per-extension editor with two rows: a User scope row and a Project scope row. Each row has a three-state checkbox:

* `[x]` on — writes `enabled: true` to that settings file.
* `[ ]` off — writes `enabled: false` to that settings file.
* `[~]` unset — removes the `enabled` field so lower layers win.

Press `Enter` or `Space` to cycle the focused row. The title bar updates to show the predicted next-session state and which layer is winning. Press `Escape` to save and close.

To apply the change:

* Run `/reload` to re-run the extension list against the fresh settings while keeping your conversation history.
* Run `/new` to start a fresh session.
* Restart Otto for a full reset.

## Configure an extension

The settings file accepts either a boolean shorthand or an object with `enabled` plus any extension-specific config. Both forms can coexist in the same file:

```json title="extensions.json" wrap theme={null}
{
  "memory": false,
  "dag-validation": {
    "enabled": true,
    "timeout": 60000
  },
  "subagent": {
    "enabled": true,
    "tiers": { "fast": "gpt-5.4-nano", "deep": "gpt-5.4" }
  }
}
```

Project settings shadow user settings per extension (whole-entry replace, not a deep merge).

### Configurable fields

| Extension        | Field        | Type   | Default        | Description                                                                                         |
| ---------------- | ------------ | ------ | -------------- | --------------------------------------------------------------------------------------------------- |
| `dag-validation` | `timeout`    | number | `30000`        | Timeout in milliseconds for `af dags errors`.                                                       |
| `subagent`       | `tiers.fast` | string | `gpt-5.4-nano` | Model used when a subagent call uses `tier: "fast"`.                                                |
| `subagent`       | `tiers.deep` | string | parent's model | Model used when a subagent call uses `tier: "deep"`. Omit to inherit the main conversation's model. |

Extensions that don't appear in the table take no configuration — they're either on or off.


# Investigate with Otto
Source: https://astronomer.io/docs/astro/otto-investigate

Use Otto to investigate Dag failures on Astro with Airflow, infrastructure, and observability context from the Astro UI, Astro alerts, or the Astro API.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

Otto investigates Dag failures on Astro using proprietary Airflow, Astro, and Observe context that no general-purpose agent has access to. Each investigation produces a structured diagnosis with a root cause type, severity, suggested fix, and a checklist of Dag- and task-level checks.

This page covers Otto investigations that run in the Astro control plane and are triggered from the Astro UI, Astro alerts, or the Astro API. To investigate a Dag failure interactively from the terminal, ask Otto in the Astro CLI. See [Otto overview](/docs/astro/otto-overview).

Astronomer recommends wiring a Dag failure to an investigation automatically. The diagnosis is ready before your team starts triaging, and it can route the failure to the right escalation or remediation path.

## How to access investigations

You can trigger an investigation from any of these surfaces:

* **Astro UI**: From the **DAGs** list, click a Dag to open its detail page, then select a failed run to trigger an investigation.
* **Astro Observe homepage**: Click **Investigate** next to a Dag to investigate its most recent failed run, or open a specific run in the Dag's run history.
* **Catalog**: Open a Dag in the new Astro UI **Catalog** (the **Asset Catalog** in the legacy UI) and select a failed run from its run history.
* **Astro alerts**: When a Dag failure alert fires, open the failed Dag run from the alert notification to start an investigation. See [Set up Astro alerts](/docs/astro/alerts).
* **Astro API**: Trigger an investigation programmatically, poll for status, and read the result. Combine this with the **Dag Trigger** notification channel on an Astro alert to investigate critical Dag failures automatically.

## What Otto investigates

Otto draws on Airflow context (including Dag code and task logs), Astro context (including Deployment configuration, component logs, and recent deploys), and Observe context (including lineage, run history, and operational metrics).

Organizations that use Astro Observe get a richer investigation, since Otto has access to lineage, the Asset Catalog, and operational metrics in addition to Airflow and Astro context.

## Run an automatic investigation

To automatically investigate your critical Dags as soon as they fail, configure Astro alerts to call the [investigation API](/docs/astro/api/v-1-labs/observability/start-a-dag-failure-diagnosis-run) on Dag failure. Otto runs the investigation, and your investigation Dag handles the response.

1. Create a Dag that calls the [investigation API](/docs/astro/api/v-1-labs/observability/start-a-dag-failure-diagnosis-run) and routes the response to the right downstream action.
2. Deploy the Dag to Astro.
3. Set up a Dag failure [Astro alert](/docs/astro/alerts) that monitors your critical Dags.
4. Set the notification channel to **Dag Trigger** and select the Dag you deployed in step 2.
5. Optionally, tailor investigations by adding Otto investigation guidance at the Workspace or Deployment level. See [Customize Otto investigation guidance](#customize-otto-investigation-guidance).

When a critical Dag fails, the alert fires, the **Dag Trigger** channel calls your investigation Dag, and Otto returns the diagnosis for downstream handling. Common downstream actions include:

* **Notify**: Post the investigation results to Slack, email, or another notification channel.
* **Open a PR**: Apply the suggested fix as a pull request.
* **Open an incident**: Create an incident in ServiceNow, PagerDuty, or another incident management system.

### Example: Post the diagnosis to Slack

The following Dag accepts the **Dag Trigger** notification payload, calls the [investigation API](/docs/astro/api/v-1-labs/observability/start-a-dag-failure-diagnosis-run), reads the streamed diagnosis, and posts a Slack message with the root cause type, severity, summary, and suggested fix.

Before deploying, configure these as Airflow variables or environment variables on the Deployment:

* `astro_organization_id` or `ASTRO_ORGANIZATION_ID`: The ID of the Organization that owns the Deployment you investigate.
* `astro_deployment_id` or `ASTRO_DEPLOYMENT_ID`: The ID of the Deployment you investigate.
* `astro_api_token` or `ASTRO_API_TOKEN`: A [Deployment API token](/docs/astro/deployment-api-tokens) with permission to read the Deployment.
* `slack_webhook_url` or `SLACK_WEBHOOK_URL`: A Slack incoming webhook URL.
* `slack_channel` or `SLACK_CHANNEL`: Optional. The Slack channel to post to, if not the webhook's default.

```python expandable wrap theme={null}
"""Triggered by Astro DAG trigger notifications to investigate a failed DAG run and post the diagnosis to Slack.

Required settings. Each can be set as an environment variable OR an Airflow
Variable of the same name (set BEFORE enabling the alert):
- ASTRO_ORGANIZATION_ID   (your Astro organization ID)
- ASTRO_DEPLOYMENT_ID     (deployment to run diagnoses against)
- ASTRO_API_TOKEN         (Astro API token with labs/v1 access)
- SLACK_WEBHOOK_URL       (Slack incoming webhook URL)
- SLACK_CHANNEL           (optional; overrides the webhook's default channel)

Set these in the Astro UI under Deployment → Environment (Environment Variables
or Airflow Variables). Missing any required value will raise
`ValueError: Missing required setting ...` at task runtime.

Expected dag_run.conf keys (sent by the Astro Alert trigger):
- alertId              (Astro alert ID; surfaced in the Slack message)
- alertType            (DAG_FAILURE — other alert types are ignored)
- dagName              (target DAG being triggered — this DAG; not used, the failed source DAG comes from `message`)
- message              (free-form alert text; the failed source DAG ID is parsed from `"... for DAG <name>"`)
- airflowDagRunId      (the failed source DAG's run ID, e.g. `scheduled__2026-06-03T23:40:00+00:00`)
- logFailureSummaries  (dict keyed by failed task ID; if exactly one key, it's used as the failed task ID)

Example payload (DAG_FAILURE alert from `always_failing_dag`):
    {
      "alertId": "cmpyms67v000m01pxkfaqoe5t",
      "alertType": "DAG_FAILURE",
      "dagName": "alert_investigation_agent",
      "message": "DAG run failed for DAG always_failing_dag",
      "airflowDagRunId": "scheduled__2026-06-03T23:40:00+00:00",
      "logFailureSummaries": {"fail_on_purpose": "..."}
    }
"""

from __future__ import annotations

import json
import logging
import os
import re
from typing import Any

import requests
from airflow.sdk import Variable, dag, task
from pendulum import datetime

ASTRO_API_BASE = "https://api.astronomer.io/labs/v1"
# This demo investigates DAG-failure alerts. Other alert types are logged
# and skipped.
ALLOWED_ALERT_TYPES = {"DAG_FAILURE"}

log = logging.getLogger(__name__)


def _get_setting(key: str) -> str:
    # Resolve a setting from an environment variable or an Airflow Variable of
    # the same name. Env var wins if both are set.
    env_value = os.environ.get(key, "")
    if env_value:
        return env_value

    try:
        return Variable.get(key)
    except Exception:
        return ""


def _get_required_setting(key: str) -> str:
    value = _get_setting(key)
    if value:
        return value
    raise ValueError(f"Missing required setting: '{key}' (set it as an env var or Airflow Variable)")


def _extract_source_dag_id(message: str) -> str:
    # Astro Alert payloads don't include the failed source DAG ID as a
    # top-level key (`dagName` in conf is the *target* DAG being triggered,
    # i.e. this one). Parse it out of the DAG_FAILURE alert `message`:
    #   "DAG run failed for DAG <dag_id>"
    match = re.search(r"for DAG\s+['\"]?([a-zA-Z0-9_.-]+)['\"]?", message, re.IGNORECASE)
    if match:
        return match.group(1)

    raise ValueError(
        "Could not determine the failed source DAG ID from the alert message. "
        f"Expected a 'for DAG <name>' phrase; got message={message!r}"
    )


def _extract_run_id(conf: dict[str, Any]) -> str:
    value = conf.get("airflowDagRunId")
    if value:
        return str(value)

    raise ValueError("Missing required dag_run.conf key: airflowDagRunId")


def _extract_task_id(conf: dict[str, Any]) -> str | None:
    # DAG_FAILURE payloads include a `logFailureSummaries` dict keyed by the
    # failed task ID(s). If exactly one task failed, use it.
    summaries = conf.get("logFailureSummaries")
    if isinstance(summaries, dict) and len(summaries) == 1:
        return next(iter(summaries))

    return None


def _call_investigation_agent(
    organization_id: str,
    deployment_id: str,
    api_token: str,
    dag_id: str,
    run_id: str,
    task_id: str | None,
) -> dict[str, Any]:
    auth_headers = {"Authorization": f"Bearer {api_token}"}
    deployment_base = (
        f"{ASTRO_API_BASE}/organizations/{organization_id}"
        f"/observability/deployments/{deployment_id}/dag-failure-diagnosis/runs"
    )

    start_response = requests.post(
        deployment_base,
        headers={**auth_headers, "Content-Type": "application/json"},
        json={
            "dagId": dag_id,
            "runId": run_id,
            **({"taskId": task_id} if task_id else {}),
        },
        timeout=30,
    )
    start_response.raise_for_status()
    diagnosis_run_id = start_response.json()["runId"]

    response = requests.get(
        f"{deployment_base}/{diagnosis_run_id}/events",
        headers={**auth_headers, "Accept": "text/event-stream"},
        stream=True,
        timeout=(30, 300),
    )
    response.raise_for_status()

    event_type: str | None = None
    data_lines: list[str] = []
    text_chunks: list[str] = []
    diagnosis: dict[str, Any] | None = None

    def flush_event() -> None:
        nonlocal event_type, data_lines, diagnosis
        if not event_type:
            data_lines = []
            return

        payload = "\n".join(data_lines)
        if event_type == "rca_diagnosis" and payload:
            diagnosis = json.loads(payload)
        elif event_type == "text_delta" and payload:
            text_chunks.append(json.loads(payload).get("text", ""))
        elif event_type == "error" and payload:
            raise RuntimeError(json.loads(payload).get("message", "Investigation Agent returned an error"))

        event_type = None
        data_lines = []

    for raw_line in response.iter_lines(decode_unicode=True):
        if raw_line is None:
            continue

        line = raw_line.rstrip("\r")
        if not line:
            flush_event()
            if diagnosis:
                break
            continue

        if line.startswith(":"):
            continue
        if line.startswith("event:"):
            event_type = line.split(":", 1)[1].strip()
            continue
        if line.startswith("data:"):
            data_lines.append(line.split(":", 1)[1].lstrip())

    flush_event()

    if diagnosis:
        return diagnosis
    if text_chunks:
        return {"title": f"Investigation for {dag_id}", "summary": "".join(text_chunks).strip()}

    raise RuntimeError("Investigation Agent returned no diagnosis payload")


def _format_slack_message(
    diagnosis: dict[str, Any],
    alert_id: str,
    alert_type: str,
    dag_id: str,
    run_id: str,
    task_id: str | None,
) -> dict[str, Any]:
    title = diagnosis.get("title") or f"Investigation for {dag_id}"
    summary = diagnosis.get("summary") or "No summary returned."
    suggested_fix = diagnosis.get("suggested_fix") or "No suggested fix returned."
    severity = diagnosis.get("severity") or "UNKNOWN"
    priority = diagnosis.get("priority") or "UNKNOWN"
    root_cause_type = diagnosis.get("root_cause_type") or "UNKNOWN"
    confidence = diagnosis.get("confidence")

    fields = [
        {"type": "mrkdwn", "text": f"*Alert ID*\n`{alert_id}`"},
        {"type": "mrkdwn", "text": f"*Alert Type*\n`{alert_type}`"},
        {"type": "mrkdwn", "text": f"*DAG*\n`{dag_id}`"},
        {"type": "mrkdwn", "text": f"*Run ID*\n`{run_id}`"},
        {"type": "mrkdwn", "text": f"*Severity*\n`{severity}`"},
        {"type": "mrkdwn", "text": f"*Root Cause Type*\n`{root_cause_type}`"},
    ]
    if task_id:
        fields.append({"type": "mrkdwn", "text": f"*Task ID*\n`{task_id}`"})
    if confidence is not None:
        fields.append({"type": "mrkdwn", "text": f"*Confidence*\n`{confidence}`"})
    if priority != "UNKNOWN":
        fields.append({"type": "mrkdwn", "text": f"*Priority*\n`{priority}`"})

    blocks: list[dict[str, Any]] = [
        {"type": "header", "text": {"type": "plain_text", "text": title[:150]}},
        {"type": "section", "fields": fields[:10]},
        {"type": "section", "text": {"type": "mrkdwn", "text": f"*Summary*\n{summary[:2900]}"}},
        {"type": "section", "text": {"type": "mrkdwn", "text": f"*Suggested Fix*\n{suggested_fix[:2900]}"}},
    ]

    text = (
        f"[{severity}] {title}\n"
        f"Root cause type: {root_cause_type}\n"
        f"Summary: {summary}\n\n"
        f"Suggested fix: {suggested_fix}"
    )

    return {"text": text[:4000], "blocks": blocks}


def _post_to_slack(webhook_url: str, channel: str | None, payload: dict[str, Any]) -> None:
    body = dict(payload)
    if channel:
        body["channel"] = channel

    response = requests.post(webhook_url, json=body, timeout=30)
    response.raise_for_status()


@dag(
    dag_id="alert_investigation_agent",
    start_date=datetime(2025, 1, 1),
    schedule=None,
    catchup=False,
    tags=["alerts", "investigation-agent", "slack"],
    default_args={"owner": "Astro", "retries": 0},
    doc_md=__doc__,
)
def alert_investigation_agent():
    @task
    def handle_alert(**context) -> None:
        dag_run = context.get("dag_run")
        conf = dict(dag_run.conf or {}) if dag_run else {}

        alert_id = str(conf.get("alertId", "unknown"))
        alert_type = str(conf.get("alertType", "unknown"))
        message = str(conf.get("message", ""))

        if alert_type not in ALLOWED_ALERT_TYPES:
            log.info(
                "Ignoring alert: alertType %r is not a DAG failure. "
                "This demo only investigates DAG-failure alerts.",
                alert_type,
            )
            return

        dag_id = _extract_source_dag_id(message)
        run_id = _extract_run_id(conf)
        task_id = _extract_task_id(conf)

        organization_id = _get_required_setting("ASTRO_ORGANIZATION_ID")
        deployment_id = _get_required_setting("ASTRO_DEPLOYMENT_ID")
        api_token = _get_required_setting("ASTRO_API_TOKEN")
        slack_webhook_url = _get_required_setting("SLACK_WEBHOOK_URL")
        slack_channel = _get_setting("SLACK_CHANNEL") or None

        diagnosis = _call_investigation_agent(
            organization_id=organization_id,
            deployment_id=deployment_id,
            api_token=api_token,
            dag_id=dag_id,
            run_id=run_id,
            task_id=task_id,
        )

        log.info(
            "Diagnosis completed for %s run %s: title=%s severity=%s root_cause_type=%s summary=%s suggested_fix=%s",
            dag_id,
            run_id,
            diagnosis.get("title") or f"Investigation for {dag_id}",
            diagnosis.get("severity") or "UNKNOWN",
            diagnosis.get("root_cause_type") or "UNKNOWN",
            diagnosis.get("summary") or "No summary returned.",
            diagnosis.get("suggested_fix") or "No suggested fix returned.",
        )

        slack_payload = _format_slack_message(
            diagnosis=diagnosis,
            alert_id=alert_id,
            alert_type=alert_type,
            dag_id=dag_id,
            run_id=run_id,
            task_id=task_id,
        )
        _post_to_slack(slack_webhook_url, slack_channel, slack_payload)

    handle_alert()


alert_investigation_agent()
```

## Customize Otto investigation guidance

You can add Otto investigation guidance at the Workspace or Deployment level to tailor investigations to your environment. For example, you can instruct Otto to treat tasks that begin with `validate` as non-blocking, or to interpret specific log patterns in a particular way. Deployments inherit Workspace guidance by default and can override it.

See [Configure Otto investigation guidance for a Workspace](/docs/astro/manage-workspaces#configure-otto-investigation-guidance) and [Configure Otto investigation guidance for a Deployment](/docs/astro/deployment-settings#configure-otto-investigation-guidance).

## Related

* [Otto overview](/docs/astro/otto-overview)
* [Skills](/docs/astro/otto-skills)
* [Troubleshoot Dag failures in Astro Observe](/docs/astro/root-cause-analysis)
* [Set up Astro alerts](/docs/astro/alerts)


# Otto memory
Source: https://astronomer.io/docs/astro/otto-memory

Learn how Otto's memory system accumulates team-specific knowledge across sessions, including conventions, corrections, and environmental details.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

Otto's memory system transforms it from an agent that starts fresh every session into a knowledgeable team member that remembers your environment, your conventions, and lessons learned from past interactions. Memory is stored locally as Markdown files split between your project repository and your home directory.

## How memory works

Memory lives in two places: your project repository and your home directory.

```text wrap theme={null}
your-project/                  # Astro project root
└── .astro/
    └── memory/                # Shared project memory (committed, team-visible)
        ├── MEMORY.md
        └── conventions.md

~/                             # Your home directory
└── .astro/
    └── memory/
        ├── MEMORY.md          # Local user memory (all projects, this computer)
        ├── preferences.md
        └── <project-slug>/    # Local project memory (this project, this computer)
            ├── MEMORY.md
            └── notes.md
```

Each location serves a different purpose:

* **Shared project memory** (`.astro/memory/`): Conventions, architecture decisions, and environment details that should travel with the project. Committed to your repository so the whole team picks them up.
* **Local project memory** (`~/.astro/memory/<project-slug>/`): Your personal notes for one specific project. Lives in your home directory and isn't committed.
* **Local user memory** (`~/.astro/memory/`): Preferences and patterns that apply to every project you work on. Lives in your home directory and isn't committed.

At the start of every session, Otto loads `MEMORY.md` from each location. Other `.md` files are indexed by name so Otto knows they exist, but they're read on demand rather than loaded upfront.

<Note>
  In addition to `.astro/memory/`, Otto reads `AGENTS.md` and `CLAUDE.md` files for context. Otto loads them from `~/.astro/otto/AGENTS.md` or `~/.astro/otto/CLAUDE.md`, then walks up from the current working directory to `/`. When both files exist in the same folder, Otto prioritizes `AGENTS.md` over `CLAUDE.md`.
</Note>

## How memories get created

Otto creates memory in several ways:

### From the current conversation with `/remember`

Run `/remember` in an interactive session to have Otto review the conversation and generate memory files for the reusable learnings it identifies. Otto proposes the memories before writing them, and you approve or reject each one.

### From history with `/bootstrap`

Run `/bootstrap` to seed shared project memory from your git history and available GitHub PR history (when the GitHub CLI is installed and authenticated). This is useful when you're starting with Otto on an established project and want to capture existing team knowledge without building it up organically.

### Manually

Platform teams can add Markdown files directly to `.astro/memory/` to seed Otto with explicit constraints before organic memory accumulates. Upload files with conventions, approved operators, retry policies, and other standards, and add each file to `MEMORY.md` so Otto knows they exist.

For example:

```text wrap theme={null}
.astro/memory/
  MEMORY.md             # Index of all memory files
  conventions.md        # Naming conventions, preferred patterns
  approved-operators.md # Operators the team has approved for use
  retry-policy.md       # Standard retry configuration
```

### Autonomously during a session

Otto can also write memory on its own when it detects something worth retaining, without you running `/remember` or `/bootstrap`. This happens when Otto:

* Discovers a project convention not represented in the codebase.
* Learns important environment facts such as versions, connections, or team structure.
* Corrects a prior mistake and wants to avoid repeating it.

## Memory file format

Each memory file is a Markdown file with the following structure:

```markdown wrap theme={null}
# <Title>

### Source
user-interaction | debugging-session | customer-call | documentation | experiment | code-review

## Memory
<concise learning — the core knowledge>

## Context
<when and why this was discovered — optional>

## Evidence
<concrete instances — grows over time as the memory is reinforced>
```

The `MEMORY.md` file in each folder serves as an index. Otto reads it at the start of every session to understand what memory files are available, and adds entries for new memories as they're created.

## When Otto does not write memory

Otto avoids writing memory for:

* Information already in `README.md`, `CLAUDE.md`, or committed documentation.
* Ephemeral task state or in-progress work.
* Code patterns that are readable directly from source.

## Memory and team collaboration

Because shared project memory lives in `.astro/memory/` within your project repository, it follows your team's standard Git workflow:

1. Otto creates or updates a memory file during a session.
2. You review the change in your Git diff.
3. You commit and push the change.
4. Other team members pull the update.
5. In their next Otto session, the new memory is loaded automatically.

Local project memory and local user memory are scoped to you and aren't shared with the team.


# Models and regions
Source: https://astronomer.io/docs/astro/otto-models-and-regions

Reference for the models and hosting regions that Otto and the Astro IDE currently use through the Astronomer Gateway.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

Otto and the AI features in the [Astro IDE](/docs/astro/ide-overview) access models from OpenAI, Anthropic, and Google through the Astronomer Gateway. Astronomer hosts these models on Azure OpenAI Service and Google Cloud Vertex AI rather than calling the model providers directly.

This document is the source of truth for the specific models and hosting regions currently in use. Astronomer updates it as the available models change.

## Azure OpenAI Service

These models run in the East US 2 region.

| Model provider | Model               |
| -------------- | ------------------- |
| OpenAI         | `gpt-5.5`           |
| OpenAI         | `gpt-5.4`           |
| OpenAI         | `gpt-5.4-mini`      |
| OpenAI         | `gpt-5.4-nano`      |
| OpenAI         | `gpt-5.3-codex`     |
| Anthropic      | `claude-opus-4-8`   |
| Anthropic      | `claude-opus-4-7`   |
| Anthropic      | `claude-opus-4-6`   |
| Anthropic      | `claude-sonnet-4-6` |
| Anthropic      | `claude-haiku-4-5`  |

## Google Cloud Vertex AI

These models run with the location set to `global`, which routes each request to whichever regional resource is available.

| Model provider | Model                           |
| -------------- | ------------------------------- |
| Anthropic      | `claude-opus-4-7`               |
| Anthropic      | `claude-opus-4-6`               |
| Anthropic      | `claude-sonnet-4-6`             |
| Anthropic      | `claude-haiku-4-5`              |
| Google         | `google/gemini-3.1-pro-preview` |
| Google         | `google/gemini-3.5-flash`       |
| Google         | `google/gemini-3-flash-preview` |
| Google         | `google/gemini-2.5-pro`         |

## Choose a model

To select a model for an Otto session, run `/model` in an interactive session to browse what's available to you, or pass `--model <id>` to pin a model at launch. See [Customize Otto](/docs/astro/customize-otto#choose-a-model) and [`astro otto` model selection](/docs/cli/v1.43/astro-otto#model-selection).


# Otto overview
Source: https://astronomer.io/docs/astro/otto-overview

Otto is Astronomer's data engineering agent, purpose built for Apache Airflow and designed to get smarter the longer your team uses it.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

Otto, Astronomer's data engineering agent, is purpose built for the work data engineers do on Apache Airflow. It builds and debugs pipelines, investigates production failures, plans and executes upgrades and migrations, explores and profiles your data, and gets meaningfully better the longer your team uses it.

Otto is grounded in context that no general-purpose agent has access to. Public Airflow knowledge forms the baseline. On top of that sits a proprietary compatibility knowledge base maintained by Astronomer, built from eight years of running Airflow at enterprise scale. The third layer is your team's own accumulated memory: conventions, corrections, connection configs, and operational history that compound every session.

## Key capabilities

**Exploration**: Query your warehouse, trace lineage, and profile tables from your agent session. Otto answers business questions about your data, checks freshness, and surfaces upstream and downstream dependencies.

**Dag Authoring and Debugging**: Describe what you need in natural language. Otto writes the Dag, configures connections, triggers a run, and iterates until it works. When something breaks, it pulls task logs, reads the Dag source, inspects variables and connections, and edits the code to fix the failure.

**Investigate production failures**: When a Dag fails, ask Otto to diagnose it. Otto reads task logs, run history, connections, and Dag source, then works through the diagnosis interactively. Otto can also investigate Dag failures automatically from the Astro UI, Astro alerts, or the Astro API. See [Investigate with Otto](/docs/astro/otto-investigate).

**Code review**: Otto reviews pull requests and merge requests automatically. The Otto review action runs Otto with a dedicated reviewer persona, posts inline comments where the code has issues, and offers commit suggestions where a concrete fix is available. See [Review code with Otto](/docs/astro/otto-code-review).

**Airflow Upgrades**: Otto analyzes your Dag fleet against Astronomer's proprietary compatibility knowledge base, identifies breaking changes, proposes specific code fixes, and produces a prioritized upgrade plan.

**Migrations**: Otto helps you move workflows to Airflow on Astro from legacy and third-party orchestration systems.

Otto extends these capabilities through skills it loads on demand. See [Skills](/docs/astro/otto-skills).

## How to access Otto

Otto is available through the [`astro otto`](/docs/cli/v1.43/astro-otto) command in the Astro CLI and in the [Astro IDE](/docs/astro/ide-overview).

* **Astro CLI** — Run `astro otto` from any Astro project folder to launch the interactive terminal user interface (TUI). The command is pre-authenticated with your Astro credentials and pre-configured with skills and proprietary context. See the [quickstart](/docs/astro/get-started-otto).
* **Astro IDE** — The AI features in the [Astro IDE](/docs/astro/ide-overview) are powered by Otto. Code generation, Dag authoring, and project-aware suggestions use Otto's context and skills.

## How Otto is versioned

Otto ships as a standalone binary that versions independently of the Astro CLI. The first time you run [`astro otto`](/docs/cli/v1.43/astro-otto), the CLI downloads Otto to `~/.astro/bin/otto`. On subsequent launches, the CLI checks for newer releases and applies them automatically before launching the agent.

To opt out of automatic updates, set `otto.auto_update` to `false`. The CLI then prints an upgrade hint instead, and you apply updates manually with `astro otto update`. See [Configure the Astro CLI](/docs/cli/v1.43/configure-cli) for the config option.

You don't need to upgrade the Astro CLI to pick up new Otto features or fixes. Check your installed version with `astro otto version`.

## Next steps

* [Quickstart: Run your first Otto session](/docs/astro/get-started-otto)
* [Usage modes](/docs/cli/v1.43/astro-otto#usage-modes)
* [Investigate with Otto](/docs/astro/otto-investigate)
* [Review code with Otto](/docs/astro/otto-code-review)
* [Tools](/docs/astro/otto-tools)
* [Skills](/docs/astro/otto-skills)
* [Memory](/docs/astro/otto-memory)
* [Customize Otto](/docs/astro/customize-otto)


# Otto permissions
Source: https://astronomer.io/docs/astro/otto-permissions

Configure how Otto asks, allows, and denies tool calls, including file and command rules, permission modes, and built-in safety checks.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

Otto ships a first-party permission layer that gates every tool call. Each call resolves to **allow**, **ask** (prompt the user), or **deny** based on the active permission mode and your configured rules.

Out of the box, Otto asks before running destructive Astro and Airflow commands, prompts before writing to sensitive files like `.env` or `.ssh/*`, and restricts writes outside your project folder. You can extend or relax these behaviors through config files and permission modes.

## Permission modes

Otto runs in one of five modes. Cycle through modes interactively with `Ctrl+]` or the `/permissions` command.

| Mode                | Behavior                                                                                                                                                         |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `default`           | Normal behavior. Tools are allowed, denied, or prompted based on your rules.                                                                                     |
| `acceptEdits`       | Auto-allows `edit` and `write` inside the project. Other tools fall through to the normal rules. Safety checks still fire.                                       |
| `confirmEdits`      | Prompts before `edit`, `write`, and non-read-only `bash`. Allow rules can't bypass the prompt. Read tools and read-only `bash` fall through to the normal rules. |
| `plan`              | Blocks `edit` and `write` entirely. Restricts `bash` to a read-only allowlist (`ls`, `cat`, `git`, `rg`, `af`, `astro`, etc.).                                   |
| `bypassPermissions` | Allows everything except bypass-immune safety checks on sensitive files and out-of-project writes.                                                               |

Set the starting mode with the `--permission-mode <mode>` flag or the `OTTO_PERMISSION_MODE` environment variable.

## Where configuration lives

Otto reads permissions from three files, merged in order (later wins):

| Scope   | Path                                 | Purpose                                         |
| ------- | ------------------------------------ | ----------------------------------------------- |
| User    | `~/.astro/otto/permissions.json`     | Your personal rules across all projects         |
| Project | `.astro/otto/permissions.json`       | Team-wide rules committed to Git                |
| Local   | `.astro/otto/permissions.local.json` | Your local-only overrides (gitignore this file) |

Commit project-scope rules to your repository so every engineer on the team gets the same posture.

## Configuration file shape

```json expandable wrap theme={null}
{
  "enabled": true,
  "defaultMode": "default",
  "restrictToProjectDir": true,
  "rules": {
    "allow": ["Bash(ls:*)"],
    "deny": ["Bash(rm -rf /:*)"],
    "ask": ["Bash(git push:*)"]
  },
  "pathDeny": [
    {
      "id": "no-root-writes",
      "patterns": ["/etc/**", "/usr/**"],
      "tools": ["write", "edit", "bash"],
      "message": "Refusing to write outside user-space."
    }
  ],
  "commandPatterns": [
    {
      "id": "my-dangerous",
      "pattern": "\\bdrop\\s+database\\b",
      "regex": true,
      "behavior": "ask",
      "description": "drop database"
    }
  ],
  "contentPatterns": [
    {
      "id": "no-eval-in-dags",
      "pattern": "eval\\(",
      "regex": true,
      "onTool": ["write", "edit"],
      "pathGlobs": ["**/dags/**/*.py"],
      "behavior": "deny",
      "message": "Avoid eval() inside Dag code."
    }
  ]
}
```

### Rule strings

Rules use the format `Tool` or `Tool(content)`. Content can be a command prefix, a path glob, or `*` to match everything.

| Rule                   | Effect                                                  |
| ---------------------- | ------------------------------------------------------- |
| `Bash`                 | Matches every `bash` call                               |
| `Bash(astro deploy:*)` | Matches bash commands starting with `astro deploy`      |
| `Write(/etc/**)`       | Matches `write` calls whose path glob-matches `/etc/**` |
| `Read(~/.ssh/**)`      | Matches `read` calls with `~` expansion                 |

### Path deny

`pathDeny` blocks writes and edits against a glob list, including bash redirects. Use it to protect folders regardless of the command used to reach them.

### Command patterns

Bash-only patterns that match the full command string. Use `regex: true` for regular expressions. `commandPatterns` with `behavior: "ask"` fire even in `bypassPermissions` mode.

### Content patterns

Scan the body of `write` and `edit` calls against a pattern. `behavior: "deny"` blocks the write before it hits disk. `behavior: "warn"` appends an advisory without blocking.

## Built-in safety checks

Otto ships compiled-in protections that fire regardless of configuration:

* **Sensitive files** — prompts before any tool touches `.env*`, `~/.ssh/**`, `~/.aws/**`, shell rc files, or similar common secret locations.
* **Destructive Astro and Airflow commands** — prompts before `astro deploy`, `astro deployment delete`, `astro workspace delete`, `astro organization delete`, `astro dev kill`, `af dags delete`, `af runs delete`, `af tasks clear`, `af connections delete`, `af variables delete`, and similar destructive commands.
* **Out-of-project writes** — prompts before `write`, `edit`, or a `bash` command targets a path outside your project folder.

These checks are bypass-immune — they fire even when the mode is `bypassPermissions` or `--skip-permissions` is set.

Disable an individual destructive-command entry in your config by id:

```json wrap theme={null}
{
  "commandPatterns": [
    { "id": "astro-deploy", "enabled": false }
  ]
}
```

## Inspect the current state

Run `/permissions` in an Otto session to see the current mode, rule counts, and the file paths for each config scope.

## Disable permissions for a session

```sh wrap theme={null}
astro otto --skip-permissions
```

This coerces the mode to `bypassPermissions` and prevents `Ctrl+]` from cycling out of it for the session. Bypass-immune safety checks still fire. Use it only for scripted flows you're reviewing by hand.


# Otto settings
Source: https://astronomer.io/docs/astro/otto-settings

Reference for Otto configuration files, environment variables, CLI overrides, settings precedence, session storage, and auto-update behavior.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

This page is the reference for every place Otto reads configuration from. For deep dives on specific subsystems, see [Permissions](/docs/astro/otto-permissions), [Extensions](/docs/astro/otto-extensions), [Memory](/docs/astro/otto-memory), and [Skills](/docs/astro/otto-skills).

## File layout

Otto reads from your project repository and your home directory:

```text expandable wrap theme={null}
your-project/                          # Astro project root
├── .agents/
│   └── skills/                        # Project skills shared across agent harnesses
│       └── <skill-name>/SKILL.md
└── .astro/
    ├── config.yaml                    # Astro CLI project config (Otto reads otto.* and dev.mode)
    ├── memory/                        # Shared project memory (committed, team-visible)
    │   ├── MEMORY.md
    │   └── *.md
    └── otto/
        ├── permissions.json           # Allow, ask, and deny rules
        ├── extensions.json            # Per-project extension toggles
        └── skills/                    # Project skills authored by your team
            └── <skill-name>/SKILL.md

~/                                     # Your home directory
├── .agents/
│   └── skills/                        # User skills shared across agent harnesses
│       └── <skill-name>/SKILL.md
├── .astro/
│   ├── config.yaml                    # Astro CLI global config
│   ├── bin/
│   │   └── otto                       # Otto binary, downloaded on first launch
│   ├── memory/
│   │   ├── MEMORY.md                  # Local user memory (all projects, this computer)
│   │   ├── *.md
│   │   └── <project-slug>/            # Local project memory (this project, this computer)
│   │       └── *.md
│   └── otto/
│       ├── settings.json              # Otto user settings
│       ├── sessions/                  # Session history (JSONL)
│       ├── skills/                    # User skills authored by you
│       │   └── <skill-name>/SKILL.md
│       ├── cache/skills/              # Hosted skill cache (1-hour TTL)
│       └── logs/cli.log               # CLI-side log (rotates at 1 MiB)
```

## Settings precedence

When the same setting is defined in multiple places, Otto resolves it in this order, with earlier entries overriding later ones:

1. CLI flag passed to `astro otto` (for example, `--model`, `--allowed-tools`, `--no-extension`)
2. Environment variable (for example, `OTTO_DISABLED_EXTENSIONS`)
3. Project file (`.astro/otto/permissions.json`, `.astro/otto/extensions.json`, `.astro/config.yaml`)
4. User file (`~/.astro/otto/settings.json`, `~/.astro/config.yaml`)
5. Built-in default

## Environment variables

The `astro otto` command sets several environment variables automatically based on your `astro login` context and the current project. You don't need to configure these manually.

| Variable             | Description                                                   |
| -------------------- | ------------------------------------------------------------- |
| `ASTRO_TOKEN`        | Bearer token for the Astronomer Gateway                       |
| `ASTRO_DOMAIN`       | Astronomer domain (for example, `astronomer.io`)              |
| `ASTRO_ORGANIZATION` | Organization ID for gateway routing                           |
| `AIRFLOW_API_URL`    | Local Airflow REST API URL, auto-discovered from proxy routes |
| `AIRFLOW_USERNAME`   | Defaults to `admin` when Airflow is connected                 |
| `AIRFLOW_PASSWORD`   | Defaults to `admin` when Airflow is connected                 |

The CLI also keeps your Astro access token refreshed in the background for the duration of the session, so long-lived sessions don't hit authentication errors.

You can set the following variables yourself to control Otto behavior:

| Variable                   | Description                                                                      |
| -------------------------- | -------------------------------------------------------------------------------- |
| `OTTO_EXTENSIONS`          | Comma-separated allowlist of extensions to enable. Disables anything not listed. |
| `OTTO_DISABLED_EXTENSIONS` | Comma-separated denylist of extensions to disable.                               |
| `OTTO_LOG_LEVEL`           | Log verbosity. Useful for debugging.                                             |

## Auto-update

The first time you run `astro otto`, the Astro CLI downloads the Otto binary to `~/.astro/bin/otto`. After that, the CLI checks for new Otto releases once per day in the background and prints a hint to stderr when an update is available. Updates are applied manually with `astro otto update`.

Otto versions independently of the Astro CLI. You don't need to upgrade the Astro CLI to pick up new Otto features or fixes. Check your installed version with `astro otto version`.

To opt out of the daily update check, set `astro config set -g otto.auto_update false`.

## See also

* [Permissions](/docs/astro/otto-permissions) — Allow, ask, and deny rules for tool calls.
* [Extensions](/docs/astro/otto-extensions) — Toggle bundled extensions.
* [Memory](/docs/astro/otto-memory) — Memory tiers and file format.
* [Skills](/docs/astro/otto-skills) — Skill catalog and local skill authoring.
* [`astro otto` CLI reference](/docs/cli/v1.43/astro-otto) — Flags and subcommands.
* [Configure the Astro CLI](/docs/cli/v1.43/configure-cli) — Full Astro CLI config reference.


# Otto skills
Source: https://astronomer.io/docs/astro/otto-skills

Author skills for Otto using the Agent Skills format, and explore the skills Otto ships with for Airflow operations, Dag authoring, dbt, lineage, and more.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

Skills are structured workflows Otto loads on demand, following the open [Agent Skills](https://agentskills.io) format. A skill is a folder containing a `SKILL.md` file with YAML frontmatter and Markdown instructions, plus any scripts or supporting files the workflow needs. Skills encode domain knowledge for a specific task such as authoring a Dag, diagnosing a failure, or upgrading Airflow.

Otto's bundled skills come from two sources:

* **Open-source skills** in the [astronomer/agents](https://github.com/astronomer/agents) repository cover Airflow operations, Dag authoring and testing, dbt and Cosmos, lineage and metadata, data discovery, and Airflow extensions.
* **Proprietary skills** are available only through Otto. They cover production failure investigation, Airflow upgrades, and migrations from legacy orchestration systems, and they rely on Astronomer's proprietary compatibility knowledge base.

You can also author your own skills for team-specific workflows. To browse what's loaded in your current session, run `/skills` in an interactive Otto session.

## Author a skill

<Tip>
  For comprehensive guidance on skill design, including grounding skills in real expertise, scoping, and writing descriptions that trigger correctly, see [Agent Skills best practices](https://agentskills.io/skill-creation/best-practices).
</Tip>

### Ask Otto to write a skill

The most effective skills are extracted from real work. Complete a task in an Otto session, providing context and corrections as you go, then describe the workflow you want codified:

```text wrap theme={null}
> Write a skill called deploy-to-staging that verifies the current branch is main, runs astro deploy against the staging Deployment, and confirms the deploy succeeded in the Astro UI. Use it whenever I ask to deploy, push, or test against staging.
```

Otto generates a `SKILL.md` under `.astro/otto/skills/<skill-name>/` and shows you the draft for review. Edit the file in your editor or ask Otto to revise it before you commit.

### Skill anatomy

Each skill is a folder with a `SKILL.md` file. The frontmatter declares the skill's name and trigger description, and the body holds the workflow Otto follows when the skill fires.

```markdown title="SKILL.md" wrap theme={null}
---
name: deploy-to-staging
description: Deploy the current project to the staging Deployment. Use when the user asks to deploy to staging, push to staging, or test against staging.
---

# Deploy to staging

Follow this workflow to deploy the project:

1. Verify the current branch is `main`.
2. Run `astro deploy --deployment-id <staging-deployment-id>`.
3. Confirm the deploy succeeded in the Astro UI.
```

The `description` field is the most important part: it determines when Otto loads the skill. Include one sentence on what the skill does and the phrasing that should trigger it, prefixed with "Use when...".

Skills can also include scripts, templates, or reference docs alongside `SKILL.md`. For example, the [analyzing-data](https://github.com/astronomer/agents/tree/main/skills/analyzing-data) skill ships scripts that run a full Jupyter kernel for data exploration.

Commit `.astro/otto/skills/` to your repository so every engineer on the team picks up the same skills.

### Test and iterate

Run a prompt that should fire the skill and confirm Otto loads it. If the skill doesn't fire, refine the description. If it fires for the wrong prompts, narrow the trigger phrasing. Ask Otto to update the skill the same way you'd refine a prompt: describe what's off, and Otto edits the file for you.

## Where skills live

Otto loads skills from your project repository and your home directory:

* **Project skills** (`.astro/otto/skills/`): Skills committed alongside your Astro project so the whole team picks them up. Otto walks up from the current folder to the git repository root, so skills in any ancestor directory also load.
* **User skills** (`~/.astro/otto/skills/`): Your personal skills, available across every project on this computer. These live in your home directory and stay out of your project repository.
* **Cross-harness skills** (`.agents/skills/` and `~/.agents/skills/`): Skills following the standard Agent Skills layout, available to Otto and any other Agent Skills-compatible tool. Use these when you want the same skills to load across multiple agents.

Otto also recognizes top-level `.md` files in the `.astro/otto/skills/` paths as single-file skills. The `.agents/skills/` paths require a directory with a `SKILL.md` file.

To point Otto at skills from other agent harnesses, such as `~/.claude/skills` or `~/.codex/skills`, list the paths in the `skills` array in `~/.astro/otto/settings.json`. See [Otto settings](/docs/astro/otto-settings).

## How Otto loads skills

When a prompt matches a skill's `description`, Otto loads the skill's `SKILL.md` content and follows the workflow.

You can also invoke a skill directly. Type `/` and start typing the skill name to filter the autocomplete, or run `/skills` to browse the full list.


# Otto tools
Source: https://astronomer.io/docs/astro/otto-tools

Reference for the tools Otto uses, including file and shell operations and the af CLI for managing Airflow.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

Otto uses several tools when working in an Astro project. This page covers what each tool does and how Otto uses it.

Tools are the primitive actions Otto takes, such as reading a file, running a shell command, or querying Airflow. For multi-step workflows that compose these actions into structured playbooks, such as Dag authoring or upgrades, Otto loads [skills](/docs/astro/otto-skills).

## File and shell tools

Otto reads, writes, and edits files in your project, runs shell commands, and searches your codebase:

| Tool    | Description                                    |
| ------- | ---------------------------------------------- |
| `read`  | Read a file's contents                         |
| `write` | Create a new file or overwrite an existing one |
| `edit`  | Modify an existing file with targeted changes  |
| `bash`  | Run shell commands                             |
| `grep`  | Search file contents with regular expressions  |
| `find`  | Locate files by name pattern                   |
| `ls`    | List folder contents                           |

All file and shell tools are sandboxed to your project folder. Otto won't write or edit files outside the project root.

## af CLI

For Airflow state — Dags, runs, tasks, connections, variables, and health checks — Otto uses the `af` CLI, an open-source command-line wrapper for the Airflow REST API maintained in the [astronomer/agents](https://github.com/astronomer/agents) repository. The `af` CLI supports Airflow 2.x and 3.x with automatic version detection. When Otto is launched from an Astro project with a connected Airflow instance, `AIRFLOW_API_URL` is set automatically.

<Note>
  If no Airflow instance is connected, Otto can still inspect and edit local Dag code, but it doesn't run `af` commands by default. To test against a local Airflow environment, start one with `astro dev start` or ask Otto to start it for you. Once it is detected, `af` commands become available.
</Note>

Before creating or editing Dags, Otto checks your project's `requirements.txt` and installed Airflow providers so it doesn't suggest operators, hooks, or sensors that aren't available in your environment.

The `af` CLI covers system health, Dag and run management, tasks, assets, connections, variables, instance management, and the provider registry, along with direct API access for endpoints not wrapped by a dedicated subcommand. For the full command reference, see the [`af` CLI documentation](https://github.com/astronomer/agents/blob/main/astro-airflow-mcp/README.md#airflow-cli-tool) in the `astronomer/agents` repository.

## Example workflows

You interact with these tools by asking Otto questions in natural language. Otto selects and runs the appropriate `af` commands automatically.

### Debug a failed Dag run

> "The `etl_orders` Dag failed last night. What happened?"

Otto checks system health, looks for import errors, finds the failed run, reads task logs, and identifies the root cause. If the fix is a code change, Otto proposes the edit and validates it.

### Explore an unfamiliar Dag

> "Walk me through the `customer_pipeline` Dag — what does it do and how has it been running?"

Otto reads the Dag source, inspects its tasks and dependencies, checks recent run history, and summarizes the connections it uses.

### Check upgrade readiness

> "Which of my Dags would break if I upgrade to Airflow 3?"

Otto evaluates your Dags against Astronomer's compatibility knowledge base and produces a prioritized list of changes needed, grouped by risk level. Available to Team, Business, and Enterprise tier customers.

### Trigger and monitor

> "Trigger the `daily_etl` Dag with config `{'env': 'staging'}` and tell me when it finishes."

Otto triggers the run and monitors it until completion, reporting the outcome.

## Automatic Dag validation

After Otto edits or creates a Dag file, it runs `af dags errors` to check for import errors when an Airflow instance is connected. If errors are found, Otto attempts to fix them in the same turn. If no Airflow instance is connected, Otto can still edit Dag files, but it can't verify imports until you connect an environment or start a local one with `astro dev start`.

## Astro CLI

Otto can use the [Astro CLI](/docs/cli/v1.43/overview) through the `bash` tool for project-level operations. For example, Otto can run `astro dev pytest` to execute the tests in your project's `tests/` folder against your local Airflow environment. Any Astro CLI command available in your shell is available to Otto.


# Upgrade Airflow with Otto
Source: https://astronomer.io/docs/astro/otto-upgrades

Use Otto to evaluate Airflow version upgrades, provider package upgrades, and Astro Runtime upgrades against Astronomer's proprietary compatibility knowledge base.

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

Otto upgrades Airflow projects against a proprietary compatibility knowledge base maintained by Astronomer. The knowledge base captures which upgrade paths are safe, which changes break Dags in practice, and which provider versions are compatible with which Airflow releases. Every finding traces back to a real Apache Airflow GitHub issue, provider changelog, or production incident, so Otto's evaluations are grounded in evidence rather than generic LLM inference.

Airflow 2 to 3 alone introduced over 121 breaking changes, and the 80+ provider packages each ship on independent release cadences. The compatibility knowledge base is what makes evaluating a specific upgrade path tractable.

## Upgrade compatibility coverage

### Airflow core versions

Otto evaluates:

* 2.x minor upgrades within the 2.5+ range, including 2.5 to 2.10.
* The 2.x to 3.0 migration, which introduces the largest set of breaking changes in Airflow's history.
* 3.x point releases, individually tracked from 3.0.1 through 3.2.1.

For each transition, Otto identifies removed and renamed APIs (operators, hooks, context variables), silent behavior changes that compile but produce different results, configuration default changes, and plugin, timetable, and executor interface breaks.

### Provider packages

The knowledge base covers 16 providers, prioritized by support volume and usage: Google, Amazon, Kubernetes, Snowflake, FAB, Slack, Databricks, Standard, Azure, Common SQL, SMTP, HTTP, SSH, Celery, Postgres, and MySQL.

For each provider, Otto tracks breaking changes at the parameter level: deprecated parameters, removed operators, base-class swaps, behavior changes within a version range, and the minimum Airflow version required.

### Astro Runtime

Otto coordinates Runtime upgrades with the corresponding Airflow core and bundled provider versions, and detects Runtime-specific issues such as namespace package precedence on Astro Runtime images and registry changes that pure Airflow analysis would miss.

### Cross-package compatibility

Beyond per-version analysis, Otto flags known incompatible package combinations: dependency diamond conflicts, version locks between core and providers, and pinning requirements that surface only at install time.

## What Otto catches

The knowledge base goes beyond what static linters and Airflow's own migration tooling detect:

* **Cross-package dependency conflicts**: combinations that fail to resolve or crash at runtime.
* **Silent behavior changes**: code that compiles but executes differently after the upgrade.
* **Custom plugin, timetable, and callback breaks**: interface changes in plugin and serialization APIs.
* **Provider-level breaking changes**: deprecated parameters, removed classes, and signature changes across the 16 covered providers.
* **Astro Runtime-specific issues**: Runtime image changes that Airflow core analysis alone misses.

## How findings are scored

Every finding carries a severity (critical, high, medium, or low) and a confidence label:

* **Verified**: backed by production incident data, a referenced GitHub issue, or a provider changelog. The fix is known to work.
* **AI-suggested**: inferred from documented patterns when the knowledge base lacks an exact match. Review before applying.

Each finding includes the file and line, the specific change, and a source reference such as a GitHub issue, provider changelog entry, or migration guide section.

## Where the knowledge base comes from

Astronomer curates the compatibility knowledge base from sources tied to real production behavior: public Apache Airflow development history, provider release artifacts, internal migration experience, and the patterns surfaced through Astronomer's years of support and engineering work. Every entry traces back to a concrete source rather than generic LLM inference.

## Run an upgrade with Otto

Ask Otto to evaluate your project in natural language. Examples:

> "Which of my Dags would break if I upgrade to Airflow 3?"

> "Assess my project for an Airflow 2.7 to 2.10 upgrade."

> "The Snowflake provider has a new release I want. Is my project ready for it?"

> "Walk me from Runtime 11 to Runtime 13."

Otto detects your current Airflow core version, Runtime version, and installed providers, then analyzes the project against the subset of the knowledge base relevant to your transition. The output is a prioritized findings list with file locations, severity, confidence, and proposed fixes.

Otto can apply mechanical fixes directly, including import rewrites, parameter renames, and version bumps. For semantic rewrites such as moving direct ORM access to Task SDK methods, Otto walks through the change with you. Before any patched dependencies are written, Otto verifies that the new dependency set resolves cleanly.

## Related

* [Skills](/docs/astro/otto-skills): The full skill catalog, including the upgrade skill.


# Astro Documentation
Source: https://astronomer.io/docs/astro/overview

Astro is a fully managed Apache Airflow® service for building, running, and observing data pipelines in the cloud.

<Info>
  Astro is a cloud solution that helps you focus on your data pipelines and spend less time managing Apache Airflow®, with capabilities enabling you to build, run, and observe data all in one place.

  <div>
    <a href="https://www.astronomer.io/lp/signup/">
      Try Astro <Icon icon="arrow-right" />
    </a>

    <a href="https://www.astronomer.io/?referral=docs-what-astro-banner">
      Learn More
    </a>
  </div>
</Info>

## Run on the cloud

<CardGroup>
  <Card title="Create a Deployment" icon="rocket-launch" href="/astro/create-deployment">
    A Deployment is an instance of Apache Airflow hosted on Astro.
  </Card>

  <Card title="Deploy code" icon="code" href="/astro/deploy-code">
    Get your dags up and running on Astro.
  </Card>

  <Card title="Automate with CI/CD" icon="circle-play" href="/astro/ci-cd-templates/template-overview">
    Push code to Astro using templates for popular CI/CD tools.
  </Card>
</CardGroup>

## Get started

<CardGroup>
  <Card title="I'm unfamiliar with Apache Airflow" icon="circle-question" href="/learn/">
    Use tutorials and concepts to learn everything you need to know about running Airflow.
  </Card>

  <Card title="I'm ready to create my first project" icon="arrow-right" href="/cli/v1.43/get-started-cli">
    Learn how to create an Astro project and run it locally with the Astro command-line interface (CLI).
  </Card>
</CardGroup>

## More Astro Tools

<CardGroup>
  <Card title="Astro API" icon="gear-code" href="/astro/api/v-1/overview">
    Develop applications and scripts for Astro components with a standard REST API.
  </Card>

  <Card title="Airflow Registry" icon="bookmark" href="https://airflow.apache.org/registry/">
    Browse Airflow providers, modules, and plugins.
  </Card>
</CardGroup>

<CardGroup>
  <Card title="Dag Factory" icon="screwdriver-wrench" href="https://astronomer.github.io/dag-factory/latest/">
    Build Apache Airflow® workflows using YAML files.
  </Card>

  <Card title="Terraform Provider" icon="puzzle-piece-simple" href="https://registry.terraform.io/providers/astronomer/astro/latest/docs">
    Automate, scale, and manage your Astro infrastructure with Terraform.
  </Card>

  <Card title="Cosmos" icon="galaxy" href="https://astronomer.github.io/astronomer-cosmos/">
    Orchestrate your dbt projects in Airflow.
  </Card>
</CardGroup>


# Autoscale Remote Execution Agent workers on queue depth
Source: https://astronomer.io/docs/astro/remote-agents-autoscale-workers

Scale Remote Execution Agent worker Pods on the number of queued tasks, so that I/O-bound tasks do not exhaust the task slots on each worker.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Each Remote Execution Agent worker Pod runs a fixed number of concurrent tasks. The `syncSlots` value sets that number, and each concurrent task uses one slot. A worker that has no free slot does not accept more tasks, and the extra tasks stay in the queue.

Tasks that wait on an external system, such as a warehouse query or an API call, use very little CPU and memory. These tasks fill every slot on a worker while CPU utilization stays low. A Horizontal Pod Autoscaler (HPA) that scales on CPU or memory does not add replicas in this state, so the queue grows and task latency increases.

This document shows how to scale worker Pods on queue depth instead. Queue depth is the number of tasks in the `queued` and `running` states, which each agent worker reports in the `astro_agent_client_queue_stats` metric. It covers two setups: [prometheus-adapter](#scale-with-prometheus-adapter) with the HPA that the Helm chart creates, and [KEDA](#scale-with-keda) for clusters that do not run prometheus-adapter.

## Slot capacity and queue depth

The task capacity of a worker Deployment is `syncSlots` multiplied by the number of replicas. A Deployment with `syncSlots: 20` and two replicas runs 40 tasks at the same time.

Use this capacity to size the autoscaler:

* Set `maxReplicaCount` to your peak number of concurrent tasks divided by `syncSlots`, rounded up. For a peak of 200 concurrent tasks and `syncSlots: 20`, set `maxReplicaCount: 10`.
* Set the metric target to `syncSlots`. The autoscaler then adds one replica for each `syncSlots` tasks of work in the queue.
* Set `minReplicaCount` to the capacity you want available before the autoscaler reacts. Each scaling decision takes at least one metric interval, and a new worker Pod takes time to start.

Raising `syncSlots` is the other way to add capacity for I/O-bound tasks. A slot holds a task process, so more slots need more memory on the worker Pod. Test a higher value against your own tasks before you use it in production.

## Prerequisites

* A Remote Execution Agent that runs Agent Client version `1.7.0` or later, which exposes the `/metrics` endpoint. See [Register and configure agents](/docs/astro/remote-execution-configure-agents).
* A Prometheus instance that scrapes the agent worker Pods. See [Scrape metrics from Remote Execution Agents](/docs/astro/remote-agents-metrics).
* Permission to install cluster components and to run `helm upgrade` against the agent release.
* One of the following:
  * prometheus-adapter, which serves the metric through the Kubernetes custom metrics API.
  * KEDA, which reads Prometheus directly and creates its own HPA.

## Choose a scaling method

| Method             | How it works                                                                                                                     | Choose it when                                                                                                                                      |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| prometheus-adapter | An adapter publishes the queue metric on the Kubernetes custom metrics API. The Helm chart creates the HPA from `workers[].hpa`. | Your platform runs prometheus-adapter, or you can install it. All autoscaling configuration stays in `values.yaml`.                                 |
| KEDA               | A `ScaledObject` queries Prometheus directly. KEDA creates and owns the HPA.                                                     | Your platform has no custom metrics API, or another component already occupies it. Only one adapter can serve `custom.metrics.k8s.io` in a cluster. |

Do not use both methods for the same worker Deployment. Two autoscalers that target one Deployment overwrite each other's decisions.

## Scale with prometheus-adapter

<Steps>
  <Step title="Scrape the workers with the Deployment name in the job label">
    The adapter rule in the next step maps the `job` label to a Kubernetes Deployment, so `job` must hold the name of the worker Deployment. The Helm chart creates one Service for each worker, labeled `deploymentName: <resourceNamePrefix>-worker-<name>`, which a `ServiceMonitor` can copy into `job`:

    ```yaml title="worker-servicemonitor.yaml" wrap theme={null}
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: astro-agent-workers
      namespace: re
    spec:
      jobLabel: deploymentName
      selector:
        matchLabels:
          app: astro-agent
          component: worker
      endpoints:
        - port: http
          path: /metrics
    ```

    If you use a standalone Prometheus with static scrape configs, relabel the target so that `job` holds the same value.
  </Step>

  <Step title="Publish the queue metric through prometheus-adapter">
    Add the following rule to your prometheus-adapter configuration:

    ```yaml title="prometheus-adapter-values.yaml" wrap theme={null}
    rules:
      - seriesQuery: '{__name__="astro_agent_client_queue_stats",container!="POD",namespace!="",pod!=""}'
        resources:
          overrides:
            job:
              resource: deployment
            namespace:
              resource: namespace
        metricsQuery: sum by (job) (max by (job, queue, state) (<<.Series>>{state=~"queued|running", <<.LabelMatchers>>}))
        name:
          matches: ".*astro_agent_client_queue_stats.*"
          as: "astro_agent_client_queued_or_running_tasks"
    ```

    The counts come from the Astro orchestration plane in the worker heartbeat response, so every Pod of a worker Deployment reports the same values. `max by (job, queue, state)` removes the duplicate series, and `sum by (job)` adds the `queued` and `running` series together into one queue-depth value for each worker Deployment.
  </Step>

  <Step title="Confirm that the custom metric is available">
    Query the custom metrics API for the worker Deployment. Replace `re` with your namespace and `astro-worker-default-worker` with your Deployment name:

    ```bash wrap theme={null}
    kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/re/deployments.apps/astro-worker-default-worker/astro_agent_client_queued_or_running_tasks"
    ```

    The response contains a `value` field with the current queue depth. An error means that the adapter rule or the `job` label does not match. Fix this before you enable the HPA.
  </Step>

  <Step title="Enable the HPA on the worker">
    Configure the worker in `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    workers:
      - name: default-worker
        queues: "default"
        syncSlots: 20
        hpa:
          enabled: true
          minReplicaCount: 2
          maxReplicaCount: 10
          metric:
            enabled: true
            name: astro_agent_client_queued_or_running_tasks
            target:
              type: AverageValue
              averageValue: "20"
          behavior:
            scaleUp:
              stabilizationWindowSeconds: 0
              policies:
                - type: Pods
                  value: 1
                  periodSeconds: 60
            scaleDown:
              stabilizationWindowSeconds: 300
        terminationGracePeriodSeconds: 600
    ```

    The `averageValue` target of `20` matches `syncSlots: 20`, so the HPA runs one replica for each 20 tasks in the queue. See [Metric target types](#metric-target-types) for why queue depth needs an `AverageValue` target.

    The chart ignores `replicas` when `hpa.enabled` is `true`.
  </Step>

  <Step title="Apply the configuration and check the HPA">
    ```bash wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    kubectl get hpa -n re
    ```

    The `TARGETS` column shows the current queue depth against the target. If it shows `<unknown>`, run `kubectl describe hpa -n re` to see which metric the HPA cannot read.
  </Step>
</Steps>

## Scale with KEDA

KEDA queries Prometheus directly, so the cluster needs no custom metrics adapter. KEDA creates the HPA for the worker Deployment and owns it.

<Steps>
  <Step title="Turn off the HPA in the Helm chart">
    Set `hpa.enabled: false` for the worker, so that the chart and KEDA do not create two autoscalers for one Deployment:

    ```yaml title="values.yaml" wrap theme={null}
    workers:
      - name: default-worker
        queues: "default"
        syncSlots: 20
        replicas: 2
        hpa:
          enabled: false
    ```

    Keep `replicas` equal to the `minReplicaCount` you set in the next step. Each `helm upgrade` writes `replicas` back to the Deployment, and KEDA restores the scaled count at its next poll.
  </Step>

  <Step title="Create a ScaledObject for the worker Deployment">
    ```yaml title="worker-scaledobject.yaml" wrap theme={null}
    apiVersion: keda.sh/v1alpha1
    kind: ScaledObject
    metadata:
      name: astro-worker-default-worker
      namespace: re
    spec:
      scaleTargetRef:
        name: astro-worker-default-worker
      minReplicaCount: 2
      maxReplicaCount: 10
      pollingInterval: 30
      advanced:
        horizontalPodAutoscalerConfig:
          behavior:
            scaleDown:
              stabilizationWindowSeconds: 300
      triggers:
        - type: prometheus
          metricType: AverageValue
          metadata:
            serverAddress: http://prometheus-operated.monitoring.svc.cluster.local:9090
            threshold: "20"
            query: |-
              sum(max by (queue, state) (astro_agent_client_queue_stats{namespace="re",job="astro-worker-default-worker",state=~"queued|running"}))
    ```

    `scaleTargetRef.name` is the name of the worker Deployment, which the chart builds as `<resourceNamePrefix>-worker-<worker name>`.

    `threshold` matches `syncSlots`, and the default `AverageValue` metric type divides the query result by the threshold. KEDA therefore runs one replica for each 20 tasks in the queue.

    The query must return a single value. Adjust the label matchers to your own scrape configuration, and add a matcher for the worker Deployment when one Prometheus instance scrapes several of them. Read the labels from a worker's `/metrics` endpoint before you write the query.
  </Step>

  <Step title="Apply the ScaledObject and check the scaling">
    ```bash wrap theme={null}
    kubectl apply -f worker-scaledobject.yaml
    kubectl get scaledobject -n re
    kubectl get hpa -n re
    ```

    The `READY` and `ACTIVE` columns of the `ScaledObject` report whether KEDA can read the metric. KEDA names the HPA that it creates `keda-hpa-<scaled-object-name>`.
  </Step>
</Steps>

<Warning>
  Do not set `minReplicaCount: 0` for an agent worker. The queue metric comes from the worker Pods, so a Deployment that scales to zero exposes no metric and never scales back up.
</Warning>

### Other KEDA scalers

The `prometheus` trigger fits any platform that keeps the agent metrics in Prometheus or in a Prometheus-compatible service, such as Amazon Managed Service for Prometheus, Azure Monitor managed service for Prometheus, or Google Cloud Managed Service for Prometheus.

If your queue depth lives in another system, KEDA can read it with an `external` trigger, which calls a gRPC service that you run. The arithmetic stays the same: report the number of queued and running tasks, and set the threshold to `syncSlots`. For the list of triggers, see the [KEDA scalers reference](https://keda.sh/docs/latest/scalers/).

## Metric target types

Kubernetes computes the replica count differently for each target type. Use `AverageValue` for queue depth.

| Target type    | Replica count                                    | Use it for                                                              |
| -------------- | ------------------------------------------------ | ----------------------------------------------------------------------- |
| `AverageValue` | `metric / target`, rounded up                    | A metric that counts work for the whole Deployment, such as queue depth |
| `Value`        | `metric / target × current replicas`, rounded up | A metric that already describes a single replica                        |

A `Value` target multiplies the ratio by the current replica count. With a queue-depth metric, the replica count therefore multiplies again in each scaling cycle for as long as the queue stays above the target, until the HPA reaches `maxReplicaCount`. For the full algorithm, see the [Kubernetes HPA documentation](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details).

## Combine queue depth with resource metrics

Queue depth measures outstanding work, not the load on a Pod. Tasks that use a lot of memory can still exhaust a worker before its slots are full. Add resource metrics with `hpa.extraMetrics`, which takes Kubernetes HPA metric objects:

```yaml title="values.yaml" wrap theme={null}
    hpa:
      enabled: true
      minReplicaCount: 2
      maxReplicaCount: 10
      metric:
        enabled: true
        name: astro_agent_client_queued_or_running_tasks
        target:
          type: AverageValue
          averageValue: "20"
      extraMetrics:
        - type: Resource
          resource:
            name: memory
            target:
              type: Utilization
              averageUtilization: 80
```

The HPA evaluates every metric and uses the highest replica count that any of them recommends. Resource metrics need `resources.requests` on the worker, because the HPA calculates utilization against the request.

With KEDA, add a second trigger of type `memory` or `cpu` to the same `ScaledObject`.

## Scale down without task failures

A worker that stops during a task fails that task. Two settings protect running work:

* `terminationGracePeriodSeconds` on the worker gives a Pod time to finish its tasks before Kubernetes stops it. The default is `600`. Set it higher than your longest task if your tasks run for more than 10 minutes.
* `behavior.scaleDown.stabilizationWindowSeconds` makes the autoscaler wait before it removes replicas. The examples on this page use `300`.

A queue of short tasks therefore makes the value dip and recover between scaling cycles. `scaleDown.stabilizationWindowSeconds` makes the HPA use the highest replica count it recommended over the trailing window, so a brief dip does not remove a worker.

## Related documentation

* [Helm chart configuration reference](/docs/astro/remote-agents-helm-reference)
* [Scrape metrics from Remote Execution Agents](/docs/astro/remote-agents-metrics)
* [Register and configure agents](/docs/astro/remote-execution-configure-agents)
* [Remote Execution Agent failure scenarios](/docs/astro/remote-agents-failure-scenarios)


# Configure AWS PrivateLink for Remote Execution Agents
Source: https://astronomer.io/docs/astro/remote-agents-aws-privatelink

Configure private connectivity between Remote Execution Agents and the Astro orchestration plane using AWS PrivateLink.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

AWS PrivateLink enables private connectivity between your Remote Execution Agents and the Astro orchestration plane without exposing traffic to the public internet. This guide explains how to set up a VPC Endpoint in your AWS environment to establish secure communication with Astro.

## Overview

By default, Remote Execution Agents communicate with the Astro orchestration plane over the public internet. With AWS PrivateLink, you can route this traffic through a private connection within AWS, which provides enhanced security and can simplify network configurations for organizations with strict security requirements.

The setup involves creating a VPC Endpoint in your AWS account that connects to Astronomer's VPC Endpoint Service. Once configured, your Remote Execution Agents can communicate with Astro through this private connection.

## Prerequisites

* An Astro Deployment configured for Remote Execution.
* Remote Execution Agents installed in an AWS environment.
* Access to the AWS Console with permissions to create VPC Endpoints and modify Route53 configurations.

## Astro-side configuration

Before you can create a VPC Endpoint, Astronomer must configure the VPC Endpoint Service on the Astro side. Contact [Astronomer Support](https://support.astronomer.io) with the following information:

* Your Astro Cluster ID.
* The AWS Account ID where your Remote Execution Agents are running.
* The AWS Region where your Remote Execution Agents are running.

Astronomer Support will provide you with the **VPC Endpoint Service name**, **Service region**, and supported **Availability Zones** required to create your VPC Endpoint.

<Info>
  If your Remote Execution Agents run in a different AWS region than the Astro orchestration plane, inform Astronomer Support. Additional configuration may be required on the Astro side, such as adding your region to the VPC Endpoint Service cross-region configuration or adding your AWS account to the allowed principals list.
</Info>

## Create a VPC Endpoint

After receiving the VPC Endpoint Service name from Astronomer Support, create a VPC Endpoint in your AWS account.

<Steps>
  <Step title="Navigate to VPC Endpoints">
    In the AWS Console, go to **VPC** > **Endpoints**.
  </Step>

  <Step title="Start the endpoint creation wizard">
    Click **Create endpoint** to begin the configuration.
  </Step>

  <Step title="Configure the endpoint">
    Set the following values:

    * **Name tag**: Enter a descriptive name, such as `astro-privatelink`.
    * **Type**: Select **Endpoint services that use NLBs and GWLBs**.
    * **Service name**: Enter the VPC Endpoint Service name provided by Astronomer Support, and click **Verify service** to confirm the service name is valid.
    * **Cross-Region**: Enable if required (optional).
    * **VPC**: Select the VPC where your Remote Execution Agents are running.
    * **Subnets**: Select at least one subnet. For high availability, select subnets in multiple Availability Zones.
    * **Security group**: Select or create a security group that allows inbound traffic on HTTPS port 443.
  </Step>
</Steps>

<Warning>
  Subnet selection shows your subnets in the Availability zones supported by the VPCe Service. If there is a mismatch, you must create subnet(s) in the zones provided by Astronomer Support.
</Warning>

## Configure DNS resolution

After creating the VPC Endpoint, configure DNS so that your Remote Execution Agents resolve the Astro orchestration plane hostname to the private endpoint IP addresses.

### Configure Route53 private hosted zone

<Steps>
  <Step title="Create a private hosted zone">
    1. In the AWS Console, go to **Route 53** > **Hosted zones**.
    2. Click **Create hosted zone**.
    3. Enter `external.astronomer.run` as the domain name.
    4. Select **Private hosted zone**.
    5. Associate the hosted zone with the VPC where your VPC Endpoint was created.
  </Step>

  <Step title="Create an alias record">
    1. In the hosted zone, click **Create record**.
    2. For **Record name**, enter your Astro Cluster ID.
    3. Select **Alias**.
    4. For **Route traffic to**, select **Alias to VPC endpoint**.
    5. Select your region and the VPC Endpoint you created.
    6. Click **Create records**.
  </Step>
</Steps>

## Verify the connection

After completing the configuration, verify that your Remote Execution Agents can communicate with Astro through the private endpoint. Validate in the Astro UI that the agents are heart beating and reporting a **Healthy** status. You can also verify from within your network using the below instructions.

1. Connect to a host within your VPC that has network access to the VPC Endpoint.
2. Run a DNS lookup to confirm the hostname resolves to a private IP address:

```sh wrap theme={null}
nslookup <AstroClusterId>.external.astronomer.run
```

The response should show the private IP addresses assigned to your VPC Endpoint rather than public IP addresses.

3. Test connectivity to the endpoint:

```sh wrap theme={null}
curl -v https://<AstroClusterId>.external.astronomer.run
```

The expected response is `404 page not found`. If the connection is successful, your Remote Execution Agents will use the private endpoint for all communication with the Astro orchestration plane.

## Multiple Remote Execution Agents

If you have multiple Remote Execution Agents across different VPCs, you can either create a VPC Endpoint in each VPC, or use a single VPC Endpoint and configure network routing between VPCs.

The following table summarizes the actions required based on your configuration:

| Configuration         | Yes                   | No                                            |
| --------------------- | --------------------- | --------------------------------------------- |
| Same AWS region       | No additional actions | Attach VPC to the Route53 private hosted zone |
| Different AWS region  | No additional actions | Contact Astronomer Support                    |
| Different AWS account | No additional actions | Contact Astronomer Support                    |

<Info>
  If you previously created a Route53 private hosted zone, you can associate additional VPCs with the same hosted zone rather than creating new zones for each VPC.
</Info>

## Restrict traffic to the private endpoint

After verifying that the private endpoint works correctly, you can optionally configure your Remote Execution API (Orchestration plane) to only allow traffic originating from your VPC Endpoint. This ensures that all communication with Astro uses the private connection.

To restrict traffic:

1. Take note of your Astro Cluster VPC Subnet Range, under **Settings** > **Clusters** > **Cluster details** (in the legacy UI, **Organization Settings** > **Clusters** > **Cluster details**).
2. In the Astro UI, navigate to your Deployment and go to **Settings**.
3. In your Deployment Advanced settings, add the cluster CIDR range to the **Allowed IP address ranges** list.

This configuration ensures that only traffic coming through the VPC Endpoint Service can reach the Deployment.

## Troubleshooting

### VPC Endpoint shows "pending acceptance"

The VPC Endpoint Service may require manual acceptance of endpoint connections, if still in pending state after 5 minutes. Contact Astronomer Support to approve your endpoint connection request.

### DNS resolution returns public IP addresses

Verify that your Route53 private hosted zone is correctly configured and associated with the VPC where you are testing.

### Connection timeouts

Check that the security group attached to the VPC Endpoint allows inbound traffic on port 443 from the subnets where your Remote Execution Agents are running.


# Configure Azure Private Link for Remote Execution Agents
Source: https://astronomer.io/docs/astro/remote-agents-azure-privatelink

Configure private connectivity between Remote Execution Agents and the Astro orchestration plane using Azure Private Link.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Azure Private Link enables private connectivity between your Remote Execution Agents and the Astro orchestration plane without exposing traffic to the public internet. This guide explains how to set up a Private Endpoint in your Azure environment to establish secure communication with Astro.

## Overview

By default, Remote Execution Agents communicate with the Astro orchestration plane over the public internet. With Azure Private Link, you can route this traffic through a private connection within Azure, which provides enhanced security and can simplify network configurations for organizations with strict security requirements.

The setup involves creating a Private Endpoint in your Azure subscription that connects to Astronomer's Private Link Service. Once configured, your Remote Execution Agents can communicate with Astro through this private connection.

## Prerequisites

* An Astro Deployment configured for Remote Execution.
* Remote Execution Agents installed in an Azure environment (Azure Kubernetes Service (AKS)).
* Access to the Azure portal with permissions to create Private Endpoints, Private DNS zones, and modify networking configurations.

## Astro-side configuration

Before you can create a Private Endpoint, Astronomer must configure the Private Link Service on the Astro side. Contact [Astronomer support](https://support.astronomer.io) with the following information:

* Your Astro Cluster ID.

Astronomer support will provide you with the **Private Link Service alias** required to create your Private Endpoint.

## Create a Private Endpoint

After receiving the Private Link Service alias from Astronomer Support, create a Private Endpoint in your Azure subscription.

<Steps>
  <Step title="Gather required information">
    Before starting, collect the following:

    * The Private Link Service alias provided by Astronomer support.
    * The Remote Execution API URL for your Deployment, which you can find in the Deployment details in the Astro UI.
  </Step>

  <Step title="Navigate to Private Endpoints">
    In the Azure portal, go to **Private Link Overview** > **Private endpoints**.
  </Step>

  <Step title="Start the endpoint creation wizard">
    Click **Create** to begin the configuration.
  </Step>

  <Step title="Configure the endpoint">
    Set the following values:

    * **Resource group**: Select the resource group where your AKS cluster with Remote Execution Agents is deployed.
    * **Name**: Enter a descriptive name, such as `astro-privatelink`.
    * **Network Interface Name**: Accept the default or enter a custom name.
    * **Region**: Select the region where your Remote Execution Agents are running.
  </Step>

  <Step title="Configure the resource connection">
    Set the following values:

    * **Connection method**: Select **Connect to an Azure resource by resource ID or alias**.
    * **Resource ID or alias**: Enter the Private Link Service alias provided by Astronomer support.
  </Step>

  <Step title="Configure networking">
    Set the following values:

    * **Virtual Network**: Select the VNet where your AKS cluster with Remote Execution Agents is deployed.
    * **Subnet**: Select the appropriate subnet based on your internal network preferences.
  </Step>

  <Step title="Create the endpoint">
    Review your configuration and click **Create**.
  </Step>
</Steps>

After creating the Private Endpoint, contact Astronomer support to approve your endpoint connection request. You can proceed with the DNS configuration while waiting for approval.

## Configure DNS resolution

After creating the Private Endpoint, configure DNS so that your Remote Execution Agents resolve the Astro orchestration plane hostname to the Private Endpoint IP address.

### Create a private DNS zone

<Steps>
  <Step title="Navigate to Private DNS zones">
    In the Azure portal, go to **Private DNS zones**.
  </Step>

  <Step title="Create a new zone">
    1. Click **Create**.
    2. Enter `external.astronomer.run` as the zone name.
    3. Select the resource group and click **Create**.
  </Step>

  <Step title="Link the DNS zone to your VNet">
    1. In the newly created DNS zone, go to **Virtual network links**.
    2. Click **Add**.
    3. Enter a link name and select the VNet where your AKS cluster is deployed.
    4. Click **OK**.
  </Step>

  <Step title="Create an A record">
    1. In the DNS zone, click **Record set**.
    2. For **Name**, enter the first subdomain from your Remote Execution API URL. For example, if your API URL is `clxxxxxxxxx.external.astronomer.run`, enter `clxxxxxxxxx`.
    3. For **Type**, select **A**.
    4. For **IP address**, enter the Private Endpoint IP address. You can find this in the Private Endpoint's **Network interface** settings in the Azure portal.
    5. Click **OK**.
  </Step>
</Steps>

## Verify the connection

After Astronomer support approves your endpoint connection, verify that your Remote Execution Agents can communicate with Astro through the Private Endpoint. Validate in the Astro UI that the agents are heartbeating and reporting a **Healthy** status. You can also verify from within your network using the following instructions.

1. Connect to a host within your VNet that has network access to the Private Endpoint.
2. Run a DNS lookup to confirm the hostname resolves to a private IP address:

```sh wrap theme={null}
nslookup <AstroClusterId>.external.astronomer.run
```

The response should show the private IP address assigned to your Private Endpoint rather than a public IP address.

3. Test connectivity to the endpoint:

```sh wrap theme={null}
curl -v https://<AstroClusterId>.external.astronomer.run
```

The expected response is `404 page not found`. If the connection is successful, your Remote Execution Agents will use the Private Endpoint for all communication with the Astro orchestration plane.

## Multiple Remote Execution Agents

Only one Private Link Service is required per Astro cluster. If you have multiple Remote Execution Agents across different VNets, you can either create a Private Endpoint in each VNet (the Private Link Service alias remains unchanged) or use a single Private Endpoint across your network.

<Info>
  If you previously created a private DNS zone, you can associate additional VNets with the same zone rather than creating new zones for each VNet.
</Info>

## Restrict traffic to the Private Endpoint

After verifying that the Private Endpoint works correctly, you can optionally configure your Remote Execution Agents to only allow traffic through the Private Endpoint. This ensures that all communication with Astro uses the private connection.

To restrict traffic:

1. Take note of your Astro Cluster VPC Subnet Range, under **Settings** > **Clusters** > **Cluster details** (in the legacy UI, **Organization Settings** > **Clusters** > **Cluster details**).
2. In the Astro UI, navigate to your Deployment and go to **Settings**.
3. In your Deployment Advanced settings, add the cluster CIDR range to the **Allowed IP address ranges** list.

This configuration ensures that only traffic coming through the Private Link Service can reach the Deployment.

## Troubleshooting

### Private Endpoint shows "pending" connection state

The Private Link Service requires manual acceptance of endpoint connections. Contact Astronomer support to approve your endpoint connection request.

### DNS resolution returns public IP addresses

Verify that your private DNS zone is correctly configured and linked to the VNet where you are testing. Ensure the A record points to the correct Private Endpoint IP address.

### Connection timeouts

Check that your network security group (NSG) rules allow outbound traffic on port 443 from the subnets where your Remote Execution Agents are running to the Private Endpoint.


# Remote Execution Agent failure and recovery scenarios
Source: https://astronomer.io/docs/astro/remote-agents-failure-scenarios

Understand how Astro handles Remote Execution Agent and API server failures and task recovery.

When the heartbeat between the API server and a Remote Execution Agent is disrupted, the Astro executor prevents task duplication by marking queued tasks from that agent as `failed`. This makes tasks eligible for reassignment to healthy agents. To ensure safe task execution, an agent must receive explicit confirmation from the API server before starting any task. If an agent loses connectivity with the API server, the agent continues executing any tasks that the API server already confirmed and marked as `running`, but the agent will not start new tasks until heartbeat communication is restored.

### Agent failure

The API Server marks an agent as failed if the API server misses three consecutive heartbeat intervals. When that happens, the API server checks whether the Agent has any "queued" tasks, or tasks the agent already picked up and started running, but has not yet reported as complete. If a worker agent fails, the API server marks those tasks as failed and makes them available for reassignment. If a triggerer Agent fails, the API server immediately reassigns the tasks, since triggerer tasks are short-lived and idempotent.

#### Dag scheduling and retention during Agent disconnection

The Airflow scheduler retains all dags that were most recently parsed and sent by the dag processor agent. If the dag processor agent or any Remote Execution Agent disconnects or fails, the scheduler continues to use these previously parsed dags. The scheduler will keep creating dag runs on schedule or in response to events, such as dataset updates, for all retained dags.

* New or updated dags are not detected until a healthy dag processor Agent reconnects and provides an updated set of dags.
* All tasks and dag runs remain pending until a healthy Remote Execution Agent, worker or triggerer, is available for execution.

<Tip>
  If no healthy Remote Execution Agents are connected, the scheduler continues to create dag runs for known dags but those tasks remain in queued state and will not execute until an agent becomes available.
  If a task stays in queued state for more than 600 seconds (default) or the value set via the [`AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#task-queued-timeout) environment variable on your Astro deployment, it will be marked as failed.
</Tip>

### API Server failure

If an agent's heartbeats can't reach the API server, the agent assumes that the API server and other agents remain healthy. In this case:

* A **worker** continues running any tasks that the API server already marked as `running`, but the worker doesn't start new tasks until it reconnects with the API server. This prevents two agents from running the same task.
* A **triggerer** stops processing tasks entirely until it restores connectivity. Since triggerer workloads are designed to be reassigned immediately when disconnected, trigger execution stops during the partition.

This behavior preserves task safety and prevents duplication for both workers and triggerers, even during partial failures or network partitions.


# Helm chart configuration reference
Source: https://astronomer.io/docs/astro/remote-agents-helm-reference

Reference for required and optional Helm chart values used to install the Remote Execution Agent.

This reference describes configuration values for the Remote Execution Agent Helm chart. For complete configuration options, see the `values.yaml` file downloaded from the Astro UI.

For a history of chart changes, see [Helm chart release notes](/docs/astro/agent-helm-chart-release-notes). For which chart versions are compatible with which Astro Agent versions, see the [Helm chart to Astro agent compatibility matrix](/docs/astro/agent-maintenance-policy#helm-chart-to-astro-agent-compatibility-matrix).

## Required configuration values

The following values must be configured before installing the Helm chart:

### Agent authentication

**agentToken** / **agentTokenSecretName** / **agentTokenFile**

You must specify exactly one of these to provide the agent token generated in the Astro UI.

* **agentToken**: Token value as plain text in `values.yaml` (not recommended for production)
* **agentTokenSecretName**: Name of existing Kubernetes secret containing the token
* **agentTokenFile**: Path to file containing the token (agent reads at runtime)

See [Agent token configuration](/docs/astro/remote-execution-configure-agents#agent-token-configuration) for detailed instructions.

### Image registry access

**imagePullSecretName** / **imagePullSecretData**

You must specify exactly one of these to allow agents to pull images from the registry.

* **imagePullSecretName**: Name of existing Kubernetes secret with Docker credentials
* **imagePullSecretData**: Docker config JSON as string (Helm creates secret named `image-pull-secret`)

See [Image pull secret configuration](/docs/astro/remote-execution-configure-agents#image-pull-secret-configuration) for detailed instructions.

### Kubernetes namespace

**namespace**

Kubernetes namespace where the agent will be deployed.

* If `createNamespace: true`, Helm creates the namespace
* If `createNamespace: false`, namespace must exist before installation

<Note>
  If using `agentTokenSecretName` and `imagePullSecretName`, set `createNamespace: false` and create the namespace manually with secrets already present.
</Note>

See [Install in restricted Kubernetes namespace](/docs/astro/remote-agents-restricted-kubernetes) for restricted namespace configuration.

### Resource name prefix

**resourceNamePrefix**

Name prefix for all Kubernetes resources (Deployments, ConfigMaps, Secrets) created by the Helm chart.

### Secrets backend

**secretBackend**

Airflow secrets backend class for accessing connections and variables. Required for agent operation.

Supported backends:

* `airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend`
* `airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend`
* `airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend`
* `airflow.providers.hashicorp.secrets.vault.VaultBackend`
* `airflow.secrets.local_filesystem.LocalFilesystemBackend` (not recommended for production)

See [Configure secrets backend](/docs/astro/secrets-backend) for detailed configuration instructions.

### XCom backend

**xcomBackend**

Airflow XCom backend class for passing data between tasks. Required for agent operation.

Typically set to: `airflow.providers.common.io.xcom.backend.XComObjectStorageBackend`

See [Configure XCom backend](/docs/astro/remote-execution-configure-xcom-backend) for detailed configuration instructions.

### State store backend

**stateStoreBackend**

Airflow worker-side state store backend class for persisting task and asset state on Astro Runtime 3.3 and later. Available in Helm chart 2.3.0 and later.

Defaults to `airflow.providers.common.io.state_store.backend.StateStoreObjectStorageBackend` with a local file path. For production, point the path at shared object storage.

See [Configure state store backend](/docs/astro/remote-execution-configure-state-store-backend) for detailed configuration instructions.

### DAG bundles

**dagBundleConfigList**

JSON string defining how agents access dag code. Required for running dags.

See [Configure DAG sources](/docs/astro/remote-execution-configure-dag-sources) for detailed configuration instructions.

## Common environment variables

**commonEnv**

Environment variables applied to all agent components (worker, DAG processor, triggerer). Used to configure secrets backend parameters, XCom paths, logging settings, and other Airflow configuration.

Example:

```yaml title="values.yaml" wrap theme={null}
commonEnv:
  - name: AIRFLOW__SECRETS__BACKEND_KWARGS
    value: '{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}'
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
    value: "s3://bucket/xcom"
```

## Worker resource configuration

**workers**

Workers are configured as a list in `values.yaml`. Each entry defines a worker Deployment with its own name, resource allocation, replica count, and optional queue assignment.

| Parameter                       | Description                                                                                                   | Default          |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------- | ---------------- |
| `name`                          | Unique identifier for the worker Deployment. Used in Kubernetes resource names.                               | `default-worker` |
| `replicas`                      | Number of worker Pod replicas. Ignored when `hpa.enabled` is `true`.                                          | `1`              |
| `queues`                        | Comma-separated list of Airflow queues this worker listens on.                                                | `default`        |
| `syncSlots`                     | Number of tasks the worker Pod runs at the same time. Each concurrent task uses one slot.                     | `20`             |
| `resources.requests.cpu`        | Minimum CPU allocated to the worker Pod.                                                                      | —                |
| `resources.requests.memory`     | Minimum memory allocated to the worker Pod.                                                                   | —                |
| `resources.limits.cpu`          | Maximum CPU the worker Pod can use.                                                                           | —                |
| `resources.limits.memory`       | Maximum memory the worker Pod can use.                                                                        | —                |
| `env`                           | List of environment variables specific to this worker.                                                        | `[]`             |
| `volumes`                       | Additional volumes to mount on the worker Pod.                                                                | `[]`             |
| `volumeMounts`                  | Mount paths for the additional volumes.                                                                       | `[]`             |
| `nodeSelector`                  | Kubernetes node selector for scheduling worker Pods.                                                          | `{}`             |
| `tolerations`                   | Kubernetes tolerations for scheduling worker Pods.                                                            | `[]`             |
| `serviceAccount.name`           | Custom service account name. Overrides the default `{{ resourceNamePrefix }}-worker-{{ worker.name }}`.       | —                |
| `serviceAccount.create`         | Whether the Helm chart creates the service account. Set to `false` when using a pre-existing service account. | `true`           |
| `terminationGracePeriodSeconds` | The grace period for the worker Pod to finish existing tasks before terminating.                              | `600`            |

Example with two workers:

```yaml title="values.yaml" wrap theme={null}
workers:
  - name: default-worker
    replicas: 2
    queues: "default"
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "2"
        memory: "4Gi"
  - name: high-memory-worker
    replicas: 1
    queues: "high-memory"
    resources:
      requests:
        cpu: "1"
        memory: "4Gi"
      limits:
        cpu: "4"
        memory: "16Gi"
```

<Note>
  When you configure multiple workers, each worker creates a separate Kubernetes Deployment. The service account name for each worker defaults to `{{ resourceNamePrefix }}-worker-{{ worker.name }}`. If you use IRSA (AWS), Workload Identity (GCP), or managed identity (Azure), annotate each worker's service account.
</Note>

## Horizontal Pod Autoscaler

**workers\[].hpa**

Each worker supports a Horizontal Pod Autoscaler (HPA) configuration to automatically scale the number of worker Pod replicas based on resource utilization or custom metrics.

When `hpa.enabled` is `true`, the Helm chart creates a `HorizontalPodAutoscaler` resource named `{{ resourceNamePrefix }}-worker-{{ worker.name }}-hpa` for the worker Deployment. The `replicas` value is ignored because the HPA controls replica count.

| Parameter             | Description                                                                                                                                          | Default                                      |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- |
| `hpa.enabled`         | Enable the Horizontal Pod Autoscaler for this worker.                                                                                                | `false`                                      |
| `hpa.minReplicaCount` | Minimum number of worker Pod replicas. Required when `hpa.enabled` is `true`.                                                                        | `1`                                          |
| `hpa.maxReplicaCount` | Maximum number of worker Pod replicas. Required when `hpa.enabled` is `true`.                                                                        | `1`                                          |
| `hpa.metric.enabled`  | Add a metric that describes the worker Deployment object to the HPA. Set it to `false` to scale on `hpa.extraMetrics` only.                          | `true`                                       |
| `hpa.metric.name`     | Name of the metric in the Kubernetes custom metrics API.                                                                                             | `astro_agent_client_queued_or_running_tasks` |
| `hpa.metric.target`   | Target for `hpa.metric.name`. Follows the Kubernetes [HPA metrics spec](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/). | `{type: Value, value: 20}`                   |
| `hpa.extraMetrics`    | Additional Kubernetes HPA metric objects, such as CPU or memory targets.                                                                             | `[]`                                         |
| `hpa.behavior`        | Kubernetes HPA scaling behavior, which controls how fast replicas are added and removed.                                                             | See below\*                                  |

\* The chart's default `hpa.behavior` is:

```yaml theme={null}
behavior:
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
      - type: Pods
        value: 1
        periodSeconds: 60
  scaleDown:
    stabilizationWindowSeconds: 300
```

<Warning>
  The `hpa.metric` block is required when `hpa.enabled` is `true`. Because `workers` is a list, Helm replaces your whole worker entry and applies none of the defaults in this table. If you enable the HPA without a `metric` block, the command fails with `nil pointer evaluating interface {}.enabled`.
</Warning>

Example with CPU-based autoscaling:

```yaml title="values.yaml" wrap theme={null}
workers:
  - name: default-worker
    queues: "default"
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "2"
        memory: "4Gi"
    hpa:
      enabled: true
      minReplicaCount: 1
      maxReplicaCount: 5
      metric:
        enabled: false
      extraMetrics:
        - type: Resource
          resource:
            name: cpu
            target:
              type: Utilization
              averageUtilization: 80
```

<Note>
  You must set `resources.requests` for the metrics you use in HPA targets. For example, CPU-based autoscaling requires `resources.requests.cpu` to be set. Without resource requests, the HPA cannot calculate utilization percentages.
</Note>

You can combine multiple metrics to define more sophisticated scaling behavior. The HPA evaluates all specified metrics and scales to the highest recommended replica count.

Example with CPU and memory metrics:

```yaml wrap theme={null}
    hpa:
      enabled: true
      minReplicaCount: 2
      maxReplicaCount: 10
      metric:
        enabled: false
      extraMetrics:
        - type: Resource
          resource:
            name: cpu
            target:
              type: Utilization
              averageUtilization: 75
        - type: Resource
          resource:
            name: memory
            target:
              type: Utilization
              averageUtilization: 80
```

CPU and memory targets under-scale workers that run I/O-bound tasks. Such tasks fill every slot on a worker while CPU utilization stays low, so the HPA adds no replicas and tasks stay queued. To scale on the number of queued tasks instead, see [Autoscale Remote Execution Agent workers on queue depth](/docs/astro/remote-agents-autoscale-workers).

## Triggerer resource configuration

**triggerer**

The triggerer runs deferred tasks asynchronously. Configure the triggerer to control replica count, async capacity, resource allocation, and Pod-level settings.

| Parameter                              | Description                                                                                                                  | Default               |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | --------------------- |
| `replicas`                             | Number of triggerer Pod replicas.                                                                                            | `1`                   |
| `asyncSlots`                           | Number of concurrent triggers the triggerer Pod can compute.                                                                 | `1000`                |
| `image`                                | Docker image for the triggerer. Defaults to the top-level `image` if not set.                                                | —                     |
| `imagePullPolicy`                      | Image pull policy for the triggerer. Defaults to the top-level `imagePullPolicy` if not set.                                 | —                     |
| `resources.limits.cpu`                 | Maximum CPU the triggerer Pod can use.                                                                                       | `1`                   |
| `resources.limits.ephemeral-storage`   | Maximum ephemeral storage the triggerer Pod can use.                                                                         | `1Gi`                 |
| `resources.limits.memory`              | Maximum memory the triggerer Pod can use.                                                                                    | `2Gi`                 |
| `resources.requests.cpu`               | Minimum CPU allocated to the triggerer Pod.                                                                                  | `1`                   |
| `resources.requests.ephemeral-storage` | Minimum ephemeral storage allocated to the triggerer Pod.                                                                    | `1Gi`                 |
| `resources.requests.memory`            | Minimum memory allocated to the triggerer Pod.                                                                               | `2Gi`                 |
| `env`                                  | List of environment variables specific to the triggerer.                                                                     | `[]`                  |
| `livenessProbe`                        | Liveness probe configuration for the triggerer Pod.                                                                          | See following example |
| `readinessProbe`                       | Readiness probe configuration for the triggerer Pod.                                                                         | See following example |
| `podSecurityContext`                   | Pod security context for the triggerer Pod. By default, the agent runs as a non-root user with UID 50000 and group ID 50000. | `~`                   |
| `containerSecurityContext`             | Security context for the triggerer container.                                                                                | `{}`                  |
| `initContainers`                       | Init containers to add to the triggerer Pod.                                                                                 | `[]`                  |
| `extraContainers`                      | Sidecar containers to add to the triggerer Pod.                                                                              | `[]`                  |
| `volumes`                              | Additional volumes to mount on the triggerer Pod.                                                                            | `[]`                  |
| `volumeMounts`                         | Mount paths for the additional volumes.                                                                                      | `[]`                  |
| `nodeSelector`                         | Kubernetes node selector for scheduling triggerer Pods.                                                                      | `~`                   |
| `affinity`                             | Affinity rules for the triggerer Pod.                                                                                        | `{}`                  |
| `tolerations`                          | Kubernetes tolerations for scheduling triggerer Pods.                                                                        | `[]`                  |

Example:

```yaml title="values.yaml" expandable wrap theme={null}
triggerer:
  replicas: 2
  asyncSlots: 500
  resources:
    limits:
      cpu: "1"
      ephemeral-storage: "1Gi"
      memory: "2Gi"
    requests:
      cpu: "1"
      ephemeral-storage: "1Gi"
      memory: "2Gi"
  livenessProbe:
    httpGet:
      path: healthz
      port: 39091
    initialDelaySeconds: 30
    periodSeconds: 5
    failureThreshold: 3
    successThreshold: 1
    timeoutSeconds: 5
  readinessProbe:
    httpGet:
      path: healthz
      port: 39091
    initialDelaySeconds: 30
    periodSeconds: 5
    failureThreshold: 3
    successThreshold: 1
    timeoutSeconds: 5
  podSecurityContext:
    runAsUser: 50000
    fsGroup: 50000
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containerSecurityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL
  initContainers:
    - name: vault-agent
      image: vault:1.13.0
      command: ["vault", "agent", "-config=/etc/vault/config.hcl"]
      volumeMounts:
        - name: vault-config
          mountPath: /etc/vault
      env:
        - name: VAULT_ADDR
          value: "https://vault.example.com"
        - name: VAULT_TOKEN
          valueFrom:
            secretKeyRef:
              key: token
              name: vault-token
  extraContainers:
    - name: logging-sidecar
      image: timberio/vector:0.45.0-debian
      env:
        - name: VECTOR_CONFIG
          value: /etc/vector/vector.yaml
      volumeMounts:
        - name: vector-config
          mountPath: /etc/vector
      resources:
        limits:
          cpu: "0.5"
          memory: "256Mi"
        requests:
          cpu: "0.5"
          memory: "256Mi"
  volumes:
    - name: task-logs
      emptyDir: {}
    - name: vector-config
      configMap:
        name: vector-config
  volumeMounts:
    - name: task-logs
      mountPath: /var/log/airflow
      readOnly: true
  nodeSelector:
    node.kubernetes.io/instance-type: c5.large
```

<Note>
  To run multiple triggerers, increase `replicas`. The configured number of replicas runs continuously. For restricted namespaces with Pod security standards set to restricted, configure `podSecurityContext` and `containerSecurityContext` to meet your cluster's requirements.
</Note>

## Optional configuration

### Logging sidecar

**loggingSidecar**

Optional sidecar for exporting task logs to external platforms or viewing logs in the Airflow UI before task completion.

See [Configure logging sidecar](/docs/astro/remote-agents-logging-sidecar) for configuration instructions.

### OpenLineage

**openLineage**

Optional configuration for data lineage collection.

<Note>You must configure OpenLineage to use [Astro Observe](/docs/astro/astro-observe) with Remote Execution Deployments.</Note>

See [Configure OpenLineage](/docs/astro/remote-execution-configure-openlineage) for configuration instructions.

### Sentinel monitoring

**sentinel**

Monitoring service for agent health reporting (agent version 1.2.0+). Astronomer recommends enabling Sentinel for all deployments.

See [Sentinel for Remote Execution Agents](/docs/astro/remote-agents-sentinel) for configuration instructions.

### Cloud provider annotations

**annotations** and **labels**

Kubernetes annotations and labels to configure Pods to run using a specific IAM role (AWS), workload identity (GCP) or managed identity (Azure).

## Helm commands

After the Remote Execution Agent is installed, any updates to the agent  use the `helm upgrade` command.

### Install agent

```sh wrap theme={null}
helm repo add astronomer https://helm.astronomer.io
helm repo update
helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml
```

### Update agent

```sh wrap theme={null}
helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
```

### View current configuration

```sh wrap theme={null}
helm get values astro-agent
```


# Configure LoggingSidecar in a Remote Execution Agent
Source: https://astronomer.io/docs/astro/remote-agents-logging-sidecar

Ship Airflow task logs from a Remote Execution Agent to external systems using a logging sidecar.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Airflow task logs are generated when tasks execute in the Worker and Triggerer components. The logging sidecar is a container that runs alongside these components to collect and ship task logs to external systems like Splunk, Elasticsearch, AWS CloudWatch, or other log aggregation services.

The following procedure describes how to configure your Remote Execution Agent to use the logging sidecar. This process configures the `loggingSidecar` section in your `values.yaml` file, which controls the deployment of a sidecar container that collects and forwards task logs.

## Prerequisites

* You must have permissions for `deployment create` or `pod create` in the kubernetes Namespace where your Remote Execution Agent is installed.

## Enable the Logging Sidecar

1. Configure volumes in your Agent Worker and Agent Triggerer components of your `values.yaml` file to collect task logs:

```yaml title="values.yaml" wrap theme={null}
workers:
  - name: default-worker
    volumes:
      - name: task-logs
        emptyDir: {}
    volumeMounts:
      - name: task-logs
        mountPath: /usr/local/airflow/logs

triggerer:
  volumes:
    - name: task-logs
      emptyDir: {}
  volumeMounts:
    - name: task-logs
      mountPath: /usr/local/airflow/logs
```

2. To enable the logging sidecar, set `enabled` to `true` in your Remote Execution Agent's `values.yaml` file, and define the name of your logging sidecar and the image you want to use. Astronomer recommends using [Vector](https://vector.dev/docs/) for exporting task logs and the following example uses the [Timber](https://hub.docker.com/r/timberio/vector) docker image for it.

```yaml title="values.yaml" wrap theme={null}
loggingSidecar:
  enabled: true
  name: vector-logging-sidecar
  image: timberio/vector:0.45.0-debian
```

3. Allocate resources for your sidecar container in the `values.yaml` file:

```yaml title="values.yaml" wrap theme={null}
loggingSidecar:
  resources:
    limits:
      cpu: "0.5"
      memory: "1Gi"
    requests:
      cpu: "0.5"
      memory: "1Gi"
```

## Example logging sidecar configuration

The following YAML file shows a full configuration example for a logging sidecar that uses Vector to export task log data to the [Splunk Cloud Platform](https://docs.splunk.com/Documentation/SplunkCloud).

```yaml title="values.yaml" wrap expandable theme={null}
loggingSidecar:
  enabled: true
  name: vector-logging-sidecar
  image: timberio/vector:0.45.0-debian

  # Mount the task logs directory to access log files
  volumeMounts:
    - name: task-logs
      mountPath: /etc/vector/task_logs

  # Resource allocation for the sidecar container
  resources:
    limits:
      cpu: "0.5"
      memory: "1Gi"
    requests:
      cpu: "0.5"
      memory: "1Gi"

  # Vector configuration
  config: |

    data_dir: /etc/vector/task_logs

    # Define log sources
    sources:
      task_logs:
        type: file
        include:
          - /etc/vector/task_logs/**/*.log

    transforms:
      parse_task_log_file:
        type: remap
        inputs:
          - task_logs
        source: |

          parsed = parse_regex!(.file, r'/dag_id=(?P<dagID>[0-9a-z-_]+)/run_id=(?P<runID>[^/]+)/task_id=(?P<taskID>[0-9a-z-_]+)/(?:map_index=(?P<mapIndex>-?[0-9]+)/)?attempt=(?P<attempt>[0-9]+)/(?P<tiID>[0-9a-z-]+)(?:\.log\.trigger\.[0-9]+)?\.log$')
          .tiID = parsed.tiID
          .attempt = parsed.attempt
          .taskID = parsed.taskID
          .runID = parsed.runID
          .dagID = parsed.dagID
          .mapIndex = parsed.mapIndex ?? -1

    sinks:
      splunk:
        type: splunk_hec_logs
        inputs:
          - parse_task_log_file
        endpoint: https://<your-domain>.splunkcloud.com
        default_token: <token>
        index: <your-index>
        indexed_fields:
          - tiID
          - attempt
          - taskID
          - runID
          - dagID
        encoding:
          codec: "text"

workers:
  - name: default-worker
    volumes:
      - name: task-logs
        emptyDir: {}
    volumeMounts:
      - name: task-logs
        mountPath: /usr/local/airflow/logs

triggerer:
  volumes:
    - name: task-logs
      emptyDir: {}
  volumeMounts:
    - name: task-logs
      mountPath: /usr/local/airflow/logs
```


# Scrape metrics from Remote Execution Agents
Source: https://astronomer.io/docs/astro/remote-agents-metrics

Scrape metrics from Astro Remote Execution Agents.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

This guide lists available metrics when running Astro's Remote Execution Agents and explains how to scrape metrics using OpenTelemetry and Prometheus.

## What you can monitor

A self-managed Prometheus instance can collect the following classes of metrics from agent components:

* **Agent client metrics** that each agent component exposes on its own `/metrics` endpoint, including a `component_health` gauge and heartbeat, agent proxy, and Python runtime metrics. The Dag processor also exposes parsing-pipeline metrics. See [Agent client metrics](#agent-client-metrics) for the full list.
* **Airflow application metrics** that Airflow emits in StatsD format from the worker, Dag processor, and triggerer. Examples include scheduler heartbeats, task instance state counts, and Dag parse times.
* **Kubernetes infrastructure metrics** for the agent Pods, such as CPU, memory, and Pod status. These come from `kube-state-metrics`, `cAdvisor`, and `node-exporter` on your cluster.
* **Sentinel runtime metrics** Sentinel reports agent and integration health back to Astro, and you can scrape its Pod for local observability. Requires [Sentinel to be enabled](/docs/astro/remote-agents-sentinel).

For metrics that Astro generates outside your cluster, such as orchestration plane scheduler activity, see [Export metrics from Astro](/docs/astro/export-metrics).

## Prerequisites

* A Remote Execution Agent (running Agent Client version `1.7.0` and above) installed in your Kubernetes cluster. See [Register and configure agents](/docs/astro/remote-execution-configure-agents).
* A running Prometheus deployment in the same cluster, or one that can reach the agent namespace over the network. The Prometheus Operator with `PodMonitor` or `ServiceMonitor` resources is supported, and so is a standalone Prometheus that uses static scrape configs.
* Cluster-level access to deploy Helm chart updates and, if you use the Prometheus Operator, to create custom resources.
* `kube-state-metrics` and `cAdvisor` available in your cluster, if you want to collect Kubernetes infrastructure metrics.

## Monitor Airflow application metrics

Airflow can expose metrics to [OpenTelemetry](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#setup-opentelemetry) or [StatsD](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#setup-statsd).

This guide covers a minimal monitoring example using OpenTelemetry. Features such as Kubernetes namespace isolation are out of scope for this guide. Code examples assume the components are deployed in a Kubernetes namespace named `re`.

See additional configuration here:

* [Airflow StatsD metrics example](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#setup-statsd)
* [OpenTelemetry Collector configuration](https://opentelemetry.io/docs/collector/configuration)

### Step 1: Install OpenTelemetry Dependency

Add `apache-airflow[otel]` to `requirements-client.txt` and deploy the Remote Execution agents (using `astro remote deploy`).

### Step 2: Install OpenTelemetry

This step installs an OpenTelemetry Collector on your Kubernetes cluster using a Helm chart. If you already have OpenTelemetry running, you can skip this step.

From your terminal, add the `open-telemetry` Helm repository and download the latest metadata:

```sh wrap theme={null}
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
```

Create a YAML file named for example `otel-values.yaml` for OpenTelemetry configuration:

```yaml title="otel-values.yaml" expandable wrap theme={null}
mode: deployment

image:
  repository: otel/opentelemetry-collector-contrib

command:
  name: otelcol-contrib

config:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318

  exporters:
    debug:
      verbosity: detailed
    prometheus:
      endpoint: 0.0.0.0:8889

  service:
    pipelines:
      metrics:
        receivers: [otlp]
        exporters: [debug, prometheus]

ports:
  otlp-http:
    enabled: true
    containerPort: 4318
    servicePort: 4318
    protocol: TCP

  prometheus:
    enabled: true
    containerPort: 8889
    servicePort: 8889
    protocol: TCP
```

Install the OpenTelemetry Collector:

```sh wrap theme={null}
helm install otel-collector open-telemetry/opentelemetry-collector -n re -f otel-values.yaml
```

### Step 3: Configure Airflow to ship metrics to OpenTelemetry

Configure these environment variables in your Remote Execution `values.yaml` under `commonEnv`:

```yaml title="values.yaml" wrap theme={null}
commonEnv:
  - name: AIRFLOW__METRICS__OTEL_ON
    value: "True"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://otel-collector-opentelemetry-collector.re.svc.cluster.local:4318
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: http/protobuf
```

This ensures Airflow services push metrics to the OTLP endpoint.

Upgrade your Remote Execution Helm chart:

```sh wrap theme={null}
helm upgrade astro-agent astronomer/astro-remote-execution-agent --values values.yaml
```

### Step 4: Verify metrics

At this stage, Airflow application metrics should arrive in the OTLP collector. You can verify this by port forwarding the otel-collector:

```sh wrap theme={null}
kubectl port-forward -n re deploy/otel-collector-opentelemetry-collector 8889:8889
```

Browse to [http://localhost:8889/metrics](http://localhost:8889/metrics) and check if you observe any metrics. Note that you might need to run some Airflow tasks for more metrics to show. Here’s an example of metrics that you might observe:

```text wrap expandable theme={null}
# HELP airflow_airflow_io_load_filesystems 
# TYPE airflow_airflow_io_load_filesystems gauge
airflow_airflow_io_load_filesystems{job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version=""} 209.85527300035756
# HELP airflow_dag_example_astronauts_get_astronauts_duration 
# TYPE airflow_dag_example_astronauts_get_astronauts_duration gauge
airflow_dag_example_astronauts_get_astronauts_duration{job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version=""} 3903.361
# HELP airflow_operator_successes_pythondecoratedoperator_total 
# TYPE airflow_operator_successes_pythondecoratedoperator_total counter
airflow_operator_successes_pythondecoratedoperator_total{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 1
# HELP airflow_operator_successes_total 
# TYPE airflow_operator_successes_total counter
airflow_operator_successes_total{dag_id="example_astronauts",job="airflow",operator="_PythonDecoratedOperator",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 1
# HELP airflow_serde_load_serializers 
# TYPE airflow_serde_load_serializers gauge
airflow_serde_load_serializers{job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version=""} 0.8434970004600473
# HELP airflow_task_duration 
# TYPE airflow_task_duration gauge
airflow_task_duration{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 3903.361
# HELP airflow_ti_finish_example_astronauts_get_astronauts_success_total 
# TYPE airflow_ti_finish_example_astronauts_get_astronauts_success_total counter
airflow_ti_finish_example_astronauts_get_astronauts_success_total{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 1
# HELP airflow_ti_finish_total 
# TYPE airflow_ti_finish_total counter
airflow_ti_finish_total{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",state="success",task_id="get_astronauts"} 1
# HELP airflow_ti_start_example_astronauts_get_astronauts_total 
# TYPE airflow_ti_start_example_astronauts_get_astronauts_total counter
airflow_ti_start_example_astronauts_get_astronauts_total{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 1
# HELP airflow_ti_start_total 
# TYPE airflow_ti_start_total counter
airflow_ti_start_total{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 1
# HELP airflow_ti_successes_total 
# TYPE airflow_ti_successes_total counter
airflow_ti_successes_total{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 1
```

If you see metrics similar to the above, your setup is successful.

From here, you can configure your metrics backend such as Prometheus to scrape the metrics from `otel-collector:8889`. A common metrics/monitoring setup is Prometheus as the metrics backend, plus Grafana for visualization.

## Agent client metrics

Each agent component (worker, Dag processor, and triggerer) runs an internal HTTP server that exposes Prometheus-format metrics directly on a `/metrics` endpoint, on port `39091` by default (configurable through the setting `http_server.port`, or using the environment variable `ASTRO_AGENT_CLIENT_HTTP_SERVER__PORT`). These metrics describe the agent client's own runtime, including its health, its heartbeat traffic with the Astro orchestration plane, and the Python process it runs in. Agent client metrics are independent of the Airflow application metrics described in [Monitor Airflow application metrics](#monitor-airflow-application-metrics), and you can scrape both endpoints from the same Prometheus instance.

The most important health signal is the `component_health` gauge. Each internal subsystem reports `1` when healthy and `0` when unhealthy, for example:

```text wrap theme={null}
component_health{component="TriggererHeartbeater"} 1.0
component_health{component="TriggererProc"} 1.0
component_health{component="Server"} 1.0
```

The `component` label values vary by agent client. For example, the worker reports its own set of subsystems, and the Dag processor reports subsystems for the parsing pipeline. Alert when any `component_health` series drops to `0`.

If you are using Prometheus to scrape the metrics of your Agent clients, you can configure the Helm chart's `annotations` with the following annotations to ensure that your metrics get collected:

```yaml title="values.yaml" wrap theme={null}
# inside your values.yaml file
# make sure to update the port annotation if you changed the default value
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: "/metrics"
  prometheus.io/scheme: "http"
  prometheus.io/port: "39091"
```

### Generic metrics

All three agent clients ship a common set of metrics that cover Python runtime, process resources, heartbeat traffic with the API server, queue state, and the agent proxy.

| Metric                                                  | Type      | Description                                                                                                                                                                                 |
| ------------------------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `component_health`                                      | Gauge     | Health status of agent subsystems. `1` is healthy and `0` is unhealthy.                                                                                                                     |
| `heartbeat_requests_total`                              | Counter   | Total heartbeat attempts, with `component` and `outcome` labels (`success`, `timeout`, `error`).                                                                                            |
| `heartbeat_duration_seconds`                            | Histogram | Time from the start of a heartbeat request to the receipt of the provider response.                                                                                                         |
| `heartbeat_payload_bytes`                               | Histogram | Size in bytes of the serialized heartbeat request body before it is sent.                                                                                                                   |
| `heartbeat_requests_received_total`                     | Counter   | Total heartbeat requests received from tasks.                                                                                                                                               |
| `heartbeat_requests_sent_total`                         | Counter   | Total heartbeat requests sent to the API server.                                                                                                                                            |
| `heartbeat_requests_error_total`                        | Counter   | Total heartbeat requests that failed to be sent to the API server.                                                                                                                          |
| `matched_proxy_errors_total`                            | Counter   | Total proxy failures for matched routes, with `route` and `error_type` labels.                                                                                                              |
| `astro_agent_client_queue_stats`                        | Gauge     | Number of tasks in each state in each queue. Use it to scale workers on queue depth. See [Autoscale Remote Execution Agent workers on queue depth](/docs/astro/remote-agents-autoscale-workers). |
| `astro_agent_proxy_http_requests_total`                 | Counter   | Total agent proxy requests, with `method`, `status`, and `handler` labels.                                                                                                                  |
| `astro_agent_proxy_http_request_duration_seconds`       | Histogram | Agent proxy request latency, with `handler` and `method` labels. Use this when aggregation by handler matters.                                                                              |
| `astro_agent_proxy_http_request_duration_highr_seconds` | Histogram | High-resolution agent proxy request latency for accurate percentile calculations.                                                                                                           |
| `astro_agent_proxy_http_request_size_bytes`             | Summary   | Content length of incoming agent proxy requests, by `handler`.                                                                                                                              |
| `astro_agent_proxy_http_response_size_bytes`            | Summary   | Content length of outgoing agent proxy responses, by `handler`.                                                                                                                             |
| `python_info`                                           | Gauge     | Python platform information, including the interpreter implementation and version labels.                                                                                                   |
| `python_gc_objects_collected_total`                     | Counter   | Objects collected during garbage collection, by `generation`.                                                                                                                               |
| `python_gc_objects_uncollectable_total`                 | Counter   | Uncollectable objects found during garbage collection, by `generation`.                                                                                                                     |
| `python_gc_collections_total`                           | Counter   | Number of times each generation was collected.                                                                                                                                              |
| `process_cpu_seconds_total`                             | Counter   | Total user and system CPU time, in seconds.                                                                                                                                                 |
| `process_virtual_memory_bytes`                          | Gauge     | Virtual memory size, in bytes.                                                                                                                                                              |
| `process_resident_memory_bytes`                         | Gauge     | Resident memory size, in bytes.                                                                                                                                                             |
| `process_start_time_seconds`                            | Gauge     | Start time of the process since the Unix epoch, in seconds.                                                                                                                                 |
| `process_open_fds`                                      | Gauge     | Number of open file descriptors.                                                                                                                                                            |
| `process_max_fds`                                       | Gauge     | Maximum number of open file descriptors.                                                                                                                                                    |

### Dag processor metrics

In addition to the generic metrics, the Dag processor exposes parsing-pipeline metrics.

| Metric                                    | Type    | Description                                                                                         |
| ----------------------------------------- | ------- | --------------------------------------------------------------------------------------------------- |
| `dag_processor_heartbeat_dags_sent_total` | Counter | Total Dags included in heartbeat requests.                                                          |
| `dag_processor_results_queue_depth`       | Gauge   | Number of parsed Dag results waiting in the coordinator queue. Use it as a back-pressure indicator. |
| `dag_processor_cache_hits_total`          | Counter | Total Dags skipped because their `dag_hash` matched the cached value.                               |
| `dag_processor_cache_misses_total`        | Counter | Total Dags processed because they were new, changed, or cold in the cache.                          |
| `dag_processor_cache_size`                | Gauge   | Number of entries in the Dag processor `dag_hashes` cache.                                          |

**Note** that the `heartbeat_*` metrics described earlier in the `Generic Metrics` section are also available for the Dag Processor component, and can be filtered by `component=dag_processor`.

## Related documentation

* [Sentinel for Remote Execution Agents](/docs/astro/remote-agents-sentinel)
* [Autoscale Remote Execution Agent workers on queue depth](/docs/astro/remote-agents-autoscale-workers)
* [Helm chart configuration reference](/docs/astro/remote-agents-helm-reference)
* [Export metrics from Astro](/docs/astro/export-metrics)
* [Register and configure agents](/docs/astro/remote-execution-configure-agents)


# Install Remote Execution Agents in a restricted kubernetes namespace
Source: https://astronomer.io/docs/astro/remote-agents-restricted-kubernetes

Install a Remote Execution Agent in a Kubernetes namespace with restricted Pod security standards.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

You can install the Remote Execution Agent in a Kubernetes namespace with restricted pod security standards. Your organization might have different security standards for infrastructure supporting internal-only sandboxes compared to production environments.

Kubernetes Pod Security Standards define different security levels for Pods:

* **Privileged**: No restrictions (least secure)
* **Baseline**: Prevents known privilege escalations
* **Restricted**: Highly-constrained settings following security best practices (most secure)

The **Restricted** profile enforces the following limitations:

* Runs containers as non-root users
* Prevents privilege escalation
* Drops all Linux capabilities
* Uses read-only root filesystems when possible
* Requires a runtime default seccomp profile

However, because of these limitations, you need to complete the following additional Remote Execution Agent configuration set up.

## Step 1: Create a restricted namespace

Create a Namespace in your Kubernetes manifest with the following `restricted` Pod security standards:

```yaml title="namespace.yaml" wrap theme={null}
apiVersion: v1
kind: Namespace
metadata:
  name: astro-agent-restricted
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
```

## Step 2: Configure Global Security Settings

Modify your Agent's `values.yaml` file to set global security context settings that apply to all Agent components' Pods and containers:

```yaml title="values.yaml" wrap theme={null}
# Global pod security context for all components
podSecurityContext:
  seccompProfile:
    type: RuntimeDefault
  runAsUser: 50000
  fsGroup: 50000
  runAsNonRoot: true

# Global container security context for all components
containerSecurityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL
```

## Step 3: Configure component-specific settings

When using the Agent in a restricted namespace, you must configure volume mounts because:

1. The container security context sets `readOnlyRootFilesystem: true`
2. These directories need write access during runtime
3. Using `emptyDir` volumes provides isolated, writable storage that meets security requirements

### Worker configuration

```yaml title="values.yaml" wrap theme={null}
workers:
  - name: default-worker
    # Other worker settings...

    # Required volumes for filesystem access
    volumes:
      - name: tmp
        emptyDir: {}
      - name: task-logs
        emptyDir: {}

    volumeMounts:
      - name: tmp
        mountPath: /tmp # This folder is used by the Astro Agent to maintain the socket file
      - name: task-logs
        mountPath: /usr/local/airflow/logs # Or the configured folder that holds the task logs
```

### Dag processor configuration

```yaml title="values.yaml" wrap theme={null}
dagProcessor:
  # Other Dag processor settings...

  # Required volumes for filesystem access
  volumes:
    - name: tmp
      emptyDir: {}

  volumeMounts:
    - name: tmp
      mountPath: /tmp # This folder is used by the Astro Agent to maintain the socket file
```

### Triggerer configuration

```yaml title="values.yaml" wrap theme={null}
triggerer:
  # Other triggerer settings...

  # Required volumes for filesystem access
  volumes:
    - name: tmp
      emptyDir: {}
    - name: task-logs
      emptyDir: {}

  volumeMounts:
    - name: tmp
      mountPath: /tmp # This folder is used by the Astro Agent to maintain the socket file
    - name: task-logs
      mountPath: /usr/local/airflow/logs # Or the configured folder that holds the task logs
```

## Step 4: (Optional) Add logging sidecar configuration

```yaml title="values.yaml" wrap theme={null}
loggingSidecar:
  # Other logging sidecar settings...

  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL
  volumeMounts:
    - name: task-logs
      mountPath: /etc/airflow/logs # Or the configured folder that holds the task logs
```


# Sentinel for Remote Execution Agents
Source: https://astronomer.io/docs/astro/remote-agents-sentinel

Overview of the Sentinel service for Astro Remote Execution Agents.

## Overview

The **Sentinel** service provides real-time monitoring and operational visibility for Astro Remote Execution Agents running in your Kubernetes cluster. Sentinel automates health checks and reports agent status back to the Astro orchestration plane. This enables Astronomer support to provide proactive support and improved triage for support issues.

Sentinel is available as part of the Remote Execution Agent Helm chart starting in version **1.2.0**.

<Info>
  Sentinel is included in the Remote Execution Agent Helm chart starting in version 1.2.0. Astronomer recommends enabling Sentinel for all Remote Execution deployments.
</Info>

Sentinel provides the following key benefits:

* Detects Pod and component issues before they impact task execution.
* Monitors essential integrations such as XCom and secrets backends.

## How Sentinel works

Sentinel runs as a Pod alongside your agent components in Kubernetes. It:

* Watches for issues with agent Pods in its namespace.
* Checks the health of key integrations and reports status.
* Sends regular "heartbeat" reports to Astro's API, where Deployment health can be reviewed.
* Only monitors agent-managed Pods, default is Pods labeled `app=astro-agent`. No user workload data, dag code, or unrelated Pod information leaves your environment.

## Set up

To enable Sentinel, set the following in your Helm chart configuration:

```yaml title="values.yaml" wrap theme={null}
sentinel:
  enabled: true
```

Astronomer recommends that you host the Sentinel image in your organization's registry and update the image reference in your Helm chart configuration.

To customize which agent pods Sentinel observes, you can change the `agent_component_app_label` in the Helm chart values. By default, Sentinel only monitors pods labeled `app=astro-agent`, but you can restrict or broaden this scope as needed.

## Security and scope

* Sentinel only observes Pods with a specific label in its namespace.
* All status data flows outbound to Astro. No inbound connectivity is required.
* No dag, task logs, or business data is transmitted.


# Register and configure agents
Source: https://astronomer.io/docs/astro/remote-execution-configure-agents

Register Remote Execution Agents with an Astro Deployment and install the Helm chart.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Remote Execution Agents execute Airflow tasks in your Kubernetes infrastructure. This guide covers registering agents with your Astro Deployment and installing the Helm chart.

## Prerequisites

* Astro Deployment configured for Remote Execution mode. See [Create a Deployment](/docs/astro/create-deployment).
* Kubernetes 1.30 or later

<Tip>
  **Recommended Kubernetes configuration**

  Configure `singleProcessOOMKill: true` in your kubelet configuration. With this setting,
  Kubernetes kills only the process that runs out of memory instead of the entire Pod. Without it, one task's
  out-of-memory error kills all tasks on the worker and loses all task logs.

  See [Kubernetes documentation](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/) for details.
</Tip>

* Helm 3 or later
* [Deployment API token](/docs/astro/deployment-api-tokens) with Deployment Admin role to pull the base Astro Remote Execution Agent Image

## Step 1: Create agent token

The agent token authenticates your agent to the Astro orchestration plane. Create this token before installing the Helm chart.

<Note>
  Save the token value in a secure location immediately after creation. You cannot retrieve it again. The limit is 50 agent tokens per Deployment.
</Note>

<Tabs>
  <Tab title="Astro UI">
    <Steps>
      <Step title="Navigate to Deployment">
        In the Astro UI, click **Deployments**, then select your Remote Execution Deployment (in the legacy UI, select a Workspace first).
      </Step>

      <Step title="Open tokens view">
        Select the **Remote Agents** tab and toggle to the **Tokens** view.
      </Step>

      <Step title="Create token">
        1. Click **+Agent Token**
        2. Enter a **Name** and **Expiration** period
        3. Optionally add a **Description**
        4. Click **Create**
      </Step>

      <Step title="Save token">
        Copy the agent token and save it securely. You will use this token in the Helm chart configuration.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Astro API">
    <Steps>
      <Step title="Get bearer token">
        Retrieve your [Deployment API token](/docs/astro/deployment-api-tokens). This authenticates API requests.
      </Step>

      <Step title="Get organization ID">
        Make a GET request to the organizations endpoint:

        ```sh wrap theme={null}
        curl https://api.astronomer.io/platform/v1beta1/organizations \
          -H "Authorization: Bearer <token>"
        ```

        Locate the `id` field in the response.
      </Step>

      <Step title="Get Deployment ID">
        Make a GET request to the deployments endpoint using your organization ID:

        ```sh wrap theme={null}
        curl https://api.astronomer.io/platform/v1beta1/organizations/<organizationId>/deployments \
          -H "Authorization: Bearer <token>"
        ```

        Locate the `id` field for your Remote Execution Deployment in the response.
      </Step>

      <Step title="Create agent token">
        Make a POST request to create the token.  Replace the `organizationId`, `deploymentId`, and `your-API-Deployment-token` placeholders with your own values. You can optionally set a description and expiration period for the token.This example sets a 30-day expiration:

        ```sh wrap theme={null}
        curl -X POST https://api.astronomer.io/iam/v1beta1/organizations/<organizationId>/deployments/<deploymentId>/agent-tokens \
          -H "Authorization: Bearer <your-API-Deployment-token>" \
          -H "Content-Type: application/json" \
          -d '{
          "description": "Production agent token",
          "name": "prod-agent",
          "tokenExpiryPeriodInDays": 30
        }'
        ```
      </Step>

      <Step title="Save token">
        Copy the `token` field from the response and save it securely.
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Step 2: Install Helm chart

<Info>
  Astronomer recommends pulling both the Remote Execution Agent image and the Sentinel image and storing them in your private registry. Sentinel provides advanced monitoring and reporting for Remote Execution Agents, starting from version 1.2.0. The Agent base images are minimal, so you might need to add packages for your pipelines to function properly. Use either an [Organization API token](/docs/astro/organization-api-tokens) with the `Org Owner` role or a [Deployment API token](/docs/astro/deployment-api-tokens) with the `Deployment Admin` role to authenticate.
</Info>

<Steps>
  <Step title="Download values file">
    1. In the Astro UI, go to the **Remote Agents** tab
    2. Toggle to the **Agents** view
    3. Click **Register a Remote Agent**
    4. Click **Download** to get the `values.yaml` file
  </Step>

  <Step title="Configure required values">
    Update the following values in `values.yaml`. All other values have working defaults.

    You must configure these values before installing the Helm chart:

    * `agentToken`, `agentTokenSecretName`, or `agentTokenFile` - See [Agent token configuration](#agent-token-configuration)
    * `imagePullSecretName` or `imagePullSecretData` - See [Image pull secret configuration](#image-pull-secret-configuration)
    * `namespace` - Kubernetes namespace for agent deployment
    * `resourceNamePrefix` - Name prefix for Kubernetes resources
    * `secretBackend` - Must be configured before agents can execute tasks. See [Configure secrets backend](/docs/astro/remote-execution-configure-secrets-backend)
    * `xcomBackend` - Must be configured before agents can execute tasks. See [Configure XCom backend](/docs/astro/remote-execution-configure-xcom-backend)
    * `stateStoreBackend` - Required on Astro Runtime 3.3 and later, where agents don't start without it. Helm chart 2.3.0 and later set a working default that isn't suitable for production. See [Configure state store backend](/docs/astro/remote-execution-configure-state-store-backend)

    See the Helm chart comments and [Helm chart configuration reference](/docs/astro/remote-agents-helm-reference) for descriptions of values.
  </Step>

  <Step title="Pull agent image for private registries">
    If self-hosting the image, log in to the image registry with your Deployment API token:

    ```sh wrap theme={null}
    docker login images.astronomer.cloud -u cli -p <your-token>
    ```

    <Note>
      **Sentinel image available with 1.2.0 and later**

      Starting with Remote Execution Agent 1.2.0, a Sentinel image is published alongside the agent images to provide monitoring for Remote Execution Agents. The Sentinel image must be pulled separately. Astronomer recommends enabling Sentinel for all deployments. To enable Sentinel, configure the service in your `values.yaml` file. See [Sentinel for Remote Execution Agents](/docs/astro/remote-agents-sentinel).
    </Note>

    After you log in, you can pull the Remote Execution Agent and Sentinel images directly. To find the latest version and image path, refer to the [Remote Execution Agent release notes](/docs/astro/agent-release-notes) for all currently hosted images and [Remote Execution Agent image reference](/docs/astro/agent-images) for their full URLs. For example:

    ```sh wrap theme={null}
    docker pull images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.0
    ```

    ```sh wrap theme={null}
    docker pull images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.2.0
    ```

    <Tip>
      **Configure scope for registry proxies**

      If you use JFrog Artifactory or a similar registry management tool to mirror or proxy `images.astronomer.cloud`, you need to configure specific include patterns instead of using the default `**/*` pattern.

      The Deployment API token has limited scope and cannot fetch manifests for all repositories. Configure your remote registry to include only these specific paths:

      * `baseimages/astro-remote-execution-agent`
      * `baseimages/astro-remote-execution-sentinel`

      Without these specific patterns, you might encounter `403 Forbidden` errors when JFrog attempts to crawl all repositories in the registry.
    </Tip>

    Pull the Remote Execution Agent image, apply customizations that your dags require, and push it to your private registry. Then update the `values.yaml` file to reference your customized image.
  </Step>

  <Step title="Install Helm chart">
    <Warning>
      You must configure `secretBackend` in your `values.yaml` before running the Helm install. The installation fails if `secretBackend` has no value. See [Configure secrets backend](/docs/astro/remote-execution-configure-secrets-backend).
    </Warning>

    Run the following commands to install the agent:

    ```sh wrap theme={null}
    helm repo add astronomer https://helm.astronomer.io
    helm repo update
    helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Step>
</Steps>

## Step 3: Optionally set allowed IP ranges

Restrict Deployment access to specific IP address ranges for additional security or network isolation between environments.

<Steps>
  <Step title="Open Deployment settings">
    In the Astro UI, click the options menu for your Deployment and select **Edit**.
  </Step>

  <Step title="Add IP ranges">
    1. In the **Advanced** section, click **+Add IP**.
    2. Enter an IP address range in CIDR format.
    3. Click **Add**.
    4. Repeat to add multiple ranges.
  </Step>
</Steps>

## Step 4: Verify agent heartbeat

Confirm the agent is connected and healthy.

<Steps>
  <Step title="Check agent status">
    In the Astro UI, go to the **Remote Agents** tab. A healthy agent shows:

    * Health status: **Healthy**
    * Last heartbeat: Within the past minute

    You can also verify locally that all agent client deployment Pods are running with `kubectl get pods -n <namespace>`. For more in-depth validation, check pod logs for heartbeat activity.

    To verify that your agents can communicate with your Astro Orchestration plane:

    1. Connect to a host or Pod within your VPC that has your Remote Execution Agent running.

    2. Run a DNS lookup to confirm the hostname resolves successfully:

       ```sh wrap theme={null}
       nslookup <AstroClusterId>.external.astronomer.run
       ```

       The response should show the Astro cluster's public load balancer's public IP addresses, or the private IP addresses assigned to your VPC Endpoint if you configured [AWS PrivateLink](/docs/astro/remote-agents-aws-privatelink).

    3. Test connectivity to the endpoint:

       ```sh wrap theme={null}
       curl -v https://<AstroClusterId>.external.astronomer.run
       ```

       The expected response is `404 page not found`. A successful connection confirms your Remote Execution Agents are able to communicate with the Astro orchestration plane over a public connection or via your private VPC endpoint.

    <Tip>
      Temporarily remove any configured [allowed IP ranges](#step-3-optionally-set-allowed-ip-ranges) if the agent is not starting up and reporting Healthy. If connecting using a public connection, your network team may need to allowlist the Astro cluster's public load balancer's public IP addresses (step 2) for outbound access from your VPC.
    </Tip>
  </Step>

  <Step title="Configure dag bundles">
    After verifying agent health, configure how agents access DAG code. See [Configure DAG sources](/docs/astro/remote-execution-configure-dag-sources).
  </Step>

  <Step title="Run test dag">
    Trigger a test DAG run to verify the agent executes tasks successfully.

    <Warning>
      If you expect tasks to run longer than the default grace period of 10 minutes, update the `terminationGracePeriodSeconds` parameter for your workers in `values.yaml`. This ensures that worker Pods have enough time to finish existing tasks before terminating. See [Worker resource configuration](/docs/astro/remote-agents-helm-reference#worker-resource-configuration).
    </Warning>
  </Step>
</Steps>

<Note>
  **HTTP/HTTPS proxy server support**

  Starting with Remote Execution Agent 1.3.2, the agents support running behind an HTTP(S) proxy server. Configure proxy settings using the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment variables.

  For Remote Execution Agent versions earlier than 1.3.2, proxy servers are not supported. If your Kubernetes environment automatically adds a proxy configuration to Pods, the agents will fail to establish an outbound connection to the orchestration plane. You might see errors similar to these in worker logs:

  * `"exc_type":"ReadError","exc_value":"[Errno 104] Connection reset by peer"`
  * `"exc_type":"HTTPStatusError","exc_value":"Client error '400 Bad Request' for url ...`

  **Workaround:** Remove the proxy configuration from the agent Pods, or upgrade to Agent 1.3.2 or later.
</Note>

## Agent token configuration

Provide the agent token using one of these methods:

### agentToken

Store the token directly in `values.yaml`:

```yaml title="values.yaml" wrap theme={null}
agentToken: "<your-token-value>"
```

<Warning>
  Storing tokens directly in values files exposes them in version control. Use `agentTokenSecretName` or `agentTokenFile` for better security.
</Warning>

### agentTokenSecretName

Reference an existing Kubernetes secret containing the token:

```sh wrap theme={null}
kubectl -n <namespace> create secret generic agent-token \
  --from-file=token=token.txt
```

In `values.yaml`:

```yaml title="values.yaml" wrap theme={null}
agentTokenSecretName: "agent-token"
```

### agentTokenFile

Mount a file containing the token. The agent reads the token at runtime:

```yaml title="values.yaml" wrap theme={null}
agentTokenFile: "/path/to/token/file"
```

## Image pull secret configuration

Configure image pull secrets to authenticate with your container registry. The configuration differs depending on whether you pull images directly from Astronomer's registry or from a self-hosted registry.

<Warning>
  The image pull secret requires an **Astro API token**, not an agent token. Use either an [Organization API token](/docs/astro/organization-api-tokens) with the `Org Owner` role or a [Deployment API token](/docs/astro/deployment-api-tokens) with the `Deployment Admin` role. The agent token created in Step 1 authenticates the agent to the Astro orchestration plane and cannot be used for pulling images.
</Warning>

<Tabs>
  <Tab title="Astronomer registry">
    Use this configuration when pulling images directly from `images.astronomer.cloud`.

    ### imagePullSecretName (Astronomer registry)

    Reference an existing Kubernetes secret in your namespace:

    ```sh wrap theme={null}
    kubectl create secret docker-registry -n <namespace> <secretName> \
      --docker-server=images.astronomer.cloud \
      --docker-username=cli \
      --docker-password=<your-astro-api-token>
    ```

    In `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    imagePullSecretName: "<secretName>"
    ```

    ### imagePullSecretData (Astronomer registry)

    Alternatively, provide Docker config JSON directly. The Helm chart creates a secret named `image-pull-secret`:

    ```yaml title="values.yaml" wrap theme={null}
    imagePullSecretData: |
      {
        "auths": {
          "images.astronomer.cloud": {
            "auth": "<base64-encoded-credentials>",
            "email": "<email>"
          }
        }
      }
    ```
  </Tab>

  <Tab title="Self-hosted registry">
    Use this configuration when pulling images from a self-hosted registry, proxy, or mirror.

    <Note>
      If you use a proxy or mirror for `images.astronomer.cloud`, you still need an Astro API token to authenticate with the upstream Astronomer registry. Configure this in your proxy settings. See [Pull agent image for private registries](#image-pull-secret-configuration).
    </Note>

    ### imagePullSecretName (self-hosted registry)

    Reference an existing Kubernetes secret in your namespace:

    ```sh wrap theme={null}
    kubectl create secret docker-registry -n <namespace> <secretName> \
      --docker-server=<your-registry-endpoint> \
      --docker-username=<your-registry-username> \
      --docker-password=<your-registry-password>
    ```

    In `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    imagePullSecretName: "<secretName>"
    ```

    ### imagePullSecretData (self-hosted registry)

    Alternatively, provide Docker config JSON directly. The Helm chart creates a secret named `image-pull-secret`:

    ```yaml title="values.yaml" wrap theme={null}
    imagePullSecretData: |
      {
        "auths": {
          "<your-registry-endpoint>": {
            "auth": "<base64-encoded-username:password>",
            "email": "<email>"
          }
        }
      }
    ```
  </Tab>
</Tabs>

## Manage Remote Execution Agents

You can take the following actions on your registered Remote Execution Agents:

* **Cordon:** Cordoning a Remote Execution Agent marks it as unavailable for scheduling new tasks, while allowing it to continue running and complete any tasks already in progress.

This allows you to gracefully remove the Agent from service without interrupting current workloads. For example, you can cordon an Agent to delete or perform maintenance, such as an upgrade, on the Agent or underlying infrastructure.

A cordoned Agent will not receive new work, but it remains active until all running tasks have finished. Once ready to reintroduce the Agent to the task pool, it can be uncordoned to resume normal operation.

* **Uncordon:** Uncordoning a Remote Execution Agent re-enables it to receive new tasks and resume normal scheduling.

* **Delete:** Deletes the Remote Execution Agent from the Deployment.

## Remote Execution Agent maintenance policy

Each Remote Execution Agent minor version is maintained for **6 months** from the release month.

See [Agent maintenance policy](/docs/astro/agent-maintenance-policy) for more details about versioning, support, and upgrade recommendations.

## Next steps

After registering agents, configure the required components:

* [Configure secrets backend](/docs/astro/remote-execution-configure-secrets-backend) - Required for agent operation
* [Configure XCom backend](/docs/astro/remote-execution-configure-xcom-backend) - Required for passing data between tasks
* [Configure DAG sources](/docs/astro/remote-execution-configure-dag-sources) - Required for accessing DAG code

## Related documentation

* [Remote Execution overview](/docs/astro/remote-execution-overview)
* [Remote Execution Agent failure scenarios](/docs/astro/remote-agents-failure-scenarios)
* [Helm chart configuration reference](/docs/astro/remote-agents-helm-reference)
* [Install agents in restricted Kubernetes namespaces](/docs/astro/remote-agents-restricted-kubernetes)


# Configure DAG sources
Source: https://astronomer.io/docs/astro/remote-execution-configure-dag-sources

Configure GitDagBundle or LocalDagBundle as the dag source for Remote Execution Agents.

Remote Execution Agents require configuration to access your DAG code. This guide covers configuring DAG bundles, which are collections of DAG files and supporting code introduced in Airflow 3.

<Note>
  This feature requires Airflow 3.x Deployments. Configuring multiple DAG bundles in a single Deployment is only supported in Remote Execution mode.
</Note>

## DAG bundle types

Choose between two types of DAG bundles:

* **GitDagBundle**: Dags stored in a Git repository (recommended for production)
* **LocalDagBundle**: Dags stored in the container image or persistent volume (default)

### When to use each bundle type

**Use GitDagBundle when**:

* Running production deployments
* Tracking DAG versions with full rerun capabilities
* Storing dags in version control systems
* Managing multiple teams or DAG repositories

**Use LocalDagBundle when**:

* Running development or testing environments
* Building dags into container images
* Using existing PVC-based DAG management
* Preferring simpler configuration

See [GitDagBundle compared to LocalDagBundle](#gitdagbundle-compared-to-localdagbundle) for functional differences.

## Dag hashing

The Dag processor computes a hash of each Dag and caches it. On each processing cycle, the Dag processor compares the current hash of a Dag against the cached value:

* If the hash matches, the Dag is unchanged, so the Dag processor skips it instead of re-sending it to the Astro orchestration plane.
* If the Dag is new, changed, or not yet in the cache, the Dag processor processes it and updates the cache.

Because the Dag processor skips unchanged Dags, Dag hashing lowers memory utilization and network bandwidth, and speeds up Dag updates in Deployments with many Dags.

To monitor cache behavior, use the `dag_processor_cache_hits_total`, `dag_processor_cache_misses_total`, and `dag_processor_cache_size` metrics. See [Dag processor metrics](/docs/astro/remote-agents-metrics#dag-processor-metrics).

<Note>
  Dag hashing functionality is enabled by default, starting on Astro Agent client release `1.8.0` and later. You can disable that feature at any time with the following environment variable: `ASTRO_AGENT_CLIENT_DAG_PROCESSOR__ENABLE_DAG_CACHING=False`.
</Note>

## Configure GitDagBundle

GitDagBundle fetches dags from Git repositories and provides automatic versioning capabilities.

<Tip>
  GitDagBundle is recommended for production Remote Execution deployments.
</Tip>

### Supported authentication methods

GitDagBundle supports the following authentication methods:

* Access tokens (personal access tokens, OAuth tokens)
* SSH keys
* SSH agent

Choose the method that aligns with your security requirements and infrastructure.

### Required token permissions by provider

When you create an access token, grant the minimum permissions required to read repository contents:

| Provider  | Token type       | Required permissions                                                                            |
| --------- | ---------------- | ----------------------------------------------------------------------------------------------- |
| GitHub    | Fine-grained PAT | `Contents: Read-only` on the target repository. `Metadata: Read-only` is included automatically |
| GitHub    | Classic PAT      | `repo` scope for private repositories. Public repositories require no scope                     |
| GitLab    | PAT              | `read_repository`                                                                               |
| Bitbucket | App password     | `Repositories: Read`                                                                            |

### Configure public repository

For public repositories, no authentication configuration is required. Configure only the repository URL and tracking reference:

```yaml title="values.yaml" wrap theme={null}
dagBundleConfigList: '[{"name": "public-dags", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"repo_url": "https://github.com/your-org/public-dags", "tracking_ref": "main", "subdir": "dags"}}]'
```

### Configure private repository

For private repositories, configure both the DAG bundle and an Airflow connection for authentication.

<Steps>
  <Step title="Create Git connection">
    Add an Airflow connection environment variable in `values.yaml`. The connection name suffix must match the `git_conn_id` value in your DAG bundle configuration.

    <Tabs>
      <Tab title="Access token (HTTPS)">
        Use this method with a Personal Access Token (PAT) or OAuth token. Set `login` to your Git username and `password` to the token value.

        ```yaml title="values.yaml" wrap theme={null}
        commonEnv:
          - name: AIRFLOW_CONN_GIT_REPO
            value: >-
              {
                "conn_type": "git",
                "login": "<git-username>",
                "password": "<personal-access-token>",
                "host": "github.com",
                "schema": "https",
                "extra": {
                  "repo": "<your-org>/<private-repo>",
                  "branch": "main"
                }
              }
        ```

        See [Required token permissions by provider](#required-token-permissions-by-provider) for the minimum permissions each provider requires.
      </Tab>

      <Tab title="SSH key (deploy key)">
        Use this method with an SSH deploy key. Set `login` to `git` and provide the private key in the `extra` field.

        ```yaml title="values.yaml" wrap theme={null}
        commonEnv:
          - name: AIRFLOW_CONN_GIT_REPO
            value: >-
              {
                "conn_type": "git",
                "login": "git",
                "host": "<your-git-host>",
                "schema": "ssh",
                "extra": {
                  "private_key": "<private-ssh-key>"
                }
              }
        ```
      </Tab>
    </Tabs>

    <Note>
      The connection name `AIRFLOW_CONN_GIT_REPO` creates a connection with ID `git_repo`. This ID must match the `git_conn_id` value in your DAG bundle configuration.
    </Note>

    For production environments, store connection credentials in a [secrets backend](/docs/astro/remote-execution-configure-secrets-backend) instead of `values.yaml`. See [Use a secrets backend for Git credentials](#use-a-secrets-backend-for-git-credentials) for an example using Azure Key Vault.
  </Step>

  <Step title="Configure DAG bundle">
    Configure the DAG bundle with matching `git_conn_id`:

    ```yaml title="values.yaml" wrap theme={null}
    dagBundleConfigList: '[{"name": "private-dags", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"tracking_ref": "main", "subdir": "dags", "repo_url": "https://github.com/<your-org>/<private-repo>.git", "git_conn_id": "git_repo"}}]'
    ```

    Note that `git_conn_id: "git_repo"` matches the connection ID from the `AIRFLOW_CONN_GIT_REPO` environment variable.
  </Step>

  <Step title="Update Helm release">
    Apply the configuration:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Step>
</Steps>

### Configure refresh interval

Control how frequently agents check for repository updates using the `refresh_interval` parameter:

```yaml title="values.yaml" wrap theme={null}
dagBundleConfigList: '[{"name": "private-dags", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"repo_url": "https://github.com/your-org/private-dags", "tracking_ref": "main", "subdir": "dags", "git_conn_id": "git_repo", "refresh_interval": 300}}]'
```

The default refresh interval is 300 seconds. Reducing this value across many bundles may increase the risk of hitting Git provider rate limits.

### Use a secrets backend for Git credentials

For production environments, use a secrets backend to store Git connection credentials instead of hardcoding them in `values.yaml`. The following example shows how to configure Azure Key Vault with workload identity authentication on Azure AKS.

<Steps>
  <Step title="Configure Azure Key Vault as secrets backend">
    Add the secrets backend configuration to your `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    secretBackend: airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend

    commonEnv:
      - name: AIRFLOW__SECRETS__BACKEND_KWARGS
        value: '{"connections_prefix": "airflow-connection", "variables_prefix": "airflow-variable", "vault_url": "<your-vault-url>", "workload_identity_tenant_id": "<your-tenant-id>", "managed_identity_client_id": "<your-managed-identity-client-id>"}'
    ```

    This configuration uses Azure workload identity for authentication, which is the recommended approach for Azure AKS environments. For other authentication methods, see [Azure Key Vault secrets backend](/docs/astro/secrets-backend/azure-key-vault).
  </Step>

  <Step title="Store Git connections in Azure Key Vault">
    Create secrets in Azure Key Vault for each Git connection. The secret name must follow the pattern `<connections_prefix>-<connection-id>`. For example, to create a connection with ID `git-repo1-conn`:

    1. In Azure Key Vault, create a secret named `airflow-connection-git-repo1-conn`.
    2. Set the secret value to a JSON connection string:

    ```json wrap theme={null}
    {
      "conn_type": "git",
      "login": "<git-username>",
      "password": "<personal-access-token>",
      "host": "github.com",
      "schema": "https",
      "extra": {
        "repo": "<your-org>/<your-repo>",
        "branch": "main"
      }
    }
    ```

    Repeat this process for each Git repository connection you need.
  </Step>

  <Step title="Configure DAG bundles with connection references">
    Configure your DAG bundles to reference the connections stored in Azure Key Vault:

    ```yaml title="values.yaml" wrap theme={null}
    dagBundleConfigList: '[
      {
        "name": "dags-folder",
        "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
        "kwargs": {
          "repo_url": "https://github.com/<your-org>/<repo-1>.git",
          "tracking_ref": "main",
          "subdir": "dags",
          "refresh_interval": 120,
          "git_conn_id": "git-repo1-conn"
        }
      },
      {
        "name": "dags-folder2",
        "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
        "kwargs": {
          "repo_url": "https://github.com/<your-org>/<repo-2>.git",
          "tracking_ref": "main",
          "subdir": "dags",
          "refresh_interval": 120,
          "git_conn_id": "git-repo2-conn"
        }
      }
    ]'
    ```

    The `git_conn_id` values must match the connection IDs you created in Azure Key Vault (without the `airflow-connection-` prefix).
  </Step>

  <Step title="Update Helm release">
    Apply the configuration:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Step>
</Steps>

## Configure LocalDagBundle

LocalDagBundle reads dags from the local filesystem. This is the default dag bundle type.

### DAG storage options

Choose one of two methods to provide dags to agents:

**Option 1: Include dags in container image**

Build a custom agent image that includes your DAG files. Copy dags into the `/dags` folder during image build.

**Option 2: Mount Persistent Volume Claim**

Create a PVC containing your dags and mount it into all agent components (Dag Processor, Worker, and Triggerer) at the same path.

### Configure DAG path

LocalDagBundle looks for dags in `/dags` by default. Specify a different path using the `path` parameter:

```yaml title="values.yaml" wrap theme={null}
dagBundleConfigList: '[{"name": "local-dags", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/opt/airflow/dags"}}]'
```

### Configure with container image

<Steps>
  <Step title="Build custom image">
    Create a Dockerfile extending the base agent image:

    ```dockerfile title="Dockerfile" wrap theme={null}
    FROM images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.0

    # Copy dags into the image
    COPY dags/ /dags/

    # Install additional dependencies if needed
    COPY requirements.txt /tmp/requirements.txt
    RUN pip install -r /tmp/requirements.txt
    ```
  </Step>

  <Step title="Update values file">
    Reference your custom image in `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    workers:
      - name: default-worker
        image: your-registry.example.com/custom-agent:1.0.0

    dagProcessor:
      image: your-registry.example.com/custom-agent:1.0.0

    triggerer:
      image: your-registry.example.com/custom-agent:1.0.0

    dagBundleConfigList: '[{"name": "local-dags", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/dags"}}]'
    ```
  </Step>

  <Step title="Update Helm release">
    Apply the configuration:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Step>
</Steps>

### Configure with Persistent Volume Claim

<Steps>
  <Step title="Create PVC">
    Create a PersistentVolumeClaim in your Kubernetes namespace:

    ```yaml title="pvc.yaml" wrap theme={null}
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: dags-pvc
      namespace: <your-namespace>
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 20Gi
      storageClassName: <your-storage-class>
    ```

    Apply the PVC:

    ```sh wrap theme={null}
    kubectl apply -f pvc.yaml
    ```
  </Step>

  <Step title="Configure volume mounts">
    Update `values.yaml` to mount the PVC into all components:

    ```yaml title="values.yaml" expandable wrap theme={null}
    workers:
      - name: default-worker
        volumes:
          - name: dags-volume
            persistentVolumeClaim:
              claimName: dags-pvc
        volumeMounts:
          - name: dags-volume
            mountPath: /opt/airflow/dags
            readOnly: true

    dagProcessor:
      volumes:
        - name: dags-volume
          persistentVolumeClaim:
            claimName: dags-pvc
      volumeMounts:
        - name: dags-volume
          mountPath: /opt/airflow/dags
          readOnly: true

    triggerer:
      volumes:
        - name: dags-volume
          persistentVolumeClaim:
            claimName: dags-pvc
      volumeMounts:
        - name: dags-volume
          mountPath: /opt/airflow/dags
          readOnly: true

    dagBundleConfigList: '[{"name": "local-dags", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/opt/airflow/dags"}}]'
    ```
  </Step>

  <Step title="Update Helm release">
    Apply the configuration:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Step>
</Steps>

## GitDagBundle compared to LocalDagBundle

Both bundle types support DAG versioning in the Airflow UI, but GitDagBundle provides additional capabilities:

| Scenario                             | LocalDagBundle                                  | GitDagBundle                                          |
| ------------------------------------ | ----------------------------------------------- | ----------------------------------------------------- |
| **Viewing previous DAG runs**        | Displays DAG as it existed at run time          | Displays DAG as it existed at run time                |
| **Creating new DAG runs**            | Uses current DAG code                           | Uses current DAG code                                 |
| **Rerunning whole previous DAG run** | Uses current DAG code                           | Uses DAG version from original run time               |
| **Rerunning individual tasks**       | Uses latest version for rerun tasks             | Uses task code from original run time                 |
| **Code changes during DAG run**      | Uses current DAG code at task start time        | Completes using bundle version from run start         |
| **Running backfills**                | Uses current DAG code                           | Uses latest bundle version                            |
| **Version creation**                 | Every structural DAG change creates new version | Every committed structural change creates new version |

## DAG versioning

Airflow 3 automatically tracks DAG versions when you use DAG bundles. Each DAG run associates with a specific DAG version visible in the Airflow UI.

Key behaviors:

* New versions are created for structural changes (tasks, dependencies, schedules)
* The scheduler uses the latest DAG version to create new runs
* You can view code for any previous DAG version in the UI
* GitDagBundle allows rerunning tasks with their original code version

See [Airflow DAG versioning](/docs/learn/airflow-dag-versioning) for detailed information about versioning behavior.

## Next steps

After configuring DAG sources:

* [Deploy Remote Execution project](/docs/astro/deploy-project-remote-execution) - Build and deploy your Airflow project
* [Configure logging](/docs/astro/remote-execution-logging-overview) - Set up task log collection
* [Configure OpenLineage](/docs/astro/remote-execution-configure-openlineage) - Enable data lineage tracking

## Related documentation

* [Remote Execution overview](/docs/astro/remote-execution-overview)
* [DAG versioning documentation](/docs/astro/dag-versioning)
* [Secrets backend configuration](/docs/astro/remote-execution-configure-secrets-backend)


# Configure OpenLineage for a Remote Execution Agent
Source: https://astronomer.io/docs/astro/remote-execution-configure-openlineage

Configure an OpenLineage API key on your Remote Execution Agent so that lineage events flow to Astro Observe and Astro Alerts.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

OpenLineage enables you to access data lineage and provenance across your Airflow workflows for your Remote Execution Agent. Features like [Observe](/docs/astro/astro-observe) and [Astro Alerts](/docs/astro/alerts) require that you enable OpenLineage for your data pipelines.

When you create your Remote Execution Agent, Astro automatically generates a Helm `values.yaml` file with OpenLineage configurations pre-filled. To set up OpenLineage, you need to configure an access credential for OpenLineage. This can be an [Astro Deployment API token](/docs/astro/deployment-api-tokens) used as your OpenLineage API key. There are three methods you can use to add your API key to your Helm values:

* Method 1 - Configure the API key as plain text. This stores your API key in your `values.yaml` file as plaintext, which is the simplest but least secure option. It's appropriate for development or testing environments.
* Method 2 - Use a pre-created Kubernetes secret. This procedure stores your API key separately from your `values.yaml` file, which provides more security than storing as plaintext. This option provides security with standard Kubernetes features.
* Method 3 - Inject your API key with a secrets manager. This approach uses an init container to inject the Agent token into the Remote Execution Agent component Pods. This example uses the [HashiCorp Vault](https://developer.hashicorp.com/vault) Agent, but you can use your own secrets manager. This option provides enhanced security with the potential for secret rotation.

## Prerequisites

* A Kubernetes cluster
* [Helm](https://helm.sh/docs/helm/helm_install/)
* The `values.yaml` file downloaded from the Remote Execution Agent registration modal in the [Astro UI](https://cloud.astronomer.io)
* A [Deployment API Token](/docs/astro/deployment-api-tokens) that uses a [Custom Deployment role](/docs/astro/customize-deployment-roles) with **Observe Ingest** permissions

## Setup

### Step 1: Retrieve your OpenLineage variables

<Steps>
  <Step title="Open the Remote Agent registration">
    In the [Astro UI](https://cloud.astronomer.io), go to the **Deployment** page and choose **Agent**. Then click **Register Remote Agent**.

    #### Download the values file

    Click **Download `values.yaml` file**.
  </Step>
</Steps>

The downloaded Helm values file includes most OpenLineage variables pre-filled, so you only need to configure the OpenLineage API key.

### Step 2: Configure the OpenLineage API key

<Tabs>
  <Tab title="Configure key as plaintext">
    This method stores an Astro [Deployment API token](/docs/astro/deployment-api-tokens) as your OpenLineage API key in plain text in your `values.yaml` file, so that the Remote Execution Agent Helm chart can use it to create a Kubernetes secret named `openlineage-api-key-secret`. This API key is base64-encoded in the Kubernetes secret.

    All Remote Execution Agent components — the worker, Dag processor, and triggerer — use this API key to authenticate with the OpenLineage endpoint.

    <Steps>
      <Step title="Update your values file">
        Add the following OpenLineage configuration to your `values.yaml` file:

        ```yaml title="values.yaml" wrap theme={null}
        openLineage:
          # Enable OpenLineage integration
          enabled: true

          # Set your OpenLineage API key directly in the values file
          apiKey: "<OPENLINEAGE_API_KEY>"

          # Do NOT set apiKeySecret when using apiKey
          # apiKeySecret: ~

          # The following fields are prefilled in the values.yaml downloaded from the Astro UI
          url: "<OPENLINEAGE_URL>"
          namespace: "<ASTRO_DEPLOYMENT_NAMESPACE>"
          endpoint: "<OPENLINEAGE_ENDPOINT>"
          facetsEnvironmentVariables: '<OPENLINEAGE_FACETS_ENV_VARS>'
        ```

        #### Install the Helm chart

        Apply the chart using the `values.yaml` file with the following command:

        ```sh wrap theme={null}
        helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml
        ```
      </Step>
    </Steps>
  </Tab>

  <Tab title="Use pre-created Kubernetes secret">
    When you use this method, the Remote Execution Agent Helm chart doesn't create a new secret for the OpenLineage API key. Instead, it configures all Agent components — the worker, Dag processor, and triggerer — to use the existing secret to authenticate with the OpenLineage endpoint.

    The secret must have a key named `api-key` containing the OpenLineage API key. You can use an Astro [Deployment API token](/docs/astro/deployment-api-tokens) as your OpenLineage API key.

    <Steps>
      <Step title="Create the Kubernetes secret">
        Create a Kubernetes secret containing your OpenLineage API key:

        ```sh wrap theme={null}
        kubectl create secret generic openlineage-api-key-secret \
          --from-literal=api-key=<OPENLINEAGE_API_KEY> \
          --namespace <YOUR_NAMESPACE>
        ```

        #### Reference the secret from your values file

        Configure your `values.yaml` file so that OpenLineage uses your pre-created secret:

        ```yaml title="values.yaml" wrap theme={null}
        openLineage:
          # Enable OpenLineage integration
          enabled: true

          # Do NOT set apiKey when using apiKeySecret
          # apiKey: ~

          # Reference the pre-created secret you created in the previous step
          apiKeySecret: "openlineage-api-key-secret"

          # The following fields are prefilled in the values.yaml downloaded from the Astro UI
          url: "<OPENLINEAGE_URL>"
          namespace: "<ASTRO_DEPLOYMENT_NAMESPACE>"
          endpoint: "<OPENLINEAGE_ENDPOINT>"
          facetsEnvironmentVariables: '<OPENLINEAGE_FACETS_ENV_VARS>'
        ```

        #### Install the Helm chart with the referenced secret

        Apply the chart using the `values.yaml` file with the following command:

        ```sh wrap theme={null}
        helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml
        ```
      </Step>
    </Steps>
  </Tab>

  <Tab title="Use secrets manager">
    You can also use a secrets manager to securely store your API keys. The following procedure specifically uses the [HashiCorp Vault Agent](https://developer.hashicorp.com/vault/tutorials/vault-agent).

    The Vault Agent init container runs before the main Remote Execution Agent container. The Vault Agent authenticates with Vault to retrieve the OpenLineage API key and then writes the API key to a file in the shared volume. The Remote Execution Agent container can then read the OpenLineage API key from the file using the `OPENLINEAGE_API_KEY` environment variable. Use your Astro [Deployment API token](/docs/astro/deployment-api-tokens) as your OpenLineage API key.

    <Steps>
      <Step title="Configure OpenLineage and init containers">
        Configure OpenLineage and add init containers for the `dagProcessor`, `workers`, and `triggerer` components:

        ```yaml title="values.yaml" expandable wrap theme={null}
        openLineage:
          # Enable OpenLineage integration
          enabled: true

          # Don't set apiKey or apiKeySecret
          # apiKey: ~
          # apiKeySecret: ~

          # The following fields are prefilled in the values.yaml downloaded from the Astro UI
          url: "<OPENLINEAGE_URL>"
          namespace: "<ASTRO_DEPLOYMENT_NAMESPACE>"
          endpoint: "<OPENLINEAGE_ENDPOINT>"
          facetsEnvironmentVariables: '<OPENLINEAGE_FACETS_ENV_VARS>'

        # Configure each component to use the Vault init container
        dagProcessor:
          initContainers:
            - name: vault-openlineage
              image: hashicorp/vault:1.13.1
              command: ["/bin/sh", "-c"]
              args:
                - |
                  export VAULT_ADDR=https://vault.example.com
                  vault agent -config=/vault/config/agent.hcl
              volumeMounts:
                - name: vault-config
                  mountPath: /vault/config
                - name: openlineage-volume
                  mountPath: /vault/secrets
          volumes:
            - name: vault-config
              configMap:
                name: vault-agent-config
            - name: openlineage-volume
              emptyDir:
                medium: Memory
          volumeMounts:
            - name: openlineage-volume
              mountPath: /vault/secrets
          env:
            - name: OPENLINEAGE_API_KEY
              valueFrom:
                fileRef:
                  path: /vault/secrets/openlineage-api-key

        triggerer:
          initContainers:
            - name: vault-openlineage
              image: hashicorp/vault:1.13.1
              command: ["/bin/sh", "-c"]
              args:
                - |
                  export VAULT_ADDR=https://vault.example.com
                  vault agent -config=/vault/config/agent.hcl
              volumeMounts:
                - name: vault-config
                  mountPath: /vault/config
                - name: openlineage-volume
                  mountPath: /vault/secrets
          volumes:
            - name: vault-config
              configMap:
                name: vault-agent-config
            - name: openlineage-volume
              emptyDir:
                medium: Memory
          volumeMounts:
            - name: openlineage-volume
              mountPath: /vault/secrets
          env:
            - name: OPENLINEAGE_API_KEY
              valueFrom:
                fileRef:
                  path: /vault/secrets/openlineage-api-key

        workers:
          initContainers:
            - name: vault-openlineage
              image: hashicorp/vault:1.13.1
              command: ["/bin/sh", "-c"]
              args:
                - |
                  export VAULT_ADDR=https://vault.example.com
                  vault agent -config=/vault/config/agent.hcl
              volumeMounts:
                - name: vault-config
                  mountPath: /vault/config
                - name: openlineage-volume
                  mountPath: /vault/secrets
          volumes:
            - name: vault-config
              configMap:
                name: vault-agent-config
            - name: openlineage-volume
              emptyDir:
                medium: Memory
          volumeMounts:
            - name: openlineage-volume
              mountPath: /vault/secrets
          env:
            - name: OPENLINEAGE_API_KEY
              valueFrom:
                fileRef:
                  path: /vault/secrets/openlineage-api-key
        ```

        #### Create a ConfigMap for the Vault Agent

        ```sh wrap theme={null}
        cat <<EOF | kubectl apply -f -
        apiVersion: v1
        kind: ConfigMap
        metadata:
          name: vault-agent-config
          namespace: <YOUR_NAMESPACE>
        data:
          agent.hcl: |
            auto_auth {
              method "kubernetes" {
                mount_path = "auth/kubernetes"
                config = {
                  role = "astro-agent"
                }
              }
            }

            template {
              destination = "/vault/secrets/openlineage-api-key"
              contents = "{{ with secret \"secret/data/openlineage/api-key\" }}{{ .Data.data.key }}{{ end }}"
            }
        EOF
        ```

        #### Install the Helm chart with the Vault Agent

        Apply the chart with the `values.yaml` file:

        ```sh wrap theme={null}
        helm install astro-agent astronomer/astro-remote-execution-agent -f values.yaml
        ```
      </Step>
    </Steps>

    Read more about [secrets backends on Astro](/docs/astro/secrets-backend).
  </Tab>
</Tabs>

### Step 3: Set OpenLineage environment variables on the orchestration plane

<Steps>
  <Step title="Open the Deployment">
    In the Astro UI, open your Deployment.

    #### Open the Environment tab

    Click the **Environment** tab.

    #### Add the OpenLineage variable

    Click **Edit Deployment Variables** (or **+ New Environment Variable** if you have no variables configured yet), then click **Add Variable** and add the following environment variable:

    ```sh wrap theme={null}
    OPENLINEAGE_DISABLED=False
    ```

    #### Apply changes

    Click **Update Environment Variables** to save your changes.
  </Step>
</Steps>

Setting this variable ensures that all required OpenLineage events, including task and Dag run events, are collected from the scheduler, workers, Dag processor, and triggerer components. This provides complete lineage in Observe and Astro Alerts.


# Configure secrets backend
Source: https://astronomer.io/docs/astro/remote-execution-configure-secrets-backend

Configure AWS, Azure, GCP, or HashiCorp secrets backends for Remote Execution Agents.

Remote Execution Agents require a secrets backend to securely access Airflow connections and variables. A secrets backend stores sensitive information like passwords, API keys, and database credentials outside of your Deployment and agent configuration files.

`secretBackend` is the Airflow secrets backend class to use for the Agent. Each supported integration includes the Remote Execution Agent-specific implementation steps:

* [AWS Secrets Manager](/docs/astro/secrets-backend/aws-secretsmanager#remote-execution)
* [AWS Systems Manager (SSM) Parameter Store](/docs/astro/secrets-backend/aws-paramstore#remote-execution)
* [Azure Key Vault](/docs/astro/secrets-backend/azure-key-vault#remote-execution)
* [Google Cloud Secret Manager](/docs/astro/secrets-backend/gcp-secretsmanager#remote-execution)
* [HashiCorp Vault](/docs/astro/secrets-backend/hashicorp-vault#remote-execution)

It is not recommended for production use cases, but you can also use Airflow’s local filesystem backend as a simpler alternative to an external secrets provider by setting:

```yaml title="values.yaml" wrap theme={null}
secretBackend: "airflow.secrets.local_filesystem.LocalFilesystemBackend"
```

and setting the following environment variable in the Remote Execution Agent's Helm chart:

```yaml title="values.yaml" wrap theme={null}
AIRFLOW__SECRETS__BACKEND_KWARGS: '{"variables_file_path": "/files/var.json", "connections_file_path": "/files/conn.json"}'
```

<Note>The rest of the configuration should be passed as environment variables to the Agent components.</Note>


# Configure state store backend
Source: https://astronomer.io/docs/astro/remote-execution-configure-state-store-backend

Configure the state store backend that Remote Execution Agents use to store task and asset state on Astro Runtime 3.3 and later.

Starting with Airflow 3.3 (Astro Runtime 3.3), tasks and asset watchers can persist state between runs through the Airflow worker-side state store. Remote Execution Agents store this state in a backend that you control, so it stays in your execution plane. This guide covers the default configuration that the Remote Execution Agent Helm chart applies and how to configure object storage for production.

## Overview

Airflow 3.3 introduces a worker-side state store that tasks and asset watchers use to persist small state values between runs. For example, an asset watcher uses it to track its position in an event stream. By default, Airflow stores this state in the metadata database. With Remote Execution, the metadata database resides in the Astro orchestration plane, so a custom state store backend keeps the data in storage that you control instead. This is the same pattern as the [XCom backend](/docs/astro/remote-execution-configure-xcom-backend).

Agent Client 1.8.0 and later enforce this on Astro Runtime 3.3 and later images: agents don't start until you set the `AIRFLOW__WORKERS__STATE_STORE_BACKEND` environment variable. Runtime versions earlier than 3.3 don't include the state store, and agents apply no state store configuration there.

## Prerequisites

* Remote Execution Agent Helm chart 2.3.0 or later. For earlier chart versions, see [Configure the backend on chart versions earlier than 2.3.0](#configure-the-backend-on-chart-versions-earlier-than-2-3-0).
* For production use, an object storage bucket and workload identity configured for your Kubernetes cluster, as described in [Configure XCom backend](/docs/astro/remote-execution-configure-xcom-backend).

## Default configuration

Remote Execution Agent Helm chart 2.3.0 and later configure the state store backend automatically. The `stateStoreBackend` value defaults to the object storage backend from the Common IO provider:

```yaml title="values.yaml" wrap theme={null}
stateStoreBackend: airflow.providers.common.io.state_store.backend.StateStoreObjectStorageBackend
```

The chart sets `AIRFLOW__WORKERS__STATE_STORE_BACKEND` to this class on every agent component. If you don't configure a storage path yourself, the chart also applies a local file path default:

```text wrap theme={null}
AIRFLOW__COMMON_IO__STATE_STORE_OBJECTSTORAGE_PATH=file:///usr/local/airflow/state-store
```

<Warning>
  **Local path default is for evaluation only**

  With the local file path, each agent Pod writes state to its own container filesystem: state isn't shared between Pods and doesn't survive Pod restarts. Configure shared object storage for production deployments.
</Warning>

## Configure object storage for production

Point the state store at shared object storage by setting the storage path in `commonEnv`:

```yaml title="values.yaml" wrap theme={null}
stateStoreBackend: airflow.providers.common.io.state_store.backend.StateStoreObjectStorageBackend

commonEnv:
  - name: AIRFLOW__COMMON_IO__STATE_STORE_OBJECTSTORAGE_PATH
    value: "s3://<connection-id>@<bucket-name>/<path-to-state-store>"
```

The path format and authentication follow the same patterns as the XCom backend, and you can reuse the same bucket and connection with a different path prefix. See [Configure XCom backend](/docs/astro/remote-execution-configure-xcom-backend) for cloud-specific instructions for AWS S3, Azure Blob Storage, and GCP Cloud Storage, including bucket setup, workload identity, and connection configuration.

After updating your values, apply the configuration:

```sh wrap theme={null}
helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
```

## Configure the backend on chart versions earlier than 2.3.0

Helm chart versions earlier than 2.3.0 don't include the `stateStoreBackend` value. Before you move agents to a Runtime 3.3 image, either upgrade to chart 2.3.0 or later, or set the environment variables on all agent components through `commonEnv`:

```yaml title="values.yaml" wrap theme={null}
commonEnv:
  - name: AIRFLOW__WORKERS__STATE_STORE_BACKEND
    value: "airflow.providers.common.io.state_store.backend.StateStoreObjectStorageBackend"
  - name: AIRFLOW__COMMON_IO__STATE_STORE_OBJECTSTORAGE_PATH
    value: "s3://<connection-id>@<bucket-name>/<path-to-state-store>"
```

## Custom backend classes

If you use a different state store backend class, set `stateStoreBackend` to your class and configure its settings through `commonEnv`. The chart only applies the local path default for the object storage backend.

## Verify configuration

After applying your configuration:

1. Confirm all agent Pods start successfully. On Runtime 3.3 images, an agent fails to start with a validation error if `AIRFLOW__WORKERS__STATE_STORE_BACKEND` isn't set.
2. Trigger a Dag that uses the state store, for example through an asset watcher.
3. Verify that state objects appear under your configured storage path.

## Next steps

* [Configure XCom backend](/docs/astro/remote-execution-configure-xcom-backend) - Object storage for passing data between tasks
* [Configure Dag sources](/docs/astro/remote-execution-configure-dag-sources) - Define how agents access Dag code

## Related documentation

* [Remote Execution overview](/docs/astro/remote-execution-overview)
* [Helm chart configuration reference](/docs/astro/remote-agents-helm-reference)
* [Remote Execution Agent release notes](/docs/astro/agent-release-notes)


# Configure XCom backend
Source: https://astronomer.io/docs/astro/remote-execution-configure-xcom-backend

Configure AWS S3, Azure Blob Storage, or GCP Cloud Storage as the XCom backend for Remote Execution.

Remote Execution Agents require a custom XCom backend to pass data between tasks. This guide covers configuring AWS S3, Azure Blob Storage, or GCP Cloud Storage as your XCom backend.

## Overview

XCom (cross-communication) enables data exchange between Airflow tasks. By default, Airflow stores XCom values in the metadata database. With Remote Execution, the metadata database resides in the Astro orchestration plane while agents execute tasks in your infrastructure.

A custom XCom backend stores XCom data in object storage accessible to all agents in your execution plane. This allows tasks running on different agents to share data through a common storage location.

<Warning>
  **XCom limitations with callbacks**

  In Airflow 3, XCom values are stored externally in storage backends. The database only holds the location of each XCom value, and retrieving the actual XCom data requires a call to the execution API with a valid task token.

  Currently, callbacks do not receive a task token, so they cannot retrieve XCom values from external backends. As a result, using XCom in callbacks is not supported at this time when using Remote Execution with custom XCom backends.
</Warning>

<Info>
  **XCom support in triggers**

  XCom triggerer support is enabled with Agent Client 1.5.0+. Triggers running on the agent can retrieve and push XCom values when using a compatible client version.
</Info>

## Prerequisites

* Remote Execution Agent installed and registered
* Object storage bucket in your cloud provider
* Workload identity configured for your Kubernetes cluster (IRSA for AWS, Managed Identity for Azure, Workload Identity for GCP)
* Required Python packages installed in your agent image (see cloud-specific sections below)

## Configure XCom backend

All XCom backend configurations use the `XComObjectStorageBackend` class from the Common IO provider. Choose your cloud provider below for specific configuration instructions.

<Tabs>
  <Tab title="AWS S3">
    ### Prerequisites (AWS S3)

    The Remote Execution Agent image must include these Python packages:

    * `apache-airflow-providers-amazon`
    * `apache-airflow-providers-common-io`
    * `s3fs`

    ### Create S3 bucket

    Create an S3 bucket for XCom storage with the following recommended settings:

    * **Versioning**: Enable for data recovery
    * **Encryption**: Enable server-side encryption (SSE-S3 or SSE-KMS)
    * **Lifecycle policy**: Configure automatic deletion of old XCom objects

    ### Configure IAM role

    Create an IAM role with permissions to read and write to your XCom bucket. Attach this policy to the role:

    ```json wrap theme={null}
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:GetObject",
            "s3:PutObject",
            "s3:DeleteObject",
            "s3:ListBucket"
          ],
          "Resource": [
            "arn:aws:s3:::your-xcom-bucket",
            "arn:aws:s3:::your-xcom-bucket/*"
          ]
        }
      ]
    }
    ```

    Configure the trust relationship to allow your Kubernetes service accounts to assume the role. See [AWS IRSA documentation](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) for details.

    ### Update Helm values (AWS S3)

    Update your agent's `values.yaml` file with the XCom backend configuration:

    ```yaml title="values.yaml" wrap theme={null}
    xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

    commonEnv:
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
        value: "s3://<connection-id>@<bucket-name>/<path-to-xcom>"
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
        value: "0"  # Always store XComs in object storage
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
        value: "zip"

    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME>
    ```

    Replace:

    * `<connection-id>`: Airflow connection ID for S3 (for example, `aws_xcom`)
    * `<bucket-name>`: Your S3 bucket name
    * `<path-to-xcom>`: Path prefix for XCom objects (for example, `xcom`)
    * `<AWS_ACCOUNT_ID>`: Your AWS account ID
    * `<IAM_ROLE_NAME>`: IAM role name created above

    ### Configure the AWS connection

    The `<connection-id>` in the XCom path must correspond to a configured Airflow connection. You can define this connection directly in `values.yaml` or through a secrets backend.

    <Tabs>
      <Tab title="values.yaml">
        Add an `AIRFLOW_CONN_<CONNECTION_ID>` environment variable to `commonEnv` in your `values.yaml`. The suffix after `AIRFLOW_CONN_` becomes the connection ID. For example, `AIRFLOW_CONN_AWS_XCOM` creates a connection with ID `aws_xcom`:

        ```yaml title="values.yaml" wrap theme={null}
        commonEnv:
          - name: AIRFLOW_CONN_AWS_XCOM
            value: '{"conn_type":"aws"}'
        ```

        Because the agent uses IRSA for authentication, you don't need to specify explicit credentials in the connection. The IAM role annotation on the pod provides access to S3 automatically.
      </Tab>

      <Tab title="Secrets backend">
        For production environments, store the connection in a [secrets backend](/docs/astro/remote-execution-configure-secrets-backend) instead of `values.yaml`. See [Set up AWS Secrets Manager as your secrets backend](/docs/astro/secrets-backend/aws-secretsmanager#remote-execution) or [Set up AWS Systems Manager Parameter Store as your secrets backend](/docs/astro/secrets-backend/aws-paramstore#remote-execution) for configuration steps.

        When using a secrets backend, create a secret that matches your connection ID. For example, if your XCom path uses `aws_xcom` as the connection ID, store the connection under the path corresponding to `aws_xcom` in your secrets backend.
      </Tab>
    </Tabs>

    ### Apply configuration (AWS S3)

    Update your Helm release:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Tab>

  <Tab title="Azure Blob Storage">
    ### Prerequisites (Azure Blob Storage)

    The Remote Execution Agent image must include these Python packages:

    * `apache-airflow-providers-microsoft-azure`
    * `apache-airflow-providers-common-io`

    ### Create storage account and container

    Create an Azure Storage account and container for XCom storage:

    1. Create a storage account with these recommended settings:
       * **Performance**: Standard
       * **Replication**: Locally-redundant storage (LRS) or higher
       * **Secure transfer**: Required
    2. Create a container named `xcom` (or your preferred name)
    3. Note the storage account name

    ### Configure managed identity

    Create a managed identity and grant it access to your storage account:

    1. Create a managed identity in Azure
    2. Assign the **Storage Blob Data Contributor** role to the managed identity on your storage account
    3. Configure workload identity federation to allow your Kubernetes service accounts to use the managed identity

    See [Azure Workload Identity documentation](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) for details.

    ### Update Helm values (Azure Blob Storage)

    Update your agent's `values.yaml` file with the XCom backend configuration:

    ```yaml title="values.yaml" wrap theme={null}
    xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

    commonEnv:
      - name: AIRFLOW_CONN_WASB_XCOM
        value: '{"conn_type":"azure","login":"<storage-account-name>","extra":"{\"anon\": false, \"account_name\":\"<storage-account-name>\",\"managed_identity_client_id\":\"<managed-identity-client-id>\",\"workload_identity_tenant_id\":\"<tenant-id>\"}"}'
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
        value: "abfs://wasb_xcom@<container-name>/<path-to-xcom>"
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
        value: "0"  # Always store XComs in object storage
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
        value: "zip"

    labels:
      azure.workload.identity/use: "true"

    annotations:
      azure.workload.identity/client-id: "<managed-identity-client-id>"
    ```

    Replace:

    * `<storage-account-name>`: Your Azure storage account name
    * `<managed-identity-client-id>`: Client ID of your managed identity
    * `<tenant-id>`: Your Azure tenant ID
    * `<container-name>`: Container name (e.g., `xcom`)
    * `<path-to-xcom>`: Path prefix for XCom objects

    ### Apply configuration (Azure Blob Storage)

    Update your Helm release:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Tab>

  <Tab title="GCP Cloud Storage">
    ### Prerequisites (GCP Cloud Storage)

    The Remote Execution Agent image must include these Python packages:

    * `apache-airflow-providers-google`
    * `apache-airflow-providers-common-io`
    * `gcsfs`

    ### Create GCS bucket

    Create a GCS bucket for XCom storage with the following recommended settings:

    * **Location type**: Region or Multi-region based on your requirements
    * **Storage class**: Standard
    * **Access control**: Uniform (bucket-level permissions)
    * **Encryption**: Google-managed or customer-managed encryption keys

    ### Configure IAM service account

    Create a GCP service account and grant it access to your GCS bucket:

    1. Create a service account in your GCP project
    2. Grant the service account the **Storage Object Admin** role (`roles/storage.objectAdmin`) on your GCS bucket
    3. Configure Workload Identity to allow your Kubernetes service accounts to use the GCP service account

    To link your Kubernetes service account (KSA) to your GCP service account (GSA):

    ```sh wrap theme={null}
    gcloud iam service-accounts add-iam-policy-binding <your-service-account>@<project>.iam.gserviceaccount.com \
      --role roles/iam.workloadIdentityUser \
      --member "serviceAccount:<project>.svc.id.goog[<namespace>/<ksa-name>]"
    ```

    See [GKE Workload Identity documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) for details.

    ### Update Helm values (GCP Cloud Storage)

    Update your agent's `values.yaml` file with the XCom backend configuration:

    ```yaml title="values.yaml" wrap theme={null}
    xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"

    commonEnv:
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
        value: "gs://<bucket-name>/<path-to-xcom>"
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
        value: "0"  # Always store XComs in object storage
      - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION
        value: "zip"

    annotations:
      iam.gke.io/gcp-service-account: <your-service-account>@<project>.iam.gserviceaccount.com
    ```

    Replace:

    * `<bucket-name>`: Your GCS bucket name
    * `<path-to-xcom>`: Path prefix for XCom objects (e.g., `xcom`)
    * `<your-service-account>`: Your GCP service account name
    * `<project>`: Your GCP project ID

    ### Apply configuration (GCP Cloud Storage)

    Update your Helm release:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Tab>
</Tabs>

## Configuration options

### XCom path format

The `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH` parameter defines where XCom objects are stored:

```text wrap theme={null}
<protocol>://<connection-id>@<bucket-or-container>/<path-prefix>
```

* **protocol**: `s3://` for AWS, `abfs://` for Azure, `gs://` for GCP
* **connection-id**: Airflow connection ID for authentication
* **bucket-or-container**: Storage bucket or container name
* **path-prefix**: Optional path prefix for organizing XCom objects

### XCom threshold

The `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD` parameter controls when XCom values are stored in object storage:

* `0`: Always store in object storage (recommended for Remote Execution)
* `>0`: Store in object storage only if value size exceeds threshold in bytes

Set this to `0` to ensure all XCom values are accessible to remote agents.

### Compression

The `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION` parameter controls compression:

* `zip`: Compress XCom values (recommended)
* `None`: No compression

Compression reduces storage costs and transfer time for large XCom values.

## Verify configuration

After applying your configuration, verify XCom backend functionality:

1. Trigger a test DAG that uses XCom to pass data between tasks
2. Check that tasks complete successfully
3. Verify XCom objects appear in your storage bucket
4. Review agent logs for any XCom-related errors

## Next steps

After configuring XCom backend:

* [Configure secrets backend](/docs/astro/remote-execution-configure-secrets-backend) - Store Airflow connections and variables securely
* [Configure state store backend](/docs/astro/remote-execution-configure-state-store-backend) - Store task and asset state on Astro Runtime 3.3 and later
* [Configure dag sources](/docs/astro/remote-execution-configure-dag-sources) - Define how agents access DAG code
* [Configure logging](/docs/astro/remote-execution-logging-overview) - Export task logs to external platforms

## Related documentation

* [XCom backend strategies](/docs/learn/custom-xcom-backend-strategies)
* [Remote Execution overview](/docs/astro/remote-execution-overview)
* [Helm chart configuration reference](/docs/astro/remote-agents-helm-reference)


# Set up custom timetable support for Remote Execution
Source: https://astronomer.io/docs/astro/remote-execution-custom-timetables

Configure the scheduler to access your secrets backend when using custom timetables on Remote Execution.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

When Dags use custom timetables that connect to external data sources, such as querying Snowflake for scheduling metadata, the scheduler must retrieve connection credentials from your secrets backend at schedule time.

When you [configure Customer Managed Identity](/docs/astro/authorize-deployments-to-your-cloud) for a Remote Execution Deployment, the setup only authorizes the apiserver to access your cloud resources for reading task logs. To support custom timetables, the scheduler also needs authorization to access your secrets backend.

Without this configuration, the scheduler can't retrieve connections from the secrets backend, causing errors like:

```text wrap theme={null}
airflow.exceptions.AirflowNotFoundException: The conn_id `<connection-id>` isn't defined
```

## Prerequisites

* A Remote Execution Deployment with [Customer Managed Identity configured](/docs/astro/authorize-deployments-to-your-cloud).
* A [secrets backend configured](/docs/astro/remote-execution-configure-secrets-backend) for your Remote Execution Agent.

## Authorize the scheduler

Extend your existing Customer Managed Identity configuration to include the scheduler service account. This is the same process used when you first configured workload identity for the apiserver.

<Tabs>
  <Tab title="AWS">
    No additional configuration is required. The default Customer Managed Identity setup for AWS uses a wildcard pattern in the IAM trust policy that authorizes all service accounts in the Deployment namespace, including the scheduler:

    ```json wrap theme={null}
    "<clusterOIDCIssuerUrl>:sub": "system:serviceaccount:<deployment-namespace>:*"
    ```

    If you specified individual service accounts instead of using a wildcard, add the scheduler service account to your IAM trust policy:

    ```json wrap theme={null}
    "<clusterOIDCIssuerUrl>:sub": "system:serviceaccount:<deployment-namespace>:<deployment-namespace>-scheduler-serviceaccount"
    ```
  </Tab>

  <Tab title="GCP">
    The Customer Managed Identity setup for GCP authorizes only the apiserver by default. Run the following command to add the scheduler:

    ```sh wrap theme={null}
    gcloud iam service-accounts add-iam-policy-binding \
      --role roles/iam.workloadIdentityUser \
      --member "serviceAccount:<gke-project-id>.svc.id.goog[<deployment-namespace>/<deployment-namespace>-scheduler-serviceaccount]" \
      <your-service-account>@<your-project>.iam.gserviceaccount.com \
      --project <your-project>
    ```

    Replace the following values:

    * `<gke-project-id>`: The GCP project ID of the GKE cluster running your Remote Execution Agent
    * `<deployment-namespace>`: Your Deployment's Kubernetes namespace
    * `<your-service-account>`: The GCP service account configured as your Deployment's Customer Managed Identity
    * `<your-project>`: The GCP project containing your service account

    <Tip>Use the same values from the command you ran when configuring Customer Managed Identity for the apiserver.</Tip>
  </Tab>

  <Tab title="Azure">
    The Customer Managed Identity setup for Azure creates federated identity credentials only for the apiserver by default. Run the following command to add the scheduler:

    ```sh wrap theme={null}
    az identity federated-credential create \
      --name <deployment-namespace>-scheduler \
      --identity-name <managed-identity-name> \
      --resource-group <resource-group> \
      --issuer <aks-oidc-issuer-url> \
      --subject system:serviceaccount:<deployment-namespace>:<deployment-namespace>-scheduler-serviceaccount
    ```

    Replace the following values:

    * `<deployment-namespace>`: Your Deployment's Kubernetes namespace
    * `<managed-identity-name>`: Name of your user-assigned managed identity
    * `<resource-group>`: Resource group containing your managed identity
    * `<aks-oidc-issuer-url>`: The OIDC issuer URL for your AKS cluster, available in the **Customer Managed Identity** modal in the Astro UI

    <Tip>Use the same `--identity-name`, `--resource-group`, and `--issuer` values from the command you ran when configuring Customer Managed Identity for the apiserver.</Tip>
  </Tab>
</Tabs>

## Verify the configuration

After updating your workload identity configuration, verify that the scheduler can retrieve connections:

1. Check the scheduler logs for authentication errors. Cloud-specific errors such as `AADSTS700213: No matching federated identity record found` (Azure) or `AccessDenied` (AWS/GCP) should no longer appear.
2. Trigger a Dag that uses a custom timetable dependent on a connection from your secrets backend. The Dag should schedule without `AirflowNotFoundException` errors.

## See also

* [Authorize an Astro Deployment to cloud resources using workload identity](/docs/astro/authorize-deployments-to-your-cloud)
* [Configure secrets backend for Remote Execution](/docs/astro/remote-execution-configure-secrets-backend)


# Deploy a dbt project to Remote Execution Agents
Source: https://astronomer.io/docs/astro/remote-execution-deploy-dbt

Learn how to make dbt project files available to Remote Execution Agents so that Cosmos can orchestrate dbt models as Airflow tasks.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Remote Execution Agents run tasks in your own Kubernetes infrastructure. Because Remote Execution mode doesn't support `astro dbt deploy`, you must make your dbt project files available to agents through other methods.

This document covers two approaches for deploying dbt code to Remote Execution Agents, ordered by Astronomer's recommendation.

## Prerequisites

* A [Remote Execution Deployment](/docs/astro/remote-execution-overview) with at least one registered [agent](/docs/astro/remote-execution-configure-agents).
* A [secrets backend](/docs/astro/remote-execution-configure-secrets-backend) and [XCom backend](/docs/astro/remote-execution-configure-xcom-backend) configured for your agents.
* A dbt project in a Git repository.
* [Cosmos](https://astronomer.github.io/astronomer-cosmos/) installed in your agent client image for orchestrating dbt models.

<a />

## Option 1: Include the dbt project as a Git submodule (recommended)

Use Git submodules to include an external dbt project repository inside your Dag repository. This approach keeps your dbt and Airflow code in separate repositories while combining them at deploy time.

This is the recommended approach because it:

* Preserves separation of ownership between dbt and Airflow teams.
* Pins the dbt project to a specific commit, giving you control over which version of dbt code runs.
* Works with `GitDagBundle` for automatic Dag versioning.

### How it works

Your Dag repository includes the dbt project as a Git submodule. When agents fetch the Dag bundle from Git, the submodule contents are fetched alongside your Dags. Cosmos then reads the dbt project from the submodule folder to orchestrate dbt models as Airflow tasks.

### Configure the Dag repository

<Steps>
  <Step title="Add the dbt project as a submodule">
    From your Dag repository root, run:

    ```sh wrap theme={null}
    git submodule add <dbt-repository-url> dbt/<project-name>
    ```

    For example, to add a dbt project called `jaffle-shop`:

    ```sh wrap theme={null}
    git submodule add https://github.com/your-org/jaffle-shop dbt/jaffle-shop
    ```

    This creates a `.gitmodules` file and clones the dbt repository into the `dbt/jaffle-shop` folder.
  </Step>

  <Step title="Verify the repository structure">
    Your Dag repository should look similar to the following:

    ```text wrap theme={null}
    dag-repo/
    ├── .gitmodules
    ├── dags/
    │   └── dbt_dag.py
    └── dbt/
        └── jaffle-shop/       # Git submodule
            ├── models/
            ├── seeds/
            └── dbt_project.yml
    ```
  </Step>

  <Step title="Create a Cosmos Dag">
    Create a Dag that uses Cosmos to orchestrate the dbt project. Set the `dbt_project_path` to reference the submodule location within the Dag bundle:

    ```python title="dags/dbt_dag.py" wrap theme={null}
    from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig
    from pathlib import Path
    from datetime import datetime

    dbt_dag = DbtDag(
        project_config=ProjectConfig(
            dbt_project_path=Path("/usr/local/airflow/dags/dbt/jaffle-shop"),
        ),
        profile_config=ProfileConfig(
            profile_name="jaffle_shop",
            target_name="dev",
            profiles_yml_filepath=Path("/usr/local/airflow/dags/dbt/jaffle-shop/profiles.yml"),
        ),
        execution_config=ExecutionConfig(
            dbt_executable_path="/home/astro/dbt_venv/bin/dbt",
        ),
        schedule="@daily",
        start_date=datetime(2024, 1, 1),
        catchup=False,
        dag_id="dbt_jaffle_shop",
    )
    ```

    <Note>
      The exact path depends on your `GitDagBundle` configuration. If your bundle uses a `subdir` parameter, adjust the path accordingly.
    </Note>
  </Step>

  <Step title="Commit and push">
    ```sh wrap theme={null}
    git add .gitmodules dbt/jaffle-shop dags/dbt_dag.py
    git commit -m "Add dbt project as submodule with Cosmos DAG"
    git push
    ```
  </Step>
</Steps>

### Configure agents to fetch submodules

By default, `GitDagBundle` does not recursively fetch Git submodules. You must configure the `git_conn_id` connection to include submodule support and ensure the agent can authenticate to the submodule repository.

If the dbt submodule repository uses the same authentication as the Dag repository, configure the Dag bundle with the same `git_conn_id`. If the submodule is in a public repository, no additional authentication is required.

For private submodule repositories that require separate credentials, configure an additional Git connection in your secrets backend or `values.yaml`.

### Update the dbt submodule

When the dbt team pushes new changes, update the submodule reference in your Dag repository:

```sh wrap theme={null}
git submodule update --remote dbt/jaffle-shop
git add dbt/jaffle-shop
git commit -m "Update jaffle-shop submodule to latest"
git push
```

Agents pick up the change on the next `GitDagBundle` refresh cycle.

For more information about working with Git submodules in an Astro context, see [Use Git submodules with an Astro project](/docs/astro/best-practices/git-submodules).

<a />

## Option 2: Include dbt code in the agent client image

Build the dbt project directly into your custom agent client image. This approach works well when dbt code changes infrequently and you want a self-contained image.

<Steps>
  <Step title="Create a custom Dockerfile">
    Extend the base agent image to include your dbt project and dependencies:

    ```dockerfile title="Dockerfile.client" wrap theme={null}
    FROM images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.0

    # Install dbt into a virtual environment
    RUN python -m venv /home/astro/dbt_venv && \
        source /home/astro/dbt_venv/bin/activate && \
        pip install --no-cache-dir dbt-postgres && \
        deactivate

    # Install Cosmos
    RUN pip install astronomer-cosmos

    # Copy dbt project into the image
    COPY dbt/jaffle-shop /usr/local/airflow/dbt/jaffle-shop
    ```
  </Step>

  <Step title="Build and push the image">
    ```sh wrap theme={null}
    docker build -f Dockerfile.client -t your-registry.example.com/custom-agent:1.0.0 .
    docker push your-registry.example.com/custom-agent:1.0.0
    ```
  </Step>

  <Step title="Update the Helm values">
    Reference the custom image in your `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    workers:
      - name: default-worker
        image: your-registry.example.com/custom-agent:1.0.0

    dagProcessor:
      image: your-registry.example.com/custom-agent:1.0.0

    triggerer:
      image: your-registry.example.com/custom-agent:1.0.0
    ```
  </Step>

  <Step title="Apply changes">
    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Step>
</Steps>

This approach requires rebuilding and redeploying the agent image every time dbt code changes. For teams that update dbt models frequently, consider Option 1 instead.

## Compare approaches

| Criteria                                      | Git submodule |       Agent image      |
| --------------------------------------------- | :-----------: | :--------------------: |
| dbt and Airflow code in separate repositories |      Yes      |           Yes          |
| Independent dbt deploy cycle                  |    Partial    |           No           |
| Works with `GitDagBundle` versioning          |      Yes      |           No           |
| Requires image rebuild for dbt changes        |       No      |           Yes          |
| Additional infrastructure                     |      None     |          None          |
| Best for                                      |   Most teams  | Infrequent dbt changes |

## Related documentation

* [Remote Execution overview](/docs/astro/remote-execution-overview)
* [Configure Dag sources](/docs/astro/remote-execution-configure-dag-sources)
* [Deploy Remote Execution project](/docs/astro/deploy-project-remote-execution)
* [Deploy dbt projects to Astro](/docs/astro/deploy-dbt-project)
* [Use Git submodules with an Astro project](/docs/astro/best-practices/git-submodules)
* [Orchestrate dbt Core jobs with Airflow and Cosmos](/docs/learn/airflow-dbt)


# Get started with Remote Execution
Source: https://astronomer.io/docs/astro/remote-execution-get-started

Set up Remote Execution on Astro with this prerequisite checklist and step-by-step setup guide.

This guide provides everything you need to set up Remote Execution on Astro, including prerequisites, configuration requirements, and a step-by-step checklist.

## What you'll configure

Remote Execution requires both required and optional components. Understand what you need before starting setup.

### Required components

You must configure these components for Remote Execution to function:

| Component                   | Purpose                                                     | When configured                                                                        |
| --------------------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| **Remote Execution Agents** | Execute Airflow tasks in your infrastructure                | [Register and configure agents](/docs/astro/remote-execution-configure-agents)              |
| **Secrets Backend**         | Securely store and access Airflow connections and variables | [Configure secrets backend](/docs/astro/remote-execution-configure-secrets-backend)         |
| **XCom Backend**            | Pass data between tasks using object storage                | [Configure XCom backend](/docs/astro/remote-execution-configure-xcom-backend)               |
| **State Store Backend**     | Store task and asset state on Astro Runtime 3.3 and later   | [Configure state store backend](/docs/astro/remote-execution-configure-state-store-backend) |
| **Dag sources**             | Define how agents access Dag code (Git or local)            | [Configure Dag sources](/docs/astro/remote-execution-configure-dag-sources)                 |

<Info>
  All required components must be configured before agents can successfully execute tasks. The setup checklist below provides the recommended order.
</Info>

### Recommended components

| Component    | Purpose                                                           | Documentation                                    |
| ------------ | ----------------------------------------------------------------- | ------------------------------------------------ |
| **Sentinel** | Monitor agent health and report status to the orchestration plane | [Enable Sentinel](/docs/astro/remote-agents-sentinel) |

### Optional components

These components enhance functionality but aren't required for basic operation:

| Component              | Purpose                                                  | Documentation                                                          |
| ---------------------- | -------------------------------------------------------- | ---------------------------------------------------------------------- |
| **Logging**            | Export task logs to external platforms or object storage | [Configure logging](/docs/astro/remote-execution-logging-overview)          |
| **OpenLineage**        | Enable data lineage and observability in Astro Observe   | [Configure OpenLineage](/docs/astro/remote-execution-configure-openlineage) |
| **AWS PrivateLink**    | Enable private connectivity between agents and Astro     | [Configure AWS PrivateLink](/docs/astro/remote-agents-aws-privatelink)      |
| **Azure Private Link** | Enable private connectivity between agents and Astro     | [Configure Azure Private Link](/docs/astro/remote-agents-azure-privatelink) |

## Setup

Follow these steps in order to set up Remote Execution:

### 1. Meet prerequisites

Before you begin, ensure you have:

* An Astro Deployment configured for Remote Execution mode. See [Create a deployment](/docs/astro/create-deployment).
* Kubernetes 1.30 or later

<Tip>
  **Recommended Kubernetes configuration**

  Configure `singleProcessOOMKill: true` in your kubelet configuration. With this setting,
  Kubernetes kills only the process that runs out of memory instead of the entire Pod. Without it, one task's
  out-of-memory error kills all tasks on the worker and loses all task logs.

  See [Kubernetes documentation](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/) for details.
</Tip>

* Helm 3 or later
* Permissions to create Astro agents
* A Deployment API token with Deployment Admin permissions
* Network access from your Kubernetes cluster to Astro. Add `https://<your-cluster-id>.external.astronomer.run/` to your organization's allowlist. To find your cluster ID, go to **Settings** > **Clusters** in the Astro UI.

### 2. Register and configure agents

Install the Remote Execution Agent Helm chart in your Kubernetes cluster:

1. Create an Agent Token in the Astro UI
2. Download and configure the `values.yaml` file
3. Install the Helm chart with your configuration
4. Verify agent heartbeat in the Astro UI

See [Register and configure agents](/docs/astro/remote-execution-configure-agents).

### 3. Configure secrets backend

Configure a secrets backend so agents can access Airflow connections and variables:

* AWS Secrets Manager
* Azure Key Vault
* Google Cloud Secret Manager
* HashiCorp Vault

See [Configure secrets backend](/docs/astro/remote-execution-configure-secrets-backend).

### 4. Configure XCom backend

Configure object storage for passing data between tasks:

* AWS S3
* Azure Blob Storage
* GCP Cloud Storage

See [Configure XCom backend](/docs/astro/remote-execution-configure-xcom-backend).

### 5. Configure Dag sources

Define how agents access your Dag code:

* **GitDagBundle** (recommended): Dags in a Git repository with automatic versioning
* **LocalDagBundle**: Dags in container images or persistent volumes

See [Configure DAG sources](/docs/astro/remote-execution-configure-dag-sources).

### 6. Enable Sentinel

Enable Sentinel to monitor agent health and report status to the orchestration plane. Sentinel detects Pod and component issues and monitors essential integrations such as XCom and secrets backends.

See [Sentinel for Remote Execution Agents](/docs/astro/remote-agents-sentinel).

### 7. Optionally configure logging

Set up task log export to view logs in external platforms or the Airflow UI:

* Export logs with a logging sidecar
* Link Airflow UI to external logging platform
* Show logs in Airflow UI from object storage

See [Configure logging](/docs/astro/remote-execution-logging-overview).

### 8. Deploy your project

Deploy your Airflow project to the Remote Execution deployment:

1. Initialize a Remote Execution project with Astro CLI
2. Build and push the server image (orchestration plane)
3. Build and push client images (execution plane)
4. Update agents to use the new client image

See [Deploy a Remote Execution project](/docs/astro/deploy-project-remote-execution).

### 9. Verify agents are running

Confirm agents are healthy and processing tasks:

1. Check agent heartbeat status in the Astro UI
2. View agent Pods in your Kubernetes cluster
3. Trigger a test Dag run and verify task execution
4. Check task logs to confirm logging configuration

## Next steps

After completing setup:

* [Configure OpenLineage](/docs/astro/remote-execution-configure-openlineage) to enable data lineage and Astro Observe features

## Related documentation

* [Remote Execution overview](/docs/astro/remote-execution-overview)
* [Execution modes](/docs/astro/execution-mode)
* [Helm chart configuration reference](/docs/astro/remote-agents-helm-reference)


# Configure logging
Source: https://astronomer.io/docs/astro/remote-execution-logging-overview

Compare logging options for Remote Execution Agents, including sidecars and object storage.

Remote Execution Agents generate task logs in your Kubernetes cluster. By default, these logs remain in agent Pods and are lost when Pods terminate. Configure logging to preserve and access these logs.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

## Logging approaches

### External logging provider (recommended)

Export logs to your logging platform (Splunk, Elasticsearch, CloudWatch, etc.) using a logging sidecar. Configure the Airflow UI to display links to external logs.

* See [Configure logging sidecar](/docs/astro/remote-agents-logging-sidecar)
* See [Export logs to external platforms](/docs/astro/remote-task-logs-external)

### Object storage (Airflow UI display)

Store logs in object storage (S3, GCS, Azure Blob) and configure the Astro API server to fetch and display them in the Airflow UI. Logs appear after task completion.

* See [Display logs in Airflow UI](/docs/astro/remote-task-logs-af-ui)

### Real-time streaming to object storage

Extend object storage logging with a Vector sidecar to stream partial logs while tasks run. Provides real-time log visibility in the Airflow UI.

* See [Display logs in Airflow UI](/docs/astro/remote-task-logs-af-ui#display-task-logs-during-task-execution)

## Comparison

| Feature              | External Logging          | Object Storage          | Real-Time Streaming       |
| -------------------- | ------------------------- | ----------------------- | ------------------------- |
| **Data location**    | Your logging platform     | Your object storage     | Your object storage       |
| **UI experience**    | Link to external platform | View in Airflow UI      | View in Airflow UI (live) |
| **Log availability** | Near real-time            | After task completion   | During task execution     |
| **Setup complexity** | Medium                    | Medium                  | High                      |
| **Storage costs**    | Platform-dependent        | Standard object storage | Higher (many small files) |

## Prerequisites

* Remote Execution Agent installed and registered
* Workload identity configured for your Kubernetes cluster
* One of the following:
  * **External logging**: Logging platform endpoint
  * **Object storage**: S3, GCS, or Azure Blob container
  * **Real-time streaming**: Object storage + Vector configuration

## Next steps

* [Configure logging sidecar](/docs/astro/remote-agents-logging-sidecar)
* [Export logs to external platforms](/docs/astro/remote-task-logs-external)
* [Display logs in Airflow UI](/docs/astro/remote-task-logs-af-ui)


# Remote Execution overview
Source: https://astronomer.io/docs/astro/remote-execution-overview

Run Airflow tasks in your own Kubernetes infrastructure while Astro manages orchestration.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Remote Execution is an execution mode on Astro that separates task execution from orchestration. With Remote Execution, you run Airflow tasks in your own Kubernetes infrastructure while Astro manages the orchestration components in the cloud. Use Remote Execution to keep your data and code within your own infrastructure for security or compliance requirements.

<Frame>
  <img alt="Astro Remote architecture overview" />
</Frame>

## How Remote Execution works

Remote Execution uses a decoupled architecture with two planes:

**Orchestration plane (Astro-managed)**

The orchestration plane runs in Astro's cloud infrastructure and includes:

* **Scheduler**: Determines when dags and tasks should run
* **Web Server/API Server**: Provides the Airflow UI and REST API
* **Metadata Database**: Stores Dag and task metadata
* **Remote Execution API**: Manages agent communication and task distribution

The orchestration plane assigns tasks to agents, monitors their health via heartbeats, and provides visibility through logging and observability features.

**Execution plane (Customer-managed)**

The execution plane runs in your Kubernetes cluster and includes:

* **Remote Execution Agents**: Deployed via Helm charts, each agent includes:
  * **DAG Processor**: Parses and serializes Dag code
  * **Triggerer**: Manages deferrable tasks
  * **Worker**: Executes Airflow tasks
  * **Sentinel**: Monitors agent health and reports status to the orchestration plane

Agents pull tasks from the orchestration plane via secure HTTPS connections, execute them locally, and report status back. Agents maintain frequent heartbeats with the API Server. If an agent loses its heartbeat, the orchestration plane automatically reroutes tasks to healthy agents. See [Remote Execution Agent failure scenarios](/docs/astro/remote-agents-failure-scenarios) for more information.

## Key concepts

### Remote Execution Agents

Agents are the core component of Remote Execution. Each agent is a collection of Airflow components (Worker, Dag Processor, Triggerer) deployed as a single unit in your Kubernetes cluster. You can deploy multiple agents with unique configurations across different clusters, regions, or node types to meet your workload requirements.

Agents communicate with Astro using:

* **Agent tokens**: Authenticate agents to the orchestration plane
* **Outbound-only connections**: Enable communication from your infrastructure to Astro without requiring inbound traffic
* **Heartbeat mechanism**: Monitor agent health with regular status checks

### Dag bundles

Dag bundles are collections of Dag files and supporting code. Remote Execution supports two types:

* **GitDagBundle**: Dags stored in a Git repository (recommended for production)
* **LocalDagBundle**: Dags stored in the container image or persistent volume

GitDagBundle provides automatic Dag versioning, allowing you to track changes and view different versions in the Airflow UI.

### XCom backend

XCom (cross-communication) allows Airflow tasks to share data. With Remote Execution, you must configure an object storage backend (AWS S3, Azure Blob Storage, or GCP Cloud Storage) to pass XCom data between tasks running on different agents.

### Secrets backend

Remote Execution Agents must be configured with a secrets backend to securely access Airflow connections and variables. This can be AWS Secrets Manager, Azure Key Vault, Google Cloud Secret Manager, or HashiCorp Vault.

### State store backend

Starting with Airflow 3.3 (Astro Runtime 3.3), tasks and asset watchers can persist state between runs through the Airflow worker-side state store. With Remote Execution, agents store this state in your execution plane instead of the Astro-hosted metadata database. The Remote Execution Agent Helm chart (2.3.0 and later) sets a working default, and you can point it at object storage for production.

## What you need to configure

Configure the following required and optional components for Remote Execution.

### Required components

* [**Remote Execution Agents**](/docs/astro/remote-execution-configure-agents): Deployed via Helm in your Kubernetes cluster
* [**Secrets Backend**](/docs/astro/remote-execution-configure-secrets-backend): To securely store and access Airflow connections and variables
* [**XCom Backend**](/docs/astro/remote-execution-configure-xcom-backend): Object storage for passing data between tasks
* [**State Store Backend**](/docs/astro/remote-execution-configure-state-store-backend): Storage for task and asset state on Astro Runtime 3.3 and later. The Helm chart (2.3.0 and later) sets a working default
* [**Dag Sources**](/docs/astro/remote-execution-configure-dag-sources): Configure how agents access your dag code (Git or local)

### Recommended components

* [**Sentinel**](/docs/astro/remote-agents-sentinel): Monitor agent health and report status to the orchestration plane. Astronomer recommends enabling Sentinel for all production deployments.

### Optional components

* [**Logging**](/docs/astro/remote-execution-logging-overview): Export task logs to external logging platforms or object storage
* [**OpenLineage**](/docs/astro/remote-execution-configure-openlineage): Enable data lineage and observability features

## Next steps

* [Get started with Remote Execution](/docs/astro/remote-execution-get-started) - Set up prerequisites and follow the setup checklist
* [Register and configure agents](/docs/astro/remote-execution-configure-agents) - Install and register your first agent

## Related documentation

* [Execution modes](/docs/astro/execution-mode)
* [Deployment settings](/docs/astro/deployment-settings)
* [Remote Execution Agent release notes](/docs/astro/agent-release-notes)


# Remote Execution shared responsibility model
Source: https://astronomer.io/docs/astro/remote-execution-shared-responsibility



Remote Execution uses a decoupled architecture where Astronomer manages the orchestration plane and the customer manages the execution plane. This document defines the responsibilities for each party.

For the general Astro shared responsibility model, see [Shared responsibility model](/docs/astro/shared-responsibility-model).

## Astronomer responsibilities

Astronomer is responsible for managing the orchestration plane and supporting the Remote Execution platform, including:

### Orchestration plane

* Operating and maintaining the Airflow scheduler, API server, web server, and metadata database in Astro's cloud infrastructure.
* Managing the Remote Execution API that coordinates task distribution between the orchestration plane and customer-managed agents, and performs task lifecycle management.
* Monitoring agent heartbeats and automatically rerouting tasks to healthy agents when an agent becomes unavailable.
* Providing the Astro UI and Astro API for Deployment management, agent registration, and token creation.
* Securing authentication and authorization for all orchestration plane interfaces, including the Astro UI, API, and CLI.
* Maintaining data encryption at rest and in transit for all orchestration plane components.
* In case of a Disaster Recovery failover, Astronomer is responsible for handling infrastructure operations necessary to migrate the Airflow Orchestration Plane over to the secondary site (the `Disaster Recovery` feature needs to be enabled on the Astronomer cluster).

### Agent software and support

* Publishing and maintaining Remote Execution Agent images.
* Publishing and maintaining the Remote Execution Agent Helm chart.
* Providing [Astronomer support](https://support.astronomer.io) for Remote Execution configuration and troubleshooting.

## Customer responsibilities

The customer is responsible for managing the execution plane and the infrastructure that Remote Execution Agents run on, including:

### Disaster Recovery

* Triggering a failover on the Astronomer-managed Kubernetes Cluster if the situation requires it.

### Kubernetes infrastructure

* Provisioning and maintaining Kubernetes clusters where agents run (and possible Disaster Recovery sites).
* Managing cluster capacity, node pools, and autoscaling to support agent workloads.
* Applying Kubernetes and node OS security patches and version upgrades.
* Configuring network policies, firewalls, and access controls within the cluster.
* Ensuring network connectivity from agent clusters to the Astro orchestration plane. See [Allowlist Astro domains](/docs/astro/allowlist-domains).

### Agent deployment and operations

* [Installing and configuring](/docs/astro/remote-execution-configure-agents) Remote Execution Agents using the Helm chart.
* Creating and managing [agent tokens](/docs/astro/remote-execution-configure-agents#step-1-create-agent-token) for authenticating agents to the orchestration plane.
* Pulling agent images from the Astronomer registry and storing them in a private registry when required.
* Building and maintaining custom agent images with additional Python packages and OS-level dependencies required by Dag code.
* [Upgrading agents](/docs/astro/agent-maintenance-policy#remote-execution-agent-image-upgrade-process) to latest (recommended) or other supported versions within the maintenance window.
* Monitoring agent Pod health and resource utilization in Kubernetes.

### Secrets and credentials

* [Configuring a secrets backend](/docs/astro/remote-execution-configure-secrets-backend) (AWS Secrets Manager, Azure Key Vault, Google Cloud Secret Manager, or HashiCorp Vault) for Airflow connections and variables.
* Managing and rotating credentials stored in the secrets backend.
* Configuring workload identity or service account permissions for agents to access the secrets backend.

### Data storage

* [Configuring an XCom backend](/docs/astro/remote-execution-configure-xcom-backend) (AWS S3, Azure Blob Storage, or GCP Cloud Storage).
* [Configuring a state store backend](/docs/astro/remote-execution-configure-state-store-backend) for task and asset state on Astro Runtime 3.3 and later.
* Provisioning and managing object storage buckets or containers used for XCom, the state store, and logging.
* Configuring IAM roles, managed identities, or service accounts for agent access to storage resources.
* Managing storage lifecycle policies, encryption, and access controls.

### Dag code and sources

* Developing and maintaining Dag code with security and quality coding practices.
* [Configuring Dag sources](/docs/astro/remote-execution-configure-dag-sources) (GitDagBundle or LocalDagBundle) for agent access to Dag code.
* Managing Git repository authentication credentials for GitDagBundle configurations.
* Building and deploying [Remote Execution project images](/docs/astro/deploy-project-remote-execution) for both the orchestration and execution planes.

### Logging and observability

* [Configuring task logging](/docs/astro/remote-execution-logging-overview) to preserve logs from agent Pods.
* Provisioning and managing external logging platforms or object storage for log export.
* It is recommended to enable [Sentinel](/docs/astro/remote-agents-sentinel) for agent health monitoring.
* Optionally configuring [OpenLineage](/docs/astro/remote-execution-configure-openlineage) for data lineage tracking.

### Security

* Managing user roles, permissions, and authentication assets (tokens, connections, environment variables).
* Integrating with identity providers for secure SSO/MFA.
* Managing customer-owned credentials.
* Securing outbound network connections from agent clusters to the Astro orchestration plane.
* Configuring private connectivity ([AWS PrivateLink](/docs/astro/remote-agents-aws-privatelink) or [Azure Private Link](/docs/astro/remote-agents-azure-privatelink)) when required.
* Securing network communications between agents and data resources in the execution plane, including secrets backends, object storage, and data sources.
* Managing IP allowlists and firewall rules for agent clusters.
* Implementing and maintaining secure, high-quality data pipelines, including dependency and vulnerability management.

## Related documentation

* [Shared responsibility model](/docs/astro/shared-responsibility-model)
* [Remote Execution overview](/docs/astro/remote-execution-overview)
* [Security in Astro](/docs/astro/security)
* [Remote Execution Agent maintenance policy](/docs/astro/agent-maintenance-policy)


# Show Remote Execution Agent task logs in Airflow UI
Source: https://astronomer.io/docs/astro/remote-task-logs-af-ui

Configure a Remote Execution Agent to upload task logs to object storage so they display in the Airflow UI after task completion or in real time.

You can display task logs in the Airflow UI by exporting logs to object storage and configuring the Astro API Server to retrieve them. Start by enabling log display after task completion, then optionally extend the setup to stream logs in real time as tasks run.

This guide explains configuring post-task log display and expanding that configuration to support real-time log streaming.

## Display task logs after task completion

Set up log uploading so logs are visible in the Airflow UI after task completion. This requires:

* Remote Execution Agent configuration (`values.yaml`)
* Astro UI Deployment configuration
* Workload identities: write access for the Remote Execution Agent, read access for the Astro API Server

The `commonEnv` block in the following procedures applies environment variables to all Airflow components in the Remote Execution Agent. These variables configure Airflow's [remote logging](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-tasks.html#serving-logs-from-remote-location) so that workers and the triggerer upload completed task logs to object storage:

* `AIRFLOW__LOGGING__REMOTE_LOGGING`: Turns on Airflow's remote log upload.
* `AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID`: Names the Airflow connection used to authenticate with object storage.
* `AIRFLOW_CONN_<CONN_ID>`: Defines that connection inline, so no separate connection record is required. A URI with no credentials (for example, `s3://`) causes the underlying cloud SDK to use its default credential chain, which picks up the workload identity attached to the Pod.
* `AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER`: Sets the bucket and path prefix that Airflow writes task logs to. Use a Deployment-scoped path so logs from different Deployments don't collide.
* `AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS`: Replaces Airflow's default logging configuration with the Astronomer Runtime configuration, which installs the task log handler used to write logs to and read logs from object storage.
* `ASTRONOMER_ENVIRONMENT`: Set to `cloud` for Astro Deployments. Astronomer Runtime reads this value to select Astro-specific logging defaults.

<Tabs>
  <Tab title="AWS">
    <Info>
      The Astro Orchestration Plane provides secure private connectivity with a pre-configured S3 Gateway Endpoint.
    </Info>

    1. Configure the following environment variables in the Helm chart's `values.yaml`, and replace the path for the `AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER` value with your information:

    ```yaml title="values.yaml" wrap theme={null}
    commonEnv:
      - name: AIRFLOW__LOGGING__REMOTE_LOGGING
        value: "True"
      - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
        value: "astro_aws_logging"
      - name: AIRFLOW_CONN_ASTRO_AWS_LOGGING
        value: "s3://"
      - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
        value: "s3://<bucket>/<deployment-id>"
      - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
        value: "astronomer.runtime.logging.logging_config"
      - name: ASTRONOMER_ENVIRONMENT
        value: "cloud"
    ```

    <Tip>
      **Mounting credentials manually**

      If you don't use workload identity and instead want to manually mount a credential, you must also add the following environment variable defining the location of a token file to your Remote Agent's `values.yaml` file. You can customize the file path, `/tmp/logging-token`, to the name of your logging token file.

      ```yaml title="values.yaml" wrap theme={null}
        - name: ASTRO_LOGGING_AWS_WEB_IDENTITY_TOKEN_FILE
          value: "/tmp/logging-token"
      ```
    </Tip>

    2. Run `helm upgrade` to apply the change to your Agents.

    3. In the Astro UI, navigate to your Deployment and click the **Details** tab. Click **Edit** in the **Advanced** section to access your logging configurations.

    4. Select **Bucket Storage** in the **Task Logs** field and fill in the **Bucket URL** as `s3://<bucket>/<deployment-id>`. Or, use the path that you configured for `AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER` in your Remote Agent's Helm chart's `values.yaml`.

    5. In the **Workload Identity for Bucket Storage** section, select **Customer Managed Identity** and follow the instructions to set up your Customer Managed Identity so that the identity you create has read access to the specified bucket and path. The Customer Managed Identity must have `s3:GetObject` and `s3:ListBucket` permissions on the S3 bucket. Additionally, ensure that no ACLs on the bucket restrict those actions.

    <Warning>
      **Default Identity** isn't currently supported for Task Logs Bucket Storage on AWS. You must use **Customer Managed Identity**.
    </Warning>

    6. If your log bucket is in a different region from your Astro Deployment, you need to define the AWS region in the `AIRFLOW__ASTRONOMER_PROVIDERS_LOGGING__AWS_REGION` environment variable for Astronomer-managed components. In the Astro UI, navigate to your Deployment and click the **Environment** tab. Click **Edit Deployment Variables** (or **+ New Environment Variable** if you have no variables configured yet), then click **Add Variable** to add the following environment variables to your Deployment:

    * `AIRFLOW__ASTRONOMER_PROVIDERS_LOGGING__AWS_REGION <The region in which the S3 bucket is configured>`
  </Tab>

  <Tab title="GCP">
    <Info>
      The Astro Orchestration Plane provides secure private connectivity with a pre-configured Private Service Connect endpoint to GCP Cloud Storage.
    </Info>

    1. Configure the following environment variables in the Helm chart's `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    commonEnv:
      - name: AIRFLOW__LOGGING__REMOTE_LOGGING
        value: "True"
      - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
        value: "astro_gcs_logging"
      - name: AIRFLOW_CONN_ASTRO_GCS_LOGGING
        value: "gcs://"
      - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
        value: "gs://<bucket>/<deployment-id>"
      - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
        value: "astronomer.runtime.logging.logging_config"
      - name: ASTRONOMER_ENVIRONMENT
        value: "cloud"
    ```

    <Info>The path for the `AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER` value is configurable. This is only an example format.</Info>

    2. Do a `helm upgrade` to apply the change to your Agents.

    3. In the Astro UI, navigate to your Deployment and click the **Details** tab. Click **Edit** in the **Advanced** section.

    4. In the **Task Logs** field, select **Bucket Storage** and fill in the **Bucket URL** as `gs://<bucket>/<deployment-id>`. Or, use the path that you configured for `AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER` in your Remote Agent's Helm chart's `values.yaml`.

    5. In the **Workload Identity for Bucket Storage** section, select **Customer Managed Identity** and follow the instructions to set up your Customer Managed Identity so that the identity you create has read access to the specified bucket and path.

    <Tip>On GCP, you can authorize your Deployment to cloud resources using workload identity with the alternative methods described [here](/docs/astro/authorize-deployments-to-your-cloud?tab=gcp#setup).</Tip>
  </Tab>

  <Tab title="Azure">
    <Warning>
      You must configure a connection from your Astro Orchestration Plane to your storage account for the Astro API server to read task logs. You can use either a private connection or a public connection.

      * **Private Connectivity**: Contact [Astronomer support](https://cloud.astronomer.io/open-support-request) to establish a Private Link connection.
      * **Public Connectivity**: Enable **Enabled from all networks** in the networking settings of your storage account. Azure doesn't support IP network rules for restricting access from resources deployed in the same region as the storage account. See [Restrictions for IP network rules](https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security-limitations#restrictions-for-ip-network-rules).
    </Warning>

    <Info>
      For security, Astronomer recommends you assign the **Storage Blob Data Reader** role to the workload identity used by the API server and apply the following settings for your storage account:

      * **Secure transfer required**: Enabled
      * **Allow Blob anonymous access**: Disabled
      * **Minimum TLS version**: TLS 1.2 or higher
    </Info>

    1. Authenticate to your storage account with an Azure Managed Identity, or the alternative method that uses a Storage Account Access Key.

    <AccordionGroup>
      <Accordion title="Azure Managed Identity (Recommended)">
        If you write task logs using an Azure Managed Identity, configure the following environment variables in the Helm chart's `values.yaml`:

        ```yaml title="values.yaml" wrap theme={null}
        commonEnv:
          - name: AIRFLOW__LOGGING__REMOTE_LOGGING
            value: "True"
          - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
            value: "astro_azure_logs_override"
          - name: AIRFLOW_CONN_ASTRO_AZURE_LOGS_OVERRIDE
            value: "wasb://<storage-account-name>@<storage-account-name>.blob.core.windows.net"
          - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
            value: "wasb-<storage-account-name>"
          - name: AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER
            value: "<storage-account-container-name>"
          - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
            value: "astronomer.runtime.logging.logging_config"
          - name: ASTRONOMER_ENVIRONMENT
            value: "cloud"

        labels:
          azure.workload.identity/use: "true"

        annotations:
          azure.workload.identity/client-id: "<managed-identity-client-id>"
        ```

        The `AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER` variable sets the Azure Blob Storage container that Airflow writes logs to. The `labels` and `annotations` blocks enable [Azure Workload Identity](https://azure.github.io/azure-workload-identity/docs/) on the Agent Pods so Airflow can authenticate to Azure without static credentials.
      </Accordion>

      <Accordion title="Storage Account Access Key (Alternative)">
        If writing task logs using an Storage Account Access Key, it isn't necessary to configure the `labels` and `annotations` in your Helm chart's `values.yaml`. It is **required** to configure the `AIRFLOW_CONN_ASTRO_AZURE_LOGS_OVERRIDE` environmental variable with the format `value: wasb://<storage-account-name>:<access-key>@<storage-account-name>.blob.core.windows.net`.

        ```yaml title="values.yaml" wrap theme={null}
        commonEnv:
          - name: AIRFLOW__LOGGING__REMOTE_LOGGING
            value: "True"
          - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
            value: "astro_azure_logs_override"
          - name: AIRFLOW_CONN_ASTRO_AZURE_LOGS_OVERRIDE
            value: "wasb://<storage-account-name>:<access-key>@<storage-account-name>.blob.core.windows.net"
          - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
            value: "wasb-<storage-account-name>"
          - name: AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER
            value: "<storage-account-container-name>"
          - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
            value: "astronomer.runtime.logging.logging_config"
          - name: ASTRONOMER_ENVIRONMENT
            value: "cloud"
        ```
      </Accordion>
    </AccordionGroup>

    2. Run `helm upgrade` to apply the change to your Agents.

    3. In the Astro UI, navigate to your Deployment and click the **Details** tab. Click **Edit** in the **Advanced** section.

    4. Select **Bucket Storage** in the **Task Logs** field and add the **Bucket URL** as `wasb://<container-name>@<storage-account-name>.blob.core.windows.net/wasb-<storage-account-name>`.

    5. In the **Workload Identity for Bucket Storage** section, select **Customer Managed Identity** and follow the instructions to set up your Customer Managed Identity so that the identity you create has read access to the specified bucket and path.

    6. In the Astro UI, navigate to your Deployment and click the **Environment** tab. Click **Edit Deployment Variables** (or **+ New Environment Variable** if you have no variables configured yet), then click **Add Variable** to add the following environment variables to your Deployment:

    * `ASTRO_LOGGING_AZURE_CLIENT_ID: <Client ID for the managed identity that you want the API server to use to fetch task logs>`
    * `ASTRO_LOGGING_AZURE_TENANT_ID: <Tenant ID for the managed identity that you want the API server to use to fetch task logs>`
  </Tab>
</Tabs>

## Display task logs during task execution

Once you have post-completion log visibility, you can enable real-time log display. Remote Execution prevents the Airflow API server from reading logs directly from workers until they reach object storage. Use [Vector](https://vector.dev), included in the Remote Execution Agent Helm chart, to upload partial logs while tasks are running.

### Prerequisites

Before you configure Vector, ensure that your Remote Execution Deployment is already set up to [upload task logs to object storage after task completion](#display-task-logs-after-task-completion).

### Enable Vector sidecar

Use Vector to watch for log file changes and upload updates to object storage during task execution.

In your Helm `values.yaml`:

* Set `loggingSidecar.enabled` to `true` to inject the Vector container into each worker and triggerer Pod:

```yaml title="values.yaml" wrap theme={null}
loggingSidecar:
  enabled: true
```

* Configure `loggingSidecar.volumeMounts` to give Vector read access to the Airflow task log files and a writable location for its on-disk checkpoint state:

```yaml title="values.yaml" wrap theme={null}
loggingSidecar:
  volumeMounts:
    - name: task-logs
      mountPath: /var/log/airflow/task_logs
      readOnly: true
    - name: vector-data
      mountPath: /var/lib/vector
```

The `task-logs` mount is the directory Vector reads from. The `vector-data` mount stores Vector's file checkpoints so it can resume uploads after a restart without re-sending lines.

### Configure log upload for your cloud provider

The `loggingSidecar.config` block is the Vector pipeline definition. Each cloud-specific config uses the same three-stage structure:

* `sources.airflow_task_logs`: Tails the local task log files written by Airflow workers and the triggerer. `read_from: beginning` ensures Vector starts at the top of each file so partial logs aren't missed.
* `transforms.strip_path_prefix`: Removes the local mount path from each event's `file` field and writes the result to `log_path`. This produces an object key that matches the layout Airflow uses when it uploads the complete log after task completion, so the Astro API Server can read both the partial and final logs from the same location.
* `sinks.<cloud>`: Uploads the transformed events to object storage. `key_prefix` (or `blob_prefix` on Azure) uses the `log_path` field from the transform to place each log under the right object key. `batch.max_bytes` and `batch.timeout_secs` control how often Vector flushes — smaller values stream logs to the UI faster but produce more small objects in storage. See [Small file problem](#small-file-problem).

<Tabs>
  <Tab title="AWS S3">
    Configure `loggingSidecar.config`:

    ```yaml title="values.yaml" expandable wrap theme={null}
    loggingSidecar:
      config: |
        # Vector configuration for Astronomer Remote Execution agents for uploading Airflow task logs to AWS S3

        sources:
          airflow_task_logs:
            type: file
            include:
              - /var/log/airflow/task_logs/**/*.log
            read_from: beginning

        transforms:
          strip_path_prefix:
            type: remap
            inputs: [airflow_task_logs]
            source: |
              .log_path, err = replace(.file, "/var/log/airflow/task_logs/", "")
              if err != null {
                abort
              }

        sinks:
          s3:
            # For more Vector AWS S3 configuration options, see https://vector.dev/docs/reference/configuration/sinks/aws_s3
            type: aws_s3
            inputs: [strip_path_prefix]
            bucket: # set bucket name e.g. airflow_logs
            region: # set AWS region e.g. us-east-2
            compression: none
            encoding:
              codec: text
            key_prefix: '{{ "{{" }} log_path {{ "}}" }}.'
            filename_time_format: "%Y-%m-%dT%H-%M-%S"
            filename_append_uuid: false
            batch:
              max_bytes: 1000000  # configure based on your storage costs and log frequency requirements - see Caveats section below
              timeout_secs: 10  # configure based on your storage costs and log frequency requirements - see Caveats section below
    ```

    <Tip>
      **AWS authentication with Vector**

      Above Vector config assumes a managed identity is set up for authentication, as described in [Display task logs after task completion](#display-task-logs-after-task-completion).

      If you require a different way to authenticate with AWS, such as static keys, see [https://vector.dev/docs/reference/configuration/sinks/`aws_s3`/#auth](https://vector.dev/docs/reference/configuration/sinks/aws_s3/#auth) for all available options.
    </Tip>
  </Tab>

  <Tab title="GCP Cloud Storage">
    Configure `loggingSidecar.config`:

    ```yaml title="values.yaml" expandable wrap theme={null}
    loggingSidecar:
      config: |
        # Vector configuration for Astronomer Remote Execution agents for uploading Airflow task logs to GCS

        sources:
          airflow_task_logs:
            type: file
            include:
              - /var/log/airflow/task_logs/**/*.log
            read_from: beginning

        transforms:
          strip_path_prefix:
            type: remap
            inputs: [airflow_task_logs]
            source: |
              .log_path, err = replace(.file, "/var/log/airflow/task_logs/", "")
              if err != null {
                abort
              }

        sinks:
          gcs:
            # For more Vector GCP Cloud Storage configuration options, see https://vector.dev/docs/reference/configuration/sinks/gcp_cloud_storage/
            type: gcp_cloud_storage
            inputs: [strip_path_prefix]
            bucket: # set to GCS bucket name
            compression: none
            encoding:
              codec: text
            key_prefix: '{{ "{{" }} log_path {{ "}}" }}.'  # If you place logs at a specific prefix, prepend that prefix here like 'logs/{{ "{{" }} log_path {{ "}}" }}.'
            filename_time_format: "%Y-%m-%dT%H-%M-%S"
            filename_append_uuid: false
            batch:
              max_bytes: 1000000  # configure based on your storage costs and log frequency requirements - see Caveats section below
              timeout_secs: 10  # configure based on your storage costs and log frequency requirements - see Caveats section below
    ```

    <Tip>
      **GCP authentication with Vector**

      Above Vector config assumes Workload Identity is set up for authentication, as described in [Display task logs after task completion](#display-task-logs-after-task-completion).

      If you require a different way to authenticate with GCP, such as service account keys, see [https://vector.dev/docs/reference/configuration/sinks/`gcp_cloud_storage`/#auth](https://vector.dev/docs/reference/configuration/sinks/gcp_cloud_storage/#auth) for all available options.
    </Tip>
  </Tab>

  <Tab title="Azure Blob Storage">
    Configure `loggingSidecar.env` to provide your Azure Storage connection string as a Kubernetes secret:

    ```yaml title="values.yaml" wrap theme={null}
    loggingSidecar:
      env:
        - name: AZURE_STORAGE_CONNECTION_STRING
          valueFrom:
            secretKeyRef:
              name: azure-blob-logs
              key: AZURE_STORAGE_CONNECTION_STRING
    ```

    Configure `loggingSidecar.config`:

    ```yaml title="values.yaml" expandable wrap theme={null}
    loggingSidecar:
      config: |
        # Vector configuration for Astronomer Remote Execution agents for uploading Airflow task logs to Azure Blob Storage

        sources:
          airflow_task_logs:
            type: file
            include:
              - /var/log/airflow/task_logs/**/*.log
            read_from: beginning

        transforms:
          strip_path_prefix:
            type: remap
            inputs: [airflow_task_logs]
            source: |
              .log_path, err = replace(.file, "/var/log/airflow/task_logs/", "")
              if err != null {
                abort
              }

        sinks:
          azure_blob:
            # For more Vector Azure Blob Storage configuration options, see https://vector.dev/docs/reference/configuration/sinks/azure_blob/
            type: azure_blob
            inputs: [strip_path_prefix]
            container_name: # set to Azure Blob Storage container name
            connection_string: "${AZURE_STORAGE_CONNECTION_STRING}"
            compression: none
            encoding:
              codec: text
            blob_prefix: '{{ "{{" }} log_path {{ "}}" }}.'  # If you configured a specific log path prefix, prepend that prefix here like 'wasb-<storage-account-name>/{{ "{{" }} log_path {{ "}}" }}.'
            blob_time_format: "%Y-%m-%dT%H-%M-%S"
            blob_append_uuid: false
            batch:
              max_bytes: 1000000  # configure based on your storage costs and log frequency requirements - see Caveats section below
              timeout_secs: 10  # configure based on your storage costs and log frequency requirements - see Caveats section below
    ```

    <Warning>
      Vector doesn't currently support Azure Managed Identity for the Azure Blob sink. You must authenticate using a Storage Account connection string. See the [Vector Azure Blob sink documentation](https://vector.dev/docs/reference/configuration/sinks/azure_blob/) for all available authentication options.
    </Warning>
  </Tab>
</Tabs>

<Tip>
  **Developing Vector Remap Language (VRL)**

  Vector expressions are written in Vector Remap Language (VRL). If you want to edit an expression in the Vector config, this [online VRL playground](https://playground.vrl.dev) is a useful debugging tool.
</Tip>

<Tip>
  **Debugging Vector**

  If you're having issues uploading logs, you can enable debug logging for the Vector sidecar by adding this to the sink configuration (so you'll have 2 sinks, e.g. an `s3` sink, and a `debug` sink):

  ```yaml title="values.yaml" wrap theme={null}
  debug:
    type: console
    inputs: [strip_path_prefix]
    encoding:
      codec: json
  ```

  With this second sink, Vector will display debug logs on the console, accessible with `kubectl logs [worker pod] -c vector-logging-sidecar`.
</Tip>

The following volume configuration creates the shared `emptyDir` volumes that the Vector sidecar and the Airflow worker or triggerer container both mount. `task-logs` is the directory Airflow writes logs into and Vector reads from. `vector-data` holds Vector's checkpoint state. The worker and triggerer mount `task-logs` at `/usr/local/airflow/logs`, which is where Airflow writes by default, while the Vector sidecar mounts the same volume at `/var/log/airflow/task_logs` to match its `sources.airflow_task_logs.include` glob.

* Configure `workers[*].volumes`:

```yaml title="values.yaml" wrap theme={null}
volumes:
  - name: task-logs
    emptyDir: {}
  - name: vector-data
    emptyDir: {}
```

* Configure `workers[*].volumeMounts`:

```yaml title="values.yaml" wrap theme={null}
volumeMounts:
  - name: task-logs
    mountPath: /usr/local/airflow/logs
```

* Configure `triggerer.volumes`:

```yaml title="values.yaml" wrap theme={null}
volumes:
  - name: task-logs
    emptyDir: {}
  - name: vector-data
    emptyDir: {}
```

* Configure `triggerer.volumeMounts`:

```yaml title="values.yaml" wrap theme={null}
volumeMounts:
  - name: task-logs
    mountPath: /usr/local/airflow/logs
```

* Set `AIRFLOW__LOGGING__DELETE_LOCAL_LOGS` in `commonEnv` so Airflow removes the local log file after it uploads the complete log to object storage. This keeps the shared `task-logs` volume from filling up on long-running Pods:

```yaml title="values.yaml" wrap theme={null}
commonEnv:
  - name: AIRFLOW__LOGGING__DELETE_LOCAL_LOGS
    value: "True"
```

### Log upload process

Partial logs are uploaded and displayed as follows:

1. Airflow worker or triggerer writes local task log files, as set by `AIRFLOW__LOGGING__LOG_FILENAME_TEMPLATE`.
2. Vector watches `/var/log/airflow/task_logs/**/*.log` and uploads log changes in chunks while the task runs.
3. Vector appends a timestamp to the file name before uploading each chunk.
4. Airflow scans object storage for log chunks when displaying the UI log view.
5. The UI displays all log content to the user.

<Note>
  **Version compatibility**

  Using Vector to upload logs assumes Airflow’s logging format is compatible. Significant changes to Airflow logging may require reconfiguration.
</Note>

### Caveats

#### Duplicate log storage

After task completion, Airflow uploads the complete log to object storage and deletes the local copy. This causes duplication:

1. Partial logs from Vector
2. Complete log from Airflow

The Airflow API server deduplicates log lines by timestamp and message. Only storage usage is affected; logs are displayed once.

#### Small file problem

High-frequency, small log file uploads can create many small objects. This may increase storage costs, load on object storage, or trigger rate limits. Adjust file size and upload frequency in your Vector config to balance performance and cost.

* AWS bills object retrieval at a 128KB minimum on certain storage classes ([source](https://aws.amazon.com/blogs/storage/optimizing-storage-costs-and-query-performance-by-compacting-small-objects)).
* A large number of small objects means more object requests (PUTs, GETs, LISTs) and more load on metadata/indexing; this can result in rate limits or latency issues.

Ensure a proper balance between filesize/timeout and log upload frequency in your Vector config.


# Send Remote Execution Agent task logs to external logging provider
Source: https://astronomer.io/docs/astro/remote-task-logs-external

Configure a Remote Execution Deployment to display links to task logs hosted in an external logging provider such as Splunk or Datadog.

For Remote Execution Deployments, the recommended approach for handling task logs is to use a logging shipper that sends logs to your external logging platform. You can configure your Remote Execution Deployment to display links to the external task logs in the Airflow UI, keeping all data within your environment.

To send logs to external logging providers, you must first [configure LoggingSidecar in your Remote Execution Agent](/docs/astro/remote-agents-logging-sidecar).

## Configure external logging provider for Remote Execution Deployments

1. In the Astro UI, navigate to your Deployment and click the **Details** tab. Click **Edit** in the **Advanced** section.

2. Select **External Logging Provider** in the **Task Logs** field.

3. In the **URL Template** field, enter a templated URL that references the [`log_filename_template`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#log-filename-template) value from your Airflow config. This inserts a dynamic link in the Airflow UI, where logs would normally appear, that points to the corresponding task logs in your external logging provider.

For example, a templated URL for task logs in Splunk might look like:

```text wrap theme={null}
https://<your-splunk-instance>/en-US/app/<your-splunk-app>/search?q=search%20index%3D<your-index-name>%20tiID%3D%22{{ ti.id }}%22&display.page.search.mode=smart&dispatch.sample_ratio=1&earliest=-24h%40h&latest=now
```


# Run your first dag on Astro
Source: https://astronomer.io/docs/astro/run-first-dag

Run your first Apache Airflow dag on Astro using the Astro CLI or GitHub Actions.

Astro is the industry's leading managed service for Apache Airflow.

You can quickly learn how Astro works by running an Apache Airflow dag with either [the Astro CLI](/docs/astro/first-dag-cli) or by using [GitHub Actions](/docs/astro/first-dag-github-actions). Either tutorial takes about 15 minutes. They each showcase different paths available for dag development, CI/CD workflows, and code deployment options.

You'll learn how to:

* Authenticate and log in to Astro.
* Create a Deployment.
* Deploy dags to Astro with either the Astro CLI or GitHub Actions.
* Trigger a run of an example dag.

## Choose a tutorial

If you prefer to develop on your local machine and use command line tools, follow the steps in [Run your first dag with the Astro CLI](/docs/astro/first-dag-cli).

If you can't install additional tools on your current machine or you prefer a no-code workflow, follow the steps in [Run your first dag with GitHub Actions](/docs/astro/first-dag-github-actions), where you can see an example of a CI/CD workflow.


# Subscribe to Astro from the AWS Marketplace
Source: https://astronomer.io/docs/astro/subscribe-aws

Learn how to subscribe to Astro from the AWS marketplace.

[Astro](https://www.astronomer.io/docs/astro) is a managed service for data orchestration that is built for the cloud and powered by Apache Airflow. Your Airflow infrastructure is managed entirely by Astronomer, enabling you to shift your focus from infrastructure to data.

Amazon Web Services (AWS) customers can subscribe to [Astro](https://www.astronomer.io/pricing/) with the AWS Marketplace. This enables you to pay for your Astro usage with your existing AWS billing configuration. You can choose to [subscribe to the Developer or Team plans for monthly, pay-as-you-go billing](https://aws.amazon.com/marketplace/pp/prodview-6lfiiphwtbhz2?trk=8d276e92-b310-40ce-908f-23a198ca7ffc\&sc_channel=el\&source=astronomer) or [work with our team](https://www.astronomer.io/pricing/) on an upfront, annual subscription.

The following steps show you how to subscribe to Astro's pay-as-you-go plans via AWS Marketplace.

## Step 1: Subscribe to Astro in AWS Marketplace

1. Log into your [AWS account](https://aws.amazon.com/console/).
2. Go to the [AWS Marketplace](http://aws.amazon.com/marketplace).
3. Search for **Astro by Astronomer - Pay As You Go**, or go directly to the [product listing](https://aws.amazon.com/marketplace/pp/prodview-6lfiiphwtbhz2?trk=8d276e92-b310-40ce-908f-23a198ca7ffc\&sc_channel=el\&source=astronomer).

<Tip>Ensure the product name search includes **Pay As You Go**. Other listings from Astronomer are available in AWS Marketplace, but do not include the plans with pay-as-you-go pricing.</Tip>

3. On the product overview page, click **View purchase options**.
4. Click **Subscribe** to complete your order.
5. Finalize your subscription by clicking **Set up your account** on the subscription confirmation page.

<Tip>If this button does not immediately appear, wait 1-2 minutes and refresh your page. Alternatively, you can [navigate to your AWS Marketplace subscriptions in the AWS Console](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/marketplace-manage-subscriptions.html) and complete the set-up process by clicking **Set up product** under **Actions**.</Tip>

6. Fill out the form on the page. Based on your [feature and support needs](https://www.astronomer.io/pricing/), select either the Developer or Team plan. Click **Complete Registration**.
7. After this step, you will be redirected to create your [Astro Organization](/docs/astro/astro-architecture#organization). If you have an existing Astro Organization, associate it to your AWS account using the [form on the Astronomer website](https://www.astronomer.io/aws-marketplace-confirmation/).

<Warning>Your subscription to Astro is not considered active until you complete *all* of the steps listed above. **If you have questions or experience issues, please contact marketplace \[at] astronomer.io for assistance.**</Warning>

## Step 2: Start Using Astro

Your new Astro Organization comes pre-loaded with a set amount of credits. You can begin to create Deployments and run Dags using these credits right away.

Within 1 business day of completing the AWS Subscription process, the Astronomer team will configure your Astro Organization to use your AWS Billing account. Astronomer will notify you when this is complete.

You can also verify your AWS Account is linked by going to your Astro Organization’s **Billing Settings** and confirm that the **Payment Type** in the **Payment Method** section shows AWS Marketplace.

See the [Astro Trial Resources](/docs/astro/trial#trial-resources) section for more information about what you can do with your Astro trial and [Run your first dag with the CLI](/docs/astro/first-dag-cli) to get started.

## Canceling your Astro Subscription

You can cancel your Astro subscription in AWS Marketplace. Before you cancel your subscription, please check that you deleted all Deployments and exported any configuration settings for your Deployment that you want to preserve. If you have any active Deployments, you might see an error or any running tasks might abruptly terminate.

<Danger>After you cancel your subscription in AWS Marketplace, Astronomer hibernates your Deployments for 30 days. After 30 days, your Deployments are permanently deleted. See [Delete a Deployment](/docs/astro/deployment-details#delete-a-deployment).</Danger>

Navigate to AWS Marketplace and follow the instructions for [canceling your SaaS subscription](https://docs.aws.amazon.com/marketplace/latest/buyerguide/cancel-subscription.html#cancel-saas-subscription).

## Unlink your AWS Billing from Astro

If you no longer want to use your AWS account for your Astro subscription, please contact billing \[аt] astronomer.io for assistance updating your payment settings.


# Start your Astro trial
Source: https://astronomer.io/docs/astro/trial

Start a free 14-day Astro trial directly or through the AWS or Azure marketplace.

Use this guide to get started with Astro, the best place to run Apache Airflow.

## Start a trial

Go to [Try Astro](https://www.astronomer.io/lp/signup/) to activate your free 14-day trial. To create your Astro user account, you'll need to provide a valid email address and create a password.

## Subscribe through a cloud provider marketplace

### Astro with AWS Marketplace

If your company uses AWS, you can subscribe to the Astro pay-as-you-go Developer plan through the AWS Marketplace. After the end of your Astro Trial, the subscription allows you to pay for your monthly Astro use with your existing AWS billing configuration without an upfront, annual subscription. See [Install Astro from the AWS Marketplace](/docs/astro/subscribe-aws).

## Create an Organization and Workspace

After you've created your Astro user account, you'll be asked to create an Organization and your first Workspace.

An *Organization* is the highest management level on Astro. An Organization contains *Workspaces*, which are collections of *Deployments*, or Airflow environments, that are typically owned by a single team. You can manage user roles and permissions both at the Organization and Workspace levels. For more information about Astro's key concepts and components, see [About Astro](/docs/astro/astro-architecture).

To start your trial, Astronomer recommends using the name of your company as the name of your Organization and naming your first Workspace after your data team or initial business use case with Airflow. You can update these names in the Astro UI after you finish activating your trial.

<Frame>
  <img alt="Create an Organization and Workspace" />
</Frame>

## Next steps

You're now ready to start deploying and running dags on Astro. See [Run your first dag on Astro](/docs/astro/run-first-dag) to choose a detailed quickstart. You'll create a local Astro project, then push that project to your Astro Deployment. The entire process takes about 5 minutes.

You have 14 days and \$20 to spend in credits before your trial ends. See [Manage billing](/docs/astro/manage-billing) to view how much credit you've used over the course of your trial. See [Pricing](https://www.astronomer.io/pricing/) for a breakdown of how much it costs to run Airflow on Astro.

### Trial resources

An Astro trial provides you a level of access to Deployment resources that Astro considers sufficient to run an initial number of Apache Airflow dags and ensure that Astro meets the needs of your Organization.

During a 14-day Astro trial, you can:

* Create one Workspace
* Create one Deployment
* Configure **Worker Count** to a maximum of 5 workers per worker queue
* Use A5, A10, or A20 workers

You cannot use high-availability (HA) mode during an Astro trial. To access additional resources or functionality, add a credit card number or other payment method to your Organization.

To learn more about Deployment resources and features available to Astro customers, see [Configure Deployment resources](/docs/astro/deployment-resources). To learn more about Astro pricing, see [Pricing](https://www.astronomer.io/pricing/).

## After your trial

After your 14-day trial ends, you can no longer access your Workspace from the Astro UI and your Deployments enter [hibernation](/docs/astro/deployment-resources#hibernate-a-development-deployment) for 3 days. You can still access your account and Astronomer support. To regain access to your Deployments and Workspace, you must enter a payment method or contact Astronomer to extend your trial. After you enter a payment method, you can wake your Deployments from hibernation and continue to run Apache Airflow dags.

All Deployment configurations are preserved during hibernation for 3 days from the last day of your Astro trial. After 3 days, your Deployment and all of its metadata are permanently deleted. When your Deployments are deleted, any code that you deployed to Astro will be lost.

If you need additional time to evaluate Astro, or you need to copy your configuration for future use, you can:

* [Add a payment method](/docs/astro/manage-billing#update-billing-details) to maintain your Deployment on one of Astro's pay-as-you-go plans. Choose between Developer or Team. [View pay-as-you-go pricing.](https://astronomer.io/pricing/)
* Contact [sales](https://astronomer.io/contact/) to request a trial extension.
* Run `astro deployment inspect` with the Astro CLI to save your existing Deployment configuration as a JSON or YAML file. See [Astro CLI command reference](/docs/cli/v1.43/astro-deployment-inspect).


# View cluster details
Source: https://astronomer.io/docs/astro/view-cluster-details

Learn how to view a Deployment's cluster details, including your cloud, region, and connection options.

After you configure a Deployment, you can view details about the cluster it runs on in the Astro UI.

## View cluster cloud and region

In the Astro UI, click **Deployments**, then select a Deployment (in the legacy UI, select a Workspace first). The cloud and region for the cluster running your Deployment appears in the **Cluster** section of your Deployment overview page.

## View dedicated cluster information

1. In the Astro UI, click **Deployments**, then select a Deployment (in the legacy UI, select a Workspace first).
2. Click **Details**. Find the dedicated cluster's name under **Name**.


# View Airflow component and task logs for a Deployment
Source: https://astronomer.io/docs/astro/view-logs

View logs for your Deployments both locally and on Astro.

View task and Airflow component logs to troubleshoot your data pipelines and better understand the health of both your tasks and their execution environment.

Airflow has two different log types:

* *Component logs* record the performance of your Airflow components.
* *Task logs* records events emitted by your dag as each task executes.

<Tip>
  You can retrieve Deployment logs programmatically with the [Get Deployment logs](/docs/astro/api/v-1/deployment/get-deployment-logs) endpoint in the Astro API. The endpoint supports filtering by log source (`scheduler`, `triggerer`, `worker`, `webserver`, `dag-processor`, or `apiserver`), time range, and free-text search, which makes it useful for building custom log workflows or integrations.
</Tip>

Task logs can additionally be exported to third-party observability platforms. See:

* [Export task logs and metrics to Datadog](/docs/astro/export-datadog)
* [Export task logs to AWS CloudWatch](/docs/astro/export-cloudwatch)
* [Export logs to a Secondary S3 Bucket](/docs/astro/export-secondary-s3-bucket)
* [Export logs to a Secondary WASB Container](/docs/astro/export-secondary-wasb)

## Airflow Component Logs

Airflow has four core components: the scheduler, triggerer, worker, and webserver/API server. Each component records its process in component logs. These logs can be used to monitor overall performance, troubleshoot errors, and optimize resources.

* *Dag processor logs* describe the performance of the Dag processor, a sub-component of the Scheduler. The Dag processor is responsible for parsing dags and turning them into dag objects that contain tasks to be scheduled. These logs can help you understand both the Dag processor and scheduler performance. Dag processor logs are not available for Small Deployments because the Dag processor and scheduler run on the same host. For more information on configuring the scheduler on Astro, see [Scheduler resources](/docs/astro/deployment-resources#scheduler).

* *Scheduler logs* describe the performance of the scheduler, which is responsible for scheduling and queueing task runs. These logs can help you understand scheduler performance and indicate if a task failed due to an issue with the scheduler. For more information on configuring the scheduler on Astro, see [Scheduler resources](/docs/astro/deployment-resources#scheduler).

* *Triggerer logs* describe the performance of the triggerer, the Airflow component responsible for running triggers and signaling tasks to resume when their conditions have been met. The triggerer is used exclusively for tasks that are run with [deferrable operators](/docs/learn/deferrable-operators).

* *Worker logs* are generated by Celery Workers and can help you monitor task execution to optimize performance. This type of log is not available when using the Kubernetes Executor.

* *Webserver logs* relate to the health and performance of [the Airflow UI](/docs/learn/airflow-components). If the Airflow UI at any point does not load, for example, webserver/API server logs might indicate why.

### Airflow component log levels

Logs and messages might also be associated with one of the following *log levels*:

* **Error**: Emitted when a process fails or does not complete. For example, these logs might indicate a missing dag file, an issue with your scheduler's connection to the Airflow database, or an irregularity with your scheduler's heartbeat.
* **Warn**: Emitted when Airflow detects an issue that may or may not be of concern but does not require immediate action. This often includes deprecation notices marked as `DeprecationWarning`. For example, Airflow might recommend that you upgrade your Deployment if there was a change to the Airflow database or task execution logic.
* **Info**: Emitted frequently by Airflow to show that a standard scheduler process, such as dag parsing, has started. These logs are frequent and can contain useful information. If you run dynamically generated dags, for example, these logs will show how many dags were created per dag file and how long it took the scheduler to parse each of them.

### View Airflow component logs in the Astro UI

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

You can access scheduler, triggerer, and task logs in the Astro UI to find the past 24 hours of logs for any Deployment on its **Logs** page.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click the **Logs** tab.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
    2. Click the **Logs** tab.
  </Tab>
</Tabs>

The maximum number of lines returned is 10,000, with 50 results displayed per page. If there are no logs available for a given Deployment, the following message appears:

```text wrap theme={null}
No matching events have been recorded in the past 24 hours.
```

Typically, this indicates that the Deployment you selected does not currently have any dags running.

#### Filter options

You can use the following options to specify the types of logs or messages that you want to view.

* **String search**: Enter a string, keyword, or phrase to find in your logs. You can also search with suffix wildcards by adding a `*` to your search query. For example, `acti*` returns results that include `action` and `acting`. The string search does not include fuzzy matching, so misspelled strings or incomplete strings without a wildcard, `*`, return zero results.

* **Time range**: Filter the logs displayed based on time.

* **Log type**: Filter based on whether the log message is from a **DAG Processor**, **Scheduler**, **Worker**, **Webserver/API Server**, or **Triggerer**.

### View Airflow component logs locally

To show logs for your Airflow scheduler, webserver, or triggerer locally, run the following Astro CLI command:

```sh wrap theme={null}
astro dev logs
```

After you run this command, the most recent logs for these components appear in your terminal window.

By default, running `astro dev logs` shows logs for all Airflow components. To see logs only for a specific component, add any of the following flags to your command:

* `--scheduler`
* `--webserver`
* `--triggerer`

To continue monitoring logs, run `astro dev logs --follow`. The `--follow` flag ensures that the latest logs continue to appear in your terminal window. For more information about this command, see [CLI Command Reference](/docs/cli/v1.43/astro-dev-logs).

## Airflow task logs

Airflow task logs can help you troubleshoot a specific task instance that failed or retried. Based on your preference, you can choose to use to access task logs in the Astro UI or the Airflow UI. Both provide filters, search, and download options for task logs and share other information about your dag performance on the same page.

Task logs for Astro Deployments are retained for 90 days. The task log retention policy is not currently configurable.

You can also access local Airflow task logs in your local [Airflow UI](/docs/learn/airflow-ui) or [printed to the terminal](/docs/learn/logging#log-locations).

### Airflow task log levels

Similar to the Airflow component log levels, task logs might also be associated with one of the following log levels, that you can search or filter with:

* **Error**
* **Warn**
* **Info**
* **Debug**
* **Critical**

### View task logs in the Airflow UI

1. Access the Airflow UI.

* To access the Airflow UI for a Deployment, open the Deployment in the Astro UI and click **Open Airflow**.
* To access the Airflow UI in a local environment, open a browser and go to `http://localhost:8080`.

2. Click a dag.
3. Click **Graph**.
4. Click a task run.
5. Click **Instance Details**.
6. Click **Log**.

## See also

* [Export task logs and metrics to Datadog](/docs/astro/export-datadog)
* [Export task logs to AWS CloudWatch](/docs/astro/export-cloudwatch)
* [Export task logs to AWS CloudWatch](/docs/astro/export-cloudwatch)
* [Export logs to a Secondary S3 Bucket](/docs/astro/export-secondary-s3-bucket)
* [Export logs to a Secondary WASB Container](/docs/astro/export-secondary-wasb)


# Astronomer Documentation
Source: https://astronomer.io/docs/index

Everything you need to know about Astronomer's modern data orchestration tool for the cloud, powered by Apache Airflow®.

<div>
  <h1>
    Astronomer Documentation
  </h1>

  <p>
    Everything you need to know about Astronomer's modern data orchestration tool for the cloud, powered by Apache Airflow®.
  </p>
</div>

<div>
  <button type="button" aria-label="Open search">
    <Icon icon="magnifying-glass" />

    <span>Search Astronomer docs...</span>
  </button>
</div>

<div>
  <h2>Explore Astronomer</h2>

  <Columns>
    <Card title="Astro" icon="rocket" href="/astro/overview">
      Build and run your critical data pipelines with Airflow.
    </Card>

    <Card title="Otto" icon="stars" href="/astro/otto-overview">
      Astronomer's data engineering agent, purpose built for Apache Airflow.
    </Card>

    <Card title="Astro Private Cloud" icon="clouds" href="/astro-private-cloud">
      Run Apache Airflow® in your environment.
    </Card>

    <Card title="Astro CLI" icon="terminal" href="/cli/v1.43/overview">
      Get started with Apache Airflow and with all Astronomer products.
    </Card>

    <Card title="Astro Runtime" icon="layer-group" href="/runtime">
      The distribution of Apache Airflow that powers Astro.
    </Card>

    <Card title="Astro Observe" icon="telescope" href="/astro/astro-observe">
      A comprehensive view of your Airflow pipeline data lineage and health.
    </Card>

    <Card title="Learn" icon="fan" href="/learn/overview">
      Tutorials and concepts for everything you need to know about running Airflow 3.
    </Card>

    <Card title="Astro API" icon="gear-code" href="/astro/api/v-1/get-started">
      Automate and integrate with Astro using a fully documented REST API.
    </Card>
  </Columns>
</div>

<div>
  <h2>Get started</h2>

  <Columns>
    <Card title="Understand Apache Airflow® concepts" icon="graduation-cap" href="https://academy.astronomer.io/">
      Free Airflow courses taught by the Astronomer experts behind the project.
    </Card>

    <Card title="Use these docs with AI" icon="robot" href="/use-docs-with-ai">
      Connect an agent to our MCP server, or read the docs as Markdown and llms.txt.
    </Card>
  </Columns>
</div>

<div>
  <h2>What's new</h2>

  <Columns>
    <Card title="Astro Release Notes" icon="stars" href="/astro/release-notes" />

    <Card title="Astro Private Cloud Release Notes" icon="clouds" href="/astro-private-cloud/v-2-x/release-notes" />

    <Card title="Astro CLI Release Notes" icon="terminal" href="/cli/v1.43/release-notes" />

    <Card title="Astronomer Runtime Release Notes" icon="file-check" href="/runtime/runtime-release-notes" />

    <Card title="Remote Execution Agent Release Notes" icon="network-wired" href="/astro/agent-release-notes" />
  </Columns>
</div>


# Use these docs with AI
Source: https://astronomer.io/docs/use-docs-with-ai

Connect an AI agent to Astronomer's documentation through an MCP server, raw Markdown pages, or llms.txt.

You can read Astronomer's documentation the way you always have, by browsing the site. If you're working with an AI agent, you have three additional ways to bring this documentation into its context: a hosted MCP server, raw Markdown pages, and `llms.txt` index files.

## MCP server

Astronomer hosts a Model Context Protocol (MCP) server at `https://www.astronomer.io/docs/mcp`. Connect an MCP-compatible client to it and the agent can search this documentation and pull specific pages into its context as it works, instead of relying on what it already knows.

<Note>
  Opening `https://www.astronomer.io/docs/mcp` in a browser returns a large JSON file. That's expected — it's an API endpoint for MCP clients, not a page meant to be viewed directly.
</Note>

The server exposes three tools:

* **`search_astronomer`** — full-text search across the documentation and API references.
* **`query_docs_filesystem_astronomer`** — read-only, shell-like queries against a virtual filesystem of every documentation page and OpenAPI spec.
* **`submit_feedback`** — report a page that's incorrect, outdated, confusing, or incomplete.

### Connect a client

<Tabs>
  <Tab title="Claude Code">
    ```bash theme={null}
    claude mcp add --transport http astronomer-docs https://www.astronomer.io/docs/mcp
    ```
  </Tab>

  <Tab title="Claude Desktop">
    Go to **Settings** > **Connectors** > **Add custom connector**, and enter `https://www.astronomer.io/docs/mcp` as the server URL.
  </Tab>

  <Tab title="Cursor">
    Open the command palette with <kbd>⌘</kbd>+<kbd>Shift</kbd>+<kbd>P</kbd> (or <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>P</kbd> on Windows/Linux), run **Open MCP settings**, then **Add custom MCP** and enter `https://www.astronomer.io/docs/mcp` as the server URL.
  </Tab>

  <Tab title="VS Code">
    Add an entry to your MCP configuration:

    ```json theme={null}
    {
      "servers": {
        "astronomer-docs": {
          "url": "https://www.astronomer.io/docs/mcp",
          "type": "http"
        }
      }
    }
    ```
  </Tab>
</Tabs>

## Markdown pages

Every documentation page is also available as raw Markdown. Append `.md` to any page's URL to fetch its plain-text source instead of the rendered HTML, for example `https://www.astronomer.io/docs/astro/overview.md`.

On the page itself, use the contextual menu in the top right to copy the page as Markdown or open it directly in ChatGPT or Claude.

## llms.txt

The root [llms.txt](/docs/llms.txt) file is an index of pointers to each product's documentation, in the format proposed by the [llms.txt standard](https://llmstxt.org/). Each product has its own nested `llms.txt`, and versioned products like Astro Private Cloud and the Astro CLI nest one level deeper, with a file per version:

```text theme={null}
/llms.txt                              Index of all products
/astro/llms.txt                        All Astro pages
/runtime/llms.txt                      All Astro Runtime pages
/cli/llms.txt                          Index of CLI versions
  /cli/v1.43/llms.txt                  Astro CLI v1.43 pages
/astro-private-cloud/llms.txt          Index of APC versions
  /astro-private-cloud/v-2-x/llms.txt  Astro Private Cloud v2.x pages
```

These files are generated to cover every supported and archived version of every product, including versions that aren't the default on the site.


# Configure Airflow email notifications on Astro
Source: https://astronomer.io/docs/astro/airflow-email-notifications

Set up email notifications for Airflow task successes and failures.

Incorporating a notification framework is critical to the health of your data pipelines. In addition to [Astro alerts](/docs/astro/alerts), you can configure the following Apache Airflow notification types on Astro:

* Slack notifications
* SLAs
* Email notifications
* Custom callbacks and notifiers

Use this guide to integrate with an SMTP service to have Astro send email notifications whenever a task run fails. To configure Dag alerts for Slack and PagerDuty, see [Astro alerts](/docs/astro/alerts). For best practices and instructions on configuring other notifications in Airflow, including notifiers and custom callbacks, see [Manage Airflow Dag notifications](/docs/learn/error-notifications-in-airflow).

## Configure Airflow email notifications

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

On Astro, setting up email notifications requires configuring an SMTP service for delivering each notification. You can use a single SMTP service for your Organization, but you must configure email notifications for each Deployment.

If your organization isn't using an SMTP service currently, Astronomer recommends one of the following:

* [SendGrid](https://sendgrid.com/)
* [Amazon SES](https://aws.amazon.com/ses/)

The following topics provide setup steps for integrating each of these external SMTP services on Astro, but any external SMTP service can be used.

### Integrate with SendGrid

[SendGrid](https://sendgrid.com/) is an email delivery service that you can use to configure Airflow notifications. A free SendGrid account grants users 40,000 free emails within the first 30 days of an account opening and 100 emails per day after that. This should be enough emails for most notification use cases.

<Tabs>
  <Tab title="New Astro UI">
    1. [Create a SendGrid account](https://signup.sendgrid.com). Be prepared to disclose some standard information about yourself and your organization.

    2. [Verify a Single Sender Identity](https://sendgrid.com/docs/ui/sending-email/sender-verification/). Because you're sending emails for only internal administrative purposes, a single sender identity is sufficient for integrating with Astro. The email address you verify here sends your Airflow notification emails.

    3. Create a Sendgrid API key. In SendGrid, go to **Email API** > **Integration Guide**. Follow the steps to generate a new API key using SendGrid's Web API and cURL.

       For more information, see [Sendgrid documentation](https://docs.sendgrid.com/ui/account-and-settings/api-keys#creating-an-api-key).

    4. Skip the step for exporting your API key to your development environment. Instead, execute the generate cURL code directly in your command line, making sure to replace `$SENDGRID_API_KEY` in the `--header` field with your copied key.

    5. Verify your integration in SendGrid to confirm that the key was activated. If you get an error indicating that SendGrid can't find the test email, try rerunning the cURL code in your terminal before retrying the verification.

    6. Add the following line to the `requirements.txt` file of your Astro project to install the [SendGrid Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-sendgrid/stable/index.html):

       ```text wrap theme={null}
       apache-airflow-providers-sendgrid
       ```

    7. In the Astro UI, click **Environment** > **Environment Variables** > **New Environment Variable** to add the following environment variables. Select your workspace from the drop-down, and then add the following variables, one at a time:

       | Environment variable key        | Environment variable value                            |
       | ------------------------------- | ----------------------------------------------------- |
       | `AIRFLOW__EMAIL__EMAIL_BACKEND` | `airflow.providers.sendgrid.utils.emailer.send_email` |
       | `AIRFLOW__EMAIL__EMAIL_CONN_ID` | `smtp_default`                                        |
       | `SENDGRID_MAIL_FROM`            | `<validated-sendgrid-sender-email-address>`           |

       For more information about these environment variables, see [Send email using SendGrid](https://airflow.apache.org/docs/apache-airflow/stable/howto/email-config.html#send-email-using-sendgrid).

    8. In the Airflow UI, [create an Airflow connection](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#creating-a-connection-with-the-ui) with the following values:

       * **Connection ID**: `smtp_default`
       * **Connection Type:**: `Email`
       * **Host**: `smtp.sendgrid.net`
       * **Login**: `apikey`
       * **Password**: `<your-sendgrid-api-key>`
       * **Port**: `587`

    9. Click **Save** to finalize your configuration.

    10. To receive email notifications for task failures within a given Dag, configure the following values in the Dag's `default_args`:

        ```python wrap theme={null}
        'email_on_failure': True,
        'email': ['<recipient-address>'],
        ```

    Repeat steps 7-10 for each Deployment where you want to configure Airflow notifications.
  </Tab>

  <Tab title="Legacy UI">
    1. [Create a SendGrid account](https://signup.sendgrid.com). Be prepared to disclose some standard information about yourself and your organization.

    2. [Verify a Single Sender Identity](https://sendgrid.com/docs/ui/sending-email/sender-verification/). Because you're sending emails for only internal administrative purposes, a single sender identity is sufficient for integrating with Astro. The email address you verify here sends your Airflow notification emails.

    3. Create a Sendgrid API key. In SendGrid, go to **Email API** > **Integration Guide**. Follow the steps to generate a new API key using SendGrid's Web API and cURL.

       For more information, see [Sendgrid documentation](https://docs.sendgrid.com/ui/account-and-settings/api-keys#creating-an-api-key).

    4. Skip the step for exporting your API key to your development environment. Instead, execute the generate cURL code directly in your command line, making sure to replace `$SENDGRID_API_KEY` in the `--header` field with your copied key.

    5. Verify your integration in SendGrid to confirm that the key was activated. If you get an error indicating that SendGrid can't find the test email, try rerunning the cURL code in your terminal before retrying the verification.

    6. Add the following line to the `requirements.txt` file of your Astro project to install the [SendGrid Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-sendgrid/stable/index.html):

       ```text wrap theme={null}
       apache-airflow-providers-sendgrid
       ```

    7. In the Deployment view of the Astro UI, add the following environment variables

       | Environment variable key        | Environment variable value                            |
       | ------------------------------- | ----------------------------------------------------- |
       | `AIRFLOW__EMAIL__EMAIL_BACKEND` | `airflow.providers.sendgrid.utils.emailer.send_email` |
       | `AIRFLOW__EMAIL__EMAIL_CONN_ID` | `smtp_default`                                        |
       | `SENDGRID_MAIL_FROM`            | `<validated-sendgrid-sender-email-address>`           |

       For more information about these environment variables, see [Send email using SendGrid](https://airflow.apache.org/docs/apache-airflow/stable/howto/email-config.html#send-email-using-sendgrid).

    8. In the Airflow UI, [create an Airflow connection](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#creating-a-connection-with-the-ui) with the following values:

       * **Connection ID**: `smtp_default`
       * **Connection Type:**: `Email`
       * **Host**: `smtp.sendgrid.net`
       * **Login**: `apikey`
       * **Password**: `<your-sendgrid-api-key>`
       * **Port**: `587`

    9. Click **Save** to finalize your configuration.

    10. To receive email notifications for task failures within a given Dag, configure the following values in the Dag's `default_args`:

        ```python wrap theme={null}
        'email_on_failure': True,
        'email': ['<recipient-address>'],
        ```

    Repeat steps 7-10 for each Deployment where you want to configure Airflow notifications.
  </Tab>
</Tabs>

### Integrate with Amazon SES

Use your existing Amazon SES instance to send Airflow notifications by email.

<Tabs>
  <Tab title="New Astro UI">
    1. Sign in to the AWS Management Console and open the [Amazon SES console](https://console.aws.amazon.com/ses/).

    2. Click the menu icon and then click **Verified Identities**.

    3. Optional. Complete one of the following tasks:

       * To confirm the email account is working and the Amazon SES service can send emails to it, click an email address in the **Identity** column and then click **Send test email**. When the owner of the email address receives the test email, they need to select the validation link in the email.
       * To add a new email address, click **Create Identity**, add the email address, and then click **Create Identity**.

         For email notifications, Astronomer recommends using one email address as the sender and a second email address as the recipient. All email addresses must be verified.

         For more information about configuring Amazon SES, read [Creating an email address identity](https://docs.aws.amazon.com/ses/latest/dg/creating-identities.html#:~:text=of%20those%20Regions.-,Creating%20an%20email%20address%20identity,-Complete%20the%20following) and [Verifying an email address identity](https://docs.aws.amazon.com/ses/latest/dg/creating-identities.html#:~:text=address%20identity.-,Verifying%20an%20email%20address%20identity,-After%20you%E2%80%99ve%20created) in the Amazon documentation.

    4. Click **Account dashboard**.

    5. In the **Simple Mail Transfer Protocol (SMTP) settings** pane, copy the value displayed below **SMTP endpoint**.

    6. Obtain your Amazon SES SMTP credentials:
       * Click **Create SMTP credentials** in the **Simple Mail Transfer Protocol (SMTP) settings** pane.
       * Enter a name for your SMTP user in the **IAM User Name** field or accept the default value.
       * Click **Create** and then **Show User SMTP Security Credentials**.
       * Click **Download Credentials** or copy them and store them in a safe place.
       * Click **Close Window**.

    7. In the Astro UI, click **Environment** > **Environment Variables** > click on the variable you want to edit, and then click **Edit**. Select your workspace from the drop-down, and then add the following variables, one at a time:

       | Environment variable key        | Environment variable value                  |
       | ------------------------------- | ------------------------------------------- |
       | `AIRFLOW__SMTP__SMTP_HOST`      | Enter the value you copied in step 5        |
       | `AIRFLOW__SMTP__SMTP_STARTTLS`  | Enter `True`                                |
       | `AIRFLOW__SMTP__SMTP_SSL`       | Enter `False`                               |
       | `AIRFLOW__SMTP__SMTP_USER`      | Enter the value you copied in step 6        |
       | `AIRFLOW__SMTP__SMTP_PASSWORD`  | Enter the value you copied in step 6        |
       | `AIRFLOW__SMTP__SMTP_PORT`      | Enter `587`                                 |
       | `AIRFLOW__SMTP__SMTP_MAIL_FROM` | Enter your from email                       |
       | `AIRFLOW__EMAIL__EMAIL_BACKEND` | Enter `airflow.utils.email.send_email_smtp` |

       See [Set environment variables on Astro](/docs/astro/environment-variables).

    8. To begin receiving Airflow notifications by email for task failures within a given Dag, configure the following values in the Dag's `default_args`:

       ```python wrap theme={null}
       'email_on_failure': True,
       'email': ['<recipient-address>'],
       ```

    Repeat steps 7-8 for each Deployment in which you want to configure Airflow notifications.
  </Tab>

  <Tab title="Legacy UI">
    1. Sign in to the AWS Management Console and open the [Amazon SES console](https://console.aws.amazon.com/ses/).

    2. Click the menu icon and then click **Verified Identities**.

    3. Optional. Complete one of the following tasks:

       * To confirm the email account is working and the Amazon SES service can send emails to it, click an email address in the **Identity** column and then click **Send test email**. When the owner of the email address receives the test email, they need to select the validation link in the email.
       * To add a new email address, click **Create Identity**, add the email address, and then click **Create Identity**.

         For email notifications, Astronomer recommends using one email address as the sender and a second email address as the recipient. All email addresses must be verified.

         For more information about configuring Amazon SES, read [Creating an email address identity](https://docs.aws.amazon.com/ses/latest/dg/creating-identities.html#:~:text=of%20those%20Regions.-,Creating%20an%20email%20address%20identity,-Complete%20the%20following) and [Verifying an email address identity](https://docs.aws.amazon.com/ses/latest/dg/creating-identities.html#:~:text=address%20identity.-,Verifying%20an%20email%20address%20identity,-After%20you%E2%80%99ve%20created) in the Amazon documentation.

    4. Click **Account dashboard**.

    5. In the **Simple Mail Transfer Protocol (SMTP) settings** pane, copy the value displayed below **SMTP endpoint**.

    6. Obtain your Amazon SES SMTP credentials:
       * Click **Create SMTP credentials** in the **Simple Mail Transfer Protocol (SMTP) settings** pane.
       * Enter a name for your SMTP user in the **IAM User Name** field or accept the default value.
       * Click **Create** and then **Show User SMTP Security Credentials**.
       * Click **Download Credentials** or copy them and store them in a safe place.
       * Click **Close Window**.

    7. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.

    8. In the environment variables area, click **Edit Deployment Variables** (or **+ New Environment Variable** if you have no variables configured yet) and add the following environment variables:

       | Environment variable key        | Environment variable value                  |
       | ------------------------------- | ------------------------------------------- |
       | `AIRFLOW__SMTP__SMTP_HOST`      | Enter the value you copied in step 5        |
       | `AIRFLOW__SMTP__SMTP_STARTTLS`  | Enter `True`                                |
       | `AIRFLOW__SMTP__SMTP_SSL`       | Enter `False`                               |
       | `AIRFLOW__SMTP__SMTP_USER`      | Enter the value you copied in step 6        |
       | `AIRFLOW__SMTP__SMTP_PASSWORD`  | Enter the value you copied in step 6        |
       | `AIRFLOW__SMTP__SMTP_PORT`      | Enter `587`                                 |
       | `AIRFLOW__SMTP__SMTP_MAIL_FROM` | Enter your from email                       |
       | `AIRFLOW__EMAIL__EMAIL_BACKEND` | Enter `airflow.utils.email.send_email_smtp` |

       See [Set environment variables on Astro](/docs/astro/environment-variables).

    9. To begin receiving Airflow notifications by email for task failures within a given Dag, configure the following values in the Dag's `default_args`:

       ```python wrap theme={null}
       'email_on_failure': True,
       'email': ['<recipient-address>'],
       ```

    Repeat steps 7-9 for each Deployment in which you want to configure Airflow notifications.
  </Tab>
</Tabs>


# Set up Astro alerts
Source: https://astronomer.io/docs/astro/alerts

Configure Astro alerts to notify you in Slack, PagerDuty, or email when Dags complete, fail, or exceed expected task durations — without changing Dag code.

Astro alerts provide an additional level of observability to Airflow's notification systems. You can configure an alert to notify you in Slack, PagerDuty, or through email when a Dag completes, if you have a Dag run failure, or if a task duration exceeds a specified time. You can also define whether alerts apply to a specific Deployment or across an entire Workspace or Organization.

Unlike Airflow callbacks and SLAs, Astro alerts don't require changes to Dag code. Follow this guide to set up your Slack, PagerDuty, or email to receive alerts from Astro and then configure your Deployment to send alerts.

<Info>For configuring Airflow notifications, see [Airflow email notifications](/docs/astro/airflow-email-notifications) and [Manage Airflow Dag notifications](/docs/learn/error-notifications-in-airflow).</Info>

## Alert types

Each Astro alert has a notification channel and a trigger type. The notification channel determines the format and destination of an alert and the trigger type defines what triggers the alert.

### Dag and task alerts

You can trigger an alert to a notification channel using one of the following trigger types:

* **DAG Failure**: The alert triggers whenever the specified Dag fails. From the alert notification, you can open the failed Dag run in the Astro UI to start an Otto investigation. To investigate Dag failures automatically when this alert fires, send it to a **Dag Trigger** notification channel that calls the Otto investigation API. See [Investigate with Otto](/docs/astro/otto-investigate).
* **DAG Success**: The alert triggers whenever the specified Dag completes.
* **DAG Timeliness**: The alert triggers whenever the specified Dag doesn't produce a successful Dag run by the given **Verification Time** including the **Look Back Period**.
* **DAG Duration**: The alert triggers whenever the specified Dag doesn't produce a successful Dag run within the given **Duration**. If the Dag fails before the specified duration is exceeded, the alert is triggered.
* **Task Failure**: The alert triggers whenever the specified task fails.
* **Task Duration**: The alert triggers when a specified task takes longer than expected to complete successfully. If the task fails, the alert is triggered even if it fails before the specified duration is exceeded.

<Info>You can only set a task duration alert for an individual task. Alerting on task group duration isn't supported.</Info>

<Warning>Timeliness alerts only support Standard Time, as opposed to Daylight Saving Time. If you want Local Time support (for example, for [time zone aware Dags](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timezone.html#time-zone-aware-dags)), you must adjust the alert's UTC time when the time changes from Standard Time to Daylight Saving Time or from Daylight Saving Time to Standard Time.</Warning>

### Deployment health alerts

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

Deployment health alerts are customizable Astro Alerts that notify you about [Deployment health incidents](/docs/astro/deployment-health-incidents#deployment-incidents) and suggest specific remediation actions. You can use these alerts to proactively monitor when Deployment health issues arise. For example, you can create an alert for when the Airflow metadata database storage is unusually high. All available Deployment health alerts are enabled by default for new Deployments and can be individually created, deleted, and edited in the **Alerts** tab of existing Deployments. If you don't want Deployment health alerts for a new Deployment, deselect **Deployment Health Alerts** in the **Advanced** section of the Deployment creation page.

Astro enables the following Deployment health alerts by default for new Deployments. You can individually create and customize these alerts for any Deployment:

* **Airflow DB Storage Unusually High**: The alert triggers when the metadata database has tables that are larger than 50GiB (Info) or 75GiB (Warning).
* **Deprecated Runtime Version**: The alert triggers when your Deployment is using a deprecated Astro Runtime version.
* **Job Scheduling Disabled**: The alert triggers when the Airflow scheduler is configured to prevent automatic scheduling of new tasks using Dag schedules.
* **Worker Queue at Capacity**: The alert triggers when at least one worker queue in this Deployment is running the maximum number of tasks and workers.

### Deployment health alert notifications

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

For new Deployments, Astro creates the default [Deployment health alerts](#deployment-health-alerts) with **Email** as the notification channel **Type**. By default, these alerts notify the **Contact Emails** of the Deployment, specified in the **Advanced** section of the Deployment creation page.

If **Contact Emails** is empty, Astro displays the [fallback email(s)](/docs/astro/deployment-details#fallback-emails) that receives alert notifications for the Deployment in the **Advanced** section of Deployment configuration. To change this notification channel, ensure the **Contact Emails** field of your Deployment isn't blank by editing **Deployment Details.**

## Create an alert in the Astro UI

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

### Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project).
* An [Astro Deployment](/docs/astro/create-deployment). Your Deployment must run Astro Runtime 7.1.0 or later to configure Astro alerts and it must have [OpenLineage enabled](/docs/astro/observe-openlineage).
* A configured [notification channel](#configure-and-add-alert-notification-channels).

<Tip>You can also work with Astro alerts and their notification channels with the [Astro API](/docs/astro/api/v-1-beta-1/platform/alerts/create-an-alert).</Tip>

Astro alerts requires OpenLineage. By default, every Astro Deployment using Hosted Execution has OpenLineage enabled. If you disabled OpenLineage in your Astro Deployment, you need to enable it to use Astro alerts. See [Disable OpenLineage](/docs/astro/observe-openlineage#disable-openlineage-completely) to find how to disable and re-enable OpenLineage.

If you use [Remote Execution](/docs/astro/execution-mode#remote-execution), you must [enable OpenLineage for Remote Execution Agents](/docs/astro/remote-execution-configure-openlineage) to use Astro Alerts or Observe.

### Step 1: Create your alert

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Alerting** > **Alerts**.

       <Accordion title="Deployment-specific alerts">
         If you don't have permissions to add a notification channel or alert at your Workspace or Organization-level, you can add one in the **Alerts** form through the **Deployment settings** page.

         1. In the Astro UI, click **Deployments** then select your Deployment.
         2. Click the **Alerts** tab.
       </Accordion>

    2. Click **New Alert**.

    3. Choose the **Alert Type**.

       * **Duration**: Enter the **Duration** for how long a Dag or task should take to run before you send an alert to your notification channels.

       * **Timeliness**: Select the **Days of Week** that the alert should observe, the **Verification Time** when it should look for a Dag success, and the **Lookback Period** for how long it should look back for a verification time. For example, if an alert has a **Verification Time** of 3:00 PM UTC and a **Lookback Period** of 60 minutes, it will trigger whenever the given Dag doesn't produce a successful Dag run from 2:00 to 3:00 PM UTC.

    <Note>
      **Initial evaluation behavior**

      Alerts only evaluate runs that occur after the alert’s creation. If a Dag ran successfully within the **Look Back Period** of a timeliness alert, the alert will still trigger on initial evaluation, since that run occurred before the alert was created.
    </Note>

    <Warning>Timeliness alerts only support Standard Time, as opposed to Daylight Saving Time. If you want Local Time support (for example, for [time zone aware Dags](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timezone.html#time-zone-aware-dags)), you must adjust the alert's UTC time when the time changes from Standard Time to Daylight Saving Time or from Daylight Saving Time to Standard Time.</Warning>

    4. Choose the alert **Severity**, either **Info**, **Warning**, or **Critical**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Alerting** > **Alerts**.

       <Accordion title="Deployment-specific alerts">
         If you don't have permissions to add a notification channel or alert at your Workspace or Organization-level, you can add one in the **Alerts** form through the **Deployment settings** page.

         1. In the Astro UI, click **Deployments** then select your Deployment.
         2. Click the **Alerts** tab.
       </Accordion>

    2. Click **+ New Alert**.

    3. Choose the **Alert Type**

       * **Duration**: Enter the **Duration** for how long a Dag or task should take to run before you send an alert to your notification channels.

       * **Timeliness**: Select the **Days of Week** that the alert should observe, the **Verification Time** when it should look for a Dag success, and the **Lookback Period** for how long it should look back for a verification time. For example, if an alert has a **Verification Time** of 3:00 PM UTC and a **Lookback Period** of 60 minutes, it will trigger whenever the given Dag doesn't produce a successful Dag run from 2:00 to 3:00 PM UTC.

    <Note>
      **Initial evaluation behavior**

      Alerts only evaluate runs that occur after the alert’s creation. If a Dag ran successfully within the **Look Back Period** of a timeliness alert, the alert will still trigger on initial evaluation, since that run occurred before the alert was created.
    </Note>

    <Warning>Timeliness alerts only support Standard Time, as opposed to Daylight Saving Time. If you want Local Time support (for example, for [time zone aware Dags](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timezone.html#time-zone-aware-dags)), you must adjust the alert's UTC time when the time changes from Standard Time to Daylight Saving Time or from Daylight Saving Time to Standard Time.</Warning>

    4. Choose the alert **Severity**, either **Info**, **Warning**, or **Critical**.
  </Tab>
</Tabs>

### Step 2: Add alert rules

1. Define the **Workspace** and **Deployment**

<Warning>
  The following alert types must be defined at the Deployment level:

  * Dag alerts (Dag Duration, Dag Failure, Dag Success, Dag Timeliness)
  * Deployment health alerts
  * Task alerts (Task Duration, Task Failure)
</Warning>

2. For Dag and Task Alerts, you can further customize the circumstances in which the alert is triggered by defining:

   * **Attribute**
     * For Dag-level alerts: **DAG ID**
     * For Task-level alerts: You can define rules for both the **DAG ID** and **Task ID** attributes.
   * **Operator**
     * **is one of**: Select specific Dags or tasks the alert should apply to. You can also choose **All dags** to apply the alert to all Dags in the Deployment and **All Tasks** to apply the alert to all tasks in the Dag.
     * **contains**: Define a substring that is in the **DAG ID** and/or **Task ID** for the Dags and/or tasks that the alert should apply to.
   * **Values**/**Substring**

     The following example is a task-level alert for all tasks with a **Task ID** that contains the substring `prod` in all Dags within a Deployment:

     <Frame>
       <img alt="Alert rules." />
     </Frame>

### Step 3: Add notification channels for the alert

Select the Notification Channels for the Alert. You can also create a new notification channel by clicking **+Notification Channel**. See [Notification Channels](#configure-and-add-alert-notification-channels) for information on configuring and adding Notification Channels.

### Step 4: (Optional) Change an alert name

After you select a Dag that you want to apply an alert to, Astro automatically generates a name for your alert. However, you can choose to change the name of your alert.

1. Expand the **Change alert names...** section.

2. Edit the **Alert Name**.

3. Click **Create Alert** to save your changes.

### Step 5: (Optional) Test your Dag failure alert

Astro alerts work whether your Dag run is manual or scheduled, so you can test your configured Astro alerts by failing your Dag manually.

1. In the Astro UI, click **DAGs**.

2. Choose the Dag that has your alert configured.

3. Trigger a Dag run.

4. Select **Mark as** and choose **Failed** to trigger an alert for a Dag failure.

   <Frame>
     <img alt="Manually marking a successful Dag run as Failed." />
   </Frame>

5. Check your Slack, PagerDuty, or Email alerts for your Dag failure alert. The alert includes information about the Dag, Workspace, Deployment, and data lineage associated with the failure as well as direct links to the Astro UI.

   <Frame>
     <img alt="Example of a Slack test alert." />
   </Frame>

## Configure and add alert notification channels

You can send Astro alerts, including Deployment Health Alerts, to the following notification channels

* Slack
* PagerDuty
* Email
* Dag trigger
* Opsgenie

<Note>For alerts triggered by a specific Dag run, Slack and email notifications include the Airflow Dag run ID. Use it to find the run in the Airflow UI.</Note>

<Warning>The **Dag Trigger** notification channel works differently from other notification channel types. Instead of sending a pre-formatted alert message, Astro makes a generic request through the Airflow REST API to trigger a Dag on Astro. You can configure the triggered Dag to complete any action, such as sending a message to your own incident management system or writing data about an incident to a table.</Warning>

### Notification channels scope

When you create a notification channel, you can define whether it is available to a specific Deployment or available to an entire Workspace or Organization. The type of scope you can use for your notification channels depends on your [user permissions](/docs/astro/user-permissions).

<Frame>
  <img alt="Choose a notification channel scope" />
</Frame>

You can view, create, and manage notification channels for your Workspace or Organization in the **Notification Channels** page or when creating a Deployment alert.

### Step 1: Configure your notification channel

<Tabs>
  <Tab title="Slack">
    To set up alerts in Slack, you need to create a Slack app in your Slack workspace. After you've created your app, you can generate a webhook URL in Slack where Astro will send alerts.

    1. Go to [Slack API: Applications](https://api.slack.com/apps/new) to create a new app in your organization's Slack workspace.

    2. Click **From scratch** when prompted to choose how you want to create your app.

    3. Enter a name for your app, like `astro-alerts`, choose the Slack workspace where you want Astro to send your alerts, and then click **Create App**.

    <Info>If you don't have permission to install apps into your Slack workspace, you can still create the app, but you will need to request that an administrator from your team completes the installation.</Info>

    4. Select **Incoming webhooks**.

    5. On the **Incoming webhooks** page, click the toggle to turn on **Activate Incoming Webhooks**. See [Sending messages using Incoming Webhooks](https://api.slack.com/messaging/webhooks).

    6. In the **Webhook URLs for your Workspace** section, click **Add new Webhook to Workspace**.

    <Info>If you don't have permission to install apps in your Slack workspace, click **Request to Add New Webhook** to send a request to your organization administrator.</Info>

    7. Choose the channel where you want to send your Astro alerts and click **Allow**.

    8. After your webhook is created, copy the webhook URL from the new entry in the **Webhook URLs for your Workspace** table.
  </Tab>

  <Tab title="PagerDuty">
    To set up an alert integration with PagerDuty, you need access to your organization's PagerDuty Service. PagerDuty uses the [Events API v2](https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTgw-events-api-v2-overview#getting-started) to create a new integration that connects your Service with Astro.

    1. Open your PagerDuty service and click the **Integrations** tab.

    <Frame>
      <img alt="Select PagerDuty integrations" />
    </Frame>

    2. Click **Add an integration**.

    3. Select **Events API v2** as the **Integration Type**.

    4. On your **Integrations** page, open your new integration and enter an **Integration Name**.

    5. Copy the **Integration Key** for your new Astro alert integration.
  </Tab>

  <Tab title="Email">
    No external configuration is required for the email integration. Astronomer recommends allowlisting `astronomer.io` with your email provider to ensure that no alerts go to your spam folder. Alerts are sent from `postmaster@astronomer.io`.
  </Tab>

  <Tab title="Dag Trigger">
    The **Dag Trigger** notification channel works differently from other notification channel types. Instead of sending a pre-formatted alert message, Astro makes a generic request through the `DagRuns` endpoint of the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/post_dag_run) to trigger any Dag in your Workspace. You can configure the triggered Dag to complete any action, such as sending a message to your own incident management system or writing data about an incident to a table.

    The alert payload includes the following parameters:

    * `conf`: This parameter holds the alert payload:
    * `alertId`: A unique alert ID.
    * `alertType`: The type of alert triggered.
    * `dagName`: The name of the Dag that the alert condition triggers.
    * `message`: The detailed message with the cause of the alert.
    * `airflowDagRunId`: The Airflow run ID of the Dag run that caused the alert. For example, `scheduled__2023-11-17T17:30:00+00:00`. This parameter is omitted when no Dag run is associated with the alert.
    * `note`: By default, this is `Triggering dag on Airflow <url>`.

    The following is an example alert payload that would be passed through the API to trigger the `alert_dag` if `fail_dag` fails:

    ```json wrap theme={null}
    {
      "dagName": "alert_dag",
      "alertType": "PIPELINE_FAILURE",
      "alertId": "d75e7517-88cc-4bab-b40f-660dd79df216",
      "airflowDagRunId": "scheduled__2023-11-17T17:30:00+00:00",
      "message": "[Astro Alerts] Pipeline failure detected on dag fail_dag. \\nStart time: 2023-11-17 17:32:54 UTC. \\nFailed at: 2023-11-17 17:40:10 UTC. \\nAlert notification time: 2023-11-17 17:40:10 UTC. \\nClick link to investigate in Astro UI: https://cloud.astronomer.io/clkya6zgv000401k8zafabcde/dags/clncyz42l6957401bvfuxn8zyxw/fail_dag/c6fbe201-a3f1-39ad-9c5c-817cbf99d123?utm_source=alert\"\\n"
    }
    ```

    These parameters are accessible in the triggered Dag using [Dag params](/docs/learn/airflow-params).

    1. Create a Dag that you want to run when the alert is triggered. For example, you can use the following Dag to run arbitrary Python code when the alert is triggered:

    ```python wrap theme={null}
    from datetime import datetime
    from typing import Any

    from airflow.models.dag import DAG
    from airflow.operators.python import PythonOperator

    with DAG(
        dag_id="register_incident",
        start_date=datetime(2023, 1, 1),
        schedule=None,
    ):

        def _register_incident(params: dict[str, Any]):
            failed_dag = params["dagName"]
            print(f"Register an incident in my system for DAG {failed_dag}.")

        PythonOperator(task_id="register_incident", python_callable=_register_incident)

    ```

    2. Deploy the Dag to any Deployment in the Workspace where you want to create the alert. The Dag that triggers the alert and the Dag that the alert runs can be in different Deployments, but they must be deployed in the same Workspace.

    3. Create a [Deployment API token](/docs/astro/deployment-api-tokens) for the Deployment where you deployed the Dag that the alert will run. Copy the token to use in the next step.
  </Tab>
</Tabs>

### Step 2: Add a notification channel in the Astro UI

You can add notification channels in the Astro UI.

1. In the Astro UI, click **Alerting** > **Notification Channels**.

2. Click **New Notification Channel**.

3. Enter a name for your notification channel.

4. Choose the Channel Type.

5. Add the notification channel information.

<Tabs>
  <Tab title="Slack">
    Paste the Webhook URL from your Slack workspace app. If you need to find a URL for an app you've already created, go to your [Slack Apps](https://api.slack.com/apps) page, select your app, and then choose the **Incoming Webhooks** page.
  </Tab>

  <Tab title="PagerDuty">
    Paste the Integration Key from your PagerDuty Integration and select the **Severity** of the alert.
  </Tab>

  <Tab title="Email">
    Enter the email addresses that should receive the alert.
  </Tab>

  <Tab title="Dag Trigger">
    Select the Deployment where your Dag is deployed, then select the Dag. Enter the Deployment API token that you created in Step 1.

    <Frame>
      <img alt="Add an email address" />
    </Frame>
  </Tab>
</Tabs>

6. Choose the scope by defining what you want to make **Notification channel available to:**

7. Click **Create notification channel**.

## Notification History

To view a historical log of all previously triggered Alerts for your Organization, click **Notification History** under the **Alerting** section of the navigation. You can filter the notification log based on:

* **Alert**: Which Alert triggered the notification
* **Time Range**: The time range the notification was triggered in
* **Notification Channel**: The notification channel that is defined for the Alert
* **Status**: Whether the notification was successfully **sent** or **failed**

To view the notification history for a particular Alert, click **Alerts** in the **Alerting** section of the navigation. Here, you can see the number of times an Alert sent a notification (in the "Sent" column) or failed to send a notification (in the "Failed" column). Click the number in the **Sent** column to see notification history for that Alert.


# API authentication and token security
Source: https://astronomer.io/docs/astro/api-authentication

How Astro authenticates API requests, including token types, JWT validation, default and configurable lifetimes, rotation, and revocation.

Astro authenticates programmatic and UI-driven API requests with bearer tokens. This page describes the token types Astro supports, how each is validated, default and configurable lifetimes, rotation and revocation behavior, and current limitations. Use it to evaluate the Astro API authentication model from a security perspective.

For step-by-step instructions on creating each token type, see [Organization API tokens](/docs/astro/organization-api-tokens), [Workspace API tokens](/docs/astro/workspace-api-tokens), [Deployment API tokens](/docs/astro/deployment-api-tokens), and [Authenticate an automation tool to Astro](/docs/astro/automation-authentication).

## Token types

Astro supports two categories of bearer tokens:

* **Customer-created API tokens**. You create these tokens in the Astro UI or with the Astro API to authenticate automation, CI/CD pipelines, and external integrations. Astro scopes each token to an Organization, Workspace, or Deployment.
* **UI-issued short-lived tokens**. The Astro UI requests these tokens from its `/token` endpoint after you sign in. The UI and Astro CLI use them to make authenticated requests on your behalf during an active session.

The following table summarizes each token type:

| Token type              | Scope        | Issuer | Default lifetime | Configurable lifetime                    |
| ----------------------- | ------------ | ------ | ---------------- | ---------------------------------------- |
| Organization API token  | Organization | Astro  | Set at creation  | Yes, at creation                         |
| Workspace API token     | Workspace    | Astro  | Set at creation  | Yes, at creation                         |
| Deployment API token    | Deployment   | Astro  | Set at creation  | Yes, at creation                         |
| UI-issued session token | User session | Auth0  | 8 hours          | Up to 30 days through refresh token flow |

## Token validation

All Astro bearer tokens are JSON Web Tokens (JWTs). Astro validates each request by verifying the token's signature, issuer, audience, and expiration (`exp`) claim before authorizing the request.

* Customer-created Organization, Workspace, and Deployment API tokens are JWTs that Astro signs and validates.
* UI-issued session tokens are JWTs that Auth0 signs. Astro validates these tokens through the [Auth0 Identifier First Authentication flow](https://auth0.com/docs/authenticate/login/auth0-universal-login/identifier-first).

Astro rejects tokens with invalid signatures, missing or incorrect claims, or expired `exp` values.

## Token lifetime

You control how long customer-created API tokens remain valid. When you create an Organization, Workspace, or Deployment API token, set the **Expiration** field to the number of days that the token can be used. After the expiration, Astro rejects the token. Astronomer recommends setting the shortest expiration that meets your automation requirements.

UI-issued session tokens have a default lifetime of 8 hours. The Astro UI can extend an active session up to 30 days through a refresh token flow. After this maximum, you must sign in again to obtain a new session token.

## Token rotation and revocation

Astro provides rotation and revocation controls for customer-created API tokens:

* **Rotation**. You can rotate an Organization, Workspace, or Deployment API token from the Astro UI or with the Astro API. Rotation issues a new token value and invalidates the previous value. See [Rotate an Organization API token](/docs/astro/organization-api-tokens#rotate-an-organization-api-token) for an example.
* **Revocation**. Deleting an API token immediately revokes it. Astro rejects any subsequent request that presents the deleted token.

UI-issued session tokens can't be rotated directly. To invalidate an active session, sign out of the Astro UI, which ends the session and invalidates the associated refresh token.

<Note>
  Token rotation replaces the previous token value immediately. The previous value stops working as soon as Astro issues the new value. Plan rotations to minimize the gap between updating the new value in your automation and the previous value becoming invalid.
</Note>

## Current limitations

The Astro API authentication model has the following current limitations:

* **No Organization-wide maximum-lifetime policy**. Organization Owners can't enforce a maximum expiration across all API tokens created in an Organization. Each token creator sets the expiration at creation time. Astronomer recommends defining and communicating an internal policy for maximum token lifetimes, and auditing existing tokens through the Astro UI.
* **No grace period during rotation**. The public rotation endpoint doesn't allow the previous token value to remain valid alongside the new value for a grace period. To minimize automation downtime, update the new token value in all consumers as soon as you rotate the token.

## See also

* [Organization API tokens](/docs/astro/organization-api-tokens)
* [Workspace API tokens](/docs/astro/workspace-api-tokens)
* [Deployment API tokens](/docs/astro/deployment-api-tokens)
* [Authenticate an automation tool to Astro](/docs/astro/automation-authentication)
* [User permissions reference](/docs/astro/user-permissions)


# Assets overview
Source: https://astronomer.io/docs/astro/assets-overview

About data assets in Astro Observe.

The **Asset Catalog** captures Airflow assets (Dags, tasks, datasets) and data assets (tables) that are automatically emitted during Airflow job runs. Assets in Astro represent the storage and movements of data through Airflow jobs.

<Note>
  Astro Assets are distinct from [Airflow Assets](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/assets.html), a user-defined logical grouping of data. Astro Assets include but aren't limited to Airflow Assets.
</Note>

You can find a list of assets in your Organization in the **Asset Catalog** page of Observe.

## Add assets to Observe

After your Dags run at least once, Observe automatically stores information about the tasks, datasets, warehouse tables, and local files you use in your Dags as Assets. You can then find these assets in the **Asset Catalog**.

## View the Asset Catalog

To view the **Asset Catalog**:

In the Astro UI, click **Observe** > **Catalog** (**Asset Catalog** in the legacy UI).

The Asset Catalog displays all assets in your Organization. Each asset shows:

* Asset name and type
* Deployment name
* Dag ID for Airflow assets
* Operator Class for Airflow tasks
* Previous 14 runs status
* Last updated timestamp
* Owner
* Tags

## Search and filter assets

Use the search and filter controls to find specific assets:

* **Search**: Use the search box to filter assets by name.
* **Sort**: Sort assets by last updated time.
* **Asset Type**: Filter by asset type, including:
  * Airflow Task
  * Airflow Dag
  * Airflow Dataset
  * OpenLineage Dataset
  * Databricks Table
  * Snowflake Table
* **Namespaces**: Filter assets by namespace.
* **DAGs**: Filter assets by Dag.
* **Tags**: Filter assets by tag.
* **Dependencies**: Filter by dependency type:
  * **Leaf Assets**: Assets that have no downstream dependencies.
  * **Root Assets**: Assets that have no upstream dependencies.

## Access asset information

Click an asset from the Asset Catalog to view detailed information about that asset.

After you open a specific asset, you can see detailed information about the Asset and its performance in each tab.

### Details

The **Details** tab includes metadata and information about your asset, organized in the following sections:

* Basic: information about Asset metadata
* Task metadata (Airflow tasks only): Specific task metadata, such as the Dag ID, owner, operator class, and task arguments
* Source: The last accessor of the asset

### Event timeline

The **Event Timeline** tab shows a chronological view of all events associated with the asset, including task successes, failures, SLA breaches, and dataset writes. Use the timeline to track asset activity, troubleshoot issues, and understand historical performance patterns. See [Event timeline](/docs/astro/assets-timeline).

### Lineage

The **Lineage** tab displays an interactive graph showing the upstream and downstream dependencies for the asset. This visualization helps you understand data flow and relationships in your pipelines. See [Asset lineage](/docs/astro/lineage-graph).

### Metrics

The **Metrics** tab displays performance metrics for the asset, including task retries, failures, and duration over time. These metrics help you identify trends and potential issues in your data pipelines.

### Data products

The **Data Products** tab shows which data products include this asset. This helps you understand the business context and impact of the asset across your organization. See [Data products](/docs/astro/create-data-products).


# Event timeline
Source: https://astronomer.io/docs/astro/assets-timeline

View asset event timeline in Astro Observe.

The event timeline provides a chronological view of all events associated with an asset in Astro Observe. Use the event timeline to track asset activity, troubleshoot issues, and understand historical performance patterns.

## View the event timeline

1. In the Astro UI, click **Observe** > **Catalog** (**Asset Catalog** in the legacy UI).
2. Select an asset from the list.
3. Click the **Event Timeline** tab.

The timeline displays a visual bar chart showing events over time, with a detailed event list showing timestamps, event types, and affected entities.

## Filter events

Use the filtering controls to narrow down the events displayed:

* **Run selection**: Choose **All Runs** or select a specific run to view its events.
* **Time period**: Select a time range and specify a date and time to view events before.
* **Event status**: Filter by **Success**, **Neutral**, or **Failure** events. Click a status indicator to toggle that event type on or off.
* **Only final asset events**: Toggle this option to show only events for final assets in your data product, excluding intermediate processing steps.

Click **+ filters** to access additional filtering options:

* **All Event Groups**: Filter events by event group.
* **Filter by namespace**: Filter events by namespace.
* **Filter by DAG**: Filter events by Dag.
* **Filter by Data Product**: Filter events by data product.
* **Filter by Deployment**: Filter events by Deployment.

## View event details

Click any event in the timeline to open a side panel with detailed information about that event, including:

* Event timestamp and metadata: Deployment name, DAG ID, Run ID, Owner
* **Root Cause**: For failure events, view root cause analysis or generate an AI log summary to diagnose the failure. See [Root cause analysis and AI log summaries](/docs/astro/root-cause-analysis).
* **Upstream Dependencies**: View any upstream failures or anomalies that may have contributed to the event
* **Downstream Dependencies**: See assets that depend on this asset, including Asset ID, Asset Name, and Asset Type

## Event types

Events are categorized by status to help you quickly identify issues and track successful operations.

### Success events

Success events indicate normal, expected operations:

* **SLA Success**: An SLA requirement was met
* **Task Success**: An Airflow task completed successfully
* **Task Start**: An Airflow task began execution

### Neutral events

Neutral events represent data operations that don't indicate success or failure:

* **Airflow dataset write**: Data was written to an Airflow dataset
* **OpenLineage dataset write**: Data was written and captured by OpenLineage

### Failure events

Failure events indicate problems that require attention:

* **Alert notification**: An alert was triggered
* **SLA Breach**: An SLA requirement wasn't met
* **Task failure**: An Airflow task failed

## Understand the timeline visualization

The bar chart at the top of the timeline shows event distribution over time:

* **Green bars**: Represent successful events
* **Red bars**: Represent failed events
* **Gray bars**: Represent neutral events

Taller bars indicate multiple events occurring in that time period. Use this visualization to quickly identify patterns, such as recurring failures or periods of high activity.


# Astro executor
Source: https://astronomer.io/docs/astro/astro-executor

Learn how the Astro executor distributes Airflow 3 tasks to agents for improved scaling and reliability.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

## Overview

The Astro executor is the default for all Airflow 3.x Deployments, and is the only executor you can use for Deployments in remote execution mode. The Astro executor consists of Agents that pull work from an API server. The API server manages the Agent lifecycle and controls task assignment logic, enabling more efficient workload distribution and scaling. This design differs from the Celery executor, where workers fetch tasks directly from a queue, and the Kubernetes executor, where the scheduler launches pods for each task.

By centrally managing both Agent scaling and task assignment, the Astro executor offers increased reliability, improved performance, and cost efficiency compared to other Airflow execution models.

## Remote Execution mode

In a remote execution mode Deployment, the Remote Execution Agents, data sources, code, secrets, and logs are housed in your environment while the API server is hosted in Astro's Orchestration Plane. To configure workers and scaling in Remote Execution mode, see [Remote Execution Agents](/docs/astro/remote-execution-configure-agents).

For more information on Remote versus Hosted execution mode, see [Execution mode](/docs/astro/execution-mode).

## Astro worker autoscaling logic in Hosted execution mode

If your Deployment is in [Hosted execution mode](/docs/astro/execution-mode), the number of Astro workers running per worker queue on your Deployment at a given time is based on two values:

* The total number of tasks in a `queued` or `running` state.
* The worker queue's setting for **Concurrency**.

The calculation is made based on the following expression:

`[Number of workers]= ([Queued tasks]+[Running tasks])/(Concurrency)`

[Kubernetes Event Driven Autoscaling](https://keda.sh/) (KEDA) computes these calculations every ten seconds. When KEDA determines that it can scale down a worker, it waits for five minutes after the last running task on the worker finishes before terminating the worker Pod.

When you push a new image to a Deployment, workers running tasks from before the code push don't shut down until those tasks are complete. To learn more about how changes to a Deployment can affect worker resource allocation, see [What happens during a code deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy).

## Configure the Astro executor in Hosted execution mode

In Hosted execution mode, you can configure Astro executor in the following ways with the Astro UI:

* The type and size of your workers.
* The minimum and maximum number of workers that your Deployment can run at a time.
* The number of tasks that each worker can run at a time.

You can set these configurations per [worker queue](/docs/astro/configure-worker-queues). With the Astro executor, you can configure multiple worker queues for different types of tasks and assign tasks to those queues in your Dag code.

The following document explains basic Astro executor configurations for a single worker queue. For instructions on how to configure multiple worker queues, see [Create a worker queue](/docs/astro/configure-worker-queues#create-a-worker-queue).

<Tip>If you plan to use only the `KubernetesPodOperator` in your Deployment, set your worker resources to the lowest possible amounts, because the worker is only required for launching your Pods. See [`KubernetesPodOperator`](/docs/astro/kubernetespodoperator) for more information.</Tip>

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Configure Astro worker scaling in Hosted execution mode

For each worker queue on your Deployment, you have to specify certain settings that affect worker scaling behavior. If you're new to Airflow, Astronomer recommends using the defaults in Astro for each of these settings.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**.

    2. Click the Deployment's **More actions** menu (⋯), then click **Edit Deployment**.

    3. In the **Execution** section, edit the worker queue and configure the following settings:

       * **Worker type**: Choose the amount of resources that each worker will have.
       * **Concurrency**: The maximum number of tasks that a single worker can run at a time. If the number of queued and running tasks exceeds this number, a new worker is added to run the remaining tasks. This value is equivalent to the Apache Airflow [worker concurrency](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#worker-concurrency) setting. It is 16 by default.
       * **Storage**: Choose the amount of ephemeral storage in GiB that each worker has. This storage volume is transient and allows for temporary storage and processing of data within the worker. The worker is assigned the minimum 10 GiB by default. The maximum quota is 100 GiB. Only ephemeral storage requests that are greater than the default minimum of 10 GiB are chargeable.
       * **Worker Count (Min-Max)**: The minimum and maximum number of workers that can run at a time. The number of running workers changes based on **Concurrency** and the current number of tasks in a queued or running state. By default, the minimum number of workers is 1 and the maximum is 10.

       <Info>The number of running workers might temporarily exceed the max when longer duration tasks delay scaled-down workers from shutting down.</Info>

    4. Click **Update Deployment**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
    2. Click the **Details** tab and then click **Edit** in the **Execution** section to edit a worker queue.
    3. Configure the following settings:

       * **Worker type**: Choose the amount of resources that each worker will have.
       * **Concurrency**: The maximum number of tasks that a single worker can run at a time. If the number of queued and running tasks exceeds this number, a new worker is added to run the remaining tasks. This value is equivalent to the Apache Airflow [worker concurrency](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#worker-concurrency) setting. It is 16 by default.
       * **Storage**: Choose the amount of ephemeral storage in GiB that each worker has. This storage volume is transient and allows for temporary storage and processing of data within the worker. The worker is assigned the minimum 10 GiB by default. The maximum quota is 100 GiB. Only ephemeral storage requests that are greater than the default minimum of 10 GiB are chargeable.
       * **Worker Count (Min-Max)**: The minimum and maximum number of workers that can run at a time. The number of running workers changes based on **Concurrency** and the current number of tasks in a queued or running state. By default, the minimum number of workers is 1 and the maximum is 10.

    <Info>The number of running workers might temporarily exceed the max when longer duration tasks delay scaled-down workers from shutting down.</Info>

    4. Click **Update Queue**.
  </Tab>
</Tabs>

## See also

* [Configure worker queues](/docs/astro/configure-worker-queues).


# Astro Observe overview
Source: https://astronomer.io/docs/astro/astro-observe

Gain insight into the health and performance of your data pipelines and lifecycle.

Astro Observe delivers pipeline-aware data observability purpose-built for Airflow, giving your team visibility into the health, quality, and performance of your business critical data products across the modern data stack.
Using Observe, you can monitor business-critical pipelines as data products, track SLAs, troubleshoot pipeline- or data-level failures, and understand health across your data ecosystem.

<Warning>If you use [Remote Execution](/docs/astro/execution-mode#remote-execution), you must [enable OpenLineage for Remote Execution Agents](/docs/astro/remote-execution-configure-openlineage) to use Astro Alerts or Observe.</Warning>

## Health

The **Data Pipelines Health Overview** shows a high-level view of operational and business health across your pipelines and data products.
The dashboard includes pipeline-level run metrics, like failed runs and retries, to identify top silent failures and problematic jobs. The dashboard also shows reliability metrics across the platform, like SLA evaluations and triggered alerts, to help you identify issues at a glance and understand the overall health of your data ecosystem.

<Frame>
  <img alt="At-a-glance view of the Snowflake cost and a summary of Task and dag success rates." />
</Frame>

## Data products

In Observe, a data product is a composition of assets that deliver a result with business relevance. Observe automatically infers upstream dependencies for the assets that define a data product, like an Airflow dag that populates a dashboard or a Snowflake table powering a recommendation engine.
Data products in Observe can include data assets across Airflow deployments. For example, a set of five dags across different deployments that all populate an executive dashboard with product analytics can be grouped together into a single data product in Observe.

<Info>
  For guidance on identifying data products in your organization, see [how to identify data products in your organization](/docs/learn/data-products#how-to-identify-data-products-in-your-organization).
</Info>

* Dependencies. A real-time lineage graph displays the relationships between the final assets in a data product and the upstream assets that feed into it.
* SLA evaluations. You can create custom business-level Service Level Agreements for evaluating the on-time delivery of a data product or its freshness. You can also view all SLAs in effect for the data product along with their current statuses at a glance.
* Metrics. You can see key metrics for assets across your data product, like task failures and retries.

<Frame>
  <img alt="Example dashboard in the Overview page for a specific data product." />
</Frame>

See [Create a Data Product](/docs/astro/create-data-products) to refer to steps to create your own data products. Or, to follow a quickstart, see [Get started with Astro Observe](/docs/learn/astro-observe-quickstart).

## Monitors

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

Monitors in Astro Observe allow you to automatically track the health of your data products and pipelines. When configured, monitors can trigger alerts for failures or critical events in your data products, enabling rapid detection and resolution of issues.

See [Monitors in Astro Observe](/docs/astro/observe-monitors) for set-up instructions and details.

## Asset Catalog

Assets include information about the attribute details and historical data regarding task runs. In the Astro UI, when you click **Catalog** (in the legacy UI, click **Observe** and then **Asset Catalog** in the Organization menu), you can view and filter the different data assets in your Workspace.

In this view you can see the following information about your Assets, including:

* Name
* Asset type
* Namespace
* Dag ID
* Owner

You can click on any asset to view more detailed information about it, such as historical metrics and metadata. To learn more about using assets and accessing metrics, see [Assets overview](/docs/astro/assets-overview).


# Export Astro audit logs
Source: https://astronomer.io/docs/astro/audit-logs

Export Astro audit logs to track administrative activities and meet compliance requirements.

Astro audit logs record administrative activities and events in your Organization. You can use the audit logs to determine who did what, where, and when. You can also use audit logs to ensure your Organization is meeting its security and regulatory requirements. For example, you can use audit logs to determine when users were added to your Organization or to individual Workspaces.

See [Reference: Astro audit log fields](#audit-logs-reference) for a complete list of available fields in audit logs.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Export audit logs

Audit logs are retained for 90 days. The [Organization Owner](/docs/astro/user-permissions#organization-roles) role is required to export audit logs.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Export…** next to **Audit Logs**.
    2. Select the number of days of audit data to export, then click **Export Logs**.
  </Tab>

  <Tab title="Legacy UI">
    1. Click **Organization Settings** in the Astro UI.
    2. In the **Audit Logs** menu, select the number days of audit data to export and then click **Export**.
  </Tab>
</Tabs>

The extracted audit log data is saved as a [newline delimited JSON (ndjson)](https://github.com/ndjson/ndjson-spec) file with the default filename `<astro-organization-name>-logs-<number-of-days>-days-<date>.ndjson`.

<Info>
  **CLI**

  You can also export logs using the Astro CLI. Run the following command to export audit logs as a GZIP file to your current directory:

  ```bash wrap theme={null}
  astro organization audit-logs export --organization-name="<your-organization-name>"
  ```
</Info>

## Audit logs reference

This section is a reference for all available fields in [Astro audit logs](/docs/astro/audit-logs).

The following categories of event data are available in an audit log:

* **API events**: The data generated by user actions in the Astro UI, Astro CLI, or from internal processing actions in the Astro control plane.
* **Airflow UI access**: The data generated when users access the Airflow UI.
* **Astronomer container registry access**: The data generated when users access the Astronomer container registry with the Astro CLI.

The audit log file is provided as a newline delimited JSON (NDJSON) file. Every entry in the audit log file corresponds to an event, and event attributes provide additional information about each event.

### Common fields

The following table lists the common fields shared by all categories of event data.

#### Table: Common fields in audit logs

| Field            | Description                                                                                                           |
| ---------------- | --------------------------------------------------------------------------------------------------------------------- |
| `subjectId`      | A unique identifier that identifies the initiator of the request. The value can be user ID.                           |
| `subjectType`    | A unique identifier that identifies the subject of the request. The value can be `USER` or `SERVICEKEY`.              |
| `organizationId` | A unique identifier that identifies your organization. This is the value displayed in the Astro UI **Settings** page. |
| `timestamp`      | The date and time the event occurred.                                                                                 |
| `sourceIp`       | The IP address of the originating request.                                                                            |
| `userAgent`      | The application used to make the request.                                                                             |

<Info>To link the user ID back to a user, [Organization Owners](/docs/astro/user-permissions) can use the Astro Access Management UI by clicking **Access Management** in their **Organization Settings**.</Info>

### API event fields

Audit log events can be generated from the v1 API or the v2 API. Each API generates different fields for the same actions, and your audit log might include events from both APIs. Audit log event frequency for the v1 API are expected to decline as Astronomer transitions to the v2 API.

#### Table: v1 API event fields

| Field               | Description                                                                      |
| ------------------- | -------------------------------------------------------------------------------- |
| `correlationId`     | A unique identifier for the request.                                             |
| `graphqlClientName` | The type of client making the request. The values are `cloud-ui` or `cli`.       |
| `operationName`     | The name of the API event. For example, `createUserInvite` or `workspaceCreate`. |
| `requestBody`       | Raw graphQL for the event.                                                       |
| `requestInput`      | The input for the API request.                                                   |

The following table maps some common `operationName` attributes to their corresponding `requestInput` attributes.

#### Table: `operationName` attribute mappings

| Event                                 | `operationName` attribute   | `requestInput` attributes |
| ------------------------------------- | --------------------------- | ------------------------- |
| A new Deployment is created.          | `createDeployment`          | `label`, `workspaceId`    |
| A Deployment is updated.              | `updateDeployment`          | `deploymentSpec`          |
| A Deployment variable is updated.     | `updateDeploymentVariables` | `isSecret`, `key`         |
| The code for a Deployment is updated. | `ImageCreate`               | `deploymentId`            |

#### Table: v2 API event fields

| Field             | Description                                     |
| ----------------- | ----------------------------------------------- |
| `method`          | The HTTP request type sent to REST API.         |
| `path`            | The path to the invoked REST API.               |
| `requestBody`     | The parameters passed as input to the API call. |
| `requestId`       | A unique identifier for the request.            |
| `response status` | The HTTP response status code.                  |

The following table maps some common `path` attributes to their corresponding `requestBody` attributes.

#### Table: `path` attribute mappings

| Event                                                                   | `path` attribute                                                                                                                                      | `requestBody` attributes                                                                                                                                                                                                      |
| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| An Organization is created.                                             | `/v1alpha1/organizations`                                                                                                                             | `metadata`, `name`                                                                                                                                                                                                            |
| An Organization is updated.                                             | `/v1alpha1/organizations/{orgShortNameId}`                                                                                                            | `billingEmail`, `isScimEnabled`, `metadata`, `name`                                                                                                                                                                           |
| A managed domain for an Organization is created.                        | `/v1alpha1/organizations/{orgShortNameId}/domains`                                                                                                    | `enforcedLogins` `name`                                                                                                                                                                                                       |
| A managed domain for an Organization is updated.                        | `/v1alpha1/organizations/{orgShortNameId}/domains/{domainId}`                                                                                         | `enforcedLogins`                                                                                                                                                                                                              |
| A managed domain for an Organization is deleted.                        | `/v1alpha1/organizations/{orgShortNameId}/domains/{domainId}`                                                                                         |                                                                                                                                                                                                                               |
| A managed domain for an Organization is verified.                       | `/v1alpha1/organizations/{orgShortNameId}/domains/{domainId}/verify`                                                                                  |                                                                                                                                                                                                                               |
| An SSO Bypass Key for an Organization is created.                       | `/v1alpha1/organizations/{orgShortNameId}/sso-bypass-key`                                                                                             |                                                                                                                                                                                                                               |
| An SSO Bypass Key for an Organization is deleted.                       | `/v1alpha1/organizations/{orgShortNameId}/sso-bypass-key`                                                                                             |                                                                                                                                                                                                                               |
| An SSO Connection for an Organization is created.                       | `/v1alpha1/organizations/{orgShortNameId}/sso-connections`                                                                                            |                                                                                                                                                                                                                               |
| An SSO Connection for an Organization is updated.                       | `/v1alpha1/organizations/{orgShortNameId}/sso-connections/{connectionId}`                                                                             |                                                                                                                                                                                                                               |
| An SSO Connection for an Organization is deleted.                       | `/v1alpha1/organizations/{orgShortNameId}/sso-connections/{connectionId}`                                                                             |                                                                                                                                                                                                                               |
| A new user is invited to an Organization.                               | `/v1alpha1/organizations/{orgShortNameId}/invites`                                                                                                    | `inviteEmail`, `role`                                                                                                                                                                                                         |
| A invite to the Organization is updated.                                | `/v1alpha1/users/self/invites/{inviteId}`                                                                                                             | `inviteStatus`                                                                                                                                                                                                                |
| A invite to the Organization is deleted.                                | `/v1alpha1/organizations/{orgShortNameId}/invites/{inviteId}`                                                                                         |                                                                                                                                                                                                                               |
| A user is deleted from an Organization.                                 | `/v1alpha1/organizations/{orgShortNameId}/users/{userId}`                                                                                             |                                                                                                                                                                                                                               |
| A user is assigned a new Organization role.                             | `/v1alpha1/organizations/{orgShortNameId}/users/{userId}/role`                                                                                        | `role`                                                                                                                                                                                                                        |
| A user's Organization, Workspace, Deployment, or DAG roles are updated. | `/v1alpha1/organizations/{orgShortNameId}/users/{userId}/roles`                                                                                       | `dagRoles`, `deploymentRoles`, `organizationRole`, `workspaceRoles`                                                                                                                                                           |
| A user is assigned a new workspace role.                                | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/users/{userId}/role`                                                               | `role`                                                                                                                                                                                                                        |
| A user is removed from a workspace.                                     | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/users/{userId}`                                                                    |                                                                                                                                                                                                                               |
| A Workspace is created.                                                 | `/v1alpha1/organizations/{orgShortNameId}/workspaces`                                                                                                 | `apiKeyOnlyDeploymentsDefault`, `description`, `name`                                                                                                                                                                         |
| A Workspace is updated.                                                 | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}`                                                                                   | `apiKeyOnlyDeploymentsDefault`, `description`, `name`                                                                                                                                                                         |
| A Workspace is deleted.                                                 | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}`                                                                                   |                                                                                                                                                                                                                               |
| A deployment is transferred to another workspace.                       | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/deployments/{deploymentId}`                                                        | `workspaceIdTarget`                                                                                                                                                                                                           |
| An AWS cluster is created.                                              | `/v1alpha1/organizations/{orgShortNameId}/clusters/aws`                                                                                               | `dbInstanceType`, `k8sTags`, `name`, `nodePools`, `templateVersion`                                                                                                                                                           |
| An AWS cluster is updated.                                              | `/v1alpha1/organizations/{orgShortNameId}/clusters/aws/{clusterId}`                                                                                   | `dbInstanceType`, `k8sTags`, `name`, `nodePools`, `templateVersion`                                                                                                                                                           |
| An Azure cluster is created.                                            | `/v1alpha1/organizations/{orgShortNameId}/clusters/azure`                                                                                             | `dbInstanceType`, `k8sTags`, `name`, `nodePools`, `providerAccount`, `region`, `templateVersion`, `tenantId`, `type`, `vpcSubnetRange`                                                                                        |
| An Azure cluster is updated.                                            | `/v1alpha1/organizations/{orgShortNameId}/clusters/azure/{clusterId}`                                                                                 | `dbInstanceType`, `k8sTags`, `name`, `nodePools`, `templateVersion`,                                                                                                                                                          |
| A GCP cluster is created.                                               | `/v1alpha1/organizations/{orgShortNameId}/clusters/gcp`                                                                                               | `dbInstanceType`, `k8sTags`, `name`, `nodePools`, `podSubnetRange` `providerAccount`, `region`, `servicePeeringRange`, `serviceSubnetRange`, `templateVersion`, `type`, `vpcSubnetRange`                                      |
| A GCP cluster is updated.                                               | `/v1alpha1/organizations/{orgShortNameId}/clusters/gcp/{clusterId}`                                                                                   | `dbInstanceType`, `k8sTags`, `name`, `nodePools`, `templateVersion`                                                                                                                                                           |
| A cluster is deleted.                                                   | `/v1alpha1/organizations/{orgShortNameId}/clusters/{clusterId}`                                                                                       |                                                                                                                                                                                                                               |
| An Organization API token is created.                                   | `/v1alpha1/organizations/{orgShortNameId}/api-tokens`                                                                                                 | `description`, `name`, `role`, `tokenExpiryPeriodInDays`                                                                                                                                                                      |
| An Organization API token is updated.                                   | `/v1alpha1/organizations/{orgShortNameId}/api-tokens/{apiTokenId}`                                                                                    | `description`, `name`, `roles`                                                                                                                                                                                                |
| An Organization API token is deleted.                                   | `/v1alpha1/organizations/{orgShortNameId}/api-tokens/{apiTokenId}`                                                                                    |                                                                                                                                                                                                                               |
| An Organization API token is rotated.                                   | `/v1alpha1/organizations/{orgShortNameId}/api-tokens/{apiTokenId}/rotate`                                                                             |                                                                                                                                                                                                                               |
| A Workspace API token is created.                                       | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/api-tokens`                                                                        | `description`, `name`, `role`, `tokenExpiryPeriodInDays`                                                                                                                                                                      |
| A Workspace API token is updated.                                       | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/api-tokens/{apiTokenId}`                                                           | `description`, `name`, `role`                                                                                                                                                                                                 |
| A Workspace API token is deleted.                                       | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/api-tokens/{apiTokenId}`                                                           |                                                                                                                                                                                                                               |
| A Workspace API token is rotated.                                       | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/api-tokens/{apiTokenId}/rotate`                                                    |                                                                                                                                                                                                                               |
| A Team is created.                                                      | `/v1alpha1/organizations/{orgShortNameId}/teams`                                                                                                      | `description`, `name`, `memberIds`                                                                                                                                                                                            |
| A Team is updated.                                                      | `/v1alpha1/organizations/{orgShortNameId}/teams/{teamId}`                                                                                             | `description`, `name`                                                                                                                                                                                                         |
| A Team is deleted.                                                      | `/v1alpha1/organizations/{orgShortNameId}/teams/{teamId}`                                                                                             |                                                                                                                                                                                                                               |
| A Team is created via SCIM.                                             | `/scim/v2/{orgShortNameId}/Groups`                                                                                                                    | `displayName`, `members`, `value`, `display`                                                                                                                                                                                  |
| A Team is updated via SCIM.                                             | `/scim/v2/{orgShortNameId}/Groups/{teamId}`                                                                                                           | `displayName`, `members`, `value`, `display`                                                                                                                                                                                  |
| A Team is deleted via SCIM.                                             | `/scim/v2/{orgShortNameId}/Groups/{teamId}`                                                                                                           |                                                                                                                                                                                                                               |
| A User is created via SCIM.                                             | `/scim/v2/{orgShortNameId}/Users`                                                                                                                     | `userName`, `name`, `emails`, `displayName`                                                                                                                                                                                   |
| A User is updated via SCIM.                                             | `/scim/v2/{orgShortNameId}/Users/{userId}`                                                                                                            | `userName`, `name`, `emails`, `displayName`                                                                                                                                                                                   |
| A User is deleted via SCIM.                                             | `/scim/v2/{orgShortNameId}/Users/{userId}`                                                                                                            |                                                                                                                                                                                                                               |
| A user is added to a Team.                                              | `/v1alpha1/organizations/{orgShortNameId}/teams/{teamId}/members`                                                                                     | `memberIds`                                                                                                                                                                                                                   |
| A user is removed from a Team                                           | `/v1alpha1/organizations/{orgShortNameId}/teams/{teamId}/members`                                                                                     |                                                                                                                                                                                                                               |
| A Team is removed from a Workspace.                                     | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/teams/{teamId}`                                                                    |                                                                                                                                                                                                                               |
| A Team's role in a Workspace is updated.                                | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/teams/{teamId}/role`                                                               | `role`                                                                                                                                                                                                                        |
| A dag is updated.                                                       | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/runtimes/{runtimeId}/pipelines/{pipelineName}`                                     | `isPaused`                                                                                                                                                                                                                    |
| A dag run is created.                                                   | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/runtimes/{runtimeId}/pipelines/{pipelineName}/runs`                                | `logicalDate`                                                                                                                                                                                                                 |
| A dag run has its state updated.                                        | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/runtimes/{runtimeId}/pipelines/{pipelineName}/runs/{pipelineRunId}`                | `state`                                                                                                                                                                                                                       |
| A dag run is cleared.                                                   | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/runtimes/{runtimeId}/pipelines/{pipelineName}/runs/{pipelineRunId}/clear`          | `isDryRun`                                                                                                                                                                                                                    |
| A task instance for a dag run is cleared.                               | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/runtimes/{runtimeId}/pipelines/{pipelineName}/clear-task-instances`                | `endDate`, `includeDownstream`, `includeFuture`, `includeParentPipelines`, `includePast`, `includeSubPipelines`, `includeUpstream`, `onlyFailed`, `onlyRunning`, `pipelineRunId`, `resetPipelineRuns`, `startDate`, `taskIds` |
| A task instance's state is set.                                         | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/runtimes/{runtimeId}/pipelines/{pipelineName}/runs/{pipelineRunId}/tasks/{taskId}` | `state`                                                                                                                                                                                                                       |
| A task instance's state is updated.                                     | `/v1alpha1/organizations/{orgShortNameId}/workspaces/{workspaceId}/runtimes/{runtimeId}/pipelines/{pipelineName}/update-task-instances-state`         | `executionDate`, `includeDownstream`, `includeFuture`, `includePast`, `includeUpstream`, `isDryRun`, `pipelineRunId`, `state`, `taskId`                                                                                       |

Use your analytics or audit tool to view additional attribute mapping information.

### Airflow UI access event fields

The following table lists the fields that are unique to Airflow UI access events.

#### Table: Airflow UI access event fields

| Field       | Description                                                                                                                                                        |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `method`    | The HTTP method for the request.                                                                                                                                   |
| `path`      | The relative path of the page being accessed. For example, `/dzl9chy4/configuration`.                                                                              |
| `targetUrl` | The URL for the canonical name record in the Airflow webserver that runs in the Astro data plane. For example, `https://cl8gwrnw601f10tyxhgrhaayw.astronomer.run`. |

The following table lists the fields that are unique to Astronomer container registry access events.

#### Table: Astronomer container registry access event fields

| Field          | Description                                                                     |
| -------------- | ------------------------------------------------------------------------------- |
| `deploymentId` | A unique identifier that identifies the Deployment on which the event occurred. |
| `method`       | The HTTP method for the request.                                                |
| `path`         | The path to the image in the registry.                                          |

## Astro CLI access event fields

The following table lists the fields that are unique to Astro CLI access events.

#### Table: Astro CLI access event fields

| Field                | Description                                                                                                                              |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `astroClient`        | A unique identifier for the API client. `cli` means the request came from the Astro CLI.                                                 |
| `astroClientVersion` | A unique identifier for the API client version.                                                                                          |
| `userAgent`          | A unique identifier for the API client type and version. `astro-cli/1.15.1` means the request came from version 1.15.1 of the Astro CLI. |


# Authorize an Astro Deployment to cloud resources using workload identity
Source: https://astronomer.io/docs/astro/authorize-deployments-to-your-cloud

Give Astro Deployments access to your cloud resources using a Kubernetes workload identity

When you create an Airflow connection from a Deployment to access cloud resources, Airflow uses your connection details to access those services. You can add credentials to your Airflow connections to authenticate, but it can be risky to add secrets like passwords to your Airflow environment.

To avoid adding secrets to your Airflow connection, you can directly authorize your Astro Deployment to access AWS, GCP, and Azure cloud services using workload identity. Astronomer recommends using a workload identity in most cases to improve security and avoid managing credentials across your Deployments. If you have less strict security requirements, you can still use any of the methods described in [Airflow connection guides](/docs/learn/connections) to manage your connection authorization.
This guide explains how to authorize your Deployment to a cloud using workload identity. For each Deployment, you will:

* Authorize your Deployment to your cloud services.
* Create an Airflow connection to access your cloud services.

<Tip>Watch the Astro Academy [Customer Workload Managed Identity](https://academy.astronomer.io/learning-bytes-customer-workload-managed-identity) Learning Byte video to learn more about managed identities and how to set up passwordless authentication for GCP.</Tip>

## Prerequisites

The Astro cluster running your Deployment must be connected to your cloud's network. See [Networking overview](/docs/astro/networking-overview).

## What is workload identity?

A workload identity is a Kubernetes service account that provides an identity to your Deployment. The Deployment can use this identity to authenticate to a cloud's API server, and the cloud can use this identity to authorize the Deployment to access different resources.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Setup

<Tabs>
  <Tab title="AWS">
    <Tip>
      You can also use either the Astro CLI or the Astro API to configure workload identity when you create or update a Deployment by providing the AWS ARN value.

      See the following pages for more detail:

      * [`astro deployment create`](/docs/cli/v1.43/astro-deployment-create#options-astro) CLI reference
      * [`astro deployment update`](/docs/cli/v1.43/astro-deployment-update#options-astro) CLI reference
      * [Create Deployment with the Astro API](/docs/astro/api/v-1/deployment/create-a-deployment)
      * [Update Deployment with the Astro API](/docs/astro/api/v-1/deployment/update-a-deployment)
    </Tip>

    ### Attach an IAM role to your Deployment

    You can attach an AWS IAM role to your Deployment to grant the Deployment all of the role's permissions.

    Using IAM roles provides the greatest amount of flexibility for authorizing Deployments to your cloud. For example, you can use existing IAM roles on new Deployments, or your can attach a single IAM role to multiple Deployments that all require the same level of access to your cloud.

    #### Prerequisites (AWS)

    * Minimum Astro Runtime version:
      * 9.15.0
      * 10.9.0
      * 11.5.0
    * A new or existing IAM role in your data sources with the required permissions you want your Deployment to have.
    * If using [AWS CloudShell](https://aws.amazon.com/cloudshell/),
      the required CLIs are enabled by default.
    * If you use a local terminal, the following CLIs are required:
      * [AWS CLI](https://aws.amazon.com/cli/)
      * [jq](https://jqlang.github.io/jq/)
      * [openSSL](https://www.openssl.org/source/)

    #### Step 1: Authorize the Deployment to your IAM role (AWS)

    To authorize your Deployment, create an IAM role to assign as your Deployment's workload identity:

    1. [Create an IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html) to delegate permissions to in an AWS service. Grant the role any permission that the Deployment will need in your AWS account. Copy the IAM role ARN to use later in this setup.
    2. In the Astro UI, select your Deployment and then click **Details**. In the **Advanced** section, click **Edit**.
    3. In the **Workload Identity** menu, select **Customer Managed Identity**.
    4. Enter your IAM role ARN when prompted, then copy and run the provided CLI command. Click **Save Configuration** to save the IAM role as a selectable configuration.

       <Info>
         **About the AWS CLI command**

         The command performs the following actions in your AWS account:

         * Creates an [IAM OIDC identity provider](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html) for the Astro cluster's EKS OIDC issuer URL, if one doesn't already exist. You can view existing providers in the AWS console at **IAM** > **Identity providers**.
         * Updates the trust policy of the IAM role you specified to allow the Astro cluster's service accounts to assume the role through `sts:AssumeRoleWithWebIdentity`. You can view the updated trust policy in the AWS console at **IAM** > **Roles** > *your role* > **Trust relationships**.

         The IAM principal that runs the command needs permissions to create OIDC providers and update IAM role trust policies, such as `iam:CreateOpenIDConnectProvider`, `iam:GetRole`, and `iam:UpdateAssumeRolePolicy`. If the command fails, verify these permissions and check the resources listed previously in the AWS console.
       </Info>
    5. Click **Update Deployment** to apply the selected IAM role to the Deployment.
    6. (Optional) Repeat these steps for each Astro Deployment that needs to access your AWS resources. Or, you can edit the `<DeploymentNamespace>` value in `Condition` when setting up the Workload Identity for one of the following scenarios to apply to multiple Deployments.

    <AccordionGroup>
      <Accordion title="Specify Kubernetes service accounts">
        Available for both Standard and Dedicated clusters. If your organization doesn't allow you to use a wildcards in your IAM Trust Policies, change the `<DeploymentNamespace>` value in `Condition` to specify the Kubernetes service accounts. The following shows an example:

        ```json wrap theme={null}
        {
          "Condition": {
            "StringLike": {
              "<cluster-oidc-issuer-url>:aud": "sts.amazonaws.com",
              "<cluster-oidc-issuer-url>:sub": [
                "system:serviceaccount:<deployment-namespace>:<deployment-namespace>-kpo",
                "system:serviceaccount:<deployment-namespace>:<deployment-namespace>-dag-processor-serviceaccount",
                "system:serviceaccount:<deployment-namespace>:<deployment-namespace>-scheduler-serviceaccount",
                "system:serviceaccount:<deployment-namespace>:<deployment-namespace>-triggerer-serviceaccount",
                "system:serviceaccount:<deployment-namespace>:<deployment-namespace>-apiserver-serviceaccount",
                "system:serviceaccount:<deployment-namespace>:<deployment-namespace>-worker-serviceaccount"
              ]
            }
          }
        }
        ```

        <Note>For Airflow 2 Deployments, `apiserver-serviceaccount` is named `webserver-serviceaccount`.</Note>
      </Accordion>

      <Accordion title="Dedicated clusters only: Share or re-use a managed identity using a wildcard">
        If you want to share or re-use the same customer managed identity on static or ephemeral Deployments for dedicated clusters, without having to update your Trust Policy in your AWS account for every net new Deployment, change the `<DeploymentNamespace>` value in `Condition` to include a wildcard. You should only use a wildcard in dedicated clusters for security purposes. The following shows an example:

        ```json wrap theme={null}
        {
          "Condition": {
            "StringLike": {
              "<cluster-oidc-issuer-url>:aud": "sts.amazonaws.com",
              "<cluster-oidc-issuer-url>:sub": "system:serviceaccount:*:*"
            }
          }
        }
        ```
      </Accordion>
    </AccordionGroup>

    #### Step 2: Create an Airflow connection (AWS, IAM role)

    Now that your Deployment is authorized, you can connect it to your cloud using an Airflow connection. Create an **Amazon Web Services** connection in either the [Astro UI](/docs/astro/create-and-link-connections) or the Airflow UI for your Deployment and specify the following fields:

    * **Connection Id**: Enter a name for the connection.

    If you don't see **Amazon Web Services** as a connection type in the Airflow UI, ensure you have installed its provider package in your Astro project's `requirements.txt` file. See **Use Provider** in the [Airflow Registry](https://airflow.apache.org/registry/providers/amazon) for the latest package.

    <Tip>
      If you use a mix of strategies for managing connections and define the same connection in multiple ways, Airflow uses the following order of precedence:

      * Secrets Backend
      * Environment Manager
      * Environment Variables
      * Airflow UI using the Airflow metadata database
    </Tip>

    ### Alternative setup: Authorize your Deployment with AWS IAM roles

    #### Step 1: Authorize the Deployment in your cloud (AWS, alternative setup)

    To grant a Deployment access to a service that is running in an AWS account not managed by Astronomer, use AWS IAM roles to authorize your Deployment's workload identity. IAM roles on AWS are often used to manage the level of access a specific user, object, or group of users has to a resource, such as Amazon S3 buckets, Redshift instances, and secrets backends.

    To authorize your Deployment, create an IAM role that is assumed by the Deployment's workload identity:

    1. In the Astro UI, select your Deployment and then click **Details**. Copy the Deployment's **Workload Identity**.

    2. In the AWS account that contains your AWS service, create an IAM role. See [Creating a role to delegate permissions to an AWS service](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-service.html).

    3. In the AWS Management Console, go to the Identity and Access Management (IAM) dashboard.

    4. Click **Roles** and in the **Role name** column, select the role you created in Step 2.

    5. Click **Trust relationships**.

    6. Click **Edit trust policy** and paste the workload identity you copied from Step 1 in the trust policy. Your policy should look like the following:

       ```json wrap theme={null}
       {
         "Version": "2012-10-17",
         "Statement": [
           {
             "Effect": "Allow",
             "Principal": {
               "AWS": ["<default-workload-identity-role>"]
             },
             "Action": "sts:AssumeRole"
           }
         ]
       }
       ```

    7. Click **Update policy**.

    Repeat these steps for each Astro Deployment that needs to access your AWS resources.

    #### Step 2: Create an Airflow connection (AWS, alternative setup)

    Now that your Deployment is authorized, you can connect it to your cloud using an Airflow connection. Either create an **Amazon Web Services** connection in the [Astro UI](/docs/astro/create-and-link-connections) or the Airflow UI for your Deployment and specify the following fields:

    * **Connection Id**: Enter a name for the connection.
    * **Extra**:

      ```json wrap theme={null}
      {
        "role_arn": "<your-role-arn>",
        "region_name": "<your-region>"
      }
      ```

    If you don't see **Amazon Web Services** as a connection type in the Airflow UI, ensure you have installed its provider package in your Astro project's `requirements.txt` file. See **Use Provider** in the [Airflow Registry](https://airflow.apache.org/registry/providers/amazon) for the latest package.

    <Tip>
      If you use a mix of strategies for managing connections, if you define the same connection in multiple ways, Airflow uses the following order of precedence:

      * Secrets Backend
      * Environment Manager
      * Environment Variables
      * Airflow UI using the Airflow metadata database
    </Tip>
  </Tab>

  <Tab title="GCP">
    <Tip>
      You can also use either the Astro CLI or the Astro API to configure workload identity when you create or update a Deployment by providing the GCP service account email.

      See the following pages for more detail:

      * [`astro deployment create`](/docs/cli/v1.43/astro-deployment-create#options-astro) CLI reference
      * [`astro deployment update`](/docs/cli/v1.43/astro-deployment-update#options-astro) CLI reference
      * [Create Deployment with the Astro API](/docs/astro/api/v-1/deployment/create-a-deployment)
      * [Update Deployment with the Astro API](/docs/astro/api/v-1/deployment/update-a-deployment)
    </Tip>

    ### Attach a service account to your Deployment

    You can attach a custom GCP service account to your Deployment to grant the Deployment all of the service account's permissions.

    Using service accounts provides the greatest amount of flexibility for authorizing Deployments to your cloud. For example, you can use existing service accounts on new Deployments, or your can attach a single service account to multiple Deployments that all have the same level of access to your cloud.

    1. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create) in the GCP project that you want your Deployment to access. Grant the service account any permissions that the Deployment will need in your GCP project. Copy the service account ID to use later in this setup.

    2. In the Astro UI, select your Deployment, then click **Details**. In the **Advanced** section, click **Edit**.

    3. In the **Workload Identity** menu, select **Customer Managed Identity**

    4. Enter your GCP service account ID when prompted, then copy and run the provided gcloud CLI command.

       <Info>
         **About the gcloud CLI command**

         The command adds an IAM policy binding to your GCP service account that grants the `roles/iam.workloadIdentityUser` role to each of the Deployment's Kubernetes service accounts (`scheduler`, `triggerer`, `worker`, `apiserver`, `dag-processor`, and `kpo`). This allows those workloads to impersonate your GCP service account through workload identity federation.

         You can view the updated bindings in the Google Cloud console at **IAM & Admin** > **Service Accounts** > *your service account* > **Permissions**.

         The principal that runs the command needs the `iam.serviceAccountAdmin` role, or equivalent permissions including `iam.serviceAccounts.setIamPolicy`, on the GCP service account. If the command fails, verify these permissions and check the service account's IAM policy in the Google Cloud console.
       </Info>

    5. Click **Update Deployment**. The service account is now selectable as a workload identity for the Deployment.

    6. Complete one of the following options for your Deployment to access your cloud resources:

       * Create a **Google Cloud** connection type in Airflow and configure the following values:

         * **Connection Id**: Enter a name for the connection.
         * **Impersonation Chain**: Enter the ID of the service account that your Deployment should impersonate.

       * To access resources in a secrets backend, run the following command to create an environment variable that grants access to the secrets backend:

       ```bash wrap theme={null}
       astro deployment variable create \
         --deployment-id <your-deployment-id> \
         'AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "project_id": "<your-secret-manager-project-id>", "impersonation_chain": "<your-gcp-service-account>"}'
       ```

    #### Dedicated clusters only: Share or reuse a managed identity using a wildcard

    <Warning>You can only use wildcard `principalSet` bindings with hosted dedicated clusters. Never use wildcard bindings with standard Deployments, which run on shared clusters.</Warning>

    If you want to share or reuse the same customer managed identity across multiple Deployments, for example with ephemeral Deployments, on a dedicated cluster without creating a per-Deployment IAM policy binding, you can use a wildcard `principalSet` binding. This authorizes all service accounts across all namespaces in the cluster to use your GCP service account, so you can specify the same custom workload identity on any Deployment in the cluster, including at Deployment creation time.

    You can find the GKE project number and project ID for your cluster. Go to **Settings** > **Clusters** in the Astro UI and click the **Details** tab. You can also use a [GET cluster API call](https://www.astronomer.io/docs/astro/api/v-1/cluster/get-a-cluster) with your cluster ID.

    ```sh wrap theme={null}
    gcloud iam service-accounts add-iam-policy-binding <service-account> \
      --role roles/iam.workloadIdentityUser \
      --member "principalSet://iam.googleapis.com/projects/<project-number>/locations/global/workloadIdentityPools/<project-id>.svc.id.goog/kubernetes.cluster/https://container.googleapis.com/v1/projects/<project-id>/locations/<location>/clusters/<cluster-id>" \
      --project <service-account-project>
    ```

    Replace the following values:

    * `<LOCATION>`: The cluster's region.
    * `<CLUSTER_ID>`: ID of the cluster.
    * `<SERVICE_ACCOUNT>`: ID of the serviceAccount or fully qualified identifier for the serviceAccount you want your Deployments to use.
    * `<PROJECT_NUMBER>`: The GCP project number for the Astro-managed GKE cluster.
    * `<PROJECT_ID>`: The GCP project ID for the Astro-managed GKE cluster.
    * `<SERVICE_ACCOUNT_PROJECT>`: The GCP project that contains your service account.

    ### Alternative setup: Authorize your Deployment through GCP service account impersonation

    If your organization has requirements over how service accounts are managed outside of your cloud, you can manually configure [GCP service account impersonation](https://cloud.google.com/docs/authentication/use-service-account-impersonation) to allow your Deployment's default workload identity to impersonate a service account in your GCP project.

    1. [Create a service account](https://cloud.google.com/iam/docs/service-accounts-create) in the GCP project that you want your Deployment to access. Grant the service account any permissions that the Deployment will need in your GCP project. Copy the service account ID to use later in this setup.
    2. In the Astro UI, select your Deployment, then click **Details**. Copy the Deployment's **Workload Identity**.
    3. In the Google Cloud Console, open the **IAM & Admin > Service Accounts** menu, then open the service account you just created.
    4. In the **Actions** column, click **Manage Permissions**, then click **Grant Access**. In the modal that appears, enter your Deployment's workload identity service account in the **Add Principals** field and select the [`Service Account Token Creator`](https://cloud.google.com/iam/docs/understanding-roles#iam.serviceAccountTokenCreator) in the **Assign Roles** field.
    5. Complete one of the following options for your Deployment to access your cloud resources:

       * Create a **Google Cloud** connection type in Airflow and configure the following values:
         * **Connection Id**: Enter a name for the connection.
         * **Impersonation Chain**: Enter the ID of the service account that your Deployment should impersonate.

       Note that this implementation requires `apache-airflow-providers-google >= 10.8.0`. See [Add Python, OS-level packages, and Airflow providers](/docs/cli/v1.43/add-providers-packages).

       * Specify the impersonation chain in code when you instantiate a Google Cloud operator. See [Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html#direct-impersonation-of-a-service-account). Note that if you configure both a connection type and an operator, the operator-level configuration takes precedence.
       * To access resources in a secrets backend, run the following command to create an environment variable that grants access to the secrets backend:

       ```bash wrap theme={null}
       astro deployment variable create \
         --deployment-id <your-deployment-id> \
         'AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "project_id": "<your-secret-manager-project-id>", "impersonation_chain": "<your-gcp-service-account>"}'
       ```

    ### Alternative setup: Grant an IAM role to your Deployment workload identity

    Complete this alternative setup if you don't have an existing Google service account that your Deployment workload identity can impersonate.

    #### Step 1: Authorize the Deployment in your cloud (GCP)

    To grant a Deployment access to a service that is running in a GCP account not managed by Astronomer, use your Deployment's workload identity. Workload identity is a service account in GCP that's used to manage the level of access for a specific user, object, or group of users to a resource, such as Google BigQuery or a GCS bucket.

    To authorize your Deployment, grant the required access to your Deployment's workload identity:

    1. In the Astro UI, select your Deployment, then click **Details**. In the **Workload Identity** dropdown menu, select **Default Identity**. Then, copy the workload identity that appears next to the dropdown menu.

    2. Grant your Deployment's workload identity an IAM role that has access to your external data service. To do this with the Google Cloud CLI, run:

       ```bash wrap theme={null}
       gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
         --member=serviceAccount:<workload-identity> \
         --role=<your-role>
       ```

       To grant your workload identity an IAM role using the Google Cloud console, see [Grant an IAM role](https://cloud.google.com/iam/docs/grant-role-console#grant_an_iam_role).

    Repeat these steps for each Deployment that needs to access your GCP resources.

    #### Step 2: Create an Airflow connection (GCP)

    Now that your Deployment is authorized, you can connect it to your cloud using an Airflow connection. Either create a **Google Cloud** connection in the [Astro UI](/docs/astro/create-and-link-connections) or the Airflow UI for your Deployment and specify the following fields:

    * **Connection Id**: Enter a name for the connection.
    * **Project Id**: Enter the ID of your Google Cloud Project where your services are running.

    If you don't see **Google Cloud** as a connection type in the Airflow UI, ensure you have installed its provider package in your Astro project's `requirements.txt` file. See **Use Provider** in the [Airflow Registry](https://airflow.apache.org/registry/providers/google/) for the latest package.

    <Tip>
      If you use a mix of strategies for managing connections, if you define the same connection in multiple ways, Airflow uses the following order of precedence:

      * Secrets Backend
      * Environment Manager
      * Environment Variables
      * Airflow UI using the Airflow metadata database
    </Tip>
  </Tab>

  <Tab title="Azure">
    In this setup, you'll authorize an existing user-assigned managed identity to a resource on Azure, then give permissions to your Deployment to assume that managed identity.

    #### Prerequisites (Azure)

    * A [Microsoft Entra ID tenant](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-create-new-tenant) with Global Administrator or Application Administrator privileges.
    * A user-assigned managed identity on Azure. See [Azure documentation](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?source=recommendations\&pivots=identity-mi-methods-azp#create-a-user-assigned-managed-identity).
    * The [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli).

    <Warning>You can only use the same user-assigned managed identity for up to four Deployments. If you need to authorize more than four Deployments to Azure, you need to create more than one user-managed identity. For more information, see [Microsoft Entra documentation](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation-considerations#general-federated-identity-credential-considerations).</Warning>

    #### Step 1: Authorize the managed identity in Azure

    1. In your Azure portal, open the resource that your managed identity needs access to. Then, select **Access control (IAM)**.
    2. Click **Add** > **Add role assignment**.
    3. Select the role for your managed identity, then click **Next**.
    4. In the **Assign access to** section, select **Managed identity**. Click **+ Select Members** and choose your managed identity. After you add your managed identity, click **Next**.
    5. Review and finalize the assignment.

    #### Step 2: Configure your Deployment

    1. In your Azure portal, open the **Managed Identities** menu.

    2. Search for your managed identity, click **Properties**, then copy its **Name**, **Client ID**, **Tenant ID**, and **Resource group** name.

    3. In the Astro UI, select your Deployment, click **Details**, then click **How to Configure...** under **Workload Identity**.

    4. In **Managed Identity**, enter the Name of the managed identity you assigned to the resource.

    5. In **Resource Group**, enter the **Resource group** name that your managed identity belongs to.

    6. Using the Azure CLI, copy and run the provided command in your local terminal.

       <Info>
         **About the Azure CLI command**

         The command creates [federated identity credentials](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation) on your user-assigned managed identity, one for each of the Deployment's Kubernetes service accounts (`scheduler`, `triggerer`, `worker`, `apiserver`, `dag-processor`, and `kpo`). Each federated credential trusts the Astro cluster's OIDC issuer and maps a specific Kubernetes service account to your managed identity.

         You can view the created credentials in the Azure portal at **Managed Identities** > *your managed identity* > **Settings** > **Federated credentials**.

         The principal that runs the command needs the [Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#contributor) or [Owner](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#owner) role on the managed identity. If the command fails, verify these permissions and check the **Federated credentials** page in the Azure portal.
       </Info>

    7. After the command completes, click **Close** on the window in the Astro UI.

    8. (Optional) repeat Steps 4 - 8 for any other Deployments that need to be authorized to Azure.

    #### Step 3: Create an Airflow connection (Azure)

    <Tabs>
      <Tab title="New Astro UI">
        1. In the Astro UI, go to **Environment** > **Connections**.
        2. Click **+ New Connection** to add a new connection for your Workspace.
        3. Search for **Azure**, then select the **Managed identity** option.
        4. Configure your Airflow connection with the information you copied in the previous steps.
        5. Link the connection to the Deployment(s) where you configured your managed identity.
      </Tab>

      <Tab title="Legacy UI">
        1. In the Astro UI, click **Environment** in the main menu to open the **Connections** page.
        2. Click **+ Connection** to add a new connection for your Workspace.
        3. Search for **Azure**, then select the **Managed identity** option.
        4. Configure your Airflow connection with the information you copied in the previous steps.
        5. Link the connection to the Deployment(s) where you configured your managed identity.
      </Tab>
    </Tabs>

    Any Dag that uses your connection will now be authorized to Azure through your managed identity.

    <Tip>
      If you use a mix of strategies for managing connections, if you define the same connection in multiple ways, Airflow uses the following order of precedence:

      * Secrets Backend
      * Environment Manager
      * Environment Variables
      * Airflow UI using the Airflow metadata database
    </Tip>
  </Tab>
</Tabs>

## See also

* [Manage Airflow connections and variables](/docs/astro/manage-connections-variables)
* [Deploy code to Astro](/docs/astro/deploy-code)


# Authorize Workspaces to a cluster
Source: https://astronomer.io/docs/astro/authorize-workspaces-to-a-cluster

Learn how to configure a cluster so that only specific Workspaces can use it.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

To provide greater control over your cloud resources, you can set a rule so that only specific Workspaces can use a specific cluster. For example, you can configure a cluster so that only production-level Workspaces can create Deployments in the cluster.

Use this document to learn restrict a cluster so that only authorized Workspaces can use it.

## Prerequisites

* [Organization Owner](/docs/astro/user-permissions#organization-roles) permissions.
* A [dedicated cluster](/docs/astro/create-dedicated-cluster).

## Authorize your workspace

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Clusters**, then select a cluster.
    2. Go to the **Workspace Authorization** tab and then click **Edit Workspace Authorization**.
    3. Click **Restricted** and select the Workspaces that you want to authorize to the cluster.
    4. Click **Update**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI's **Organization** section, click **Organization Settings**, then click **Clusters** and then select a cluster.
    2. Go to the **Workspace Authorization** tab and then click **Edit Workspace Authorization**.
    3. Click **Restricted** and select the Workspaces that you want to authorize to the cluster.
    4. Click **Update**.
  </Tab>
</Tabs>

After you authorize Workspaces to a cluster, Astro treats the cluster as restricted. Restricted clusters do not appear in the dedicated cluster drop-down menu within the **Create Deployment** view of Workspaces that have not been authorized.

<Info>
  A restricted cluster can't host Deployments from an unauthorized Workspace. To restrict a cluster that's currently running Deployments from unauthorized Workspaces, you must transfer the Deployments from these Workspaces to the Workspaces you want to authorize.

  Similarly, to unauthorize a Workspace but still keep its Deployments in the cluster, you must transfer your Deployments to a Workspace that is still authorized to the cluster. See [Transfer a Deployment to another Workspace](/docs/astro/transfer-a-deployment).
</Info>


# Authenticate an automation tool to Astro
Source: https://astronomer.io/docs/astro/automation-authentication

Learn about all possible ways that you can authenticate your automation tool to Astro.

Before you can automate actions on Astro, you must prove to Astro that your automation tool has the correct identity and access to interact with specific Astro resources. Complete the following actions to authenticate to Astro using the Astro CLI and API tokens:

* Create an API token in Astro.
* Install the Astro CLI in your automation environment, such as a GitHub Actions.
* Make the token accessible to the Astro CLI installed in your automation environment.

After you complete this setup, your automation environment is authenticated to Astro. You can then write and run scripts to manage your Deployments through CI/CD.

Astro's authentication process is based on the [Auth0 Identifier First Authentication flow](https://auth0.com/docs/authenticate/login/auth0-universal-login/identifier-first). This process doesn't provide authorization and isn't affected by what a user can do in Astro. To manage authorization in Astro, see [User permissions](/docs/astro/user-permissions).

## Step 1: Create an API token

You can use any of the following credentials to authenticate in an automated process:

* A Deployment API token. See [Deployment API tokens](/docs/astro/deployment-api-tokens).
* A Workspace API token. See [Create a Workspace API token](/docs/astro/workspace-api-tokens).
* An Organization API token. See [Create an Organization API token](/docs/astro/organization-api-tokens).

When you create an API token for your environment, keep the following best practices in mind:

* Always give your API token the minimum permissions required to perform an action. This improves control and security over your Astro components. For example, instead of creating an Organization API token to automate actions across two  Workspaces, create a separate Workspace API token for each Workspace.

* Always set an expiration date for your API tokens.

* Always [rotate your API tokens](/docs/astro/workspace-api-tokens#rotate-a-workspace-api-token) for enhanced security.

## Step 2: Install the Astro CLI in your automation tool

To manage your Astro workflows programmatically, you must install the Astro CLI in the environment which will run the workflows. Typically, this requires running `curl -sSL install.astronomer.io | sudo bash -s` or an equivalent installation command before your process starts. See [CI/CD templates](/docs/astro/ci-cd-templates/template-overview) for examples of how to install the Astro CLI in different version management and workflow automation environments.

## Step 3: Add your API token to your environment

To make your API token accessible to the Astro CLI, you need to set specific environment variables in your CI/CD tool or automation environment.

<Warning>Because these environment variables store sensitive credentials, Astronomer recommends encrypting the variable values before using them in your script. You can do this either directly in your automation tool or in a secrets backend.</Warning>

To use a Deployment, Workspace, or Organization API token as an authentication credential, set the following environment variable in your script:

```text wrap theme={null}
ASTRO_API_TOKEN=<your-api-token>
```

## See also

* [API authentication and token security](/docs/astro/api-authentication)
* [Develop a CI/CD workflow](/docs/astro/set-up-ci-cd)
* [Manage Deployments as code](/docs/astro/manage-deployments-as-code)


# Automate actions on Astro
Source: https://astronomer.io/docs/astro/automation-overview

Learn how you can automate various actions on Astro to quickly build and manage your data ecosystem.

As an administrator or head of your team, you can use the Astro CLI to automate the management of Deployments and Workspaces. Some common actions you can automate include:

* Managing users.
* Deploying code in a CI/CD pipeline.
* Creating Deployments.

Automating actions allows your team to interact with Astro in a predictable way that improves reliability and security. For example, when you automate code deploys with CI/CD, your can have your users deploy from a source where all of their work is tracked and reviewed, such as GitHub.

This section of documentation covers how to automate processes on Astro using Astro CLI. To start automating, you'll first [programmatically authenticate to Astro](/docs/astro/automation-authentication) using an API token. Then, you'll write and run a script to perform your action.

## Common actions to automate

### Deployment actions

* Deploy code to your Deployment using [CI/CD](/docs/astro/set-up-ci-cd).
* Update your Deployment using a [Deployment file](/docs/astro/manage-deployments-as-code).
* Make a request to your Deployment using the [Airflow REST API](/docs/astro/airflow-api).

### Workspace actions

* [Manage users, Teams, and tokens](/docs/cli/v1.43/astro-workspace-list) in your Workspace.
* Create [preview Deployments](/docs/astro/ci-cd-templates/github-actions-deployment-preview) using CI/CD.
* Perform all Deployment-level actions on any Deployment in a Workspace.

### Organization actions

* [Manage Organization users, Teams, and tokens](/docs/cli/v1.43/astro-organization-list).
* Export [audit logs](/docs/astro/audit-logs#export-audit-logs).


# Configure the Celery executor
Source: https://astronomer.io/docs/astro/celery-executor

Configure the Celery executor on Astro by setting worker size, count, and concurrency per queue.

On Astro, you can configure Celery executor in the following ways:

* The type and size of your workers.
* The minimum and maximum number of workers that your Deployment can run at a time.
* The number of tasks that each worker can run at a time.

You can set these configurations per [worker queue](/docs/astro/configure-worker-queues). With the Celery executor, you can configure multiple worker queues for different types of tasks and assign tasks to those queues in your Dag code.

The following document explains basic Celery executor configurations for a single worker queue. For instructions on how to configure multiple worker queues, see [Create a worker queue](/docs/astro/configure-worker-queues#create-a-worker-queue).

<Tip>If you plan to use only the `KubernetesPodOperator` in your Deployment, set your Celery executor resources to the lowest possible amounts, because the executor is only required for launching your Pods. See [`KubernetesPodOperator`](/docs/astro/kubernetespodoperator) for more information.</Tip>

## Celery worker autoscaling logic

The number of Celery workers running per worker queue on your Deployment at a given time is based on two values:

* The total number of tasks in a `queued` or `running` state
* The worker queue's setting for **Concurrency**

The calculation is made based on the following expression:

`[Number of workers]= ([Queued tasks]+[Running tasks])/(Concurrency)`

Deployment [parallelism](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#parallelism) is the maximum number of tasks that can run concurrently across worker queues. To ensure that you can always run as many tasks as your worker queues allow, parallelism is calculated with the following expression:

`[Parallelism]= ([The sum of all 'Max Worker Count' values for all worker queues] * [The sum of all 'Concurrency' values for all worker queues])`.

[Kubernetes Event Driven Autoscaling](https://keda.sh/) (KEDA) computes these calculations every ten seconds. When KEDA determines that it can scale down a worker, it waits for five minutes after the last running task on the worker finishes before terminating the worker Pod.

When you push code to a Deployment, workers running tasks from before the code push don't scale down until those tasks is complete. To learn more about how changes to a Deployment can affect worker resource allocation, see [What happens during a code deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy).

## Configure Celery worker scaling

For each worker queue on your Deployment, you have to specify certain settings that affect worker autoscaling behavior. If you're new to Airflow, Astronomer recommends using the defaults in Astro for each of these settings.

1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
2. Click the **Details** tab and then click **Edit** in the **Execution** section to edit a worker queue.
3. Configure the following settings:

   * **Worker type**: Choose the amount of resources that each worker will have.
   * **Concurrency**: The maximum number of tasks that a single worker can run at a time. If the number of queued and running tasks exceeds this number, a new worker is added to run the remaining tasks. This value is equivalent to the Apache Airflow [worker concurrency](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#worker-concurrency) setting. It is 16 by default.
   * **Storage**: Choose the amount of ephemeral storage in GiB that each worker has. This storage volume is transient and allows for temporary storage and processing of data within the worker. The worker is assigned the minimum 10 GiB by default. The maximum quota is 100 GiB. Only ephemeral storage requests that are greater than the default minimum of 10 GiB are chargeable.
   * **Worker Count (Min-Max)**: The minimum and maximum number of workers that can run at a time. The number of running workers changes based on **Concurrency** and the current number of tasks in a queued or running state. By default, the minimum number of workers is 1 and the maximum is 10.

<Info>The number of running workers might temporarily exceed the max when longer duration tasks delay scaled-down workers from shutting down.</Info>

4. Click **Update Queue**.

## See also

* [Configure worker queues](/docs/astro/configure-worker-queues).
* [Airflow Executors explained](/docs/learn/airflow-executors-explained)


# Astro CI/CD templates for AWS CodeBuild
Source: https://astronomer.io/docs/astro/ci-cd-templates/aws-codebuild

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using AWS CodeBuild.

Use the following CI/CD templates to automate deploying Apache Airflow dags from a Git repository to Astro with [AWS CodeBuild](https://aws.amazon.com/codebuild/).

The templates for AWS CodeBuild use [image deploy](/docs/astro/ci-cd-templates/template-overview) templates. If you have one Deployment and one environment on Astro, use the [single branch implementation](#single-branch-implementation). If you have multiple Deployments that support development and production environments, use the [multiple branch implementation](#multiple-branch-implementation).

If you use the [dag-only deploy feature](/docs/astro/deploy-dags) on Astro and are interested in a dag deploy CI/CD template, see [Template overview](/docs/astro/ci-cd-templates/template-overview) to configure your own. To learn more about CI/CD on Astro, see [Choose a CI/CD strategy](/docs/astro/set-up-ci-cd).

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) hosted in a Git repository that AWS CodeBuild can access. See [Plan a build in AWS CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/planning.html).
* An [Astro Deployment](/docs/astro/create-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).
* Access to AWS CodeBuild. See [Getting started with CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/getting-started-overview.html).

Each CI/CD template implementation might have additional requirements.

## Single branch implementation

To automate code deploys from a single branch to a single Deployment using AWS CodeBuild, complete the following setup in the Git-based repository that hosts your Astro project:

1. In your AWS CodeBuild pipeline configuration, add the following environment variables:

   * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
   * `ASTRO_DEPLOYMENT_ID`: The ID for your Deployment.

   Be sure to set the value of your API token as secret.

2. At the root of your Git repository, add a [`buildspec.yml`](https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-example) file that includes the following script:

   ```yaml title="buildspec.yml" wrap theme={null}
   version: 0.2

   phases:
     install:
       runtime-versions:
         python: latest

     build:
       commands:
         - echo "${CODEBUILD_WEBHOOK_HEAD_REF}"
         - export ASTRO_API_TOKEN="${ASTRO_API_TOKEN}"
         - curl -sSL install.astronomer.io | sudo bash -s
         - astro deploy "${ASTRO_DEPLOYMENT_ID}" -f
   ```

3. In your AWS CodeBuild project, create a [webhook event](https://docs.aws.amazon.com/codebuild/latest/userguide/webhooks.html) for the Git repository where your Astro project is hosted. If you use GitHub, see [GitHub webhook events](https://docs.aws.amazon.com/codebuild/latest/userguide/github-webhook.html). When you configure the webhook, select an event type of `PUSH`.

Your `buildspec.yml` file now triggers a code push to an Astro Deployment every time a commit or pull request is merged to the `main` branch of your repository.

## Multiple branch implementation

To automate code deploys across multiple Deployments using AWS CodeBuild, complete the following setup.

This setup requires two Deployments on Astro and two branches in your Git repository. The example assumes that one Deployment is a development environment, and that the other Deployment is a production environment. To learn more, see [Multiple environments](/docs/astro/set-up-ci-cd#multiple-environments).

1. In your AWS CodeBuild pipeline configuration, add the following environment variables:

   * `PROD_ASTRO_API_TOKEN`: The value for your production Workspace or Organization API token.
   * `PROD_DEPLOYMENT_ID`: The Deployment ID of your production Deployment.
   * `DEV_ASTRO_API_TOKEN`: The value for your development Workspace or Organization API token.
   * `DEV_DEPLOYMENT_ID`: The Deployment ID of your development Deployment.

   Be sure to set the values for your API tokens as secret.

2. At the root of your Git repository, add a [`buildspec.yml`](https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-example) that includes the following script:

   ```yaml title="buildspec.yml" wrap theme={null}
   version: 0.2

   phases:
     install:
       runtime-versions:
         python: latest

     build:
       commands:
         - |
           if expr "${CODEBUILD_WEBHOOK_HEAD_REF}" : "refs/heads/main" >/dev/null; then
             export ASTRO_API_TOKEN="${PROD_ASTRO_API_TOKEN}"
             curl -sSL install.astronomer.io | sudo bash -s
             astro deploy "${PROD_DEPLOYMENT_ID}" -f
           fi
         - |
           if expr "${CODEBUILD_WEBHOOK_HEAD_REF}" : "refs/heads/dev" >/dev/null; then
             export ASTRO_API_TOKEN="${DEV_ASTRO_API_TOKEN}"
             curl -sSL install.astronomer.io | sudo bash -s
             astro deploy "${DEV_DEPLOYMENT_ID}" -f
           fi
   ```

3. In your AWS CodeBuild project, create a [webhook event](https://docs.aws.amazon.com/codebuild/latest/userguide/webhooks.html) for the Git repository where your Astro project is hosted. If you use GitHub, see [GitHub webhook events](https://docs.aws.amazon.com/codebuild/latest/userguide/github-webhook.html). When you configure the webhook, select an event type of `PUSH`.

Your `buildspec.yml` file now triggers a code push to your development Deployment every time a commit or pull request is merged to the `dev` branch of your repository, and a code push to your production Deployment every time a commit or pull request is merged to the `main` branch of your repository.


# Deploy dags from an AWS S3 bucket to Astro using AWS Lambda
Source: https://astronomer.io/docs/astro/ci-cd-templates/aws-s3

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using AWS S3 and Lambda.

Use the following CI/CD template to automate deploying Apache Airflow dags from an S3 bucket to Astro using AWS Lambda.

## Prerequisites

* An AWS S3 bucket
* An [Astro Deployment](/docs/astro/create-deployment) with [dag-only deploys enabled](/docs/astro/deploy-dags#enable-or-disable-dag-only-deploys-on-a-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).
* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) containing your project configurations.

### Dag deploy template

This CI/CD template can be used to deploy dags from a single S3 bucket to a single Astro Deployment. When you create or modify a dag in the S3 bucket, a Lambda function triggers and initializes an `astro` project to deploy your dags using Astro CLI.

<Info>To deploy any non-dag code changes to Astro, you need to trigger a standard image deploy with your Astro project. When you do this, your Astro project must include the latest version of your dags from your S3 bucket. If your Astro project `dags` folder isn't up to date with your S3 dags bucket when you trigger this deploy, you will revert your dags back to the version hosted in your Astro project.</Info>

1. Download the latest Astro CLI binary from [GitHub releases](https://github.com/astronomer/astro-cli/releases), then rename the file to, `astro_cli.tar.gz`. For example, to use Astro CLI version 1.13.0 in your template, download `astro_1.13.0_linux_amd64.tar.gz` and rename it to `astro_cli.tar.gz`.

2. In your S3 bucket, create the following new folders:

   * `dags`
   * `cli_binary`

3. Add `astro_cli.tar.gz` to `cli_binary`.

4. In the AWS IAM console, [create a new role for AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/lambda-intro-execution-role.html#permissions-executionrole-console) with the following permissions. Replace `<account_id>`, `<lambda_function_name>`, and `<bucket_name>` with your values.

   ```json expandable wrap theme={null}
   {
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "lambdacreateloggroup",
           "Effect": "Allow",
           "Action": "logs:CreateLogGroup",
           "Resource": "arn:aws:logs:us-east-1:<account_id>:*"
       },
       {
           "Sid": "lambdaputlogevents",
           "Effect": "Allow",
           "Action": [
               "logs:CreateLogStream",
               "logs:PutLogEvents"
           ],
           "Resource": [
               "arn:aws:logs:us-east-1:<account_id>:log-group:/aws/lambda/<lambda_function_name>:*"
           ]
       },
       {
           "Sid": "bucketpermission",
           "Effect": "Allow",
           "Action": [
               "s3:GetObject",
               "s3:ListBucket"
           ],
           "Resource": [
               "arn:aws:s3::<bucket_name>",
               "arn:aws:s3::<bucket_name>/*"
           ]
       }
   ]
   }
   ```

5. Author a new AWS Lambda function from scratch with the following configurations:

   * **Function name**: Any
   * **Runtime**: Python 3.9
   * **Architecture**: Any
   * **Execution role**: Click **Use an existing role** and enter the role you created.

6. Configure the following [Lambda environment variables](https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html) for your Lambda function:

   * `ASTRO_HOME`: `\tmp`
   * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
   * `ASTRO_DEPLOYMENT_ID`: Your Deployment ID.

   For production Deployments, Astronomer recommends storing your API credentials in AWS Secrets Manager and referencing them from Lambda. See [https://docs.aws.amazon.com/lambda/latest/dg/configuration-`database.html`](https://docs.aws.amazon.com/lambda/latest/dg/configuration-database.html)

7. Add the following code to `lambda_function.py`. Replace `<bucket_name>` with your value.

   ```python title="lambda_function.py" expandable wrap theme={null}
   import os
   import subprocess
   import tarfile

   import boto3

   BUCKET = os.environ.get("BUCKET", "<bucket_name>")
   s3 = boto3.resource('s3')
   deploymentId = os.environ.get('ASTRO_DEPLOYMENT_ID')

   def untar(filename: str, destination: str) -> None:
       with tarfile.open(filename) as file:
           file.extractall(destination)

   def run_command(cmd: str) -> None:
       p = subprocess.Popen("set -x; " + cmd, shell=True)
       p.communicate()

   def download_to_local(bucket_name: str, s3_folder: str, local_dir: str = None) -> None:
       """
       Download the contents of a folder directory
       Args:
           bucket_name: the name of the s3 bucket
           s3_folder: the folder path in the s3 bucket
           local_dir: a relative or absolute directory path in the local file system
       """
       bucket = s3.Bucket(bucket_name)
       for obj in bucket.objects.filter(Prefix=s3_folder):
           target = obj.key if local_dir is None \
               else os.path.join(local_dir, os.path.relpath(obj.key, s3_folder))
           if not os.path.exists(os.path.dirname(target)):
               os.makedirs(os.path.dirname(target))
           if obj.key[-1] == '/':
               continue
           bucket.download_file(obj.key, target)
       print("downloaded file")

   def lambda_handler(event, context) -> None:
       """Triggered by a change to a Cloud Storage bucket.
       :param event: Event payload.
       :param context: Metadata for the event.
       """
       base_dir = '/tmp/astro'
       if not os.path.isdir(base_dir):
           os.mkdir(base_dir)

       download_to_local(BUCKET, 'dags/', f'{base_dir}/dags')
       download_to_local(BUCKET, 'cli_binary/', base_dir)

       os.chdir(base_dir)
       untar('./astro_cli.tar.gz', '.')

       run_command('echo y | ./astro dev init')
       run_command(f"./astro deploy {deploymentId} --dags")

       return {"statusCode": 200}
   ```

8. [Create a trigger](https://docs.aws.amazon.com/lambda/latest/dg/lambda-invocation.html) for your Lambda function with the following configuration:

   * **Source**: Select **S3**.
   * **Bucket**: Select the bucket that contains your `dags` directory.
   * **Event types**: Select **All object create events**.
   * **Prefix**: Enter `dags/`.
   * **Suffix**: Enter `.py`.

9. If you haven't already, deploy your complete Astro project to your Deployment. See [Deploy code](/docs/astro/deploy-code).

<Info>
  If you stage multiple commits to dag files and push them all at once to your remote branch, the template only deploys dag code changes from the most recent commit. It will miss any code changes made in previous commits.

  To avoid this, either push commits individually or configure your repository to **Squash commits** for pull requests that merge multiple commits simultaneously.
</Info>

10. Add your dags to the `dags` folder in your storage bucket
11. In the Astro UI, click **Deployments**, then select your Deployment. Confirm that your Lambda function worked by checking the Deployment **Dag bundle version**. The version's name should include the time that you added the dags to your S3 bucket.


# Astro CI/CD templates for Azure DevOps
Source: https://astronomer.io/docs/astro/ci-cd-templates/azure-devops

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using Azure DevOps.

Use the following CI/CD templates to automate deploying Apache Airflow Dags from a Git repository to Astro with [Azure DevOps](https://dev.azure.com/).

The template for Azure DevOps is based on the [image deploy template](/docs/astro/ci-cd-templates/template-overview#image-deploy-templates) with a [single branch implementation](#single-branch-implementation), which requires only one Astro Deployment.

If you use the [Dag-only deploy feature](/docs/astro/deploy-dags) on Astro or you're interested in a multiple-branch implementation, see [Template overview](/docs/astro/ci-cd-templates/template-overview#multiple-branch-implementation) to configure your own. To learn more about CI/CD on Astro, see [Choose a CI/CD strategy](/docs/astro/set-up-ci-cd).

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) hosted in a Git repository that Azure DevOps can access.
* An [Astro Deployment](/docs/astro/create-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).
* Access to [Azure DevOps](https://dev.azure.com/).

## Single branch implementation

Complete the following setup in an Azure repository that hosts an Astro project:

1. Set the following environment variable as a [DevOps pipeline variable](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops\&tabs=yaml%2Cbatch):

   * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
   * `ASTRO_DEPLOYMENT_ID`: The ID for your Deployment.

   For production Deployments, ensure that you set the value for your API token as a [secret](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops\&tabs=yaml%2Cbatch#secret-variables).

2. Create a new Azure DevOps pipeline named `astro-devops-cicd.yaml` at the root of the repository that includes the following configuration:

   ```yaml title="astro-devops-cicd.yaml" wrap theme={null}
   trigger:
   - main

   pr: none

   stages:
   - stage: deploy
     jobs:
     - job: deploy_image
       pool:
         vmImage: 'ubuntu-latest'
       steps:
       - script: |
           curl -sSL install.astronomer.io | sudo bash -s
           astro deploy ${ASTRO_DEPLOYMENT_ID}
         env:
           ASTRO_API_TOKEN: $(ASTRO_API_TOKEN)
           ASTRO_DEPLOYMENT_ID: $(ASTRO_DEPLOYMENT_ID)
   ```


# Astro CI/CD templates for Bitbucket
Source: https://astronomer.io/docs/astro/ci-cd-templates/bitbucket

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using BitBucket.

Use the following CI/CD templates to automate deploying Apache Airflow dags from a Git repository to Astro with [Bitbucket](https://bitbucket.org/product).

The templates for Bitbucket use the [image deploy](/docs/astro/ci-cd-templates/template-overview) process with a [single branch implementation](#single-branch-implementation), which requires only one Astro Deployment.

If you use the [dag-only deploy feature](/docs/astro/deploy-dags) on Astro or you're interested in a multiple-branch implementation, see [Template overview](/docs/astro/ci-cd-templates/template-overview#multiple-branch-implementation) to configure your own. To learn more about CI/CD on Astro, see [Choose a CI/CD strategy](/docs/astro/set-up-ci-cd).

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) hosted in a Git repository that Bitbucket can access.
* An [Astro Deployment](/docs/astro/create-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).
* Access to [Bitbucket](https://bitbucket.org/product).

## Single branch implementation

To automate code deploys to a Deployment using [Bitbucket](https://bitbucket.org/), complete the following setup in a Git-based repository that hosts an Astro project:

1. Set the following environment variable as a [Bitbucket pipeline variable](https://support.atlassian.com/bitbucket-cloud/docs/variables-and-secrets/):

   * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
   * `ASTRO_DEPLOYMENT_ID`: Your Deployment ID.

   For production Deployments, be sure to set the value of your API token as **secured**.

2. Create a new YAML file in `bitbucket-pipelines.yml` at the root of the repository that includes the following configuration:

   ```yaml title="bitbucket-pipelines.yml" wrap theme={null}
   pipelines:
     pull-requests: # The branch pattern under pull requests defines the *source* branch.
       dev:
         - step:
             name: Deploy to Production
             deployment: Production
             script:
               - curl -sSL install.astronomer.io | sudo bash -s
               - astro deploy ${ASTRO_DEPLOYMENT_ID} -f
             services:
               - docker
   ```


# Astro CI/CD templates for CircleCI
Source: https://astronomer.io/docs/astro/ci-cd-templates/circleci

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using CircleCI.

[CircleCI](https://circleci.com/) is a continuous integration and continuous delivery platform that can be used to implement DevOps practices. This document provides sample CI/CD templates to automate deploying Apache Airflow dags from a GitHub repository to Astro using CircleCI.

If you have one Deployment and one environment on Astro, use the [single branch implementation](/docs/astro/ci-cd-templates/template-overview#single-branch-implementation). If you have multiple Deployments that support development and production environments, use the [multiple branch implementation](/docs/astro/ci-cd-templates/template-overview#multiple-branch-implementation). If your team builds custom Docker images, use the [custom image implementation](/docs/astro/ci-cd-templates/template-overview#custom-image-implementation).

Refer to [Template overview](/docs/astro/ci-cd-templates/template-overview) to see generic templates expressed as simple shell scripts or configure your own. To learn more about CI/CD on Astro, see [Choose a CI/CD strategy](/docs/astro/set-up-ci-cd).

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) hosted in a Git repository that CircleCI can access.
* An [Astro Deployment](/docs/astro/create-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).
* A [CircleCI](https://circleci.com/vcs-authorize/) account.

## Image deploy templates

[Image deploy templates](/docs/astro/ci-cd-templates/template-overview) build a Docker image and push it to Astro whenever you update any file in your Astro project.

<Tabs>
  <Tab title="Single branch">
    To automate code deploys to a Deployment using CircleCI for a single branch implementation, complete the following setup in a Git-based repository that hosts an Astro project:

    #### Configuration requirements (Single branch, code deploy)

    * You have a `main` branch of an Astro project hosted in a single GitHub repository.
    * You have a production Deployment on Astro where you want to deploy your `main` GitHub branch.
    * You have a production CircleCI context that stores environment variables for your CI/CD workflows.

    #### Implementation (Single branch, code deploy)

    1. Set the following environment variables in a [CircleCI context](https://circleci.com/docs/guides/security/contexts/):

    * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
    * `ASTRO_DEPLOYMENT_ID`: The ID for your Deployment.

    2. Create a new YAML file in `.circleci/config.yml` that includes the following configuration:

    ```yaml title=".circleci/config.yml" expandable wrap theme={null}
    # Use the latest CircleCI pipeline process engine version.
    # See: https://circleci.com/docs/2.0/configuration-reference
    version: 2.1

    orbs:
      docker: circleci/docker@2.0.1
      github-cli: circleci/github-cli@2.0.0

    # Define a job to be invoked later in a workflow.
    # See: https://circleci.com/docs/2.0/configuration-reference/#jobs
    jobs:

      build_image_and_deploy:
        docker:
          - image: cimg/base:stable
        # Add steps to the job
        # See: https://circleci.com/docs/2.0/configuration-reference/#steps
        steps:
          - setup_remote_docker:
              version: 20.10.11
          - checkout
          - run:
              name: "Deploy to Astro"
              command: |
                curl -sSL install.astronomer.io | sudo bash -s
                astro deploy ${ASTRO_DEPLOYMENT_ID} -f

    # Invoke jobs with workflows
    # See: https://circleci.com/docs/2.0/configuration-reference/#workflows
    workflows:
      version: 2.1
      wf-build-and-deploy:
        jobs:
          - build_image_and_deploy:
              context:
                - <YOUR-CIRCLE-CI-CONTEXT>
              filters:
                branches:
                  only:
                    - <YOUR-BRANCH-NAME>
    ```
  </Tab>

  <Tab title="Multiple branch">
    The following template can be used to create a multiple branch CI/CD pipeline using CircleCI. A multiple branch pipeline can be used to test dags in a development Deployment and promote them to a production Deployment.

    #### Configuration requirements (Multiple branch, code deploy)

    * You have both a `dev` and `main` branch of an Astro project hosted in a single GitHub repository.
    * You have respective development and production Deployments on Astro where you deploy your GitHub branches to.
    * You have respective development and production CircleCI contexts that store environment variables to use in your CI/CD workflows.

    #### Implementation (Multiple branch, code deploy)

    1. Set the following environment variables in both your production and development [CircleCI contexts](https://circleci.com/docs/guides/security/contexts/):

    * `ASTRO_API_TOKEN` = `<your-workspace-or-organization-api-token-with-access-to-prod-deployment>`
    * `ASTRO_DEPLOYMENT_ID` = `<your-prod-deployment-id>`

    2. Create a new YAML file in `.circleci/config.yml` that includes the following configuration:

    ```yaml title=".circleci/config.yml" expandable wrap theme={null}
    # Use the latest CircleCI pipeline process engine version.
    # See: https://circleci.com/docs/2.0/configuration-reference
    version: 2.1

    orbs:
      docker: circleci/docker@2.0.1
      github-cli: circleci/github-cli@2.0.0

    # Define a job to be invoked later in a workflow.
    # See: https://circleci.com/docs/2.0/configuration-reference/#jobs
    jobs:

      build_image_and_deploy:
        docker:
          - image: cimg/base:stable
        # Add steps to the job
        # See: https://circleci.com/docs/2.0/configuration-reference/#steps
        steps:
          - setup_remote_docker:
              version: 20.10.11
          - checkout
          - run:
              name: "Deploy to Astro"
              command: |
                curl -sSL install.astronomer.io | sudo bash -s
                astro deploy ${ASTRO_DEPLOYMENT_ID} -f

    # Invoke jobs with workflows
    # See: https://circleci.com/docs/2.0/configuration-reference/#workflows
    workflows:
      version: 2.1
      wf_build-and-deploy:
        jobs:
          - build_image_and_deploy:
              context:
                - <YOUR-PROD-CIRCLE-CI-CONTEXT>
              filters:
                branches:
                  only:
                    - <YOUR-PRODUCTION-BRANCH-NAME>
        jobs:
          - build_image_and_deploy:
              context:
                - <YOUR-DEV-CIRCLE-CI-CONTEXT>
              filters:
                branches:
                  only:
                    - <YOUR-DEVELOPMENT-BRANCH-NAME>
    ```

    Read more about multiple workflows in the [CircleCI documentation](https://circleci.com/docs/schedule-pipelines-with-multiple-workflows/).
  </Tab>

  <Tab title="Custom Image">
    If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments in `docker build` command and then use the `image tag` to deploy to Astro. See [`docker build`](https://docs.docker.com/build/guide/build-args/) for reference.

    #### Configuration requirements (Custom Image, code deploy)

    * You have a `main` branch of an Astro project hosted in a single GitHub repository.
    * You have a production Deployment on Astro where you want to deploy your `main` GitHub branch.
    * You have a production CircleCI context where you store environment variables to use in your CI/CD workflows.

    #### Implementation (Custom Image, code deploy)

    1. Set the following environment variables in a [CircleCI context](https://circleci.com/docs/guides/security/contexts/):

    * `ASTRO_API_TOKEN` = `<your-workspace-or-organization-api-token-with-access-to-prod-deployment>`
    * `ASTRO_DEPLOYMENT_ID` = `<your-deployment-id>`

    2. Create a new YAML file in `.circleci/config.yml` that includes the following configuration:

    ```yaml title=".circleci/config.yml" expandable wrap theme={null}
    # Use the latest CircleCI pipeline process engine version.
    # See: https://circleci.com/docs/2.0/configuration-reference
    version: 2.1

    orbs:
      docker: circleci/docker@2.0.1
      github-cli: circleci/github-cli@2.0.0

    # Define a job to be invoked later in a workflow.
    # See: https://circleci.com/docs/2.0/configuration-reference/#jobs
    jobs:

      build_image_and_deploy:
        docker:
          - image: cimg/base:stable
        # Add steps to the job
        # See: https://circleci.com/docs/2.0/configuration-reference/#steps
        steps:
          - setup_remote_docker:
              version: 20.10.11
          - checkout
          - run:
              name: "Build image and deploy"
              command: |
                set -e
                echo "export image_tag=astro-$(date +%Y%m%d%H%M%S)" >> $BASH_ENV
                source "$BASH_ENV"
                docker build -t ${image_tag} --build-arg="<your-build-arg>=<your-build-arg-value>" .
                curl -sSL install.astronomer.io | sudo bash -s
                astro deploy --image-name ${image_tag} ${ASTRO_DEPLOYMENT_ID} -f

      # Invoke jobs with workflows
      # See: https://circleci.com/docs/2.0/configuration-reference/#workflows
      workflows:
        version: 2.1
        build-and-deploy-prod:
          jobs:
            - build_image_and_deploy_prod:
                context:
                  - <YOUR-CIRCLE-CI-CONTEXT>
                filters:
                  branches:
                    only:
                      - <YOUR-BRANCH-NAME>
    ```

    <Info>If you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to [Astronomer support](https://support.astronomer.io/).</Info>
  </Tab>
</Tabs>

## Dag deploy templates

A [dag deploy template](/docs/astro/ci-cd-templates/template-overview#dag-deploy-templates) uses the `--dags` flag in the `astro deploy` command in the Astro CLI  to push only dags to your Deployment.

This CI/CD pipeline deploys your dags to Astro when one or more files in your `dags` folder are modified. It deploys the rest of your Astro project as a Docker image when other files or directories are also modified. For more information about the benefits of this workflow, see [Deploy dags only](/docs/astro/deploy-dags).

### Configuration requirements (dag deploy)

For each Deployment that you use with dag deploy templates, you must [enable dag deploys](/docs/astro/deploy-dags).

<Info>
  If you stage multiple commits to dag files and push them all at once to your remote branch, the template only deploys dag code changes from the most recent commit. It will miss any code changes made in previous commits.

  To avoid this, either push commits individually or configure your repository to **Squash commits** for pull requests that merge multiple commits simultaneously.
</Info>

### Single branch implementation

To automate code deploys to a Deployment using [CircleCI](https://circleci.com/), complete the following setup in a Git-based repository that hosts an Astro project:

1. Set the following environment variables in a [CircleCI context](https://circleci.com/docs/guides/security/contexts/):

* `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
* `ASTRO_DEPLOYMENT_ID`: The ID for your Deployment.

2. In your project repository, create a new YAML file in `.circleci/config.yml` that includes the following configuration:

```yaml title=".circleci/config.yml" expandable wrap theme={null}
# Use the latest CircleCI pipeline process engine version.
# See: https://circleci.com/docs/2.0/configuration-reference
version: 2.1

orbs:
  docker: circleci/docker@2.0.1
  github-cli: circleci/github-cli@2.0.0

# Define a job to be invoked later in a workflow.
# See: https://circleci.com/docs/2.0/configuration-reference/#jobs
jobs:

  build_image_and_deploy:
    docker:
      - image: cimg/base:stable
    # Add steps to the job
    # See: https://circleci.com/docs/2.0/configuration-reference/#steps
    steps:
      - setup_remote_docker:
          version: 20.10.11
      - checkout
      - run:
          name: "Build image and deploy"
          command: |
            curl -sSL install.astronomer.io | sudo bash -s
            files=($(git diff-tree HEAD --name-only --no-commit-id))
            echo ${files}
            find="dags"
            if [[ ${files[*]} =~ (^|[[:space:]])"$find"($|[[:space:]]) && ${#files[@]} -eq 1 ]]; then
              echo "only deploying dags"
              astro deploy ${ASTRO_DEPLOYMENT_ID} --dags -f;
            else
              echo "image deploy"
              astro deploy ${ASTRO_DEPLOYMENT_ID} -f;
            fi

# Invoke jobs with workflows
# See: https://circleci.com/docs/2.0/configuration-reference/#workflows
workflows:
  version: 2.1
  wf-build-and-deploy:
    jobs:
      - build_image_and_deploy:
          context:
            - <YOUR-CIRCLE-CI-CONTEXT>
          filters:
            branches:
              only:
                - <YOUR-BRANCH-NAME>
```

This script checks the diff between your current commit and the HEAD of your branch to which you are pushing the changes to. If the changes are only in `dags` then it executes a `dag-only` deploy. Otherwise, it executes an image-based deploy. Make sure to customize the script to use your specific branch and context.

You can customize this script to work for multiple branches as shown in the [image-based multi-branch deploy template](/docs/astro/ci-cd-templates/circleci?tab=multibranch#image-deploy-templates) by creating separate `job` and `workflow` for each branch.


# dbt deploy action for deploying dbt code to Astro
Source: https://astronomer.io/docs/astro/ci-cd-templates/dbt-deploy-action

Use pre-built Astronomer CI/CD templates to automate deploying dbt code to Astro using GitHub Actions.

<Note>There is a hard limit of 10 dbt bundles per Astro Deployment</Note>

If you have a dbt project that you want to use with the deploy action for a GitHub action, you can choose to use either the dbt deploy on its own, or you can create GitHub actions that combine the `infer` action with a dbt deploy.

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) hosted in a GitHub repository.
* An [Astro Deployment](/docs/astro/create-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).
* Access to [GitHub Actions](https://github.com/features/actions).

Each CI/CD template implementation might have additional requirements.

<Warning>
  If you use a [self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners) to execute jobs from GitHub Actions, the Astro CLI's `config.yaml` file, which stores default deploy details, might be shared across your organization and hence multiple CI/CD pipelines. To reduce the risk of accidentally deploying to the wrong Deployment, ensure the following:

  * Add `ASTRO_API_TOKEN` to your repository and include a check in your GitHub workflow to verify that it exists.
  * Use Deployment API tokens, which are scoped only to one Deployment, instead of Workspace or Organization API tokens.
  * Specify `deployment-id` or `deployment-name` in your action. For example, `astro deploy <deployment-id>` or `astro deploy -n <deployment-name>`.
  * Add the command `astro logout` at the end of your workflow to ensure that your authentication token is cleared from the `config.yaml` file
</Warning>

### Setup

<Note>Each template installs the Astro CLI with [`setup-astro-cli`](https://github.com/astronomer/setup-astro-cli) before `deploy-action` runs. Astronomer recommends installing the CLI in a separate step so that you can pin or upgrade the CLI version independently of the action, including with Dependabot, and reuse a single installation across multiple deploy steps. To pin a version, set the `version` input, for example `version: "1.40.1"`. If you omit this step, `deploy-action` installs the latest version of the Astro CLI itself.</Note>

<Tabs>
  <Tab title="Single branch">
    #### Prerequisites (Single branch)

    * The root folder name for the directory that contains your dbt project.

    #### Implementation (Single branch)

    To automate code deploys to a single Deployment using [GitHub Actions](https://github.com/features/actions) for a dbt project, complete the following setup in a Git-based repository that hosts an Astro project:

    1. Set the following as a [GitHub secret](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository):

    * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.

    2. In your project repository, create a new YAML file in `.github/workflows` that includes the following configuration:

    ```yaml wrap theme={null}
    name: Astronomer CI - Deploy dbt code

    on:
      push:
        branches:
          - main

    env:
      ## Sets Deployment API credentials as environment variables
      ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}

    jobs:
      build:
        runs-on: ubuntu-latest
        steps:
        - name: Install Astro CLI
          uses: astronomer/setup-astro-cli@v0.0.1
        - name: Deploy to Astro
          uses: astronomer/deploy-action@v0.14.0
          with:
            deployment-id: <deployment-id>
            deploy-type: dbt
            root-folder: <dbt-root-folder>
    ```

    3. (Optional) You can add [optional configurations](https://github.com/astronomer/deploy-action/blob/main/README.md#configuration-options) to customize your workflow. For example, if you add [`wake-on-deploy`](https://github.com/astronomer/deploy-action/blob/main/README.md#wake-on-deploy) to your configuration, the Deploy Action wakes a hibernating Deployment before deploying code to it.

    <Warning>Using `wake-on-deploy` takes precedence over any existing Deployment hibernation overrides that you configured through the Astro UI or `config.yaml` file.</Warning>
  </Tab>

  <Tab title="Multiple branch">
    The following template can be used to create a multiple branch CI/CD pipeline using GitHub Actions. A multiple branch pipeline can be used to test dags in a development Deployment and promote them to a production Deployment.

    #### Configuration requirements

    * You have both a `dev` and `main` branch of an Astro project hosted in a single GitHub repository.
    * You have respective `dev` and `prod` Deployments on Astro where you deploy your GitHub branches to.
    * You have at least one API token with access to both of your Deployments.
    * The root folder name for the directory that contains your dbt project.

    #### Implementation (Multiple branch)

    1. Set the following as [GitHub secrets](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository):

    * `PROD_ASTRO_API_TOKEN`: The value for your production Workspace or Organization API token.
    * `DEV_ASTRO_API_TOKEN`: The value for your development Workspace or Organization API token.

    2. In your project repository, create a new YAML file in `.github/workflows` that includes the following configuration:

    ```yaml expandable wrap theme={null}
    name: Astronomer CI - Deploy dbt code (Multiple Branches)

    on:
      push:
        branches: [dev]
      pull_request:
        types:
          - closed
        branches: [main]

    jobs:
      dev-push:
        if: github.ref == 'refs/heads/dev'
        env:
          ## Sets DEV Deployment API credential as an environment variable
          ASTRO_API_TOKEN: ${{ secrets.DEV_ASTRO_API_TOKEN }}
        runs-on: ubuntu-latest
        steps:
        - name: Install Astro CLI
          uses: astronomer/setup-astro-cli@v0.0.1
        - name: Deploy to Astro
          uses: astronomer/deploy-action@v0.14.0
          with:
            deployment-id: <dev-deployment-id>
            deploy-type: dbt
            root-folder: <dbt-root-folder>
      prod-push:
        if: github.event.action == 'closed' && github.event.pull_request.merged == true
        env:
          ## Sets Prod Deployment API credential as an environment variable
          ASTRO_API_TOKEN: ${{ secrets.PROD_ASTRO_API_TOKEN }}
        runs-on: ubuntu-latest
        steps:
        - name: Install Astro CLI
          uses: astronomer/setup-astro-cli@v0.0.1
        - name: Deploy to Astro
          uses: astronomer/deploy-action@v0.14.0
          with:
            deployment-id: <prod-deployment-id>
            deploy-type: dbt
            root-folder: <dbt-root-folder>
    ```

    3. (Optional) You can add [optional configurations](https://github.com/astronomer/deploy-action/blob/main/README.md#configuration-options) to customize your workflow. For example, if you add [`wake-on-deploy`](https://github.com/astronomer/deploy-action/blob/main/README.md#wake-on-deploy) to your configuration, the Deploy Action wakes a hibernating Deployment before deploying code to it.

    <Warning>Using `wake-on-deploy` takes precedence over any existing Deployment hibernation overrides that you configured through the Astro UI or `config.yaml` file.</Warning>
  </Tab>

  <Tab title="Combine dbt and Astro project">
    In addition to configuring the deploy action to deploy just your dbt changes, you can also configure the action to do a `dbt deploy` and an `infer` deploy.

    #### Prerequisites (Combine dbt and Astro project)

    * The root folder name for the directory that `contct.ains` your dbt project

    #### Implementation (Combine dbt and Astro project)

    To automate code deploys to a single Deployment using [GitHub Actions](https://github.com/features/actions) for a dbt project, complete the following setup in a Git-based repository that hosts both an Astro project and a dbt project:

    1. Set the following as a [GitHub secret](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository):

    * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.

    2. In your project repository, create a new YAML file in `.github/workflows` that includes the following configuration:

    ```yaml wrap theme={null}
    name: Astronomer CI - Deploy dbt and Astro project code

    on:
      push:
        branches:
          - main

    env:
      ## Sets Deployment API credentials as environment variables
      ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}

    jobs:
      build:
        runs-on: ubuntu-latest
        steps:
        - name: Install Astro CLI
          uses: astronomer/setup-astro-cli@v0.0.1
        - name: DBT Deploy to Astro
          uses: astronomer/deploy-action@v0.14.0
          with:
            deployment-id: <deployment-id>
            deploy-type: dbt
            root-folder: <dbt-root-folder>
        - name: DAGs/Image Deploy to Astro
          uses: astronomer/deploy-action@v0.14.0
          with:
            deployment-id: <deployment-id>
            root-folder: <astro-project-root-folder>
            parse: true
    ```

    3. (Optional) You can add [optional configurations](https://github.com/astronomer/deploy-action/blob/main/README.md#configuration-options) to customize your workflow. For example, if you add [`wake-on-deploy`](https://github.com/astronomer/deploy-action/blob/main/README.md#wake-on-deploy) to your configuration, the Deploy Action wakes a hibernating Deployment before deploying code to it.

    <Warning>Using `wake-on-deploy` takes precedence over any existing Deployment hibernation overrides that you configured through the Astro UI or `config.yaml` file.</Warning>
  </Tab>
</Tabs>


# Default deploy action for deploying code to Astro
Source: https://astronomer.io/docs/astro/ci-cd-templates/default-deploy-action

Use pre-built Astronomer CI/CD templates to automate deploying code to Astro using GitHub Actions.

By default, the deploy action uses the `infer` deploy type, which enables the action to determine whether to use either a `dags-only` deploy or an `image-and-dags` deploy, depending on the files you change.

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) hosted in a GitHub repository.
* An [Astro Deployment](/docs/astro/create-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).
* Access to [GitHub Actions](https://github.com/features/actions).

Each CI/CD template implementation might have additional requirements.

<Warning>
  If you use a [self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners) to execute jobs from GitHub Actions, the Astro CLI's `config.yaml` file, which stores default deploy details, might be shared across your organization and hence multiple CI/CD pipelines. To reduce the risk of accidentally deploying to the wrong Deployment, ensure the following:

  * Add `ASTRO_API_TOKEN` to your repository and include a check in your GitHub workflow to verify that it exists.
  * Use Deployment API tokens, which are scoped only to one Deployment, instead of Workspace or Organization API tokens.
  * Specify `deployment-id` or `deployment-name` in your action. For example, `astro deploy <deployment-id>` or `astro deploy -n <deployment-name>`.
  * Add the command `astro logout` at the end of your workflow to ensure that your authentication token is cleared from the `config.yaml` file.
</Warning>

<Info>
  If you stage multiple commits to dag files and push them all at once to your remote branch, the template only deploys dag code changes from the most recent commit. It will miss any code changes made in previous commits.

  To avoid this, either push commits individually or configure your repository to **Squash commits** for pull requests that merge multiple commits simultaneously.
</Info>

### Setup

<Note>Each template installs the Astro CLI with [`setup-astro-cli`](https://github.com/astronomer/setup-astro-cli) before `deploy-action` runs. Astronomer recommends installing the CLI in a separate step so that you can pin or upgrade the CLI version independently of the action, including with Dependabot, and reuse a single installation across multiple deploy steps. To pin a version, set the `version` input, for example `version: "1.40.1"`. If you omit this step, `deploy-action` installs the latest version of the Astro CLI itself.</Note>

<Tabs>
  <Tab title="Single branch">
    To automate code deploys to a single Deployment using [GitHub Actions](https://github.com/features/actions), complete the following setup in a Git-based repository that hosts an Astro project:

    1. Set the following as a [GitHub secret](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository):

    * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.

    2. In your project repository, create a new YAML file in `.github/workflows` that includes the following configuration:

    ```yaml wrap theme={null}
    name: Astronomer CI - Deploy code

    on:
      push:
        branches:
          - main

    env:
      ## Sets Deployment API credentials as environment variables
      ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}

    jobs:
      build:
        runs-on: ubuntu-latest
        steps:
        - name: Install Astro CLI
          uses: astronomer/setup-astro-cli@v0.0.1
        - name: Deploy to Astro
          uses: astronomer/deploy-action@v0.14.0
          with:
            deployment-id: <your-deployment-id>
    ```

    3. (Optional) You can add [optional configurations](https://github.com/astronomer/deploy-action/blob/main/README.md#configuration-options) to customize your workflow. For example, if you add [`wake-on-deploy`](https://github.com/astronomer/deploy-action/blob/main/README.md#wake-on-deploy) to your configuration, the Deploy Action wakes a hibernating Deployment before deploying code to it.

    <Warning>Using `wake-on-deploy` takes precedence over any existing Deployment hibernation overrides that you configured through the Astro UI or `config.yaml` file.</Warning>
  </Tab>

  <Tab title="Multiple branch">
    The following template can be used to create a multiple branch CI/CD pipeline using GitHub Actions. A multiple branch pipeline can be used to test dags in a development Deployment and promote them to a production Deployment.

    #### Configuration requirements

    * You have both a `dev` and `main` branch of an Astro project hosted in a single GitHub repository.
    * You have respective `dev` and `prod` Deployments on Astro where you deploy your GitHub branches to.
    * You have at least one API token with access to both of your Deployments.

    #### Implementation (Multiple branch)

    1. Set the following as [GitHub secrets](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository):

    * `PROD_ASTRO_API_TOKEN`: The value for your production Workspace or Organization API token.
    * `DEV_ASTRO_API_TOKEN`: The value for your development Workspace or Organization API token.

    2. In your project repository, create a new YAML file in `.github/workflows` that includes the following configuration:

    ```yaml expandable wrap theme={null}
    name: Astronomer CI - Deploy code (Multiple Branches)

    on:
      push:
        branches: [dev]
      pull_request:
        types:
          - closed
        branches: [main]

    jobs:
      dev-push:
        if: github.ref == 'refs/heads/dev'
        env:
          ## Sets DEV Deployment API credential as an environment variable
          ASTRO_API_TOKEN: ${{ secrets.DEV_ASTRO_API_TOKEN }}
        runs-on: ubuntu-latest
        steps:
        - name: Install Astro CLI
          uses: astronomer/setup-astro-cli@v0.0.1
        - name: Deploy to Astro
          uses: astronomer/deploy-action@v0.14.0
          with:
            deployment-id: <dev-deployment-id>
      prod-push:
        if: github.event.action == 'closed' && github.event.pull_request.merged == true
        env:
          ## Sets Prod Deployment API credential as an environment variable
          ASTRO_API_TOKEN: ${{ secrets.PROD_ASTRO_API_TOKEN }}
        runs-on: ubuntu-latest
        steps:
        - name: Install Astro CLI
          uses: astronomer/setup-astro-cli@v0.0.1
        - name: Deploy to Astro
          uses: astronomer/deploy-action@v0.14.0
          with:
            deployment-id: <prod-deployment-id>
    ```

    3. (Optional) You can add [optional configurations](https://github.com/astronomer/deploy-action/blob/main/README.md#configuration-options) to customize your workflow. For example, if you add [`wake-on-deploy`](https://github.com/astronomer/deploy-action/blob/main/README.md#wake-on-deploy) to your configuration, the Deploy Action wakes a hibernating Deployment before deploying code to it.

    <Warning>Using `wake-on-deploy` takes precedence over any existing Deployment hibernation overrides that you configured through the Astro UI or `config.yaml` file.</Warning>
  </Tab>

  <Tab title="Custom Image">
    If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker's [`build-push-action`](https://github.com/docker/build-push-action).

    #### Prerequisites

    * An Astro project that requires additional build-time arguments to build the Runtime image.

    #### Implementation (Custom Image)

    1. Set the following as a [GitHub secret](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository):

    * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.

    2. In your project repository, create a new YAML file in `.github/workflows` that includes the following configuration:

    ```yaml expandable wrap theme={null}
    name: Astronomer CI - Additional build-time args

    on:
      push:
        branches:
          - main

    jobs:
      build:
        runs-on: ubuntu-latest
        env:
          ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
        steps:
        - name: Check out the repo
          uses: actions/checkout@v3
        - name: Create image tag
          id: image_tag
          run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
        - name: Build image
          uses: docker/build-push-action@v2
          with:
            deployment-id: <your-deployment-id>
            tags: ${{ steps.image_tag.outputs.image_tag }}
            load: true
            # Define your custom image's build arguments, contexts, and connections here using
            # the available GitHub Action settings:
            # https://github.com/docker/build-push-action#customizing .
            # This example uses `build-args` , but your use case might require configuring
            # different values.
            build-args: |
              <your-build-arguments>
        - name: Install Astro CLI
          uses: astronomer/setup-astro-cli@v0.0.1
        - name: Deploy to Astro
          uses: astronomer/deploy-action@v0.14.0
          with:
            deployment-id: <your-deployment-id>
            image-name: ${{ steps.image_tag.outputs.image_tag }}
    ```

    For example, to create a CI/CD pipeline that deploys a project which [installs Python packages from a private GitHub repository](/docs/cli/v1.43/private-python-packages), you would use the following configuration:

    ```yaml expandable wrap theme={null}
    name: Astronomer CI - Custom base image

    on:
      push:
        branches:
          - main

    jobs:
      build:
        runs-on: ubuntu-latest
        env:
          ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
        steps:
        - name: Check out the repo
          uses: actions/checkout@v3
        - name: Create image tag
          id: image_tag
          run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
        - name: Create SSH Socket
          uses: webfactory/ssh-agent@v0.5.4
          with:
            # GITHUB_SSH_KEY must be defined as a GitHub secret.
            ssh-private-key: ${{ secrets.GITHUB_SSH_KEY }}
        - name: (Optional) Test SSH Connection - Should print hello message.
          run: (ssh git@github.com) || true
        - name: Build image
          uses: docker/build-push-action@v2
          with:
            tags: ${{ steps.image_tag.outputs.image_tag }}
            load: true
            ssh: |
              github=${{ env.SSH_AUTH_SOCK }}
        - name: Install Astro CLI
          uses: astronomer/setup-astro-cli@v0.0.1
        - name: Deploy to Astro
          uses: astronomer/deploy-action@v0.14.0
          with:
            deployment-id: <your-deployment-id>
            image-name: ${{ steps.image_tag.outputs.image_tag }}
    ```

    <Info>If you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to [Astronomer support](https://support.astronomer.io/).</Info>

    3. (Optional) You can add [optional configurations](https://github.com/astronomer/deploy-action/blob/main/README.md#configuration-options) to customize your workflow. For example, if you add [`wake-on-deploy`](https://github.com/astronomer/deploy-action/blob/main/README.md#wake-on-deploy) to your configuration, the Deploy Action wakes a hibernating Deployment before deploying code to it.

    <Warning>Using `wake-on-deploy` takes precedence over any existing Deployment hibernation overrides that you configured through the Astro UI or `config.yaml` file.</Warning>
  </Tab>
</Tabs>


# Astro CI/CD templates for Drone
Source: https://astronomer.io/docs/astro/ci-cd-templates/drone

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using Drone CI.

Use the following CI/CD templates to automate deploying Apache Airflow dags from a Git repository to Astro with [Drone CI](https://www.drone.io/).

The template for DroneCI is based on the [image deploy template](/docs/astro/ci-cd-templates/template-overview) with a [single branch implementation](#single-branch-implementation), which requires only one Astro Deployment.

If you use the [dag-only deploy feature](/docs/astro/deploy-dags) on Astro or you're interested in a multiple-branch implementation, see [Template overview](/docs/astro/ci-cd-templates/template-overview#multiple-branch-implementation) to configure your own. To learn more about CI/CD on Astro, see [Choose a CI/CD strategy](/docs/astro/set-up-ci-cd).

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) hosted in a Git repository that Drone can access.
* An [Astro Deployment](/docs/astro/create-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).
* A functional Drone [server](https://docs.drone.io/server/overview/).
* A user with admin privileges to your Drone server.
* A [Docker runner](https://docs.drone.io/runner/docker/overview/).

## Single branch implementation

1. Set the following environment variable as a repository-level [secret](https://docs.drone.io/secret/repository/) on Drone:

* `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.

2. In your Drone server, open your Astro project repository and go to **Settings** > **General**. Under **Project Settings**, turn on the **Trusted** setting.

3. In the top level of your Git repository, create a file called `.drone.yml` that includes the following configuration:

```yaml title=".drone.yml" expandable wrap theme={null}
---
kind: pipeline
type: docker
name: deploy

steps:
  - name: install
    image: debian
    commands:
    - apt-get update
    - apt-get -y install curl
    - curl -sSL install.astronomer.io | sudo bash -s
  - name: wait
    image: docker:dind
    volumes:
    - name: dockersock
      path: /var/run
    commands:
    - sleep 5
  - name: deploy
    image: docker:dind
    volumes:
    - name: dockersock
      path: /var/run
    commands:
    - astro deploy <your-deployment-id> -f
    depends_on:
    - wait
    environment:
      ASTRO_API_TOKEN:
        from_secret: ASTRO_API_TOKEN

services:
- name: docker
  image: docker:dind
  privileged: true
  volumes:
  - name: dockersock
    path: /var/run

volumes:
- name: dockersock
  temp: {}

trigger:
  branch:
  - main
  event:
  - push
```


# Deploy Dags from Google Cloud Storage to Astro
Source: https://astronomer.io/docs/astro/ci-cd-templates/gcs

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow Dags to Astro using Google Cloud Storage.

## Prerequisites

* A Google Cloud Storage (GCS) bucket.
* An [Astro Deployment](/docs/astro/create-deployment) with [Dag-only deploys enabled](/docs/astro/deploy-dags#enable-or-disable-dag-only-deploys-on-a-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).
* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) containing your project configurations.

## Dag deploy template

This CI/CD template can be used to deploy Dags from a single GCS bucket to a single Astro Deployment. When you create or modify a Dag in the GCS bucket, a Cloud Function triggers and initializes an Astro project to deploy your Dags using Astro CLI.

<Info>
  To deploy any non-Dag code changes to Astro, you need to trigger a standard image deploy with your Astro project. When you do this, your Astro project must include the latest version of your Dags from your GCS bucket. If your Astro project `dags` folder isn't up to date with your GCS Dags bucket when you trigger this deploy, you revert your Dags back to the version hosted in your Astro project.
</Info>

1. Download the latest Astro CLI binary from [GitHub releases](https://github.com/astronomer/astro-cli/releases), then rename the file to `astro_cli.tar.gz`. For example, to use Astro CLI version 1.40.0 in your template, download `astro_1.40.0_linux_amd64.tar.gz` and rename it to `astro_cli.tar.gz`.

2. In your GCS bucket, create the following new folders:

   * `dags`
   * `cli_binary`

3. Add `astro_cli.tar.gz` to `cli_binary`.

4. Create a [Cloud Run Function](https://docs.cloud.google.com/run/docs/quickstarts/functions/deploy-functions-console) with the Python 3.12 runtime in the same region as your storage bucket. Use the inline editor to create your function.

5. Create a [Cloud Storage trigger](https://docs.cloud.google.com/run/docs/triggering/storage-triggers) with the following configuration:

   * **Event provider**: Select **Cloud Storage**.
   * **Event**: Select **google.cloud.storage.object.v1.finalized**.
   * **Bucket**: Select your storage bucket.
   * **Service Account**: Ensure the service account you use has the Cloud Run Invoker role.

6. Choose the runtime service account on the **Security** tab of the Cloud Run Functions settings. Ensure that the service account has **Storage Object Viewer** (`storage.objects.list`) access to the Google Cloud Storage bucket.

7. Under the **Containers** settings, set the following [environment variables](https://docs.cloud.google.com/run/docs/configuring/services/environment-variables#setting_runtime_environment_variables) for your Cloud Function:

   * `ASTRO_HOME` = `/tmp`
   * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
   * `ASTRO_DEPLOYMENT_ID`: Your Deployment ID.
   * `BUCKET`: Your GCS bucket.

   For production Deployments, ensure that you store the value for your API token in a secrets backend. See [Secret Manager overview](https://cloud.google.com/secret-manager/docs/overview).

8. When editing the function source, change the function entry point to `astro_deploy`.

9. Add the following code to `main.py`:

```python title="main.py" expandable wrap theme={null}
import os
import shutil
import subprocess
import tarfile

from google.cloud import storage
import functions_framework

BUCKET = os.environ.get("BUCKET", "missing-bucket")
deploymentId = os.environ.get("ASTRO_DEPLOYMENT_ID", "missing-deployment-id")


def clear_dir(path: str) -> None:
    if os.path.exists(path):
        print(f"Clearing directory: {path}")
        shutil.rmtree(path)
    os.makedirs(path, exist_ok=True)
    print(f"Re-created directory: {path}")

def untar(filename: str, destination: str) -> None:
    with tarfile.open(filename) as file:
        file.extractall(destination)

def run_command(cmd: str) -> None:
    print(f'running command: {cmd}')
    p = subprocess.Popen("set -x; " + cmd, shell=True)
    p.communicate()

def download_to_local(bucket_name: str, gcs_folder: str, local_dir: str = None) -> None:
    """Download the contents of a folder directory
    :param bucket_name: the name of the gcs bucket
    :param gcs_folder: the folder path in the gcs bucket
    :param local_dir: a relative or absolute directory path in the local file system
    """

    ## create a storage client to access GCS objects
    storage_client = storage.Client()
    source_bucket = storage_client.bucket(bucket_name)

    ## get a list of all the files in the bucket folder
    blobs = source_bucket.list_blobs(prefix=gcs_folder)

    ## download each of the dag to local
    for blob in blobs:
        if blob.name.endswith('/'):
            continue

        target = blob.name if local_dir is None \
            else os.path.join(local_dir, os.path.relpath(blob.name, gcs_folder))
        print(target)
        if not os.path.exists(os.path.dirname(target)):
            os.makedirs(os.path.dirname(target))

        blob.download_to_filename(target)
        print("downloaded file")

@functions_framework.cloud_event
def astro_deploy(cloud_event) -> None:
    base_dir = '/tmp/astro'
    dags_dir = f'{base_dir}/dags'

    clear_dir(dags_dir)

    # --- Download DAGs ---
    print('downloading dags')
    download_to_local(BUCKET, 'dags/', f'{base_dir}/dags')  # NOTE: use "dags/" prefix

    # --- Download CLI ---
    print('downloading cli')
    download_to_local(BUCKET, 'cli_binary/', base_dir)

    # --- Initialize project ---
    os.chdir(base_dir)
    untar('./astro_cli.tar.gz', '.')
    run_command('echo y | ./astro dev init')

    # --- Remove generated example DAG(s) ---
    example_paths = [
        "dags/example_dag.py",
        "dags/exampledag.py",
    ]

    for path in example_paths:
        full_path = os.path.join(base_dir, path)
        if os.path.exists(full_path):
            print(f"Removing generated example DAG: {full_path}")
            os.remove(full_path)
        else:
            print(f"Example DAG not found: {full_path}")

    # --- Deploy ----
    run_command(f'./astro deploy {deploymentId} --dags')
```

10. Add the dependency `google-cloud-storage` to the `requirements.txt` file for your Cloud Function. See [Specifying Dependencies in Python](https://docs.cloud.google.com/run/docs/runtimes/python-dependencies).

11. (Optional) If you want the function to trigger when Dags are deleted as well as created/modified, create another [Cloud Storage trigger](https://docs.cloud.google.com/run/docs/triggering/storage-triggers) with the following configuration:

    * **Event provider**: Select **Cloud Storage**.
    * **Event**: Select **google.cloud.storage.object.v1.deleted**.
    * **Bucket**: Select your storage bucket.
    * **Service Account**: Ensure the service account you use has the Cloud Run Invoker role.

12. If you haven't already, deploy your complete Astro project to your Deployment. See [Deploy code](/docs/astro/deploy-code).

13. Add your Dags to the `dags` folder in your storage bucket.

14. In the Astro UI, click **Deployments**, then select your Deployment. Confirm that your deploy worked by checking the Deployment **Dag bundle version**. The version's name should include the time that you added the Dags to your GCS bucket.


# GitHub Actions templates for deploying code to preview Deployments on Astro
Source: https://astronomer.io/docs/astro/ci-cd-templates/github-actions-deployment-preview

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to a preview Deployment using GitHub Actions.

<Tip>The Astro GitHub integration can automatically deploy code from a GitHub repository to Astro without you needing to configure a GitHub action. In addition, the Astro UI shows Git metadata for each deploy on your Deployment information screen. See [Deploy code with the Astro GitHub integration](/docs/astro/deploy-github-integration) for setup steps.</Tip>

The Astronomer [deploy action](https://github.com/astronomer/deploy-action/blob/main/README.md#deployment-preview-templates) includes several sub-actions that can be used together to create a complete [Deployment preview](/docs/astro/ci-cd-templates/preview-deployments) pipeline, a configuration that allows you to test your code changes in an ephemeral development Deployment before promoting your changes to a production Astro Deployment.

The Deployment preview templates use GitHub secrets to manage the credentials needed for GitHub to authenticate to Astro. You can specify the credentials for your [secrets backend](/docs/astro/secrets-backend) so that preview Deployments have access to secret Airflow variables or connections during tests. See [Deployment preview template with secrets backend implementation](#deployment-preview-template-with-secrets-backend-implementation).

Deployment preview templates use Astronomer's [`deploy-action`](/docs/astro/ci-cd-templates/template-overview) to automates the deploy process, meaning it can selectively deploy parts of your project based on which files you changed. See [Standard deploy templates](/docs/astro/ci-cd-templates/template-overview) for more information about the `deploy-action`.

<Note>Each template installs the Astro CLI with [`setup-astro-cli`](https://github.com/astronomer/setup-astro-cli) before `deploy-action` runs. Astronomer recommends installing the CLI in a separate step so that you can pin or upgrade the CLI version independently of the action, including with Dependabot, and reuse a single installation across multiple deploy steps. To pin a version, set the `version` input, for example `version: "1.40.1"`. If you omit this step, `deploy-action` installs the latest version of the Astro CLI itself.</Note>

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) hosted in a GitHub repository.
* An [Astro Deployment](/docs/astro/create-deployment).
* A [Workspace API token](/docs/astro/workspace-api-tokens) or [Organization API token](/docs/astro/organization-api-tokens).
* Access to [GitHub Actions](https://github.com/features/actions).

Specific templates might have additional requirements.

<Warning>Creating preview Deployments for Deployments that use a private image registry is currently unsupported.</Warning>

<Warning>
  If you use a [self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners) to execute jobs from GitHub Actions, the Astro CLI's `config.yaml` file, which stores default deploy details, might be shared across your organization and hence multiple CI/CD pipelines. To reduce the risk of accidentally deploying to the wrong Deployment, ensure the following:

  * Add `ASTRO_API_TOKEN` to your repository and include a check in your GitHub workflow to verify that it exists.
  * Specify `deployment-id` or `deployment-name` in your action. For example, `astro deploy <deployment-id>` or `astro deploy -n <deployment-name>`.
  * Add the command `astro logout` at the end of your workflow to ensure that your authentication token is cleared from the `config.yaml` file.
</Warning>

## Deployment preview template

The standard Deployment preview template uses GitHub secrets and an Astro Workspace or Organization API token to create a preview Deployment whenever you create a new feature branch off of your main branch.

### Setup

1. Copy and save the Deployment ID for your Astro Deployment.

<Info>Replace `<main-deployment-id>` with this Deployment ID in all the scripts created in the following steps. Even though some scripts take action on the preview Deployment, the `<main-deployment-id>` should be same for each script.</Info>

2. Set the following [GitHub secret](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository) in the repository hosting your Astro project:

* Key: `ASTRO_API_TOKEN`
* Secret: `<your-token>`

3. In your project repository, create a new YAML file in `.github/workflows` named `deploy-to-preview.yml` that includes the following configuration:

```yaml title="deploy-to-preview.yml" wrap theme={null}
name: Astronomer CI - Deploy code to preview

on:
  pull_request:
    branches:
      - main

env:
  ## Set your API token as a GitHub secret
  ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Install Astro CLI
      uses: astronomer/setup-astro-cli@v0.0.1
    - name: Create preview Deployment
      uses: astronomer/deploy-action@v0.14.0
      continue-on-error: true  # Create action fails if deploy preview already exist, which is expected for subsequent commits in the PR
      with:
        action: create-deployment-preview
        deployment-id: <main-deployment-id>
        wait-time: 10m # Max wait time for the preview deployment to be completed by the deploy action. The workflow will fail if the deployment fails to create within give time period.
    - name: Deploy code to preview
      uses: astronomer/deploy-action@v0.14.0
      with:
        action: deploy-deployment-preview
        deployment-id: <main-deployment-id>
```

4. In the same folder, create a new YAML file named `delete-preview-deployment.yml` that includes the following configuration:

```yaml title="delete-preview-deployment.yml" wrap theme={null}
name: Astronomer CI - Delete Preview Deployment

on:
  delete:
    branches:
      - "**"
env:
  ## Set your API token as a GitHub secret
  ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Install Astro CLI
      uses: astronomer/setup-astro-cli@v0.0.1
    - name: Delete preview Deployment
      uses: astronomer/deploy-action@v0.14.0
      with:
        action: delete-deployment-preview
        deployment-id: <main-deployment-id>
```

5. In the same folder, create a new YAML file named `deploy-to-main-deployment.yml` that includes the following configuration:

```yaml title="deploy-to-main-deployment.yml" wrap theme={null}
name: Astronomer CI - Deploy code to main Deployment

on:
  push:
    branches:
      - main

env:
  ## Set your API token as a GitHub secret
  ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Install Astro CLI
      uses: astronomer/setup-astro-cli@v0.0.1
    - name: Deploy code to main Deployment
      uses: astronomer/deploy-action@v0.14.0
      with:
        deployment-id: <main-deployment-id>
```

6. (Optional) You can add [optional configurations](https://github.com/astronomer/deploy-action/blob/main/README.md#configuration-options) to customize your workflow.

All three workflow files must have the same Deployment ID specified. The actions use this Deployment ID to create and delete preview Deployments based on your main Deployment.

## Deployment preview template with secrets backend implementation

If you use a [secrets backend](/docs/astro/secrets-backend) to manage Airflow objects such as variables and connections, you can configure your action to grant preview Deployments access to your secrets backend. This means that dags in the preview Deployment can access your secret Airflow objects for testing purposes.

This template makes use of the `AIRFLOW__SECRETS__BACKEND_KWARGS` environment variable to store information and credentials for your secrets backend.

### Prerequisites

* A [secrets backend](/docs/astro/secrets-backend), such as Hashicorp Vault.

### Setup

1. Copy and save the Deployment ID for your Astro deployment.

<Info>Replace `<main-deployment-id>` with this Deployment ID in all the scripts created in the following steps. Even though some scripts take action on the preview Deployment, the `<main-deployment-id>` should be same for each script.</Info>

2. Set the following [GitHub secrets](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository) in the repository hosting your Astro project. This includes your Astro API Token, so that GitHub has permissions to deploy code to your Deployments or Workspaces, and your secrets backend information stored in `AIRFLOW__SECRETS__BACKEND_KWARGS`. See [Configure a secrets backend](/docs/astro/secrets-backend) for more information about configuring your secrets backend as an environment variable.

* **Key 1**: `ASTRO_API_TOKEN`
* **Secret 1**: `<your-token>`
* **Key 2**: `AIRFLOW__SECRETS__BACKEND_KWARGS`
* **Secret 2**: `<your-kwargs>`

3. In your project repository, create a new YAML file in `.github/workflows` named `create-deployment-preview.yml` that includes the following configuration.

```yaml title="create-deployment-preview.yml" wrap theme={null}
name: Astronomer CI - Create preview Deployment with Secrets Backend

on:
  create:
    branches:
      - "**"

env:
  ## Sets Deployment API token credentials as environment variables
  ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Install Astro CLI
      uses: astronomer/setup-astro-cli@v0.0.1
    - name: Create Deployment Preview
      uses: astronomer/deploy-action@v0.14.0
      id: create-dep-prev
      with:
        action: create-deployment-preview
        deployment-name: "test"
    - name: Create Secret Variables
      run: |
        astro deployment variable update --deployment-id ${{ steps.create-dep-prev.outputs.preview-id }} AIRFLOW__SECRETS__BACKEND_KWARGS=${{ secrets.AIRFLOW__SECRETS__BACKEND_KWARGS }} --secret
```

4. In the same folder, create a new YAML file named `deploy-to-preview.yml` that includes the following configuration:

```yaml title="deploy-to-preview.yml" wrap theme={null}
name: Astronomer CI - Deploy code to preview

on:
  pull_request:
    branches:
      - main

env:
  ## Set your API token as a GitHub secret
  ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Install Astro CLI
      uses: astronomer/setup-astro-cli@v0.0.1
    - name: Deploy code to preview
      uses: astronomer/deploy-action@v0.14.0
      with:
        action: deploy-deployment-preview
        deployment-id: <main-deployment-id>
```

5. In the same folder, create a new YAML file named `delete-preview-deployment.yml` that includes the following configuration:

```yaml title="delete-preview-deployment.yml" wrap theme={null}
name: Astronomer CI - Delete Preview Deployment

on:
  delete:
    branches:
      - "**"
env:
  ## Set your API token as a GitHub secret
  ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Install Astro CLI
      uses: astronomer/setup-astro-cli@v0.0.1
    - name: Delete preview Deployment
      uses: astronomer/deploy-action@v0.14.0
      with:
        action: delete-deployment-preview
        deployment-id: <main-deployment-id>
```

6. In the same folder, create a new YAML file named `deploy-to-main-deployment.yml` that includes the following configuration:

```yaml title="deploy-to-main-deployment.yml" wrap theme={null}
name: Astronomer CI - Deploy code to main Deployment

on:
  push:
    branches:
      - main

env:
  ## Set your API token as a GitHub secret
  ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Install Astro CLI
      uses: astronomer/setup-astro-cli@v0.0.1
    - name: Deploy code to main Deployment
      uses: astronomer/deploy-action@v0.14.0
      with:
        deployment-id: <main-deployment-id>
```

All four workflow files must have the same Deployment ID specified. The actions use this Deployment ID to create and delete preview Deployments based on your main Deployment.


# GitHub Actions templates for deploying to Astro from private networks
Source: https://astronomer.io/docs/astro/ci-cd-templates/github-actions-private-network

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro on private networks using GitHub Actions.

<Tip>The Astro GitHub integration can automatically deploy code from a GitHub repository to Astro without you needing to configure a GitHub action. In addition, the Astro UI shows Git metadata for each deploy on your Deployment information screen. See [Deploy code with the Astro GitHub integration](/docs/astro/deploy-github-integration) for setup steps.</Tip>

If you don't have access to the Astronomer [deploy action](https://github.com/astronomer/deploy-action) because you can't access the public internet from your GitHub repository, use one of the following private network templates to deploy to Astro.

Read the following sections to choose the right template for your use case. If you have one Deployment and one environment on Astro, use the [single branch implementation](/docs/astro/ci-cd-templates/template-overview#single-branch-implementation). If you have multiple Deployments that support development and production environments, use the [multiple branch implementation](/docs/astro/ci-cd-templates/template-overview#multiple-branch-implementation). If your team builds custom Docker images, use the [custom image implementation](/docs/astro/ci-cd-templates/template-overview#custom-image-implementation). If you want to deploy a prebuilt image from an artifact registry without rebuilding, use the [prebuilt image implementation](/docs/astro/deploy-project-image#deploy-a-prebuilt-docker-image).

You can configure your CI/CD pipelines to deploy a full project image or your `dags` directory. To learn more about CI/CD on Astro, see [Choose a CI/CD strategy](/docs/astro/set-up-ci-cd).

<Warning>
  If you use a [self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners) to execute jobs from GitHub Actions, the Astro CLI's `config.yaml` file, which stores default deploy details, might be shared across your organization and hence multiple CI/CD pipelines. To reduce the risk of accidentally deploying to the wrong Deployment, ensure the following:

  * Add `ASTRO_API_TOKEN` to your repository and include a check in your GitHub workflow to verify that it exists.
  * Use Deployment API tokens, which are scoped only to one Deployment, instead of Workspace or Organization API tokens.
  * Specify `deployment-id` or `deployment-name` in your action. For example, `astro deploy <deployment-id>` or `astro deploy -n <deployment-name>`.
  * Add the command `astro logout` at the end of your workflow to ensure that your authentication token is cleared from the `config.yaml` file.
</Warning>

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) hosted in a GitHub repository.
* An [Astro Deployment](/docs/astro/create-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).
* Access to [GitHub Actions](https://github.com/features/actions).

## Setup

<Tabs>
  <Tab title="Single branch">
    To automate code deploys to a Deployment using [GitHub Actions](https://github.com/features/actions), complete the following setup in a Git-based repository that hosts an Astro project:

    1. Set the following as [GitHub secrets](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository):

    * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.

    2. In your project repository, create a new YAML file in `.github/workflows` that includes the following configuration. When you make a commit to a specified branch, this workflow sets your Deployment API credentials as environment variables, installs the latest version of the Astro CLI, checks to see if your `dags` folder has changes, and then either completes a full code deploy or a dag-only code deploy.

    ```yaml expandable wrap theme={null}
    name: Astronomer CI - Deploy code

    on:
      push:
        branches:
          - main

    env:
      ## Sets Deployment API credentials as environment variables
      ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}

    jobs:
      build:
        runs-on: ubuntu-latest # add the appropriate image
        steps:
            # Install the Astro CLI (current version)
            - name: checkout repo
              uses: actions/checkout@v3
              with:
                fetch-depth: 2
                clean: false
            - name: Install the CLI
              run: curl -sSL install.astronomer.io | sudo bash -s
            # Determine if only dag files have changes
            - name: Deploy to Astronomer
              run: |
                files=$(git diff --name-only $(git rev-parse HEAD~1) -- .)
                dags_only=1
                for file in $files; do
                if [[ $file != dags/* ]]; then
                    echo "$file is not a dag, triggering a full image build"
                    dags_only=0
                    break
                fi
                done
                ### If only dags changed deploy only the dags in your 'dags' folder to your Deployment
                if [ $dags_only == 1 ]
                then
                    astro deploy --dags
                fi
                ### If any other files changed build your Astro project into a Docker image, push the image to your Deployment, and then push and dag changes
                if [ $dags_only == 0 ]
                then
                    astro deploy
                fi
    ```
  </Tab>

  <Tab title="Multiple branch">
    The following setup can be used to create a multiple branch CI/CD pipeline using GitHub Actions to push a [full image deploy](/docs/astro/deploy-project-image) to Astro. A multiple branch pipeline can be used to test dags in a development Deployment and promote them to a production Deployment.

    #### Prerequisites (Multiple branch)

    * You have both a `dev` and `main` branch of an Astro project hosted in a single GitHub repository.
    * You have respective `dev` and `prod` Deployments on Astro where you deploy your GitHub branches to.
    * You have at least one API token with access to both of your Deployments.

    #### Setup (Multiple branch)

    1. Set the following as [GitHub secrets](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository):

    * `PROD_ASTRO_API_TOKEN`: The value for your production Workspace or Organization API token.
    * `DEV_ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.

    2. In your project repository, create a new YAML file in `.github/workflows` that includes the following configuration:

    ```yaml expandable wrap theme={null}
    name: Astronomer CI - Deploy code (Multiple Branches)

    on:
      push:
        branches: [dev]
      pull_request:
        types:
          - closed
        branches: [main]

    jobs:
      dev-push:
        if: github.ref == 'refs/heads/dev'
        env:
          ## Sets DEV Deployment API token credential as an environment variable
          ASTRO_API_TOKEN: ${{ secrets.DEV_ASTRO_API_TOKEN }}
        runs-on: ubuntu-latest
        steps:
        - name: checkout repo
          uses: actions/checkout@v3
        - name: Deploy to Astro
          run: |
            curl -sSL install.astronomer.io | sudo bash -s
            astro deploy <your-dev-deployment-id>
      prod-push:
        if: github.event.action == 'closed' && github.event.pull_request.merged == true
        env:
          ## Sets PROD Deployment API token credential as an environment variable
          ASTRO_API_TOKEN: ${{ secrets.PROD_ASTRO_API_TOKEN }}
        runs-on: ubuntu-latest
        steps:
        - name: checkout repo
          uses: actions/checkout@v3
        - name: Deploy to Astro
          run: |
            curl -sSL install.astronomer.io | sudo bash -s
            astro deploy <your-prod-deployment-id>
    ```
  </Tab>

  <Tab title="Custom Image">
    If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker's [`build-push-action`](https://github.com/docker/build-push-action). This template always pushes your entire project [as an image](/docs/astro/deploy-project-image) to Astro.

    #### Prerequisites (Custom Image)

    * An Astro project that requires additional build-time arguments to build the Runtime image.

    #### Setup (Custom Image)

    1. Set the following as [GitHub secrets](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository):

    * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.

    2. In your project repository, create a new YAML file in `.github/workflows` that includes the following configuration:

    ```yaml expandable wrap theme={null}
    name: Astronomer CI - Additional build-time args

    on:
      push:
        branches:
          - main

    jobs:
      build:
        runs-on: ubuntu-latest
        env:
          ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
        steps:
        - name: Check out the repo
          uses: actions/checkout@v3
        - name: Create image tag
          id: image_tag
          run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
        - name: Build image
          uses: docker/build-push-action@v4
          with:
            tags: ${{ steps.image_tag.outputs.image_tag }}
            load: true
            # Define your custom image's build arguments, contexts, and connections here using
            # the available GitHub Action settings:
            # https://github.com/docker/build-push-action#customizing .
            # This example uses `build-args` , but your use case might require configuring
            # different values.
            build-args: |
              <your-build-arguments>
        - name: Deploy to Astro
          run: |
            curl -sSL install.astronomer.io | sudo bash -s
            astro deploy <your-deployment-id> --image-name ${{ steps.image_tag.outputs.image_tag }}
    ```

    For example, to create a CI/CD pipeline that deploys a project which [installs Python packages from a private GitHub repository](/docs/cli/v1.43/private-python-packages), you would use the following configuration:

    ```yaml expandable wrap theme={null}
    name: Astronomer CI - Custom base image

    on:
      push:
        branches:
          - main

    jobs:
      build:
        runs-on: ubuntu-latest
        env:
          ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
        steps:
        - name: Check out the repo
          uses: actions/checkout@v3
        - name: Create image tag
          id: image_tag
          run: echo ::set-output name=image_tag::astro-$(date +%Y%m%d%H%M%S)
        - name: Create SSH Socket
          uses: webfactory/ssh-agent@v0.5.4
          with:
            # GITHUB_SSH_KEY must be defined as a GitHub secret.
            ssh-private-key: ${{ secrets.GITHUB_SSH_KEY }}
        - name: (Optional) Test SSH Connection - Should print hello message.
          run: (ssh git@github.com) || true
        - name: Build image
          uses: docker/build-push-action@v2
          with:
            tags: ${{ steps.image_tag.outputs.image_tag }}
            load: true
            ssh: |
              github=${{ env.SSH_AUTH_SOCK }}
        - name: Deploy to Astro
          run: |
            curl -sSL install.astronomer.io | sudo bash -s
            astro deploy <your-deployment-id> --image-name ${{ steps.image_tag.outputs.image_tag }}
    ```

    <Info>If you need guidance configuring a CI/CD pipeline for a more complex use case involving custom Runtime images, reach out to [Astronomer support](https://support.astronomer.io/).</Info>
  </Tab>

  <Tab title="Prebuilt image">
    If your organization builds Astro Runtime images in a separate build pipeline and stores them in an artifact registry, use this template to pull and deploy the prebuilt image without rebuilding it. This template skips the build step entirely and deploys the image directly from your registry to Astro.

    #### Prerequisites (Prebuilt image)

    * A prebuilt Docker image based on Astro Runtime, stored in an artifact registry such as Google Artifact Registry, Amazon ECR, Azure Container Registry, or Docker Hub.
    * Registry credentials with pull access to the image.

    #### Setup (Prebuilt image)

    1. Set the following as [GitHub secrets](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository):

    * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
    * Registry credentials for your artifact registry. See the following tabs for registry-specific secrets.

    2. In your project repository, create a new YAML file in `.github/workflows` that includes the following configuration for your registry:

    <Tabs>
      <Tab title="Google Artifact Registry">
        ```yaml wrap theme={null}
        name: Astronomer CI - Deploy prebuilt image from Google Artifact Registry

        on:
          push:
            branches:
              - main

        env:
          ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
          REGISTRY: <your-region>-docker.pkg.dev/<your-project-id>/<your-repository>
          IMAGE_NAME: <your-image-name>
          IMAGE_TAG: <your-image-tag>

        jobs:
          deploy:
            runs-on: ubuntu-latest
            steps:
            - name: Authenticate to Google Cloud
              uses: google-github-actions/auth@v2
              with:
                credentials_json: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
            - name: Configure Docker for Google Artifact Registry
              run: gcloud auth configure-docker <your-region>-docker.pkg.dev --quiet
            - name: Pull image from registry
              run: docker pull ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
            - name: Install the Astro CLI
              uses: astronomer/setup-astro-cli@v0.0.1
            - name: Deploy to Astro
              run: |
                astro deploy <your-deployment-id> --image-name ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
        ```

        Set `GCP_SERVICE_ACCOUNT_KEY` as a GitHub secret containing the JSON key for a Google Cloud service account with read access to the registry.
      </Tab>

      <Tab title="Amazon ECR">
        ```yaml expandable wrap theme={null}
        name: Astronomer CI - Deploy prebuilt image from Amazon ECR

        on:
          push:
            branches:
              - main

        env:
          ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
          REGISTRY: <your-account-id>.dkr.ecr.<your-region>.amazonaws.com
          IMAGE_NAME: <your-image-name>
          IMAGE_TAG: <your-image-tag>

        jobs:
          deploy:
            runs-on: ubuntu-latest
            steps:
            - name: Configure AWS credentials
              uses: aws-actions/configure-aws-credentials@v4
              with:
                aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
                aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
                aws-region: <your-region>
            - name: Authenticate to Amazon ECR
              uses: aws-actions/amazon-ecr-login@v2
            - name: Pull image from registry
              run: docker pull ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
            - name: Install the Astro CLI
              uses: astronomer/setup-astro-cli@v0.0.1
            - name: Deploy to Astro
              run: |
                astro deploy <your-deployment-id> --image-name ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
        ```

        Set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` as GitHub secrets with permissions to pull from the ECR repository.
      </Tab>

      <Tab title="Azure Container Registry">
        ```yaml wrap theme={null}
        name: Astronomer CI - Deploy prebuilt image from Azure Container Registry

        on:
          push:
            branches:
              - main

        env:
          ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
          REGISTRY: <your-registry-name>.azurecr.io
          IMAGE_NAME: <your-image-name>
          IMAGE_TAG: <your-image-tag>

        jobs:
          deploy:
            runs-on: ubuntu-latest
            steps:
            - name: Authenticate to Azure Container Registry
              uses: docker/login-action@v3
              with:
                registry: ${{ env.REGISTRY }}
                username: ${{ secrets.ACR_USERNAME }}
                password: ${{ secrets.ACR_PASSWORD }}
            - name: Pull image from registry
              run: docker pull ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
            - name: Install the Astro CLI
              uses: astronomer/setup-astro-cli@v0.0.1
            - name: Deploy to Astro
              run: |
                astro deploy <your-deployment-id> --image-name ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
        ```

        Set `ACR_USERNAME` and `ACR_PASSWORD` as GitHub secrets. You can use a service principal or an admin account with pull access to the registry.
      </Tab>

      <Tab title="Docker Hub">
        ```yaml wrap theme={null}
        name: Astronomer CI - Deploy prebuilt image from Docker Hub

        on:
          push:
            branches:
              - main

        env:
          ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
          IMAGE_NAME: <your-dockerhub-org>/<your-image-name>
          IMAGE_TAG: <your-image-tag>

        jobs:
          deploy:
            runs-on: ubuntu-latest
            steps:
            - name: Authenticate to Docker Hub
              uses: docker/login-action@v3
              with:
                username: ${{ secrets.DOCKERHUB_USERNAME }}
                password: ${{ secrets.DOCKERHUB_TOKEN }}
            - name: Pull image from registry
              run: docker pull ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
            - name: Install the Astro CLI
              uses: astronomer/setup-astro-cli@v0.0.1
            - name: Deploy to Astro
              run: |
                astro deploy <your-deployment-id> --image-name ${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}
        ```

        Set `DOCKERHUB_USERNAME` and `DOCKERHUB_TOKEN` as GitHub secrets with pull access to the repository.
      </Tab>
    </Tabs>

    These workflows authenticate to the registry, pull the prebuilt image, and deploy it to Astro using `--image-name`. No build step occurs. The image must be based on Astro Runtime.

    To learn more about deploying prebuilt images, see [Deploy a prebuilt Docker image](/docs/astro/deploy-project-image#deploy-a-prebuilt-docker-image).
  </Tab>
</Tabs>


# GitHub Actions templates for deploying code to Astro
Source: https://astronomer.io/docs/astro/ci-cd-templates/github-actions-template

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using GitHub Actions.

<Tip>The Astro GitHub integration can automatically deploy code from a GitHub repository to Astro without you needing to configure a GitHub action. In addition, the Astro UI shows Git metadata for each deploy on your Deployment information screen. See [Deploy code with the Astro GitHub integration](/docs/astro/deploy-github-integration) for setup steps.</Tip>

GitHub Action templates use the Astronomer-maintained `deploy-action`, which is available in the [GitHub Marketplace](https://github.com/marketplace/actions/deploy-apache-airflow-dags-to-astro). This action automates the deploy process and includes additional features for more complex automation workflows. Specifically, the action can automatically:

* Choose a deploy type based on the files that were changed in a commit. This allows you to use the same template for dag deploys and image deploys.
* Test dags as part of the deploy process and prevent deploying if any of the tests fail. These tests are defined in the `tests` directory of your Astro project.
* Create a [preview Deployment](/docs/astro/ci-cd-templates/github-actions-deployment-preview) to test your code before deploying to production. A Deployment preview is an Astro Deployment that mirrors the configuration of an existing Deployment.
* Allows you to choose the type of code deploys used by the automation:
  * (Default) `infer`
  * `image-and-dags`
  * `image-only`
  * `dags-only`
  * `dbt`

If you have one Deployment and one environment on Astro, use the [single branch implementation](/docs/astro/ci-cd-templates/template-overview#single-branch-implementation). If you have multiple Deployments that support development and production environments, use the [multiple branch implementation](/docs/astro/ci-cd-templates/template-overview#multiple-branch-implementation). If your team builds custom Docker images, use the [custom image implementation](/docs/astro/ci-cd-templates/template-overview#custom-image-implementation). If you don't have access to Astronomer's `deploy-action`, use the [private network templates](/docs/astro/ci-cd-templates/github-actions-private-network).

To learn more about CI/CD on Astro, see [Choose a CI/CD strategy](/docs/astro/set-up-ci-cd).

<Info>If you use GitHub Enterprise and cannot access the Astronomer Deploy Action, see [Private network templates](/docs/astro/ci-cd-templates/github-actions-private-network).</Info>

## Deploy type

The `deploy-action` includes several deploy types for you to choose a specific type of code deploy for your CI/CD processes.

<Info>See the [Deploy Action README](https://github.com/astronomer/deploy-action#readme) to learn more about using and customizing this action, like creating a GitHub action that can support dag and dbt deploys.</Info>

### (Default) infer

By default, the `deploy-action` uses `infer`, which allows it to determine the type of code deploy to use based on the types of files changed in a commit: Dag or Astro project. If you committed changes only to dag files, the action triggers a dag deploy. If you committed changes to any other file, the action triggers an image deploy, or `image-and-dags`. This setting does not include `dbt` deploy types. See [default deploy action](/docs/astro/ci-cd-templates/default-deploy-action).

### image-and-dags

The `image-and-dags` option enables the Deploy Action to make a full project deploy, which includes both images and dags. This option will also include code in `includes/`, `plugins/`, or any other directories in the root of the repository. For example, if you have a dbt project in a `dbt/` directory, it will be included when using this Deploy Action.

### image-only

The `image-only` option enables the Deploy Action to deploy only the Docker image of your Astro project without updating existing dags on your Deployment. Use this option when you want to push dependency or configuration changes independently of your dag code. This deploy type skips dag parsing and pytest validation. Requires Astro CLI version 1.21.0 or later.

### dags-only

The `dags-only` deploy option enables the deploy action to deploy only the dags in your Astro project's `dags` directory to your Deployment.

### dbt

The `dbt` deploy option enables the Deploy Action to deploy dbt projects to Astro, when you provide the path to a directory in your GitHub repo that contains your dbt project. See [dbt deploy action](/docs/astro/ci-cd-templates/dbt-deploy-action). This is most commonly used when your dbt code/logic does not live in the same repository as your Astro project.

## Monorepo support

If your Astro project lives in a monorepo, use the following inputs to control how the action checks out your repository and determines whether to trigger a deploy:

* `sparse-checkout`: Limits the GitHub checkout to specific folders, typically the same value as `root-folder`. Use this input to reduce checkout size in large monorepos.
* `image-trigger-paths`: Lists paths outside `root-folder` that also trigger a deploy when changed, for example generated files that get copied into the image at build time. Applies to the `infer`, `image-and-dags`, and `image-only` deploy types.
* `skip-unchanged`: Set to `false` to always run the deploy dictated by `deploy-type`, instead of skipping when no changed files fall under `root-folder` or `image-trigger-paths`. This input replaces the deprecated `deploy-image: true` input.

See [Configuration options](https://github.com/astronomer/deploy-action#configuration-options) in the Deploy Action README for full details and examples.


# Astro CI/CD templates for GitLab
Source: https://astronomer.io/docs/astro/ci-cd-templates/gitlab

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using GitLab.

Use the following CI/CD templates to automate deploys to Astro from a [GitLab](https://gitlab.com/) repository.

Read the following sections to choose the right template for your project. The templates for GitLab include the image deploy templates and dag deploy templates.

If you have one Deployment and one environment on Astro, use the [single branch implementation](#single-branch). If you have multiple Deployments that support development and production environments, use the [multiple branch implementation](#multiple-branch). If you want your CI/CD process to automatically decide which deploy strategy to choose, see [Dag deploy templates](#dag-deploy-templates).

To learn more about CI/CD on Astro, see [Choose a CI/CD strategy](/docs/astro/set-up-ci-cd).

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) hosted in a GitLab repository.
* An [Astro Deployment](/docs/astro/create-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).

Each CI/CD template implementation might have additional requirements.

## Image deploy templates

<Tabs>
  <Tab title="Single branch">
    Use this template to push code from a GitLab repository to Astro.

    1. Set the following [environment variables](https://docs.gitlab.com/ee/ci/variables/#for-a-project) in your GitLab project:

    * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
    * `DEPLOYMENT_ID`: The ID of your Astro Deployment. You can copy the **ID** from your Deployment's home page in the Astro UI.

    Astronomer recommends that you always [mask](https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable) your API token to prevent it from being accessible in plain text. You can also set the API token as an [external secret](https://docs.gitlab.com/ee/ci/secrets/index.html) for an extra layer of security.

    2. Go to **Build** > **Pipeline Editor** and commit the following:

    ```yaml title=".gitlab-ci.yml" wrap theme={null}
    astro_deploy:
      stage: deploy
      image: docker:latest
      services:
        - docker:dind

      variables:
        ASTRO_API_TOKEN: ${ASTRO_API_TOKEN}
        DEPLOYMENT_ID: ${DEPLOYMENT_ID}

      before_script:
        - apk add --update curl && rm -rf /var/cache/apk/*
        - apk add bash
      script:
        - (curl -sSL install.astronomer.io | bash -s)
        - astro deploy -f $DEPLOYMENT_ID
      only:
        - main
    ```
  </Tab>

  <Tab title="Multiple branch">
    Use this template to push code to a development and a production Deployment in Astro based on your GitLab project's branch name.

    1. Set the following [environment variables](https://docs.gitlab.com/ee/ci/variables/#for-a-project) in your GitLab project:

    * `PROD_ASTRO_API_TOKEN`: The value of your production Workspace or Organization API token.
    * `PROD_DEPLOYMENT_ID`: The ID of your Astro Deployment. You can copy the **ID** from your production Deployment's page in the Astro UI.
    * `DEV_ASTRO_API_TOKEN`: The value of your development Workspace or Organization API token.
    * `DEV_DEPLOYMENT_ID`: The ID of your Astro Deployment. You can copy the **ID** from your development Deployment's page in the Astro UI.

    Astronomer recommends that you always [mask](https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable) your API token to prevent it from being accessible in plain text. You can also set the API token as an [external secret](https://docs.gitlab.com/ee/ci/secrets/index.html) for an extra layer of security.

    <Tip>When you create a CI/CD variable that will be used in multiple branches, you might want to [protect the variable](https://docs.gitlab.com/ee/ci/variables/#protect-a-cicd-variable) so that it can only be accessed from the relevant branches.</Tip>

    2. Go to the **Editor** option in your project's CI/CD section and commit the following:

    ```yaml title=".gitlab-ci.yml" expandable wrap theme={null}
    astro_deploy_dev:
      stage: deploy
      image: docker:latest
      services:
        - docker:dind
      variables:
        ASTRO_API_TOKEN: ${DEV_ASTRO_API_TOKEN}
        DEPLOYMENT_ID: ${DEV_DEPLOYMENT_ID}
      before_script:
        - apk add --update curl && rm -rf /var/cache/apk/*
        - apk add bash
      script:
        - (curl -sSL install.astronomer.io | bash -s)
        - astro deploy -f $DEPLOYMENT_ID
      only:
        - dev

    astro_deploy_prod:
      stage: deploy
      image: docker:latest
      services:
        - docker:dind
      variables:
        ASTRO_API_TOKEN: ${PROD_ASTRO_API_TOKEN}
        DEPLOYMENT_ID: ${PROD_DEPLOYMENT_ID}
      before_script:
        - apk add --update curl && rm -rf /var/cache/apk/*
        - apk add bash
        - apk add jq
      script:
        - (curl -sSL install.astronomer.io | bash -s)
        - astro deploy -f $DEPLOYMENT_ID
      only:
        - main
    ```
  </Tab>
</Tabs>

## Dag deploy templates

The dag deploy template uses the `--dags` flag in the Astro CLI to push dag changes to Astro. These CI/CD pipelines deploy your dags only when files in your `dags` folder are modified, and they deploy the rest of your Astro project as a Docker image when other files or directories are modified. For more information about the benefits of this workflow, see [Deploy dags only](/docs/astro/deploy-code).

<Info>
  If you stage multiple commits to dag files and push them all at once to your remote branch, the template only deploys dag code changes from the most recent commit. It will miss any code changes made in previous commits.

  To avoid this, either push commits individually or configure your repository to **Squash commits** for pull requests that merge multiple commits simultaneously.
</Info>

### Single branch implementation

Use this template to push code from a GitLab repository to Astro.

1. Set the following [environment variables](https://docs.gitlab.com/ee/ci/variables/#for-a-project) in your GitLab project:

* `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
* `DEPLOYMENT_ID`: The ID of your Astro Deployment. You can copy the **ID** from your Deployment's page in the Astro UI.

Astronomer recommends that you always [mask](https://docs.gitlab.com/ee/ci/variables/#mask-a-cicd-variable) your API token to prevent it from being accessible in plain text. You can also set the API token as an [external secret](https://docs.gitlab.com/ee/ci/secrets/index.html) for an extra layer of security.

2. Go to the **Editor** option in your project's CI/CD section and commit the following:

```yaml title=".gitlab-ci.yml" expandable wrap theme={null}
astro_smart_deploy:
  stage: deploy
  image: docker:latest
  services:
    - docker:dind
  variables:
    ASTRO_API_TOKEN: ${ASTRO_API_TOKEN}
    DAG_FOLDER: "dags"
    DEPLOYMENT_ID: ${DEPLOYMENT_ID}
  before_script:
    - apk add --update curl && rm -rf /var/cache/apk/*
    - apk add git
    - apk add bash
  script:
    - (curl -sSL install.astronomer.io | bash -s)
    - files=$(git diff --name-only $(git rev-parse HEAD~1) -- .)
    - dags_only=1
    - echo "$DAG_FOLDER"
    - echo "$files"
    - for file in $files; do
    -   echo "$file"
    -   if [[ "$file" != "$DAG_FOLDER"* ]]; then
    -     echo "$file is not a dag, triggering a full image build"
    -     dags_only=0
    -     break
    -   else
    -     echo "just a dag"
    -   fi
    - done
    - if [[ $dags_only == 1 ]]; then
    -   echo "doing dag-only deploy"
    -   astro deploy --dags $DEPLOYMENT_ID
    - elif [[ $dags_only == 0 ]]; then
    -   echo "doing image deploy"
    -   astro deploy -f $DEPLOYMENT_ID
    - fi
  only:
    - main
```


# Astro CI/CD templates for Harness
Source: https://astronomer.io/docs/astro/ci-cd-templates/harness

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using Harness.

Use the following CI/CD template to automate a deploy with a [single branch implementation](/docs/astro/ci-cd-templates/template-overview#single-branch-implementation), which requires only one Astro Deployment, from a Git repository to Astro with [Harness](https://harness.org/product).

If you use the [dag-only deploy feature](/docs/astro/deploy-dags) on Astro or you're interested in a multiple-branch implementation, see [Template overview](/docs/astro/ci-cd-templates/template-overview#multiple-branch-implementation) to configure your own. To learn more about CI/CD on Astro, see [Choose a CI/CD strategy](/docs/astro/set-up-ci-cd).

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) hosted in a Git repository that Harness can access.
* An [Astro Deployment](/docs/astro/create-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).
* Access to [Harness](https://www.harness.io/).

## Deploy template

1. [Create a Harness CI pipeline](https://developer.harness.io/docs/continuous-integration/use-ci/prep-ci-pipeline-components/) and configure it as **Remote** with a **Third-party Git provider**.

2. Authenticate to your Git provider.

3. Choose **Cloud** for infrastructure, and **Linux** and **ARM64** for platform. If you want to use other infrastructure, see [Harness’s guide for implementation](https://developer.harness.io/docs/platform/delegates/delegate-concepts/delegate-overview/).

4. Add a step, then choose **Run** as the template type from the step library. Use `sh` for your shell.  Only one step is needed to deploy code from your Git repository to Astro, but you can add more if your use case requires.

5. Paste the following code into the command prompt and apply changes.

```bash wrap expandable theme={null}
#!/bin/bash
set -ex
# Prerequisites: Set variables
ORGANIZATION_ID=$ASTRO_ORG_ID
DEPLOYMENT_ID=$ASTRO_DEPLOYMENT_ID
ASTRO_API_TOKEN=$ASTRO_API_TOKEN
ASTRO_PROJECT_PATH=$ASTRO_PROJECT_PATH
# Step 1: Initialize deploy
echo -e "Initiating Deploy Process for deployment $DEPLOYMENT_ID\n"
CREATE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/platform/v1beta1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys" \
--header "X-Astro-Client-Identifier: script" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $ASTRO_API_TOKEN" \
--data '{
"type": "IMAGE_AND_DAG"
}' | jq '.')
DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id')
REPOSITORY=$(echo $CREATE_DEPLOY | jq -r '.imageRepository')
TAG=$(echo $CREATE_DEPLOY | jq -r '.imageTag')
DAGS_UPLOAD_URL=$(echo $CREATE_DEPLOY | jq -r '.dagsUploadUrl')
# Step 2: Log in to Docker
docker login $REPOSITORY -u cli -p $ASTRO_API_TOKEN
echo -e "\nBuilding Docker image $REPOSITORY:$TAG for $DEPLOYMENT_ID from $ASTRO_PROJECT_PATH"
# Step 3: Build image
docker build -t $REPOSITORY:$TAG --platform=linux/amd64 $ASTRO_PROJECT_PATH
# Step 4: Push image
echo -e "\nPushing Docker image $REPOSITORY:$TAG to $DEPLOYMENT_ID"
docker push $REPOSITORY:$TAG
# Step 5: Create tar file wd
echo -e "\nCreating a dags tar file from $ASTRO_PROJECT_PATH/dags and stored in $ASTRO_PROJECT_PATH/dags.tar\n"
cd $ASTRO_PROJECT_PATH
tar -cvf "$ASTRO_PROJECT_PATH/dags.tar" "dags"
# Step 6: Upload dags tar file
echo -e "\nUploading tar file $ASTRO_PROJECT_PATH/dags.tar\n"
VERSION_ID=$(curl -i --request PUT $DAGS_UPLOAD_URL \
--header 'x-ms-blob-type: BlockBlob' \
--header 'Content-Type: application/x-tar' \
--upload-file "$ASTRO_PROJECT_PATH/dags.tar" | grep x-ms-version-id | awk -F': ' '{print $2}')
VERSION_ID=$(echo $VERSION_ID | sed 's/\r//g') # Remove unexpected carriage return characters
echo -e "\nTar file uploaded with version: $VERSION_ID\n"
# Step 7: Finalizing Deploy
FINALIZE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/platform/v1beta1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys/$DEPLOY_ID/finalize" \
--header "X-Astro-Client-Identifier: script" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $ASTRO_API_TOKEN" \
--data '{"dagTarballVersion": "'$VERSION_ID'"}')
ID=$(echo $FINALIZE_DEPLOY | jq -r '.id')
if [[ "$ID" != null ]]; then
       echo -e "\nDeploy is Finalized. Image and dag changes for deployment $DEPLOYMENT_ID should be live in a few minutes"
       echo "Deployed Image tag: $TAG"
       echo "Deployed dag Tarball Version: $VERSION_ID"
else
       MESSAGE=$(echo $FINALIZE_DEPLOY | jq -r '.message')
       if  [[ "$MESSAGE" != null ]]; then
               echo $MESSAGE
       else
               echo "Something went wrong. Reach out to astronomer support for assistance"
       fi
fi
# Cleanup
echo -e "\nCleaning up the created tar file from $ASTRO_PROJECT_PATH/dags.tar"
rm -rf "$ASTRO_PROJECT_PATH/dags.tar"
```

6. Add the following environment variables in Harness:

`ORGANIZATION_ID`: The ID for your Organization.
`DEPLOYMENT_ID`: The ID for your Deployment.
`ASTRO_API_TOKEN`: The value for your API token as a secret.
`ASTRO_PROJECT_PATH`: The default value is `/harness/` and is followed by the folder name used in the Git provider for Astro, if applicable.

7. Run the pipeline. Harness requires you to save changes before executing. Logs will display any errors.


# Astro CI/CD templates for Jenkins
Source: https://astronomer.io/docs/astro/ci-cd-templates/jenkins

Use pre-built Astronomer CI/CD templates to automate deploying Apache Airflow dags to Astro using Jenkins.

Use the following CI/CD templates to automate deploying Apache Airflow dags from a Git repository to Astro with [Jenkins](https://www.jenkins.io/).

The following templates for Jenkins are available:

* [Image deploy templates](/docs/astro/ci-cd-templates/template-overview#image-deploy-templates)
* [dag deploy templates](/docs/astro/ci-cd-templates/template-overview#dag-deploy-templates)

Each template type supports multiple implementations. If you have one Deployment and one environment on Astro, use the [single branch implementation](/docs/astro/ci-cd-templates/template-overview#single-branch-implementation). If you have multiple Deployments that support development and production environments, use the [multiple branch implementation](/docs/astro/ci-cd-templates/template-overview#multiple-branch-implementation). If your team builds custom Docker images, use the [custom image implementation](/docs/astro/ci-cd-templates/template-overview#custom-image-implementation).

For more information on each template or to configure your own, see [Template overview](/docs/astro/ci-cd-templates/template-overview). To learn more about CI/CD on Astro, see [Choose a CI/CD strategy](/docs/astro/set-up-ci-cd).

## Prerequisites

* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) hosted in a Git repository that Jenkins can access.
* An [Astro Deployment](/docs/astro/create-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens).
* Access to [Jenkins](https://www.jenkins.io/).

Each CI/CD template implementation might have additional requirements.

## Image deploy templates

<Tabs>
  <Tab title="Single branch">
    To automate code deploys to a single Deployment using [Jenkins](https://www.jenkins.io/), complete the following setup in a Git-based repository hosting an Astro project:

    1. In your Jenkins pipeline configuration, add the following environment variables:

       * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
       * `ASTRONOMER_DEPLOYMENT_ID`: The Deployment ID of your production deployment

       To set environment variables in Jenkins, on the Jenkins Dashboard go to **Manage Jenkins** > **Configure System** > **Global Properties** > **Environment Variables** > **Add**. To see Jenkins documentation on environment variables click [here](https://www.jenkins.io/doc/pipeline/tour/environment/)

       Be sure to set the value for your API token as secret.

    2. At the root of your Astro Git repository, add a [Jenkinsfile](https://www.jenkins.io/doc/book/pipeline/jenkinsfile/) that includes the following script:

    ```groovy title="Jenkinsfile" wrap theme={null}
    pipeline {
        agent any
        stages {
            stage('Deploy to Astronomer') {
                when {
                    expression {
                        return env.GIT_BRANCH == "origin/main"
                    }
                }
                steps {
                    checkout scm
                    sh '''
                    curl -LJO https://github.com/astronomer/astro-cli/releases/download/v1.38.0/astro_1.38.0_linux_amd64.tar.gz
                    tar -zxvf astro_1.38.0_linux_amd64.tar.gz astro && rm astro_1.38.0_linux_amd64.tar.gz
                    ./astro deploy env.ASTRONOMER_DEPLOYMENT_ID
                    '''
                }
            }
        }
        post {
            always {
                cleanWs()
            }
        }
    }
    ```

    This `Jenkinsfile` triggers a code push to Astro every time a commit or pull request is merged to the `main` branch of your repository.
  </Tab>

  <Tab title="Multiple branch">
    To automate code deploys across multiple Deployments using [Jenkins](https://www.jenkins.io/), complete the following setup in a Git-based repository hosting an Astro project:

    1. In Jenkins, add the following environment variables:

       * `PROD_ASTRO_API_TOKEN`: The value for your production Workspace or Organization API token.
       * `PROD_DEPLOYMENT_ID`: The Deployment ID of your production Deployment
       * `DEV_ASTRO_API_TOKEN`: The value for your development Workspace or Organization API token.
       * `DEV_DEPLOYMENT_ID`: The Deployment ID of your development Deployment

       To set environment variables in Jenkins, on the Jenkins Dashboard go to **Manage Jenkins** > **Configure System** > **Global Properties** > **Environment Variables** > **Add**. To see Jenkins documentation on environment variables click [here](https://www.jenkins.io/doc/pipeline/tour/environment/)

       Be sure to set the values for your API credentials as secret.

    2. At the root of your Git repository, add a [`Jenkinsfile`](https://www.jenkins.io/doc/book/pipeline/jenkinsfile/) that includes the following script:

    ```groovy title="Jenkinsfile" expandable wrap theme={null}
    pipeline {
        agent any
        stages {
            stage('Set Environment Variables') {
                steps {
                    script {
                        if (env.GIT_BRANCH == 'main') {
                            echo "The git branch is ${env.GIT_BRANCH}";
                            env.ASTRO_API_TOKEN = env.PROD_ASTRO_API_TOKEN;
                            env.ASTRONOMER_DEPLOYMENT_ID = env.PROD_DEPLOYMENT_ID;
                        } else if (env.GIT_BRANCH == 'dev') {
                            echo "The git branch is ${env.GIT_BRANCH}";
                            env.ASTRO_API_TOKEN = env.DEV_ASTRO_API_TOKEN;
                            env.ASTRONOMER_DEPLOYMENT_ID = env.DEV_DEPLOYMENT_ID;
                        } else {
                            echo "This git branch ${env.GIT_BRANCH} is not configured in this pipeline."
                        }
                    }
                }
            }
            stage('Deploy to Astronomer') {
                steps {
                    checkout scm
                    sh '''
                    curl -LJO https://github.com/astronomer/astro-cli/releases/download/v1.38.0/astro_1.38.0_linux_amd64.tar.gz
                    tar -zxvf astro_1.38.0_linux_amd64.tar.gz astro && rm astro_1.38.0_linux_amd64.tar.gz
                    ./astro deploy env.ASTRONOMER_DEPLOYMENT_ID
                    '''
                }
            }
        }
        post {
            always {
                cleanWs()
            }
        }
    }
    ```

    This `Jenkinsfile` triggers a code push to an Astro Deployment every time a commit or pull request is merged to the `dev` or `main` branch of your repository.
  </Tab>

  <Tab title="Custom Image">
    If your Astro project requires additional build-time arguments to build an image, you need to define these build arguments using Docker's [`build-push-action`](https://github.com/docker/build-push-action).

    #### Configuration requirements

    * An Astro project that requires additional build-time arguments to build the Runtime image.

    1. In your Jenkins pipeline configuration, add the following environment variables:

       * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
       * `ASTRONOMER_DEPLOYMENT_ID`: The Deployment ID of your production deployment

       To set environment variables in Jenkins, on the Jenkins Dashboard go to **Manage Jenkins** > **Configure System** > **Global Properties** > **Environment Variables** > **Add**. To see Jenkins documentation on environment variables click [here](https://www.jenkins.io/doc/pipeline/tour/environment/)

       Be sure to set the value for your API token as secret.

    2. At the root of your Astro Git repository, add a [Jenkinsfile](https://www.jenkins.io/doc/book/pipeline/jenkinsfile/) that includes the following script:

    ```groovy title="Jenkinsfile" wrap theme={null}
    pipeline {
            agent any
            stages {
                stage('Deploy to Astronomer') {
                    when {
                        expression {
                            return env.GIT_BRANCH == "origin/main"
                        }
                    }
                    steps {
                        checkout scm
                        sh '''
                        export astro_id=$(date +%Y%m%d%H%M%S)
                        docker build -f Dockerfile --progress=plain --build-arg <your-build-arguments> -t $astro_id .
                        curl -LJO https://github.com/astronomer/astro-cli/releases/download/v1.37.0/astro_1.37.0_linux_amd64.tar.gz
                        tar -zxvf astro_1.37.0_linux_amd64.tar.gz astro && rm astro_1.37.0_linux_amd64.tar.gz
                        ./astro deploy env.ASTRONOMER_DEPLOYMENT_ID --image-name $astro_id
                        '''
                    }
                }
            }
            post {
                always {
                    cleanWs()
                }
            }
        }
    ```

    This `Jenkinsfile` triggers a code push to Astro every time a commit or pull request is merged to the `main` branch of your repository.
  </Tab>
</Tabs>

## Dag deploy templates

The dag deploy template uses the `--dags` flag in the Astro CLI to push dag changes to Astro. These CI/CD pipelines deploy your dags only when files in your `dags` folder are modified, and they deploy the rest of your Astro project as a Docker image when other files or directories are modified. For more information about the benefits of this workflow, see [Deploy dags only](/docs/astro/deploy-dags).

<Info>
  If you stage multiple commits to dag files and push them all at once to your remote branch, the template only deploys dag code changes from the most recent commit. It will miss any code changes made in previous commits.

  To avoid this, either push commits individually or configure your repository to **Squash commits** for pull requests that merge multiple commits simultaneously.
</Info>

### Single branch implementation

Use the following template to implement dag-only deploys to a single Deployment using Jenkins.

1. In your Jenkins pipeline configuration, add the following parameters:

   * `ASTRO_API_TOKEN`: The value for your Workspace or Organization API token.
   * `ASTRONOMER_DEPLOYMENT_ID`: The Deployment ID of your production deployment

   Be sure to set the values for your API token as secret.

2. At the root of your Git repository, add a [`Jenkinsfile`](https://www.jenkins.io/doc/book/pipeline/jenkinsfile/) that includes the following script:

```groovy title="Jenkinsfile" expandable wrap theme={null}
pipeline {
    agent any
    stages {
        stage('Dag Only Deploy to Astronomer') {
            when {
                expression {
                    return env.GIT_BRANCH == "origin/main"
                }
            }
            steps {
                checkout scm
                sh '''
                curl -LJO https://github.com/astronomer/astro-cli/releases/download/v1.38.0/astro_1.38.0_linux_amd64.tar.gz
                tar -zxvf astro_1.38.0_linux_amd64.tar.gz astro && rm astro_1.38.0_linux_amd64.tar.gz
                files=($(git diff-tree HEAD --name-only --no-commit-id))
                find="dags"
                if [[ ${files[*]} =~ (^|[[:space:]])"$find"($|[[:space:]]) && ${#files[@]} -eq 1 ]]; then
                ./astro deploy env.ASTRONOMER_DEPLOYMENT_ID --dags;
                else
                ./astro deploy env.ASTRONOMER_DEPLOYMENT_ID;
                fi
                '''
            }
        }
    }
    post {
        always {
            cleanWs()
        }
    }
}
```


# Automate preview Deployments with any CI/CD tool
Source: https://astronomer.io/docs/astro/ci-cd-templates/preview-deployments

Use shell scripts to automate creating, updating, and deleting preview Deployments on Astro with any CI/CD tool.

A *preview Deployment* is an Astro Deployment that a CI/CD workflow automatically creates and deletes based on feature branches in your Git repository. The workflow creates the preview Deployment when you create a temporary feature branch and deletes it when you delete the branch. Astronomer recommends using preview Deployments if you regularly need to test a small set of dags on Astro before promoting those dags to a base, production Deployment. This lowers the infrastructure cost of Deployments that are dedicated to development and testing.

To implement this feature, you need a CI/CD workflow that:

* Creates the preview Deployment when you create a new branch.
* Deploys code changes to Astro when you make updates in the branch.
* Deletes the preview Deployment when you delete the branch.
* Deploys your changes to your base Deployment after you merge your changes into your main branch.

If you use GitHub Actions as your CI/CD tool, you can find preview Deployment templates as part of the [Astronomer GitHub action in the GitHub Marketplace](https://github.com/astronomer/deploy-action?tab=readme-ov-file#deployment-preview-templates). This GitHub action includes sub-actions for each of these four steps. To learn more, see [GitHub Actions templates for preview Deployments](/docs/astro/ci-cd-templates/github-actions-deployment-preview).

To configure your own automated workflow for preview Deployments with another CI/CD tool, use the following scripts. Each of the following four shell scripts is equivalent to the steps required to implement this feature with [GitHub Actions](/docs/astro/ci-cd-templates/github-actions-deployment-preview).

## Create a preview Deployment

In a Deployment preview CI/CD pipeline, you run this script when you create a feature branch off of the main branch of your Astro project.

```bash wrap theme={null}
# Install the Astro CLI
curl -sSL https://install.astronomer.io | sudo bash -s

# Get preview Deployment name
DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --key configuration.name)"
BRANCH_DEPLOYMENT_NAME=$BRANCH_NAME_$DEPLOYMENT_NAME
BRANCH_DEPLOYMENT_NAME="$BRANCH_DEPLOYMENT_NAME// /_"

# Create template of Deployment to be copied
astro deployment inspect $DEPLOYMENT_ID --template > deployment-preview-template.yaml # automatically creates deployment-preview-template.yaml file

# Add name to Deployment template file
sed -i "s|  name:.*|  name: $BRANCH_DEPLOYMENT_NAME}|g"  deployment-preview-template.yaml

# Create new preview Deployment based on the Deployment template file
astro deployment create --deployment-file deployment-preview-template.yaml

# Get the ID of the new preview Deployment
PREVIEW_DEPLOYMENT_ID="$(astro deployment inspect -n $BRANCH_DEPLOYMENT_NAME --key metadata.deployment_id)"

# Copy connections, Airflow variables, and pools from the source Deployment
astro deployment connection copy --source-id $DEPLOYMENT_ID --target-id $PREVIEW_DEPLOYMENT_ID
astro deployment airflow-variable copy --source-id $DEPLOYMENT_ID --target-id $PREVIEW_DEPLOYMENT_ID
astro deployment pool copy --source-id $DEPLOYMENT_ID --target-id $PREVIEW_DEPLOYMENT_ID

# Deploy new code to the deployment preview
astro deploy -n $BRANCH_DEPLOYMENT_NAME
```

Creating a Deployment from a template does not copy the connections, Airflow variables, or pools from the source Deployment, so the script copies them separately with `astro deployment connection copy`, `astro deployment airflow-variable copy`, and `astro deployment pool copy`.

<Note>
  The copy commands fail if the source Deployment is hibernating. Resume the source Deployment before you run them.
</Note>

## Update a preview Deployment

In a Deployment preview CI/CD pipeline, you run this script whenever you make changes in your feature branch.

```bash wrap theme={null}
# Install the Astro CLI
curl -sSL https://install.astronomer.io | sudo bash -s

# Get preview Deployment name
DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --key configuration.name)"
BRANCH_DEPLOYMENT_NAME=$BRANCH_NAME_$DEPLOYMENT_NAME
BRANCH_DEPLOYMENT_NAME="$BRANCH_DEPLOYMENT_NAME// /_"

# Deploy new code to the preview Deployment
astro deploy -n $BRANCH_DEPLOYMENT_NAME
```

## Delete a preview Deployment

In a Deployment preview CI/CD pipeline, you run this script when you delete your feature branch.

```bash wrap theme={null}
# Install the Astro CLI
curl -sSL https://install.astronomer.io | sudo bash -s

DEPLOYMENT_NAME="$(astro deployment inspect $DEPLOYMENT_ID --key configuration.name)"
BRANCH_DEPLOYMENT_NAME=$BRANCH_NAME_$DEPLOYMENT_NAME
BRANCH_DEPLOYMENT_NAME="$BRANCH_DEPLOYMENT_NAME// /_"

# Delete preview Deployment
astro deployment delete -n $BRANCH_DEPLOYMENT_NAME -f
```

## Deploy changes from a preview Deployment to a base Deployment

In a Deployment preview CI/CD pipeline, you run this script when you merge your feature branch into your main branch.

```bash wrap theme={null}
# Install the Astro CLI
curl -sSL https://install.astronomer.io | sudo bash -s

# Deploy new code to base Deployment
astro deploy $DEPLOYMENT_ID
```


# Template options
Source: https://astronomer.io/docs/astro/ci-cd-templates/template-overview

Use pre-built templates to get started with automating code deploys

Astronomer CI/CD templates are customizable, pre-built code samples that help you configure automated workflows with popular CI/CD tools, such as GitHub Actions or Jenkins. Use the templates to create a workflow that automates deploying code to Astro according to your team's CI/CD requirements and strategy.

Template types differ based on the deploy method they use and how many branches or environments they require. This document contains information about the following template types:

* *Dag deploy templates* that use the [dag-only deploy feature](/docs/astro/deploy-dags) in Astro and either deploy dags or your entire Astro project depending on the files that you update.
* *dbt deploy templates* that use the [dbt deploy feature](/docs/astro/deploy-dags) in Astro to push dbt code from your specified dbt project directory, by default `/dbt`, to make the code accessible in your Deployment.
* *Image deploy templates* that build a Docker image and push it to Astro whenever you update any file in your Astro project, including your dag directory.
* *Preview Deployment templates* that automatically create and delete Deployments when you create or delete a feature branch from your main Astro project branch.

Astronomer maintains a dedicated guide with templates for select CI/CD tools. Most guides include image based templates for a single-branch implementation. Astronomer recommends reconfiguring the templates to work with your own directory structures, tools, and processes.

If you're interested in documentation for a CI/CD tool or template type that does not exist, configure your own or [contact Astronomer support](https://cloud.astronomer.io/open-support-request). To learn more about single-branch and multiple-branch implementations and decide which template is right for you, see [Choose a CI/CD strategy](/docs/astro/set-up-ci-cd).

## Implementation options

CI/CD templates support different implementations based on how many branches and Astro Deployments your team maintains. Choose the implementation that matches your branching strategy, then apply it with one of the deploy templates described in the following sections. For a detailed comparison of environment and repository strategies, see [Develop a CI/CD workflow](/docs/astro/set-up-ci-cd#choose-a-deploy-strategy).

### Single-branch implementation

A single-branch implementation uses one permanent branch, such as `main`, that deploys to a single Astro Deployment. Use this implementation when you have one Deployment and one environment on Astro. See [Single environment](/docs/astro/set-up-ci-cd#single-environment).

### Multiple-branch implementation

A multiple-branch implementation uses multiple permanent branches that each deploy to a separate Astro Deployment, such as a `dev` branch for development and a `main` branch for production. Use this implementation when you maintain multiple Deployments that support development and production environments. See [Multiple environments](/docs/astro/set-up-ci-cd#multiple-environments).

### Custom image implementation

A custom image implementation builds a custom Docker image and deploys it to Astro. Use this implementation when your team builds custom Docker images, for example to install additional dependencies or apply custom build logic. Build the image for the `linux/amd64` platform. Astro rejects images built for any other platform when you deploy them.

## Dag deploy templates

*Dag deploy templates* check the changes in your Astro project and trigger either a dag deploy or image deploy based on the files changed, allowing for faster deploys. This template deploys your dags when only the files in your `dags` folder are modified, and it deploys the rest of your Astro project as a Docker image when any other files or directories are modified. To learn more about the benefits of this workflow, see [Deploy dags](/docs/astro/deploy-dags).

CI/CD templates that use the dag deploy workflow:

* Require that each Deployment have the dag-only deploy feature enabled. See [Enable/disable dag-only deploys on a Deployment](/docs/astro/deploy-dags#enable-or-disable-dag-only-deploys-on-a-deployment).
* Use a [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens). This value must be set using the `ASTRO_API_TOKEN` environment variable.
* Install the latest version of the Astro CLI.
* Trigger the following Astro CLI commands depending on which files were updated by the commit:
  * If only dag files in the `dags` folder have changed, run `astro deploy --dags`. This pushes your `dags` folder to your Deployment.
  * If any file not in the `dags` folder has changed, run `astro deploy`. This triggers two subprocesses. One that creates a Docker image for your Astro project, authenticates to Astro using your Deployment API token, and pushes the image to your Deployment. A second that pushes your `dags` folder to your Deployment.

<Info>
  If you stage multiple commits to dag files and push them all at once to your remote branch, the template only deploys dag code changes from the most recent commit. It will miss any code changes made in previous commits.

  To avoid this, either push commits individually or configure your repository to **Squash commits** for pull requests that merge multiple commits simultaneously.
</Info>

This process is equivalent to the following shell script:

```bash wrap theme={null}
# Set Deployment API token credentials as environment variables
export ASTRO_API_TOKEN="<your-api-token>"
export DAG_FOLDER="<path to dag folder ie. dags/>"
# Install the latest version of Astro CLI
curl -sSL install.astronomer.io | sudo bash -s
# Determine if only dag files have changes
files=$(git diff --name-only $(git rev-parse HEAD~1) -- .)
dags_only=1
for file in $files; do
  if [[ $file != "$DAG_FOLDER"* ]]; then
    echo "$file is not a dag, triggering a full image build"
    dags_only=0
    break
  fi
done
# If only dags changed deploy only the dags in your 'dags' folder to your Deployment
if [ $dags_only == 1 ]
then
    astro deploy --dags
fi
# If any other files changed build your Astro project into a Docker image, push the image to your Deployment, and then push and dag changes
if [ $dags_only == 0 ]
then
    astro deploy
fi
```

## Image deploy templates

*Image based templates* build a Docker image and push it to Astro whenever you update any file in your Astro project. This type of template works well for development workflows that include complex Docker customization or logic.

CI/CD templates that use image based workflows:

* Use a [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens) or [Organization API token](/docs/astro/organization-api-tokens). This value must be set using the `ASTRO_API_TOKEN` environment variable.
* Install the latest version of the Astro CLI.
* Run the `astro deploy` command. This creates a Docker image for your Astro project, authenticates to Astro using your Deployment API token, and pushes the image to your Deployment.

This is equivalent to running the following shell script:

```sh wrap theme={null}
# Set Deployment API token credentials as environment variables
export ASTRO_API_TOKEN="<your-api-token>"
# Install the latest version of Astro CLI
curl -sSL install.astronomer.io | sudo bash -s
# Build your Astro project into a Docker image and push the image to your Deployment
astro deploy <your-deployment-id> -f
```

## dbt deploy templates

*dbt deploy templates* use [dbt deploy](/docs/cli/v1.43/astro-dbt-deploy) to deploy just your dbt code to your Astro project, allowing for faster deploys. Or, it uses the [Deploy Action with GitHub Actions](/docs/astro/ci-cd-templates/dbt-deploy-action) to deploy dbt code. These templates deploy your dbt code when only the files in your dbt project folder, by default `/dbt`, are modified.

CI/CD templates that use image based workflows:

* Use a [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens) or [Organization API token](/docs/astro/organization-api-tokens). This value must be set using the `ASTRO_API_TOKEN` environment variable.
* Install the latest version of the Astro CLI.
* Run the `astro dbt deploy` command from your dbt project directory. This bundles all files in your dbt project and pushes them to Astro, where they are mounted on your Airflow containers so that your dags can access them.

This is equivalent to running the following shell script from your dbt directory:

```sh wrap theme={null}
# Set Deployment API token credentials as environment variables
export ASTRO_API_TOKEN="<your-api-token>"
# Install the latest version of Astro CLI
curl -sSL install.astronomer.io | sudo bash -s
# Push your dbt code to your Deployment
astro dbt deploy <your-deployment-id> -f
```

## Preview Deployment templates

*Preview Deployment* templates automate creating and deleting Deployments based on feature branches in your Git repository. A CI/CD workflow creates a preview Deployment when you create a feature branch and deletes it when you delete the branch. Use preview Deployments to test a small set of dags on Astro before you promote them to a base, production Deployment.

* For GitHub Actions, see [GitHub Actions templates for preview Deployments](/docs/astro/ci-cd-templates/github-actions-deployment-preview).
* For any other CI/CD tool, see [Preview Deployments](/docs/astro/ci-cd-templates/preview-deployments) for equivalent shell scripts.


# Set up authentication and single sign-on for Astro
Source: https://astronomer.io/docs/astro/configure-idp

Configure federated authentication from third-party identity providers on Astro.

This guide provides the steps for integrating identity providers on Astro to enable single sign-on (SSO) for your users. After you complete the integration for your organization:

* Users will automatically be authenticated to Astro if they're already logged in to your identity provider (IdP).
* You no longer have to repeatedly sign in and remember credentials for your account.
* You will have complete ownership over credential configuration and management on Astro.
* You can require multi-factor authentication (MFA) by enforcing SSO and configuring MFA in your IdP.
* You can use services such as [Adaptive Authentication](https://www.okta.com/identity-101/adaptive-authentication/) and [Conditional Access](https://learn.microsoft.com/en-us/azure/active-directory/conditional-access/overview) to create advanced access policies that enforce trusted IP ranges or limit access to authorized devices.

To manage Organization users after you have configured SSO, see [Manage Organization users](/docs/astro/manage-organization-users).

In addition to SSO, users can authenticate to Astro in three ways:

* Basic authentication (username and password)
* Google social login
* GitHub social login

To require users to sign in through your IdP and disable the other methods, see [SSO enforcement](#sso-enforcement).

## Supported SSO identity providers

Single sign-on (SSO) allows users to sign in using their company credentials, managed through an IdP. This provides a streamlined login experience for your Astro users, as they are able to use the same credentials across multiple applications. In addition, this provides improved security and control for organizations to manage access from a single source. Astro supports integrations with the following IdPs:

* [Microsoft Entra ID](https://www.microsoft.com/en-us/security/business/microsoft-entra-pricing)
* [Okta](https://www.okta.com/)
* [OneLogin](https://www.onelogin.com/)
* [Ping Identity](https://www.pingidentity.com/en.html)

<Info>You can configure multiple SSO connections for a single Organization. This requires having a unique [verified domain](/docs/astro/manage-domains) for each new SSO connection.</Info>

## Configure your SSO identity provider

At a high level, to configure an SSO identity provider (IdP) you will:

1. Create a connection between Astro and your IdP.
2. Map a managed domain to your SSO connection.
3. Invite users to Astro through your IdP.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

<Tabs>
  <Tab title="Okta">
    This section provides setup steps for setting up Okta as your IdP on Astro. After completing this setup, all users in your organization can use Okta to sign in to Astro.

    ### Supported Okta features

    The Astro integration with Okta supports the following authentication options:

    * IdP-initiated SSO
    * Service provider (SP)-initiated SSO
    * Just-In-Time provisioning

    #### Prerequisites (Okta)

    * [Organization Owner](/docs/astro/user-permissions) privileges in the Organization you're configuring.
    * An [Okta account](https://www.okta.com/) with administrative access.
    * At least one [verified domain](/docs/astro/manage-domains).

    #### Configure Okta as your identity provider

    <Steps>
      <Step title="Create a SAML-based connection to Okta">
        To set up Okta as your IdP, you will create a Security Assertion Markup Language (SAML) connection to Okta.

        <Tabs>
          <Tab title="New Astro UI">
            1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.

            2. Click **+ New SSO Configuration**.

            3. Configure the following values for your connection:

               * **Connection type**: Select **SAML**.
               * **Verified Domains**: Enter the verified domain(s) that you want to map to Okta.
               * **Automatic Membership**: Set the default role for users who join your Organization through Okta without an explicit invite from Astro.

            4. Copy the **Connection Name** from the pop-out.

            5. Open a new tab and go to Okta. In the Okta Admin Console, go to **[Applications](https://help.okta.com/en-us/content/topics/apps/apps_apps_page.htm)** and click **Browse App Catalogue**. Then, search the catalogue, select the **Astro** app integration, and click **Add Integration**. After configuring a label for the integration, the application appears in **Applications**.

               <Info>
                 When you create your application, Okta automatically maps the following attributes to Astro user account values:

                 | Attribute | Value            |
                 | --------- | ---------------- |
                 | email     | `user.email`     |
                 | firstName | user.firstName   |
                 | lastName  | user.lastName    |
                 | name      | user.displayName |
               </Info>

            6. Open the Astro application you just configured, click **Sign On**, then click **Edit**. Configure the following values:

               * **Connection Name**: Enter the **Connection Name** you copied from the Astro UI.
               * **Application username format**: **Email**.
               * **Update application username on**: `Create and update`.

            7. Copy the values for **Sign-on URL**, **Sign out URL**, and **X.509 Certificate** from the **Metadata Details** section.

            8. Assign yourself to the Astro app integration from Okta. See [Assign an app integration to a user](https://help.okta.com/en-us/Content/Topics/Provisioning/lcm/lcm-assign-app-user.htm).

            9. Return to the Astro UI. In the configuration screen for your SAML connection, configure the following values:

               * **Identity Provider Single Sign-on URL**: Enter your **Single Sign-on URL**.
               * **Identity Provider Sign-out URL**: Enter your **Single Sign-out URL**.
               * **X.509 Certificate**: Enter your **X.509 Certificate**.

            10. Click **Create**. Your Okta integration appears as an entry in **SSO Configuration**.

            11. &#x20;In **SSO Configuration**, click **Activate**. You are redirected to Okta to test your configuration. After you have successfully authenticated, you are redirected to Astro.

            12. Click **Activate SSO**.
          </Tab>

          <Tab title="Legacy UI">
            1. In the Astro UI, click **Organization Settings**, then click **Authentication**.

            2. In the **SSO Configuration** menu, click **Configure SSO**.

            3. Configure the following values for your connection:

               * **Connection type**: Select **SAML**.
               * **SSO Domain(s)**: Enter the verified domain(s) that you want to map to Okta.
               * **Automatic Membership**: Set the default role for users who join your Organization through Okta without an explicit invite from Astro.

            4. Copy the **Connection Name**.

            5. Open a new tab and go to Okta. In the Okta Admin Console, go to **[Applications](https://help.okta.com/en-us/content/topics/apps/apps_apps_page.htm)** and click **Browse App Catalogue**. Then, search the catalogue, select the **Astro** app integration, and click **Add Integration**. After configuring a label for the integration, the application appears in **Applications**.

               <Info>
                 When you create your application, Okta automatically maps the following attributes to Astro user account values:

                 | Attribute | Value            |
                 | --------- | ---------------- |
                 | email     | `user.email`     |
                 | firstName | user.firstName   |
                 | lastName  | user.lastName    |
                 | name      | user.displayName |
               </Info>

            6. Open the Astro application you just configured, click **Sign On**, then click **Edit**. Configure the following values:

               * **Connection Name**: Enter the **Connection Name** you copied from the Astro UI.
               * **Application username format**: **Email**.
               * **Update application username on**: `Create and update`.

            7. Copy the values for **Sign-on URL**, **Sign out URL**, and **X.509 Certificate** from the **Metadata Details** section.

            8. Assign yourself to the Astro app integration from Okta. See [Assign an app integration to a user](https://help.okta.com/en-us/Content/Topics/Provisioning/lcm/lcm-assign-app-user.htm).

            9. Return to the Astro UI. In the configuration screen for your SAML connection, configure the following values:

               * **Identity Provider Single Sign-on URL**: Enter your **Single Sign-on URL**.
               * **Identity Provider Sign-out URL**: Enter your **Single Sign-out URL**.
               * **X.509 Certificate**: Enter your **X.509 Certificate**.

            10. Click **Create**. Your Okta integration appears as an entry in **SSO Configuration**.

            11. In **SSO Configuration**, click **Activate**. You are redirected to Okta to test your configuration. After you have successfully authenticated, you are redirected to Astro.

            12. Click **Activate SSO**.
          </Tab>
        </Tabs>
      </Step>

      <Step title="Copy your SSO bypass link">
        <Warning>Don't share your single sign-on (SSO) bypass link. With an SSO bypass link, anyone with an email and a password can sign in to Astro. Astronomer recommends periodically regenerating the link from the **Settings** tab in the Astro UI.</Warning>

        An SSO bypass link allows you to authenticate to your Organization without using SSO. This link should be used to access your Organization only when you can't access Astro due to an issue with your identity provider.

        <Tabs>
          <Tab title="New Astro UI">
            1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.
            2. In the **Advanced Settings** section, click **Edit Settings**, turn on **SSO Bypass Link**, then click **Copy**. Save this link for when you need to sign in to Astro without using SSO.

            To remove the bypass link, turn off **SSO Bypass Link**.
          </Tab>

          <Tab title="Legacy UI">
            1. In the Astro UI, click **Organization Settings**, then click **Authentication**.
            2. In the **SSO Bypass Link** field, click **Copy**. Save this link for when you need to sign in to Astro without using SSO.

            If you don't want to maintain an SSO bypass link, click **Delete**. You can always regenerate a link if you need one in the future.
          </Tab>
        </Tabs>
      </Step>

      <Step title="Assign users to your Okta application">
        On the page for your Okta app integration, open the **Assignments** tab. Ensure that all users who will use Astro are assigned to the integration. For more information, see [Assign applications to users](https://help.okta.com/en/prod/Content/Topics/users-groups-profiles/usgp-assign-apps.htm).
      </Step>

      <Step title="(Optional) Configure SCIM provisioning">
        SCIM provisioning allows you to manage Astro users from your identity provider platform. See [Set up SCIM provisioning](/docs/astro/set-up-scim-provisioning) for setup steps.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Microsoft Entra ID">
    This section provides setup steps for setting up Microsoft Entra ID as your IdP on Astro. After completing this setup, your organization's users can use Microsoft Entra ID to sign in to Astro.

    #### Prerequisites (Microsoft Entra ID)

    To integrate Azure as your IdP for Astro you must have:

    * An Azure subscription.
    * [Cloud Application Administrator](https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/permissions-reference#cloud-application-administrator) permissions on Microsoft Entra ID.
    * [Organization Owner](/docs/astro/user-permissions) permissions in the Organization you're configuring.
    * At least one [verified domain](/docs/astro/manage-domains).

    #### Configure Microsoft Entra ID as your identity provider

    <Steps>
      <Step title="Register Astro as an application on Azure">
        Follow the [Microsoft documentation](https://learn.microsoft.com/en-us/entra/identity/saas-apps/astro-tutorial#add-astro-from-the-gallery) to add Astro from the gallery to your list of managed SaaS applications.
      </Step>

      <Step title="Configure SSO on Astro">
        <Tabs>
          <Tab title="New Astro UI">
            1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.

            2. Click **+ New SSO Configuration**.

            3. Configure the following values for your connection:

               * **Connection Type**: Select **SAML**. Astro also supports **Azure AD** as a connection type for Microsoft Entra ID. Astronomer recommends **SAML** because Microsoft's [Astro tutorial](https://learn.microsoft.com/en-us/entra/identity/saas-apps/astro-tutorial) only documents the SAML setup path. If you prefer to use the **Azure AD** connection type instead, see [Alternative: Configure SSO using the Azure AD connection type](#alternative-configure-sso-using-the-azure-ad-connection-type).
               * **Verified Domains**: Enter the verified domain(s) that you want to map to Microsoft Entra ID.
               * **Automatic Membership**: Set the default role for users who join your Organization through Microsoft Entra ID without an explicit invite from Astro.

            4. Copy the **Single Sign-On URL** and **Audience URI (SP Entity ID)** for the next step.

            5. Keep this configuration window open. You'll return to it later in this procedure.
          </Tab>

          <Tab title="Legacy UI">
            1. In the Astro UI, click **Organization Settings**, then click **Authentication**.

            2. In the **SSO Configuration** menu, click **Configure SSO**.

            3. Configure the following values for your connection:

               * **Connection Type**: Select **SAML**. Astro also supports **Azure AD** as a connection type for Microsoft Entra ID. Astronomer recommends **SAML** because Microsoft's [Astro tutorial](https://learn.microsoft.com/en-us/entra/identity/saas-apps/astro-tutorial) only documents the SAML setup path. If you prefer to use the **Azure AD** connection type instead, see [Alternative: Configure SSO using the Azure AD connection type](#alternative-configure-sso-using-the-azure-ad-connection-type).
               * **SSO Domain(s)**: Enter the verified domain(s) that you want to map to Microsoft Entra ID.
               * **Automatic Membership**: Set the default role for users who join your Organization through Microsoft Entra ID without an explicit invite from Astro.

            4. Copy the **Single Sign-On URL** and **Audience URI (SP Entity ID)** for the next step.

            5. Keep this configuration window open. You'll return to it later in this procedure.
          </Tab>
        </Tabs>
      </Step>

      <Step title="Configure SSO on Azure">
        In a new tab, open the [Microsoft Entra admin center](https://entra.microsoft.com/) and follow the steps in [Microsoft documentation](https://learn.microsoft.com/en-us/entra/identity/saas-apps/astro-tutorial#configure-microsoft-entra-sso) to configure Microsoft Entra SSO for Astro. In the **Basic SAML Configuration** section of your configuration, set the following values:

        * **Identifier (Entity ID)**: Enter the value for **Audience URI (SP Entity ID)** you copied in the previous step.
        * **Reply URL (Assertion Consumer Service URL)**: Enter the value for **Single Sign-On URL** you copied in the previous step.

        After you complete the configuration, download the PEM certificate from the **SAML Signing Certificate** section for the next step.
      </Step>

      <Step title="Finalize the SSO connection in Astro">
        1. Return to the Astro UI. In the configuration screen for your SSO connection, configure the following values:

           * **X.509 Certificate**: Enter the PEM certificate you downloaded in the previous step.
           * **Identity Provider Single Sign-on URL**: Enter the URL you copied in the previous step.

        2. Click **Create**. Your Entra ID integration appears as an entry in **SSO Configuration**.
      </Step>

      <Step title="Copy your SSO bypass link">
        <Warning>Don't share your single sign-on (SSO) bypass link. With an SSO bypass link, anyone with an email and a password can sign in to Astro. Astronomer recommends periodically regenerating the link from the **Settings** tab in the Astro UI.</Warning>

        An SSO bypass link allows you to authenticate to your Organization without using SSO. This link should be used to access your Organization only when you can't access Astro due to an issue with your identity provider.

        <Tabs>
          <Tab title="New Astro UI">
            1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.
            2. In the **Advanced Settings** section, click **Edit Settings**, turn on **SSO Bypass Link**, then click **Copy**. Save this link for when you need to sign in to Astro without using SSO.

            To remove the bypass link, turn off **SSO Bypass Link**.
          </Tab>

          <Tab title="Legacy UI">
            1. In the Astro UI, click **Organization Settings**, then click **Authentication**.
            2. In the **SSO Bypass Link** field, click **Copy**. Save this link for when you need to sign in to Astro without using SSO.

            If you don't want to maintain an SSO bypass link, click **Delete**. You can always regenerate a link if you need one in the future.
          </Tab>
        </Tabs>
      </Step>

      <Step title="Assign users to your Microsoft Entra ID application">
        Follow [Microsoft documentation](https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/assign-user-or-group-access-portal?pivots=portal) to assign users from your organization to your new application.

        When a user assigned to the application accesses Astro, they will be brought automatically to Microsoft Entra ID after entering their email in the Astro UI.
      </Step>
    </Steps>

    #### Alternative: Configure SSO using the Azure AD connection type

    Instead of SAML, you can connect Microsoft Entra ID to Astro using the **Azure AD** connection type, which uses OAuth 2.0 / OpenID Connect with an Azure AD app registration. Use this path if your organization prefers OIDC-based SSO over SAML.

    <Steps>
      <Step title="Register Astro as an app in Microsoft Entra ID">
        1. Sign in to the [Microsoft Entra admin center](https://entra.microsoft.com/) as a Cloud Application Administrator.

        2. Follow [Microsoft documentation](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app) to register a new application. When prompted, set the **Redirect URI** type to **Web** and the value to:

           ```text wrap theme={null}
           https://auth.astronomer.io/login/callback
           ```

        3. After the app is created, from the app's **Overview** page, copy the following values for the next step:

           * **Application (client) ID**
           * **Directory (tenant) ID** (your primary Microsoft Azure AD domain, for example `your-org.onmicrosoft.com`, is also available on the tenant's overview page)

        4. In the app, open **Certificates & secrets**, click **New client secret**, and create a secret. Copy the secret **Value** immediately — it's only shown once. Save it for the next step.

        5. In the app, open **API permissions** and ensure the default `Microsoft Graph > User.Read` delegated permission is present so users can sign in and read their profile. Grant admin consent if your tenant requires it.
      </Step>

      <Step title="Start the SSO configuration in Astro">
        <Tabs>
          <Tab title="New Astro UI">
            1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.

            2. Click **+ New SSO Configuration**.

            3. Configure the following values for your connection:

               * **Connection Type**: Select **Azure AD**.
               * **Verified Domain(s)**: Select the verified domain(s) that you want to map to Microsoft Entra ID. Users with a matching email domain can sign in through SSO.
               * **Automatic Membership**: Set the default Organization role for users who sign in through SSO without an explicit invite from Astro.

            4. Confirm that the **Redirect URI** shown in the Astro UI matches the redirect URI you configured on the Azure app registration in the previous step:

               ```text wrap theme={null}
               https://auth.astronomer.io/login/callback
               ```
          </Tab>

          <Tab title="Legacy UI">
            1. In the Astro UI, click **Organization Settings**, then click **Authentication**.

            2. In the **SSO Configuration** menu, click **Configure SSO**.

            3. Configure the following values for your connection:

               * **Connection Type**: Select **Azure AD**.
               * **Verified Domain(s)**: Select the verified domain(s) that you want to map to Microsoft Entra ID. Users with a matching email domain can sign in through SSO.
               * **Automatic Membership**: Set the default Organization role for users who sign in through SSO without an explicit invite from Astro.

            4. Confirm that the **Redirect URI** shown in the Astro UI matches the redirect URI you configured on the Azure app registration in the previous step:

               ```text wrap theme={null}
               https://auth.astronomer.io/login/callback
               ```
          </Tab>
        </Tabs>
      </Step>

      <Step title="Provide your Azure AD values to Astro">
        In the same Astro configuration screen, under **Get the following values from your IdP and input them here**, enter the values you copied from your Azure app registration:

        * **Microsoft Azure AD Domain**: Your tenant's primary domain (for example, `your-org.onmicrosoft.com`).
        * **Client ID**: The **Application (client) ID** from the app's **Overview** page.
        * **Client Secret**: The client secret **Value** you created under **Certificates & secrets**.

        Click **Create**. Your Azure AD integration appears as an entry in **SSO Configuration**.
      </Step>

      <Step title="Assign users and test">
        1. In the Microsoft Entra admin center, follow [Microsoft documentation](https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/assign-user-or-group-access-portal?pivots=portal) to assign users or groups from your organization to the Astro app registration.
        2. In the Astro UI, activate the new SSO connection and complete the test sign-in flow.
        3. Copy your SSO bypass link from the Astro UI **Authentication** page and store it securely in case you ever need to sign in without SSO.

        When a user with an email in a verified domain signs in to Astro, they're redirected to Microsoft Entra ID to authenticate.
      </Step>
    </Steps>
  </Tab>

  <Tab title="OneLogin">
    This section provides setup steps for setting up OneLogin as your IdP on Astro. After completing this setup, your organization's users can use OneLogin to sign in to Astro.

    #### Prerequisites (OneLogin)

    * A [OneLogin account](https://www.onelogin.com/) with administrative access.
    * [Organization Owner](/docs/astro/user-permissions) privileges in the Organization you're configuring.
    * At least one [verified domain](/docs/astro/manage-domains).

    #### Configure OneLogin as your identity provider

    <Steps>
      <Step title="Create a SAML-based connection to OneLogin">
        To set up OneLogin as your IdP, you will create a Security Assertion Markup Language (SAML) connection to OneLogin.

        <Tabs>
          <Tab title="New Astro UI">
            1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.

            2. Click **+ New SSO Configuration**.

            3. Configure the following values for your connection:

               * **Connection Type**: Select **SAML**.
               * **Verified Domains**: Enter the verified domain(s) that you want to map to OneLogin.
               * **Automatic Membership**: Set the default role for users who join your Organization through OneLogin without an explicit invite from Astro.

            4. Copy the **Single Sign-On URL** and **Audience URI (SP ENTITY ID)** for use later in this procedure.

            5. Open a new tab and go to OneLogin. In the OneLogin administrator dashboard, click **Applications** > **Applications** and then click **Add App**.

            6. In the **Search** field, enter **SAML Custom**, and then select **SAML Custom Connector (Advanced)**.

            7. In the **Display Name** field, enter **Astro** and then click **Save**.

            8. Click **Configuration** in the left menu and complete the following fields:

               * **Audience (EntityID)**: `<your-audience-uri>`
               * **ACS (Consumer) URL Validator**: `<your-sso-url>`
               * **ACS (Consumer) URL**: `<your-sso-url>`

            9. Select the **Sign SLO Request** and **Sign SLO Response** checkboxes. Then, click **Save**.

            10. Click **Parameters** in the left menu, and add the following four parameters, using the same capitalization shown in the **Value** column:

                | Field name | Value      |
                | ---------- | ---------- |
                | email      | Email      |
                | firstName  | First Name |
                | lastName   | Last Name  |
                | name       | Name       |

                Select the **Include in SAML assertion** checkbox for every parameter that you add and then click **Save**.

            11. Click **SSO** in the left menu, click **View Details** below the **X.509 Certificate** field and then click **Download**.

            12. Select **SHA-256** in the **SAML Signature Algorithm** list.

            13. Copy and save the value displayed in the **SAML 2.0 Endpoint (HTTP)** field.

            14. Assign yourself to the Astro app integration from OneLogin. See [Assigning apps to users](https://onelogin.service-now.com/kb_view_customer.do?sysparm_article=KB0010387).

            15. Return to the Astro UI. In the configuration screen for your SAML connection, configure the following values:

                * **Identity Provider Single Sign-on URL**: Enter the value you copied from **SAML 2.0 Endpoint (HTTP)**.
                * **X.509 Certificate**: Enter the **X.509 Certificate** that you downloaded.

            16. Click **Create**. Your OneLogin integration appears as an entry in **SSO Configuration**.

            17. &#x20;In **SSO Configuration**, click **Activate**. You are redirected to OneLogin to test your configuration. After you have successfully authenticated, you are redirected to Astro. Then, click **Activate SSO**.
          </Tab>

          <Tab title="Legacy UI">
            1. In the Astro UI, click **Organization Settings**, then click **Authentication**.

            2. In the **SSO Configuration** menu, click **Configure SSO**.

            3. Configure the following values for your connection:

               * **Connection Type**: Select **SAML**.
               * **SSO Domain(s)**: Enter the verified domain(s) that you want to map to OneLogin.
               * **Automatic Membership**: Set the default role for users who join your Organization through OneLogin without an explicit invite from Astro.

            4. Copy the **Single Sign-On URL** and **Audience URI (SP ENTITY ID)** for use later in this procedure.

            5. Open a new tab and go to OneLogin. In the OneLogin administrator dashboard, click **Applications** > **Applications** and then click **Add App**.

            6. In the **Search** field, enter **SAML Custom**, and then select **SAML Custom Connector (Advanced)**.

            7. In the **Display Name** field, enter **Astro** and then click **Save**.

            8. Click **Configuration** in the left menu and complete the following fields:

               * **Audience (EntityID)**: `<your-audience-uri>`
               * **ACS (Consumer) URL Validator**: `<your-sso-url>`
               * **ACS (Consumer) URL**: `<your-sso-url>`

            9. Select the **Sign SLO Request** and **Sign SLO Response** checkboxes. Then, click **Save**.

            10. Click **Parameters** in the left menu, and add the following four parameters, using the same capitalization shown in the **Value** column:

                | Field name | Value      |
                | ---------- | ---------- |
                | email      | Email      |
                | firstName  | First Name |
                | lastName   | Last Name  |
                | name       | Name       |

                Select the **Include in SAML assertion** checkbox for every parameter that you add and then click **Save**.

            11. Click **SSO** in the left menu, click **View Details** below the **X.509 Certificate** field and then click **Download**.

            12. Select **SHA-256** in the **SAML Signature Algorithm** list.

            13. Copy and save the value displayed in the **SAML 2.0 Endpoint (HTTP)** field.

            14. Assign yourself to the Astro app integration from OneLogin. See [Assigning apps to users](https://onelogin.service-now.com/kb_view_customer.do?sysparm_article=KB0010387).

            15. Return to the Astro UI. In the configuration screen for your SAML connection, configure the following values:

                * **Identity Provider Single Sign-on URL**: Enter the value you copied from **SAML 2.0 Endpoint (HTTP)**.
                * **X.509 Certificate**: Enter the **X.509 Certificate** that you downloaded.

            16. Click **Create**. Your OneLogin integration appears as an entry in **SSO Configuration**.

            17. In **SSO Configuration**, click **Activate**. You are redirected to OneLogin to test your configuration. After you have successfully authenticated, you are redirected to Astro. Then, click **Activate SSO**.
          </Tab>
        </Tabs>
      </Step>

      <Step title="Copy your SSO bypass link">
        <Warning>Don't share your single sign-on (SSO) bypass link. With an SSO bypass link, anyone with an email and a password can sign in to Astro. Astronomer recommends periodically regenerating the link from the **Settings** tab in the Astro UI.</Warning>

        An SSO bypass link allows you to authenticate to your Organization without using SSO. This link should be used to access your Organization only when you can't access Astro due to an issue with your identity provider.

        <Tabs>
          <Tab title="New Astro UI">
            1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.
            2. In the **Advanced Settings** section, click **Edit Settings**, turn on **SSO Bypass Link**, then click **Copy**. Save this link for when you need to sign in to Astro without using SSO.

            To remove the bypass link, turn off **SSO Bypass Link**.
          </Tab>

          <Tab title="Legacy UI">
            1. In the Astro UI, click **Organization Settings**, then click **Authentication**.
            2. In the **SSO Bypass Link** field, click **Copy**. Save this link for when you need to sign in to Astro without using SSO.

            If you don't want to maintain an SSO bypass link, click **Delete**. You can always regenerate a link if you need one in the future.
          </Tab>
        </Tabs>
      </Step>

      <Step title="Assign users to your OneLogin Astro application">
        1. In the OneLogin administrator dashboard, click **Applications** > **Applications** and then click **Astro**.
        2. Click **Users** in the left menu.
        3. Make sure that all users who need to use Astro are assigned to the Astronomer application.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Ping Identity">
    This section provides setup steps for setting up Ping Identity as your IdP on Astro. After completing this setup, your organization's users can use Ping Identity to sign in to Astro.

    #### Prerequisites (Ping Identity)

    * A [Ping Identity account](https://www.pingidentity.com/) with administrative access.
    * [Organization Owner](/docs/astro/user-permissions) privileges in the Organization you're configuring.
    * At least one [verified domain](/docs/astro/manage-domains).

    #### Configure Ping Identity as your identity provider

    <Steps>
      <Step title="Configure Ping Identity">
        <Tabs>
          <Tab title="New Astro UI">
            1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.

            2. Click **+ New SSO Configuration**.

            3. Configure the following values for your connection:

               * **Connection type**: Select **SAML**.
               * **Verified Domains**: Enter the verified domain(s) that you want to map to Ping Identity.
               * **Automatic Membership**: Set the default role for users who join your Organization through Ping Identity without an explicit invite from Astro.

            4. Copy the **Single Sign-On URL** and **Audience URI (SP ENTITY ID)** for use later in this procedure.

            5. Open a new tab and go to PingIdentity. In the PingIdentity Administrator Console, click **Connections** in the left pane, and then click the **+** icon next to **Applications**.

            6. In the **Application Name** field, enter `Astro`. Optionally, add a description and an icon.

            7. Click **SAML Application**, and then click **Configure**.

            8. Click **Manually Enter** and then complete the following fields:

               * **ACS URLs**: `<your-sso-url>`
               * **Entity ID**: `<your-audience-uri>`

            9. Click **Save**.

            10. Click **Edit** on the **Overview** page, and then enter `<your-sso-url>` in the **Sign on URL** field. Then, click **Save**.

            11. Click the **Configuration** tab, and then click **Edit**.

            12. Select **Sign Assertion & Response** and confirm `RSA_SHA256` is selected in the **Signing Algorithm** list. Then, click **Save**.

            13. On the **Configuration** page, click **Download Signing Certificate** and select `X509 PEM (.crt)` to download the X.509 certificate for your application.

            14. Copy and save the URL in the **Single Sign-on Service** field.

            15. Click the **Attribute Mappings** tab, click **Edit**, and add the following attributes, using the capitalization shown in both columns:

                | Astronomer     | PingOne       |
                | -------------- | ------------- |
                | `saml_subject` | User ID       |
                | email          | Email Address |
                | firstName      | Given Name    |
                | lastName       | Family Name   |
                | name           | Formatted     |

            16. Click **Save**.

            17. (Optional) If you configured your application on a PingFederate server, enable the **Include the certificate in the signature `<KeyInfo>` element** setting for your server. See [Ping documentation](https://docs.pingidentity.com/pingfederate/latest/administrators_reference_guide/pf_certificate_key_management.html).

            18. Click the toggle in the top right to enable the application.

            19. Assign yourself to Astro from Ping Identity. See [Editing a user](https://docs.pingidentity.com/pingone/directory/p1_edituser.html).

            20. Return to the Astro UI. In the configuration screen for your SAML connection, configure the following values:

                * **Identity Provider Single Sign-on URL**: Enter the value you copied from the **Single Sign-on Service** field.
                * **X.509 Certificate**: Enter the X.509 Certificate that you downloaded.

            21. Click **Create**. Your Ping Identity integration appears as an entry in **SSO Configuration**.

            22. &#x20;In **SSO Configuration**, click **Activate**. You are redirected to Ping Identity to test your configuration. After you have successfully authenticated, you are redirected to Astro.

            23. Click **Activate SSO**.
          </Tab>

          <Tab title="Legacy UI">
            1. In the Astro UI, click **Organization Settings**, then click **Authentication**.

            2. In the **SSO Configuration** menu, click **Configure SSO**.

            3. Configure the following values for your connection:

               * **Connection type**: Select **SAML**.
               * **SSO Domain(s)**: Enter the verified domain(s) that you want to map to Ping Identity.
               * **Automatic Membership**: Set the default role for users who join your Organization through Ping Identity without an explicit invite from Astro.

            4. Copy the **Single Sign-On URL** and **Audience URI (SP ENTITY ID)** for use later in this procedure.

            5. Open a new tab and go to PingIdentity. In the PingIdentity Administrator Console, click **Connections** in the left pane, and then click the **+** icon next to **Applications**.

            6. In the **Application Name** field, enter `Astro`. Optionally, add a description and an icon.

            7. Click **SAML Application**, and then click **Configure**.

            8. Click **Manually Enter** and then complete the following fields:

               * **ACS URLs**: `<your-sso-url>`
               * **Entity ID**: `<your-audience-uri>`

            9. Click **Save**.

            10. Click **Edit** on the **Overview** page, and then enter `<your-sso-url>` in the **Sign on URL** field. Then, click **Save**.

            11. Click the **Configuration** tab, and then click **Edit**.

            12. Select **Sign Assertion & Response** and confirm `RSA_SHA256` is selected in the **Signing Algorithm** list. Then, click **Save**.

            13. On the **Configuration** page, click **Download Signing Certificate** and select `X509 PEM (.crt)` to download the X.509 certificate for your application.

            14. Copy and save the URL in the **Single Sign-on Service** field.

            15. Click the **Attribute Mappings** tab, click **Edit**, and add the following attributes, using the capitalization shown in both columns:

                | Astronomer     | PingOne       |
                | -------------- | ------------- |
                | `saml_subject` | User ID       |
                | email          | Email Address |
                | firstName      | Given Name    |
                | lastName       | Family Name   |
                | name           | Formatted     |

            16. Click **Save**.

            17. (Optional) If you configured your application on a PingFederate server, enable the **Include the certificate in the signature `<KeyInfo>` element** setting for your server. See [Ping documentation](https://docs.pingidentity.com/pingfederate/latest/administrators_reference_guide/pf_certificate_key_management.html).

            18. Click the toggle in the top right to enable the application.

            19. Assign yourself to Astro from Ping Identity. See [Editing a user](https://docs.pingidentity.com/pingone/directory/p1_edituser.html).

            20. Return to the Astro UI. In the configuration screen for your SAML connection, configure the following values:

                * **Identity Provider Single Sign-on URL**: Enter the value you copied from the **Single Sign-on Service** field.
                * **X.509 Certificate**: Enter the X.509 Certificate that you downloaded.

            21. Click **Create**. Your Ping Identity integration appears as an entry in **SSO Configuration**.

            22. In **SSO Configuration**, click **Activate**. You are redirected to Ping Identity to test your configuration. After you have successfully authenticated, you are redirected to Astro.

            23. Click **Activate SSO**.
          </Tab>
        </Tabs>
      </Step>

      <Step title="Copy your SSO bypass link">
        <Warning>Don't share your single sign-on (SSO) bypass link. With an SSO bypass link, anyone with an email and a password can sign in to Astro. Astronomer recommends periodically regenerating the link from the **Settings** tab in the Astro UI.</Warning>

        An SSO bypass link allows you to authenticate to your Organization without using SSO. This link should be used to access your Organization only when you can't access Astro due to an issue with your identity provider.

        <Tabs>
          <Tab title="New Astro UI">
            1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.
            2. In the **Advanced Settings** section, click **Edit Settings**, turn on **SSO Bypass Link**, then click **Copy**. Save this link for when you need to sign in to Astro without using SSO.

            To remove the bypass link, turn off **SSO Bypass Link**.
          </Tab>

          <Tab title="Legacy UI">
            1. In the Astro UI, click **Organization Settings**, then click **Authentication**.
            2. In the **SSO Bypass Link** field, click **Copy**. Save this link for when you need to sign in to Astro without using SSO.

            If you don't want to maintain an SSO bypass link, click **Delete**. You can always regenerate a link if you need one in the future.
          </Tab>
        </Tabs>
      </Step>

      <Step title="Assign users to your Ping Identity application">
        Assign users from your organization to your new application. See [Managing user groups](https://docs.pingidentity.com/pingone/directory/p1_managing_groups.html).

        When a user assigned to the application accesses Astro, they are automatically signed in to Ping Identity after entering their email in the Astro UI.
      </Step>
    </Steps>
  </Tab>
</Tabs>

## SSO enforcement

<Note> This is feature is only available if you are on the **Business** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

Configuring an SSO connection lets your users sign in through your IdP, but doesn't require them to. By default, users in your verified domains can still sign in with basic authentication, Google, or GitHub. To require them to sign in through your IdP, enforce SSO.

Enforcement:

* Limits sign-in to SSO for users in your verified domains.
* Disables basic authentication, Google, and GitHub for those users.
* Is available on Business plans or higher.

<Note>
  Astro doesn't provide native MFA for basic or social logins. To require MFA for all users, enforce SSO on a Business plan or higher and configure MFA in your IdP — for example, with [Okta Adaptive Authentication](https://www.okta.com/identity-101/adaptive-authentication/) or [Microsoft Conditional Access](https://learn.microsoft.com/en-us/azure/active-directory/conditional-access/overview).
</Note>

To enforce SSO:

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.
    2. In the **Advanced Settings** section, click **Edit Settings**, then select **Allow only Single Sign-On (SSO)** from the **Login Method** list.

    You can also restrict sign-in to only Google or only GitHub from the same list.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**, then click **Authentication**.
    2. In the **Advanced Settings** menu, select **Allow only Single Sign-On (SSO)** from the **Login Methods** list.

    <Frame>
      <img alt="The Edit Advanced Settings dialog with &#x22;Allow only Single Sign-On (SSO)&#x22; selected in the Login Methods list." />
    </Frame>

    You can also restrict sign-in to only Google or only GitHub from the same list.
  </Tab>
</Tabs>

## Reconfigure SSO to a new identity provider

If your organization needs to switch from one SSO identity provider to another, you can remove the existing SSO connection and configure a new one. This process requires a brief period of downtime during which users can't sign in through the old IdP.

<Warning>After you delete the old SSO connection, users can't sign in using the old IdP. Schedule this change during a maintenance window to minimize disruption.</Warning>

### Prerequisites

* [Organization Owner](/docs/astro/user-permissions) permissions in the Organization you're reconfiguring.
* A new IdP account with administrative access.
* At least one [verified domain](/docs/astro/manage-domains).

### Switch to your new identity provider

<Steps>
  <Step title="Register Astro with the new identity provider">
    Before you remove the old SSO connection, complete the initial registration step for your new IdP. This ensures that your new IdP is ready to connect to Astro as soon as the old SSO connection is removed.

    Follow the instructions in the relevant tab under [Configure your SSO identity provider](#configure-your-sso-identity-provider) and complete only the first step for your new IdP. For example, if you're switching to Microsoft Entra ID, complete the **Register Astro as an application on Azure** step.
  </Step>

  <Step title="Delete the old SSO connection">
    <Tabs>
      <Tab title="New Astro UI">
        1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.
        2. &#x20;Delete the SSO connection.
        3. Confirm the deletion.
      </Tab>

      <Tab title="Legacy UI">
        1. In the Astro UI, click **Organization Settings**, then click **Authentication**.
        2. In the **SSO Configuration** menu, click **Delete SSO Connection**.
        3. Confirm the deletion.
      </Tab>
    </Tabs>

    After you delete the old connection, users can only sign in through an SSO bypass link or other enabled authentication methods until you activate the new SSO connection.
  </Step>

  <Step title="Complete the new SSO configuration">
    Return to the instructions for your new IdP under [Configure your SSO identity provider](#configure-your-sso-identity-provider) and complete the remaining steps, starting from the step where you configure SSO on Astro.
  </Step>
</Steps>

## Advanced setup

### Configure just-in-time provisioning

Astro supports just-in-time provisioning by default for all single sign-on (SSO) integrations. This means that if someone without an Astro account tries logging into Astro with an email address from a domain that you manage, they are automatically granted a default role in your Organization without needing an invite. Users with emails outside of this domain need to be invited to your Organization to access it.

To enable or disable just-in-time provisioning:

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.
    2. &#x20;Edit your SSO connection.
    3. In **Automatic Membership**, select the default Organization role for users who sign in to Astro for the first time through your identity provider. To disable just-in-time provisioning, select **Disabled**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**, then click **Authentication**.
    2. In the **SSO Configuration** menu, click the pencil icon to edit your SSO connection.
    3. In **Automatic Membership**, select the default Organization role for users who sign in to Astro for the first time through your identity provider. To disable just-in-time provisioning, select **Disabled**.
  </Tab>
</Tabs>

### Regenerate an SSO bypass link

Regenerating your SSO bypass link voids your existing SSO bypass link so that any former users with the existing link can't sign in to Astro.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**, then in the **Advanced Settings** section, click **Edit Settings**.
    2. &#x20;Click **Regenerate** to create a new bypass link and void the old one.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click the **Settings** tab.
    2. Click **Regenerate** to create a new bypass link and void the old one.
  </Tab>
</Tabs>


# Configure worker queues
Source: https://astronomer.io/docs/astro/configure-worker-queues

Learn how to create and configure worker queues to create best-fit execution environments for your tasks.

By default, all tasks using the Astro or Celery executor run in a `default` worker queue. You can create additional worker queues to enable multiple worker types or configurations for different groups of tasks, and assign tasks to these queues in your Dag code. For more information about Airflow executors on Astro, see [Manage Airflow executors on Astro](/docs/astro/executors-overview).

Use worker queues to create optimized execution environments for different types of tasks in the same Deployment. You can use worker queues to:

* Separate resource-intensive tasks, such as those that execute machine learning models, from tasks that require minimal resources, such as those that execute SQL queries.
* Separate short-running tasks from long-running tasks.
* Isolate a single task from other tasks in your Deployment.
* Allow some workers to scale to zero but keep a minimum of 1 for other types of workers.

## Benefits

Worker queues can enhance performance, decrease cost, and increase the reliability of task execution in your Deployment. Specifically:

* Executing a task with dedicated hardware that best fits the needs of that task can result in faster performance. In some cases, this can decrease the duration of a Dag by up to 50%.
* Paying for larger workers only for select tasks means that you can lower your infrastructure cost by not paying for that worker when your tasks don't need it.
* Separating tasks that have different characteristics often means they're less likely to result in a failed or zombie state.

## Example

By configuring multiple worker queues and assigning tasks to these queues based on the requirements of the tasks, you can enhance the performance, reliability, and throughput of your Deployment. For example, consider the following scenario:

* You are running Task A and Task B in a Deployment.
* Task A and Task B are dependent on each other, so they need to run in the same Deployment.
* Task A is a long-running task that uses a lot of CPU and memory, while Task B is a short-running task that uses minimal amounts of CPU and memory.

You can assign Task A to a worker queue that is configured to use the A20 worker type, which is optimized for running compute-heavy tasks. Then, you can assign Task B to a worker queue that is configured to use the A5 worker type, which is smaller and optimized for general usage.

## Default worker queue

Each Deployment requires a worker queue named `default` to run tasks. Tasks that aren't assigned to a worker queue in your Dag code are executed by workers in the `default` worker queue.

You can change all settings of the default worker queue except for its name.

## Worker queue settings

You can configure each worker queue on Astro with the following settings:

* **Name:** The name of your worker queue. Use this name to assign tasks to the worker queue in your Dag code. Worker queue names must consist only of lowercase letters and hyphens. For example, `machine-learning-tasks` or `short-running-tasks` or `high-cpu`.
* **Concurrency:** The maximum number of tasks that a single worker can run at a time. If the number of queued and running tasks exceeds this number, a new worker is added to run the remaining tasks. This value is equivalent to [worker concurrency](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#worker-concurrency) in Apache Airflow. The default for your worker type is suitable for most use cases.
* **Min # Workers / Max # Workers**: The minimum and maximum number of workers that can run at a time. The number of workers autoscales based on **Concurrency** and the current number of tasks in a `queued` or `running` state.
* **Worker Type:** The size of workers in the worker queue, for example A5 or A20. A worker's total available CPU, memory, and storage is defined by its worker size. For more information, see the following section on worker types.

### Hosted Deployment worker types

Each Deployment worker queue has a *worker type* that determines how many resources are available to your Airflow workers for running tasks. A worker type is a virtualized instance of CPU and memory on your cluster that is specific to the Astro platform. The underlying node instance type running your worker can vary based on how Astro optimizes resource usage on your cluster.

Each virtualized instance of your worker type is a *worker*. Astro and Celery workers can run multiple tasks at once, while Kubernetes workers only scale up and down to run a single task at a time.

The following table lists all available worker types on Astro Deployments.

| Worker Type | vCPU | Memory | Ephemeral storage | Default task concurrency | Max task concurrency |
| ----------- | ---- | ------ | ----------------- | ------------------------ | -------------------- |
| A5          | 1    | 2GiB   | 10 GiB            | 5                        | 15                   |
| A10         | 2    | 4GiB   | 10 GiB            | 10                       | 30                   |
| A20         | 4    | 8GiB   | 10 GiB            | 20                       | 60                   |
| A40         | 8    | 16GiB  | 10 GiB            | 40                       | 120                  |
| A60         | 12   | 24GiB  | 10 GiB            | 60                       | 180                  |
| A120        | 24   | 48GiB  | 10 GiB            | 120                      | 360                  |
| A160        | 32   | 64GiB  | 10 GiB            | 160                      | 480                  |

All worker types additionally have 10 GiB of ephemeral storage that your tasks can use when storing small amounts of data within the worker.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Create a worker queue

<Info>
  **CLI**

  If you prefer, you can also run the `astro deployment worker-queue create` command in the Astro CLI to create a worker queue. See the [CLI Command Reference](/docs/cli/v1.43/astro-deployment-worker-queue-create).
</Info>

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**.

    2. Click the **More actions** menu (⋯) on the Deployment you want to edit, then click **Edit Deployment**.

    3. In the **Execution** section, click **+ Add Queue** to create a new worker queue, and then configure its related attributes. Note that you can't change the name of a worker queue after you create it.

    4. When you're done creating your worker queue, click **Update Deployment** to save your changes.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.

    2. Click the **Options** menu of the Deployment you want to update, and select **Edit Deployment**.

       <Frame>
         <img alt="Edit Deployment in options menu" />
       </Frame>

    3. In the **Execution** section, click **Add Queue** to create a new worker queue, and then configure its related attributes. Note that you can't change the name of a worker queue after you create it.

    4. When you're done creating your worker queue, click **Update Deployment** to save your changes.
  </Tab>
</Tabs>

<Tip>You can create, update, and delete multiple worker queues at once using a Deployment file. See [Deployments as Code](/docs/astro/manage-deployments-as-code).</Tip>

## Assign tasks to a worker queue

By default, all tasks run in the default worker queue. To run tasks on a different worker queue, assign the task to the worker queue in your Dag code.

### Step 1: Copy the name of the worker queue

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then click the Deployment you want to use.
    2. Click the **Details** tab. Under **Execution**, the worker queues are listed.
    3. Copy the name of the worker queue you want to assign a task to.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace and select a Deployment.
    2. Click the **Details** tab.
    3. From the **Execution** section, copy the name of the worker queue you want to assign a task to.
  </Tab>
</Tabs>

### Step 2: Assign the task in your Dag code

In your Dag code, add a `queue='<worker-queue-name>'` argument to the definition of the task. If a task is assigned to a queue that doesn't exist or isn't referenced properly, the task might remain in a `queued` state and fail to execute. Make sure that the name of the queue in your Dag code matches the name of the queue in the Astro UI.

Astronomer recommends using Apache Airflow's [Taskflow API](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/taskflow.html) to define your task argument. The Taskflow API is feature in Airflow 2 that includes a task [decorator](/docs/learn/airflow-decorators) and makes Dags easier to write. In the following examples, all instances of the task will run in the `machine-learning-tasks` queue. Choose an example based on whether or not you use the Taskflow API.

<Tabs>
  <Tab title="Classic Operator Example">
    ```python wrap theme={null}
    train_model = PythonOperator(
        task_id="train_model",
        python_callable=train_model_flights,
        queue="machine-learning-tasks",
    )
    ```
  </Tab>

  <Tab title="TaskFlow API Example">
    ```python wrap theme={null}
    @task(task_id="train_model", queue="machine-learning-tasks")
    def train_model_flights(x_train, y_train):
        import xgboost
        from sklearn.pipeline import Pipeline
        from sklearn.preprocessing import StandardScaler

        xgbclf = xgboost.XGBClassifier()

        pipe = Pipeline([
            ("scaler", StandardScaler(with_mean=False)),
            ("xgbclf", xgbclf),
        ])

        pipe.fit(x_train, y_train)

        return pipe
    ```
  </Tab>
</Tabs>

## Update a worker queue

<Info>
  **CLI**

  If you prefer, you can run the `astro deployment worker-queue update` command in the Astro CLI to update a worker queue. See the [CLI Command Reference](/docs/cli/v1.43/astro-deployment-worker-queue-update).
</Info>

You can update worker queue configurations at any time. The worker queue name can't be changed.

If you need to change the worker type of an existing worker queue, Astronomer recommends making the change at a time when it won't affect production pipelines. After you've changed a worker type, Astronomer recommends waiting a minimum of five minutes before pushing new code to your Deployment.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**.

    2. Click the **More actions** menu (⋯) on the Deployment you want to edit, then click **Edit Deployment**.

    3. Expand the **Execution** section if it isn't already expanded.

    4. Update the worker queue settings, and then click **Update Deployment**.

       The Airflow components of your Deployment automatically restart to apply the updated resource allocations. This action is equivalent to deploying code to your Deployment and doesn't impact running tasks that have 24 hours to complete before running workers are terminated. See [What happens during a code deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy).
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.

    2. Click the **Options** menu of the Deployment you want to update, and select **Edit Deployment**.

       <Frame>
         <img alt="Edit Deployment in options menu" />
       </Frame>

    3. Expand the **Execution** section if it isn't already expanded.

    4. Update the worker queue settings, and then click **Update Deployment**.

       The Airflow components of your Deployment automatically restart to apply the updated resource allocations. This action is equivalent to deploying code to your Deployment and doesn't impact running tasks that have 24 hours to complete before running workers are terminated. See [What happens during a code deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy).
  </Tab>
</Tabs>

<Tip>
  **Update Airflow Pool**

  If you see tasks getting stuck, it might be because the worker queue configuration doesn't align with the Airflow Pools, a component that allows you to control execution parallelism. Make sure to update your Airflow Pools to match the changes in potential maximum task parallelism caused by changes to the worker queue.

  For more information on limited parallelism, see [Airflow Pools](/docs/learn/airflow-pools).
</Tip>

## Delete a worker queue

<Info>
  **CLI**

  If you prefer, you can also run the `astro deployment worker-queue delete` command in the Astro CLI to delete a worker queue. See the [CLI Command Reference](/docs/cli/v1.43/astro-deployment-worker-queue-delete).
</Info>

All scheduled tasks that are assigned to a worker queue after the worker queue is deleted remain in a `queued` state indefinitely and won't execute. To avoid stalled task runs, ensure that you reassign all tasks from a worker queue before deleting it. You can either remove the worker queue argument or assign the task to a different queue.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**.

    2. Click the **More actions** menu (⋯) on the Deployment you want to edit, then click **Edit Deployment**.

    3. Expand the **Execution** section if it isn't already expanded.

    4. Click the trash can icon next to the worker queue you want to remove, then click **Update Deployment**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.

    2. Click the **Options** menu of the Deployment you want to update, and select **Edit Deployment**.

       <Frame>
         <img alt="Edit Deployment in options menu" />
       </Frame>

    3. Expand the **Execution** section if it isn't already expanded.

    4. Click **Remove queue** to delete the worker queue, and then click **Update Deployment**.
  </Tab>
</Tabs>


# Create a network connection between Astro and AWS
Source: https://astronomer.io/docs/astro/connect-aws

Create a network connection to AWS.

You can grant Astro cluster and its Deployments access to your external AWS resources.

Publicly accessible endpoints allow you to quickly connect your Astro clusters or Deployments to AWS through an Airflow connection. If your cloud restricts IP addresses, you can add the external IPs of your Deployment or cluster to an AWS resource's allowlist. See [Connect to a public AWS endpoint](/docs/astro/connect-aws-public)

If you have stricter security requirements, you can create a private connection to AWS in a few different ways. See [Private networking connections](#private-networking-connections) for more information.

After you create a connection from your cluster to AWS, you might also need to individually authorize Deployments to access specific resources. See [Authorize your Deployment using workload identity](/docs/astro/authorize-deployments-to-your-cloud).

## Standard and dedicated cluster support for AWS networking

Standard clusters have different connection options than dedicated clusters.

Standard clusters can connect to AWS in the following ways:

* Using [static external IP addresses](/docs/astro/connect-aws-public#allowlist-a-deployment’s-external-ip-addresses-on-aws)
* Using PrivateLink to connect with the following endpoints:
  * [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html) - Gateway Endpoint
  * [Amazon Simple Queue Service (SQS)](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-internetwork-traffic-privacy.html#sqs-vpc-endpoints) - Interface Endpoint    - [Amazon Elastic Container Registry (ECR)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html) - Interface Endpoints for ECR API and Docker Registry API
  * [Elastic Load Balancing (ELB)](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/load-balancer-vpc-endpoints.html) - Interface Endpoint
  * [AWS Security Token Service (AWS STS)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_sts_vpce.html) - Interface Endpoint

### Private networking connections

Dedicated clusters can connect to AWS in the same ways as standard clusters. Additionally, they support a number of private connectivity options including:

* [VPC peering](/docs/astro/connect-aws-vpc-peering)
* [Transit Gateways](/docs/astro/connect-aws-transit-gateways)
* [AWS PrivateLink](/docs/astro/connect-aws-privatelink)
* [VPN](/docs/astro/connect-aws-VPN)
* [Hostname resolution options](/docs/astro/connect-aws-hostname-resolution)

If you require a private connection between Astro and AWS, Astronomer recommends configuring a dedicated cluster. See [Create a dedicated cluster](/docs/astro/create-dedicated-cluster). Transitive connectivity to on-premise networks is also possible through your managed VPCs. However, architectures with a demarcation point between Astro and your on-premise network are not supported.

## See Also

* [Manage Airflow connections and variables](/docs/astro/manage-connections-variables)
* [Authorize your Deployment using workload identity](/docs/astro/authorize-deployments-to-your-cloud)


# AWS Networking: VPN
Source: https://astronomer.io/docs/astro/connect-aws-VPN

Create a VPN connection to AWS.

<Info>This connection option is only available for dedicated Astro clusters.</Info>

Use this connectivity type to access on-premises resources or resources in other cloud providers.

## Prerequisites

* An Astro Deployment with a dedicated cluster.
* Configured gateway device or application with Public IP address. You need 2 addresses for an HA setup.

Contact your internal network team or engineer and ask for the following information:

* Public IP addresses for the tunnels configuration.
* IKE pre-shared key, if your team wants to use a particular key.
* Preferable settings for phase 1 and phase 2 (BGP only) IKE negotiations.
* ASN for BGP or IP prefixes for static configuration.
* (Optional) A size /30 IPv4 CIDR block from the 169.254.0.0/16 range for the inside tunnel IPv4 addresses.

## Contact Astronomer support for VPN configuration on Astro side

Submit all collected details to [Astronomer support](https://cloud.astronomer.io/open-support-request). The Astronomer CRE team will proceed with the required steps. The CRE team will contact you using your support ticket to ask follow-up questions, request clarification, or let you know about connectivity tests.


# AWS Networking: Hostname resolution options
Source: https://astronomer.io/docs/astro/connect-aws-hostname-resolution

Hostname resolution options for AWS.

Securely connect Astro to resources running in other VPCs or on-premises through a resolving service.

Using Route 53 requires sharing a resolver rule with your Astro account. If this is a security concern, Astronomer recommends using Domain Name System (DNS) forwarding. If you have a small number of records and immutable IP addresses, the Astronomer support team can create a Private zone with DNS records, pointed to your resources.

<Tabs>
  <Tab title="Shared resolver rule">
    Use Route 53 Resolver rules to allow Astro to resolve DNS queries for resources running in other VPCs or on-premises.

    **Prerequisites**

    * An Amazon Route 53 Resolver rule. See [Managing forwarding rules](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-rules-managing.html).
    * Permission to share resources using the AWS Resource Access Manager (RAM)

    <Steps>
      <Step title="Share the Amazon Route 53 Resolver rule">
        To allow Astro to access a private hosted zone, you need to share your Amazon Route 53 Resolver rule with your Astro AWS account.

        1. In the Route 53 Dashboard, click **Rules** below **Resolver** in the navigation menu.

        2. Select a Resolver rule and then click **Details**.

        3. Click **Share** and enter `Astro` in the **Name** field.

        4. In the **Resources - optional** section, select **Resolver Rules**  in the **Select resource type** list and then select one or more rules.

        5. On the **Associate permissions** page, accept the default settings and then click **Next**.

        6. On the **Grant access to principals** page, select **Allow sharing only within your organization**, and then enter your Astro AWS account ID for your organization in the **Enter an AWS account ID** field.

           To get the Astro AWS account ID, In the Astro UI, click **Organization Settings**. From the **General** page, copy the **AWS External ID**.

        7. Click **Create resource share**.
      </Step>

      <Step Title="Contact Astronomer support for rule verification">
        To verify that the Amazon Route 53 Resolver rule was shared correctly, submit a request to [Astronomer support](https://cloud.astronomer.io/open-support-request). With your request, include the Amazon Route 53 Resolver rule ID. To locate the Resolver rule ID, open the Route 53 Dashboard, and in the left menu click **Rules** below **Resolver**. Copy the value in the Resolver **ID** column.
      </Step>
    </Steps>
  </Tab>

  <Tab title="Domain Name System forwarding">
    Use Domain Name System (DNS) forwarding to allow Astro to resolve DNS queries for resources running in other VPCs or on-premises. Unlike Route 53, you don't need to share sensitive configuration data with your Astro account. To learn more about DNS forwarding, see [Forwarding outbound DNS queries to your network](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-forwarding-outbound-queries.html).

    To use this solution, make sure Astro can connect to the DNS server using a VPC peering or transit gateway connection and then submit a request to [Astronomer support](https://cloud.astronomer.io/open-support-request). With your request, include the following information:

    * The domain name for forwarding requests
    * The IP address of the DNS server where requests are forwarded
  </Tab>

  <Tab title="Private hosted zone">
    Astronomer support can create Private hosted zones for reflecting particular DNS records in your environment without any changes or additional configurations. Private zones work well when the number of zones and records is small and stable. Otherwise, name resolution accuracy and connectivity in general can be affected.

    To use this solution, submit a request to [Astronomer support](https://cloud.astronomer.io/open-support-request). With your request, include the following information:

    * List of DNS records for the Private zone
    * IP addresses that have to be assigned to each DNS record
  </Tab>
</Tabs>

### (Optional) Create an Airflow connection to confirm connectivity

After Astronomer support confirms that DNS forwarding was successfully set up, you can confirm that it works by creating an Airflow connection to a resource running in a VPC or on-premises. See [Managing Connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html).


# AWS Networking: AWS PrivateLink
Source: https://astronomer.io/docs/astro/connect-aws-privatelink

Create a PrivateLink connection to AWS.

On Astro standard clusters, only the following AWS PrivateLink endpoints are supported:

* [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html) - Gateway Endpoint
* [Amazon Elastic Compute Cloud (Amazon EC2) Autoscaling](https://docs.aws.amazon.com/general/latest/gr/as.html) - Interface Endpoint
* [Amazon Elastic Container Registry (ECR)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/vpc-endpoints.html) - Interface Endpoints for ECR API and Docker Registry API
* [Elastic Load Balancing (ELB)](https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/load-balancer-vpc-endpoints.html) - Interface Endpoint
* [AWS Security Token Service (AWS STS)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_sts_vpce.html) - Interface Endpoint

<Info>Astro automatically supports cross-region connectivity for dedicated clusters that use AWS PrivateLink connections. Standard cross-region data transfer charges apply.</Info>

Use AWS PrivateLink to create private connections from Astro to your AWS services without exposing your data to the public internet.

All Astro clusters are pre-configured with the following AWS PrivateLink endpoints:

* Amazon S3 - Gateway Endpoint
* Amazon Elastic Compute Cloud (Amazon EC2) Autoscaling - Interface Endpoint
* Amazon Elastic Container Registry (ECR) - Interface Endpoints for ECR API and Docker Registry API
* Elastic Load Balancing (ELB)  - Interface Endpoint
* AWS Security Token Service (AWS STS) - Interface Endpoint

To enable PrivateLink connectivity between the Astronomer VPC and your VPC, you must open a support ticket with [Astronomer support](https://cloud.astronomer.io/open-support-request). PrivateLink isn't self-service.

To request additional endpoints, or assistance connecting to other AWS services, complete the following steps:

<Tabs>
  <Tab title="AWS Service Endpoint">
    * Prepare a list of your AWS Services that require Endpoints, such as SQS, Lambda, or DynamoDB.
    * Contact [Astronomer support](https://cloud.astronomer.io/open-support-request) and provide this information for next steps.

    By default, Astronomer support activates the **Enable DNS Name** option on supported AWS PrivateLink endpoint services.  With this option enabled, you can make requests to the default public DNS service name instead of the public DNS name that is automatically generated by the VPC endpoint service. For example, `*.notebook.us-east-1.sagemaker.aws` instead of `vpce-xxx.notebook.us-east-1.vpce.sagemaker.aws`. For more information about AWS DNS hostnames, see [DNS hostnames](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#:~:text=recursive%20DNS%20queries.-,DNS%20hostnames,-When%20you%20launch).
  </Tab>

  <Tab title="Custom VPC Endpoint">
    * To retrieve your cluster's AWS account ID on Astro, contact [Astronomer support](https://cloud.astronomer.io/open-support-request).
    * Go to your VPCe Service configuration and add `arn:aws:iam::<AWS account ID>:role/astronomer-remote-management` into a list of the Allowed principals.
    * Contact [Astronomer support](https://cloud.astronomer.io/open-support-request) and provide a VPCe Service name for the custom Service.
    * (Optional)  If Route53 alias is required for the proper connectivity by your service such as Snowflake or EKS, add a preferable DNS name for the Endpoint to your request. Otherwise, you can skip this step.
  </Tab>
</Tabs>

You'll incur additional AWS infrastructure costs for every AWS PrivateLink endpoint service that you use.  See [AWS PrivateLink pricing](https://aws.amazon.com/privatelink/pricing/).


# Connect to a public AWS Endpoint
Source: https://astronomer.io/docs/astro/connect-aws-public

Create a network connection to a public AWS endpoint.

All Astro clusters include a set of external IP addresses that persist for the lifetime of the cluster. When you create a Deployment in your workspace, Astro assigns the external IP addresses to it. To facilitate communication between Astro and your cloud, you can allowlist these external IPs in your cloud. If you have no other security restrictions, this means that any cluster with an allowlisted external IP address can access your AWS resources through a valid Airflow connection.

## Allowlist a Deployment's external IP addresses on AWS

1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
2. Select the **Details** tab.
3. In the **Other** section, you can find the **External IPs** associated with the Deployment.

Add the IP addresses to the allowlist of any external services that you want your Deployment to access.

When you use publicly accessible endpoints to connect to AWS, traffic moves directly between your Astro cluster and the AWS API endpoint. Data in this traffic never reaches the Astronomer managed control plane. Note that you still might also need to authorize your Deployment to some resources before it can access them. For example, you can [Authorize deployments to your cloud with workload identity](/docs/astro/authorize-deployments-to-your-cloud) so that you can avoid adding passwords or other access credentials to your Airflow connections.

<Accordion title="Dedicated cluster external IP addresses">
  If you use Dedicated clusters and want to allowlist external IP addresses at the cluster level instead of at the Deployment level, you can find the list of cluster-level external IP addresses in the **Clusters** page of the Astro UI.

  1. In the **Organization** section of the Astro UI, click **Organization Settings**, then click **Clusters**, then select a cluster.
  2. In the Details page, copy the IP addresses listed under **External IPs**.

  Add the IP addresses to the allowlist of any external services that you want your cluster to access. You can also access these IP addresses from the **Details** page of any Deployment in the cluster.

  After you allowlist a cluster's IP addresses, all Deployments in that cluster have network connectivity to AWS.
</Accordion>


# AWS Networking: Transit Gateways
Source: https://astronomer.io/docs/astro/connect-aws-transit-gateways

Create a Transit Gateway connection to AWS.

<Info>This connection option is only available for dedicated Astro clusters.</Info>

Use AWS Transit Gateway to connect one or more Astro clusters to other VPCs, AWS accounts, and on-premises networks supported by your organization.

AWS Transit Gateway is an alternative to VPC Peering on AWS. Instead of having to establish a direct connection between two VPCs, you can attach over 5,000 networks to a central transit gateway that has a single VPN connection to your corporate network.

While it can be more costly, AWS Transit Gateway requires less configuration and is often recommended for organizations connecting a larger number of VPCs. For more information, see [AWS Transit Gateway](https://aws.amazon.com/transit-gateway/).

AWS Transit Gateway doesn't provide built-in support for DNS resolution. If you need DNS integration, Astronomer recommends that you use the Route 53 Resolver service. For assistance integrating the Route 53 Resolver service with your Astronomer VPC, contact [Astronomer support](https://cloud.astronomer.io/open-support-request).

<Info>
  If your transit gateway is in a different region than your Astro cluster, contact [Astronomer support](https://cloud.astronomer.io/open-support-request). Astronomer support can create a new transit gateway in your AWS account for Astro and set up [a cross-region peering attachment](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-peering.html) with your existing transit gateway.

  If Astronomer creates a new transit gateway in your AWS account for Astro, keep in mind that your organization will incur additional AWS charges for the new transit gateway as well as the inter-region transfer costs.
</Info>

## Prerequisites

* A [dedicated Astro cluster](/docs/astro/create-dedicated-cluster). Transit Gateway connections are not available for standard clusters.
* An existing transit gateway in the same region as your Astro cluster.
* Permission to share resources using AWS Resource Access Manager (RAM).

## Setup

1. To retrieve your cluster's AWS account ID on Astro, contact [Astronomer support](https://cloud.astronomer.io/open-support-request).

2. In your AWS console, copy the ID of your existing transit gateway (TGW).

3. [Create a resource share in AWS RAM](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) and [share the TGW with your cluster's Astro AWS account](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-transit-gateways.html#tgw-sharing).

4. Contact [Astronomer support](https://cloud.astronomer.io/open-support-request) and provide the following information:

   * Your Astro cluster **ID**.
   * Your TGW ID from Step 2.
   * The CIDR block for the external VPC or on-premises network that you want to connect your Astro cluster with.

   Astronomer support approves the resource sharing request, attaches the Astro private subnets to your transit gateway, and creates routes in the Astro route tables to your transit gateway for each of the CIDR provided. Astronomer support notifies you about the process completion and provides you with the Astro CIDRs.

5. After you receive the confirmation from Astronomer support, use the Astro CIDRs to [create back routes](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-peering.html#tgw-peering-add-route) from your transit gateway to the Astro VPC.

6. Contact [Astronomer support](https://cloud.astronomer.io/open-support-request) to confirm that you have created the static route. Astronomer support then tests the connection and confirm.

7. (Optional) Repeat the steps for each Astro cluster that you want to connect to your transit gateway.


# AWS Networking: VPC Peering
Source: https://astronomer.io/docs/astro/connect-aws-vpc-peering

Create a VPC peering network connection to AWS.

Choose one of the following setups based on the security requirements of your company and your existing infrastructure.

<Info>This connection option is only available for dedicated Astro clusters.</Info>

### Prerequisites

* An external VPC on AWS
* A CIDR block for your external VPC in the RFC 1918 range
* [Organization Owner](/docs/astro/user-permissions#organization-roles) permissions

### Setup

To set up a private connection between an Astro VPC and an AWS VPC, you can create a VPC peering connection. VPC peering ensures private and secure connectivity, reduces network transit costs, and simplifies network layouts.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

<Tabs>
  <Tab title="New Astro UI">
    1. Open the AWS console of the AWS account with the external VPC and copy the following:

       * AWS account ID
       * AWS region
       * VPC ID of the external VPC
       * CIDR block of the external VPC

    2. In the Astro UI, go to **Settings** > **Clusters**, select your cluster, click **VPC Peering Connections**, then click **+ New VPC Peering Connection**.

    3. Configure the following values for your VPC peering connection using the information you copied in Step 1:

       * **Peering Name**: Provide a name for the VPC peering connection.
       * **AWS account ID**: Enter the account ID of the external VPC.
       * **Destination VPC ID**: Enter the VPC ID.
       * **Destination VPC region**: Enter the region of the external VPC.
       * **Destination VPC CIDR block**: Enter the CIDR block of the external VPC.

    4. Click **Create Connection**. The connection appears as **Pending**.

    5. Wait a few minutes for the **Complete Activation** button to appear, then click **Complete Activation**.

    6. In the modal that appears, follow the instructions to accept the connection from your external VPC and create routes from the external VPC to Astro.
  </Tab>

  <Tab title="Legacy UI">
    1. Open the AWS console of the AWS account with the external VPC and copy the following:

       * AWS account ID
       * AWS region
       * VPC ID of the external VPC
       * CIDR block of the external VPC

    2. In the Astro UI, click **Organization Settings**, then click **Clusters**, select your cluster, click **VPC Peering Connections**, then click **+ VPC Peering Connection**.

    3. Configure the following values for your VPC peering connection using the information you copied in Step 1:

       * **Peering Name**: Provide a name for the VPC peering connection.
       * **AWS account ID**: Enter the account ID of the external VPC.
       * **Destination VPC ID**: Enter the VPC ID.
       * **Destination VPC region**: Enter the region of the external VPC.
       * **Destination VPC CIDR block**: Enter the CIDR block of the external VPC.

    4. Click **Create Connection**. The connection appears as **Pending**.

    5. Wait a few minutes for the **Complete Activation** button to appear, then click **Complete Activation**.

    6. In the modal that appears, follow the instructions to accept the connection from your external VPC and create routes from the external VPC to Astro.
  </Tab>
</Tabs>

A few minutes after you complete the instructions in the modal, the connection status changes from **Pending** to **Active**. A new default route appears in **Routes** with your configured CIDR block.

<Info>
  **Troubleshooting VPC connection statuses**

  Astro might show additional information in your connection status if it has an issue when it creates the connection. The following are all possible connection statuses.

  * **Pending** (Without **Complete Activation**): Astro is sending the peering request to the external VPC. Wait 1-2 minutes for request to be created and sent.
  * **Pending** (With **Complete Activation**): The peering connection request has been created and sent. Click **Complete Activation** to finish the setup.
  * **Active**: The peering connection was successfully created and accepted.
  * **Failed**: The peering connection request was rejected. Delete the failed connection and retry using a new connection configuration. If you don't delete the failed connection, Astro will retry creating the peering request whenever you create a new VPC connection.
  * **Not Found**: Astro failed to create the peering request. Wait 5 minutes for Astro to retry. If the status hasn't changed after 5 minutes, delete the connection and retry using a new connection configuration.

  Note that a VPC connection can be listed as **Active** even when it has an incorrectly configured CIDR block. To reconfigure your CIDR block without deleting your connection, delete the route that was generated when you configured the connection and create a new route with the correct CIDR block.
</Info>

### Configure additional routes for a VPC connection

Your initial VPC connection connects Astro to your external VPC through a primary CIDR block. To connect Astro to other data services or systems within the external VPC, you can create additional routes to secondary CIDR blocks or subnets within the primary CIDR block. You can also complete this setup if you recently configured a new service in your external VPC and want to connect it with Astro without updating your base VPC connection.

<Tabs>
  <Tab title="New Astro UI">
    1. Open the **Routes** tab, then click **+ New Route**.

    2. Configure the following details for your route:

       * **Route ID**: Provide a name for the route.
       * **Destination**: Enter the subnet of the service in the external VPC.
       * **Target**: Select the VPC peering connection you configured.

    3. Click **Create Route**, then wait a few minutes for the route to be created.&#x20;
  </Tab>

  <Tab title="Legacy UI">
    1. Open the **Routes** tab, then click **+ Route**.

    2. Configure the following details for your route:

       * **Route ID**: Provide a name for the route.
       * **Destination**: Enter the subnet of the service in the external VPC.
       * **Target**: Select the VPC peering connection you configured.

    3. Click **Create Route**, then wait a few minutes for the route to be created.
  </Tab>
</Tabs>

#### DNS considerations for VPC peering

If your external VPC resolves DNS hostnames using **DNS Hostnames** and **DNS Resolution**, you must also enable the **Accepter DNS Resolution** setting on AWS. This allows Astro clusters and Deployments to resolve the public DNS hostnames of the external VPC to its private IP addresses. To configure this option, see [AWS Documentation](https://docs.aws.amazon.com/vpc/latest/peering/modify-peering-connections.html).

If your external VPC resolves DNS hostnames using [private hosted zones](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/hosted-zones-private.html), then you must associate your Route53 private hosted zone with the Astro VPC using instructions provided in [AWS Documentation](https://aws.amazon.com/premiumsupport/knowledge-center/route53-private-hosted-zone/).

To retrieve the ID of any Astro VPC, contact [Astronomer support](https://cloud.astronomer.io/open-support-request). If you have more than one Astro cluster, request the VPC ID of each cluster.


# Create a network connection between Astro and Azure
Source: https://astronomer.io/docs/astro/connect-azure

Create a network connection to Microsoft Azure.

Use this document to learn how you can grant an Astro cluster and its Deployments access to your external Azure resources.

Publicly accessible endpoints allow you to quickly connect your Astro clusters or Deployments to Azure through an Airflow connection. See [Create a connection to a public Azure endpoint](/docs/astro/connect-azure-public). If your cloud restricts IP addresses, you can add the external IPs of your Deployment or cluster to an Azure resource's allowlist.

If you have stricter security requirements, you can [create a private connection](#create-a-private-connection-between-astro-and-azure) to Azure in a few different ways.

After you create a connection from your cluster to Azure, you might also need to individually authorize Deployments to access specific resources. See [Authorize your Deployment using workload identity](/docs/astro/authorize-deployments-to-your-cloud).

## Standard and dedicated cluster support for Azure networking

Standard clusters have different connection options than dedicated clusters.

Standard clusters can connect to Azure in the following ways:

* Using [static external IP addresses](/docs/astro/connect-azure-public#allowlist-a-deployment’s-external-ip-addresses-on-azure).

### Create a private connection between Astro and Azure

Dedicated clusters can also connect to Azure using static IP addresses. Additionally, they support a number of private connectivity options including:

* [VNet peering](/docs/astro/connect-azure-vnet-peering)
* [VHub peering](/docs/astro/connect-azure-vhub-peering)
* [VPN](/docs/astro/connect-azure-vpn)
* [Azure Private Link](/docs/astro/connect-azure-private-link)
* [Hostname resolution options](/docs/astro/connect-azure-hostname-resolution)

If you require a private connection between Astro and Azure, Astronomer recommends configuring a dedicated cluster. See [Create a dedicated cluster](/docs/astro/create-dedicated-cluster).

## See Also

* [Manage Airflow connections and variables](/docs/astro/manage-connections-variables)


# Azure Networking: Hostname resolution options
Source: https://astronomer.io/docs/astro/connect-azure-hostname-resolution

Hostname resolution options for Azure.

Securely connect Astro to resources running in other VNets or on-premises through a resolving service.

Astronomer recommends using Domain Name System (DNS) forwarding as the most flexible and reliable solution. If you have a small number of records and immutable IP addresses, the Astronomer support team can create a Private zone with DNS records, pointed to your resources.

<Tabs>
  <Tab title="Domain Name System forwarding">
    Use Domain Name System (DNS) forwarding to allow Astro to resolve DNS queries for resources running in other VPCs or on-premises. You have access to internal resources through private names. All changes in zone will be available for Astro environment immediately.

    To use this solution, make sure Astro can connect to the DNS server using a VNet or VHub peering connection and then submit a request to [Astronomer support](https://cloud.astronomer.io/open-support-request). With your request, include the following information:

    * The domain name for forwarding requests
    * The IP address of the DNS server where requests are forwarded
  </Tab>

  <Tab title="Private hosted zone">
    Astronomer support can create Private hosted zones for reflecting particular DNS records in your environment without any changes or additional configurations. Private zones work well when the number of zones and records is small and stable. Otherwise, name resolution accuracy and connectivity in general can be affected.

    To use this solution, submit a request to [Astronomer support](https://cloud.astronomer.io/open-support-request). With your request, include the following information:

    * List of DNS records for the Private zone
    * IP addresses that have to be assigned respectively
  </Tab>
</Tabs>

#### (Optional) Create an Airflow connection to confirm connectivity

After Astronomer support confirms that DNS forwarding was successfully set up, you can confirm that it works by creating an Airflow connection to a resource running in a VPC or on-premises. See [Managing Connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html).


# Azure Networking: Azure PrivateLink
Source: https://astronomer.io/docs/astro/connect-azure-private-link

Create a PrivateLink connection to Azure.

<Info>This connection option is only available for dedicated Astro clusters.</Info>

Use Azure Private Link to create private connections from Astro to your Azure services without exposing your data to the public internet.

Astro clusters are pre-configured with the Azure blob private endpoint.

To request additional endpoints, or assistance connecting to other Azure services, provide [Astronomer support](https://cloud.astronomer.io/open-support-request) with the following information for the resource you want to connect to using Private Link:

* Resource name
* Resource ID
* [Group ID](https://github.com/MicrosoftDocs/azure-docs/blob/main/articles/private-link/private-endpoint-overview.md#private-link-resource)

For example, to connect with Azure Container Registry:

1. Follow the [Azure documentation](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-get-started-portal?tabs=azure-cli) to create the [container registry](https://portal.azure.com/#create/Microsoft.ContainerRegistry). Copy the name of container registry.
2. Follow the [Azure documentation](https://learn.microsoft.com/en-us/azure/container-registry/container-registry-private-link#create-a-private-endpoint---new-registry) to create a private endpoint for your container registry. Then, copy the name of the **Data endpoint**.
3. Then, from the left panel, go to **Overview** menu, and click on JSON view in **Essentials**, to copy the resource ID. You can also run Azure CLI command `az acr show -n myRegistry` to get the resource ID.
4. Contact [Astronomer Support](https://cloud.astronomer.io/open-support-request) with your request to connect. Provide the resource name, data endpoint name, and resource ID.
5. When Astronomer support adds an Azure private endpoint, corresponding private DNS zone and Canonical Name (CNAME) records are created to allow you to address the service by its private link name. Astronomer support will send the connection request in Azure Portal's [Private Link Center](https://portal.azure.com/#view/Microsoft_Azure_Network/PrivateLinkCenterBlade/~/pendingconnections).
6. Approve the connection requests from your Azure portal, then confirm that you've completed this in your support ticket. Astronomer support will then test whether the DNS resolves the endpoint correctly.

After Astronomer configures the connection, you can create Airflow connections to your resource. In some circumstances, you might need to modify your dags to address the service by its private link name (For example, `StorageAccountA.privatelink.blob.core.windows.net` instead of `StorageAccountA.blob.core.windows.net`).

Note that you'll incur additional Azure infrastructure costs for every Azure private endpoint that you use. See [Azure Private Link pricing](https://azure.microsoft.com/en-us/pricing/details/private-link/).


# Create a connection between Astro and Azure public endpoint
Source: https://astronomer.io/docs/astro/connect-azure-public

Create a network connection to a public endpoint on Microsoft Azure.

All Astro clusters include a set of external IP addresses that persist for the lifetime of the cluster. When you create a Deployment in your workspace, Astro assigns it one of these external IP addresses. To facilitate communication between Astro and your cloud, you can allowlist these external IPs in your cloud. If you have no other security restrictions, this means that any cluster with an allowlisted external IP address can access your Azure resources through a valid Airflow connection.

## Allowlist a Deployment's external IP addresses on Azure

1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
2. Select the **Details** tab.
3. In the **Other** section, you can find the **External IPs** associated with the Deployment.

Add the IP addresses to the allowlist of any external services that you want your Deployment to access.

When you use publicly accessible endpoints to connect to Azure, traffic moves directly between your Astro cluster and the Azure API endpoint. Data in this traffic never reaches the Astronomer managed control plane. Note that you still might also need to authorize your Deployment to some resources before it can access them. For example, you can [Authorize deployments to your cloud with workload identity](/docs/astro/authorize-deployments-to-your-cloud) so that you can avoid adding passwords or other access credentials to your Airflow connections.

<Accordion title="Dedicated cluster external IP addresses">
  If you use Dedicated clusters and want to allowlist external IP addresses at the cluster level instead of at the Deployment level, you can find the list of cluster-level external IP addresses in your Organization's **Clusters**.

  1. In the **Organization** section of the Astro UI, click **Organization Settings**, then click **Clusters**, then select a cluster.
  2. In the Details page, copy the IP addresses listed under **External IPs**.

  Add the IP addresses to the allowlist of any external services that you want your cluster to access. You can also access these IP addresses from the **Details** page of any Deployment in the cluster.

  After you allowlist a cluster's IP addresses, all Deployments in that cluster have network connectivity to Azure.
</Accordion>


# Azure Networking: VHub Peering
Source: https://astronomer.io/docs/astro/connect-azure-vhub-peering

Create a VHub peering network connection to Azure.

<Info>This connection option is only available for dedicated Astro clusters.</Info>

To set up a private connection between an Astro Virtual Network (VNet) and an Azure VHub, you can create a VHub peering connection. VHub peering ensures private and secure connectivity, reduces network transit costs, and attaches the Astro environment to a centralized managed network.

1. Retrieve the following information from the target Azure environment that you want to connect with:

   * Azure Tenant ID and Subscription ID.
   * VHub name.
   * Resource Group name.
   * Optional. Firewall IP address if you use any on the VHub side.

2. Prepare a `astro-vhub-peering-creator-role.json` JSON file with the following permissions. Replace `{customer-subscription-id}` with your value:

   ```json title="astro-vhub-peering-creator-role.json" wrap theme={null}
   {
       "Name": "Astro VHub Peering Contributor",
       "IsCustom": true,
       "Description": "Can create VNET peering with Astro.",
       "Actions": [
           "Microsoft.Resources/subscriptions/resourceGroups/read",
           "Microsoft.Resources/subscriptions/read",
           "Microsoft.Network/virtualHubs/hubVirtualNetworkConnections/write",
           "Microsoft.Network/virtualHubs/read",
           "Microsoft.Network/virtualWans/virtualHubs/read",
           "Microsoft.Network/virtualHubs/hubVirtualNetworkConnections/read"
       ],
       "NotActions": [

       ],
       "AssignableScopes": [
         "/subscriptions/{customer-subscription-id}"
       ]
   }
   ```

3. Run the following Azure CLI commands to give Astronomer Support temporary permissions to establish a VHub peering connection:

   ```sh wrap theme={null}
   # Add Astronomer Service Principal
   az ad sp create --id a67e6057-7138-4f78-bbaf-fd9db7b8aab0

   # Create a Custom role with permissions prepared in previous step
   az role definition create --role-definition ~/astro-vhub-peering-creator-role.json

   # Assign Custom role to the Astronomer Service Principal ({customer-subscription-id} has to be replaced with your value)
   az role assignment create \
   --assignee a67e6057-7138-4f78-bbaf-fd9db7b8aab0 \
   --role "Astro VHub Peering Contributor" \
   --scope "/subscriptions/{customer-subscription-id}"

   # Verify an assignment
   az role assignment list --assignee a67e6057-7138-4f78-bbaf-fd9db7b8aab0 --all -o table
   ```

4. Contact [Astronomer support](https://cloud.astronomer.io/open-support-request) to tell them that you granted them permissions to the Astronomer Service Principal. In addition, provide the following details in your request:

   * Astro Cluster ID
   * Azure Tenant ID and Subscription ID with a VHub
   * Resource group name
   * VHub name and preferable name for the peering
   * (Optional) Firewall IP address if you use any on the VHub side.

After receiving your request, Astronomer support creates a VHub peering connection to Astro VNet. No other actions are required from you. Astronomer support will notify you when the connection is ready to use.

When the network connection is confirmed, you can delete the temporary roles you created using the following command. Replace `{customer-subscription-id}` with your value:

```sh wrap theme={null}
az role assignment delete --assignee a67e6057-7138-4f78-bbaf-fd9db7b8aab0 --role "Astro VHub Peering Contributor" --scope "/subscriptions/{customer-subscription-id}"
```


# Azure Networking: VNet Peering
Source: https://astronomer.io/docs/astro/connect-azure-vnet-peering

Create a VNet peering network connection to Azure.

To set up a private connection between an Astro Virtual Network (VNet) and an Azure VNet, you can create a VNet peering connection. VNet peering ensures private and secure connectivity, reduces network transit costs, and simplifies network layouts.

## Prerequisites

* A dedicated Astro cluster
* Global Admin permissions in Azure to add the Astronomer Service Principal

## Set up VNet peering

1. Retrieve the following information from the target Azure environment that you want to connect with:

   * Azure Tenant ID and Subscription ID.
   * VNet name.
   * Resource Group name.

2. Prepare the `astro-vnet-peering-creator-role.json` JSON file with the following permissions. Replace `{customer-subscription-id}` with your value:

   ```json title="astro-vnet-peering-creator-role.json" wrap theme={null}
   {
       "Name": "Astro VNET Peering Contributor",
       "IsCustom": true,
       "Description": "Can create VNET peering with Astro.",
       "Actions": [
           "Microsoft.Resources/subscriptions/resourceGroups/read",
           "Microsoft.Resources/subscriptions/read",
           "Microsoft.Network/virtualNetworks/read",
           "Microsoft.Network/virtualNetworks/write",
           "Microsoft.Network/virtualNetworks/peer/action",
           "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/write",
           "Microsoft.Network/virtualNetworks/virtualNetworkPeerings/read"
       ],
       "NotActions": [

       ],
       "AssignableScopes": [
         "/subscriptions/{customer-subscription-id}"
       ]
   }
   ```

3. Run the following Azure CLI commands to give Astronomer support temporary permissions to establish a VNet peering connection:

   ```sh wrap theme={null}
   # Add Astronomer Service Principal
   az ad sp create --id a67e6057-7138-4f78-bbaf-fd9db7b8aab0

   # Create a Custom role with permissions prepared in previous step
   az role definition create --role-definition ~/astro-vnet-peering-creator-role.json

   # Assign Custom role to the Astronomer Service Principal ({customer-subscription-id} has to be replaced with your value)
   az role assignment create \
   --assignee a67e6057-7138-4f78-bbaf-fd9db7b8aab0 \
   --role "Astro VNET Peering Contributor" \
   --scope "/subscriptions/{customer-subscription-id}"

   # Verify an assignment
   az role assignment list --assignee a67e6057-7138-4f78-bbaf-fd9db7b8aab0 --all -o table
   ```

4. Contact [Astronomer support](https://cloud.astronomer.io/open-support-request) to tell them that you granted permissions to the Astronomer Service Principal. In addition, provide the following details in your request:

   * Astro Cluster ID
   * Azure Tenant ID and Subscription ID of the target VNet
   * Resource group name
   * VNet Name and preferred name for peering

After receiving your request, Astronomer support creates a VNet peering connection between the two VNets. No other actions are required from you. Astronomer support will notify you when the connection is ready to use.

When the network connection is confirmed, you can delete the temporary roles you created using the following command. Replace `{customer-subscription-id}` with your value:

```sh wrap theme={null}
az role assignment delete --assignee a67e6057-7138-4f78-bbaf-fd9db7b8aab0 --role "Astro VNET Peering Contributor" --scope "/subscriptions/{customer-subscription-id}"
```


# Azure Networking: VPN
Source: https://astronomer.io/docs/astro/connect-azure-vpn

Create a VPN connection to Azure.

<Info>This connection option is only available for dedicated Astro clusters.</Info>

Use this connectivity type to access on-premises resources or resources in other cloud providers.

## Prerequisites

Retrieve the following information about your VPN device or application:

* Public IP address
* Subnet CIDR range, multiple if needed, for your side of the connection
* Preferences regarding shared key and BGP usage
* IKE settings for the tunnel

## Contact Astronomer support for VPN configuration on Astro side

Submit all collected details to [Astronomer support](https://cloud.astronomer.io/open-support-request). The Astronomer CRE team will proceed with the required steps. The CRE team will contact you using your support ticket to ask follow-up questions, request clarification, or let you know about connectivity tests.


# Create and use data products with Astro Observe
Source: https://astronomer.io/docs/astro/create-data-products

Create observability tools on Astro to gain insight into the health and performance of your data pipelines and lifecycle.

In Observe, a data product is a composition of assets that deliver a result with business relevance. Observe automatically infers upstream dependencies for the assets that define a data product, like an Airflow dag that populates a dashboard or a Snowflake table powering a recommendation engine.

<Info>
  For guidance on identifying data products in your organization, see [how to identify data products in your organization](/docs/learn/data-products#how-to-identify-data-products-in-your-organization).
</Info>

The **Data Products** landing page provides an overview of all your data products and their SLA statuses. You can see at a glance which data products are late or stale, who owns them, and when they were last updated to see a snapshot of data product health.

<Frame>
  <img alt="An example of the data product list view." />
</Frame>

The following procedures describe how to create data products and leverage them to gain insight across pipelines and deployments. After you create a data product, then you can [view data product details](#view-your-data-product-details).

<Tip>You can follow a comprehensive walkthrough of setting up a Data Product and testing an alert in the [Get started with Observe](/docs/learn/data-products) quickstart.</Tip>

## Prerequisites

See [Observe prerequisites](/docs/astro/observe-get-started#prerequisites)

## Create a data product

<Steps>
  <Step title="Start data product creation">
    <Tabs>
      <Tab title="From the Asset Catalog">
        * In the sidebar, go to **Observe** > **Catalog** (**Asset Catalog** in the legacy UI).
        * Click **+ Create Data Product**.
      </Tab>

      <Tab title="From the Data Products page">
        * In the sidebar, go to **Observe** > **Data Products**.
        * Click **+ Data Product**.
      </Tab>
    </Tabs>
  </Step>

  <Step title="Select assets">
    * In the asset selection panel, use filters to find and select the final downstream assets for your data product.
    * As you select assets from the left panel, they will appear in the right preview area.
    * Click **Continue** after selecting one or more assets.
  </Step>

  <Step title="Define data product details">
    * Enter a **Name** and optionally a **Description** for your data product.
    * Choose a **Data Product Owner**: Select either **User** or **Team**, then specify the responsible user or team.
    * Review your selected assets and inspect the automatically generated asset lineage graph on the right side. You can still add assets to the data product after it is created.
    * Click **Create Data Product** to proceed.
  </Step>

  <Step title="Optionally configure a monitor">
    See [Create a data product monitor](/docs/astro/observe-monitors#data-product-monitors) for information on monitoring data products.
  </Step>
</Steps>

After completing these steps, your new data product will be available to view and manage under **Data Products**.

## View your data product details

After you create a data product, you can view in-depth details about the performance of your assets by selecting your data product for a closer look.

1. Click **Observe** in the Astro UI, and then click **Data Products**.

2. Choose the data product that you want to view details about.

When you see information about a specific data product, you can see summary performance data. This includes general statistics, like the average SLA success rate and information about when the data product was created and last updated. Additional details can be found in the following tabs.

### Overview

The overview tab provides summary information about the assets in your data product and the rates of overall SLA success rates, consolidated into daily, weekly, and monthly overall rates. This view allows you to quickly identify trends in historical data for your business-critical pipelines.

<Frame>
  <img alt="Example dashboard in the Overview page for a specific data product." />
</Frame>

### Event timeline

Each Data Product in Observe has an Event Timeline that reflects activity from the assets, like dag and task completion, and key issues, like breached SLAs and triggered alerts, in that data product. The event timeline supports filtering by event status along with more fine-grained filters. Selecting any particular event displays metadata about the event, including the notification history of a triggered alert.

The event timeline view allows you to see a record of events associated with your data product. These are categorized into **Success**, **Neutral**, and **Failure** events.

#### Success events

* SLA Success
* Task Success
* Task Start

#### Neutral events

* Airflow dataset write
* OpenLineage dataset write

#### Failure events

* Alert notification
* SLA Breach
* Task failure

For more information about diagnosing failures, see [Root cause analysis and AI log summaries](/docs/astro/root-cause-analysis).

### Assets

The **Assets** tab shows the assets included in the specific data product.

If you select an asset from the list to examine in-depth, you can see additional asset-specific details. See [Assets](/docs/astro/assets-overview) for descriptions.

### Metrics

Data products that include Airflow task assets report the following metrics by default:

* Task retries
* Task failures
* Task duration

### Alerts

The **Alerts** tab consolidates all alerts and SLAs that are specific to your selected data product. From this page you can [Add an alert](/docs/astro/observe-slas#create-an-alert), find an existing alert, and view the notification history for an alert.


# Create a dedicated Astro cluster
Source: https://astronomer.io/docs/astro/create-dedicated-cluster

Create a dedicated, single-tenant Astro cluster with custom regions, networking, and security controls for your Organization's Deployments.

<Note> This is feature is only available if you are on the **Team** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

A *dedicated cluster* exclusively runs Deployments from your Organization within a single-tenant environment on Astronomer's cloud. Dedicated clusters provide more configuration options for regions, connectivity, and security than standard clusters. You might want to create a dedicated cluster if:

* You need to connect to data services using private networking.
* You want more options for the region your cluster is hosted in.
* You otherwise want to keep your Deployments as isolated as possible.

Dedicated clusters offer the self-service convenience of a fully managed service while respecting the need to keep data private, secure, and within a single-tenant environment. If you don't need the aforementioned features, you can use one of the standard clusters when you [Create a Deployment](/docs/astro/create-deployment).

Each cloud provider has specific requirements for CIDR size, based on underlying infrastructure behavior and observations from customer deployments. Astro doesn't accept values smaller than these minimums.

Astronomer can't provide an exact match between network settings and workload size or the number of Deployments, because many factors influence address space utilization. Astronomer engineers provide specific recommendations and guidance after reviewing your use case requirements.

<Note>Organization Owners can require all new Deployments to use a dedicated cluster by enabling the **Enforce Dedicated Clusters** policy. See [Enforce dedicated clusters for new Deployments](/docs/astro/organization-settings#enforce-dedicated-clusters-for-new-deployments).</Note>

## Setup

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

<Tabs>
  <Tab title="AWS">
    <Tabs>
      <Tab title="New Astro UI">
        1. In the Astro UI, go to **Settings** > **Clusters**.

        2. Click **+ New Cluster**.

        3. Configure the following details about your cluster:

           * **Cloud Provider**: Select **AWS**.
           * **Name**: The name for your cluster.
           * **Region**: Select the region that you want your cluster to run in.
           * **VPC Subnet Range**: Provide a subnet range for Astro to connect to your existing AWS resources through VPC peering. The default is `172.20.0.0/20`, and the minimum size you can use is `/22`.

        4. (Optional) Expand **Advanced Configuration** to further define Kubernetes Pod Networking:

           Astro allocates a secondary VPC CIDR range specifically for Kubernetes Pod IPs within your cluster. Pods use source network address translation (SNAT) for connections outside the VPC, so your data sources see connections from the primary VPC subnet, not the Pod CIDR. Only traffic internal to the VPC, such as to RDS databases, appears from Pod IPs in the secondary CIDR.

           * **Pod CIDR Range**: Specify the range for Pod IP addresses. This range must not overlap with your VPC Subnet Range or any connected networks.

             <Info>
               The Pod CIDR range isolates Pod networking and mitigates Pod IP exhaustion. SNAT ensures that all external connections use the main VPC subnet, simplifying integration with external services.
             </Info>

        5. (Optional) In the **Disaster Recovery** section, configure cross-region disaster recovery:

           <Info>
             Cross-region disaster recovery requires the Enterprise Business Critical tier.
           </Info>

           * **Cross-Region Disaster Recovery**: Enable to create a secondary DR cluster in a different AWS region.
           * **Failover Region**: Select the AWS region for the secondary cluster.
           * **DR VPC Subnet Range**: Specify a different VPC subnet range for the secondary cluster, or leave blank to use the same range as the primary cluster.
           * **DR Pod CIDR Range**: Specify a different Pod CIDR range for the secondary cluster, or leave blank to use the same range as the primary cluster.
           * **Task Logs Replication SLA**: Enable to guarantee a 15-minute recovery point objective (RPO) for task logs. Additional charges apply.

             See [Disaster recovery](/docs/astro/disaster-recovery) for details on how these options affect your DR configuration.

        6. Click **Create Cluster**. After Astro finishes creating the cluster, users in your Organization can select the cluster when they [create a Deployment](/docs/astro/create-deployment).
      </Tab>

      <Tab title="Legacy UI">
        1. In the Astro UI, click **Organization Settings**, then click **Clusters**.

        2. Click **+ Cluster**.

        3. Configure the following details about your cluster:

           * **Cloud Provider**: Select **AWS**.
           * **Name**: The name for your cluster. You can change this name after creation.
           * **Region**: Select the region that you want your cluster to run in.
           * **VPC Subnet Range**: Provide a subnet range for Astro to connect to your existing AWS resources through VPC peering. The default is `172.20.0.0/20`, and the minimum size you can use is `/22`.

        4. (Optional) Expand **Advanced Configuration** to further define Kubernetes Pod Networking:

           Astro allocates a secondary VPC CIDR range specifically for Kubernetes Pod IPs within your cluster. Pods use source network address translation (SNAT) for connections outside the VPC, so your data sources see connections from the primary VPC subnet, not the Pod CIDR. Only traffic internal to the VPC, such as to RDS databases, appears from Pod IPs in the secondary CIDR.

           * **Pod CIDR Range**: Specify the range for Pod IP addresses. This range must not overlap with your VPC Subnet Range or any connected networks.

             <Info>
               The Pod CIDR range isolates Pod networking and mitigates Pod IP exhaustion. SNAT ensures that all external connections use the main VPC subnet, simplifying integration with external services.
             </Info>

        5. (Optional) In the **Disaster Recovery** section, configure cross-region disaster recovery:

           <Info>
             Cross-region disaster recovery requires the Enterprise Business Critical tier.
           </Info>

           * **Cross-Region Disaster Recovery**: Enable to create a secondary DR cluster in a different AWS region.
           * **Failover Region**: Select the AWS region for the secondary cluster.
           * **DR VPC Subnet Range**: Specify a different VPC subnet range for the secondary cluster, or leave blank to use the same range as the primary cluster.
           * **DR Pod CIDR Range**: Specify a different Pod CIDR range for the secondary cluster, or leave blank to use the same range as the primary cluster.
           * **Task Logs Replication SLA**: Enable to guarantee a 15-minute recovery point objective (RPO) for task logs. Additional charges apply.

             See [Disaster recovery](/docs/astro/disaster-recovery) for details on how these options affect your DR configuration.

        6. Click **Create cluster**. After Astro finishes creating the cluster, users in your Organization can select the cluster when they [create a Deployment](/docs/astro/create-deployment).
      </Tab>
    </Tabs>
  </Tab>

  <Tab title="GCP">
    <Tabs>
      <Tab title="New Astro UI">
        1. In the Astro UI, go to **Settings** > **Clusters**.

        2. Click **+ New Cluster**.

        3. Configure the following details about your cluster:

           * **Cloud Provider**: Select **GCP**.
           * **Name**: The name for your cluster.
           * **Region**: Select the region that your cluster runs in.
           * **VPC Subnet Range**: Specify the range used by nodes in your GKE cluster (Default: `172.20.0.0/22`). Astro uses this range to make private connections to your target data sources. The minimum size you can use is `/24`.

        4. (Optional) Configure the following **Advanced Configuration** details about your cluster for Private Networking.

           Astro uses source network address translation (SNAT) that performs many-to-one IP address translations for connections to your data sources and defaults secondary ranges to RFC 6598 address space (non-standard Private IP addresses), to minimize the risk and concern with IP overlap and exhaustion. When using private networking, like VPN or VPC Peering, between Astro and your target data sources, your target data sources see connections from the default **VPC Subnet Range**. If you're using private connections, confirm that the following **Advanced Configuration** network ranges don't overlap with your target data source networks.

           * **Pod Subnet Range**: Specify the range used by GKE Pods (Default: `100.64.0.0/16`).

           * **Service Subnet Range**: Specify the range used by Services in your GKE cluster (Default: `100.65.0.0/22`).

           * **Service Peering Range**: Specify the range used by Private Service connections (Default: `100.66.0.0/21`)

             If there is an overlap between the Advanced Configurations and your target data source networks, you can use the following alternative ranges:

           * **RFC 1918**:
             * 10.0.0.0/8, 10.0.0.0 – 10.255.255.255
             * 172.16.0.0/12, 172.16.0.0 – 172.31.255.255
             * 192.168.0.0/16, 192.168.0.0 – 192.168.255.255

           * **RFC 6598**: 100.64.0.0/10, specifically IP addresses from 100.64.0.0 to 100.127.255.255

        5. (Optional) In the **Disaster Recovery** section, configure cross-region disaster recovery:

           <Info>
             Cross-region disaster recovery requires the Enterprise Business Critical tier.
           </Info>

           * **Cross-Region Disaster Recovery**: Enable to create a secondary DR cluster in a different GCP region.
           * **Failover Region**: Select the GCP region for the secondary cluster. Only regions compatible with your primary region appear, based on [supported GCP dual-region pairings](https://docs.cloud.google.com/storage/docs/locations#location-dr). For example, `us-east5` pairs only with `us-central1` or `us-east1`.
           * **DR VPC Subnet Range**: Specify the VPC subnet range for the secondary cluster. This range is required and must not overlap with the primary **VPC Subnet Range**.
           * **DR Pod CIDR Range**: Specify the Pod range for the secondary cluster. This range is required and must not overlap with the primary **Pod Subnet Range**.
           * **DR Service Subnet Range**: Specify the Service range for the secondary cluster. This range is required and must not overlap with the primary **Service Subnet Range**.
           * **DR Service Peering Range**: Specify the Private Service connection range for the secondary cluster. This range is required and must not overlap with the primary **Service Peering Range**.
           * **Task Logs Replication SLA**: Enable to guarantee a 15-minute recovery point objective (RPO) for task logs. Additional charges apply.

             See [Disaster recovery](/docs/astro/disaster-recovery) for details on how these options affect your DR configuration.

        6. Click **Create Cluster**. After Astro finishes creating the cluster, users in your Organization can select the cluster when they [create a Deployment](/docs/astro/create-deployment).
      </Tab>

      <Tab title="Legacy UI">
        1. In the Astro UI, click **Organization Settings**, then click **Clusters**.

        2. Click **+ Cluster**.

        3. Configure the following details about your cluster:

           * **Cloud Provider**: Select **GCP**.
           * **Name**: The name for your cluster. You can change this name after creation.
           * **Region**: Select the region that your cluster runs in.
           * **VPC Subnet Range**: Specify the range used by nodes in your GKE cluster (Default: `172.20.0.0/22`). Astro uses this range to make private connections to your target data sources. The minimum size you can use is `/24`.

        4. (Optional) Configure the following **Advanced Configuration** details about your cluster for Private Networking.

           Astro uses source network address translation (SNAT) that performs many-to-one IP address translations for connections to your data sources and defaults secondary ranges to RFC 6598 address space (non-standard Private IP addresses), to minimize the risk and concern with IP overlap and exhaustion. When using private networking, like VPN or VPC Peering, between Astro and your target data sources, your target data sources see connections from the default **VPC Subnet Range**. If you're using private connections, confirm that the following **Advanced Configuration** network ranges don't overlap with your target data source networks.

           * **Pod Subnet Range**: Specify the range used by GKE Pods (Default: `100.64.0.0/16`).

           * **Service Subnet Range**: Specify the range used by Services in your GKE cluster (Default: `100.65.0.0/22`).

           * **Service Peering Range**: Specify the range used by Private Service connections (Default: `100.66.0.0/21`)

             If there is an overlap between the Advanced Configurations and your target data source networks, you can use the following alternative ranges:

           * **RFC 1918**:
             * 10.0.0.0/8, 10.0.0.0 – 10.255.255.255
             * 172.16.0.0/12, 172.16.0.0 – 172.31.255.255
             * 192.168.0.0/16, 192.168.0.0 – 192.168.255.255

           * **RFC 6598**: 100.64.0.0/10, specifically IP addresses from 100.64.0.0 to 100.127.255.255

        5. (Optional) In the **Disaster Recovery** section, configure cross-region disaster recovery:

           <Info>
             Cross-region disaster recovery requires the Enterprise Business Critical tier.
           </Info>

           * **Cross-Region Disaster Recovery**: Enable to create a secondary DR cluster in a different GCP region.
           * **Failover Region**: Select the GCP region for the secondary cluster. Only regions compatible with your primary region appear, based on [supported GCP dual-region pairings](https://docs.cloud.google.com/storage/docs/locations#location-dr). For example, `us-east5` pairs only with `us-central1` or `us-east1`.
           * **DR VPC Subnet Range**: Specify the VPC subnet range for the secondary cluster. This range is required and must not overlap with the primary **VPC Subnet Range**.
           * **DR Pod CIDR Range**: Specify the Pod range for the secondary cluster. This range is required and must not overlap with the primary **Pod Subnet Range**.
           * **DR Service Subnet Range**: Specify the Service range for the secondary cluster. This range is required and must not overlap with the primary **Service Subnet Range**.
           * **DR Service Peering Range**: Specify the Private Service connection range for the secondary cluster. This range is required and must not overlap with the primary **Service Peering Range**.
           * **Task Logs Replication SLA**: Enable to guarantee a 15-minute recovery point objective (RPO) for task logs. Additional charges apply.

             See [Disaster recovery](/docs/astro/disaster-recovery) for details on how these options affect your DR configuration.

        6. Click **Create cluster**. After Astro finishes creating the cluster, users in your Organization can select the cluster when they [create a Deployment](/docs/astro/create-deployment).
      </Tab>
    </Tabs>

    <Info>
      **Configure cluster maintenance windows**

      All GCP dedicated clusters are subscribed to the [GKE regular release channel](https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels), meaning that Google automatically upgrades the cluster and its nodes whenever an upgrade is available.

      After you create a GCP cluster, you can control when these upgrades happen by requesting a [maintenance window](https://cloud.google.com/kubernetes-engine/docs/how-to/maintenance-windows-and-exclusions#maintenance-window) for the cluster. Maintenance windows determine when and how Google updates your cluster. You can use maintenance windows to ensure that upgrades don't happen while critical Dags are running on your cluster.

      To set a maintenance window, first choose a maintenance window time and read through the [maintenance window considerations](https://cloud.google.com/kubernetes-engine/docs/how-to/maintenance-windows-and-exclusions#considerations) to make sure that the time is optimized for your cluster. Then, contact [Astronomer Support](https://cloud.astronomer.io/open-support-request) and provide your cluster ID and desired maintenance window.
    </Info>
  </Tab>

  <Tab title="Azure">
    <Tabs>
      <Tab title="New Astro UI">
        1. In the Astro UI, go to **Settings** > **Clusters**.

        2. Click **+ New Cluster**.

        3. Configure the following details about your cluster:

           * **Cloud Provider**: Select **Azure**.
           * **Name**: The name for your cluster.
           * **Region**: Select the region that you want your cluster to run in.
           * **VPC Subnet Range**: Provide a subnet range for Astro to connect to your existing Azure resources through a VNet connection. The default is `172.20.0.0/19`, and the minimum size you can use is `/20`.

        4. Click **Create Cluster**. After Astro finishes creating the cluster, users in your Organization can select the cluster when they [create a Deployment](/docs/astro/create-deployment).
      </Tab>

      <Tab title="Legacy UI">
        1. In the Astro UI, click **Organization Settings**, then click **Clusters**.

        2. Click **+ Cluster**.

        3. Configure the following details about your cluster:

           * **Cloud Provider**: Select **Azure**.
           * **Name**: The name for your cluster. You can change this name after creation.
           * **Region**: Select the region that you want your cluster to run in.
           * **VPC Subnet Range**: Provide a subnet range for Astro to connect to your existing Azure resources through a VNet connection. The default is `172.20.0.0/19`, and the minimum size you can use is `/20`.

        4. Click **Create cluster**. After Astro finishes creating the cluster, users in your Organization can select the cluster when they [create a Deployment](/docs/astro/create-deployment).
      </Tab>
    </Tabs>
  </Tab>
</Tabs>


# Create and assign custom Deployment roles
Source: https://astronomer.io/docs/astro/customize-deployment-roles

Customize your users' permissions for Airflow environments on Astro.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

A user entity's Deployment role determines their level of access to a specific Deployment in a Workspace. User entities with [Workspace Member permissions](/docs/astro/user-permissions#workspace-roles) or higher have some level of access to all Deployments in a Workspace, and these permissions can be increased using Deployment roles.

There are some circumstances where users should have limited access to Deployments in a Workspace. For example, all users might need full access to a development Deployment, but only administrative users need access to a production Deployment.

In situations where you need fine-grained Deployment access, you can create custom Deployment roles and assign them to users with Workspace Accessor or Workspace Member roles. If a user doesn't have a Workspace role when you assign them a Deployment role, Astro automatically gives them Workspace Accessor permissions. When you grant a user a Deployment role, they have a specific level of access to a specific Deployment. Use custom Deployment roles to enable users to collaborate in the same Workspace with only the minimum permissions they require.

<Tip>Watch the Astro Academy Learning Byte video for [Custom Deployment Roles](https://academy.astronomer.io/learning-bytes-custom-deployment-roles) to review some common use cases and learn more about how Astro implements RBAC.</Tip>

## Prerequisites

* Organization Owner permissions to create, update, and delete custom roles.
* Workspace Owner permissions or Deployment Admin permissions to assign and change Deployment roles for users.

See the [User permissions reference](/docs/astro/user-permissions) for more information about user roles.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Create a custom Deployment role

You manage and create custom Deployment roles at the Organization level. After you create a custom Deployment role, you can assign users, teams, and Deployment API tokens the role from any Deployment in the Organization.

Deployment roles are additive, meaning that a user with multiple Deployment roles has all of the permissions of each Deployment role as well as their Workspace role. For example, if a user belongs to a Team with a custom Deployment role that includes permissions to edit Airflow variables, and they also have a personal custom Deployment role that includes permissions to edit connections, then the user has permissions to edit both Airflow variables and connections in the Deployment.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Access Management** section, click **Roles & Permissions**.

    2. Click **Custom**, then click **+ New Custom Role**.

    3. In the **New Role** dialog, confirm that the **Scope** dropdown is set to **Deployment**. To create a role scoped to individual Dags instead, select **DAG**. See [Dag-level access control](/docs/astro/dag-level-access-control#create-a-custom-dag-role) for more about Dag-scoped roles.

    4. Enter a **Name** and **Description** for the role.

    5. In the **Permissions** table, select all permissions that you want the new role to have. See [Custom role permissions reference](/docs/astro/deployment-role-reference) for more information about each available permission.

       To base the new role on an existing one, use the **Copy from an existing role or template** dropdown above the table. See [Deployment role templates](#deployment-role-templates) for more information about the available default templates.

       <Tip>
         **Custom Deployment role permissions**

         To deploy with the Astro CLI when using a Deployment API token with a custom Deployment role, the minimum required permissions for the token's Deployment role are [`deployment.get`](/docs/astro/deployment-role-reference#deployment) and [`deployment.deploys.create`](/docs/astro/deployment-role-reference#deployment-deploys).
       </Tip>

    6. Click **Create Role**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings** > **Access Management**.

    2. Click **Roles**.

    3. Click **Custom** then click **+ Add Role**.

    4. In the window that appears, confirm that the **Scope** dropdown is set to **Deployment**. To create a role scoped to individual Dags instead, select **Dag**. See [Dag-level access control](/docs/astro/dag-level-access-control#create-a-custom-dag-role) for more about Dag-scoped roles.

    5. Enter a **Name** and **Description** for the role.

    6. In the **Permissions** table, check the boxes of all permissions that you want the new role to have. See [Custom role permissions reference](/docs/astro/deployment-role-reference) for more information about each available permission.

       Use the dropdown menu above the permissions table to automatically load the permissions of a templated role or an existing custom role as the basis for your new role. See [Deployment role templates](#deployment-role-templates) for more information about the available default templates.

       <Tip>
         **Custom Deployment role permissions**

         To deploy with the Astro CLI when using a Deployment API token with a custom Deployment role, the minimum required permissions for the token's Deployment role are [`deployment.get`](/docs/astro/deployment-role-reference#deployment) and [`deployment.deploys.create`](/docs/astro/deployment-role-reference#deployment-deploys).
       </Tip>

    7. Click **Create role**.
  </Tab>
</Tabs>

Your role is now available to assign at the Deployment level. See [Assign users and Teams to Deployments](#assign-users-and-teams-to-deployments) or [Create Deployment API tokens with custom Deployment roles](/docs/astro/deployment-api-tokens#create-a-deployment-api-token) for next steps.

### Deployment role templates

Astro provides a few Deployment role templates that you can use as the basis for custom roles. These roles aren't hard-coded and exist only as templates.

* **Deployment Viewer**: This is similar to the [Airflow Viewer](https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/auth-manager/access-control.html#viewer) role. It grants the user entity view-only permissions for the Airflow UI excluding the **Admin** tab.
* **Deployment Author**: This is similar to the [Airflow User](https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/auth-manager/access-control.html#user) role. It grants the user entity permissions to deploy code and manage Dag and task runs from the Airflow UI.
* **Deployment Operator**: This is similar to the [Airflow Op](https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/auth-manager/access-control.html#op) role. It grants the user entity permissions to update Deployment API tokens and Airflow objects from both the Airflow UI and the Astro UI.
* **Deployment Observe Ingest**: This permission role allows a user entity permissions to ingest OpenLineage events and metrics.

### Example: `MCP_VIEWER` role for the Airflow MCP Plugin

If you use the [Airflow MCP Plugin](/docs/astro/astro-mcp-server#airflow-mcp-plugin) to connect AI tools to your Deployment, create a custom `MCP_VIEWER` role with `deployment.get` and all `deployment.airflow.*.get` permissions. This is the least-privilege role that allows all MCP read tools to work. The MCP protocol requires POST requests, so the built-in `WORKSPACE_MEMBER` role doesn't work. See [Configure authentication](/docs/astro/astro-mcp-server#configure-authentication) for setup instructions.

## Assign users and Teams to Deployments

Using Deployment roles, you can add users and Teams directly to Deployments without first assigning them to a Workspace. If they don't already belong to the Workspace, Astro grants them the *Workspace Accessor* role. A Workspace Accessor only has permissions to access their assigned Deployments within the Workspace. All other Deployments and Workspace settings are hidden.

1. In the Astro UI, open the Deployment where you want to assign the user entity.
2. Click **Access**, then click **Users** or **Teams** depending on what kind of user entity you want to assign to the Deployment.
3. Click **+ User**/ **+ Team**.
4. In the window that appears, select the user entity you want to add, then select the role they will have in the Deployment.
5. Click **Add User**/ **Add Team**.

<Info>
  **Centralized access management for Organization Owners**

  Organization Owners can also add or update a user or Team for a Deployment from your Organization's access management settings:

  <Tabs>
    <Tab title="New Astro UI">
      1. In the Astro UI, go to **Settings** > **Access Management**.
      2. Select the user or Team.
      3. Click the **Deployments** tab, then click **+ Add Deployment** to add them to a Deployment, or **Edit Role** to change their role for an existing Deployment.
    </Tab>

    <Tab title="Legacy UI">
      1. In the Astro UI, click **Organization Settings** > **Access Management**.
      2. Select the user or Team to add or update.
      3. Click the **Deployments** tab, then click **+ Deployment** to add them to a Deployment, or open the **More actions** menu (⋯) and select **Edit role** next to an existing Deployment to change their role.
    </Tab>
  </Tabs>
</Info>

## Restrict a custom Deployment role to specific Workspaces

By default, a custom role is available to use in all Workspaces. After you create a custom Deployment role, you can restrict it so that users can only be assigned the role within specific Workspaces. Use Workspace role restriction when some Workspaces in your Organization have different requirements for how users interact with Deployments.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Access Management** section, click **Roles & Permissions**.
    2. Click **Custom**, then click the name of the custom role you want to restrict.
    3. Click the **Restricted Workspaces** tab.
    4. In the **Edit Restricted Workspaces** panel, toggle **Workspace Restriction** to **On**.
    5. Under **Restricted Workspaces**, select the checkboxes for the Workspaces you want to restrict the role to.
    6. Click **Update Restricted Workspaces**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings** > **Access Management**.
    2. Click **Roles**.
    3. Click **Custom**, then select the custom role that you want to restrict.
    4. In the menu that appears, click **Restricted Workspaces**, then click **Edit**.
    5. Click the **Workspace Restriction** toggle to on, then tick the checkbox for any Workspaces where you want the role to be usable.
    6. Click **Update Restricted Workspaces**.
  </Tab>
</Tabs>


# Dag-level access control
Source: https://astronomer.io/docs/astro/dag-level-access-control

Assign fine-grained, per-Dag roles to users, Teams, and API tokens on Astro.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

<Warning>
  **Dag-level access control requires Deployment-based forward auth URLs**

  If you use **Dag-level access control**, forward authentication must use the **Deployment-based endpoints**, not the older org-based endpoints.

  When you click **Navigate to Airflow** in the Astro UI, the correct Deployment-based URL appears in your browser address bar.

  Deployment-based URLs follow this format:

  `<deployment-id>.<last-two-characters-of-deployment-id>...`

  You only need to reference or configure these endpoints if you are setting up custom ingress, reverse proxies, or other advanced integrations.

  Dag-level access control permissions are enforced at the **Deployment level**, and using the older org-based URLs can result in incorrect permission enforcement or authentication failures. Note that you may still be feature-flagged to use the older URI format. Contact your account team to enable the new Deployment-based forward auth URLs to use Dag-level access control.
</Warning>

<Info>
  **Astro Runtime 3.1-12+**

  Dag-level access control requires Astro Runtime 3.1-12 or later. Deployments running earlier Runtime versions don't support Dag roles.
</Info>

Astro supports Dag-level role-based access control (RBAC), which adds a fourth tier to the Astro access control hierarchy: Organization > Workspace > Deployment > Dag. Dag roles grant per-Dag permissions to users, Teams, and API tokens within a specific Deployment, so you can enforce least-privilege security and enable multiple teams to collaborate in a single Deployment without exposing Dags across team boundaries.

When you assign a Dag role, you bind it to Dags using either **Dag tags** or **DAG IDs**:

* **Dag tags (recommended)**: Bind roles to one or more [Dag tags](https://airflow.apache.org/docs/apache-airflow/stable/howto/add-dag-tags.html). Any Dag with a matching tag is automatically included in the role binding. This is the recommended approach because new Dags that share the same tag are automatically covered without needing to update role assignments.
* **DAG IDs**: Bind roles to specific Dag IDs. Dag IDs are unique per Deployment. Use this approach when you need to grant access to a specific Dag that doesn't share tags with other Dags.

<Tip>
  Use Dag tags for your role bindings whenever possible. Tag-based bindings scale automatically as you add new Dags, so you won't need to update role assignments every time a new Dag is deployed. For example, tagging all Dags owned by a team with `team:analytics` lets you assign a single Dag role that covers all current and future Dags for that team.
</Tip>

## Prerequisites

* An Astro Deployment running Astro Runtime 3.1-12 or later.
* The user being assigned a Dag role must be an Organization Member. If the user doesn't already have a Workspace role, Astro automatically grants them the Workspace Accessor role when you assign them a Dag role. See [Workspace Accessor](/docs/astro/user-permissions#workspace-roles).
* Organization Owner permissions to create custom Dag roles.
* Workspace Owner or Deployment Admin permissions to assign Dag roles to users, Teams, and API tokens.

## Default Dag roles

Astro provides two default Dag roles that you can assign to users, Teams, and API tokens:

| Role           | Description                                                        |
| -------------- | ------------------------------------------------------------------ |
| **Dag Viewer** | Read-only access to a specific Dag and its resources.              |
| **Dag Author** | Read, edit, and delete access to a specific Dag and its resources. |

To create roles with more granular permissions, see [Create a custom Dag role](#create-a-custom-dag-role).

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Assign Dag roles to users

<Tip>You can also manage Dag roles from a Dag-centric view. See [View and manage Dag access from a Dag](#view-and-manage-dag-access-from-a-dag).</Tip>

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Settings**.
    2. Under **Access Management**, click **Users**, then click the user you want to manage.
    3. Click the **DAGs** tab.
    4. Click **+ Add DAG**.
    5. In the **Add User to DAG** slide-out, select a **Deployment**.
    6. Under **Target DAG by**, select **DAG Tag** or **DAG ID**. Astronomer recommends using Dag tags so that the role automatically applies to any new Dags with the same tag.
    7. Select the Dag tag or Dag ID you want to bind the role to.
    8. Select a **DAG Role** and click **Add to DAG**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings** > **Access Management**.
    2. Click **Users**, then click the user you want to manage.
    3. Click the **DAGs** tab.
    4. Click **+ Dag**.
    5. In the **Add User to DAG** slide-out, select a **Deployment**.
    6. Under **Target DAG by**, select **DAG Tag** or **DAG ID**. Astronomer recommends using Dag tags so that the role automatically applies to any new Dags with the same tag.
    7. Select the Dag tag or Dag ID you want to bind the role to.
    8. Select a **DAG Role** and click **Add to DAG**.
  </Tab>
</Tabs>

## Assign Dag roles to Teams

You can assign Dag roles to Teams so that all Team members share the same Dag-level permissions.

<Tip>You can also manage Dag roles from a Dag-centric view. See [View and manage Dag access from a Dag](#view-and-manage-dag-access-from-a-dag).</Tip>

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Settings**.
    2. Under **Access Management**, click **Teams**, then click the Team you want to manage.
    3. Click the **DAGs** tab, then click **+ Add DAG**.
    4. In the slide-out, select a **Deployment**.
    5. Under **Target DAG by**, select **DAG Tag** or **DAG ID**. Astronomer recommends using Dag tags so that the role automatically applies to any new Dags with the same tag.
    6. Select the Dag tag or Dag ID you want to bind the role to.
    7. Select a **DAG Role** and click **Add to DAG**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings** > **Access Management**.
    2. Click **Teams**, then click the Team you want to manage.
    3. Click the **DAGs** tab, then click **+ Dag**.
    4. In the slide-out, select a **Deployment**.
    5. Under **Target DAG by**, select **DAG Tag** or **DAG ID**. Astronomer recommends using Dag tags so that the role automatically applies to any new Dags with the same tag.
    6. Select the Dag tag or Dag ID you want to bind the role to.
    7. Select a **DAG Role** and click **Add to DAG**.
  </Tab>
</Tabs>

## Assign Dag roles to API tokens

You can assign Dag roles to Organization, Workspace, and Deployment API tokens to give them fine-grained access to specific Dags within a Deployment. Direct Access tokens can't be assigned Dag roles.

* **Organization API tokens**: Assign Dag roles from **Organization Settings** > **Access Management** > **API Tokens**. Click the token, then use the **DAGs** tab. See the following steps.
* **Workspace API tokens**: Assign Dag roles from the token's access management page. See [Manage Workspace API token access](/docs/astro/workspace-api-tokens#manage-workspace-api-token-access).
* **Deployment API tokens**: Assign Dag roles from the token's access management page. See [Manage Deployment API token access](/docs/astro/deployment-api-tokens#manage-deployment-api-token-access).

To assign a Dag role to an Organization API token:

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Settings**.
    2. Under **Access Management**, click **API Tokens**, then click the Organization API token you want to manage.
    3. Click the **DAGs** tab, then click **+ Add DAG**.
    4. In the slide-out, select a **Deployment**.
    5. Under **Target DAG by**, select **DAG Tag** or **DAG ID**. Astronomer recommends using Dag tags so that the role automatically applies to any new Dags with the same tag.
    6. Select the Dag tag or Dag ID you want to bind the role to.
    7. Select a **DAG Role** and click **Add to DAG**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings** > **Access Management**.
    2. Click **API Tokens**, then click the Organization API token you want to manage.
    3. Click the **DAGs** tab, then click **+ Dag**.
    4. In the slide-out, select a **Deployment**.
    5. Under **Target DAG by**, select **DAG Tag** or **DAG ID**. Astronomer recommends using Dag tags so that the role automatically applies to any new Dags with the same tag.
    6. Select the Dag tag or Dag ID you want to bind the role to.
    7. Select a **DAG Role** and click **Add to DAG**.
  </Tab>
</Tabs>

<Tip>You can also manage Dag roles from a Dag-centric view. See [View and manage Dag access from a Dag](#view-and-manage-dag-access-from-a-dag).</Tip>

## View and edit a user's Dag access

Organization Owners can view and manage all of a user's Dag role assignments from a centralized page.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Settings**.

    2. Under **Access Management**, click **Users**, then click the user whose Dag access you want to view.

    3. Click the **DAGs** tab.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings** > **Access Management**.

    2. Click **Users**, then click the user whose Dag access you want to view.

    3. Click the **DAGs** tab.
  </Tab>
</Tabs>

The **DAGs** tab lists explicit Dag role assignments across all Deployments. Users may also have additional access through their Deployment, Workspace, or Organization role, or through a Team membership. The table shows the following columns:

* **DAG ID**: The ID of the Dag the role is bound to.
* **DAG Tag**: The Dag tag the role is bound to.
* **Deployment**: The Deployment the binding belongs to.
* **DAG Role**: The Dag role assigned to the user.

To edit a user's Dag role:

1. Open the **More actions** menu (⋯) next to the Dag entry you want to update and select **Edit role**.

2. In the **Edit Dag Access** slide-out, select a new Dag role. The **DAG ID** and **Deployment** fields are read-only.

3. Click **Save changes**.

To remove a user's access to a Dag, open the **More actions** menu (⋯) and select **Remove**.

## View and manage Dag access from a Dag

Organization Owners can view and manage all role assignments for a specific Dag from the Dag's **Access Management** page. This provides a Dag-centric view of access, showing all users, Teams, and API tokens that have roles on a particular Dag.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **DAGs**, then click the Dag you want to manage.

    2. Click the **Access** tab, then toggle between **Users**, **Teams**, and **API Tokens**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open the **DAGs** page for your Deployment.

    2. Open the **More actions** menu (⋯) for the Dag you want to manage and click **Access Management**.
  </Tab>
</Tabs>

The **Access Management** page shows three tabs:

* **Users**: All users with Dag roles on this Dag.
* **Teams**: All Teams with Dag roles on this Dag.
* **API Tokens**: All API tokens with Dag roles on this Dag.

Roles assigned by **DAG ID** can be added, edited, or removed from this page. Roles assigned by **DAG Tag** are view-only because they are managed through the tag-based binding. To edit a tag-based role, go to **Organization Settings** > **Access Management** and manage the role from the entity's **DAGs** tab.

### Add a Dag role from the Dag Access Management page

1. Click the tab for the entity type you want to add (**Users**, **Teams**, or **API Tokens**).

2. Click **+ User**, **+ Team**, or **+ API Token**, depending on the selected tab.

3. In the slide-out:

   * For **Users** and **Teams**: Select the entity and a **DAG Role**, then click **Add**.
   * For **API Tokens**: Select a **Scope** (Deployment, Workspace, or Organization) to filter the available tokens. Select an **API Token** and a **DAG Role**, then click **Add**.

<Note>Direct Access tokens appear in the token dropdown but aren't selectable.</Note>

### Edit or remove a Dag role from the Dag Access Management page

1. Open the **More actions** menu (⋯) next to the entity you want to update.

2. Click **Edit role** to change the Dag role, or click **Remove from Dag** to remove the entity's access to the Dag.

<Tip>Click an entity's name on the **Access Management** page to navigate to its details page, where you can view and manage all of its role assignments.</Tip>

## Create a custom Dag role

You can create custom Dag roles with granular permissions at the Organization level. After you create a custom Dag role, you can assign it to users, Teams, and API tokens for any Dag in any Deployment in the Organization.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Settings**.

    2. Under **Access Management**, click **Roles & Permissions**.

    3. Click **Custom**, then click **+ New Custom Role**.

    4. In the slide-out that appears, set the **Scope** dropdown to **Dag**.

    5. Enter a **Name** and **Description** for the role.

    6. (Optional) Use the **Copy from an existing role** dropdown to load the permissions of a default Dag role or an existing custom role as a starting point.

    7. In the **Permissions** table, check the boxes for the permissions you want the role to have. See [Custom role permissions reference](/docs/astro/deployment-role-reference#dag-scope-permissions) for a complete list of available permissions.

    8. Click **Create Role**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**.

    2. Go to **Access Management**, then click **Roles**.

    3. Click **Custom**, then click **+ Add Role**.

    4. In the slide-out that appears, set the **Scope** dropdown to **Dag**.

    5. Enter a **Name** and **Description** for the role.

    6. (Optional) Use the **Copy from an existing role** dropdown to load the permissions of a default Dag role or an existing custom role as a starting point.

    7. In the **Permissions** table, check the boxes for the permissions you want the role to have. See [Custom role permissions reference](/docs/astro/deployment-role-reference#dag-scope-permissions) for a complete list of available permissions.

    8. Click **Create Role**.
  </Tab>
</Tabs>

Your custom Dag role is now available to assign to users, Teams, and API tokens at the Dag level in any Deployment.

### Custom Dag roles vs. custom Deployment roles

Custom Deployment roles and custom Dag roles both use the custom role creation flow in **Organization Settings** > **Access Management** > **Roles**, but they differ in scope:

* **Custom Deployment roles** grant permissions across all Dags and resources in a Deployment. See [Create and assign custom Deployment roles](/docs/astro/customize-deployment-roles).
* **Custom Dag roles** grant permissions to specific Dags within a Deployment, bound by Dag tag or Dag ID.

A user can have both a Deployment role and one or more Dag roles. Permissions are additive, meaning a user with multiple roles has the combined permissions of all their roles.

## Permission dependencies for custom Dag roles

Most Airflow operations require a specific combination of permissions to work. Assigning a resource-specific permission without the required base permission results in access being denied, even when the user appears to have the relevant permission.

Two base permission patterns apply across all resources:

* **Read operations** require `dag.airflow.dag.get` alongside the specific resource read permission. For example, to view Dag runs, a role needs both `dag.airflow.dag.get` and `dag.airflow.dagRun.get`.
* **Write operations** (create, update, delete) require `dag.airflow.dag.update` — not `dag.airflow.dag.get` — alongside the specific resource write permission. For example, to trigger a Dag run, a role needs `dag.airflow.dag.update` and `dag.airflow.dagRun.create`.

Some nested resources also require permissions for all parent resources. For example, viewing task logs requires `dag.airflow.dag.get`, `dag.airflow.dagRun.get`, `dag.airflow.taskInstance.get`, and `dag.airflow.taskLog.get`.

### Common permission sets

The following examples show minimum permission sets for common role types. These are verified against Airflow API behavior but cover a subset of operations. Not all Airflow endpoints are represented here.

<AccordionGroup>
  <Accordion title="Read-only access">
    View Dags, Dag runs, and task execution details without making changes.

    ```text wrap theme={null}
    dag.airflow.dag.get
    dag.airflow.dagRun.get
    dag.airflow.taskInstance.get
    dag.airflow.taskLog.get
    ```
  </Accordion>

  <Accordion title="Dag operator">
    Trigger, update, and delete Dag runs.

    ```text wrap theme={null}
    dag.airflow.dag.get
    dag.airflow.dag.update
    dag.airflow.dagRun.get
    dag.airflow.dagRun.create
    dag.airflow.dagRun.update
    dag.airflow.dagRun.delete
    ```
  </Accordion>

  <Accordion title="Task manager">
    View and manage individual task instances.

    ```text wrap theme={null}
    dag.airflow.dag.get
    dag.airflow.dag.update
    dag.airflow.dagRun.get
    dag.airflow.taskInstance.get
    dag.airflow.taskInstance.update
    dag.airflow.taskInstance.delete
    ```
  </Accordion>

  <Accordion title="Dag administrator">
    Full control over a Dag, including deletion.

    ```text wrap theme={null}
    dag.airflow.dag.get
    dag.airflow.dag.update
    dag.airflow.dag.delete
    dag.airflow.dagRun.get
    dag.airflow.dagRun.create
    dag.airflow.dagRun.update
    dag.airflow.dagRun.delete
    ```
  </Accordion>
</AccordionGroup>

<Warning>
  The [Airflow access control documentation](https://airflow.apache.org/docs/apache-airflow/2.8.3/security/access-control.html) covers a broader set of permissions but isn't consistently maintained and may not reflect actual enforcement behavior. Use it as a general reference and verify requirements in your Deployment.
</Warning>

## See also

* [User permissions reference](/docs/astro/user-permissions)
* [Create and assign custom Deployment roles](/docs/astro/customize-deployment-roles)
* [Custom role permissions reference](/docs/astro/deployment-role-reference)
* [Configure Teams on Astro](/docs/astro/manage-teams)
* [Create and manage Organization API tokens](/docs/astro/organization-api-tokens)


# Create and manage Deployment API tokens
Source: https://astronomer.io/docs/astro/deployment-api-tokens

Use Deployment API tokens to automate code deploys and configuration changes to a Deployment.

A Deployment API token is a credential that you can use to programmatically access a specific Deployment. They are a direct replacement for Deployment API keys, which aren't supported. Using a Deployment API token, you can:

* [Push code](/docs/astro/deploy-code) to a Deployment.
* Update the Deployment's [environment variables](/docs/astro/environment-variables).
* Update a Deployment's configurations. See [Manage Deployments as code](/docs/astro/manage-deployments-as-code).
* Make requests to update your Deployment's Airflow environment using the [Airflow REST API](/docs/astro/airflow-api).

Use this document to learn how to create and manage API tokens. To use your API token in an automated process, see [Authenticate an automation tool](/docs/astro/automation-authentication). For an overview of how Astro authenticates API requests, including JWT validation, token lifetimes, and rotation behavior, see [API authentication and token security](/docs/astro/api-authentication).

## Deployment API token permissions

Unlike Workspace API tokens and Organization API tokens, Deployment API tokens aren't scoped to a specific [user role](/docs/astro/user-permissions). Deployment API tokens have the same permissions as the [Workspace Operator](/docs/astro/user-permissions#workspace-roles) role, but only for Deployment-level operations. For example, an API token can create a Deployment [environment variable](/docs/astro/environment-variables) but, unlike a Workspace Operator, it can't create an [Astro alert](/docs/astro/alerts) because alerts apply to the whole Workspace.

<Info> You can manage Dag roles and the Deployment role for a Deployment API token from its access management page. See [Manage Deployment API token access](#manage-deployment-api-token-access). </Info>

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Create a Deployment API token

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.

    2. Click **Access** > **API Tokens**.

    3. Click **+ Add API Token** > **New Deployment API Token**.

    4. Configure the new Deployment API token:

       * **Name**: The name for the API token.
       * **Description**: (Optional) The Description for the API token.
       * **Type**: **Standard** or **Direct Access**. See [Direct access API tokens](#direct-access-token-api-tokens) for more information.
       * **Deployment Role**: (Enterprise plan only) Choose the Deployment-level role and permissions that the API token will have. See [Customize Deployment roles](/docs/astro/customize-deployment-roles).
       * **Expiration**: The number of days that the API token can be used before it expires.

    5. Click **Add API token**. A confirmation screen showing the token appears.

    6. Copy the token and store it in a safe place. You won't be able to retrieve this value from Astro again.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open your Workspace, then open the Deployment you want to create an API token for.

    2. Click **Access**.

    3. Click **API Tokens**, then click **+ Deployment API Token**. In the dropdown menu that appears, click **Add Deployment API token**.

    4. Configure the new Deployment API token:

       * **Name**: The name for the API token.
       * **Description**: (Optional) The Description for the API token.
       * **Type**: **Standard** or **Direct Access**. See [Direct access API tokens](#direct-access-token-api-tokens) for more information.
       * **Deployment Role**: (Enterprise plan only) Choose the Deployment-level role and permissions that the API token will have. See [Customize Deployment roles](/docs/astro/customize-deployment-roles).
       * **Expiration**: The number of days that the API token can be used before it expires.

    5. Click **Add API token**. A confirmation screen showing the token appears.

    6. Copy the token and store it in a safe place. You won't be able to retrieve this value from Astro again.
  </Tab>
</Tabs>

<a />

### Direct access API tokens

A direct access API token grants the ability to bypass the control plane in the event of a system outage. The role assigned to a direct access Deployment API token can't be changed after the token is created.

<Note>
  Only Organization Owners can create direct access Deployment API tokens.
</Note>

## Assign an Organization or Workspace API token to a Deployment

To centralize API token management, you can add an Organization or Workspace API token to a Deployment instead of creating a dedicated Deployment API token. Deployment-scoped API tokens are useful if you want to manage API tokens from the Organization level on a single screen, or you want to use a single API token for multiple Deployments.

Deployment-scoped API tokens are functionally identical to dedicated Deployment API tokens, except that you can only rotate, update, or delete them within their original scope.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.

    2. Click **Access** > **API Tokens**.

    3. Click **+ Add API Token**, then click **Assign Workspace API Token** or **Assign Organization API Token**.

    4. Configure the new Deployment API token:

       * **Workspace/ Organization API Token**: Select the API token you want to assign to the Deployment.
       * **Deployment Role**: Select the role that the API token has in the Deployment.

    5. Click **Update API Token**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open your Workspace, then open the Deployment you want to create an API token for.

    2. Click **Access**.

    3. Click **API Tokens**, then click **+ Deployment API Token**. In the dropdown menu that appears, click either **Assign Workspace API token** or **Assign Organization API token**.

    4. Configure the new Deployment API token:

       * **Workspace/ Organization API Token**: Select the API token you want to assign to the Deployment.
       * **Deployment Role**: Select the role that the API token has in the Deployment.

    5. Click **Update API Token**.
  </Tab>
</Tabs>

## Manage Deployment API token access

You can view and manage the roles for a Deployment API token from its access management page.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click **Access** > **API Tokens**.
    3. Click the row for the API token you want to manage.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open your Workspace, then open the Deployment that the API token belongs to.
    2. Click **Access**.
    3. Click **API Tokens**, then click the row for the API token you want to manage.
  </Tab>
</Tabs>

The token access management page shows the following information and management options:

* **Deployment Role**: View or update the token's Deployment role. Click **Edit** to change the role, then click **Update API Token**.
* **Dag Roles**: View and manage the token's Dag role assignments. Click **+ Dag** to add a Dag role. To edit or remove a Dag role, open the **More actions** menu (⋯) next to the Dag entry. See [Dag-level access control](/docs/astro/dag-level-access-control) for more information about Dag roles.

## Update or delete a Deployment API token

If you delete a Deployment API token, make sure that no existing CI/CD workflows are using it. After it's deleted, an API token can't be recovered. If you unintentionally delete an API token, create a new one and update any CI/CD workflows that used the deleted API token.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click **Access** > **API Tokens**.
    3. Open the **More actions** menu (⋯) next to your API token, then click **Edit Token**.
    4. Update the name or description of your token, then click **Update API Token**.
    5. (Optional) To delete a Deployment API token, click **Delete API Token**, enter `Delete`, and then click **Yes, Continue**. If you're editing a Deployment-scoped API token, click **Remove API token** instead to unassign the API token from the Deployment.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open your Workspace, then open the Deployment that the API token belongs to.
    2. Click **Edit** next to your API token.
    3. Update the name or description of your token, then click **Save Changes**.
    4. (Optional) To delete a Deployment API token, click **Delete API Token**, enter `Delete`, and then click **Yes, Continue**. If you're editing a Deployment-scoped API token, click **Remove API token** instead to unassign the API token from the Deployment.
  </Tab>
</Tabs>

## Rotate a Deployment API token

Rotating a Deployment API token lets you renew a token without needing to reconfigure its name, description, and permissions. You can also rotate a token if you lose your current token value and need it for additional workflows.

When you rotate a Deployment API token, you receive a new valid token from Astro that can be used in your existing workflows. The previous token value becomes invalid and any workflows using those previous values stop working.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click **Access** > **API Tokens**.
    3. Open the **More actions** menu (⋯) next to your API token, then click **Rotate Token**. Type in `ROTATE` to confirm, and click **Yes, Continue**. The Astro UI rotates the token and shows the new token value.
    4. Copy the new token value and store it in a safe place. You won't be able to retrieve this value from Astro again.
    5. In any workflows using the token, replace the old token value with the new value you copied.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open your Workspace, then open the Deployment that the API token belongs to.

    2. Click **Edit** next to your API token.

    3. Click **Rotate token**. The Astro UI rotates the token and shows the new token value.

    4. Copy the new token value and store it in a safe place. You will not be able to retrieve this value from Astro again.

    5. In any workflows using the token, replace the old token value with the new value you copied.
  </Tab>
</Tabs>

## Use a Deployment API token with the Astro CLI

To use a Deployment API token with Astro CLI, specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

```sh wrap theme={null}
export ASTRO_API_TOKEN=<your-token>
```

After you configure the `ASTRO_API_TOKEN` environment variable, you can run Astro CLI commands related to the Deployment for which the Deployment API token was created. For example, [`astro deployment inspect`](/docs/cli/v1.43/astro-deployment-inspect) or [`astro deployment logs`](/docs/cli/v1.43/astro-deployment-logs).

When using a Deployment API token for automation, Astronomer recommends storing `ASTRO_API_TOKEN` as a secret.

### Use a Deployment API token for CI/CD

You can use Deployment API tokens and the Astro CLI to automate various Deployment management actions in CI/CD.

For all use cases, you must make the following environment variable available to your CI/CD environment:

```text wrap theme={null}
ASTRO_API_TOKEN=<your-token>
```

After you set this environment variable, you can run Astro CLI commands from CI/CD pipelines without needing to manually authenticate to Astro. For more information and examples, see [Automate code deploys with CI/CD](/docs/astro/set-up-ci-cd).


# Deployment file reference
Source: https://astronomer.io/docs/astro/deployment-file-reference

View all possible values that you can include in a Deployment file when managing Deployments as code.

After you create an Astro Deployment, you can use the Astro CLI to create a Deployment file that contains its unique configurations represented as code. That includes worker queues, environment variables, Astro Runtime version, and more. You can use Deployment files to manage Deployments programmatically on Astro.

When you [inspect a Deployment](/docs/cli/v1.43/astro-deployment-inspect) to generate a Deployment file, its current configuration is generated as a YAML *Deployment file*. The file includes the name, description, and metadata that is unique to the Deployment.

A *Deployment template file* is different from the *Deployment file*. A template file does not have the `metadata` and `environment_variables` section, and the `name` and `description` fields are empty. Deployment template files are used to create new Deployments, while a Deployment file of an existing Deployment can be used to update its configuration. To create a Deployment template file in YAML format, run `astro deployment inspect <your-deployment-id> --template > your-deployment.yaml`.

Use this document as a reference for all fields in both Deployment files and Deployment template files.

## Deployment file example

<Info>To see the minimum values required to create a Deployment using a template file, see [Create a Deployment using a template file](/docs/astro/manage-deployments-as-code#create-a-deployment-using-a-template-file).</Info>

The following is an example Deployment file for Astro Hosted:

```yaml title="deployment.yaml" expandable wrap theme={null}
deployment:
    configuration:
        name: test
        description: ""
        runtime_version: 12.0.0
        dag_deploy_enabled: true
        ci_cd_enforcement: false
        scheduler_size: SMALL
        is_high_availability: false
        is_development_mode: false
        executor: CELERY
        scheduler_count: 1
        workspace_name: Demo Workspace
        deployment_type: STANDARD
        cloud_provider: AZURE
        region: westus2
        default_task_pod_cpu: "0.25"
        default_task_pod_memory: 0.5Gi
        resource_quota_cpu: "10"
        resource_quota_memory: 20Gi
        workload_identity: ""
    worker_queues:
        - name: default
          max_worker_count: 10
          min_worker_count: 0
          worker_concurrency: 5
          worker_type: A5
    metadata:
        deployment_id: cm0cvxk2r000108jt6vdw9sg2
        workspace_id: cm0cvy04p000308jt62nze0hc
        cluster_id: N/A
        release_name: N/A
        airflow_version: 2.10.0
        current_tag: 12.0.0
        status: CREATING
        created_at: 2024-08-23T20:00:40.72Z
        updated_at: 2024-08-23T20:00:40.72Z
        deployment_url: cloud.astronomer.io/cm0cvyx0m000408jt112i7aax/deployments/cm0cvz60c000508jtbgnrese4/overview
        webserver_url: org-astro-dev-ex.astronomer.run/cm0cvzely000608jt5v5p27bo
        airflow_api_url: org-astro-dev-ex.astronomer.run/cm0cvzowp000708jt2r7662bm/api/v1
    alert_emails:
        - test-emailclskz4wu5000508jz4gm25q5j@testdomain
    hibernation_schedules:
        - hibernate_at: 1 * * * *
          wake_at: 2 * * * *
          description: hibernation schedule 1
          enabled: true
```

The following sections describe configuration options for your Deployment file:

### `deployment.environment_variables`

You can create, update, or delete environment variables in the `environment_variables` section of the template file. This is equivalent to configuring environment variables in the Deployment's **Environment Variables** tab in your Deployment's **Environment** settings. Each variable in this section must include a `key` and a `value`.

By default, each variable is created as a non-secret variable. To set any new or existing environment variables as secret, specify `is_secret: true` in the same section as the key and value. For example:

```yaml wrap theme={null}
- is_secret: true
  key: PROJECT_NAME
  value: test_project
```

When you inspect a Deployment, the value of secret environment variables do not appear in the Deployment file.

To delete an environment variable, remove the lines that contain its key, its value, and other associated fields. Then, reapply the file to the Deployment. Any variables that exist on the Deployment, but are not included in the most recently applied Deployment file, are deleted.

If you commit a template file to a GitHub repository, do not add secret environment variables in the file. Instead, add them manually in the Astro UI. This ensures that you do not commit secret values to a version control tool in plain-text.

<Warning>When you add environment variables using a Deployment file, you must provide a `value` for your environment variable. Leaving this value blank or as an empty string (`""`) will cause the `astro deployment create` command to fail.</Warning>

### `deployment.configuration`

The `configuration` section contains all of the basic settings that you can configure from the Deployment **Details** page in the Astro UI. See:

* [Create a Deployment](/docs/astro/create-deployment#create-a-deployment).
* [Update a Deployment name and description](/docs/astro/deployment-details#update-a-deployment-name-and-description).
* [Scheduler](/docs/astro/deployment-resources#scheduler).

### `deployment.worker_queues`

The `worker_queues` section defines the [worker queues](/docs/astro/configure-worker-queues) for Deployments that use the Celery executor. This section is not applicable to Deployments that use Kubernetes executor.

If you don't enter specific values for the `default` worker queue for a Deployment, Astro uses default values based on the worker types available on your cluster. Each additional worker queue must include a `name` and `worker_type`. The Astro CLI uses default values for any other unspecified fields.

### Other fields

* `deployment_type` can be `HOSTED_SHARED` or `HOSTED_DEDICATED` for Astro Hosted depending on your [cluster type](/docs/cli/v1.43/astro-deployment-create#options-astro). Use `HOSTED_SHARED` for standard clusters and `HOSTED_DEDICATED` for dedicated clusters.
* `cluster_name` is the name for the cluster that appears in the Astro UI.


# View and address Deployment health incidents
Source: https://astronomer.io/docs/astro/deployment-health-incidents

A list of all possible Deployment incident types and steps for resolving each one.

Astro monitors your Deployment and displays notifications when issues arise that could affect your Deployment's functionality or performance. Use your Deployment health status to quickly check if your Deployment has any issues that need immediate attention.

## Deployment health

After you create a Deployment, its real-time health status appears at the top of the Deployment information page. Deployment health indicates if the components within your Deployment are running as expected.

<Frame>
  <img alt="Deployment Health status" />
</Frame>

The following are possible health statuses your Deployments can have:

* **Creating** (Grey): Astro is still provisioning Deployment resources. It is not yet available to run dags. See [Create a Deployment](/docs/astro/create-deployment).
* **Deploying** (Grey): A code deploy or environment update is in progress. Hover over the status indicator to view specific information about the deploy, including whether it was an image deploy or a dag-only deploy.
* **Healthy** (Green): The Airflow webserver and scheduler are both healthy and running as expected.
* **Unhealthy** (Red): Your Deployment webserver or scheduler are restarting or otherwise not in a healthy, running state.
* **Hibernating** (Grey): Your Deployment is currently hibernating.
* **Unknown** (Grey): The Deployment status can't be determined.

Your Deployment health status will also show a number next to the status if a [Deployment health incident](#deployment-incidents) is currently active. Incidents are classified as **Info**, **Warning** or **Critical** level.

<Frame>
  <img alt="A Deployment health status" />
</Frame>

If your Deployment is unhealthy or the status can't be determined, check the status of your tasks and wait for a few minutes. If your Deployment is unhealthy for more than five minutes, review the logs in the [Airflow component logs](/docs/astro/view-logs#view-airflow-component-logs-in-the-astro-ui) in the Astro UI or contact [Astronomer support](https://cloud.astronomer.io/open-support-request).

## Deployment incidents

Astro automatically monitors your Deployments and sends messages when your Deployment isn't running optimally or as expected. These messages are known as *Deployment incidents*. To view information about the incident, hover over the incident and click **View details**.

<Frame>
  <img alt="A Deployment Health incident message appearing after a user hovers over the Deployment health status" />
</Frame>

The following table contains all types of Deployment incidents. An **info** incident indicates that an issue has been identified but it will not impact the execution of dags or tasks. A **warning** incident indicates that specific tasks or dags might fail or that some action that should be taken on the Deployment. A **critical** incident indicates that your entire Deployment might not work as expected.

| Incident name                           | Severity        | Description                                                                                           |
| --------------------------------------- | --------------- | ----------------------------------------------------------------------------------------------------- |
| Scheduler Heartbeat Not Found           | Warning         | The Airflow scheduler has not sent a heartbeat for longer than 10 minutes.                            |
| Airflow Database Storage Unusually High | Info or Warning | The metadata database has tables that are larger than 50GiB (Info) or 75GiB (Warning).                |
| Deprecated Runtime Version              | Warning         | Your Deployment is using a deprecated Astro Runtime version.                                          |
| Job Scheduling Disabled                 | Warning         | The Airflow scheduler is configured to prevent automatic scheduling of new tasks using dag schedules. |
| Worker Queue at Capacity                | Warning         | At least one worker queue in this Deployment is running the maximum number of tasks and workers.      |

Use the following topics to address each of these incidents.

<Info>
  **Preview: Deployment Health Alerts**

  You can configure alerts for Deployment health incidents so that you are automatically notified via Slack, PagerDuty, or email if certain Deployment health incidents occur. See [Astro Alerts](/docs/astro/alerts) for setup information.
</Info>

### Scheduler Heartbeat Not Found

The scheduler has not sent a heartbeat for longer than 10 minutes. This could be a sign that the scheduler is down. Tasks will keep running, but new tasks will not be scheduled.

If you receive this incident notification, Astronomer Support has already been notified and no action is required from you. Ensure that you [configured a Deployment contact email](/docs/astro/deployment-details#configure-deployment-contact-emails) so that you can be notified if this issue requires additional follow-ups.

### Airflow Database Storage Unusually High

Your Deployment metadata database is currently storing tables that are larger than 50GiB (Info) or 75GiB (Warning). Click **View details** on the incident to view the affected tables. Even with large tables, Airflow will continue to operate as normal, but tables that are larger than 75GiB might cause degraded scheduler performance. As a result, you may need to clean up the relevant tables in the metadata database to avoid the risk of delayed task runs.

The tables that are currently monitored for size are:

* `dag`
* `dag_run`
* `task_instance`
* `job`
* `log`
* `xcom`

If `xcom` is listed in the affected tables, consider taking one of the following actions:

* Configure an external backend for XCom data, such as AWS S3. See the [Astronomer XCom Backend Tutorial](/docs/learn/custom-xcom-backends-tutorial).
* Implement intermediary data storage for tasks so that Airflow doesn't store large amounts of data when passing data between tasks. See [Intermediary data storage](/docs/learn/airflow-passing-data-between-tasks#intermediary-data-storage).

For additional assistance in cleaning up large tables, submit a request to [Astronomer Support](https://cloud.astronomer.io/open-support-request).

### Deprecated Runtime Version

The Astro Runtime version being used by the Deployment has been deprecated. Airflow will continue to run as normal, but you should upgrade as soon as possible.

To upgrade to a [supported version](/docs/runtime/runtime-version-lifecycle-policy), see [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime).

### Job Scheduling Disabled

The Airflow scheduler is currently disabled and will not automatically schedule new tasks. To run a new task in this state, you must manually trigger a dag run. To resume all scheduling, remove any overrides to the `AIRFLOW__SCHEDULER__USE_JOB_SCHEDULE` environment variable.

This variable might be configured in the [Astro UI](/docs/astro/manage-env-vars#use-the-astro-ui) or in your [Dockerfile](/docs/astro/manage-env-vars#using-your-dockerfile).

### Worker Queue at Capacity

At least one worker queue in your Deployment is running the maximum possible number of tasks and workers. Tasks will continue to run but new tasks will not be scheduled until worker resources become available. Click **View details** on the incident to view the affected worker queue(s).

To limit this notification for a worker queue, increase its **Max # Workers** setting or choose a larger **Worker Type**. See [Configure worker queues](/docs/astro/configure-worker-queues).

## See also

* [Deployment metrics](/docs/astro/deployment-metrics)
* [Astro alerts](/docs/astro/alerts)
* [Deployment logs](/docs/astro/view-logs)


# View metrics for Astro Deployments
Source: https://astronomer.io/docs/astro/deployment-metrics

Learn how to monitor Deployment performance, health, and total task volume in the Astro UI. These metrics can help you with resource allocation and issue troubleshooting.

The Astro UI exposes a suite of observability metrics that show real-time data related to the performance and health of your Deployments. These metrics are a useful reference as you troubleshoot issues and can inform how you allocate resources. They can also help you estimate the cost of your Deployments. This document explains each available metric and where to find them.

To view information about individual dags, see [dag and task runs](/docs/astro/manage-dags). To track Deployment health and specific incidents, see [Deployment health and incidents](/docs/astro/deployment-health-incidents).

<Info>
  **New Deployment analytics experience**

  This feature is in [Preview](/docs/astro/feature-previews) and includes expanded Airflow visibility and improved charting controls. You can switch between the new and previous experience from the **Analytics** page.
</Info>

## Deployment analytics

The **Analytics** page contains a suite of metrics for a given Deployment. This page includes metrics that give you insight into the performance of both your data pipelines and infrastructure. Because metrics are collected in real time, you can use this page to detect irregularities in your pipelines or infrastructure as they happen.

To view metrics for a Deployment, open the Deployment in the Astro UI, click **Analytics**. The following topics contain information about each available metric.

### Dag and task runs

These metrics contain information about your Deployment's dag runs and task runs over a given period of time.

<Frame>
  <img alt="dag run analytics in the Astro UI" />
</Frame>

#### Available metrics

* **Dag/ Task Runs**: This metric graphs the total number of dag/ task runs.
* **Runs per Status**: This metric graphs the number of failed and successful dag/ task runs, plotted based on the dag/ task run start time. Use this metric to see exactly when recent dag/ task runs succeeded or failed.

<Warning>
  The dag runs metric does not record dag run timeouts as failed runs. To see timed out dag runs, you must go into the Airflow UI to check on the statuses of each dag run there.
</Warning>

* **P90 Run Duration per Status**: This metric graphs the 90th percentile of execution times for dag/ task runs, plotted based on the dag/ task run start time. In the example above, the P90 Run Duration per Status for successful dag/ task runs at 5:00 was 34 seconds, which means that 90% of those dag/ task runs finished in 34 seconds or less.

  This metric can both help you understand how your pipelines are performing overall, as well as identify dag/ task runs that didn't result in a failure but still took longer to run than expected.

* **Ephemeral storage usage (*Kubernetes Executor/KubernetesPodOperator*)**: View how your Kubernetes tasks use your available ephemeral storage as a metric of the percentage used of total ephemeral storage configured. Click on **Dynamic y-axis scaling** to adjust the graph's y-axis to better fit your data or zoom in to view details.

### Airflow workers

These metrics contain infrastructure use information about your workers. Unique worker instances appear on these charts as different colored lines.

Hover over the graph to view a graph legend. If a given worker queue spins a worker down and back up again within a given interval, the newly spun up worker appears as a new color on the graph.

<Frame>
  <img alt="Worker analytics in the Astro UI" />
</Frame>

#### Available metrics

* **CPU Usage Per Pod (%)**: This metric graphs the peak CPU usage for all workers as a percentage of your maximum CPU capacity. Different worker Pods appear as differently colored lines on this chart. Hover over a given interval to view the specific number of CPUs being used.

* **Memory Usage Per Pod (MB)**: This metric graphs the peak memory usage for all workers as a percentage of your maximum memory capacity. Different worker Pods will appear as differently colored lines on this chart. This metric should be at or below 50% of your total allowed memory at any given time.

<Info>The number of workers per Deployment autoscales based on a combination of worker concurrency and the number of `running` and `queued` tasks. This means that the total available CPU and memory for a single Deployment may change at any time.</Info>

* **Network Usage Per Pod (MB)**: This metric graphs each worker/ scheduler Pod's peak network usage over time. Sudden, irregular spikes in this metric should be investigated as a possible error in your project code.

* **Pod Count per Status**: This metric graphs the number of worker and scheduler Pods in a given Kubernetes container state. Because Astro operates on a one-container-per-pod model, the state of the container state is also the Pod state. Use this metric to understand how many workers and schedulers are currently running in your Deployment. This can help you determine the values for **Worker Count (Min-Max)** that best fit your resource use.

  **Maximum Worker Count** applies only to workers in the `Running` state. This means that the number of worker Pods in **Pod Count per Status** might be greater than **Maximum Worker Count** at times if Astro scales down these workers at the same time that it creates new workers. For example, let's say that a Deployment has a **Maximum Worker Count** of 20. If you have five workers running tasks for an hour and you deploy code that requires 20 workers, Astro will trigger a scale-down event for the existing five workers and create 20 new workers to run tasks according to your new code. This means that your Deployment temporarily runs 25 workers.

  If a pod is stuck in a `Waiting` state, it can indicate that your Deployment did not successfully pull and run your Runtime image. For more information about container states, read the [Kubernetes documentation](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-states).

* **Ephemeral storage usage (*Celery workers*)**: View how your Celery worker uses your available ephemeral storage as a metric of the percentage used of total ephemeral storage configured. Click on **Dynamic y-axis scaling** to adjust the graph's y-axis to better fit your data or zoom in to view details.

### Airflow scheduler and Dag processor

These metrics contain infrastructure use information about your scheduler resources. Unique scheduler instances appear on these charts as different colored lines.

Hover over the graph to view a graph legend. If a given Deployment spins a scheduler down and back up again within a given interval, the newly spun up scheduler appears as a new color on the graph. If your Deployment includes a separate Dag processor, that resource also has its own line and appears in the legend.

<Frame>
  <img alt="Scheduler and Dag processor analytics in the Astro UI" />
</Frame>

#### Available metrics

* **CPU Usage Per Pod (%)**: This metric graphs the peak CPU usage for all schedulers as a percentage of your maximum CPU capacity. Different scheduler and Dag processor Pods appear as differently colored lines on this chart. Hover over a given interval to view the specific number of CPUs being used.

* **Memory Usage Per Pod (MB)**: This metric graphs the peak memory usage for all schedulers as a percentage of your maximum memory capacity. Different scheduler Pods appear as differently colored lines on this chart. This metric should be at or below 50% of your total allowed memory at any given time. For scheduler metrics, the maximum allowable memory for each scheduler Pod appears as a dotted red line.

* **Network Usage Per Pod (MB)**: This metric graphs each scheduler Pod's peak network usage over time. Sudden, irregular spikes in this metric should be investigated as a possible error in your project code.

* **Pod Count per Status**: This metric graphs the number of scheduler Pods in a given Kubernetes container state. Because Astro operates on a one-container-per-pod model, the state of the container is also the Pod state. Use this metric to understand how many schedulers and Dag processors are currently running in your Deployment.

  If a Pod is stuck in a `Waiting` state, it can indicate that your Deployment did not successfully pull and run your Runtime image. For more information about container states, read the [Kubernetes documentation](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-states).

* **Ephemeral storage usage per Pod**: View how your schedulers and Dag processors uses the available ephemeral storage as a metric of the percentage used of total ephemeral storage configured. Click on **Dynamic y-axis scaling** to adjust the graph's y-axis to better fit your data or zoom in to view details.

* **Scheduler Heartbeat**: A scheduler emits a heartbeat at a regular rate to signal that it's healthy to other Airflow components. This metric graphs a scheduler's average heartbeats per minute over a given time.

  On average, a scheduler should emit \~11-12 heartbeats per minute. A scheduler is considered "unhealthy" if it has not emitted a heartbeat for over 1 minute. The lack of a scheduler heartbeat is expected during a code push, but erratic restarts or an "Unhealthy" state that persists for a significant amount of time is worth investigating further.

### Airflow triggerer

These metrics provide visibility into the performance and resource use of Airflow triggerer components in your Deployment. The triggerer is responsible for handling deferred tasks and triggers in Airflow, and these metrics help you monitor their health and behavior.

<Warning>
  A known bug currently displays `%` as the unit for some Triggerer memory usage metrics. This will be resolved in an upcoming release.
</Warning>

#### Available metrics

* **Pod Count**: See the current number of triggerer Pods. Use this metric to monitor scaling events and ensure expected capacity.
* **CPU Usage Per Triggerer Pod (%)**: Visualize the peak CPU usage for each triggerer Pod as a percentage of your maximum CPU allocation. Both time series and table views of maximum and average are available.
* **Memory Usage Per Triggerer Pod (%)**: Visualize the peak memory usage for each triggerer Pod as a percentage of your maximum memory allocation. Both time series and table views of maximum and average are available.
* **Triggers per Status**: Track the number of triggers in each status over time, helping you understand how triggers progress and where issues may occur.
* **Running Triggerers**: Displays a timeline of running triggers.

<Tip>
  You can enable or disable dynamic Y-axis scaling on graphs for clearer visibility into spikes or sustained usage.
</Tip>

<Info>
  Triggerer metrics are available for export with [Universal Metrics Export](/docs/astro/export-metrics).
</Info>

### Pools

These metrics contain information about your Deployment's configured [Airflow pools](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/pools.html). They can give you insight into how your dags are handling concurrency.

<Frame>
  <img alt="Pool analytics in the Astro UI" />
</Frame>

#### Available metrics

* **Status Count for `<pool-name>`**: This metric graphs both the number of open slots in your pool and the number of tasks in each pool state:

  * **Open**: The number of available slots in the pool
  * **Queued**: The number of task instances which are occupying a pool slot and waiting to be picked up by a worker
  * **Running**: The number of tasks instances which are occupying a pool slot and running
  * **Starving**: The number of tasks that can't be scheduled when there are 0 available pool slots

  A large number of starving tasks could indicate that you should reconfigure your pools to run more tasks in parallel.

## Deployment overview

Each Deployment includes four high-level performance charts about the `default` worker queue which you can view from both the **Deployments** menu and a Deployment's **Overview** page. They include:

* dag runs
* Task Instances
* Worker CPU
* Worker Memory

<Frame>
  <img alt="Metrics dashboard in the Astro UI" />
</Frame>

The data in these four charts is recorded hourly and is displayed in both UTC and your local browser timezone. Each bar across all graphs covers a complete hour while the entire time window for a single graph is 24 hours. For example, a single bar might represent `16:00` to `17:00` while the entire time window of the graph might represent `Nov 1 16:00` to `Nov 2 16:00`.

The data for the most recent hour is for the hour to date. For example, if you are looking at this page at 16:30, then the bar for the `16:00-17:00` hour interval would show data for `16:00-16:30`.

These charts serve as high-level reports for your `default` worker queue that you can investigate further.

The following sections describe each of the 4 available charts.

### Total dag runs

The **Dag Runs** metric records successful and failed dag runs over hour-long intervals. A [dag run](https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html) is defined as an instantiation of a dag at a specific point in time.

You can hover over each bar to see the corresponding hour interval displayed in both UTC and your local timezone. Below that, you can see the number of successful dag runs and the number of failed dag runs. If a bar is partially or fully red, it means that one or more dag runs failed within that hour interval.

The bolded value above the graph denotes the total number of dag runs that have been executed in the last 24 hours.

### Task instances

The **Tasks** chart records successful and failed task instances over hour-long intervals. A [task instance](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#task-instances) is defined as an instantiation of a task at a specific point in time.

You can hover over each bar to see the corresponding hour interval displayed in both UTC and your local timezone. Below that, you can see the number of successful and failed task instances. If a bar is partially or fully red, it means that one or more task instances failed within that hour interval.

The bolded value above the graph denotes the total number of tasks that have run in the last 24 hours.

### Resource usage

The **Worker CPU** and **Worker Memory** charts in the Astro UI provide visibility into the resources being consumed by the workers in your Deployment as measured by CPU and memory consumption.

**Worker CPU** records the peak CPU usage, while **Worker Memory** records the peak memory usage by worker nodes over hour-long intervals. The bolded values above each graph show the maximum CPU/ memory usage by a single worker at any point in time over the last 24 hours.

<Info>A known issue with Celery might cause worker memory allocation to increase without a corresponding increase in task count or dag memory use. To correct this issue, redeploy your code without making any changes to restart the Celery worker and reset memory requests. See [Deploy code to Astro](/docs/astro/deploy-code). This issue should not affect your tasks or cause OutOfMemory (OOM) errors. If you experience complications, contact Astronomer Support.</Info>

## See also

* [View Organization dashboards](/docs/astro/organization-dashboard)
* [Export task logs and metrics to Datadog](/docs/astro/export-datadog)
* [Export task logs to AWS CloudWatch](/docs/astro/export-cloudwatch)
* [Export logs to a Secondary S3 Bucket](/docs/astro/export-secondary-s3-bucket)
* [Export logs to a Secondary WASB Container](/docs/astro/export-secondary-wasb)


# Environment variables
Source: https://astronomer.io/docs/astro/environment-variables

Overview of environment variables on Astro

On Astro, an *Environment Variable* is a key-value configuration that can be set at the Deployment or Workspace level. You can use environment variables to configure custom environment variables for your Deployments, customize core settings of Airflow and its pre-installed providers, or store Airflow connections and variables.

Some scenarios where you might use environment variables in your Deployment include:

* Adding a token or a URL that is required by your Airflow Dags or tasks.
* Integrating with Datadog or other third-party tooling to [export Deployment metrics](/docs/astro/export-datadog).
* Specifying a tag that's added to all resources created by the Deployment and indicates whether a resource is for development or production.

Some examples of customizing core settings of Airflow or any of its pre-installed providers include:

* Changing the import timeout of DAGBag using `AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT`.
* Setting up an SMTP service to receive [Airflow alerts](/docs/astro/airflow-email-notifications) by email.

You can also use environment variables to store [Airflow connections](/docs/learn/connections#define-connections-with-environment-variables) and [variables](/docs/learn/airflow-variables#using-environment-variables).

Some environment variables on Astro are set globally and can't be overridden for individual Deployments, while others are used by Astro Runtime to enhance your Airflow experience. For more information on these, see [Global environment variables](/docs/astro/platform-variables).

## Choose a strategy

Environment variables can be used in many different contexts on Airflow. To choose the right management and implementation strategy for your specific use case, it's helpful to know how Astro prioritizes and stores environment variables for each available management option.

### Management options

On Astro, you can manage environment variables from three different locations:

* **Workspace Environment Manager**: Create environment variables at the Workspace level and link them to multiple Deployments. This allows you to define common environment variables once and share them across Deployments, with the ability to override values per Deployment. See [Create environment variables in Astro](/docs/astro/create-and-link-environment-variables) for setup steps.
* **Deployment Environment Variables tab**: Set environment variables specific to a single Deployment through your Deployment's **Environment Variables** tab in your Deployment's **Environment** settings in the Astro UI. This is the fastest way to set a Deployment-specific environment variable. See [Using the Astro UI](/docs/astro/manage-env-vars#use-the-astro-ui) for setup steps.
* **Astro project Dockerfile**: Store environment variables in your Dockerfile to manage them as code in a version control tool like GitHub. However, environment variables stored in the Dockerfile don't appear in the Astro UI and might be harder to reference from Dag code. Using the Dockerfile is recommended for more complex production use cases, such as implementing a secrets backend. See [Using your Dockerfile](/docs/astro/manage-env-vars#using-your-dockerfile) for setup steps.

At the local development level, you can use your Astro project `.env` file to set and test environment variables. When you're ready to push these environment variables to a Deployment, you can use the Astro CLI to export and store them in the Astro UI for your Deployment.

### How environment variables are stored in the Astro UI

#### Deployment-level environment variables

When you set a non-secret environment variable in the Deployment UI, Astronomer stores the variable in a database that is hosted and managed by Astronomer.

When you set a secret environment variable in the Deployment UI, the following happens:

1. Astro generates a manifest that defines a Kubernetes secret, named `env-secrets`, that contains your variable's key and value.
2. Astro applies this manifest to your Deployment's namespace.
3. After the manifest is applied, the key and value of your environment variable are stored in a managed [etcd cluster](https://etcd.io/) at rest within Astro.

This process occurs every time you update the environment variable's key or value. The Deployment restarts when Deployment-level environment variables are saved.

#### Workspace-level environment variables

When you create an environment variable in the Workspace Environment Manager, Astro stores the environment variable in an Astronomer-hosted secrets manager and applies it to Deployments as Kubernetes Secrets. See [How environment variables are stored](/docs/astro/create-and-link-environment-variables#how-environment-variables-are-stored) for details.

<Warning>
  Environment variables marked as secret are stored securely by Astronomer and are not shown in the Astro UI. However, it's possible for a user in your organization to create or configure a Dag that exposes secret values in Airflow task logs. Airflow task logs are visible to all Workspace members in the Airflow UI and accessible in your Astro cluster's storage.

  To avoid exposing secret values in task logs, instruct users to not log environment variables in Dag code.
</Warning>

### Environment variable priority

On Astro, environment variables are applied in the following order of precedence, from highest to lowest:

1. **Deployment-level environment variables** (set in the Deployment's Environment Variables tab in the Astro UI)
2. **Workspace-level environment variables** (set in the Workspace Environment Manager)
3. **Environment variables in your Dockerfile**

For example, if you set `AIRFLOW__CORE__PARALLELISM` with one value in the Deployment UI, another value in the Workspace Environment Manager, and a third value in your `Dockerfile`, the value set in the Deployment UI takes precedence.

When you view environment variables in a Deployment's Environment Variables page, you'll see a unified view of both Deployment and Workspace environment variables, with clear indicators showing the source and any overrides.

Similarly, in local development, environment variables set in your `.env` file take precedence over environment variables set in your Dockerfile.

### Example use cases

For most use cases, Astronomer recommends using the Astro UI to store your environment variables for the following reasons:

* It's easy to use.
* It has built-in security for secret environment variables.
* You can import and export environment variables using the Astro CLI.

There are some scenarios when you might want to use a mix of methods or strategies other than the Astro UI. The following table prescribes specific methods for various common use cases.

| Scenario                                                                                                                                                              | Recommended method                                                                                                                                                                                                                                                                                |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| You are a new Astro user. You just created a Deployment and want to integrate your Hashicorp Vault secrets backend to test Dags.                                      | [Deployment UI](/docs/astro/manage-env-vars#use-the-astro-ui) for ease of use and visibility.                                                                                                                                                                                                          |
| Your team has multiple environment types and you use an `ENVIRONMENT_TYPE` environment variable in your Dags to customize the file, bucket, or database schema names. | [Deployment UI](/docs/astro/manage-env-vars#use-the-astro-ui) for visibility.                                                                                                                                                                                                                          |
| You want to share common environment variables across multiple Deployments in a Workspace, with the ability to override values per Deployment.                        | [Workspace Environment Manager](/docs/astro/create-and-link-environment-variables) to centrally manage and share environment variables.                                                                                                                                                                |
| You want to standardize a core set of environment variables for all new Deployments in a Workspace.                                                                   | [Workspace Environment Manager](/docs/astro/create-and-link-environment-variables) with auto-linking enabled to apply environment variables to all current and future Deployments.                                                                                                                     |
| You want to standardize a core set of Airflow environment configurations across multiple environments in your code.                                                   | [`Dockerfile`](/docs/astro/manage-env-vars#using-your-dockerfile) to have a file that serves as a source of truth and starting point for all environments.                                                                                                                                             |
| You want to use version control to manage your environment variables from a code repository.                                                                          | [`Dockerfile`](/docs/astro/manage-env-vars#using-your-dockerfile) to version control your environment configuration.                                                                                                                                                                                   |
| You want to carry over environment variables when you promote code from a development environment to a production environment.                                        | [`Dockerfile`](/docs/astro/manage-env-vars#using-your-dockerfile) to deploy environment variables from project code using CI/CD.                                                                                                                                                                       |
| You are part of a production support team analyzing Dag failures and want to turn on debugging-related environment variables temporarily.                             | [Deployment UI](/docs/astro/manage-env-vars#use-the-astro-ui) for ease of use.                                                                                                                                                                                                                         |
| You're developing locally, and you want to use a secret credential in your Dags.                                                                                      | [Use the `.env` file](/docs/astro/manage-env-vars#manage-environment-variables-locally) locally. This allows you to avoid accidentally sending credentials to your code repository because `.env` is part of `.gitignore`. You can then export the `.env` file to your Deployment using the Astro CLI. |
| You use environment variables to store your Airflow connections and variables, and you have to reconfigure these between Deployments based on the environment type.   | [Deployment UI](/docs/astro/manage-env-vars#use-the-astro-ui) for visibility.                                                                                                                                                                                                                          |
| You want the features of the Astro UI for environment variables, but you also keep track of the non-secret environment variables in your code repository.             | Use the [Astro CLI](/docs/astro/manage-env-vars#use-the-astro-ui) and an automation script to add or update the non-secret environment variables in your Deployment from your code repository.                                                                                                         |


# Manage Airflow executors on Astro
Source: https://astronomer.io/docs/astro/executors-overview

Compare the Astro, Celery, and Kubernetes executors to choose the right one for your Deployment.

The Airflow executor determines which worker resources run your scheduled tasks.

On Astro, every Deployment requires an executor and you can change the executor at any time. After you choose an executor for an Astro Deployment, you can configure your dags and Deployment resources to maximize the executor's efficiency and performance. Use the information provided in this topic to learn how to configure the Astro, Celery, and Kubernetes executors on Astro.

To learn more about executors in Airflow, see [Airflow executors](/docs/learn/airflow-executors-explained).

## Choose an executor

The difference between executors is based on how tasks are distributed across worker resources. The executor you choose affects the infrastructure cost of a Deployment and how efficiently your tasks execute. Astro currently supports three executors:

* Astro executor
* Celery executor
* Kubernetes executor

Read the following topics to learn about the benefits and limitations of each executor. For information about how to change the executor of an existing Deployment, see [Update the Deployment executor](/docs/astro/deployment-resources#update-the-deployment-executor).

### Astro executor

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

The [Astro executor](/docs/astro/astro-executor) is the default for all new Airflow 3.x Deployments. The Astro executor consists of agents (workers) that pull work from an API server. The API server manages the agent lifecycle and controls task assignment logic, enabling more efficient workload distribution and scaling. This design differs from the Celery executor, where workers fetch tasks directly from a queue, and the Kubernetes executor, where the scheduler launches pods for each task.

By centrally managing both agent scaling and task assignment, the Astro executor offers increased reliability, improved performance, and cost efficiency compared to other Airflow execution models.

The Astro executor is a good fit for your Deployment if:

* You need more control over agent scaling and task distribution.
* You want a balance between dynamic scaling and efficient task assignment.
* Your workload requires high reliability and performance.
* Cost optimization is a priority for your environment.

### Celery executor

The Celery executor uses a group of workers, each of which can run multiple tasks at a time. Astronomer uses [worker autoscaling logic](/docs/astro/celery-executor#celery-worker-autoscaling-logic) to determine how many workers run on your Deployment at a given time.

The Celery executor is a good option for most use cases. Specifically, the Celery executor is a good fit for your Deployment if:

* You're just getting started with Airflow.
* You want to use different worker types based on the type of task you're running. See [Configure worker queues](/docs/astro/configure-worker-queues).
* You have many short-running tasks.
* Your tasks require the shortest startup latency (often milliseconds).
* You don't require task or dependency isolation.
* You want to direct tasks to run in worker queues with varying resources. This is useful if you want to run a subset of tasks with priority for SLA reasons, or if your tasks have varied resource requirements.

If you find that some tasks consume the resources of other tasks and cause them to fail, Astronomer recommends implementing worker queues or moving to the Kubernetes executor.

See [Manage the Celery executor](/docs/astro/celery-executor) to learn more about how to configure the Celery executor.

### Kubernetes executor

The Kubernetes executor runs each task in an individual Kubernetes Pod that's defined either in your task or Deployment configuration. When a task completes, its Pod is terminated and the resources are returned to your cluster. On Astro, the infrastructure required to run the Kubernetes executor is built into every Deployment and is managed by Astronomer.

You can specify the configuration of a task's Pod, including CPU and memory, as part of your dag definition using the [Kubernetes Python Client](https://github.com/kubernetes-client/python) and the `pod_override` arg. Any task without a `pod_override` runs in a [default Pod](/docs/astro/deployment-resources#configure-kubernetes-pod-resources) as configured on your Deployment.

The Kubernetes executor is a good option for some use cases. Specifically, the Kubernetes executor is a good fit for your Deployment if:

* Your tasks are compute-intensive or you are processing large volumes of data within the task. Kubernetes executor tasks run separately in a dedicated Pod per task.
* Your tasks can tolerate some startup latency (often seconds)
* Your tasks require task or dependency isolation.
* You have had issues running certain tasks reliably with the Celery executor.

If you're running a high task volume or cannot tolerate startup latency, Astronomer recommends the Celery executor. To learn more about using the Kubernetes executor, see [Manage the Kubernetes executor](/docs/astro/kubernetes-executor).

<Info>
  **Running costs for the Kubernetes executor**

  For the Kubernetes executor, Astro bills for the number of A5 workers necessary to accommodate the total amount of CPU and Memory, rounded up to the nearest A5. One A5 worker corresponds to 1 CPU and 2 GiB Memory.
  In order to ensure reliability, Astro allocates the limit requested by each task. If a task doesn't have [specified limits](/docs/astro/kubernetes-executor#example-set-cpu-or-memory-limits-and-requests), then Astro uses the [Deployment defaults](/docs/astro/deployment-resources#configure-kubernetes-pod-resources).
</Info>

<Tip>The Kubernetes executor is primarily used for resource optimization. If you want more control over the environment and dependencies that your tasks run with, consider using the [`KubernetesPodOperator`](/docs/astro/kubernetespodoperator) with either the Celery executor or Kubernetes executor.</Tip>


# Export logs to AWS CloudWatch
Source: https://astronomer.io/docs/astro/export-cloudwatch

Configure your Deployment to forward observability data to your AWS CloudWatch instance.

By forwarding Astro data to AWS CloudWatch, you can integrate Astro into your existing observability practices by analyzing information about your Deployments' performance with CloudWatch monitoring tools. Currently, you can send the following data to AWS CloudWatch:

* Airflow task logs

Complete the following setup to authenticate your Deployments to AWS CloudWatch and forward your observability data to your AWS CloudWatch instance.

<Info>At this time, you can export only Airflow task logs to CloudWatch from Astro Deployments on AWS. You can export both Airflow metrics and task logs to [Datadog](/docs/astro/export-datadog) from Astro for all cloud providers.</Info>

## Export task logs to AWS CloudWatch

You can forward Airflow task logs from a Deployment to [AWS CloudWatch](https://aws.amazon.com/cloudwatch/) using an IAM role or user. This allows you to view and manage task logs across all Deployments from a centralized observability plane.

By default, Astro sets a unique log stream for each Deployment, and log groups are defined to include log streams from Deployments which share the same Workspace and cluster. You can override these definitions using Deployment environment variables if you want to change how your task logs are organized on CloudWatch. See [AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/Working-with-log-groups-and-streams.html) for more information about log groups and log streams.

### Prerequisites

* Your Deployment must run Astro Runtime 9 or later. See [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime).

### Setup

1. On AWS, create an IAM role and a trust policy that allows your Deployment to write logs to CloudWatch. See [Authorize Deployments to your cloud](/docs/astro/authorize-deployments-to-your-cloud#step-1-authorize-the-deployment-in-your-cloud-aws-alternative-setup).

2. Create a permissions policy with the following configuration:

   ```json wrap theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "logs:CreateLogGroup",
                   "logs:CreateLogStream",
                   "logs:PutLogEvents",
                   "logs:DescribeLogStreams"
               ],
               "Resource": "*"
           }
       ]
   }
   ```

   Attach this policy to your IAM role. See [Creating policies using the JSON editor](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create-console.html#access_policies_create-json-editor) and [Adding IAM identity permissions (console)](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html#add-policies-console).

3. Set the following [environment variables](/docs/astro/environment-variables) in your Deployment:

   * **Key 1**: `ASTRO_CLOUDWATCH_TASK_LOGS_ENABLED`

   * **Value 1**: `True`

   * **Key 2**: `ASTRO_CLOUDWATCH_ROLE_ARN`

   * **Value 2**: The ARN for your IAM role. It should look similar to `arn:aws:iam::123456789012:role/rolename`

<Info>
  If your CloudWatch instance is not in the same region as your Deployment, you must also set the following variable:

  * **Key**: `ASTRO_CLOUDWATCH_AWS_REGION`
  * **Value**: `<your-cloudwatch-region>`
</Info>

4. (Optional) Set the following environment variables if you require custom naming for your log streams or log groups. For example, you might set these names to make them more readable for CloudWatch admins who need to set targeted policies for log groups, or to organize log streams only by cluster instead of cluster and Workspace:

   * **Key 1**: `ASTRO_CLOUDWATCH_TASK_LOGS_LOG_GROUP`

   * **Value 1**: Your log group name.

   * **Key 2**: `ASTRO_CLOUDWATCH_TASK_LOGS_LOG_STREAM`

   * **Value 2**: Your log stream name.


# Export metrics and logs to Datadog
Source: https://astronomer.io/docs/astro/export-datadog

Configure your Deployment to forward observability data to your Datadog instance.

By forwarding Astro data to Datadog, you can integrate Astro into your existing observability practices by analyzing information about your Deployments' performance with Datadog cloud monitoring tools. Currently, you can send the following data to Datadog:

* Airflow task logs.
* Supported Datadog [Airflow metrics](https://docs.datadoghq.com/integrations/airflow/?tab=host#data-collected).

Complete the following setup to authenticate your Deployments to Datadog and forward your observability data to your Datadog instance.

## Export task logs to Datadog

You can forward Airflow task logs from a Deployment to [Datadog](https://www.datadoghq.com/) using a Datadog API key. This allows you to view and manage task logs across all Deployments from a centralized observability plane.

### Prerequisites

* Your Deployment must run Astro Runtime 9 (AWS) or 9.1 (Azure and GCP) or later. See [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime).
* If your Deployment runs Astro Runtime 3.x, add `apache-airflow-providers-datadog` to your Astro project `requirements.txt` file. Astro Runtime 3.x images don't include this package by default.

### Setup

1. Create a new Datadog API key or copy an existing API key. See [API and Application Keys](https://docs.datadoghq.com/account_management/api-app-keys/).
2. Set the following [environment variable](/docs/astro/environment-variables) on your Deployment:

   * **Key 1**: `DATADOG_API_KEY`

   * **Value 1**: Your Datadog API key.

   * **Key 2**: `ASTRO_DATADOG_TASK_LOGS_ENABLED`

   * **Value 2**: `true`

   Select the **Secret?** checkbox for `DATADOG_API_KEY`. This ensures that your Datadog API key is saved securely and is not available to Workspace users in plain text.

<Info>
  By default, the Astro Datadog integration also sends a Deployment's [Airflow metrics](https://docs.datadoghq.com/integrations/airflow/?tab=host#data-collected) to Datadog. To send only task logs to Datadog, add the following environment variable:

  * **Key**: `ASTRO_DATADOG_METRICS_DISABLED`
  * **Value**: `true`
</Info>

1. (Optional) Set the following [environment variable](/docs/astro/environment-variables) on your Deployment to send your logs to a specific [Datadog site](https://docs.datadoghq.com/getting_started/site/):

   * **Key**: `DATADOG_SITE`
   * **Value**: Your Datadog **Site Parameter**. For example, `datadoghq.com`.

2. (Optional) For Astro Runtime 9.2.0 and greater, set the following [environment variable](/docs/astro/environment-variables) on your Deployment to [add specific tags to your logs](https://docs.datadoghq.com/getting_started/tagging/):

   * **Key**: `ASTRO_DATADOG_TASK_LOGS_TAGS`
   * **Value**: `<tag-key-1>:<tag-value-1>,<tag-key-2>:<tag-value-2>`

By default, Astro uses the tags `source=astronomer` and `service=astronomer-task-logs`. Astro automatically adds `dag_id`, `task_id`, `run_id`, and `try_number` tags to logs sent to Datadog.

The minimum Astro Runtime versions in which logs are enriched with task instance data are 9.15.0, 10.9.0, and 11.5.0 across all cloud providers. To export logs enriched with task instance data to Datadog with a lower Astro Runtime version, upgrade `astronomer-providers-logging` to at least `1.5.1` in the `requirements.txt` file.

## Export Airflow metrics to Datadog

<Info>
  Airflow metrics for Dags or tasks with non-ASCII characters in their IDs are not supported and are not exported. Ensure all Dag IDs, task IDs, and TaskGroup IDs contain only ASCII characters (`A–Z`, `a–z`, `0–9`, `_`, `-`, `.`).
</Info>

Export over 40 Airflow metrics related to the state and performance of your Astro Deployment to [Datadog](https://www.datadoghq.com/) by adding a Datadog API key to your Deployment. These metrics include most information that is available in the Astro UI, as well as additional metrics that Datadog automatically collects, including number of queued tasks, dag processing time, and more. For a complete list of supported metrics, see [Data Collected](https://docs.datadoghq.com/integrations/airflow/?tab=host#data-collected) in the Datadog documentation.

<Info>Astro does not export any [service checks](https://docs.datadoghq.com/integrations/airflow/?tab=host#service-checks) to Datadog. Information about the general health of your Deployment is available only as part of the Astro UI's Deployment health metric.</Info>

<Note>
  If your Deployment runs Astro Runtime 3.x, add `apache-airflow-providers-datadog` to your Astro project `requirements.txt` file. Astro Runtime 3.x images don't include this package by default.
</Note>

1. Create a new Datadog API key or copy an existing API key. See [API and Application Keys](https://docs.datadoghq.com/account_management/api-app-keys/).

2. In the Astro UI, click **Deployments**, then select the Astro Deployment for which you want to export metrics.

3. Create a new [environment variable](/docs/astro/manage-env-vars#use-the-astro-ui) in your Deployment with the Datadog API key from step 1:

   * **Key:** `DATADOG_API_KEY`
   * **Value:** `<Your-Datadog-API-key>`.

   Select the **Secret?** checkbox. This ensures that your Datadog API key is saved securely and is not available to Workspace users in plain text.

4. (Optional) Add the following environment variable if your organization doesn't use the default Datadog site `datadoghq.com`:

   * **Key:** `DATADOG_SITE`
   * **Value:** `<Your-Datadog-Site>`

5. (Optional) Add the following environment variables to create [custom Datadog tags](https://docs.datadoghq.com/getting_started/tagging/) associated with your Deployment:

   * **Key 1**: `AIRFLOW__METRICS__STATSD_DATADOG_ENABLED`

   * **Value 1**: `True`

   * **Key 2**: `AIRFLOW__METRICS__STATSD_DATADOG_TAGS`

   * **Value 2**: `<tag-key-1>:<tag-value-1>,<tag-key-2>:<tag-value-2>`

6. Click **Save variable**.

After you complete this setup, Astro automatically launches a sidecar container in your Deployment that runs [DogStatsD](https://docs.datadoghq.com/developers/dogstatsd/?tab=hostagent). This container works with your Deployment's existing infrastructure to export Airflow metrics to the Datadog instance associated with your API key.

<Note>
  For [Remote Execution](/docs/astro/remote-execution-configure-agents) Deployments, you must manually configure a sidecar container with DogStatsD to export Airflow metrics for Remote Execution Agent components such as the triggerer, worker, and dag processor. This is because the [Datadog Agent's Airflow integration](https://docs.datadoghq.com/integrations/airflow/?tab=host#configure-datadog-agent-airflow-integration) does not yet support Airflow 3. You can specify the sidecar container details using `extraContainers` in the Remote Execution Agent's `values.yaml` Helm chart.
</Note>

### View metrics in Datadog

1. In the Datadog UI, go to **Metrics** > **Summary**.
2. Search for metrics starting with `airflow` and open any Airflow metric.
3. In the **Tags** table, check the values for the `namespace` tag key. The namespaces of the Deployments you configured to export logs should appear as tag values.

To check the health of a Deployment's DogStatsD container, open the `datadog.dogstatsd.running` metric in the Datadog UI. If the Deployment's namespace appears under the metric's `host` tag key, its DogStatsD container is healthy and exporting metrics to Datadog.


# Export metrics from Astro
Source: https://astronomer.io/docs/astro/export-metrics

Export Airflow infrastructure and task metrics from Astro to any Prometheus-compatible monitoring system using the Universal Metrics Exporter.

You can export comprehensive metrics about your Apache Airflow usage on Astro directly to any third-party monitoring and alerting system using the Universal Metrics Exporter. This gives you unlimited access to all infrastructure metrics that are available to Astronomer and allows you to use your preferred observability tooling, such as Grafana or CloudWatch.

While [Deployment Analytics](/docs/astro/deployment-metrics), [Deployment health incidents](/docs/astro/deployment-health-incidents), and [exporting metrics and logs to Datadog](/docs/astro/export-datadog) on Astro can all help you understand your Airflow usage and infrastructure resource consumption, the Universal Metrics Exporter is additionally valuable because it:

* Gives you access to both Kubernetes-level infrastructure metrics as well as task-level execution information specific to Apache Airflow.
* Provides metrics that aren't available to [Datadog's supported Airflow metrics](https://docs.datadoghq.com/integrations/airflow/?tab=host#data-collected).
* Uses the [Prometheus data model](https://prometheus.io/docs/concepts/data_model/) format using the remote-write capability, making it flexible and easy to use.
* Allows you to configure a metrics export at the per-Deployment level or at the Workspace level.
* Enables you to customize your observability experience with your tooling and create custom dashboards or alerts that aren't currently available in Astro.
* Offers a way to add custom metadata as labels to your exported metrics or HTTP request headers.

You can use this information to right-size Astro and Celery workers, optimize your usage of the Kubernetes executor, and stay informed about task execution status.

## Metric categories

There are two categories of metrics that you can export using the Universal Metrics Exporter:

* Airflow application level metrics
* Infrastructure level metrics

Both application and infrastructure metrics have metadata labels associated with them. The following list shows the default standard set of labels that Astro attaches to each metric:

* `cloud_provider`
* `cloud_region`
* `cluster_organization_id`
* `clusterId` (Deployments on Dedicated clusters only)
* `container`
* `namespace`
* `pod`
* `deploymentId`
* `organizationId`
* `workspaceId`

Some metrics additionally include labels that are specific and unique to that metric. Use these metadata labels to identify each individual metric with its corresponding environment in Astro.

### Custom metadata

In addition to the default metadata included by Astro with your metrics, you can also add **Custom headers** and **Custom labels** to your metrics exports.

* Custom headers add key-value pairs to the HTTP request made by Astro to your Prometheus server.
* Custom labels add key-value pairs to the export metadata, so you can filter your metrics downstream.

### Airflow application metrics

Airflow application metrics are defined by Apache Airflow and describe the health, success, and performance of the Dags that Airflow orchestrates and executes. Examples include task instance failure counts, scheduler heartbeat counts, pool slot utilization, and Dag run durations.

Astro normalizes the StatsD-format metrics that Airflow emits, then exports them to your Prometheus endpoint. For the full list of supported metrics, the Prometheus labels available on each, and how each Astro metric name maps to its upstream Airflow name, see the [Metrics reference](/docs/astro/export-metrics-reference#airflow-application-metrics).

### Infrastructure metrics

Infrastructure metrics describe the resource usage, health, and performance of the Kubernetes Pods that run each Airflow component. Use them to track CPU, memory, storage, and Pod lifecycle status across your Deployment. For the full list of supported infrastructure metrics, see the [Metrics reference](/docs/astro/export-metrics-reference#infrastructure-metrics).

## Prerequisites

* Supported Auth: Bearer token or license key, username and password, SigV4Authorization (AWS Deployments only), or custom HTTP header(s) of your target data observability server *(Optional)*
* A Prometheus data endpoint
* Network connectivity between your Astro resources and Prometheus endpoint

## Set up your Prometheus endpoint

The following list includes the setup instructions of different, commonly used Prometheus endpoints. Use these resources to set up your observability tools to receive metrics exports from Astro.

* [Chronosphere](https://docs.chronosphere.io/ingest/metrics-traces/collector/configure/prometheus-backend)
* [Coralogix](https://coralogix.com/docs/prometheus/)
* [Cribl](https://docs.cribl.io/stream/4.2/sources-prometheus-remote-write/)
* [Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-prometheus-remote_write.html)
* [Grafana Cloud](https://grafana.com/docs/grafana-cloud/send-data/metrics/metrics-prometheus/#send-data-from-a-prometheus-instance)
* [Logz.io](https://docs.logz.io/docs/shipping/other/prometheus-remote-write/)
* [New Relic](https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-remote-write/set-your-prometheus-remote-write-integration/)
* [Sysdig](https://docs.sysdig.com/en/sysdig-monitor/install-prometheus-remote-write/)
* [Prometheus](https://prometheus.io/docs/prometheus/latest/feature_flags/#remote-write-receiver)

## Enable metrics export

You can enable metrics export at both the Workspace and Deployment level.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

### Workspace metrics

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Environment** > **Metrics Exports**.
    2. Click **+ New Metric Export** to create a new export connection.
    3. Enter the required information for your export, the **Name** and **Endpoint**.

    <Tip> For the endpoint, you might need to use a format like `http://prometheus.<internal-hostname>/api/v1/write` </Tip>

    4. (Optional) Click **More Options** to add additional information for your endpoint:
       * **Authentication Type** - **None**, **Basic Authentication**, **Bearer Authentication**, or **SigV4Authorization**. **SigV4Authorization** is available only for AWS Deployments on dedicated clusters and enables you to use IAM role-based authentication for metrics export.
         * If you select **SigV4Authorization**, you must configure a trust policy that allows Astro to assume an IAM role in your AWS account. You can view and copy this trust policy from the **Trust Policies** tab.
       * **Token** - Bearer token or license key for your Prometheus endpoint for bearer authentication, if needed.
       * **Username** and **Password** - Credentials for basic authentication, if needed.
       * **AWS Role** - IAM role that Astro assumes to communicate with your Amazon Managed Prometheus.
       * **Region** - AWS region of the Amazon Managed Prometheus.
       * Custom **Headers** - Add a **Name** and **Value** pair that can be sent to your server as part of the HTTP request with each remote write. If you add a custom header, any default values are overwritten by your custom configuration. You can add Deployment-specific labels when configuring **Deployment metrics**.
       * Custom **Labels** - Add a **Name** and **Value** pair that can be sent to your server. You can add Deployment-specific labels when configuring **Deployment metrics**.
    5. Click **Create metrics export**.
    6. (Optional) Allow all Deployments to link to this metrics export configuration. See [Link metrics export](#share-metrics-exports-across-deployments).
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select **Environment** on the sidebar menu.
    2. Click the **Metrics Export** tab to configure your metrics export.
    3. Click **+ Metrics Export** to create a new export connection.
    4. Enter the required information for your export, the **Name** and **Endpoint**.

    <Tip> For the endpoint, you might need to use a format like `http://prometheus.<internal-hostname>/api/v1/write` </Tip>

    5. (Optional) Click **More Options** to add additional information for your endpoint:
       * **Authentication Type** - **None**, **Basic Authentication**, **Bearer Authentication**, or **SigV4Authorization**. **SigV4Authorization** is available only for AWS Deployments on dedicated clusters and enables you to use IAM role-based authentication for metrics export.
         * If you select **SigV4Authorization**, you must configure a trust policy that allows Astro to assume an IAM role in your AWS account. You can view and copy this trust policy from the **Trust Policies** tab.
       * **Token** - Bearer token or license key for your Prometheus endpoint for bearer authentication, if needed.
       * **Username** and **Password** - Credentials for basic authentication, if needed.
       * **AWS Role** - IAM role that Astro assumes to communicate with your Amazon Managed Prometheus.
       * **Region** - AWS region of the Amazon Managed Prometheus.
       * Custom **Headers** - Add a **Name** and **Value** pair that can be sent to your server as part of the HTTP request with each remote write. If you add a custom header, any default values are overwritten by your custom configuration. You can add Deployment-specific labels when configuring **Deployment metrics**.
       * Custom **Labels** - Add a **Name** and **Value** pair that can be sent to your server. You can add Deployment-specific labels when configuring **Deployment metrics**.
    6. Click **Create metrics export**.
    7. (Optional) Allow all Deployments to link to this metrics export configuration. See [Link metrics export](#share-metrics-exports-across-deployments).
  </Tab>
</Tabs>

### Deployment metrics

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, select a Deployment, then click the **Environment** tab.
    2. Toggle to **Metrics Exports**.
    3. Click **+ New Metric Export** to configure a new export connection.
    4. Enter the required information for your export, the **Name** and **Endpoint**.

    <Tip> For the endpoint, you might need to use a format like `http://prometheus.<internal-hostname>/api/v1/write` </Tip>

    5. (Optional) Click **More Options** to add additional information for your endpoint:
       * **Authentication Type** - **None**, **Basic Authentication**, **Bearer Authentication**, or **SigV4Authorization**. **SigV4Authorization** is available only for AWS Deployments on dedicated clusters and enables you to use IAM role-based authentication for metrics export.
         * If you select **SigV4Authorization**, you must configure a trust policy that allows Astro to assume an IAM role in your AWS account. You can view and copy this trust policy from the **Trust Policies** tab.
       * **Token** - Bearer token or license key for your Prometheus endpoint for bearer authentication, if needed.
       * **Username** and **Password** - Credentials for basic authentication, if needed.
       * **AWS Role** - IAM role that Astro assumes to communicate with your Amazon Managed Prometheus.
       * **Region** - AWS region of the Amazon Managed Prometheus.
       * Custom **Headers** - Add a **Name** and **Value** pair that can be sent to your server with as part of the HTTP request with each remote write. If you add a custom header, any default values or values configured at the Workspace level are overwritten by your custom configuration.
       * Custom **Labels** - Add a **Name** and **Value** pair that can be sent to your server. You can add Deployment-specific labels when configuring **Deployment metrics**.
    6. Click **Create metrics export**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, select a Deployment, then click the **Environment** tab.
    2. Toggle to **Metrics Exports**.
    3. Click **+ Metrics Export** to configure a new export connection.
    4. Enter the required information for your export, the **Name** and **Endpoint**.

    <Tip> For the endpoint, you might need to use a format like `http://prometheus.<internal-hostname>/api/v1/write` </Tip>

    5. (Optional) Click **More Options** to add additional information for your endpoint:
       * **Authentication Type** - **None**, **Basic Authentication**, **Bearer Authentication**, or **SigV4Authorization**. **SigV4Authorization** is available only for AWS Deployments on dedicated clusters and enables you to use IAM role-based authentication for metrics export.
         * If you select **SigV4Authorization**, you must configure a trust policy that allows Astro to assume an IAM role in your AWS account. You can view and copy this trust policy from the **Trust Policies** tab.
       * **Token** - Bearer token or license key for your Prometheus endpoint for bearer authentication, if needed.
       * **Username** and **Password** - Credentials for basic authentication, if needed.
       * **AWS Role** - IAM role that Astro assumes to communicate with your Amazon Managed Prometheus.
       * **Region** - AWS region of the Amazon Managed Prometheus.
       * Custom **Headers** - Add a **Name** and **Value** pair that can be sent to your server with as part of the HTTP request with each remote write. If you add a custom header, any default values or values configured at the Workspace level are overwritten by your custom configuration.
       * Custom **Labels** - Add a **Name** and **Value** pair that can be sent to your server. You can add Deployment-specific labels when configuring **Deployment metrics**.
    6. Click **Create metrics export**.
  </Tab>
</Tabs>

If you successfully connected your metrics export to your observability service endpoint, after five minutes, your Astro metrics begin to populate in your observability service. If the connection has issues, you see the error log messages in the **Error Logs** tab.

### View SigV4Authorization trust policy

If you're using **SigV4Authorization** to authenticate your Metrics Export for AWS-based Deployments, you must configure your Amazon Managed Prometheus endpoint with a trust policy that allows Astro to assume an IAM role.

You can view and copy this trust policy from the **Trust Policies** tab of the Metrics Export.

## Share metrics exports across Deployments

You can configure Astro to link Workspace-level metrics exports to all Deployments in the Workspace by default.

This is useful, for example, when you need to configure a metrics export for development environments that all Deployments in a Workspace should start with. Then, when you create new Deployments, they automatically have a default metrics export configuration to your development resources.

When you're ready to move your Deployments' metrics exports to production configurations, you can either replace the metrics export or [override the configuration](#override-configuration-fields) values with your production resource information.

If you change the setting from **Restricted** to **Linked to all Deployments**, Astro respects any metrics exports fields that you might have configured for existing linked Deployments.

1. In the Astro UI, go to **Environment** > **Metrics Exports**.
2. Click the name of the export target that you want to add per-Deployment field overrides to.
3. Click **Deployment Sharing** and toggle the setting to choose either:
   * **Restricted**: Only share individually to Deployments.
   * **Linked to all Deployments**: Link to all current and future Deployments in this Workspace.
4. (Optional) Change the default field values.
5. Click **Update metrics export** to save.

## Override configuration fields

If you create a metrics export at the Workspace level and link it to a Deployment, you can later edit the endpoint within the Deployment to specify field overrides. When you override a field, you specify values that you want to use for a one Deployment, but not for others. This way, you can configure a metrics export for a single time, but still have the flexibility to customize it at the Deployment level.

For example, you might have created a metrics connection to a dev or internal observability endpoint, and then later you can add field overrides to specify production details you want each Deployment to use.

1. In the Astro UI, go to **Environment** > **Metrics Exports**.
2. Click the metrics export that you want to add per-Deployment field overrides to.
3. (Optional) Click **Deployment Sharing** and choose if you want to **Restrict** or **Link to all Deployments**. You can also change the default field values. Click **Update metrics export** to save.
4. Click **Edit** to open the metrics export configurations for a specific linked Deployment.
5. Add the override values to the fields you want to edit. You might need to open **More options** to find the full list of available fields.
6. Click **Update metrics export**.

<a />

## Example: Grafana Cloud dashboard

The [Astronomer Docs resources repository](https://github.com/astronomer/astronomer-docs-resources/tree/main/astro/metrics-export/grafana-cloud) includes two example dashboard configuration files for [Grafana Cloud](https://grafana.com/products/cloud/): `grafana-cloud-dash.json` for StatsD-based metrics and `grafana-cloud-dash-otel.json` for OpenTelemetry-based metrics on Astro Runtime 3.0 or later.

<Frame>
  <img alt="Example Grafana dashboard showing available sections, collapsed. These include information about Dags and tasks, and components; scheduler, webserver, worker, triggerer, and PgBouncer." />
</Frame>

This configuration file allows you to create a dashboard that provides an at-a-glance view of your Astro resources and task execution status. The following image shows an example of the **Scheduler** dashboard view:

<Frame>
  <img alt="Example Grafana dashboard showing data about the Deployment's scheduler." />
</Frame>

Or you can view details about resources like your **Workers**, such as in the following image.

<Frame>
  <img alt="Example Grafana dashboard showing data about the Deployment's workers" />
</Frame>


# Universal Metrics Exporter metrics reference
Source: https://astronomer.io/docs/astro/export-metrics-reference

Complete list of Airflow application and infrastructure metrics that the Universal Metrics Exporter exports from Astro.

This document lists every metric that Astro exports through the [Universal Metrics Exporter](/docs/astro/export-metrics). Use this reference to identify which metrics are available, the Prometheus labels you can query against, and how each Astro metric name maps to its upstream Apache Airflow name.

<Info>
  Airflow metrics for Dags or tasks with non-ASCII characters in their IDs are not supported and are not exported. Ensure all Dag IDs, task IDs, and TaskGroup IDs contain only ASCII characters (`A–Z`, `a–z`, `0–9`, `_`, `-`, `.`).
</Info>

Astro exports two categories of metrics:

* Airflow application metrics describe the health, success, and performance of Dag execution. Astro normalizes these from the StatsD format that Airflow emits before exporting them to your Prometheus endpoint.
* Infrastructure metrics describe the resource usage and lifecycle health of the Kubernetes Pods that run each Airflow component.

Astro doesn't export metrics outside of these tables. The mapping configuration drops any metric that doesn't match a rule.

## How Astro normalizes Airflow metrics

Astro applies the following transformations to Airflow metrics before they reach your Prometheus endpoint. For the source-of-truth mapping rules, see the [Astro StatsD mappings file](https://github.com/astronomer/ap-vendor/blob/main/statsd-exporter/include/mappings-gen2.yml).

* StatsD names become Prometheus names. Astro replaces dots with underscores. For example, `airflow.dag_processing.import_errors` becomes `airflow_dag_processing_import_errors`.
* Variable name parts become Prometheus labels. High-cardinality identifiers move out of the metric name and into labels so that one metric name covers many dimensions. For example, Astro exports the Airflow metric `airflow.dag.<dag_id>.<task_id>.duration` as `airflow_task_duration` with `dag_id` and `task_id` labels.
* Legacy and current Airflow names both flow through. Astro maps metrics that Airflow renamed across versions under both their legacy and current names so that Dags running on different Astro Runtime versions both emit. For example, both `zombies_killed` (Airflow 2.x) and `task_instances_without_heartbeats_killed` (Airflow 3 and later) export when Airflow emits them.
* Astro adds default metadata labels. Every exported metric carries the standard Astro labels documented in [Export metrics](/docs/astro/export-metrics#metric-categories), such as `deploymentId`, `organizationId`, and `workspaceId`.

<Warning>
  Dag IDs, task IDs, and TaskGroup IDs flow into metric names, so they must contain only ASCII characters (`A–Z`, `a–z`, `0–9`, `_`, `-`, `.`). Airflow's own validation accepts Unicode characters such as accented letters (`ç`, `ã`, `ö`), but metric backends don't. StatsD silently drops metrics whose names contain non-ASCII characters, and OpenTelemetry rejects them as invalid under the [instrument name syntax specification](https://opentelemetry.io/docs/specs/otel/metrics/api/#instrument-name-syntax). The allowed characters for StatsD metric names are defined in [Airflow's metric validation source](https://github.com/apache/airflow/blob/main/shared/observability/src/airflow_shared/observability/metrics/validators.py).

  Audit your Dag files for non-ASCII identifiers before you enable metrics:

  ```sh wrap theme={null}
  grep -rn '[^\x00-\x7F]' dags/ --include="*.py"
  ```

  Rename any non-ASCII IDs to ASCII equivalents so that all metrics export correctly.
</Warning>

## Airflow application metrics

Apache Airflow classifies metrics into three types based on how the value behaves over time: counters, gauges, and timers. The following tables use Airflow's classification. For background on each type, see the [Apache Airflow metrics reference](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html).

### Counters

A counter records the cumulative count of events that occur over time, such as task failures or scheduler heartbeats.

| Name                                               | Airflow name                                    | Labels                       | Description                                                                                                                                                    |
| -------------------------------------------------- | ----------------------------------------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `airflow_job_start`                                | `<job_name>_start`                              | `job_name`                   | Started jobs, such as `SchedulerJob` or `LocalTaskJob`.                                                                                                        |
| `airflow_job_end`                                  | `<job_name>_end`                                | `job_name`                   | Completed jobs.                                                                                                                                                |
| `airflow_job_heartbeat_failure`                    | `<job_name>_heartbeat_failure`                  | `job_name`                   | Heartbeat failures for a given job type.                                                                                                                       |
| `airflow_operator_successes`                       | `operator_successes_<operator>`                 | `operator`                   | Successful executions of a given operator type.                                                                                                                |
| `airflow_operator_failures`                        | `operator_failures_<operator>`                  | `operator`                   | Failures of a given operator type.                                                                                                                             |
| `airflow_scheduler_heartbeat`                      | `scheduler_heartbeat`                           | `type`                       | Scheduler heartbeat occurrences. Astro sets the `type` label to `counter` so dashboards can distinguish counter-typed values from older gauge-typed emissions. |
| `airflow_dag_processor_heartbeat`                  | `dag_processor_heartbeat`                       | None                         | Standalone Dag processor heartbeat occurrences. Available on Airflow 3 and later.                                                                              |
| `airflow_triggerer_heartbeat`                      | `triggerer_heartbeat`                           | None                         | Triggerer heartbeat occurrences.                                                                                                                               |
| `airflow_ti_start`                                 | `ti.start.<dag_id>.<task_id>`                   | `dag_id`, `task_id`          | Task instance initiations within a Dag.                                                                                                                        |
| `airflow_ti_finish`                                | `ti.finish.<dag_id>.<task_id>.<state>`          | `dag_id`, `task_id`, `state` | Task instance completions, broken out by terminal state.                                                                                                       |
| `airflow_ti_failures`                              | `ti_failures`                                   | None                         | Total task instance failures across all Dags.                                                                                                                  |
| `airflow_ti_successes`                             | `ti_successes`                                  | None                         | Total task instance successes across all Dags.                                                                                                                 |
| `airflow_task_instance_created`                    | `task_instance_created_<task_type>`             | `task_type`                  | Task instances created, broken out by operator type.                                                                                                           |
| `airflow_scheduler_tasks_killed_externally`        | `scheduler.tasks.killed_externally`             | None                         | Tasks terminated by external processes.                                                                                                                        |
| `airflow_zombies_killed`                           | `zombies_killed`                                | None                         | Zombie task instances terminated by the scheduler. Replaced by `airflow_task_instances_without_heartbeats_killed` on Airflow 3 and later.                      |
| `airflow_task_instances_without_heartbeats_killed` | `task_instances_without_heartbeats_killed`      | None                         | Task instances terminated due to missing heartbeats. Replaces `airflow_zombies_killed` on Airflow 3 and later.                                                 |
| `airflow_triggers_succeeded`                       | `triggers.succeeded`                            | None                         | Triggers that successfully fired at least one event.                                                                                                           |
| `airflow_triggers_failed`                          | `triggers.failed`                               | None                         | Triggers that failed before firing.                                                                                                                            |
| `airflow_dataset_updates`                          | `dataset.updates`                               | None                         | Dataset updates. Replaced by `airflow_asset_updates` on Airflow 3 and later.                                                                                   |
| `airflow_dataset_triggered_dagruns`                | `dataset.triggered_dagruns`                     | None                         | Dag runs triggered by dataset updates. Replaced by `airflow_asset_triggered_dagruns` on Airflow 3 and later.                                                   |
| `airflow_asset_updates`                            | `asset.updates`                                 | None                         | Asset modifications. Available on Airflow 3 and later; replaces `airflow_dataset_updates`.                                                                     |
| `airflow_asset_triggered_dagruns`                  | `asset.triggered_dagruns`                       | None                         | Dag runs initiated by asset updates. Available on Airflow 3 and later; replaces `airflow_dataset_triggered_dagruns`.                                           |
| `airflow_ol_emit_failed`                           | `ol.emit.failed`                                | None                         | Failed attempts to emit OpenLineage events.                                                                                                                    |
| `airflow_astro_logging_write_failed`               | `airflow.astro_logging.<provider>.write.failed` | `provider`                   | Log-write failures from the `astronomer-providers-logging` package, broken out by provider.                                                                    |
| `astro_bundle_backend_refresh_success`             | Astronomer only                                 | `instance`, `mount_path`     | Successful refreshes of an Astro bundle backend mount.                                                                                                         |
| `astro_bundle_backend_refresh_failure`             | Astronomer only                                 | `instance`, `mount_path`     | Failed refreshes of an Astro bundle backend mount.                                                                                                             |
| `astro_bundle_backend_download_urls_success`       | Astronomer only                                 | `instance`, `mount_path`     | Successful download URL fetches by the Astro bundle backend.                                                                                                   |
| `astro_bundle_backend_download_urls_failure`       | Astronomer only                                 | `instance`, `mount_path`     | Failed download URL fetches by the Astro bundle backend.                                                                                                       |

### Gauges

A gauge measures a point-in-time value that can rise and fall, such as the number of running tasks or open executor slots.

| Name                                          | Airflow name                                      | Labels                         | Description                                                                                                                  |
| --------------------------------------------- | ------------------------------------------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| `airflow_dagbag_size`                         | `dagbag_size`                                     | None                           | Number of Dags found during the last scheduler scan.                                                                         |
| `airflow_dag_processing_import_errors`        | `dag_processing.import_errors`                    | None                           | Number of errors encountered when parsing Dag files.                                                                         |
| `airflow_dag_processing_total_parse_time`     | `dag_processing.total_parse_time`                 | None                           | Total seconds spent scanning and importing Dag files in the most recent cycle.                                               |
| `airflow_dag_processing_last_run_seconds_ago` | `dag_processing.last_run.seconds_ago.<dag_file>`  | `dag_file`                     | Seconds elapsed since the named Dag file was last evaluated.                                                                 |
| `airflow_executor_open_slots`                 | `executor.open_slots`                             | None                           | Available execution slots on the executor.                                                                                   |
| `airflow_executor_queued_tasks`               | `executor.queued_tasks`                           | None                           | Tasks awaiting execution on the executor.                                                                                    |
| `airflow_executor_running_tasks`              | `executor.running_tasks`                          | None                           | Tasks currently executing on the executor.                                                                                   |
| `airflow_pool_open_slots`                     | `pool.open_slots.<pool>`                          | `pool`                         | Open slots in a named pool.                                                                                                  |
| `airflow_pool_used_slots`                     | `pool.used_slots.<pool>`                          | `pool`                         | Slots currently in use in a named pool. Available on Airflow 2.x.                                                            |
| `airflow_pool_queued_slots`                   | `pool.queued_slots.<pool>`                        | `pool`                         | Slots held by queued tasks in a named pool.                                                                                  |
| `airflow_pool_running_slots`                  | `pool.running_slots.<pool>`                       | `pool`                         | Slots held by running tasks in a named pool.                                                                                 |
| `airflow_pool_deferred_slots`                 | `pool.deferred_slots.<pool>`                      | `pool`                         | Slots held by deferred tasks in a named pool.                                                                                |
| `airflow_pool_scheduled_slots`                | `pool.scheduled_slots.<pool>`                     | `pool`                         | Slots held by scheduled tasks in a named pool.                                                                               |
| `airflow_pool_starving_tasks`                 | `pool.starving_tasks.<pool>`                      | `pool`                         | Tasks in a named pool that can't proceed because pool resources are exhausted.                                               |
| `airflow_scheduler_tasks_running`             | `scheduler.tasks.running`                         | None                           | Tasks currently running according to the scheduler. Available on Airflow 2.x.                                                |
| `airflow_scheduler_tasks_starving`            | `scheduler.tasks.starving`                        | None                           | Tasks the scheduler can't run because pool resources are exhausted.                                                          |
| `airflow_triggers_running`                    | `triggers.running`                                | None                           | Triggers currently executing on a triggerer host.                                                                            |
| `airflow_dataset_orphaned`                    | `dataset.orphaned`                                | None                           | Datasets no longer referenced by any Dag. Replaced by `airflow_asset_orphaned` on Airflow 3 and later.                       |
| `airflow_asset_orphaned`                      | `asset.orphaned`                                  | None                           | Assets no longer referenced by any Dag schedule or task output. Available on Airflow 3 and later.                            |
| `airflow_runner_resources`                    | `airflow.executor.runner_resources.<resource>`    | `resource`                     | Percentage of a resource in use on an executor runner. Values for the `resource` label include `slots`, `cpu`, and `memory`. |
| `airflow_executor_task_resources`             | `airflow.executor.task_resources.<resource_stat>` | `resource_stat`                | Task-level resource statistics. Values for the `resource_stat` label include `memory_rss` and `cpu_times_system`.            |
| `astro_bundle_backend_num_files`              | Astronomer only                                   | `instance`, `mount_path`, `le` | Histogram of file counts in Astro bundle backend mounts.                                                                     |
| `astro_bundle_backend_tarball_size`           | Astronomer only                                   | `instance`, `mount_path`, `le` | Histogram of tarball sizes downloaded by the Astro bundle backend.                                                           |

### Timers

A timer measures the duration of an event, such as how long a task or Dag run takes to complete. Astro exports timer values in milliseconds.

| Name                                         | Airflow name                                                  | Labels                         | Description                                                                                                                |
| -------------------------------------------- | ------------------------------------------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------- |
| `airflow_task_duration`                      | `dag.<dag_id>.<task_id>.duration`                             | `dag_id`, `task_id`            | Total duration of a task instance.                                                                                         |
| `airflow_dagrun_duration`                    | `dagrun.duration.success.<dag_id>`                            | `dag_id`                       | Duration of a successful Dag run.                                                                                          |
| `airflow_dagrun_failed`                      | `dagrun.duration.failed.<dag_id>`                             | `dag_id`                       | Duration of a failed Dag run.                                                                                              |
| `airflow_dagrun_schedule_delay`              | `dagrun.schedule_delay.<dag_id>`                              | `dag_id`                       | Delay between the scheduled and actual start of a Dag run.                                                                 |
| `airflow_dagrun_first_task_scheduling_delay` | `dagrun.<dag_id>.first_task_scheduling_delay`                 | `dag_id`                       | Delay between a Dag run's start and the scheduling of its first task.                                                      |
| `airflow_dagrun_dependency_check`            | `dagrun.dependency-check`, `dagrun.dependency-check.<dag_id>` | `dag_id`                       | Time required to evaluate Dag run dependencies. The `dag_id` label is present only when Airflow emits the Dag-scoped form. |
| `airflow_dag_processing_last_duration`       | `dag_processing.last_duration.<dag_file>`                     | `dag_file`                     | Time required to parse the named Dag file in the most recent cycle.                                                        |
| `airflow_dag_processing_last_runtime`        | `dag_processing.last_runtime.<dag_file>`                      | `dag_file`                     | Legacy name for `airflow_dag_processing_last_duration`. Astro retains this mapping so older Airflow versions still emit.   |
| `airflow_collect_db_dags`                    | `collect_db_dags`                                             | None                           | Time spent fetching serialized Dags from the metadata database.                                                            |
| `airflow_ol_emit_attempts`                   | `ol.emit.attempts`                                            | None                           | Time consumed by OpenLineage event emission attempts.                                                                      |
| `astro_bundle_backend_download_time`         | Astronomer only                                               | `instance`, `mount_path`, `le` | Histogram of download durations for Astro bundle backend tarballs.                                                         |
| `astro_bundle_backend_extract_time`          | Astronomer only                                               | `instance`, `mount_path`, `le` | Histogram of extract durations for Astro bundle backend tarballs.                                                          |

### Astro event scheduler metrics

The Astro event scheduler emits metrics under the `airflow.astro_event_scheduler.*` namespace. Astro strips the `airflow.` prefix and exports each metric as `astro_event_scheduler_<rest>`. Because this is a catch-all mapping, the specific metric names emitted depend on the version of Astro Runtime running in your Deployment.

## Infrastructure metrics

Infrastructure metrics describe the Kubernetes Pods that run each Airflow component. Use them to track CPU, memory, storage, and lifecycle health.

| Name                                          | Description                                                                                                                                   |
| --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `container_cpu_usage_seconds_total`           | CPU usage of each container.                                                                                                                  |
| `container_memory_working_set_bytes`          | Memory usage of each container.                                                                                                               |
| `kubelet_stats_ephemeral_storage_pod_usage`   | Ephemeral storage usage of each Pod.                                                                                                          |
| `kube_pod_status_*`                           | Kubernetes Pod status.                                                                                                                        |
| `kube_pod_labels`                             | Kubernetes Pod labels.                                                                                                                        |
| `kube_pod_container_resource_limits`          | CPU, memory, and storage limits for Celery workers, Kubernetes executors, `KubernetesPodOperator` Pods, the scheduler, and the Dag processor. |
| `kube_pod_container_status_terminated_reason` | Reason a Kubernetes container terminated.                                                                                                     |
| `kube_resourcequota`                          | Resource quota usage and limits for namespaces in the cluster.                                                                                |


# Export logs to a Secondary GCS Bucket
Source: https://astronomer.io/docs/astro/export-secondary-gcs-bucket

Configure your Deployment to forward Airflow task logs to a secondary GCS bucket in GCP.

## Export logs to a Secondary GCS Bucket

By forwarding Airflow task logs from your Astro Deployment to **an additional, customer‑managed GCS bucket**, you can keep redundant copies of your execution history, integrate with existing log‑processing pipelines, and satisfy compliance or retention requirements that extend beyond the built‑in Astro logs.

## Prerequisites

* Your Deployment must run **Astro Runtime 11.7.0** or later. See [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime).
* Your image must include `astronomer-providers-logging==1.6.4` or later. The secondary GCS logging feature was introduced in this version.
* You need a GCP account where you can create a GCP bucket and IAM resources.
* A role with the [minimum required permissions](#minimum-required-permissions)

### Minimum Required Permissions

For all authentication methods, the service account or identity needs the following minimum permissions on the GCS bucket:

* `storage.objects.list`: List matching log files
* `storage.objects.create`: To create new log files
* `storage.objects.get`: To read existing log files (for append operations)
* `storage.objects.update`: To update existing log files

These permissions are included in the predefined `roles/storage.objectUser` role, or you can create a custom role with just these specific permissions.

## Overview of configuration options

There are three methods to configure your Astro deployment to write logs to a secondary GCS bucket:

1. **Customer‑provided Workload Identity**: Recommended if you already use this mechanism for other GCP services.
2. **Service Account Impersonation**: Use this approach if you want to use an existing service account which is different than your Deployment's workload identity.
3. **Astro-provided Workload Identity**: Use this approach if you do not have an existing service account that your Deployment can impersonate.

The following environment variables are supported by this feature:

| Variable                                                 | Required | Example value                                                    |
| -------------------------------------------------------- | -------- | ---------------------------------------------------------------- |
| `AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_ENABLED`             | Yes      | `true`                                                           |
| `AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_BASE_LOG_FOLDER`     | Yes      | `gs://your-bucket-name/logs`                                     |
| `AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_IMPERSONATION_CHAIN` | No       | `target-service-account@YOUR_PROJECT_ID.iam.gserviceaccount.com` |

### Option 1: Customer‑Provided Workload Identity

This approach uses [Google Cloud Workload Identity](https://cloud.google.com/kubernetes-engine/multi-cloud/docs/attached/eks/concepts/workload-identity) to allow your Astro deployment to authenticate with Google Cloud using its Kubernetes service account identity.

1. [Attach a service account to your Astro Deployment](/docs/astro/authorize-deployments-to-your-cloud#attach-a-service-account-to-your-deployment)
2. Set the following environment variables in the Deployment:

```text wrap theme={null}
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_ENABLED=true
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_BASE_LOG_FOLDER="gs://your-bucket-name/logs"
```

### Option 2: Service Account Impersonation

Use this approach if you want to use an existing service account which is different than your Deployment's workload identity.

1. Follow the steps to setup [service account impersonation](/docs/astro/authorize-deployments-to-your-cloud#alternative-setup-authorize-your-deployment-through-gcp-service-account-impersonation), which allows your Astro deployment to impersonate a service account with the necessary permissions.
2. Set the following environment variables in the Deployment:

```text wrap theme={null}
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_ENABLED=true
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_BASE_LOG_FOLDER="gs://your-bucket-name/logs"
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_IMPERSONATION_CHAIN="target-service-account@YOUR_PROJECT_ID.iam.gserviceaccount.com"
```

### Option 3: Astro-provided Workload Identity

Use this approach if you do not have an existing service account that your deployment can impersonate.

1. Follow instructions to [grant an IAM role to your Deployment Workload Identity](/docs/astro/authorize-deployments-to-your-cloud#alternative-setup-grant-an-iam-role-to-your-deployment-workload-identity).
2. Set the following environment variables in the Deployment:

```text wrap theme={null}
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_ENABLED=true
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_BASE_LOG_FOLDER="gs://your-bucket-name/logs"
```

## Troubleshooting

If you encounter issues with logging to the secondary GCS bucket:

1. Check that the environment variables are correctly set in your Astro deployment
2. Verify that your service account or Workload Identity has the necessary permissions
3. For Workload Identity Federation, ensure the Kubernetes service account annotation is correctly configured
4. For impersonation, check that the impersonation chain is correctly configured and the source identity has token creator permissions
5. Check for any errors in your Airflow logs related to GCS authentication or permissions
6. Test your authentication setup using the `gcloud` CLI or Google Cloud Console

If issues persist, contact Astronomer Support with the Deployment ID and any relevant error output.


# Export logs to a Secondary S3 Bucket
Source: https://astronomer.io/docs/astro/export-secondary-s3-bucket

Configure your Deployment to forward Airflow task logs to a secondary AWS S3 bucket.

## Export logs to a Secondary S3 Bucket

By forwarding Airflow task logs from your Astro Deployment to **an additional, customer‑managed S3 bucket**, you can keep redundant copies of your execution history, integrate with existing log‑processing pipelines, and satisfy compliance or retention requirements that extend beyond the built‑in Astro logs.

## Prerequisites

* Your Deployment must run **Astro Runtime 11.7.0** or later. See [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime).
* Your image must include `astronomer-providers-logging==1.6.2` or later. The secondary S3 logging feature was introduced in this version.
* You need an AWS account where you can create an S3 bucket and IAM resources.

## Overview of configuration options

Astro supports two ways to grant your Deployment permissions to write to the secondary bucket:

1. **Customer‑provided Workload Identity**: Recommended if you already use this mechanism for other AWS services.
2. **Assume Role**: Allows your Deployment to assume a dedicated IAM role that has access to the bucket.

Both options require the same base environment variables:

| Variable                                            | Required | Example value                                                        |
| --------------------------------------------------- | -------- | -------------------------------------------------------------------- |
| `AIRFLOW__ASTRO_SECONDARY_LOGS__S3_BUCKET_ENABLED`  | Yes      | `true`                                                               |
| `AIRFLOW__ASTRO_SECONDARY_LOGS__S3_BASE_LOG_FOLDER` | Yes      | `s3://my‑bucket/logs`                                                |
| `AIRFLOW__ASTRO_SECONDARY_LOGS__REGION`             |          | `us‑east‑1` (omit if bucket is in the same region as the Deployment) |

## Option 1 : Customer‑Provided Workload Identity

If your Deployment already uses a **customer‑managed AWS Workload Identity**, you can attach S3 write permissions directly to that role.

1. [Configure a Workload Identity](/docs/astro/authorize-deployments-to-your-cloud#setup) for the Deployment if you don’t have one yet.
2. Grant that identity permissions to write to the bucket. Example policy:

```json wrap theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ExternalS3Policy",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::my-bucket",
        "arn:aws:s3:::my-bucket/*"
      ]
    }
  ]
}
```

3. Set the following environment variables in the Deployment:

```text wrap theme={null}
AIRFLOW__ASTRO_SECONDARY_LOGS__S3_BUCKET_ENABLED=true
AIRFLOW__ASTRO_SECONDARY_LOGS__S3_BASE_LOG_FOLDER="s3://my-bucket/logs"
# Optional if bucket is in the same region as the deployment
AIRFLOW__ASTRO_SECONDARY_LOGS__REGION="us-east-1"
```

## Option 2: Assume Role

With this approach, your Deployment’s existing identity **assumes a separate IAM role** that has write access to the bucket.

#### Additional variable

| Variable                                  | Required | Example value                                          |
| ----------------------------------------- | -------- | ------------------------------------------------------ |
| `AIRFLOW__ASTRO_SECONDARY_LOGS__ROLE_ARN` | Yes      | `arn:aws:iam::123456789012:role/SecondaryS3LoggerRole` |

1. Create an IAM role with S3 write permissions (same JSON policy as above).
2. Configure the role’s **trust relationship** to allow your Deployment to assume it:

```json wrap theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/astro-deployment-role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
```

3. Set the environment variables:

```text wrap theme={null}
AIRFLOW__ASTRO_SECONDARY_LOGS__S3_BUCKET_ENABLED=true
AIRFLOW__ASTRO_SECONDARY_LOGS__S3_BASE_LOG_FOLDER="s3://my-bucket/logs"
AIRFLOW__ASTRO_SECONDARY_LOGS__ROLE_ARN="arn:aws:iam::123456789012:role/SecondaryS3LoggerRole"
# Optional if bucket is in the same region
AIRFLOW__ASTRO_SECONDARY_LOGS__REGION="us-east-1"
```

## Troubleshooting

If task logs are not appearing in the bucket:

1. Confirm that all required environment variables are set exactly as documented.
2. Verify that the IAM role attached to the Deployment, or the assumed role, includes **`s3:PutObject`** for the bucket path.
3. Check the role’s **trust relationship** (Assume Role options only).
4. Inspect the deployment worker logs for external S3 logging-related errors. Ensure that the Worker is selected in the dropdown - the default is Scheduler

If issues persist, contact Astronomer Support with the Deployment ID and any relevant error output.


# Export logs to a Secondary WASB Container
Source: https://astronomer.io/docs/astro/export-secondary-wasb

Configure your Deployment to forward Airflow task logs to a secondary Azure Storage (WASB) container.

## Export logs to a Secondary WASB Container

By forwarding Airflow task logs from your Astro Deployment to **an additional, customer‑managed Azure Storage (WASB) container**, you can keep redundant copies of your execution history, integrate with existing log‑processing pipelines, and satisfy compliance or retention requirements that extend beyond the built‑in Astro logs.

## Prerequisites

* Your Deployment must run **Astro Runtime 11.7.0** or later. See [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime).
* Your image must include **`astronomer-providers-logging==1.5.4`** or later. The secondary WASB logging feature was introduced in this version.
* An Azure **Storage Account** with a **container** created to hold the logs.
* An Azure **Managed Identity** (or Service Principal) that has **read/write** access to the container.

## Required environment variables

| Variable                                              | Description                                      | Example                              |
| ----------------------------------------------------- | ------------------------------------------------ | ------------------------------------ |
| `AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_ENABLED`         | Turn on secondary WASB logging                   | `true`                               |
| `AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_TENANT_ID`       | Azure AD tenant where the Managed Identity lives | `random‑86f1‑41af‑91ab‑2d7cd011db47` |
| `AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_CLIENT_ID`       | Client ID of the Managed Identity                | `a1b2c3d4‑e5f6‑1234‑5678‑random`     |
| `AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_STORAGE_ACCOUNT` | Name of the Storage Account (without protocol)   | `mystorageacct`                      |
| `AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_CONTAINER`       | Name of the container where logs are stored      | `airflow‑task‑logs`                  |

Add these [environment variables](/docs/astro/environment-variables) in the Deployment.

```text wrap theme={null}
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_ENABLED=true
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_TENANT_ID="<YOUR_TENANT_ID>"
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_CLIENT_ID="<YOUR_CLIENT_ID>"
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_STORAGE_ACCOUNT="<YOUR_STORAGE_ACCOUNT_NAME>"
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_CONTAINER="<YOUR_CONTAINER_NAME>"
```

## Step 1: Create Azure resources

1. [Create a Storage Account](https://learn.microsoft.com/azure/storage/common/storage-account-create?tabs=azure-portal).
2. Inside the account, create a **blob container** to hold the logs.

## Step 2: Configure Astro Workload Identity

Follow [Authorize Deployments to your Azure cloud](/docs/astro/authorize-deployments-to-your-cloud?tab=azure#setup) to assign a **Managed Identity** to the Deployment.

## Step 3: Grant storage permissions

In Azure, assign the Managed Identity the **Storage Blob Data Contributor** or equivalent scoped role to the Storage Account or container.

## Step 4: Add environment variables and redeploy

Set the [environment variables](/docs/astro/environment-variables) listed in [required environment variables](#required-environment-variables) and redeploy.

## Troubleshooting

If task logs do not appear in the container:

1. Confirm that **all required environment variables** are set and have no extra quotes or spaces.
2. Verify that the Managed Identity has **read/write access** to the Storage Account and container.
3. Check Deployment logs for authentication or authorization errors related to Azure.
4. Ensure the container exists, Astro will not create it automatically.

If issues persist, contact Astronomer Support with the Deployment ID and any relevant error output.


# Set up IP Access List on Astro
Source: https://astronomer.io/docs/astro/ip-access-list

Configure IP Access list to control the IP addresses from where your users can log in to Astro.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

You can restrict which IP addresses or IP address ranges can access the Astro service for your specific Organization. By default, Astro allows users to access their Organization from unsecured networks. However, by creating an IP access list, if your organization uses a VPN or other mechanism that limits the IP addresses your users might have, you can restrict access to Astro based on the IP addresses that you define in the Astro UI.

After you enable the IP access list, users and user-privileged resources can only interact with Astro while using a network with a permitted IP address, whether by using the Astro UI or programmatically with Astro API or Airflow API requests.

## Prerequisites

* Organization Owner permissions

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Set up IP Access list

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Security** section, click **IP Access List**.
    2. Click **+ New IP Address Range**.
    3. Add the IP address range or ranges in CIDR format, and click **Add Address Range**
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**. This opens the **General** Organization page.
    2. Select **Access Management** and then select the **IP Access List** tab.
    3. Click **+ IP Address Range**.
    4. Add the IP Address range or ranges in CIDR format.
  </Tab>
</Tabs>

<Warning>
  You must make sure that you include critical services in your IP range. These include:

  * Make sure the first IP range you add is inclusive of the IP you are currently using, or you will be locked out of your Astro Organization.
  * You must include the IP address of your identity provider (IdP), if you have one, or it will also be locked out.
</Warning>

## Restore access

Because the IP Access List limits access to the Astro UI only to specific IP addresses, you can't access the Astro UI if you're not connected to a corresponding VPN or authorized network.

To restore access for a user that is blocked, an Organization Owner needs to either:

* Disable the IP Access list setting by deleting the list or
* Add the specific blocked user's IP address to the IP Access list.

If you disable the IP Access List setting to resolve the user's access issue temporarily, remember to enable the setting again to maintain the IP address restrictions.

## Enhanced Support Access bypass

Enhanced Support Access is enabled by default for all Organizations to ensure faster, more effective assistance from the Astronomer Support team. It grants **Read-only** Admin access to your Organization’s details. If you have IP Access List enabled, Enhanced Support Access permits Astronomer Support to bypass the IP restriction and allows Astronomer Support to view your Organization details. Support can't make any changes to your Organization or resources.

See [Enhanced Support Access](/docs/astro/user-permissions#enhanced-support-access) for information about the permission scope and how to disable the feature.


# Configure tasks to run with the Kubernetes executor
Source: https://astronomer.io/docs/astro/kubernetes-executor

Learn how to configure the Pods that the Kubernetes executor runs your tasks in.

The Kubernetes executor runs each Airflow task in a dedicated Kubernetes [Pod](https://kubernetes.io/docs/concepts/workloads/pods/). On Astro, you can customize these Pods on a per-task basis using a `pod_override` configuration. If a task doesn't contain a `pod_override` configuration, it runs using the default Pod as configured in your Deployment resource settings.

<Info>This document describes how to configure individual task Pods for different use cases. To configure defaults for all Kubernetes executor task pods, see [Configure Kubernetes Pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources).</Info>

## Prerequisites

* An Astro Deployment using Astro Runtime version 8.1.0 or later.

<Warning>If you use the Kubernetes executor on Astro, you can't change the [PYTHONPATH](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/modules_management.html) of your Astro project from its default value. If you do, the Kubernetes executor will be unable to read `airflow_local_settings.py` and will fail to start up new Pods.</Warning>

## Customize a task's Kubernetes Pod

<Tip>By default, Astro supports a maximum `KubernetesExecutor` Pod size of 43 vCPU and 86 GiB of memory. Astro can support Pod sizes up to 86 vCPU and 172 GiB of memory on request. Contact [Astro support](/docs/astro/astro-support) if you need to run larger jobs on `KubernetesExecutor`.</Tip>

<Danger>While you can customize all values for a worker Pod, Astronomer does not recommend configuring complex Kubernetes infrastructure in your Pods, such as sidecars. These configurations have not been tested by Astronomer.</Danger>

For each task running with the Kubernetes executor, you can customize its individual worker Pod and override the defaults used in Astro by configuring a `pod_override` file.

1. Add the following import to your dag file:

   ```python wrap theme={null}
   from kubernetes.client import models as k8s
   ```

2. Add a `pod_override` configuration to the dag file containing the task. See the [`kubernetes-client`](https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Container.mdx) GitHub for a list of all possible settings you can include in the configuration.

3. Specify the `pod_override` in the task's parameters.

See the following example of a `pod_override` configuration.

### Example: Set CPU or memory limits and requests

You can request a specific amount of resources for a Kubernetes worker Pod so that a task always has enough resources to run successfully. When requesting resources, make sure that your requests don't exceed the resource limits in your Deployment's [max pod size](/docs/astro/deployment-resources#configure-kubernetes-pod-resources).

The following example shows how you can use a `pod_override` configuration in your dag code to request custom resources for a task:

```python expandable wrap theme={null}
import pendulum
import time

from airflow.models.dag import DAG
from airflow.decorators import task
from airflow.operators.bash import BashOperator
from airflow.example_dags.libs.helper import print_stuff
from kubernetes.client import models as k8s

k8s_exec_config_resource_requirements = {
    "pod_override": k8s.V1Pod(
        spec=k8s.V1PodSpec(
            containers=[
                k8s.V1Container(
                    name="base",
                    resources=k8s.V1ResourceRequirements(
                        requests={"cpu": 0.5, "memory": "1024Mi", "ephemeral-storage": "1Gi"},
                        limits={"cpu": 0.5, "memory": "1024Mi", "ephemeral-storage": "1Gi"}
                    )
                )
            ]
        )
    )
}

with DAG(
    dag_id="example_kubernetes_executor_pod_override_sources",
    schedule=None,
    start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
    catchup=False
):
    BashOperator(
      task_id="bash_resource_requirements_override_example",
      bash_command="echo hi",
      executor_config=k8s_exec_config_resource_requirements
    )

    @task(executor_config=k8s_exec_config_resource_requirements)
    def resource_requirements_override_example():
        print_stuff()
        time.sleep(60)

    resource_requirements_override_example()
```

When this dag runs, it launches a Kubernetes Pod with exactly 0.5m of CPU and 1024Mi of memory, as long as that infrastructure is available in your Deployment. After the task finishes, the Pod terminates gracefully.

<Warning>
  On Astro Hosted, Astro automatically sets resource requests equal to limits for KubernetesExecutor worker Pods. This ensures Pods receive a Kubernetes [Guaranteed Quality of Service (QoS) class](https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#guaranteed), which prevents resource contention and eviction. Because Astro uses the limit values as both requests and limits, your Pods are billed based on the limits you configure, even if actual usage is lower. To avoid unexpected charges, set limits close to the resources your task requires. Check your [Billing and usage](/docs/astro/manage-billing) to view your resource use and associated charges.
</Warning>

## Use secret environment variables in worker Pods

On Astro Deployments, secret [environment variable](/docs/astro/environment-variables) values are stored in a Kubernetes secret called `env-secrets`. These environment variables are available to your worker Pods, and you can access them in your tasks just like any other environment variable. For example, you can use `os.environ[<your-secret-env-var-key>]` or `os.getenv(<your-secret-env-var-key>, None)` in your dag code to access the variable value.

However, if you can’t use Python, or you are using a pre-defined code that expects specific keys for environment variables, you must pull the secret value from `env-secrets` and mount it to the Pod running your task as a new Kubernetes Secret.

1. Add the following import to your dag file:

   ```python wrap theme={null}
   from airflow.kubernetes.secret import Secret
   ```

2. Define a Kubernetes `Secret` in your dag instantiation using the following format:

   ```python wrap theme={null}
   secret_env = Secret(deploy_type="env", deploy_target="<VARIABLE_KEY>", secret="env-secrets", key="<VARIABLE_KEY>")
   namespace = conf.get("kubernetes", "NAMESPACE")
   ```

3. Specify the `Secret` in the `secret_key_ref` section of your `pod_override` configuration.

4. In the task where you want to use the secret value, add the following task-level argument:

   ```python wrap theme={null}
   op_kwargs={
           "env_name": secret_env.deploy_target
   },
   ```

5. In the executable for the task, call the secret value using `os.environ[env_name]`.

In the following example, a secret named `MY_SECRET` is pulled from `env-secrets` and printed to logs.

```python expandable wrap theme={null}
import pendulum
from kubernetes.client import models as k8s

from airflow.configuration import conf
from airflow.kubernetes.secret import Secret
from airflow.models import DAG
from airflow.operators.python import PythonOperator


def print_env(env_name):
    import os
    print(os.environ[env_name])

with DAG(
        dag_id="test-secret",
        start_date=pendulum.datetime(2022, 1, 1, tz="UTC"),
        end_date=pendulum.datetime(2022, 1, 5, tz="UTC"),
        schedule="@once",
) as dag:
    secret_env = Secret(deploy_type="env", deploy_target="MY_SECRET", secret="env-secrets", key="MY_SECRET")
    namespace = conf.get("kubernetes", "NAMESPACE")

    p = PythonOperator(
        python_callable=print_env,
        op_kwargs={
            "env_name": secret_env.deploy_target
        },
        task_id="test-py-env",
        executor_config={
            "pod_override": k8s.V1Pod(
                spec=k8s.V1PodSpec(
                    containers=[
                        k8s.V1Container(
                            name="base",
                            env=[
                                k8s.V1EnvVar(
                                    name=secret_env.deploy_target,
                                    value_from=k8s.V1EnvVarSource(
                                        secret_key_ref=k8s.V1SecretKeySelector(name=secret_env.secret,
                                                                               key=secret_env.key)
                                    ),
                                )
                            ],
                        )
                    ]
                )
            ),
        }
    )
```

## See also

* [Configure Kubernetes Pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources)
* [How to use cluster ConfigMaps, Secrets, and Volumes with Pods](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html#how-to-use-cluster-configmaps-secrets-and-volumes-with-pod)
* [Run the `KubernetesPodOperator` on Astro](/docs/astro/kubernetespodoperator)
* [Airflow Executors explained](/docs/learn/airflow-executors-explained)


# Lineage
Source: https://astronomer.io/docs/astro/lineage-graph

View lineage graph in Astro Observe.

Astro visualizes real-time lineage based on Airflow run metadata. You can view the lineage for a specific [data product](/docs/astro/create-data-products) or [asset](/docs/astro/assets-overview) to understand data dependencies and run history.

## View lineage

To view lineage, navigate to a [data product](/docs/astro/create-data-products) or [asset](/docs/astro/assets-overview) and click **Open Lineage**.

By default, the lineage graph displays dags, tasks, datasets, and tables that ran or were updated any time in the last 90 days. You can update this time range by changing the **Lineage captured from** filter in the top right of the lineage graph.
Click any asset to highlight its upstream and downstream dependencies in the graph and see asset metadata, like an **Event Timeline** of its run or update history and which **Data Products** the asset belongs to.

<Frame>
  <img alt="A graph of a data product's lineage." />
</Frame>

## Search and filter lineage

Use the controls to customize your view of the lineage graph:

* **Search**: Use the search box in the upper left of the graph to find specific assets by name.
* **Lineage captured from**: Determines which assets appear in the graph. Any asset with a run or update event in the selected time range will appear in the graph.
* **Hide Datasets**: Toggle to hide or show dataset nodes from the graph to reduce noise.
* **Highlight Last Run Status**: Toggle to highlight assets based on their last run status. The **Before** filter allows you to see run status before a specific point in time. For example, you can view the last run status before the timestamp of a known failure to see which exactly task in your pipeline failed. This time filter also affects which events show up in the **Event Timeline** tab of a selected asset.
* **Layers**: **Highlight Deployments** highlights Astro Deployments across the lineage graph. **Highlight Key Assets** highlights the final assets in data product graphs, and the primary asset in asset graphs.

<Frame>
  <img alt="A snapshot of the controls in the lineage graph." />
</Frame>

## Adjust graph depth

By default, data product graphs include 10 levels upstream of the selected final assets. In asset graphs, you can control how many levels of dependencies to display:

* **Upstream**: Adjust the number of upstream dependency levels to show using the **+** and **-** buttons.
* **Downstream**: Adjust the number of downstream dependency levels to show using the **+** and **-** buttons.

The graph depth controls help you focus on the most relevant dependencies for your analysis.

## View asset details in lineage

Click any asset in the lineage graph to open a side panel with detailed information, including:

* **Basic**: Asset metadata including FQDN, Name, and Namespace
* **Dag Metadata**: Owner and Tags (for Airflow assets)
* **Source**: Last updated timestamp and last updated by information

The side panel also includes tabs for:

* **Impact Analysis**: View assets downstream of the selected asset. By default, this view only shows leaf nodes, or nodes with no further downstream assets. You can change this by toggling the **Only Leaf Assets** control.
* **Upstream Issues**: See any issues in upstream dependencies. By default, this view only shows upstream issues that had a failure or anomaly. You can change this by toggling the **Only Problems** control.
* **Event Timeline**: View a history of run events or update events for this asset. Failed dag and task runs display a summary of the failure logs for troubleshooting.
* **Data Products**: See which data products this asset is part of to understand the potential impact across the business.


# Manage Deployments programmatically using Deployment files
Source: https://astronomer.io/docs/astro/manage-deployments-as-code

Manage an Astro Deployment using a Deployment file in YAML or JSON format

You can configure Deployments programmatically using Deployment files and Deployment template files. *Deployment files* are used to update the same Deployment programmatically, and *Deployment template files* are used to create new Deployments based on a single template.

Managing Deployments with files is essential to automating Deployment management at scale.  For example, you can:

* Create a template file in a central GitHub repository and use it as a source of truth for new Deployments that fit a particular use case. For example, you can standardize your team's development Deployments by creating a template file with configurations for that type of Deployment.
* Create a Deployment file that represents the configurations of an existing Deployment and store it in your GitHub repository. You can make changes to this file to update a Deployment  using CI/CD, which maintains the history of your changes.

Use this document to learn how to create and manage Deployment files and Deployment template files. See the [Deployment file reference](/docs/astro/deployment-file-reference) for a list of configurable Deployment file values. When you're ready to programmatically run Deployment file workflows, see [Authenticate your workflow](/docs/astro/automation-authentication).

## Create a template file or Deployment file

To create a template file based on an existing Deployment, run the following command:

```sh wrap theme={null}
astro deployment inspect <deployment-id> --template > <your-deployment-template-file-name>.yaml
```

To create a Deployment file based on an existing Deployment, run the following command:

```sh wrap theme={null}
astro deployment inspect <deployment-id> > <your-deployment-file-name>.yaml
```

Alternatively, you can manually create a template file without using an existing Deployment as explained in [Create a Deployment using a template file](#create-a-deployment-using-a-template-file).

## Create a Deployment using a template file

<Info>These are the minimum values required to create a Deployment using a template file. Any configurations not specified are set to default values. To add more configurations, see [Deployment file reference](/docs/astro/deployment-file-reference).</Info>

1. Copy one of the following templates to a local `yaml` file:

<Tabs>
  <Tab title="Standard cluster">
    ```yaml title="deployment.yaml" wrap theme={null}
    deployment:
        configuration:
            name: <your-deployment-name>
            deployment_type: HOSTED_SHARED
            cloud_provider: aws
            description: <deployment-description>
            runtime_version: 9.1.0
            dag_deploy_enabled: true
            executor: CeleryExecutor
            cluster_name: us-east-1
            region: us-east-1
            workspace_name: <your-workspace-name>
            scheduler_size: small
    ```

    Note that for Deployments on a standard cluster, the `region` and `cluster-name` parameters must both contain the region name for the standard cluster. See [Available regions for your cloud provider](/docs/astro/resource-reference-hosted#standard-cluster-regions).
  </Tab>

  <Tab title="Dedicated cluster">
    ```yaml title="deployment.yaml" wrap theme={null}
      deployment:
          configuration:
              name: <your-deployment-name>
              deployment_type: HOSTED_DEDICATED
              cloud_provider: aws
              description: <deployment-description>
              runtime_version: 9.1.0
              dag_deploy_enabled: true
              executor: KubernetesExecutor
              cluster_name: <your-cluster-name>
              region: us-east-1
              workspace_name: <your-workspace-name>
              scheduler_size: small
    ```

    The `cluster_name` field must include the name of the dedicated cluster that exists in your Astro Organization.
  </Tab>
</Tabs>

2. Adjust the template file values for the Deployment you want to create. When working with template files, keep the following in mind:

   * The `name` field must include a unique name within the Workspace.

   * The `workspace_name` field must include a valid Workspace name that exists in your Astro Organization.

   * The possible values for `cloud_provider`, `executor`, and `scheduler_size` are the same possible values when you create a Deployment with [`astro deployment create`](/docs/cli/v1.43/astro-deployment-create#options-astro).

   * See [Airflow and Astro Runtime version parity](/docs/runtime/runtime-image-architecture#astro-runtime-and-apache-airflow-parity) to choose your Astro Runtime version.

See [Deployment file reference](/docs/astro/deployment-file-reference) for a list of all configurable Deployment template file values.

3. Run the following command to create the Deployment:

   ```sh wrap theme={null}
   astro deployment create --deployment-file <your-deployment-file-name>
   ```

4. (Optional) Either open the Astro UI or run the following command to confirm that you successfully created your Deployment:

   ```sh wrap theme={null}
   astro deployment list
   ```

5. (Optional) Reconfigure any Airflow connections or variables from the Deployment that you copied into the template file. Airflow connections and variables cannot be configured using template files. See [Manage connections in Airflow](/docs/astro/manage-connections-variables).

## Update a Deployment using a Deployment file

A Deployment file is a complete snapshot of an existing Deployment at the point you inspected it. It's similar to a template file, but also contains your Deployment's name, description, and metadata. In the same way you use a template file to create a new Deployment, you use a Deployment file to update an existing Deployment with a new set of configurations.

When you update a Deployment with a Deployment file, keep the following in mind:

* You can’t change the cluster or Workspace the Deployment runs on. To transfer a Deployment to a different Workspace, see [Transfer a Deployment](/docs/astro/transfer-a-deployment).
* You can't change the Astro Runtime version of the Deployment. To upgrade Astro Runtime, you must update the Dockerfile in your Astro project. See [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime).
* Environment variables marked as secret in the Astro UI will be exported with a blank `value` to your Deployment file. To redeploy using the Deployment file, you either need to provide the `value` again in the Deployment file or delete the object for the variable. Otherwise, `astro deployment create` will fail. See [`deployment.environment_variables`](/docs/astro/deployment-file-reference#deployment-environment_variables) for more details.

<Danger>When you update a Deployment with a Deployment file, you must push a complete Deployment file that lists all of your existing worker queues. If a worker queue exists on Astro but doesn't exist in your Deployment file, the worker queue is deleted when you push your Deployment file.</Danger>

To update a Deployment using a Deployment file:

1. Inspect an existing Deployment and create a Deployment file for its current configurations:

```sh wrap theme={null}
astro deployment inspect <deployment-id> > <your-deployment-file-name>.yaml
```

2. Modify the Deployment file and save your changes. See [Deployment file reference](/docs/astro/deployment-file-reference) for fields that you can modify.

3. Update your Deployment according to the configurations in the Deployment file:

```sh wrap theme={null}
astro deployment update <deployment-id> --deployment-file <your-deployment-file>
```

4. (Optional) Confirm that your Deployment was updated successfully by running the following command. You can also go to the Deployment page in the Astro UI to confirm the new values.

```sh wrap theme={null}
astro deployment inspect <deployment-id>
```

## See also

* [Authenticate an automation tool to Astro](/docs/astro/automation-authentication)
* [Deploy Code](/docs/astro/deploy-code)
* [Choose a CI/CD Strategy for deploying code to Astro](/docs/astro/set-up-ci-cd)


# Create and manage domains for your Organization
Source: https://astronomer.io/docs/astro/manage-domains

Create and map domains to single sign-on platforms on Astro.

Authorization policies in Astro are based on email domains, which ensures that all users with an approved email domain authenticate with a specific authentication method. Domains can be mapped to single sign-on (SSO) methods so that users who have an email with a domain are automatically directed to your SSO platform when they sign in to Astro.

You can only configure a single domain for an SSO connection. However, if you have multiple managed domains, then you can set up multiple different SSO connections, with a separate SSO connection for each domain.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Create a domain

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.
    2. Click **+ New Managed Domain**.
    3. In the **Domain** field, enter the domain that you want to map.
    4. Click **Create Managed Domain**. The domain is added to your managed domains and marked as **Unverified**.
    5. In the entry for your domain, click **Verify Domain**. A new window appears with steps to verify your domain.
    6. Follow the steps to verify your domain. If you can't complete the steps yourself, contact the administrator for your domain and ask them to complete the steps instead.
    7. Click **Verify Domain** to notify Astro to check your DNS record.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**, then click **Authentication**.
    2. In the **Managed Domains** menu, click **Managed Domain**.
    3. In the **Domain** field, enter the domain that you want to map.
    4. Click **Create**. The domain is added to your **Managed Domains** and marked as **Unverified**.
    5. In the entry for your domain, click **Verify**. A new window appears with steps to verify your domain.
    6. Follow the steps to verify your domain. If you can't complete the steps yourself, contact the administrator for your domain and ask them to complete the steps instead.
    7. Click **Verify** to notify Astro to check your DNS record.
  </Tab>
</Tabs>

<Info>Typically, a change in the DNS record takes only minutes to propagate; however, there are cases where it may take up to 72 hours.</Info>

After your domain is verified, you can map the domain to an SSO connection. Mapping a domain ensures that all users with the same email address domain have the same authentication experience when they sign in to Astro. You can later use this mapping to enforce specific sign in methods for users with emails from a specific domain. See [Configure SSO](/docs/astro/configure-idp#configure-your-sso-identity-provider).

## Delete a domain

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.
    2. Next to the domain you want to delete, click the trash bin.
    3. Follow the prompts to confirm the deletion.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**, then click **Authentication**.
    2. In the **Managed Domains** menu, click the trash bin next to the domain you want to delete.
    3. Follow the prompts to confirm the deletion.
  </Tab>
</Tabs>


# Manage environment variables on Astro
Source: https://astronomer.io/docs/astro/manage-env-vars

Learn how to manage environment variables on Astro

This document covers how to manage **Deployment-level environment variables** on Astro. For information about creating **Workspace-level environment variables** that can be shared across multiple Deployments, see [Create environment variables in Astro](/docs/astro/create-and-link-environment-variables).

On Astro, you can create, update, or delete Deployment-level environment variables in the following ways:

* Using the Deployment's **Environment Variables** tab in your Deployment's **Environment** settings.
* Using your Astro project `Dockerfile`.

The way you manage environment variables can affect security and access for your variable data. See [Choose a strategy](/docs/astro/environment-variables#choose-a-strategy) to determine which management strategy is right for your use case.

Additionally, you can test environment variables from your local Astro environment and export them to the Astro UI. See [Manage environment variables locally](#manage-environment-variables-locally).

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Use the Astro UI

Setting environment variables using the Astro UI is the quickest and easiest way to manage Deployment-level environment variables on Astro.

<Info>
  When you view the **Environment Variables** page for a Deployment, you'll see both Deployment-level environment variables (which you can edit on this page) and Workspace-level environment variables (which are managed through the [Workspace Environment Manager](/docs/astro/create-and-link-environment-variables)). Deployment-level environment variables take precedence over Workspace-level environment variables with the same key.
</Info>

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.

    2. Click the **Environment** tab.

    3. Click **Environment variables**.

    4. Click **Edit Deployment Variables** (or **+ New Environment Variable** if you have no variables configured yet).

    5. Enter an environment variable key and value. For sensitive credentials that should be treated with an additional layer of security, select the **Secret** checkbox. This permanently hides the variable's value from all users in your Workspace.

    6. Click **Update Environment Variables** to save your changes. Your Airflow scheduler, webserver, and workers restart. After saving, it can take up to two minutes for new variables to be applied to your Deployment.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.

    2. Click the **Environment** tab.

    3. Click **Environment variables**.

    4. Click **Edit Deployment Variables** (or **+ New Environment Variable** if you have no variables configured yet).

    5. Enter an environment variable key and value. For sensitive credentials that should be treated with an additional layer of security, select the **Secret** checkbox. This permanently hides the variable's value from all users in your Workspace.

    6. Click **Update Environment Variables** to save your changes. Your Airflow scheduler, webserver, and workers restart. After saving, it can take up to two minutes for new variables to be applied to your Deployment.
  </Tab>
</Tabs>

### Edit or delete existing values

After you set an environment variable key, only the environment variable value can be modified. You can modify environment variables that are set as **Secret**, but the existing secret variable value is never shown. When you modify a secret environment variable, the existing value is erased and you are prompted to enter a new value.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.

    2. Click the **Environment** tab.

    3. Click the **Environment Variables** tab.

    4. Click **Edit Deployment Variables**.

    5. Modify the value of the variable you want to edit.

       <Frame>
         <img alt="Edit value location" />
       </Frame>

    6. Click **Update Environment Variables** to save your changes. Your Airflow scheduler, webserver, and workers restart. After saving, it can take up to two minutes for updated variables to be applied to your Deployment.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.

    2. Click the **Environment** tab.

    3. Click the **Environment Variables** tab.

    4. Click **Edit Deployment Variables**.

    5. Modify the value of the variable you want to edit.

       <Frame>
         <img alt="Edit value location" />
       </Frame>

    6. Click **Update Environment Variables** to save your changes. Your Airflow scheduler, webserver, and workers restart. After saving, it can take up to two minutes for updated variables to be applied to your Deployment.
  </Tab>
</Tabs>

## Using your Dockerfile

If you want to store environment variables with an external version control tool, Astronomer recommends setting them in your `Dockerfile`. This file is automatically created when you first initialize an Astro project using `astro dev init`.

<Warning>Environment variables set in your `Dockerfile` are stored in plain text. For this reason, Astronomer recommends storing sensitive environment variables using the Astro UI or a third-party secrets backend. For more information, see [Configure a secrets backend](/docs/astro/secrets-backend).</Warning>

1. Open your Astro project `Dockerfile`.

2. To add the environment variables, declare an ENV command with the environment variable key and value. For example, the following `Dockerfile` sets two environment variables:

   ```dockerfile title="Dockerfile" wrap theme={null}
   FROM astrocrpublic.azurecr.io/runtime:3.1-5
   ENV AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG=1
   ENV AIRFLOW_VAR_MY_VAR=25
   ```

3. Save your Dockerfile and run `astro deploy` to deploy your variables to an Astro Deployment. To apply your changes locally, use `astro dev restart` to rebuild your image.

4. (Optional) To verify if the environment variables are applied correctly to Astro Deployment or your local Airflow environment, you can use `os.getenv("AIRFLOW_VAR_MY_VAR")` inside of Airflow Dags and tasks.

To delete an environment variable from your Astro Runtime image, remove or comment the line in your `Dockerfile` that defines it.

<Info>Environment variables set in your Dockerfile aren't visible in the Astro UI.</Info>

## Manage environment variables locally

You can use the Astro CLI to set environment variables on Astro and your local Airflow environment. If you're developing locally, the best way to manage environment variables is using your Astro project `.env` file.

1. Open your Astro project `.env` file.

2. Use the following format to set your environment variables in the `.env` file:

   ```text wrap theme={null}
   KEY=VALUE
   ```

   Environment variables should be in all-caps and not include spaces.

   Alternatively, you can run `astro deployment variable list --save` to copy environment variables from an existing Deployment to a file.

3. Restart your local environment using `astro dev restart`.

To confirm that your environment variables were applied:

1. Run `astro dev bash --scheduler` to sign in to the scheduler container.
2. Run `printenv | grep <your-env-variable>` in the container to print all environment variables that are applied to your environment.
3. Run `exit` to exit the container.

To export the contents of your `.env` file to an Astro Deployment, run the following command:

```sh wrap theme={null}
astro deployment variable update --deployment-id <your-deployment-id> --load .env
```

<Warning>
  When you use the `.env` file to add or update environment variables on Astro, it will overwrite all existing variables in your Astro Deployment. To update only select environment variables, run `astro deployment variable create` without the `--load` option. For example, the following command creates two new environment variables without affecting existing Deployment environment variables:

  ```sh wrap theme={null}
  astro deployment variable create AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT=60 ENVIRONMENT_TYPE=dev --deployment-id cl03oiq7d80402nwn7fsl3dmv
  ```
</Warning>

After you deploy environment variables, your Deployment automatically restarts and applies the variables. To verify if the environment variables were applied correctly, go to **Environment** > **Environment Variables** of your Deployment settings in the Astro UI.

### Use multiple .env files

The Astro CLI looks for `.env` by default, but if you want to specify multiple files, make `.env` a top-level directory and create sub-files within that folder, as shown in the example directory below. If you make `.env` a top-level directory, make sure to specify the specific filepath(s) with the `--env` flag. For example, to specify `dev.env` as your environment variable file with the `astro dev start` command:

```sh wrap theme={null}
astro dev start --env .env/dev.env
```

```text wrap theme={null}
my_project
├── Dockerfile
├── dags
│   └── my_dag
├── include
│   └── my_operators
├── airflow_settings.yaml
└── .env
    ├── dev.env
    └── prod.env
```

## See also

* [Set Airflow connections](/docs/learn/connections#define-connections-with-environment-variables) using environment variables.
* [Set Airflow variables](/docs/learn/airflow-variables#using-environment-variables) using environment variables.
* [Import and export environment variables](/docs/astro/import-export-connections-variables#from-environment-variables)


# Manage users in an Astro Organization
Source: https://astronomer.io/docs/astro/manage-organization-users

Add, edit, or remove users within an Organization on Astro.

As an Organization Owner, you can add new team members to Astro and grant them user roles with permissions for specific actions across your Organization.

To manage users at the Workspace level, see [Manage Workspace users](/docs/astro/manage-workspace-users). To manage groups of users, see [Manage Teams](/docs/astro/manage-teams).

## Prerequisites

* Organization Owner permissions.

For more information on user roles, see [Manage user permissions on Astro](/docs/astro/user-permissions).

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Add a user to an Organization

If your Organization has [configured an identity provider (IdP) with Astro](/docs/astro/configure-idp#configure-your-sso-identity-provider), assign users to Astro from your identity provider. By default, any users that you assign can join your Organization as an Organization Member without an invite. To change this behavior, see [Disable just-in-time provisioning](/docs/astro/configure-idp#configure-just-in-time-provisioning).

You can also manually add users to an Organization. You must manually add users in the following circumstances:

* You don't have an IdP configured.
* You disabled just-in-time provisioning.
* You want to invite a user to an Organization from a domain that you don't own, such as a third party consultant.
* You want to invite someone from your company to Astro as a role other than Organization Member.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Access Management** section, click **Users**.
    2. Click **+ Invite User**.
    3. Enter the user's email.
    4. Select an Organization role for the user. See [Organization roles reference](/docs/astro/user-permissions#organization-roles).
    5. Click **Invite User**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings** > **Access Management**.
    2. Click **Invite member**:
    3. Enter the user's email.
    4. Select an Organization role for the user. See [Organization roles reference](/docs/astro/user-permissions#organization-roles).
    5. Click **Invite member**.
  </Tab>
</Tabs>

After you add the user, their information appears in the **Users** tab in **Access Management**. To access Astro, the user needs to accept the invitation sent by email and then create an Astro account.

## Update or remove an Organization user

See [User permissions](/docs/astro/user-permissions) to view the permissions for each available Organization role.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Access Management** section, click **Users**.
    2. Find the user in the **Users** list.
    3. To edit the user's role, click the user's **More actions** menu (⋯), select **Edit User**, change their access type, then click **Update User**.
    4. To remove the user, click the user's **More actions** menu (⋯), select **Remove User…**, then click **Yes, Continue**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings** > **Access Management**.
    2. Find the user in the **Users** list. To view or manage the user's Organization, Workspace, and Deployment roles, either click the user's row to open the **User Details** page, or open the action menu (⋯) on the user's row and select **Edit User**.
    3. (Optional) Edit the user's role. See [Organization roles](/docs/astro/user-permissions). If you opened the action menu, you can change the user's Organization role from the **Edit User** option. On the **User Details** page you can view and edit the user's Organization, Workspace, Deployment, and Dag roles.
    4. If you updated the user's role, click **Update member**. To delete the user, open the action menu and select **Remove user…**, or click **Remove member** on the **User Details** page.
  </Tab>
</Tabs>

<Info>To remove yourself from an Organization as an Organization Owner, one or more Organization Owners must be assigned to the Organization. If you're the only Organization Owner for your Organization, you'll need to assign another Organization Owner before removing yourself from the Organization.</Info>

## Add a group of users to Astro using the Astro CLI

You can use the Astro CLI and a shell script to add multiple users to an Organization at once. The shell script reads from a text file which contains user information. You can generate the text file for each new batch of users that need to assigned to an Organization and run the script using the Astro CLI.

1. Create a text file named `users.txt`.

2. Open the text file. On each line, add a user's email and their role separated by a space. The following is an example of how you can write a list for inviting users to an Organization:

   ```text wrap theme={null}
   user1@astronomer.io ORGANIZATION_MEMBER
   user2@astronomer.io ORGANIZATION_OWNER
   user3@astronomer.io ORGANIZATION_BILLING_ADMIN
   user4@astronomer.io ORGANIZATION_OWNER
   ```

3. Create a file named `add-users.sh` and then add the following script to it:

   ```bash title="add-users.sh" wrap theme={null}
   #!/bin/bash

   # Check if a file was provided as an argument
   if [ $# -ne 1 ]; then
       echo "Usage: $0 <file>"
       exit 1
   fi
   # Loop through the file to read each user email address and the role, and use Astro CLI to invite them
   while read line; do
       email=$(echo "$line" | cut -d' ' -f1)
       role=$(echo "$line" | cut -d' ' -f2)
       echo "Inviting $email as $role..."
       astro organization invite "$email" --role "$role"
   done < "$1"
   ```

4. Sign in to the Astro CLI using `astro login`, and then run `astro organization list` to ensure that you're in the same Organization where you want to add the users. If you're not in the right Organization, run `astro organization switch`.

5. Run the following command to execute the shell script:

   ```sh wrap theme={null}
   sh path/to/add-users.sh path/to/users.txt
   ```

6. (Optional) To use this script as part of a CI/CD pipeline, create an [Organization API token](/docs/astro/organization-api-tokens) and specify the environment variable `ASTRO_API_TOKEN=<your-token>` in your CI/CD environment.

## See also

* [Manage Workspace users](/docs/astro/manage-workspace-users)
* [Manage Teams](/docs/astro/manage-teams)
* [Manage user permissions on Astro](/docs/astro/user-permissions)
* [Dag-level access control](/docs/astro/dag-level-access-control)


# Configure Teams on Astro
Source: https://astronomer.io/docs/astro/manage-teams

Create, delete, and update Teams on Astro.

<Note> This is feature is only available if you are on the **Team** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

<Info>Team plans are limited to two Teams, but Business plans allow you to have unlimited Teams. See [pricing](https://www.astronomer.io/pricing/).</Info>

As an Organization Owner or Workspace Owner, you can use Teams to batch assign Organization and Workspace roles to groups of users. Organization Owners create, update, or delete Teams. Then, either Organization Owners or Workspace Owners can assign Teams to different Workspaces and define their [Workspace permissions](/docs/astro/user-permissions#workspace-roles). For Deployments running Astro Runtime 3.1-12 or later, you can also assign Teams to individual Dags with Dag-specific roles. See [Dag-level access control](/docs/astro/dag-level-access-control#assign-dag-roles-to-teams).

A *Team* is a group of users in an Organization that share the same Organization and Workspace permissions. You can use Teams to securely assign permissions for a large group of users across multiple Workspaces. For example, you can create a Team of dag authors, then assign that Team to each of your development Workspaces as a Workspace author.

You can also assign different roles for each Workspace. For example, you can have a group of dag authors that has full Workspace Owner permissions for development Workspaces, and that same Team can have only Workspace Member permissions for production Workspaces.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Create a Team

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Access Management** section, click **Teams**.
    2. Click **+ New Team**.
    3. In the **New Team** panel, enter a **Team Name** and **Team Description**, select an **Organization Role**, and optionally **Add Users**. If you can't find a user, you might need to [add them to your Organization](/docs/astro/manage-organization-users#add-a-user-to-an-organization) first.
    4. Click **Create Team**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings** > **Access Management**.

    2. Click **Teams**.

    3. Click **+ Team** to create a new Team.

    4. Configure the following details about your Team:

       * **Team Name**: The name for your Team.
       * **Team description**: (Optional) The description for your Team.
       * **Organization role**: The Organization role for your Team.
       * **Add users**: Choose the Organization users you want to add to the Team.

       If you don't find the user you want to add, you might need to [add the user to your Organization](/docs/astro/manage-organization-users#add-a-user-to-an-organization).

    5. After you finish adding users to the Team, click **Add Team**.
  </Tab>
</Tabs>

You can now [add your Team to a Workspace](/docs/astro/manage-teams#add-a-team-to-a-workspace) and define the Team users' permissions in the Workspace.

## Update existing Teams

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Access Management** section, click **Teams**.
    2. Click a Team to open its detail page. The **Members** tab is selected by default.
    3. To add a member, click **+ Add Member**, use the **Add Users** dropdown to find the user, then click **Add Members**.
    4. To remove a member, hover over their row and click the trash icon.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings** > **Access Management**.
    2. Click **Teams**.
    3. Click the name of the Team you want to update.
    4. Update your Team:

       * Click **+ Member** to add an existing Organization member to your Team.
       * Click the delete icon to remove Team members.
  </Tab>
</Tabs>

## Add a Team to a Workspace

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Workspaces**.
    2. Select your Workspace, then go to **Access Management** > **Teams**.
    3. Click **+ Add Team**.
    4. Select the **Team** you want to add and define their **Workspace Role**, which determines their [Workspace user permissions](/docs/astro/user-permissions#workspace-roles).
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace and click **Workspace Settings** > **Access Management**.
    2. Click **Teams**.
    3. Click **+ Team**.
    4. Select the **Team** you want to add and define their **Workspace Role**, which determines their [Workspace user permissions](/docs/astro/user-permissions#workspace-roles).
  </Tab>
</Tabs>

<Info>
  **Centralized access management for Organization Owners**

  [Organization Owners](/docs/astro/user-permissions#organization-roles) can also add or update a Team for a Workspace from your Organization's access management settings:

  <Tabs>
    <Tab title="New Astro UI">
      1. In the Astro UI, go to **Settings**, then in the **Access Management** section, click **Teams**.
      2. Select the Team.
      3. Click the **Workspaces** tab, then click **+ Add Workspace** to add them to a Workspace, or **Edit Role** to change their role for an existing Workspace.
    </Tab>

    <Tab title="Legacy UI">
      1. In the Astro UI, click **Organization Settings** > **Access Management**.
      2. Select the Team to add or update.
      3. Click the **Workspaces** tab, then click **+ Workspace** to add them to a Workspace, or open the **More actions** menu (⋯) and select **Edit role** next to an existing Workspace to change their role.
    </Tab>
  </Tabs>
</Info>

<Info>
  **CLI**

  You can add a Team to multiple Workspaces programmatically using the Astro CLI. See [`astro workspace team add`](/docs/cli/v1.43/astro-workspace-team-add) for example output and commands.
</Info>

## Add a Team to a Dag

You can assign a Team to specific Dags within a Deployment, giving all Team members the same Dag-level permissions. For complete instructions, see [Assign Dag roles to Teams](/docs/astro/dag-level-access-control#assign-dag-roles-to-teams).

## Add a Team to multiple Workspaces using the Astro CLI

You can use the Astro CLI and a shell script to add a Team to multiple Workspaces at once. The shell script reads from a text file which contains Team information. You can generate a text file for each Team that needs to be assigned to Workspaces and run a script to process the file. You must have Organization Owner or Workspace Owner level permissions to add Teams to Workspaces.

1. Create a text file named `teams.txt`.

2. Open the text file. On each line, add a Team ID, the Team's role, and the Workspace ID delimited by spaces. Your text file should look similar to the following:

   ```text wrap theme={null}
   uclk17xqgm124q01hkrgilsr49 WORKSPACE_MEMBER tbkj96wpfl913p90glqfgkrq398
   uclk17xqgm124q01hkrgilsr49 WORKSPACE_OWNER salk85voek802q89fkpefjqp287
   uclk17xqgm124q01hkrgilsr49 WORKSPACE_OPERATOR rzkj74undj791p78ejofeipo178
   vdml28yrhn235r12ilshjmts50 WORKSPACE_OWNER tbkj96wpfl913p90glqfgkrq398
   ```

3. Create a file named `add-teams.sh` and add the following script to it:

   ```bash title="add-teams.sh" wrap theme={null}
   #!/bin/bash

   # Check if a file was provided as an argument
   if [ $# -ne 1 ]; then
       echo "Usage: $0 <file>"
       exit 1
   fi

   while read line; do
       team_id=$(echo "$line" | cut -d' ' -f1)
       role=$(echo "$line" | cut -d' ' -f2)
       workspace_id=$(echo "$line" | cut -d' ' -f3)
       echo "Inviting ${team_id} to ${workspace_id} as $role..."
       astro workspace team add "$team_id" --role "$role" --workspace-id "$workspace_id"
   done < "$1"
   ```

4. (Optional) Log in to the Astro CLI using `astro login`, then run `astro workspace list` to ensure that you have access to the Workspaces where you want to add the users.

5. Run the following command to execute the shell script:

   ```sh wrap theme={null}
   sh path/to/add-teams.sh path/to/teams.txt
   ```

6. (Optional) To use this script as part of a CI/CD pipeline, create an [Organization API token](/docs/astro/organization-api-tokens) and specify the following environment variable in your CI/CD environment:

   * **Key**:  `ASTRO_API_TOKEN`
   * **Value**: `<your-api-token>`

## Teams and SCIM provisioning

To preserve a single source of truth for user group management, some Team management actions are limited when you [set up SCIM provisioning](/docs/astro/set-up-scim-provisioning). Specifically, when you set up SCIM provisioning:

* You can't create new Teams.
* You can't add users to existing Teams.

For any Teams that were created before you set up SCIM provisioning, you can still complete the following actions:

* Update the Team's permissions.
* Remove users from the Team.
* Delete the Team.


# Manage users in your Astro Workspace
Source: https://astronomer.io/docs/astro/manage-workspace-users

Add, edit, or remove users within a Workspace on Astro.

As a Workspace Owner or an Organization Owner, you can add new team members to Astro and grant them user roles with permissions for specific actions across your Workspace.

To manage users at the Organization level, see [Manage Organization users](/docs/astro/manage-organization-users). To manage groups of users, see [Manage Teams](/docs/astro/manage-teams).

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Prerequisites

To add, edit, or remove Workspace users from a given Workspace, you need either Organization Owner permissions or Workspace Owner permissions for the Workspace.

<Info>
  **Centralized access management for Organization Owners**

  Organization Owners can also add or update a user or Team for a Workspace from your Organization's access management settings:

  <Tabs>
    <Tab title="New Astro UI">
      1. In the Astro UI, go to **Settings**, then in the **Access Management** section, click **Users** or **Teams**.
      2. Select the user or Team.
      3. Click the **Workspaces** tab, then click **+ Add Workspace** to add them to a Workspace, or **Edit Role** to change their role for an existing Workspace.
    </Tab>

    <Tab title="Legacy UI">
      1. In the Astro UI, click **Organization Settings** > **Access Management**.
      2. Select the user or team to add or update.
      3. Click the **Workspaces** tab, then click **+ Workspace** to add them to a Workspace, or open the **More actions** menu (⋯) and select **Edit role** next to an existing Workspace to change their role.
    </Tab>
  </Tabs>
</Info>

## Add a user to a Workspace

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Workspaces**.
    2. Select your Workspace, then in the **Access Management** section, click **Users**.
    3. Click **+ Add User**.
    4. In the **User** field, enter the email address of the user you want to invite.
    5. From the dropdown, select the **Invite new user** option for the email you entered.
    6. Choose a **Workspace Role**. See [Workspace roles reference](/docs/astro/user-permissions#workspace-roles).
    7. Click **Invite User to Workspace**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Workspace Settings** > **Access Management**.
    2. In the **Users** tab, click **+ Member**.
    3. Select the user's name and email address in the **Organization Member** list.
    4. Select a role for the user and then click **Add member**. See [Workspace roles reference](/docs/astro/user-permissions#workspace-roles).
  </Tab>
</Tabs>

After you add the user, their information appears in the **Users** tab as a new entry in the **Members** list.

You can also add groups of users to a Workspace through Teams. See [Manage Teams](/docs/astro/manage-teams).

## Update or remove a Workspace user

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Workspaces**.
    2. Select your Workspace, then in the **Access Management** section, click **Users**.
    3. To edit the user's role, click the user's **More actions** menu (⋯), select **Edit User**, change their access type, then click **Update User**.
    4. To remove the user, click the user's **More actions** menu (⋯), select **Remove User…**, then click **Yes, Continue**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, go to **Workspace Settings** > **Access Management**.
    2. Find the user in the **Users** list. To view or manage the user's roles, either click the user's row to open the **User Details** page, or click the **More actions** button on the user's row and select **Edit User**.
    3. (Optional) Edit the user's name and role. See [Workspace roles](/docs/astro/user-permissions).
    4. If you've updated the user's role, click **Update member**. To delete the user, open the action menu and select **Remove user…**, or click **Remove member** on the **User Details** page.
  </Tab>
</Tabs>

## Add a group of users to a Workspace using the Astro CLI

You can use the Astro CLI and a shell script to add multiple users to a Workspace at once. The shell script reads from a text file which contains user information. You can generate a text file for each new batch of users that need to be assigned to a Workspace and run the script with the Astro CLI.

1. Create a text file named `users.txt`.

2. Open the text file. On each line, add a user's email and their role separated by a space. The following is an example of how you can write a list for inviting users to a Workspace:

   ```text wrap theme={null}
   user1@astronomer.io WORKSPACE_MEMBER
   user2@astronomer.io WORKSPACE_OWNER
   user3@astronomer.io WORKSPACE_OPERATOR
   user4@astronomer.io WORKSPACE_OWNER
   ```

3. Create a file named `add-users.sh` and add the following script to it:

   ```bash title="add-users.sh" wrap theme={null}
   #!/bin/bash

   # Check if a file was provided as an argument
   if [ $# -ne 1 ]; then
       echo "Usage: $0 <file>"
       exit 1
   fi

   while read line; do
       email=$(echo "$line" | cut -d' ' -f1)
       role=$(echo "$line" | cut -d' ' -f2)
       echo "Inviting $email as $role..."
       astro workspace user add "$email" --role "$role"
   done < "$1"
   ```

4. Log in to the Astro CLI using `astro login`, then run `astro workspace list` to ensure that you're in the same Workspace where you want to add the users. If you're not in the right Workspace, run `astro workspace switch`.

5. Run the following command to execute the shell script:

   ```sh wrap theme={null}
   sh path/to/add-users.sh path/to/users.txt
   ```

6. (Optional) To use this script as part of a CI/CD pipeline, create an [Organization API token](/docs/astro/organization-api-tokens) or [Workspace API token](/docs/astro/workspace-api-tokens) and specify the environment variable `ASTRO_API_TOKEN=<your-token>` in your CI/CD environment. Note that you can use Workspace API tokens to manage users only at the Workspace level.

## See also

* [Manage Organization users](/docs/astro/manage-organization-users)
* [Manage Teams](/docs/astro/manage-teams)
* [Manage user permissions on Astro](/docs/astro/user-permissions)


# Configure Workspaces on Astro
Source: https://astronomer.io/docs/astro/manage-workspaces

Create, delete, and update Workspaces on Astro.

Workspaces are collections of Deployments that can be accessed by a specific group of users. You can use Workspaces to group Deployments that share a business use case or trait.

Some common ways to implement Workspaces are:

* Using a **single Workspace** for an entire Organization, with all development and production pipelines hosted together.
* Using a **Workspace per team**. For example, you might have a Workspace for your Data Science team and a separate Workspace for your Data Engineer team, with each team's Workspace hosting both production and development pipelines.
* Using a **Workspace per project**. For example, you might have a Workspace called "Sales Analytics Project" that hosts both production and development pipelines.

This document explains how to configure Workspace details. To manage Workspace users, see [Manage Workspace users](/docs/astro/manage-workspace-users).

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Create a Workspace

<Info>To create a Workspace, you must have an [Organization-level](/docs/astro/user-permissions#organization-roles) role.</Info>

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Workspaces**.
    2. Click **+ New Workspace**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open the Workspace selection menu, then select **Manage Workspaces**.
    2. Click **Add Workspace**.
  </Tab>
</Tabs>

During this initial setup, you can configure the Workspace's name and description. You can update all other Workspace settings later from the Workspace's settings.

## Navigate your Workspace

<Tabs>
  <Tab title="New Astro UI">
    The new Astro UI doesn't have a separate Workspace view. To scope to a specific Workspace, use the Workspace dropdown in the **Deployments**, **DAGs**, and **Environment** sections. To manage a Workspace's settings, go to **Settings** > **Workspaces**, then select the Workspace.
  </Tab>

  <Tab title="Legacy UI">
    To enter your Workspace, click the Workspace in the **Overview** tab. When you click into a Workspace and see a list of that Workspace's Deployments, you are in the **Workspace view** of the Astro UI. The Workspace view contains several pages for managing your Workspace which are accessible from a sidebar on the left of the screen:

    * **Home**: View the status of your Deployments and select recently accessed Deployments.
    * **Deployments:** Create new Deployments and see key metrics about existing Deployments in the Workspace. For more information, see [Create a Deployment](/docs/astro/create-deployment).
    * **Dags:** View metrics about individual dags across your Workspace. For more information, see [Deployment metrics](/docs/astro/deployment-metrics#dag-and-task-runs).
    * **Workspace Settings:** Update Workspace details, including Workspace user permissions, the Workspace name, and the Workspace description.
  </Tab>
</Tabs>

## Update general Workspace settings

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Workspaces**, then select your Workspace.
    2. In the **General** section, click **Edit**, update the **Name** and **Description** as needed, then click **Save**.
    3. To set the CI/CD enforcement default, toggle **CI/CD Enforcement** in the **Deployment Defaults** section. This determines whether new Deployments in the Workspace enforce CI/CD deploys by default, and can be overridden at the Deployment level. See [Enforce CI/CD deploys](/docs/astro/deployment-details#enforce-ci/cd-deploys).
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace.

    2. Click **Workspace Settings** and then click the **Details** tab.

    3. Click **Edit Details**, then update the following settings as needed:

       * **Name**: The name of your Workspace
       * **Description**: The description of your Workspace
       * **CI/CD Enforcement Default**: Determines whether new Deployments in the Workspace enforce CI/CD deploys by default. This default can be overridden at the Deployment level. See [Enforce CI/CD deploys](/docs/astro/deployment-details#enforce-ci/cd-deploys).
  </Tab>
</Tabs>

## Configure Otto investigation guidance

Otto investigation guidance is custom text you provide to tailor [Otto investigations](/docs/astro/otto-investigate) to your environment. For example, you can instruct Otto to treat tasks that begin with `validate` as non-blocking data quality tests, or to interpret a specific log pattern in a particular way. Guidance set at the Workspace level applies to every Deployment in the Workspace. Individual Deployments can override it. See [Configure Otto investigation guidance for a Deployment](/docs/astro/deployment-settings#configure-otto-investigation-guidance).

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Workspaces**, then select your Workspace.
    2. In the **AI Agents** section, click the **Edit** icon next to **Otto Investigation Guidance**.
    3. Enter up to 10,000 characters of markdown guidance, then save your changes.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace.

    2. Click **Workspace Settings** and then click the **AI Agents** tab.

    3. Enter up to 10,000 characters of markdown guidance, then save your changes.
  </Tab>
</Tabs>

## Delete a Workspace

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Workspaces**, then select your Workspace.

    2. Scroll to the **Danger Zone** section, then click **Delete Workspace**. This option isn't available when there are active Deployments in the Workspace.

    3. In the confirmation dialog, enter `delete` and then click **Yes, Continue**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace.

    2. Click **Workspace Settings**.

    3. Click the option menu at the top of the page and select **Delete Workspace**. This option isn't available when there are active Deployments in the Workspace.

       <Frame>
         <img alt="Delete Workspace button" />
       </Frame>

    4. In the confirmation dialog, enter `delete` and then click **Yes, Continue**.
  </Tab>
</Tabs>


# Create network connections between Astro and external resources
Source: https://astronomer.io/docs/astro/networking-overview

Learn about the fundamentals of creating network connections from Astro.

To maximize the power of Airflow, your dags need to be able access data and services that exist outside of Astro. A *network connection* is the basic requirement for accessing external resources from Astro. After you create a network connection, you can configure an Airflow connection to access specific resources that are available through the connection.

Network connections can be either public or private, and each type of connection has a different implementation for security and authorization.

In a public connection, data travels over the public internet to publicly accessible IP addresses on either side of the connection. For example, consider a Deployment that accesses an S3 bucket using an AWS connection with a configured AWS access key and secret. Because the only limitation for accessing the S3 bucket is API authentication, this is an example of a public connection.

In a private connection, data travels over a private network through private IP addresses. Private connections have significantly more security requirements and are recommended whenever you're accessing sensitive or private data.

<Info>Astronomer can support alternative networking solutions that are not covered in documentation. If you have specific networking requirements that aren't covered in documentation, or you need help to create a custom network connection, reach out to [Astronomer support](https://cloud.astronomer.io/open-support-request).</Info>

## Network connection recommendations

If you're just starting out on Astro and you're working with publicly available services and testing data, you only need a public connection. For example, if you're accessing a publicly available API, you only need to configure an [HTTP Airflow connection](https://airflow.apache.org/docs/apache-airflow-providers-http/stable/connections/http.html) to establish a connection between your Deployment and the API.

To access or write data on your company's cloud, Astronomer strongly recommends establishing a private network connection between Astro and your cloud. For most use cases, Astronomer recommends creating a VPC peering connection between Astro and your cloud. After the connection is established, you can authorize individual Deployments to specific resources using workload identity. This method is simple to set up and ensures private and secure connectivity between Astro and any support cloud provider.

To create a VPC peering connection to Astro, you must use a dedicated cluster. In general, dedicated clusters support more secure networking types, such as AWS PrivateLink and Azure VNet peering. See:

* [AWS: Create a private connection between Astro and AWS](/docs/astro/connect-aws#private-networking-connections)
* [GCP: Create a private connection between Astro and GCP](/docs/astro/connect-gcp#create-a-private-connection-between-astro-and-gcp)
* [Azure: Create a private connection between Astro and Azure](/docs/astro/connect-azure#create-a-private-connection-between-astro-and-azure)

After you create your VPC peering connection, follow the steps in [Authorize your Deployment to cloud resources](/docs/astro/authorize-deployments-to-your-cloud) for each Deployment that needs access to your cloud.

<Info>Astronomer monitors the health of Deployments and dags, but it doesn't monitor the status of network connections because they exist outside of Astronomer's observable control plane and data plane.</Info>


# Add custom metadata to OpenLineage assets
Source: https://astronomer.io/docs/astro/observe-custom-metadata

Use the astroCustomMetadata OpenLineage facet to attach links to assets displayed in Astro Observe.

Astro Observe supports a custom OpenLineage facet, `astroCustomMetadata`, that you can use to attach links to assets sourced from OpenLineage, such as datasets and tasks. The links appear on the asset in the **Details** tab and in the **Asset Overview** in [Lineage](/docs/astro/lineage-graph). Use them to surface contextual resources, such as documentation, dashboards, or runbooks, alongside your assets.

Astro Observe supports up to five links per asset.

## Prerequisites

* A Deployment that emits OpenLineage events to Astro Observe. See [Configure OpenLineage on Astro](/docs/astro/observe-openlineage).
* A Dag that emits OpenLineage events for an Observe asset (supported for now: `Airflow Task`, `OpenLineage Dataset`).

## Facet structure

Attach the facet using the `facets` keyword argument. The facet body contains an `observeAssetMetadata` array, where each item describes a single link.

| Field                               | Description                                                                        |
| ----------------------------------- | ---------------------------------------------------------------------------------- |
| `observeAssetMetadata`              | Array of metadata items to display on the asset.                                   |
| `observeAssetMetadata[].value`      | The URL to link to.                                                                |
| `observeAssetMetadata[].type`       | The metadata type. Must be `url`. Astro Observe ignores items with any other type. |
| `observeAssetMetadata[].typeParams` | Optional. For `url`, set `displayText` to the text shown for the link.             |

## Example

The following custom facet attaches one link to an OpenLineage dataset:

```python wrap theme={null}
from airflow.providers.common.compat.openlineage.facet import Dataset as OLDataset


def observe_facet():
    return {
        "astroCustomMetadata": {
            "observeAssetMetadata": [
                {
                    "value": "https://www.astronomer.io/docs/astro/astro-observe",
                    "type": "url",
                    "typeParams": {"displayText": "Go to Astro Observe documentation"},
                },
            ],
        }
    }


# See https://openlineage.io/docs/spec/naming#dataset-naming
ol_dataset = OLDataset(
    name="<dataset-name>",
    namespace="<dataset-namespace>",
    facets=observe_facet(),
)
task = BashOperator(
    task_id="task",
    bash_command="exit 0;",
    outlets=[ol_dataset],
)
```

Use this dataset as an `outlets` value on any task that writes to it. The OpenLineage event for that task includes the dataset with this facet, and Astro Observe attaches the link to the corresponding dataset asset.

To attach the facet to a task run instead of a dataset, emit it as a custom run facet. See [Custom facets](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/configurations-ref.html#custom-facets) in the Apache Airflow documentation.


# Data quality
Source: https://astronomer.io/docs/astro/observe-data-quality

Monitor tables and columns for volume, schema changes, and null percentages. View an overall data quality dashboard and investigate triggered issues.

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

Astro Observe data quality helps you monitor tables to ensure data accuracy, completeness, and integrity across your pipelines. It automatically tracks key metrics such as column null percentages, schema changes, and table row counts to detect anomalies or unexpected shifts in your data.

## Configure permissions

Before connecting to Astro Observe, configure the necessary permissions for your data platform.

<Tabs>
  <Tab title="Snowflake">
    For Snowflake connections, the Observe role must have access to both the `ACCOUNT_USAGE` and `INFORMATION_SCHEMA` system tables. The service user must have a default warehouse configured to support discovery and ongoing data quality monitoring.

    <Steps>
      <Step title="Log into Snowflake">
        Log into Snowflake using a high-privilege role such as `ACCOUNTADMIN`.
      </Step>

      <Step title="Create a dedicated role for Observe">
        ```sql title="Create Observe role" wrap theme={null}
        CREATE ROLE IF NOT EXISTS ASTRO_OBSERVE_ROLE;
        ```
      </Step>

      <Step title="Create a read-only service user">
        Create a service user that Observe will use.

        ```sql title="Create Observe service user" wrap theme={null}
        CREATE USER IF NOT EXISTS ASTRO_OBSERVE_USER
          DEFAULT_ROLE = ASTRO_OBSERVE_ROLE
          TYPE = SERVICE;
        ```
      </Step>

      <Step title="Assign the role to the user">
        ```sql title="Grant role to user" wrap theme={null}
        GRANT ROLE ASTRO_OBSERVE_ROLE TO USER ASTRO_OBSERVE_USER;
        ```
      </Step>

      <Step title="Grant the role the privileges Observe requires.">
        Replace `YOUR_DB` and other example names to match your Snowflake environment.

        ```sql title="Example privileges for ASTRO_OBSERVE_ROLE" wrap theme={null}
        -- Grant warehouse access (replace COMPUTE_WH with your warehouse)
        GRANT USAGE ON WAREHOUSE "COMPUTE_WH" TO ROLE ASTRO_OBSERVE_ROLE;
        ALTER USER ASTRO_OBSERVE_USER SET DEFAULT_WAREHOUSE = 'COMPUTE_WH';

        -- Metadata and object access:
        GRANT USAGE, MONITOR ON DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;
        GRANT USAGE, MONITOR ON ALL SCHEMAS IN DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;

        -- Read access:
        GRANT SELECT ON ALL TABLES IN DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;
        GRANT SELECT ON ALL VIEWS IN DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;
        GRANT SELECT ON ALL EXTERNAL TABLES IN DATABASE YOUR_DB TO ROLE ASTRO_OBSERVE_ROLE;

        -- Optional: access to Snowflake usage views:
        GRANT SELECT ON SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY TO ROLE ASTRO_OBSERVE_ROLE;
        GRANT SELECT ON SNOWFLAKE.ACCOUNT_USAGE.TABLE_DML_HISTORY TO ROLE ASTRO_OBSERVE_ROLE;

        -- Or use:
        -- GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE ASTRO_OBSERVE_ROLE;
        ```

        <Note>These are example grants. Replace database, schema, and warehouse names with values appropriate for your account and security policies.</Note>
      </Step>
    </Steps>

    ## Setup key-pair authentication in Snowflake (recommended)

    Astronomer recommends key-pair authentication for Snowflake service users. Generate an RSA key pair, then assign the public key to the Observe service user to enable secure authentication.

    <Steps>
      <Step title="Generate a password-protected private key and a public key">
        Run the following commands on a secure host to create an encrypted private key and a public key:

        ```sh title="Generate RSA key pair (example)" wrap theme={null}
        # Create encrypted private key
        openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 des3 -inform PEM -out rsa_key.p8

        # Create public key
        openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
        ```
      </Step>

      <Step title="Validate the key">
        ```sh wrap theme={null}
        openssl pkey -in rsa_key.p8 -check -noout
        ```
      </Step>

      <Step title="Assign the public key to the Snowflake user">
        Remove the `-----BEGIN PUBLIC KEY-----` and `-----END PUBLIC KEY-----` headers and newlines so the key is a compact single string. Assign the cleaned public key to the service user:

        ```sql title="Assign public key to Snowflake user" wrap theme={null}
        ALTER USER ASTRO_OBSERVE_USER SET RSA_PUBLIC_KEY = '<your_cleaned_public_key>';
        ```
      </Step>
    </Steps>

    All Snowflake integrations require that the Observe role has access to both `ACCOUNT_USAGE` and `INFORMATION_SCHEMA` system tables. The service user must have a default warehouse configured for all discovery and monitoring operations.
  </Tab>

  <Tab title="Databricks">
    Observe's Databricks Connector requires [Unity Catalog enabled](https://docs.databricks.com/aws/en/admin/system-tables#enable-system-tables) in your Databricks environment.

    <Steps>
      <Step title="Get your Metastore ID">
        Get your Metastore ID using the Databricks CLI.

        ```sh title="Get Metastore ID" wrap theme={null}
        databricks metastores current --profile <DATABRICKS_CONFIG_PROFILE>
        ```
      </Step>

      <Step title="List available schemas (optional)">
        List available system schemas to see what's available.

        ```sh title="List available schemas" wrap theme={null}
        databricks system-schemas list <METASTORE_ID>
        ```
      </Step>

      <Step title="Enable the required system schemas">
        Unity Catalog system tables provide metadata and logging information. These must be explicitly enabled before Observe can read from them.

        ```sh title="Enable system schemas" wrap theme={null}
        databricks system-schemas enable <METASTORE_ID> query
        ```
      </Step>

      <Step title="Grant permissions to the service principal">
        Grant the necessary read permissions to the Observe Service Principal. Replace `<astro_observe_service_principal>` with the actual name of your service principal.

        ```sql title="Grant permissions to service principal" wrap theme={null}
        GRANT USE SCHEMA ON system;

        GRANT SELECT ON system.access.table_lineage TO <astro_observe_service_principal>;
        GRANT SELECT ON system.access.column_lineage TO <astro_observe_service_principal>;
        GRANT SELECT ON system.query.history TO <astro_observe_service_principal>;
        GRANT SELECT ON system.lakeflow.jobs TO <astro_observe_service_principal>;
        GRANT SELECT ON system.lakeflow.job_tasks TO <astro_observe_service_principal>;
        GRANT SELECT ON system.lakeflow.job_run_timeline TO <astro_observe_service_principal>;
        GRANT SELECT ON system.lakeflow.job_task_run_timeline TO <astro_observe_service_principal>;
        ```

        <Note>
          * These steps must be repeated per workspace that uses Unity Catalog
          * Make sure the Databricks CLI is configured (via profile or access token)
          * The above SQL commands must be run by a user with Unity Catalog admin privileges
        </Note>
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Set up a connection

After you configure permissions for your data platform, create an Observe connection.

<Steps>
  <Step title="Open Connections">
    In the Astro UI main navigation, go to **Dashboards** > **Data Quality** and click **+ New Connection** (in the legacy UI, navigate to **Data Quality** > **Connections** and click **+ Connection**).
  </Step>

  <Step title="Fill in connection details">
    <Tabs>
      <Tab title="Snowflake">
        Complete the following fields:

        * **Name:** A name for the connection.
        * **Description:** Optional description.
        * **Connection Type:** Select **Snowflake**.
        * **Polling Schedule:** How frequently Observe polls Snowflake for metrics (examples: every 1 hour, 6 hours, 1 day). Polling frequency is the maximum rate at which Observe updates data quality metrics and monitors; more frequent polling may increase Snowflake compute costs.
        * **Account Identifier:** Your Snowflake account identifier (for example, `FY02423-GP2141`). Observe maps assets to a connection by account identifier.
        * **Username:** The Snowflake service user (`ASTRO_OBSERVE_USER`).
        * **Private Key:** Paste your private key for key-pair authentication if using key-pair auth.

        <Note>
          Only one Observe connection is allowed per Snowflake account identifier. If you have multiple Snowflake accounts, create a separate connection for each account identifier.
        </Note>
      </Tab>

      <Tab title="Databricks">
        Complete the following fields:

        * **Name:** A name for the connection.
        * **Description:** Optional description to help identify the purpose of the connection.
        * **Connection Type:** Select **Databricks**.
        * **Polling Schedule:** How frequently Observe polls Databricks for metrics (examples: every 1 hour, 6 hours, 1 day). Polling frequency is the maximum rate at which Observe updates data quality metrics and monitors; more frequent polling may increase Databricks compute costs.
        * **Host:** Enter the server hostname for your Databricks warehouse.
        * **HTTP Path:** Enter the HTTP Path for your Databricks warehouse.
        * **Password:** This can be either a personal access token or an OAuth secret from a service principal. Astronomer recommends [creating a service principal and using it to generate an OAuth secret](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m#prerequisite-create-a-service-principal).

        <Note>
          Your Databricks role must have Account Admin or Workspace Admin permissions in order to generate a Service Principal and secret.
        </Note>

        <Info>
          To find **Host** and **HTTP Path** for your Databricks warehouse, navigate to **SQL Warehouses** in Databricks, select the warehouse you want to connect, and click **Connection details**. Here you will find Host and HTTP Path.
        </Info>
      </Tab>
    </Tabs>
  </Step>

  <Step title="Save and start discovery">
    Click **Create**. Observe begins the metadata extraction process and will discover your data assets and surface discovered tables in the Asset Catalog.
  </Step>
</Steps>

## Navigating data quality in Astro Observe

### Asset Catalog

Navigate to **Catalog** (**Asset Catalog** in the legacy UI), filter by your data platform (for example, **Snowflake tables** or **Databricks tables**), and select the desired table.

<Info>
  You can sort tables by *popularity* to quickly identify frequently used tables. Popularity rankings are based on query frequency and the number of unique users accessing each table.
</Info>

### Schema

The **Schema** tab shows table structure details:

* Column names
* Data types
* Completeness status
* Nullability
* Default values

You can enable monitoring for specific columns to actively track completeness.

### Event Timeline

The **Event Timeline** tab shows data quality events for a selected timeframe. Events are color-coded by severity: **Success**, **Neutral**, and **Failure**. Click an event to view details, historical patterns, and affected metrics.

### Data quality

The **data quality** tab provides visualizations for monitored metrics:

* **Table Volume:** track changes in row counts and percent change over time to identify unexpected fluctuations.
* **Completeness:** visualize column null percentages against thresholds to surface completeness problems.

### Monitors

To create and manage data quality monitors, see [Monitors in Astro Observe](/docs/astro/observe-monitors#data-quality-monitors).

### Triggered monitor overview

To see a high-level overview of your organization's data quality, click **Data Quality** in the navigation. Here you can see a summary of triggered data quality monitors from the last week or month, grouped by severity and check type.

Click any triggered monitor to investigate it and see the underlying data that triggered the monitor's conditions.

<Frame>
  <img alt="Data quality issues overview dashboard" />
</Frame>


# Get started with Observe
Source: https://astronomer.io/docs/astro/observe-get-started

High-level technical prerequisites and setup steps for Astro Observe.

Astro Observe delivers pipeline-aware data observability purpose-built for Airflow, giving your team visibility into the health, quality, and performance of your business critical data products across the modern data stack. This guide outlines the fundamental requirements and steps to onboard.

## Prerequisites

* Astro Deployment running **Astro Runtime 9** or later (`apache-airflow>=2.7.0`)
* **OpenLineage Airflow provider** (`apache-airflow-providers-openlineage>=1.12.1`) and **OpenLineage client** (`openlineage-python>=1.38.0`) specified in project dependencies (for example, in `requirements.txt`)
* At least one [asset](/docs/astro/assets-overview) running in Airflow
* **Observe Member, Billing Admin, or higher** role required to create data products (see [user permissions reference](/docs/astro/user-permissions#organization-roles))
* [OpenLineage enabled for Remote Execution Agents](/docs/astro/remote-execution-configure-openlineage) if using **Remote Execution**

<Note>
  Astronomer recommends using the latest possible [OpenLineage provider version](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/changelog.html) and the latest [OpenLineage client](https://openlineage.io/docs/) version. The client can be updated regardless of your Airflow version. See [Configure OpenLineage on Astro](/docs/astro/observe-openlineage#recommended-openlineage-versions) for more information and upgrade instructions.
</Note>

## Set up Astro Observe

<Steps>
  <Step title="Enable and verify OpenLineage versions">
    Confirm your Deployments are running the recommended OpenLineage client and provider versions by checking your Astro project's `requirements.txt`. See [Configure OpenLineage on Astro](/docs/astro/observe-openlineage#recommended-openlineage-versions) for instructions on upgrading to the recommended client and provider versions.
  </Step>

  <Step title="Review assets and lineage">
    * Check that the Asset Catalog includes assets you expect. Observe captures Airflow assets based on run data from the last 90 days.
    * If you are missing assets for dags and tasks that ran in the last 90 days, ensure OpenLineage is enabled in your Deployments. See [how to enable or disable OpenLineage](/docs/astro/observe-openlineage#disable-openlineage-completely).
    * If OpenLineage is enabled, check whether the assets are produced by jobs with supported operators. Supported operators and hooks are listed in the [OpenLineage documentation](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/supported_classes.html).
    * For operators not supported out of the box, or for custom operators, see [options for emitting OpenLineage events](/docs/astro/observe-openlineage).
  </Step>

  <Step title="Create your first data product">
    <Info>
      A **data product** is a composition of assets that, taken together, deliver a result with business relevance.

      For guidance on identifying data products in your organization, see [how to identify data products in your organization](/docs/learn/data-products#how-to-identify-data-products-in-your-organization).
    </Info>

    In the Astro UI, navigate to **Observe > Data Products**. Select **+ Data Product** and follow the prompts to define your product and choose assets.

    See [Create a data product](/docs/astro/create-data-products#create-a-data-product) for more information.
  </Step>

  <Step title="Set a Service Level Agreement (SLA)">
    Add an SLA to your data product to monitor freshness or timeliness of your data product. See [Create an SLA](/docs/astro/observe-slas#create-an-sla).
  </Step>

  <Step title="Configure alerts">
    Set up alerts to be notified of SLA violations or potential failures. To monitor the SLA you just created, click the **SLAs** tab in your data product, and select the SLA you want to monitor. Click **+ Alert** to set up a **Data Product SLA Violation** Alert. See [Create an alert](/docs/astro/create-data-products#alerts) for details on alert configuration settings.
  </Step>

  <Step title="Grant permissions to team members">
    To allow others to create data products, SLAs, and monitors in Observe, grant the **Observe Member** or **Observe Admin** role to any teammates who need to set up and manage these features.
    See [user permissions reference](/docs/astro/user-permissions#organization-roles) for instructions on managing roles.
  </Step>
</Steps>

## Next steps and resources

* For in-depth guides, advanced configuration, and best practices, see the full [Create and use data products](/docs/astro/create-data-products) documentation.
* Explore how to [work with assets](/docs/astro/assets-overview), [monitor data products](/docs/astro/observe-monitors), and set up [notifications](/docs/astro/create-data-products#alerts).
* Review [permissions](/docs/astro/user-permissions#organization-roles) to ensure your team has the right access.


# Monitors in Astro Observe
Source: https://astronomer.io/docs/astro/observe-monitors

Monitor data product and pipeline health in Astro Observe.

<Info>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Info>

Astro Observe uses monitors to continuously assess the health of your data products and pipelines. When a monitor detects a failure, it can automatically alert your team through your [notification channels](/docs/astro/alerts#configure-and-add-alert-notification-channels) such as email or Slack. Monitors help you rapidly identify and resolve operational issues in your data workflows.

## Data product monitors

<Info>
  Only one **Data Product** monitor can be created per data product.
</Info>

The **Proactive Failure Monitor** monitors a specific [data product](/docs/astro/create-data-products) and sends an alert when any upstream or final dag in the specified data product fails. This enables you to quickly respond to failures, address the root cause, and understand the downstream impact.

When a monitored failure occurs, the alert:

* Provides a direct link to a lineage view, showing the failing DAG and its failing task.
* Identifies all downstream data products impacted by the failure.

### Create a data product monitor

<Steps>
  <Step title="Start monitor creation">
    <Tabs>
      <Tab title="For a new data product">
        You will be automatically prompted to create a Proactive Failure Monitor as the fourth step in the [Data Product creation process](/docs/astro/create-data-products#create-a-data-product).
        Configure the monitor by selecting a severity level and notification channel(s) to receive alerts as described in the following steps.
      </Tab>

      <Tab title="For an existing data product">
        In the Astro UI, you can create a data product monitor in two ways:

        * **Data Product header:** On the Data Product page, click **Create Monitor** in the **Monitored** field in the header.

          <Frame>
            <img alt="Create Monitor button in Monitored field" />
          </Frame>

        * **Monitoring tab:** On the Data Product page, go to the **Monitoring** tab, select the **Monitors** toggle, and click **+ Monitor**.

          <Frame>
            <img alt="Create Monitor button in Monitoring tab" />
          </Frame>
      </Tab>
    </Tabs>
  </Step>

  <Step title="Name and describe the monitor">
    * A suggested name is auto-generated, but you can edit it as needed.
    * Optionally, add a description to clarify this monitor’s purpose for your team.
  </Step>

  <Step title="Failure detection behavior">
    * **If:** Any upstream or final DAG in the data product fails, an alert will be triggered automatically. This failure condition is managed by Observe and will update dynamically if your pipeline changes.
    * **Then:** Select alert severity level: **Info**, **Warning**, or **Critical**.
    * **And:** Choose one or more notification channels to receive the alerts. Supported channels include email, Slack, and custom integrations. For more on configuration, see [Alert Notification Channels](/docs/astro/alerts#configure-and-add-alert-notification-channels).
  </Step>

  <Step title="Create monitor">
    Click **Create Monitor** to create and enable the configured monitor.
  </Step>
</Steps>

## Data quality monitors

Data quality monitors help you track the health and integrity of your data assets. You can create monitors for tables and columns to detect anomalies in data volume, completeness, and schema changes. To learn more about data quality and configure a connection, see [Data Quality](/docs/astro/observe-data-quality).

<Frame>
  <img alt="Data quality monitor creation" />
</Frame>

### Create a data quality monitor

Follow these steps to create a monitor for a table or column.

<Tabs>
  <Tab title="From asset catalog">
    <Steps>
      <Step title="Select a table">
        In the **Catalog** (**Asset Catalog** in the legacy UI), find the table you want to monitor. Filter the catalog to the table type you want to see (for example, **Snowflake tables**).

        Tables are sorted by popularity to help you find the most relevant tables to monitor. Popularity rankings are based on query frequency and the number of unique users accessing each table.
      </Step>

      <Step title="Open monitors">
        Select the table you want to monitor and click the **Monitors** tab.
      </Step>

      <Step title="Create a monitor">
        Click **+ Monitor** and select the monitor type: **Column Null Percentage**, **Row Volume Change**, **Schema Change**, or **Custom SQL**. Details about each monitor type are in the section below.
      </Step>

      <Step title="Define monitor settings">
        Set alert thresholds and notification preferences.
      </Step>

      <Step title="Create monitor">
        Click **Create Monitor** to activate the monitor.
      </Step>
    </Steps>
  </Tab>

  <Tab title="From monitors page">
    <Steps>
      <Step title="Go to Monitors page">
        In the side navigation, go to **Alerting** > **Monitors** (in the legacy UI, click **Monitors**).

        <Frame>
          <img alt="Monitors list page" />
        </Frame>
      </Step>

      <Step title="Start monitor creation">
        Click **+ Monitor**.
      </Step>

      <Step title="Select monitor scope">
        Select the monitor scope **Table**.
      </Step>

      <Step title="Select a table">
        Select the table you want to monitor.
      </Step>

      <Step title="Select monitor type">
        Select the monitor type: **Column Null Percentage**, **Row Volume Change**, **Schema Change**, or **Custom SQL**. Details about each monitor type are in the sections below.
      </Step>

      <Step title="Configure monitor settings">
        Set alert thresholds and notification preferences.
      </Step>

      <Step title="Create monitor">
        Click **Create Monitor** to activate the monitor.
      </Step>
    </Steps>
  </Tab>
</Tabs>

### Row Volume Change

The **Row Volume Change** monitor tracks changes in table row counts over time to identify unexpected fluctuations in data volume.

When configuring this monitor:

* Specify thresholds based on percentage changes or absolute row-count changes.
* Monitors execute checks according to the schedule you define. If you set the monitor to run every 6 hours, it evaluates whether row counts exceed the configured thresholds within that interval.

### Column Null Percentage

The **Column Null Percentage** monitor tracks the percentage of null values in a specific column to surface completeness problems.

When configuring this monitor:

* Select the column to monitor and define the null percentage threshold.
* The monitor evaluates the column at the interval you specify. If the null percentage exceeds the threshold, the monitor triggers.

### Table Schema Change

The **Table Schema Change** monitor detects when the structure of a table changes, such as when columns are added, removed, or modified. This helps you identify unexpected schema changes that could impact downstream processes.

### Custom SQL

The **Custom SQL** monitor compares the numeric output of a SQL query to a defined threshold. Using this monitor, you can define data quality checks specific to your business requirements (for example, checking for missing records, duplicates, broken joins, or invalid values).
Custom SQL monitors can be triggered on a set schedule, manually, or programmatically.

When configuring this monitor:

<Steps>
  <Step title="Select connection">
    Select the data warehouse connection (for example, Snowflake) that the query will run against.
  </Step>

  <Step title="Write SQL query">
    Write your SQL query. The SQL can query multiple tables and columns but must return one row and one column with a numeric (scalar) value. Example SQL queries are in the section below.

    <Frame>
      <img alt="Custom SQL monitor configuration" />
    </Frame>
  </Step>

  <Step title="Define alert condition">
    Define the numeric threshold of the returned value that triggers an alert (for example, `missing_orders > 10`).
  </Step>

  <Step title="Test query">
    Test the query before creating the monitor to verify that the SQL works as expected.
  </Step>
</Steps>

Common use cases include:

* Flagging duplicate records
* Checking for foreign key violations
* Validating business rules (for example, order amount cannot be negative)
* Ensuring data freshness (for example, last update within 1 hour)

#### Example SQL queries

```sql wrap theme={null}
-- Missing data
SELECT COUNT(*) FROM orders WHERE order_id IS NULL

-- Duplicates
SELECT COUNT(*) FROM (
  SELECT order_id, COUNT(*) 
  FROM orders 
  GROUP BY order_id 
  HAVING COUNT(*) > 1
)

-- Invalid values
SELECT COUNT(*) FROM orders WHERE total_amount < 0

-- Orphaned rows
SELECT COUNT(*) 
FROM orders o 
LEFT JOIN customers c ON o.customer_id = c.customer_id 
WHERE c.customer_id IS NULL

-- Staleness
SELECT DATEDIFF('hour', MAX(updated_at), CURRENT_TIMESTAMP()) 
FROM orders
```

#### Trigger Custom SQL monitors with events

In addition to a fixed schedule, Custom SQL monitors can be triggered by events. For example, when a specific Airflow dag completes successfully or when data lands in a Snowflake table. This ensures that checks are evaluated immediately after new data lands, allowing downstream consumers to be notified of data quality issues immediately rather than waiting for the next scheduled check.

You can trigger a Custom SQL monitor by calling the following API endpoint:

```text wrap theme={null}
POST https://api.astronomer.io/v1alpha1/organizations/{ORG_ID}/observability/monitors/{MONITOR_ID}/trigger
```

When triggered, Observe:

1. Runs the configured SQL monitor immediately against the connected data warehouse.
2. Evaluates the result against the monitor's condition (for example, `invalid_rows > 0`).
3. Logs the result in the monitor timeline.
4. Sends alerts through configured channels if thresholds are breached.

**Example: Airflow DAG trigger**

You can include this API call directly in your Airflow dag to trigger validation right after a load completes:

```python wrap theme={null}
trigger_sql_monitor = PythonOperator(
   task_id="trigger_sql_monitor",
   python_callable=lambda: requests.post(
        f"https://api.astronomer.io/v1alpha1/organizations/{ORG_ID}/observability/monitors/{MONITOR_ID}/trigger",
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {API_TOKEN}",
        },
        timeout=30,
    )
)
```

Chain this operator after your load task:

```python wrap theme={null}
load_data_to_snowflake >> trigger_sql_monitor
```

to ensure that quality checks run immediately after data load, enabling faster issue detection and downstream notification.

#### Custom SQL requirements and limitations

* **Read-only queries:** Only read queries are allowed. Write queries will return an error.
* **Query format:** Queries must return a single row and a single column. Otherwise, Observe returns an error.
* **Numeric output:** Query output must be a numeric value.
* **Table references:** Tables in queries must use the format `<database>.<schema>.<table>` since the default database and schema might not be set.
* **Joins:** You can perform multiple join queries as long as they meet the above criteria.
* **Rate limits:** To prevent system abuse, the following limitations apply:
  * You cannot test the same query more than 3 times in a minute.
  * You cannot test more than 10 queries in a minute.
  * You cannot manually re-trigger the monitor for the next 5 minutes after a trigger request is initiated.
* **Initial trigger:** Wait at least 1 minute before manually triggering the Custom SQL monitor after the monitor is created.


# Configure OpenLineage on Astro
Source: https://astronomer.io/docs/astro/observe-openlineage

How Astro automatically configures OpenLineage and how to add additional backends.

Astro uses the [OpenLineage Airflow Provider](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) (pre-installed on [Astro Runtime](/docs/runtime/runtime-image-architecture)) to extract lineage metadata from Airflow.
Astro also automatically configures OpenLineage to send metadata from your Deployments to Astro for [Observe](/docs/astro/astro-observe) and  [alerts](/docs/astro/alerts).
This ensures lineage data is captured and delivered without any additional user setup. You can also forward lineage events to additional backends if needed.

For information about data lineage concept, OpenLineage, and how it works with Airflow, see [Integrate OpenLineage and Airflow](/docs/learn/airflow-openlineage).

In the Astro UI, the **Observe** section allows you to view the different **Data Products** and **Assets** (dags, tasks, and datasets) that your Organization has created. When you view a specific Data Product, you can click **Open Graph** to render the lineage metadata generated by your dags as a dynamic graph. For more information on using the lineage graph, see [Leveraging Data Products](/docs/learn/data-products).

<Warning>If you use [Remote Execution](/docs/astro/execution-mode#remote-execution), you must [enable OpenLineage for Remote Execution Agents](/docs/astro/remote-execution-configure-openlineage) to use Astro Alerts or Observe.</Warning>

## Recommended OpenLineage versions

### OpenLineage client

The [`openlineage-python`](https://openlineage.io/docs/client/python) package is responsible for sending lineage metadata from Airflow to Astro.
It can be safely upgraded at any time, independent of Airflow and OpenLineage provider versions, to take advantage of the latest fixes, performance improvements, and features.

Use **OpenLineage client version 1.38 or later** for maximum compatibility and access to the latest features on Astro. Astronomer recommends always using the latest available release.

Add the following to your `requirements.txt` to upgrade:

```text title="requirements.txt" wrap theme={null}
openlineage-python>=1.38.0  # check the latest available version at https://pypi.org/project/openlineage-python/
```

### OpenLineage provider

The [`apache-airflow-providers-openlineage`](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) package serves as the Airflow integration layer for OpenLineage, extracting metadata from tasks and DAGs and passing it to the OpenLineage client.
Keep the provider at the latest available version supported by your Airflow version to ensure accurate and complete lineage capture.

Astronomer recommends always using the latest available release of the OpenLineage provider.

Add the following to your `requirements.txt` to upgrade:

```text title="requirements.txt" wrap theme={null}
apache-airflow-providers-openlineage>=2.8.0  # check the latest version supported by your Airflow at https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/changelog.html
```

## Default Astro configuration

Astro automatically configures a set of environment variables for OpenLineage setup. These variables are grouped by their function.

### OpenLineage transport variables

Astro sets the following transport-related variables for compatibility with all supported OpenLineage client versions:

#### Legacy http transport

These three legacy transport variables are used by older OpenLineage client and Astro supplies them for backward compatibility. If you use [recommended OpenLineage client version](#recommended-openlineage-versions) or newer, these variables are ignored in favor of the composite transport configuration below.

| Variable                                                                                  | Value / Example                            | Purpose                                                       |
| ----------------------------------------------------------------------------------------- | ------------------------------------------ | ------------------------------------------------------------- |
| [`OPENLINEAGE_URL`](https://openlineage.io/docs/client/python#environment-variables)      | `https://o11y.astronomer.io`               | Astro Observe ingestion URL.                                  |
| [`OPENLINEAGE_API_KEY`](https://openlineage.io/docs/client/python#environment-variables)  | `<deployment_api_key>`                     | Authentication key for your Deployment.                       |
| [`OPENLINEAGE_ENDPOINT`](https://openlineage.io/docs/client/python#environment-variables) | `api/v1/lineage?ASTRO_ORGANIZATION_ID=...` | Astro Observe ingestion endpoint with deployment identifiers. |

#### New composite transport

For newer OpenLineage clients, Astro configures a composite transport with explicit sub-transports. This structure allows you to easily append other backends by defining additional sub-transports, described below. Make sure to upgrade to [recommended OpenLineage client version](#recommended-openlineage-versions).

| Variable                                                                                                                              | Value / Example | Description                                                                                         |
| ------------------------------------------------------------------------------------------------------------------------------------- | --------------- | --------------------------------------------------------------------------------------------------- |
| [`OPENLINEAGE__TRANSPORT__TYPE`](https://openlineage.io/docs/client/python#composite)                                                 | `composite`     | Specifies use of the composite transport, enabling multiple sub-transports.                         |
| [`OPENLINEAGE__TRANSPORT__SORT_TRANSPORTS`](https://openlineage.io/docs/client/python#composite)                                      | `true`          | Ensures sub-transport execution is sorted by priority.                                              |
| [`OPENLINEAGE__TRANSPORT__TRANSPORTS__DEFAULT_HTTP`](https://openlineage.io/docs/client/python#dynamic-alias-for-transport-variables) | `{}`            | Internally used to avoid duplicate event sending. Safe to ignore warnings on older client versions. |
| [`OPENLINEAGE__TRANSPORT__TRANSPORTS__ASTRO__*`](https://openlineage.io/docs/client/python#composite)                                 |                 | Set of variables configuring the transport for Astro Observe delivery.                              |

Astro transport internally is also of composite type and consists of:

* **Primary HTTP transport:** sends events to public Astro Observe ingestion URL.
* **Backup HTTP transport:** sends events to local ingestion URL within the cluster, invoked only if the primary transport fails.
  * For OpenLineage client version 1.38 or higher, the backup transport is only called if the primary transport fails. For older client versions, the backup transport may be called unnecessarily even when the primary transport succeeds.

<Info>
  **Forwarding OpenLineage transport configuration to external jobs**

  If you use automated forwarding of OpenLineage configuration to external jobs, such as Spark or dbt, from your Airflow DAGs, be aware that Astro’s backup transport uses an internal URL accessible only within the Airflow cluster. When forwarding OpenLineage configuration, to jobs outside of Astro or your Airflow deployment, ensure you only use globally accessible endpoints such as `https://o11y.astronomer.io/`. The backup transport may not be reachable from outside the cluster, and should be omitted from the forwarded transport configuration.
</Info>

### OpenLineage non-transport variables

| Variable                                                                                                                                                           | Description                                                                                                                                                                                                                                   |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `OPENLINEAGE_NAMESPACE`                                                                                                                                            | Unique job namespace for your Deployment.                                                                                                                                                                                                     |
| [`OPENLINEAGE__FACETS__ENVIRONMENT_VARIABLES`](https://openlineage.io/docs/client/python#environment-variables-run-facet)                                          | Includes `ASTRO_ORGANIZATION_ID`, `ASTRO_WORKSPACE_ID`, `ASTRO_DEPLOYMENT_ID`. If you specify any environment variables names here, Astro also appends `ASTRO_ORGANIZATION_ID`, `ASTRO_WORKSPACE_ID`, and `ASTRO_DEPLOYMENT_ID` to your list. |
| [`AIRFLOW__OPENLINEAGE__EXECUTION_TIMEOUT`](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/configurations-ref.html#execution-timeout) | Maximum seconds to wait for OpenLineage listener to perform lineage metadata extraction.                                                                                                                                                      |
| [`AIRFLOW__CORE__TASK_SUCCESS_OVERTIME`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#task-success-overtime)                      | Ensures Airflow allows OpenLineage enough time to finish metadata extraction.                                                                                                                                                                 |

<Note>
  If you define any of these [environment variables](/docs/astro/manage-env-vars) manually, Astro preserves your values and does not overwrite them.
</Note>

<Warning>
  Do not change or overwrite the values of `OPENLINEAGE_NAMESPACE` or `AIRFLOW__OPENLINEAGE__NAMESPACE`. These variables are reserved for Astro's internal use to ensure Deployment-level alerting and event traceability. Overwriting them can cause alerting and lineage to break.
</Warning>

## Sending lineage to additional backends

To send OpenLineage events to an external OpenLineage backend alongside Astro Observe, define an environment variable with the additional transport configuration.
Example:

```text wrap theme={null}
OPENLINEAGE__TRANSPORT__TRANSPORTS__MY_TRANSPORT='{"type":"http","url":"https://example.com/api/v1/lineage","auth":{"type":"api_key","apiKey":"<KEY>"}}'
```

<Warning>
  **Additional transport name**

  When defining additional transports make sure not to use reserved `ASTRO` name of the transport as it may overwrite astro default transport causing potential disruption to lineage, alerting and other Observe functionalities. Use of any variable starting with `OPENLINEAGE__TRANSPORT__TRANSPORTS__ASTRO` is not recommended.
</Warning>

* Astro’s default transport remains active.
* Transport's `priority` attribute determines the order in which transports are executed. Astro’s transport priority is set to 1.
* Do not set or overwrite any other transport related variables as it may cause some transports not to send events properly.
* Do not overwrite `OPENLINEAGE_NAMESPACE` or `AIRFLOW__OPENLINEAGE__NAMESPACE`.

### Example: Send data to Atlan

<Info>
  **Custom namespace for Atlan**

  Astro does not allow overwriting `OPENLINEAGE_NAMESPACE` or `AIRFLOW__OPENLINEAGE__NAMESPACE`.

  If Atlan requires a different namespace, use a [TransformTransport](/docs/astro/observe-transformtransport) to set a custom namespace only for Atlan events.
</Info>

1. Update the placeholder values and add the following [environment variable](/docs/astro/manage-env-vars) to your Deployment:

   ```text wrap theme={null}
   OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN='{"type":"http","url":"https://<instance>.atlan.com/events/openlineage/airflow-astronomer/", "auth":{"type":"api_key","apiKey":"<key>"}, "compression":"gzip"}'
   ```

2. Click Update Environment Variables to save your changes.

3. Verify that Atlan receives lineage events alongside Astro Observe.

## Opting out of Astro OpenLineage delivery

<Warning>
  Astronomer relies on OpenLineage events to provide Astro Observe and Deployment-level alerting. Any changes that prevent OpenLineage events from being delivered to Astronomer may result in loss of lineage visibility, alerting malfunction and degraded Astro Observe functionality.
</Warning>

### Stop sending data to Astronomer but send to your own backend

To override Astronomer's default OpenLineage configuration and send events only to your backend, provide a full OpenLineage transport configuration using *one* of the following [environment variables](/docs/astro/manage-env-vars). These take precedence over Astro defaults:

* `AIRFLOW__OPENLINEAGE__TRANSPORT`
* `AIRFLOW__OPENLINEAGE__CONFIG_PATH` (YAML config path)

Your configuration must include at least a `type` key to be valid. Once configured, the Deployment does not send OpenLineage events to Astronomer.

### Disable OpenLineage completely

By default, OpenLineage is enabled for all Astro Deployments. To disable all OpenLineage emission from your Deployment, set the following [environment variable](/docs/astro/manage-env-vars):

```text wrap theme={null}
AIRFLOW__OPENLINEAGE__DISABLED=true
```

<Warning>
  This disables all lineage capture for the deployment. Astro Observe and alerting will not function.
</Warning>

If you want to re-enable OpenLineage, simply remove the above variable or set its value to `false`.

## Expected log messages and warnings

You may encounter the following messages in your Airflow or OpenLineage logs. These log outputs are expected and safe to ignore in the context of Astro's managed OpenLineage configuration:

* **"DEFAULT\_HTTP already found in environment variables, skipping aliasing OPENLINEAGE\_URL"**
  This means Astro is pre-setting modern composite transport variables and suppressing legacy duplicate event sending. Upgrade to the latest OpenLineage client to suppress this message.

* **DeprecationWarning for `'api_key' option is deprecated, please use 'apiKey'`.**
  Astro populates both versions of the API key to maximize compatibility. Upgrade to the latest OpenLineage client to suppress this message.

* **"Stopping OpenLineage CompositeTransport emission after the first successful delivery because continue\_on\_success=False. Transport that emitted the event: `<HttpTransport(name=astro_primary OR astro_backup …)>`"**
  Indicates that the Astro composite transport has stopped as it successfully delivered the event to Astro Observe backend. Any user-configured transports still run normally. Upgrade to the latest OpenLineage client to suppress this message.

## Known limitations

* Astro accepts OpenLineage events up to a maximum size of 5 MB. Events exceeding this limit will result in an `HTTP 413 Content Too Large` error.


# Service Level Agreements (SLAs)
Source: https://astronomer.io/docs/astro/observe-slas

Create and manage SLAs in Astro Observe.

Service Level Agreements (SLAs) define expectations for data product delivery and freshness. Create SLAs to monitor whether your data products meet business requirements and configure alerts to notify your team when SLAs are at risk or violated. SLAs check for a successful run or update of the final assets in a data product.

<Note>
  Data products with tables as their final assets do not support SLAs.
</Note>

## Create an SLA

Astro allows you to create **Freshness**, **Timeliness** and **Custom** SLAs on your data products. Based on the timeliness or freshness expectations defined in your SLAs, Astro uses SLA success rates to send proactive alerts and generate insights into your data pipelines.

1. In the **Data Products** page, click the specific data product for which you want to create an SLA.
2. Click the **SLA Evaluations** tab and click **+ Add SLA**.
3. Add a **Name** for your SLA and optionally add a **Description**.
4. Choose a **Template**:
   * **Timeliness**: Data Product updates by a specific time. Define the window of time by choosing **Days of the week**, **Verification time**, and **Lookback period**.
   <Info>
     Timeliness SLAs only support Standard Time. If you want Local Time support, you must adjust the SLA's UTC time when the time changes from Standard Time to Daylight Savings Time or from Daylight Savings Time to Standard Time. All evaluation times are in UTC, consistent with Airflow.
   </Info>
   * **Freshness**: Data Product updates on a certain frequency. Configure this SLA by defining a **Freshness Policy** by the number of minutes, hours, or days.
   * **Custom**: Configure your own SLA parameters with full control over evaluation schedules and freshness windows. For example, you can set an SLA to check whether a retail reporting pipeline ran on the first 10 days of every fiscal quarter with the following cron expression: `0 0 1-10 1,4,7,10 *`
5. Configure the **Evaluation Schedule**, which defines how often the SLA checks for a successful run of the final assets :

   * For simple schedules, use the predefined options
   * For complex schedules, select **Cron** from the dropdown and enter a cron expression. The UI will display a human-readable translation of your cron expression and show the next scheduled evaluation time.

   <Note>
     **Evaluation schedule guidelines**

     * The minimum interval is 15 minutes to prevent excessive evaluation frequency.

     * The maximum freshness window is 31 days.

     * Evaluation schedules longer than 1 day evaluate on the nth day of the month and begin on the first of the month. For example, an evaluation schedule of 9 days set on December 31 will evaluate on Jan 1st, Jan 10th, Jan 19th, Jan 28th, Feb 1st, Feb 10th, Feb 19th, Feb 28th, March 1st, etc.
   </Note>
6. Set the **Freshness Window**: Define how far back to look for successful runs before each evaluation time.
7. Click **Create SLA**.

After you create an SLA, you can configure alerts and proactive alerts.

## Create an alert

After you create an SLA, Astro keeps a record of the rate at which your data product hits or misses the SLA. You must configure an **Alert** or a **Proactive Alert** to receive notifications when your pipeline experiences an SLA miss or when an upstream process might cause an SLA miss or a failure.

1. In the **Data Products** page, click the specific data product you want to create an SLA for.
2. Click the **Alerts** tab and click **+ Add Alert**.
3. Choose the **Type** of alert and **Severity**. The following alert types are available:
   * Data Product SLA Violation: Send an alert when a data product asset has violated its SLA definition.
   * Data Product Proactive SLA: Astro monitors the upstream dependencies of the data product assets, and proactively sends an alert if delays in the upstream dependencies might eventually cause SLA misses.
   * Data Product Proactive Failure: Send an alert when a dependent asset upstream of your data product has failed.
4. Define the conditions that the alert applies to. These conditions vary depending on the type of alert you want to set up.
5. Select or add a **Notification Channel** where you want to send your alert. For more information about configuring notification channels, see [Alert Notification Channels](/docs/astro/alerts#configure-and-add-alert-notification-channels).
6. (Optional) Customize the alert name.
7. Click **Create alerts**.


# Use TransformTransport to customize OpenLineage events
Source: https://astronomer.io/docs/astro/observe-transformtransport

Customize OpenLineage events per backend using TransformTransport with Astro Observe.

[TransformTransport](https://openlineage.io/docs/client/python/#transform) allows you to modify OpenLineage events just before they are sent, on a per-transport basis. This lets you send customized or transformed events to specific backends without changing the global OpenLineage configuration.

Use OpenLineage client version **1.38** or higher. Upgrading the OpenLineage client version is safe and independent of your Airflow version. Astronomer recommends always using the latest version of OpenLineage client.

Add the following to your `requirements.txt` to upgrade:

```text title="requirements.txt" wrap theme={null}
openlineage-python>=1.38.0
```

You can define your own transformer class to modify events as needed, or use one of the predefined transformers available for common use cases like renaming job namespaces.

If you use TransformTransport, it is important to note:

* TransformTransport only modifies the event right before sending.
* TransformTransport does not change the global OpenLineage configuration or variables such as the namespace. It means that the OpenLineage namespace remains unchanged globally.
* If you use global OpenLineage namespace elsewhere, such as passing it to Spark operators or OpenLineage macros, those references will still have the original namespace value. In such cases, you might need to manually update those references to ensure consistency or implement a similar approach for Spark integration.

This guide presents an example configuration that demonstrates how to use TransformTransport to customize OpenLineage events and send them to both Astro Observe and Atlan with different job namespaces. This approach can be adapted for other use cases or backends that require event transformation before delivery. See [OpenLineage documentation](https://openlineage.io/docs/client/python/#transform) for more examples.

## Send OpenLineage events to Observe and Atlan with different namespaces

An example use case of TransformTransport for OpenLineage is you want to [send OpenLineage events to both Astro Observe and Atlan](/docs/astro/observe-openlineage), but need the OpenLineage namespace to be different only for Atlan.

This can be done using the predefined `JobNamespaceReplaceTransformer` transformer. This transformer lets you dynamically override the job namespace for specific backends using the TransformTransport interface.

Astro already configures OpenLineage transport for the default Astro Observe destination, so you don't need to configure it again.

To add a secondary Atlan backend with a different job namespace, define the following environment variables:

```text wrap theme={null}
# Transform applied only to Atlan transport
OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TYPE=transform
OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TRANSFORMER_CLASS=openlineage.client.transport.transform.JobNamespaceReplaceTransformer
OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TRANSFORMER_PROPERTIES={"new_job_namespace": "<atlan-namespace-value>"}

# HTTP transport configuration for Atlan, to be used after transformation
OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TRANSPORT__TYPE=http
OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TRANSPORT__URL=https://<instance>.atlan.com/events/openlineage/airflow-astronomer/
OPENLINEAGE__TRANSPORT__TRANSPORTS__ATLAN__TRANSPORT__AUTH={"type":"api_key", "apiKey":"<API_token>"}
```

### Verify the setup

1. Trigger a DAG that emits OpenLineage events.
2. Confirm events appear in Astro Observe, the default backend.
3. Check your Atlan instance to confirm receipt of events with the updated namespace.
4. Check Airflow logs for any errors related to the OpenLineage transport layer.

If you do not overwrite the default value of `OPENLINEAGE__TRANSPORT__CONTINUE_ON_FAILURE=true`, a failed Atlan transport will not affect Astro delivery.


# Export Astro reporting data
Source: https://astronomer.io/docs/astro/org-dash-exports

Export data from Organization dashboards or configure conditional exports as alerts.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

As an Astro administrator, you can export reporting data to share with other team members or to keep a record of key performance indicators. Astro supports several different methods for exporting reporting data based on how and when you want to receive the data. Specifically, you can export data:

* Manually through the Astro UI.
* On a regular schedule.
* Whenever a certain condition is met, such as a metric reaching a specific threshold. This is known as a *dashboard alert*.

When you export reporting data, Astro exports the last [one million rows of data](https://help.sigmacomputing.com/docs/download-export-and-upload-limitations) from a specific dashboard in the file format of your choice. Use this document to learn about the different ways you can trigger the export process.

<Tip>Watch the Astro Academy [Reporting](https://academy.astronomer.io/learning-bytes-reporting) Learning Byte video to learn more about how observe your teams' Airflow Deployments.</Tip>

## Prerequisites

* Only users with **Organization Billing Admin** [user permissions](/docs/astro/user-permissions#organization-roles) can access Organization dashboards.

## Download reporting data

1. To view Organization dashboards, click **Dashboards** in the Astro UI menu. You can also access this page directly at [https://cloud.astronomer.io/dashboards](https://cloud.astronomer.io/dashboards).

2. Click the **Underlying Data** tab. This tab contains the raw data that powers each dashboard chart in a standardized financial format, including the following tables:

   * **Deployment Activity**: Deployment task, Dag, operator, and code push metrics aggregated by day, week, or month time periods.
   * **Workspace Activity**: Workspace task, Dag, operator, and code push metrics aggregated by day, week, or month time periods.
   * **Daily Cost Breakdown**: Costs broken down by day. Includes a second table in the FinOps FOCUS format.

   Each table includes filters so you can narrow the data before exporting.

   <Frame>
     <img alt="The Underlying Data tab shows raw data tables for Deployment Activity, Workspace Activity, and Daily Cost Breakdown." />
   </Frame>

   <Note>For the **Daily Cost Breakdown** table, export a larger range of dates and upsert the data, because costs can arrive late.</Note>

3. Click the three-dot menu on the table that you want to export, then click **Export**. Under **DOWNLOAD**, choose the file format that you want to download your data in. Astro generates your data export and saves it to your local computer.

   <Frame>
     <img alt="Click the three-dot menu to access options for downloading data, emailing data, scheduling a data export, and setting up a data alert." />
   </Frame>

## Export reporting data

Instead of downloading your data directly, you can email it or send it to an external service through a webhook.

1. To view Organization dashboards, click **Dashboards** in the Astro UI menu. You can also access this page directly at [https://cloud.astronomer.io/dashboards](https://cloud.astronomer.io/dashboards).

2. Click the **Underlying Data** tab, then click the three-dot menu for the table you want to export. Click **Export**, then under **SEND**, click **Export...**.

   <Frame>
     <img alt="Click the three-dot menu to access options for downloading data, emailing data, scheduling a data export, and setting up a data alert." />
   </Frame>

3. Choose where you want to send the data report.
   * **Email** - Enter the recipient's email address and optionally edit the subject line and email body message.
   * **Webhook** - ([Sigma feature in beta](https://help.sigmacomputing.com/docs/webhook-exports)) Add the webhook URL where you want to export your data to.

4. In the **Attachments** section, select the report you want to send and file format to include it as. You can choose from the same file types that are available for a one-time data download.

5. (Optional) To send more than one report, click **+ Add**, and then select the report you want to send and its file format.

When you receive a report via email, the sender appears as **Sigma Computing**. The subject line also includes the name of the dashboard element's data you exported.

<Frame>
  <img alt="Example email alert shows the sender as Sigma with information about the report in the subject line." />
</Frame>

### Schedule a data report

Astronomer recommends scheduling exports from the **Underlying Data** tab instead of from individual charts. The standardized format provides consistent data that replaces previous chart-specific export customizations.

1. To view Organization dashboards, click **Dashboards** in the Astro UI menu. You can also access this page directly at [https://cloud.astronomer.io/dashboards](https://cloud.astronomer.io/dashboards).

2. Click the **Underlying Data** tab, then click the three-dot menu for the table you want to schedule an export for. Click **Export**, then under **SEND**, click **Schedule exports...**.

   <Frame>
     <img alt="Click the three-dot menu to access options for downloading data, emailing data, scheduling a data export, and setting up a data alert." />
   </Frame>

3. Enter the where you want to send the data report.
   * **Email** - Enter the recipient's email address and optionally edit the subject line and email body message.
   * **Webhook** - ([Sigma feature in beta](https://help.sigmacomputing.com/docs/webhook-exports)) Add the webhook URL where you want to export your data to.

If you toggle **Condition** when setting up your schedule, it allows you to configure a conditional data export, or *dashboard alert*. Instead of sending a report at a specific time interval, it sends a data report when your data meets criteria that you define.

## Create a dashboard alert

A dashboard alert contains a message and a data export that Astro sends when specific criteria are met in one of your reporting metrics. Use dashboard alerts to quickly receive messages and data when your metrics reach a specific threshold, such as when task failures exceed a certain amount.

1. To view Organization dashboards, click **Dashboards** in the Astro UI menu. You can also access this page directly at [https://cloud.astronomer.io/dashboards](https://cloud.astronomer.io/dashboards).

2. Click the three-dot menu for the reporting element you want to set up an alert for. Click **Alert when...**.

   <Frame>
     <img alt="Click the three-dot menu to access options for downloading data, emailing data, scheduling a data export, and setting up a data alert." />
   </Frame>

3. Configure the export destination and attachments, then in the **Frequency** section, select **if a condition is met**. In the **Condition** section that appears, define when you want the dashboard to send the data report.

   <Frame>
     <img alt="Configure a scheduled export with a condition that triggers the dashboard to send you a data report for a particular dashboard element." />
   </Frame>

When you receive a dashboard alert in an email, the sender appears as **Sigma Computing**. The subject line also includes the name of the dashboard element's data you exported.


# View Organization dashboards
Source: https://astronomer.io/docs/astro/organization-dashboard

View information about your Organization, Deployments, dags, and costs.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

Astro provides dashboards that share important metrics about your Organization's use of Astro, which you can use to manage Deployments and resources. These dashboards include:

* Organization Overview
* Deployment Detail
* Cost Breakdown
* Operator Use
* Contract Details

Organization dashboards help you quickly identify opportunities to optimize how your team uses Airflow at different organizational levels, such as across your Organization, within Workspaces, in specific Deployments, and operators within dags. For example, you can use the **Organization Overview** or **Deployment Detail** dashboard to identify unexpected activity, without requiring you to examine the dags results in the Airflow UI. Instead, you can quickly check your dashboards to identify trends like unusually high or low rates of task successes, and then check the **Cost Breakdown** dashboard to identify any associated costs incurred by that behavior.

<Tip>Watch the Astro Academy [Reporting](https://academy.astronomer.io/learning-bytes-reporting) Learning Byte video to learn more about how observe your teams' Airflow Deployments.</Tip>

To view Organization dashboards, click **Dashboards** in the Astro UI menu. You can also access this page directly at [https://cloud.astronomer.io/dashboards](https://cloud.astronomer.io/dashboards). In addition to viewing the dashboards in the Astro UI, you can also [export and email individual reports](/docs/astro/org-dash-exports).

Use the tabs at the bottom of the Astro UI to switch between dashboards. Each dashboard shows the last time that it was updated in the **Data available through** section.

<Info>Only users with **Organization Billing Admin** [user permissions](/docs/astro/user-permissions#organization-roles) can access Organization dashboards.</Info>

## Organization overview

The **Organization Overview** dashboard provides at-a-glance summaries about activity across your Organization's Deployments. You can also filter to view summaries for specific Workspaces or Deployments.

This dashboard allows you to compare the activity and performances of Deployments to one another. You can identify Deployments, dags, or tasks that have had recent behavior changes or are performing in an unexpected way. For example, you can filter the data shown by time period, Workspace name, or Deployment name, to view data such as the number of successful or failed tasks within Workspaces or Deployments. Hovering your cursor over any of the charts brings up a detailed view of the data, indexed by date.

By examining your data at the Organization level, you can identify Deployments with large numbers of failing tasks by looking at the graphs or charts for outliers. Then, you can filter by time period to see if there have been similar events in the past and when.

<Frame>
  <img alt="The main section of the Organization Overview dashboard, showing Deployments and task counts over time as a bar chart" />
</Frame>

## Deployment Detail

The **Deployment Detail** dashboard shows a table of all Deployments in your Organization, indexed by Workspace name. For each Deployment, the dashboard shows your Deployment configurations as well as use metrics like task run counts, dag run counts, and operator counts.

Use this dashboard to check that your Deployment has the appropriate resources based on the number of dags it runs. You can also use this dashboard to check whether a Deployment's Astro Runtime version is currently supported.

<Frame>
  <img alt="View data about Deployment health and dag success." />
</Frame>

## Cost Breakdown

The **Cost Breakdown** dashboard displays your Astro spending over time across the following cost categories, which you can filter by time period, Workspace name, or Deployment name:

* Deployment
* Cluster
* Compute
* Network
* Remote Execution
* AI Tokens
* Observe
* Triggerer

Use this data alongside your other dashboards to identify the biggest opportunities for cost reduction in your Organization.

<Frame>
  <img alt="View the cost of your different Astro resources." />
</Frame>

### Detailed Costs

The **Detailed Costs** view provides granular breakdowns for each Workspace, Deployment, Deployment worker queue, or each dedicated cluster by billable item (the same as seen on invoices) within the selected time period. Use the **Deployments**, **Compute Types**, **Worker Queues**, and **Clusters** tabs to switch between breakdowns.

Each breakdown displays a **Cost Type** table that groups your spending into billable categories such as **Compute**, **Deployment**, and **AI Tokens**, alongside the total amount for the selected time period. Select a cost type to see a description or relevant notes about what that cost type includes. For example, the **Deployment** cost type covers costs related to Scheduler configuration and ephemeral storage.

<Frame>
  <img alt="View detailed cost breakdowns by Deployment, Compute Type, Worker Queue, or Cluster." />
</Frame>

<Frame>
  <img alt="Hover over the info icon to see descriptions of each cost type." />
</Frame>

## Operator Use

The **Operator Use** dashboard shows how your Deployments and Workspaces use Operators, as well as how often tasks succeed and fail when using specific operators. Use this data to identify types of operators that could be replaced with more efficient alternatives, or to find operators that fail more than expected. This dashboard provides data to answer the questions, *What are the top operators used across my organization?* and *Which workspace is using the selected operators the most?*.

## Contract Details

The **Contract Details** dashboard shows your Organization's credit or dollar usage against your current contract. Use this dashboard to monitor your remaining balance, view your projected burn rate through the end of your contract term or until your next grant, and review individual credit and deduction history.

<Frame>
  <img alt="View your remaining balance and projected burn rate." />
</Frame>


# Astro Hosted resource reference
Source: https://astronomer.io/docs/astro/resource-reference-hosted

Reference of all supported infrastructure for new Astro Hosted clusters.

<Warning>
  This document applies only to [Astro Hosted](/docs/astro/astro-architecture) and does not apply to Astro Hybrid.

  To learn more about Astro solutions, see [Remote Execution](/docs/astro/execution-mode) and [Create a Deployment](/docs/astro/create-deployment).
</Warning>

This page contains reference information for all supported Astro Hosted Deployment and cluster resource configurations. Use this information to determine whether Astro supports the type of Airflow environment you want to run.

If you're interested in a cloud region or resource size that's not mentioned here, reach out to [Astronomer support](https://cloud.astronomer.io/open-support-request).

## Astro worker types

Astro Deployments use the worker types in the following table.

If you use the [Astro executor](/docs/astro/astro-executor) or Celery executor, you can configure a [worker queue](/docs/astro/configure-worker-queues) to run tasks on a specific worker type. The Astro executor is the default for all Airflow 3.x Deployments.
If you use the Kubernetes executor, Astro uses the number of A5 workers necessary to accommodate the total amount of CPU and Memory needed for the tasks.

With the Kubernetes executor, you can configure ephemeral storage [per task](/docs/astro/kubernetes-executor) otherwise the
[Deployment default setting](/docs/astro/deployment-resources#configure-kubernetes-pod-resources) is applied. With the Astro executor or Celery executor,
you can configure ephemeral storage and concurrency for tasks per worker queue. See [Configure the Astro executor](/docs/astro/astro-executor#configure-astro-worker-scaling-in-hosted-execution-mode) or [Configure the Celery executor](/docs/astro/celery-executor#configure-celery-worker-scaling).

| Worker Type | vCPU | Memory | Ephemeral storage | Default task concurrency | Max task concurrency |
| ----------- | ---- | ------ | ----------------- | ------------------------ | -------------------- |
| A5          | 1    | 2GiB   | 10 GiB            | 5                        | 15                   |
| A10         | 2    | 4GiB   | 10 GiB            | 10                       | 30                   |
| A20         | 4    | 8GiB   | 10 GiB            | 20                       | 60                   |
| A40         | 8    | 16GiB  | 10 GiB            | 40                       | 120                  |
| A60         | 12   | 24GiB  | 10 GiB            | 60                       | 180                  |
| A120        | 24   | 48GiB  | 10 GiB            | 120                      | 360                  |
| A160        | 32   | 64GiB  | 10 GiB            | 160                      | 480                  |

## Standard cluster regions

A *standard cluster* is a multi-tenant cluster that's hosted and managed by Astronomer. Astronomer maintains standard clusters in a limited regions and clouds, with support for more regions and clouds coming soon.

Currently, standard clusters are available on the following clouds and regions:

<Tabs>
  <Tab title="AWS">
    | Code             | Region                   |
    | ---------------- | ------------------------ |
    | `ap-southeast-1` | Asia Pacific (Singapore) |
    | `eu-central-1`   | Europe (Frankfurt)       |
    | `eu-west-1`      | Europe (Ireland)         |
    | `us-east-1`      | US East (N. Virginia)    |
    | `us-west-2`      | US West (Oregon)         |
  </Tab>

  <Tab title="GCP">
    | Code           | Region                  |
    | -------------- | ----------------------- |
    | `europe-west4` | Netherlands, Europe     |
    | `us-central1`  | Iowa, North America     |
    | `us-east4`     | Virginia, North America |
  </Tab>

  <Tab title="Azure">
    | Code            | Region                     |
    | --------------- | -------------------------- |
    | `australiaeast` | New South Wales, Australia |
    | `eastus2`       | Virginia, North America    |
    | `westus2`       | Washington, North America  |
    | `westeurope`    | Netherlands, Europe        |
  </Tab>
</Tabs>

## Dedicated cluster regions

A *dedicated cluster* is cluster that Astronomer provisions solely for use by your Organization. You can create new dedicated clusters from the Astro UI in a variety of clouds and regions. To configure dedicated clusters, see [Create a dedicated cluster](/docs/astro/create-dedicated-cluster).

Currently, dedicated clusters are available on the following clouds and regions:

<Tabs>
  <Tab title="AWS">
    | Code             | Name                      |
    | ---------------- | ------------------------- |
    | `ap-northeast-1` | Asia Pacific (Tokyo)      |
    | `ap-southeast-1` | Asia Pacific (Singapore)  |
    | `ap-southeast-2` | Asia Pacific (Sydney)     |
    | `ap-south-1`     | Asia Pacific (Mumbai)     |
    | `ca-central-1`   | Canada (Central)          |
    | `eu-central-1`   | Europe (Frankfurt)        |
    | `eu-west-1`      | Europe (Ireland)          |
    | `eu-west-2`      | Europe (London)           |
    | `sa-east-1`      | South America (São Paulo) |
    | `us-east-1`      | US East (N. Virginia)     |
    | `us-east-2`      | US East (Ohio)            |
    | `us-west-1`      | US West (N. California)   |
    | `us-west-2`      | US West (Oregon)          |
  </Tab>

  <Tab title="Azure">
    | Code                 | Region               |
    | -------------------- | -------------------- |
    | `australiaeast`      | Australia East       |
    | `brazilsouth`        | Brazil South         |
    | `canadacentral`      | Canada Central       |
    | `centralindia`       | Central India        |
    | `centralus`          | Central US           |
    | `eastus`             | East US              |
    | `eastus2`            | East US 2            |
    | `francecentral`      | France Central       |
    | `germanywestcentral` | Germany West Central |
    | `japaneast`          | Japan East           |
    | `northeurope`        | North Europe         |
    | `uaenorth`           | United Arab Emirates |
    | `uksouth`            | UK South             |
    | `southafricanorth`   | South Africa North   |
    | `southcentralus`     | South Central US     |
    | `westeurope`         | West Europe          |
    | `westus2`            | West US 2            |
    | `westus3`            | West US 3            |
  </Tab>

  <Tab title="GCP">
    | Code                      | Name                          |
    | ------------------------- | ----------------------------- |
    | `asia-east1`              | Taiwan, Asia                  |
    | `asia-northeast1`         | Tokyo, Asia                   |
    | `asia-northeast2`         | Osaka, Asia                   |
    | `asia-northeast3`         | Seoul, Asia                   |
    | `asia-south1`             | Mumbai, Asia                  |
    | `asia-south2`             | Delhi, Asia                   |
    | `asia-southeast1`         | Singapore, Asia               |
    | `asia-southeast2`         | Jakarta, Asia                 |
    | `australia-southeast1`    | Sydney, Australia             |
    | `australia-southeast2`    | Melbourne, Australia          |
    | `europe-north1`           | Finland, Europe               |
    | `europe-southwest1`       | Madrid, Europe                |
    | `europe-west1`            | Belgium, Europe               |
    | `europe-west2`            | England, Europe               |
    | `europe-west3`            | Frankfurt, Europe             |
    | `europe-west4`            | Netherlands, Europe           |
    | `europe-west6`            | Zurich, Switzerland           |
    | `europe-west8`            | Milan, Europe                 |
    | `europe-west9`            | Paris, Europe                 |
    | `northamerica-northeast1` | Montreal, North America       |
    | `northamerica-northeast2` | Toronto, North America        |
    | `southamerica-east1`      | São Paulo, South America      |
    | `southamerica-west1`      | Santiago, South America       |
    | `us-central1`             | Iowa, North America           |
    | `us-east1`                | South Carolina, North America |
    | `us-east4`                | Virginia, North America       |
    | `us-east5`                | Columbus, North America       |
    | `us-west1`                | Oregon, North America         |
    | `us-west2`                | Los Angeles, North America    |
    | `us-west4`                | Nevada, North America         |
  </Tab>
</Tabs>


# Diagnose Dag failures in Astro Observe
Source: https://astronomer.io/docs/astro/root-cause-analysis

Diagnose Dag failures in Astro Observe with Otto investigations, AI log summaries, and upstream root cause analysis.

Astro Observe gives you several ways to diagnose Dag and task failures: Otto's investigation agent, AI log summaries, and upstream root cause analysis.

## Investigate failures with Otto

When a Dag fails, you can run a structured investigation that combines Airflow context (Dag code, task logs), Astro context (Deployment configuration, component logs, recent deploys), and Observe context (lineage, run history, operational metrics). Each investigation returns a root cause type, severity, suggested fix, and a checklist of Dag- and task-level checks.

To run an investigation from Astro Observe:

* Click **Investigate** next to a Dag on the Observe homepage to investigate its most recent failed run.
* Open a Dag in the **Catalog** (the **Asset Catalog** in the legacy UI) and select a failed run from its run history.
* Open a specific run in a Dag's run history to investigate that run.

For the full feature, including all access points, the recommended automation pattern with Astro alerts, and Otto investigation guidance, see [Investigate with Otto](/docs/astro/otto-investigate).

## AI log summaries

For failed tasks and failure events that depend on these tasks, such as Dag failures and data product SLAs, Astro Observe summarizes the task failure logs into a human-readable description of the issue. AI log summaries are visible from the event timeline of a task failure, the event timeline of an SLA breach with a failed upstream task, and the event timeline tab in the lineage view.

AI log summaries are powered by Otto. See [Otto overview](/docs/astro/otto-overview) for Otto's full set of capabilities.

<Frame>
  <img alt="Create Monitor button in Monitored field" />
</Frame>

## Upstream root cause analysis

For failure events, Astro Observe detects the upstream root cause by scanning upstream dependencies and surfacing any anomalies or failures. Failure events also show downstream dependencies to help you understand the potential impact of the issue.

<Note>
  AI features can be enabled or disabled Organization-wide by an Organization Owner. When disabled, AI log summaries and Otto investigations are unavailable for all users. To configure this setting, see [Toggle AI features for your Organization](/docs/astro/enable-disable-astro-ai).
</Note>


# Configure a secrets backend
Source: https://astronomer.io/docs/astro/secrets-backend

Learn to configure a secrets backend on Astro to store Airflow connections and variables

Apache Airflow [variables](/docs/learn/airflow-variables) and [connections](/docs/learn/connections) often contain sensitive information about your external systems that you need to keep in a *secrets backend* tool, which stores secrets in a secure and centralized location. Unlike other management strategies, such as using Environment Variables or working with connections and variables in the Airflow UI, secrets backends require a third-party secrets manager. This means that you can use a secrets manager administered by your organization for existing security protocols, or you need to choose and set up a secrets backend.

This document explains the available secrets backend integrations supported by Astro and how Airflow finds connections and variables if you use multiple strategies to manage them.

See [Manage connections and variables](/docs/astro/manage-connections-variables) to learn more about your available options and decide whether using a secrets backend complies with your organization's security requirements.

## Available integrations

Secrets backend integrations can be configured individually with each Astro Deployment by someone with [**Workspace Operator**](/docs/astro/user-permissions#workspace-roles) permissions.

Using secrets to set Airflow connections requires knowledge of how to generate Airflow connections in URI or JSON format. See [Import and export Airflow connections and variables](/docs/astro/import-export-connections-variables) for guidance on how to export your connections and variables based on where they are stored.

Astro integrates with the following secrets backend tools:

* [AWS Secrets Manager](/docs/astro/secrets-backend/aws-secretsmanager)
* [AWS Systems Manager Parameter Store](/docs/astro/secrets-backend/aws-paramstore)
* [Azure Key Vault](/docs/astro/secrets-backend/azure-key-vault)
* [Google Cloud Secret Manager](/docs/astro/secrets-backend/gcp-secretsmanager)
* [Hashicorp Vault](/docs/astro/secrets-backend/hashicorp-vault)

### Remote execution integration

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

You can also set up a secrets backend integration with your Remote Execution Agent in your Execution plane. Each supported integration includes the Remote Execution Agent-specific implementation steps.

## How Airflow finds connections or variables

<Tip>If you need to access your secrets backend from your local Airflow, you can mount your user credentials to a local Airflow environment. While this implementation is not recommended for Astro Deployments, it lets you quickly test pipelines with data hosted in your cloud. See [Authenticate to cloud services](/docs/cli/v1.43/authenticate-to-clouds).</Tip>

If you configure a secrets backend on Astro, you can still continue to define Airflow variables and connections as [environment variables](/docs/astro/environment-variables), with the [Astro Environment Manager](/docs/astro/create-and-link-variables) or in the Airflow UI. The order of precedence for connections is:

1. Secrets Backend
2. Astro Environment Manager
3. Environment Variables
4. Airflow's metadata database (Airflow UI)


# Set up AWS Systems Manager (SSM) Parameter Store
Source: https://astronomer.io/docs/astro/secrets-backend/aws-paramstore

Configure AWS Systems Manager Parameter Store as a secrets backend for Airflow variables and connections on Astro.

In this section, you'll learn how to use [AWS Systems Manager (SSM) Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) as a secrets backend on Astro.

If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see [Configure a Secrets Backend](/docs/astro/secrets-backend).

## Prerequisites

* A [Deployment](/docs/astro/create-deployment).
* The [Astro CLI](/docs/cli/v1.43/overview).
* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) with version 5.1.0+ of `apache-airflow-providers-amazon`. See [Add Python and OS-level packages](/docs/cli/v1.43/add-providers-packages).
* An IAM role with access to the [Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-paramstore-access.html) that your Astro cluster can assume.
* (Remote Execution Only) [Helm installed](https://helm.sh/docs/intro/install/)
* (Remote Execution Only) The `values.yaml` file from the **Register Agents** modal in your **Deployments**>**Agents** page.

## Step 1: Create Airflow secrets directories in Parameter Store

Create directories for Airflow variables and connections in Parameter Store that you want to store as secrets.

Variables and connections should be stored in `/airflow/variables` and `/airflow/connections`, respectively. For example, if you're setting a secret variable with the key `my_secret`, it should be stored in the `/airflow/connections/` directory. If you modify the directory paths, make sure you change the values for `variables_prefix` and `connections_prefix` in Step 2.

For instructions, see the [AWS Systems Manager Console](https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-create-console.html), the [AWS CLI](https://docs.aws.amazon.com/systems-manager/latest/userguide/param-create-cli.html), or the [Tools for Windows PowerShell](https://docs.aws.amazon.com/systems-manager/latest/userguide/param-create-ps.html) documentation.

## Step 2: Set up Parameter Store locally

<Tabs>
  <Tab title="Astro">
    Add the following environment variables to your Astro project's `.env` file:

    ```text wrap theme={null}
    AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
    AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables",  "role_arn": "<your-role-arn>", "region_name": "<your-region>"}
    ```

    You can now run a dag locally to check that your variables are accessible using `Variable.get("<your-variable-key>")`.
  </Tab>

  <Tab title="Remote Execution">
    In your Astro project, add the [AWS Systems Manager (SSM) Parameter Store](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-ssm-parameter-store.html) to your project by adding the following to your `values.yaml` file to set the secrets backend class to use the provider and configure your secrets backend kwargs:

    ```yaml title="values.yaml" wrap theme={null}
    secretBackend: "airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend"

    commonEnv:
      - name: AIRFLOW__SECRETS__BACKEND_KWARGS
        value: '{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables",  "role_arn": "<your-role-arn>", "region_name": "<your-region>"}'
    ```

    You need to run the Remote Execution Agent with AWS credentials to fetch from your secrets manager.

    <Tip>
      For secure production environments, you can store sensitive Kwargs containing secret ID and app role ID in a secret:

      ```yaml title="values.yaml" wrap theme={null}
      commonEnv:
        - name: AIRFLOW__SECRETS__BACKEND_KWARGS
          valueFrom:
            secretKeyRef:
              name: airflow-secret-backend
              key: '{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables",  "role_arn": "<your-role-arn>", "region_name": "<your-region>"}'
      ```
    </Tip>
  </Tab>
</Tabs>

## Step 3: Deploy configuration

<Tabs>
  <Tab title="Astro">
    1. Run the following commands to export your secrets backend configurations as environment variables to Astro.

       ```sh wrap theme={null}
       astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend

       astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables",  "role_arn": "<your-role-arn>", "region_name": "<your-region>"}' --secret
       ```

    2. (Optional) Remove the environment variables from your `.env` file or store your `.env` file in a safe location to protect your credentials in `AIRFLOW__SECRETS__BACKEND_KWARGS`.
  </Tab>

  <Tab title="Remote Execution">
    1. Run the following command to update your Remote Execution Agent with your new configurations.

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Tab>
</Tabs>


# Set up AWS Secrets Manager as your secrets backend
Source: https://astronomer.io/docs/astro/secrets-backend/aws-secretsmanager

Configure AWS Secrets Manager as a secrets backend for Airflow variables and connections on Astro.

This topic provides setup steps for configuring [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) as a secrets backend on Astro.

For more information about Airflow and AWS connections, see [Amazon Web Services Connection](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html).

If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see [Configure a Secrets Backend](/docs/astro/secrets-backend).

## Prerequisites

* A [Deployment](/docs/astro/create-deployment).
* The [Astro CLI](/docs/cli/v1.43/overview).
* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project) with `apache-airflow-providers-amazon` version 5.1.0 or later. See [Add Python and OS-level packages](/docs/cli/v1.43/add-providers-packages).
* An IAM role with the `SecretsManagerReadWrite` policy that your Astro cluster can assume. See [AWS IAM roles](/docs/astro/authorize-deployments-to-your-cloud).
* (Remote Execution Only) [Helm installed](https://helm.sh/docs/intro/install/)
* (Remote Execution Only) The `values.yaml` file from the **Register Agents** modal in your **Deployments**>**Agents** page.

## Step 1: Add Airflow secrets to Secrets Manager

Create directories for Airflow variables and connections in AWS Secrets Manager that you want to store as secrets. You can use real or test values.

* When setting the secret type, choose `Other type of secret` and select the `Plaintext` option.
* If creating a connection URI or a non-dict variable as a secret, remove the brackets and quotations that are pre-populated in the plaintext field.
* The secret name is assigned after providing the plaintext value and clicking `Next`.

Secret names must correspond with the `connections_prefix` and `variables_prefix` set below in step 2. Specifically:

* If you use `"variables_prefix": "airflow/variables"`, you must set Airflow variable names as:

  ```text wrap theme={null}
  airflow/variables/<variable-key>
  ```

* The `<variable-key>` is how you will retrieve that variable's value in a dag. For example:

  ```python wrap theme={null}
  my_var = Variable.get("<variable-key>")
  ```

* If you use `"connections_prefix": "airflow/connections"`, you must set Airflow connections as:

  ```text wrap theme={null}
  airflow/connections/<connection-id>
  ```

* The `<connection-id>` is how you will retrieve that connection's URI in a dag. For example:

  ```python wrap theme={null}
  conn = BaseHook.get_connection(conn_id="<connection-id>")
  ```

* Be sure to not include a leading `/` at the beginning of your variable or connection name

For more information on adding secrets to Secrets Manager, see [AWS documentation](https://docs.aws.amazon.com/secretsmanager/latest/userguide/manage_create-basic-secret.html).

## Step 2: Set up Secrets Manager locally

<Tabs>
  <Tab title="Astro">
    Add the following environment variables to your Astro project's `.env` file:

    ```text wrap theme={null}
    AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
    AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "role_arn": "<your-role-arn>"}
    AWS_DEFAULT_REGION=<region>
    ```

    After you configure an Airflow connection to AWS, can run a dag locally to check that your variables are accessible using `Variable.get("<your-variable-key>")`.
  </Tab>

  <Tab title="Remote Execution">
    In your Astro project, add the [AWS Secrets Manager Backend](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-secrets-manager.html) to your project by adding the following to your `values.yaml` file to set the secrets backend class to use the AWS provider and configure your secrets backend kwargs:

    ```yaml title="values.yaml" wrap theme={null}
    secretBackend: "airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend"

    commonEnv:
      - name: AIRFLOW__SECRETS__BACKEND_KWARGS
        value: '{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "role_arn": "<your-role-arn>"}'
      - name: AWS_DEFAULT_REGION
        value: '<region>'
    ```

    You need to run the Remote Execution Agent with AWS credentials to fetch from your secrets manager.
  </Tab>
</Tabs>

## Step 3: Deploy environment variables to Astro

<Tabs>
  <Tab title="Astro">
    1. Run the following commands to export your secrets backend configurations as environment variables to Astro.

       ```sh wrap theme={null}
       astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

       astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables",  "role_arn": "<your-role-arn>", "region_name": "<your-region>"}' --secret
       ```

    2. (Optional) Remove the environment variables from your `.env` file or store your `.env` file in a safe location to protect your credentials.

    <Info> If you delete the `.env` file, the Secrets Manager backend won't work locally. </Info>

    3. Open the Airflow UI for your Deployment and create an [Amazon Web Services connection](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html) without credentials. When you use this connection in a dag, Airflow will automatically fall back to using the credentials in your configured environment variables.
  </Tab>

  <Tab title="Remote Execution">
    1. Run the following command to update your Remote Execution Agent with your new configurations.

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Tab>
</Tabs>

To further customize the Airflow and AWS SSM Parameter Store integration, see the [full list of available kwargs](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/secrets/systems_manager/index.html).


# Set up Azure Key Vault as your secrets backend
Source: https://astronomer.io/docs/astro/secrets-backend/azure-key-vault

Configure Azure Key Vault as a secrets backend for Airflow variables and connections on Astro.

This topic provides setup steps for configuring [Azure Key Vault](https://azure.microsoft.com/en-gb/services/key-vault/#getting-started) as a secrets backend on Astro.

If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see [Configure a Secrets Backend](/docs/astro/secrets-backend).

## Prerequisites

* A [Deployment](/docs/astro/create-deployment).
* The [Astro CLI](/docs/cli/v1.43/overview).
* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project).
* An existing Azure Key Vault linked to a resource group.
* Your Key Vault URL. To find this, go to your Key Vault overview page > **Vault URI**.
* (Remote Execution Only) [Helm installed](https://helm.sh/docs/intro/install/)
* (Remote Execution Only) The `values.yaml` file from the **Register Agents** modal in your **Deployments**>**Agents** page.

If you don't already have Key Vault configured, read [Microsoft Azure documentation](https://docs.microsoft.com/en-us/azure/key-vault/general/quick-create-portal).

## Step 1: Register Astro as an app on Azure

<Note>
  Steps 1 and 2 are only required if you are using service principal (client secret) authentication. If you prefer to use managed identity authentication, skip to Step 3 and follow the **Managed Identity** tab instructions.
</Note>

Follow the [Microsoft Azure documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-credentials) to register a new application for Astro.

At a minimum, you need to add a [secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-credentials) that Astro can use to authenticate to Key Vault.

Note the value of the application's client ID and secret for Step 3.

## Step 2: Create an access policy

<Note>If you use a managed identity to authenticate to Key Vault, skip to [Step 3](#step-3-set-up-key-vault-locally). Ensure your managed identity has an access policy or Azure RBAC role that grants it access to your Key Vault secrets.</Note>

Follow the [Microsoft documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-credentials) to create a new access policy for the application that you just registered. The settings you need to configure for your policy are:

* **Configure from template**: Select `Key, Secret, & Certificate Management`.
* **Select principal**: Select the name of the application that you registered in Step 1.

## Step 3: Set up Key Vault locally

<Tabs>
  <Tab title="Astro">
    In your Astro project, add the following line to your `requirements.txt` file:

    ```text title="requirements.txt" wrap theme={null}
    apache-airflow-providers-microsoft-azure
    ```

    Add the following environment variables to your `.env` file. Choose the option that matches your authentication method:

    **Client secret authentication:**

    ```text wrap theme={null}
    AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
    AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "tenant_id": "<your-tenant-id>", "client_id": "<your-client-id>", "client_secret": "<your-client-secret>"}
    ```

    For client secret authentication, find your client ID in Azure Portal at **App Registration page** > **Application (Client) ID**. To find your tenant ID, go to **App Registration page** > **Directory (tenant) ID**. To find your client secret, go to **App Registration Page** > **Certificates and Secrets** > **Client Secrets** > **Value**.

    **Managed identity authentication:**

    Before using managed identity authentication, you must configure your Deployment with a workload identity. See the [Azure tab](/docs/astro/authorize-deployments-to-your-cloud?language=azure#setup) in [Authorize a Deployment to cloud resources using workload identity](/docs/astro/authorize-deployments-to-your-cloud#setup) to set up your managed identity and authorize it to your Deployment.

    ```text wrap theme={null}
    AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
    AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "managed_identity_client_id": "<your-managed-identity-client-id>", "workload_identity_tenant_id": "<your-tenant-id>"}
    ```
  </Tab>

  <Tab title="Remote Execution">
    Add the [Azure Key Vault Backend](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/secrets-backends/azure-key-vault.html) to your project by updating your `values.yaml` file. Choose an authentication method:

    <Tabs>
      <Tab title="Service Principal">
        Add the following to your `values.yaml` file to set the secrets backend class to use the Vault provider and configure your secrets backend kwargs:

        ```yaml title="values.yaml" wrap theme={null}
        secretBackend: "airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend"

        commonEnv:
          - name: AIRFLOW__SECRETS__BACKEND_KWARGS
            value: '{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "tenant_id": "<your-tenant-id>", "client_id": "<your-client-id>", "client_secret": "<your-client-secret>"}'
        ```

        You need to run the Remote Execution Agent with your Azure credentials to fetch from your secrets manager.
      </Tab>

      <Tab title="Managed Identity">
        Managed identity authentication is more secure because it eliminates the need to manage client secrets.

        #### Configure a managed identity

        Before updating your Helm values, configure a managed identity with access to your Key Vault:

        1. Create a user-assigned managed identity in Azure, or use an existing one.
        2. Grant the managed identity access to your Key Vault. In your Key Vault, create an access policy with **Get** and **List** permissions for secrets, and select your managed identity as the principal (see [Step 2](#step-2-create-an-access-policy)).
        3. Configure [Azure AD Workload Identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) to allow your Kubernetes service accounts to use the managed identity.

        #### Update Helm values

        Add the following to your `values.yaml` file:

        ```yaml title="values.yaml" wrap theme={null}
        secretBackend: "airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend"

        commonEnv:
          - name: AIRFLOW__SECRETS__BACKEND_KWARGS
            value: '{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>", "managed_identity_client_id": "<your-managed-identity-client-id>", "workload_identity_tenant_id": "<your-tenant-id>"}'

        labels:
          azure.workload.identity/use: "true"

        annotations:
          azure.workload.identity/client-id: "<your-managed-identity-client-id>"
        ```

        Replace:

        * `<your-vault-url>`: Your Azure Key Vault URL
        * `<your-managed-identity-client-id>`: Client ID of your managed identity
        * `<your-tenant-id>`: Your Azure tenant ID
      </Tab>
    </Tabs>
  </Tab>
</Tabs>

For managed identity authentication, find your managed identity client ID in Azure Portal at **Managed Identities** > your identity > **Client ID**. To find your tenant ID, go to **Microsoft Entra ID** > **Overview** > **Tenant ID**.

This configuration tells Airflow to look for variable information at the `airflow/variables/*` path in Azure Key Vault and connection information at the `airflow/connections/*` path. You can now run a dag locally to check that your variables are accessible using `Variable.get("<your-variable-key>")`.

By default, this setup requires that you prefix any secret names in Key Vault with `airflow-connections` or `airflow-variables`. If you don't want to use prefixes in your Key Vault secret names, set the values for `sep`, `"connections_prefix"`, and `"variables_prefix"` to `""` within `AIRFLOW__SECRETS__BACKEND_KWARGS`.

## Step 4: Deploy configuration

<Tabs>
  <Tab title="Astro">
    1. Run the following commands to export your environment variables to Astro.

       ```sh wrap theme={null}
       astro deployment variable create --deployment-id <your-deployment-id> --load --env .env
       ```

       In the Astro UI, mark `AIRFLOW__SECRETS__BACKEND_KWARGS` as **Secret**. See [Set environment variables in the Astro UI](/docs/astro/manage-env-vars#use-the-astro-ui).

    2. Run the following command to push your updated `requirements.txt` file to Astro:

       ```sh wrap theme={null}
       astro deploy --deployment-id <your-deployment-id>
       ```

    3. (Optional) Remove the environment variables from your `.env` file, or store your `.env` file so that your credentials are hidden, for example with GitHub secrets.
  </Tab>

  <Tab title="Remote Execution">
    1. Run the following command to update your Remote Execution Agent with your new configurations.

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Tab>
</Tabs>


# Set up Google Cloud Secret Manager as your secrets backend
Source: https://astronomer.io/docs/astro/secrets-backend/gcp-secretsmanager

Configure Google Cloud Secret Manager as a secrets backend for Airflow variables and connections on Astro.

This topic provides setup steps for configuring [Google Cloud Secret Manager](https://cloud.google.com/secret-manager/docs/configuring-secret-manager) as a secrets backend on Astro.

If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see [Configure a Secrets Backend](/docs/astro/secrets-backend).

## Prerequisites

* A [Deployment](/docs/astro/create-deployment).
* The [Astro CLI](/docs/cli/v1.43/overview).
* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project).
* [Cloud SDK](https://cloud.google.com/sdk/gcloud).
* A Google Cloud environment with [Secret Manager](https://cloud.google.com/secret-manager/docs/configuring-secret-manager) configured.
* A [service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) with the [Secret Manager Secret Accessor](https://cloud.google.com/secret-manager/docs/access-control) role on Google Cloud.
* (Optional) A [JSON service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys) for the service account. This is required to provide access to a secrets backend from a local machine, or when you're not using Workload Identity.
* (Remote Execution Only) [Helm installed](https://helm.sh/docs/intro/install/)
* (Remote Execution Only) The `values.yaml` file from the **Register Agents** modal in your **Deployments**>**Agents** page.

## Step 1: Create an Airflow variable or connection in Google Cloud Secret Manager

To start, create an Airflow variable or connection in Google Cloud Secret Manager that you want to store as a secret. You can use the Cloud Console or the gcloud CLI.

Secrets must be formatted such that:

* Airflow variables are set as `airflow-variables-<variable-key>`.
* Airflow connections are set as `airflow-connections-<connection-id>`.

For example, to add an Airflow variable with a key `my-secret-variable`, you run the following gcloud CLI command:

```sh wrap theme={null}
gcloud secrets create airflow-variables-<my-secret-variable> \
    --replication-policy="automatic"
```

For more information on creating secrets in Google Cloud Secret Manager, read the [Google Cloud documentation](https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#create).

## Step 2: Set up GCP Secret Manager locally

<Tabs>
  <Tab title="Astro">
    1. Copy the complete JSON service account key for the service account that you want to use to access Secret Manager.

    2. Add the following environment variables to your Astro project's `.env` file, replacing `<your-service-account-key>` with the key you copied in Step 1:

       ```text wrap theme={null}
       AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
       AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": "<your-service-account-key>"}
       ```

    3. (Optional) Run `Variable.get("<your-variable-key>")` to run a dag locally and confirm that your variables are accessible.
  </Tab>

  <Tab title="Remote Execution">
    In your Astro project, add the [Google Cloud Secret Manager Backend](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/secrets-backends/google-cloud-secret-manager-backend.html) to your project by adding the following to your `values.yaml` file to set the secrets backend class to use the Vault provider and configure your secrets backend kwargs:

    ```yaml title="values.yaml" wrap theme={null}
    secretBackend: "airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend"

    commonEnv:
      - name: AIRFLOW__SECRETS__BACKEND_KWARGS
        value: '{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": "<your-service-account-key>"}'
    ```

    You need to run the Remote Execution Agent with GCP credentials to fetch from your secrets manager.
  </Tab>
</Tabs>

## Step 3: (Astro Only) Configure Secret Manager on Astro using Workload Identity (Recommended)

1. Set up Workload Identity for your Airflow Deployment. See [Connect Astro to GCP data sources](/docs/astro/authorize-deployments-to-your-cloud#setup).

2. Run the following commands to set the secrets backend for your Astro Deployment:

   ```sh wrap theme={null}
   astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend

   astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "project_id": "<your-secret-manager-project-id>"}'
   ```

3. (Optional) Remove the environment variables from your `.env` file or store your `.env` file in a safe location to protect your credentials in `AIRFLOW__SECRETS__BACKEND_KWARGS`.

To ensure the security of secrets, the `.env` variable is only available in your local environment and not in the Astro UI . See [Set Environment Variables Locally](/docs/cli/v1.43/develop-project#set-environment-variables-locally).

## Step 4: Configure Secret Manager on Astro using a service account JSON key file

<Tabs>
  <Tab title="Astro">
    1. Set up the Secret Manager locally. See [Set up GCP Secret Manager locally](#step-2-set-up-gcp-secret-manager-locally).

    2. Run the following command to set the `SECRET_VAR_SERVICE_ACCOUNT` environment variable on your Astro Deployment:

       ```sh wrap theme={null}
       astro deployment variable create --deployment-id <your-deployment-id> SECRET_VAR_SERVICE_ACCOUNT="<your-service-account-key>" --secret
       ```

    3. (Optional) Remove the environment variables from your `.env` file or store your `.env` file in a safe location to protect your credentials in `AIRFLOW__SECRETS__BACKEND_KWARGS`.
  </Tab>

  <Tab title="Remote Execution">
    1. Run the following command to update your Remote Execution Agent with your new configurations.

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Tab>
</Tabs>


# Set up HashiCorp Vault as your secrets backend
Source: https://astronomer.io/docs/astro/secrets-backend/hashicorp-vault

Configure HashiCorp Vault as a secrets backend for Airflow variables and connections, for both local development and on Astro.

This page provides steps for using [HashiCorp Vault](https://www.vaultproject.io/) as a secrets backend for both local development and on Astro.

To do this, you will:

* Create an AppRole in Vault which grants Astro minimal required permissions.
* Write a test Airflow variable or connection as a secret to your Vault server.
* Configure your Astro project to pull the secret from Vault.
* Test the backend in a local environment.
* Deploy your changes to Astro.

If you use a different secrets backend tool or want to learn the general approach on how to integrate one, see [Configure a Secrets Backend](/docs/astro/secrets-backend).

## Prerequisites

* A [Deployment](/docs/astro/create-deployment) on Astro.
* [The Astro CLI](/docs/cli/v1.43/overview).
* A local or hosted Vault server. See [Starting the Server](https://learn.hashicorp.com/tutorials/vault/getting-started-dev-server?in=vault/getting-started) or [Create a Vault Cluster on HCP](https://developer.hashicorp.com/vault/tutorials/cloud/get-started-vault).
* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project).
* [The Vault CLI](https://www.vaultproject.io/docs/install).
* Your Vault Server's URL. If you're using a local server, this should be `http://127.0.0.1:8200/`.
* (Remote Execution Only) [Helm installed](https://helm.sh/docs/intro/install/)
* (Remote Execution Only) The `values.yaml` file from the **Register Agents** modal in your **Deployments**>**Agents** page.

If you don't already have a Vault server deployed but would like to test this feature, Astronomer recommends that you either:

* Sign up for a Vault trial on [HashiCorp Cloud Platform (HCP)](https://cloud.hashicorp.com/products/vault) or
* Deploy a local Vault server. See [Starting the server](https://learn.hashicorp.com/tutorials/vault/getting-started-dev-server?in=vault/getting-started) in HashiCorp documentation.

## Step 1: Create a policy and AppRole in Vault

To use Vault as a secrets backend, Astronomer recommends configuring a Vault AppRole with a policy that grants only the minimum necessary permissions for Astro. For Remote Execution Deployments, you can use any Vault authentication method you prefer, for example Kubernetes auth if your agents and Vault are running on Kubernetes.

To do this:

1. Run the following command to [create a Vault policy](https://www.vaultproject.io/docs/concepts/policies) that Astro can use to access a Vault server:

   ```sh wrap theme={null}
   vault auth enable approle
   vault policy write astro_policy - <<EOF
   path "secret/*" {
     capabilities = ["create", "read", "update", "patch", "delete", "list"]
   }
   EOF
   ```

2. Run the following command to [create a Vault AppRole](https://www.vaultproject.io/docs/auth/approle):

   ```sh wrap theme={null}
   vault auth enable approle
   vault write auth/approle/role/astro_role \
       role_id=astro_role \
       secret_id_ttl=0 \
       secret_id_num_uses=0 \
       token_num_uses=0 \
       token_ttl=24h \
       token_max_ttl=24h \
       token_policies=astro_policy
   ```

3. Run the following command to retrieve the `secret-id` for your AppRole:

   ```sh wrap theme={null}
   vault write -f auth/approle/role/<your-approle>/secret-id
   ```

   Save this value. You'll use this later to complete the setup.

## Step 2: Create an Airflow variable or connection in Vault

To start, create an Airflow variable or connection in Vault that you want to store as a secret. It can be either a real or test value. You will use this secret to test your backend's functionality.

You can use an existing mount point or create a new one to store your Airflow connections and variables. For example, to create a new mount point called `airflow`, run the following Vault CLI command:

```sh wrap theme={null}
vault secrets enable -path=airflow -version=2 kv
```

To store an Airflow variable in Vault as a secret at the path `variables`, run the following Vault CLI command with your own values:

```sh wrap theme={null}
vault kv put -mount=airflow variables/<your-variable-name> value=<your-value>
```

To store an Airflow connection in Vault as a secret at the path `connections`, first format the connection as a URI. Then, run the following Vault CLI command with your own values:

```sh wrap theme={null}
vault kv put -mount=airflow connections/<your-connection-name> conn_uri=<connection-type>://<connection-login>:<connection-password>@<connection-host>:<connection-port>
```

To format existing connections in URI format, see [Import and export connections](/docs/astro/import-export-connections-variables#using-the-astro-cli-local-environments-only).

<Warning>Don't use custom key names for your secrets. Airflow requires the key name `value` for all Airflow variables and the key name `conn_uri` for all Airflow connections as shown in the previous commands.</Warning>

To confirm that your secret was written to Vault successfully, run:

```sh wrap theme={null}
# For variables
vault kv get -mount=airflow variables/<your-variable-name>

# For connections
vault kv get -mount=airflow connections/<your-connection-name>
```

## Step 3: Set up Vault locally

<Tabs>
  <Tab title="Astro">
    In your Astro project, add the [HashiCorp Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-hashicorp/stable/index.html) to your project by adding the following to your `requirements.txt` file:

    ```text title="requirements.txt" wrap theme={null}
    apache-airflow-providers-hashicorp
    ```

    Then, add the following environment variables to your `.env` file:

    ```text wrap theme={null}
    AIRFLOW__SECRETS__BACKEND=airflow.providers.hashicorp.secrets.vault.VaultBackend
    AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_path": "connections", "variables_path": "variables",  "mount_point": "airflow", "url": "http://host.docker.internal:8200", "auth_type": "approle", "role_id":"astro_role", "secret_id":"<your-approle-secret>"}
    ```
  </Tab>

  <Tab title="Remote Execution">
    In your Astro project, add the [HashiCorp Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-hashicorp/stable/index.html) to your project by adding the following to your `values.yaml` file to set the secrets backend class to use the Vault provider and configure your secrets backend kwargs:

    ```yaml title="values.yaml" wrap theme={null}
    secretBackend: "airflow.providers.hashicorp.secrets.vault.VaultBackend"

    commonEnv:
      - name: AIRFLOW__SECRETS__BACKEND_KWARGS
        value: '{"connections_path": "connections", "variables_path": "variables", "config_path": null, "url": "<vault-url>", "auth_type": "approle", "role_id":"<your-approle-id>", "secret_id":"<your-approle-secret>"}'
    ```

    You need to run the Remote Execution Agent with Vault credentials to fetch from your secrets manager.

    <Tip>
      For more security, you can store sensitive Kwargs containing secret ID and app role ID in a secret:

      ```yaml title="values.yaml" wrap theme={null}
      commonEnv:
        - name: AIRFLOW__SECRETS__BACKEND_KWARGS
          valueFrom:
            secretKeyRef:
              name: airflow-secret-backend
              key: '{"connections_path": "connections", "variables_path": "variables", "config_path": null, "url": "<vault-url>", "auth_type": "approle", "role_id":"<your-approle-id>", "secret_id":"<your-approle-secret>"}'
      ```
    </Tip>
  </Tab>
</Tabs>

<Info>
  If you run Vault on HashiCorp Cloud Platform (HCP):

  * Replace `http://host.docker.internal:8200` with `https://<your-cluster>.hashicorp.cloud:8200`.
  * Add `"namespace": "admin"` as an argument after `url`.
</Info>

This tells Airflow to look for variable and connection information at the `airflow/variables/*` and `airflow/connections/*` paths in your Vault server. You can now run a Dag locally to check that your variables are accessible using `Variable.get("<your-variable-key>")`.

### (Optional) Authenticate to Vault with AWS Assume Role (STS)

If your Astro environment runs in AWS, you can authenticate to Vault using AWS IAM with an STS assume role instead of an AppRole. This lets Vault verify the assumed AWS IAM role rather than requiring you to manage a long-lived AppRole secret. For more details, see [Vault authentication with AWS Assume Role STS](https://airflow.apache.org/docs/apache-airflow-providers-hashicorp/stable/secrets-backends/hashicorp-vault.html#vault-authentication-with-aws-assume-role-sts) in the Apache Airflow documentation.

To use this authentication method, set `auth_type` to `aws_iam` and provide `assume_role_kwargs` with the IAM role to assume. For example:

```text wrap theme={null}
AIRFLOW__SECRETS__BACKEND=airflow.providers.hashicorp.secrets.vault.VaultBackend
AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_path": "connections", "variables_path": "variables", "mount_point": "airflow", "url": "<your-hashicorpvault-url>", "auth_type": "aws_iam", "assume_role_kwargs": {"RoleArn": "arn:aws:iam::<account-id>:role/<role-name>", "RoleSessionName": "Airflow"}}
```

For the full list of supported parameters in `assume_role_kwargs`, see the [AWS STS `assume_role` reference](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sts/client/assume_role.html).

## Step 4: Deploy configuration

<Tabs>
  <Tab title="Astro">
    1. Run the following commands to export your environment variables to Astro:

       ```sh wrap theme={null}
       astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND=airflow.providers.hashicorp.secrets.vault.VaultBackend

       astro deployment variable create --deployment-id <your-deployment-id> AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_path": "connections", "variables_path": "variables", "mount_point": "airflow", "url": "<your-hashicorpvault-url>", "auth_type": "approle", "role_id":"astro_role", "secret_id":"<your-approle-secret>"}' --secret
       ```

    2. Run the following command to push your updated `requirements.txt` file to Astro:

       ```sh wrap theme={null}
       astro deploy --deployment-id <your-deployment-id>
       ```

    3. (Optional) Remove the environment variables from your `.env` file or store your `.env` file in a safe location to protect your credentials in `AIRFLOW__SECRETS__BACKEND_KWARGS`.
  </Tab>

  <Tab title="Remote Execution">
    1. Run the following command to update your Remote Execution Agent with your new configurations.

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Tab>
</Tabs>

Now, any Airflow variable or connection that you write to your Vault server can be successfully accessed and pulled by any Dag in your Deployment on Astro.


# Develop a CI/CD workflow for deploying code to Astro
Source: https://astronomer.io/docs/astro/set-up-ci-cd

Plan a CI/CD strategy for deploying Apache Airflow code to Astro Deployments.

Continuous Integration and Continuous Delivery (CI/CD) pipelines are programmatic workflows that automate key parts of the software development lifecycle, including code changes, builds, and testing. CI/CD helps your organization develop faster, more securely, and more reliably.

There are many strategies for organizing your source code and building CI/CD pipelines for Apache Airflow on Astro, and each has its own benefits and limitations. Use this document to:

* Learn the benefits of CI/CD.
* Determine what kind of Deployment and CI/CD strategy you want your data team to use.

When you're ready to implement your CI/CD strategy, see [CI/CD templates](/docs/astro/ci-cd-templates/template-overview) for setup steps and examples for third-party CI/CD tools, like GitHub Actions and CircleCI.

## Benefits of CI/CD on Astro

On Astro, you can use [Deployment API tokens](/docs/astro/deployment-api-tokens) to automate your code deploys. Astronomer recommends hosting your Astro project source code in a version control tool, such as [GitHub](https://github.com/) or [GitLab](https://about.gitlab.com/), and setting up a CI/CD workflow for all production environments.

There are many benefits to configuring a CI/CD workflow on Astro. Specifically, you can:

* Avoid manually running `astro deploy` every time you make a change to your Astro project.
* Ensure that all changes to your Astro project are reviewed and approved by your team before they get pushed to Astro.
* Automate promoting code across development and production environments on Astro when pull requests to certain branches are merged.
* Enforce automated testing, which increases code quality and allows your team to respond quickly in case of an error or a failure.

<Tip>If you use [custom Deployment roles](/docs/astro/customize-deployment-roles) and Deployment API tokens, you might need to configure custom permissions instead of using a template. See [Create a custom Deployment role](/docs/astro/customize-deployment-roles#create-a-custom-deployment-role) for more information.</Tip>

## Choose a deploy strategy

You can set up CI/CD pipelines to manage multiple Deployments and repositories based on your team structure. Before you create your pipeline, you need to determine how many project environments and repositories you want to maintain.

When deciding on a strategy, keep the size of your data team and the importance of your Airflow dags in mind while answering the following questions:

* How many different environments do you need and want to invest in to adequately test your dags before deploying them to production?
* Do you need to limit your dag authors' access to sensitive project configurations? If so, do you need to host your Astro project in multiple repositories?

Read the following topics to learn which combination of environments and repositories is right for your team. Advanced data teams might follow a custom CI/CD strategy that is not described here and is optimized for a particular use case or team structure. If you're not sure which CI/CD strategy is right for you, contact your customer success representative.

### Environments

When you create a CI/CD pipeline, you must first determine how many environments you need to test your dags in. This typically depends on how critical these dags are to your business and how much your team expects to spend in infrastructure.

Each environment requires a Deployment on Astro, as well as its own GitHub branch and implementation in your CI/CD pipeline.

#### Single environment

The most cost-effective way to get started with CI/CD on Astro is to maintain a single Deployment on Astro for one Astro project. This solution is best for simple projects where you can tolerate testing and bug fixing in production.

This method assumes that you have:

* One Astro project.
* One Astro Workspace for your project.
* One Astro Deployment.
* One primary and permanent branch in your Git repository. Astronomer recommends `main`.

With this method, you can configure a CI/CD pipeline that deploys code to Astro every time a change is made to an Astro project file in your Git repository. To deploy a change to your project:

* Create a new, temporary branch in your Git repository against the `main` branch.
* Make a change to your Astro project and push those changes to the branch.
* Test all project changes locally with the Astro CLI. See [Test your Astro project locally](/docs/cli/v1.43/test-your-astro-project-locally) and [Troubleshoot your local Airflow environment](/docs/cli/v1.43/run-airflow-locally).
* Open a pull request for review.
* When the pull request is approved and merged to the `main` branch, the change is automatically deployed to your Deployment on Astro and available in the Airflow UI.

```mermaid actions={true} theme={null}
flowchart LR;
    classDef subgraph_padding fill:none,stroke:none
    classDef astro fill:#dbcdf6,stroke:#333,stroke-width:2px;
    id1[Local Astro project]-->|Push code change|id2[Feature branch]
    subgraph Git ["Git repository"]
    id2-->|Pull request| id3[Main branch]
    end
    id3-->|Deploy through CI/CD| id4[Astro Deployment]
```

Running all of your data pipelines in a single environment means that you don't have to pay for the infrastructure of multiple Deployments, but it limits your ability to test changes on Astro or on development datasets before they are deployed. Astronomer does not recommend this method for production use cases.

#### Multiple environments

For data teams running dags which are critical to your business, Astronomer recommends developing a CI/CD pipeline that supports multiple environments for running and testing different versions of your project, such as a production and development environment.

With this method, you maintain one Git repository with multiple permanent branches that each represent an environment in which you want to test your Astro project. You also have multiple Astro Deployments in a single Workspace that each correspond to one of the permanent branches. If you work at a larger organization, you can adapt this method by creating a Workspace and Astro project for each team or business use case.

The multiple environment method assumes that you have:

* One Astro project.
* One Astro Workspace for your project.
* At least two permanent branches in your Git repository that each represent an environment. Astronomer recommends naming the branches `main` and `dev`.
* At least two Astro Deployments.

```mermaid actions={true} theme={null}
%%{ init: { 'flowchart': { 'curve': 'linear' } } }%%
flowchart LR;
    classDef subgraph_padding fill:none,stroke:none
    classDef astro fill:#dbcdf6,stroke:#333,stroke-width:2px;
    id1[Local Astro project]-->|Push code change|id2[Feature branch]
    subgraph Git ["Git repository"]
    id2-->|Pull request| id3[Dev branch]
    id3 --> id5[Main branch]
    end
    id3-->|Deploy through CI/CD| id4[Dev Deployment]
    id5-->|Deploy through CI/CD| id6[Main Deployment]
    subgraph Astro [Astro]
    id4
    id6
    end
```

This method provides your team with at least two environments on Astro to test before pushing changes to production. Each Deployment can contain separate versions of your code, as well as separate environment configurations. If you use Snowflake, for example, your development Deployment on Astro can use a virtual data warehouse for development (`DWH Dev`), and your production Deployment can use a different virtual data warehouse for production (`DWH Prod`).

### Repositories

Astro supports deploying dag code changes separately from project configuration changes. This means that when you create a CI/CD pipeline for an Astro project, you can choose between the following strategies:

* Maintain a single Git repository for all files in your Astro project.
* Separate your dags from other files and maintain multiple Git repositories for a single Astro project.
* Store your dags in a cloud provider storage solution, such as AWS S3, and the rest of your Astro project files in a dedicated Git repository.

Develop this strategy in conjunction with your strategy for managing environments. Your CI/CD pipeline can manage deploying multiple branches across multiple repositories. See [Environments](#environments).

#### One repository

For most teams, Astronomer recommends creating a single Git repository for each Astro project. This means that your team can make changes to your dags, Python packages, and Deployment configuration in a single place. This strategy keeps your code history centralized, makes it easy for developers to contribute code, and avoids synchronization problems across files.

This strategy is recommended if:

* A single team is responsible for both developing dags and maintaining Deployments on Astro.
* You don't require a dedicated repository for dags across your organization.
* Less than 30 people interact with or contribute to your Astro project.

#### Multiple repositories

Depending on your organization, you might be required to maintain multiple repositories for a single Astro project. Data teams that implement this strategy typically manage dags in one repository and Astro project settings, such as Python packages and worker configuration, in another repository.

This strategy is recommended if:

* You have strict security requirements for who can update specific project files.
* You want to minimize complexity for project contributors at the expense of a more complex CI/CD pipeline.

```mermaid actions={true} wrap theme={null}
%%{ init: { 'flowchart': { 'curve': 'linear' } } }%%
flowchart LR;
classDef astro fill:#dbcdf6,stroke:#333,stroke-width:2px;
    id1[Admin's local Astro project]-->|Push project changes|id5[Git repository]
    id4[DAG author's local Astro project]-->|Push DAG changes|id2[Git repository]
    id5-->|Full project deploy|id3
    id2-->|DAG-only deploy| id3[Astro Deployment]
```

One limitation of this strategy is that you must keep any local copies of the Astro project synchronized with both repositories in order to test Deployment code locally and ensure that updates from your Admin repo doesn't erase your dags. Your team members might have inconsistencies in their local environments if they can't access code changes from other team members. Astronomer recommends setting up a `dev` Deployment where dag authors can see and modify project configurations for testing purposes.

This strategy requires [dag-only deploys](/docs/astro/deploy-dags#enable-or-disable-dag-only-deploys-on-a-deployment) on the target Deployment and setting up your CI/CD pipeline on both Git repositories.

### Store dags in storage bucket

Similar to the multiple repository strategy, this strategy separates the management of dags and project configuration. Dags are stored in an [S3](https://aws.amazon.com/s3/) or [Google Cloud Storage (GCS)](https://cloud.google.com/storage) bucket, while Astro project configuration files are stored in a Git repository.

```mermaid actions={true} wrap theme={null}
flowchart LR;
    classDef subgraph_padding fill:none,stroke:none
    classDef astro fill:#dbcdf6,stroke:#333,stroke-width:2px;
    id1[Admin's local Astro project]-->|Push project changes|id5[Git repository]
    id4[dag author's local Astro project]-->|Push dag changes|id2[dag bucket]
    id2-->|"dag-only deploy" | id3[Astro Deployment]
    id5-->|"Image-only deploy </br> (CI/CD)"|id3
```

If you migrated to Astro from Amazon Managed Workflows for Apache Airflow (MWAA) or Google Cloud Composer (GCC), this strategy is useful for maintaining a similar workflow for dag authors. For example, you can set up a Lambda function to push dags to your Astronomer Deployment whenever dag files are updated in your specific S3 bucket.

## Create a CI/CD pipeline

When you set up a CI/CD workflow on Astro, you will:

* Select a CI/CD strategy after reviewing this document and the requirements of your team.
* Set up your repositories and permissions based on your CI/CD strategy.
* Add an Astronomer [CI/CD template](/docs/astro/ci-cd-templates/template-overview) to your repositories.
* Modify the Astronomer template or GitHub action to meet the requirements of your organization.

If you use GitHub, Astronomer recommends using the [`deploy-action` GitHub action](https://github.com/astronomer/deploy-action) that is maintained by Astronomer.

## Enforce CI/CD

When you use a CI/CD pipeline, all code pushes to your Deployment are tested, standardized, and observable through your pipeline. For Deployments where these qualities are a priority, Astronomer recommends enabling CI/CD enforcement so that code pushes can be completed only when using a Deployment or Workspace token. See:

* [Enforce CI/CD deploys](/docs/astro/deployment-details#enforce-ci/cd-deploys)
* [Update general Workspace settings](/docs/astro/manage-workspaces#update-general-workspace-settings)

## Test and validate dags in your CI/CD pipeline

Astronomer recommends that you pytest all Python code in your dags. The Astro CLI includes [pytests](/docs/cli/v1.43/test-your-astro-project-locally#unit-test-dags) to validate that your dags do not have import or syntax errors. You can implement this parse test with the [Astro CLI](/docs/cli/v1.43/astro-dev-parse) or the [Deploy Action](https://github.com/astronomer/deploy-action).

The default test may not work on all dags, especially if they access the Airflow metadata database. In this case, you can write your own parse test using example pytests provided in the default Astro project.


# Set up SCIM provisioning on Astro
Source: https://astronomer.io/docs/astro/set-up-scim-provisioning

Configure SCIM provisioning to import groups of users from your identity provider to Astro as Teams.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

Astro supports integration with the open standard System for Cross-Domain Identity Management (SCIM). Using the SCIM protocol with Astro allows you to automatically provision and deprovision users and [Teams](/docs/astro/manage-teams) based on templates for access and permissions. It also provides better observability through your identity provider for when users and Teams are created or modified across your organization. Specifically, you can utilize SCIM provisioning to complete the following Astro actions from your identity provider platform:

* Create and remove users in your Organization.
* Update user profile information.
* Create and remove Astro Teams.
* Add and remove Team members.
* Retrieve user and Team information.

<Info>
  Some user management features on Astro behave differently after you set up SCIM provisioning. See [Manage Teams](/docs/astro/manage-teams#teams-and-scim-provisioning) for more information.
  Astro doesn't support group nesting for SCIM provisioning. Access levels assigned to parent groups don't automatically propagate to child groups, so each group must be individually assigned the required access levels.
</Info>

## Supported SSO identity providers

Astro supports SCIM provisioning with the following IdPs:

* [Microsoft Entra ID](https://www.microsoft.com/en-us/security/business/microsoft-entra)
* [Okta](https://www.okta.com/)

### Supported Okta features

Okta's Astro integration supports the following SCIM actions:

* Create users
* Update user attributes
* Deactivate users
* Group push

## Prerequisites

* A configured identity provider. See [Set up SSO](/docs/astro/configure-idp).

## Setup

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

<Tabs>
  <Tab title="Okta - Astro integration (Recommended)">
    <Tabs>
      <Tab title="New Astro UI">
        1. Create an Organization API token with Organization Owner permissions. See [Organization API tokens](/docs/astro/organization-api-tokens). Copy the token to use later in this setup.

        2. In the Astro UI, go to **Settings**.

        3. Copy your **Organization ID** to use later in this setup.

        4. Go to **Settings**, then in the **Security** section, click **Authentication**, then in the **Advanced Settings** section, click **Edit Settings** and turn on the **SCIM integration** toggle.

        5. In the Okta admin dashboard, open your Astro app integration and click **Provisioning**.

        6. Click **Configure API integration**, check **Enable API integration**, then configure the following values:

           * **Organization ID**: Enter your **Organization ID**.
           * **API token**: Enter your Organization API token.

        7. Test your API credentials, then click **Save**.

        8. In the **Provisioning** menu, click **To App** and configure the following:

        * **Provisioning to App**: Select only **Create Users**, **Update User Attributes**, and **Deactivate Users**.

          See [Okta documentation](https://developer.okta.com/docs/guides/scim-provisioning-integration-connect/main/#to-app) for more information on configuring these values.

        9. Create user groups and push them to Astro. User groups pushed to Astro appear as [Teams](/docs/astro/manage-teams) in the Astro UI. See [Okta documentation](https://help.okta.com/en-us/Content/Topics/users-groups-profiles/usgp-enable-group-push.htm) for setup steps.
      </Tab>

      <Tab title="Legacy UI">
        1. Create an Organization API token with Organization Owner permissions. See [Organization API tokens](/docs/astro/organization-api-tokens). Copy the token to use later in this setup.

        2. In the Astro UI, click **Organization Settings**.

        3. On the **General** page, copy your **Organization ID** to use later in this setup.

        4. Go to **Settings > Authentication**. In the **Advanced Settings** menu, click **Edit Settings**, then click the **SCIM integration** toggle to on.

        5. In the Okta admin dashboard, open your Astro app integration and click **Provisioning**.

        6. Click **Configure API integration**, check **Enable API integration**, then configure the following values:

           * **Organization ID**: Enter your **Organization ID**.
           * **API token**: Enter your Organization API token.

        7. Test your API credentials, then click **Save**.

        8. In the **Provisioning** menu, click **To App** and configure the following:

        * **Provisioning to App**: Select only **Create Users**, **Update User Attributes**, and **Deactivate Users**.

          See [Okta documentation](https://developer.okta.com/docs/guides/scim-provisioning-integration-connect/main/#to-app) for more information on configuring these values.

        9. Create user groups and push them to Astro. User groups pushed to Astro appear as [Teams](/docs/astro/manage-teams) in the Astro UI. See [Okta documentation](https://help.okta.com/en-us/Content/Topics/users-groups-profiles/usgp-enable-group-push.htm) for setup steps.
      </Tab>
    </Tabs>
  </Tab>

  <Tab title="Okta - Manual">
    Complete the manual setup if you configured your existing Astro app without using the Okta app catalogue.

    <Tabs>
      <Tab title="New Astro UI">
        1. Create an Organization API token with Organization Owner permissions. See [Organization API tokens](/docs/astro/organization-api-tokens). Copy the token to use later in this setup.

        2. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.

        3. In the **Advanced Settings** menu, click **Edit Settings**, then turn on the **SCIM integration** toggle.

        4. Copy the **SCIM Integration URL** that appears.

        5. In the Okta admin dashboard, add SCIM provisioning to your existing Astro app integration. Then, open your app in Okta and go to **Provisioning** > **Integration** to configure the following values:

           * **Supported provisioning actions**: Select **Push New Users**, **Push Profile Updates**, and **Push Groups**.
           * **SCIM connector base URL**: Enter the SCIM integration URL you copied from the Astro UI.
           * **Unique identifier field for users**: `email`.
           * **Authentication Mode**: Choose **HTTP Header** and paste your Organization API token in the **Bearer** field.

             See [Okta documentation](https://help.okta.com/en-us/Content/Topics/Apps/Apps_App_Integration_Wizard_SCIM.htm) for more information about setting up SCIM provisioning.

        6. In the **Provisioning** menu, click **To App** and configure the following:

           * **Provisioning to App**: Select **Create Users**, **Update User Attributes**, and **Deactivate Users**.
           * **Astro Attribute Mappings**: Configure the following mappings:

             | Attribute                    | Attribute Type | Value                          | Apply On          |
             | ---------------------------- | -------------- | ------------------------------ | ----------------- |
             | Username (`userName`)        | Personal       | Configured in sign-on settings |                   |
             | Given name  (`givenName`)    | Personal       | user.firstName                 | Create and update |
             | Family name (`familyName`)   | Personal       | user.lastName                  | Create and update |
             | Email  (`email`)             | Personal       | `user.email`                   | Create and update |
             | Display name (`displayName`) | Personal       | user.displayName               | Create and update |
             | Profile Url  (`profileUrl`)  | Personal       | user.profileUrl                | Create and update |

             See [Okta documentation](https://developer.okta.com/docs/guides/scim-provisioning-integration-connect/main/#to-app) for more information on configuring these values.

        7. Create user groups and push them to Astro. User groups pushed to Astro appear as [Teams](/docs/astro/manage-teams) in the Astro UI. See [Okta documentation](https://help.okta.com/en-us/Content/Topics/users-groups-profiles/usgp-enable-group-push.htm) for setup steps.
      </Tab>

      <Tab title="Legacy UI">
        1. Create an Organization API token with Organization Owner permissions. See [Organization API tokens](/docs/astro/organization-api-tokens). Copy the token to use later in this setup.

        2. In the Astro UI, click **Organization Settings**, then click **Authentication**.

        3. In the **Advanced Settings** menu, click **Edit Settings**, then click the **SCIM integration** toggle to on.

        4. Copy the **SCIM Integration URL** that appears.

        5. In the Okta admin dashboard, add SCIM provisioning to your existing Astro app integration. Then, open your app in Okta and go to **Provisioning** > **Integration** to configure the following values:

           * **Supported provisioning actions**: Select **Push New Users**, **Push Profile Updates**, and **Push Groups**.
           * **SCIM connector base URL**: Enter the SCIM integration URL you copied from the Astro UI.
           * **Unique identifier field for users**: `email`.
           * **Authentication Mode**: Choose **HTTP Header** and paste your Organization API token in the **Bearer** field.

             See [Okta documentation](https://help.okta.com/en-us/Content/Topics/Apps/Apps_App_Integration_Wizard_SCIM.htm) for more information about setting up SCIM provisioning.

        6. In the **Provisioning** menu, click **To App** and configure the following:

           * **Provisioning to App**: Select **Create Users**, **Update User Attributes**, and **Deactivate Users**.
           * **Astro Attribute Mappings**: Configure the following mappings:

             | Attribute                    | Attribute Type | Value                          | Apply On          |
             | ---------------------------- | -------------- | ------------------------------ | ----------------- |
             | Username (`userName`)        | Personal       | Configured in sign-on settings |                   |
             | Given name  (`givenName`)    | Personal       | user.firstName                 | Create and update |
             | Family name (`familyName`)   | Personal       | user.lastName                  | Create and update |
             | Email  (`email`)             | Personal       | `user.email`                   | Create and update |
             | Display name (`displayName`) | Personal       | user.displayName               | Create and update |
             | Profile Url  (`profileUrl`)  | Personal       | user.profileUrl                | Create and update |

             See [Okta documentation](https://developer.okta.com/docs/guides/scim-provisioning-integration-connect/main/#to-app) for more information on configuring these values.

        7. Create user groups and push them to Astro. User groups pushed to Astro appear as [Teams](/docs/astro/manage-teams) in the Astro UI. See [Okta documentation](https://help.okta.com/en-us/Content/Topics/users-groups-profiles/usgp-enable-group-push.htm) for setup steps.
      </Tab>
    </Tabs>
  </Tab>

  <Tab title="Microsoft Entra ID">
    <Tabs>
      <Tab title="New Astro UI">
        1. Create an Organization API token with Organization Owner permissions. See [Organization API tokens](/docs/astro/organization-api-tokens). Copy the token to use later in this setup.
        2. In the Astro UI, go to **Settings**, then in the **Security** section, click **Authentication**.
        3. In the **Advanced Settings** menu, click **Edit Settings**, then turn on the **SCIM integration** toggle.
        4. Copy the **SCIM Integration URL** that appears.
        5. Append the [Microsoft Entra ID feature flag parameter](https://learn.microsoft.com/en-us/azure/active-directory/app-provisioning/application-provisioning-config-problem-scim-compatibility#flags-to-alter-the-scim-behavior) `?aadOptscim062020` to your **SCIM Integration URL** and recopy it. For example, if your SCIM Integration URL is `https://api.astronomer.io/scim/v2/cknaqyipv05731evsry6cj4n0`, your final URL would be `https://api.astronomer.io/scim/v2/cknaqyipv05731evsry6cj4n0?aadOptscim062020`. The feature flag is required for fully compliant SCIM behavior in Microsoft Entra ID.
        6. In the Microsoft Entra ID management dashboard, [create a new enterprise application](https://learn.microsoft.com/en-us/azure/active-directory/manage-apps/add-application-portal#add-an-enterprise-application) with the third option, **Integrate any other application you don't find in the gallery**.
        7. In the menu for your new application, click **Provisioning** and configure the following values:

           * **Provisioning mode**: Set to **Automatic**.
           * **Admin Credentials** > **Tenant URL**: Enter the **SCIM integration URL** including the Microsoft Entra ID feature flag parameter.
           * **Secret Token**: Enter your Organization API token.
           * **Authentication Method**: Select **Bearer Authentication**.

        <Info>
          **Only provision users**

          If you prefer to only provision users with SCIM, and you don't want to provision Teams, skip Steps 8-10.
        </Info>

        8. In **Mappings**, open the **Groups** mapping configuration.
        9. In **Target Object Actions**, tick the checkboxes for **Create**, **Update**, and **Delete**.
        10. In the **Attribute Mappings** table, add the following mappings:

            | Microsoft Entra ID Attribute | Astro Attribute |
            | ---------------------------- | --------------- |
            | displayName                  | displayName     |
            | members                      | members         |

            Delete any other group attributes not listed in the previous table. You should have exactly two attributes as shown in the following screenshot:

        <Frame>
          <img alt="Azure group mappings with only the correct 2 attributes listed" />
        </Frame>

        11. Go back to the **Mappings** menu and open the **User** mapping configuration.
        12. In **Target Object Actions**, tick the checkboxes for **Create**, **Update**, and **Delete**.
        13. In the **Attribute Mappings** table, add the following mappings:

            | Microsoft Entra ID Attribute                                  | Astro Attribute |
            | ------------------------------------------------------------- | --------------- |
            | `mail`                                                        | `userName`      |
            | `Switch([IsSoftDeleted], , "False", "True", "True", "False")` | `active`        |
            | `displayName`                                                 | `displayName`   |

            To successfully configure the `active` mapping in the table, you must first complete these steps:

            If not already present, create the `active` Attribute:

            * Click **Show advanced options**.
            * Select **Edit attribute list**.
            * Add a new attribute:
              * **Name**: `active`
              * **Type**: Boolean
            * Click **Save**.

            Add the Attribute Mapping using an Expression:

            * Go back to **Attribute Mappings**.

            * Click **Add New Mapping**.

            * Change the **Type** to **Expression**.

            * In the **Entra ID Attribute** field, enter:

              ```text wrap theme={null}
              Switch([IsSoftDeleted], , "False", "True", "True", "False")
              ```

            * In the **Attribute** field, select `active`.

            * Click **Save**.

        <Warning>
          This setup assumes that `mail` contains your users' email. If you use a field other than `mail` to define your user email, replace `mail` with the attribute you use.

          Whether you choose to use `mail` or a different attribute like `userPrincipleName`, you must use the same Microsoft Entra ID Attribute for both your SSO and SCIM configurations.
        </Warning>

        Delete any other user attributes not listed in the previous table. You should have exactly three attributes as shown in the following screenshot:

        <Frame>
          <img alt="Azure user mappings with only the correct 3 attributes listed" />
        </Frame>

        14. Click **Test connection** in the Microsoft Entra ID application management menu to confirm your connection to the SCIM endpoint.
      </Tab>

      <Tab title="Legacy UI">
        1. Create an Organization API token with Organization Owner permissions. See [Organization API tokens](/docs/astro/organization-api-tokens). Copy the token to use later in this setup.
        2. In the Astro UI, click **Organization Settings**, then click **Authentication**.
        3. In the **Advanced Settings** menu, click **Edit Settings**, then click the **SCIM integration** toggle to on.
        4. Copy the **SCIM Integration URL** that appears.
        5. Append the [Microsoft Entra ID feature flag parameter](https://learn.microsoft.com/en-us/azure/active-directory/app-provisioning/application-provisioning-config-problem-scim-compatibility#flags-to-alter-the-scim-behavior) `?aadOptscim062020` to your **SCIM Integration URL** and recopy it. For example, if your SCIM Integration URL is `https://api.astronomer.io/scim/v2/cknaqyipv05731evsry6cj4n0`, your final URL would be `https://api.astronomer.io/scim/v2/cknaqyipv05731evsry6cj4n0?aadOptscim062020`. The feature flag is required for fully compliant SCIM behavior in Microsoft Entra ID.
        6. In the Microsoft Entra ID management dashboard, [create a new enterprise application](https://learn.microsoft.com/en-us/azure/active-directory/manage-apps/add-application-portal#add-an-enterprise-application) with the third option, **Integrate any other application you don't find in the gallery**.
        7. In the menu for your new application, click **Provisioning** and configure the following values:

           * **Provisioning mode**: Set to **Automatic**.
           * **Admin Credentials** > **Tenant URL**: Enter the **SCIM integration URL** including the Microsoft Entra ID feature flag parameter.
           * **Secret Token**: Enter your Organization API token.
           * **Authentication Method**: Select **Bearer Authentication**.

        <Info>
          **Only provision users**

          If you prefer to only provision users with SCIM, and you don't want to provision Teams, skip Steps 8-10.
        </Info>

        8. In **Mappings**, open the **Groups** mapping configuration.
        9. In **Target Object Actions**, tick the checkboxes for **Create**, **Update**, and **Delete**.
        10. In the **Attribute Mappings** table, add the following mappings:

            | Microsoft Entra ID Attribute | Astro Attribute |
            | ---------------------------- | --------------- |
            | displayName                  | displayName     |
            | members                      | members         |

            Delete any other group attributes not listed in the previous table. You should have exactly two attributes as shown in the following screenshot:

        <Frame>
          <img alt="Azure group mappings with only the correct 2 attributes listed" />
        </Frame>

        11. Go back to the **Mappings** menu and open the **User** mapping configuration.
        12. In **Target Object Actions**, tick the checkboxes for **Create**, **Update**, and **Delete**.
        13. In the **Attribute Mappings** table, add the following mappings:

            | Microsoft Entra ID Attribute                                  | Astro Attribute |
            | ------------------------------------------------------------- | --------------- |
            | `mail`                                                        | `userName`      |
            | `Switch([IsSoftDeleted], , "False", "True", "True", "False")` | `active`        |
            | `displayName`                                                 | `displayName`   |

            To successfully configure the `active` mapping in the table, you must first complete these steps:

            If not already present, create the `active` Attribute:

            * Click **Show advanced options**.
            * Select **Edit attribute list**.
            * Add a new attribute:
              * **Name**: `active`
              * **Type**: Boolean
            * Click **Save**.

            Add the Attribute Mapping using an Expression:

            * Go back to **Attribute Mappings**.

            * Click **Add New Mapping**.

            * Change the **Type** to **Expression**.

            * In the **Entra ID Attribute** field, enter:

              ```text wrap theme={null}
              Switch([IsSoftDeleted], , "False", "True", "True", "False")
              ```

            * In the **Attribute** field, select `active`.

            * Click **Save**.

        <Warning>
          This setup assumes that `mail` contains your users' email. If you use a field other than `mail` to define your user email, replace `mail` with the attribute you use.

          Whether you choose to use `mail` or a different attribute like `userPrincipleName`, you must use the same Microsoft Entra ID Attribute for both your SSO and SCIM configurations.
        </Warning>

        Delete any other user attributes not listed in the previous table. You should have exactly three attributes as shown in the following screenshot:

        <Frame>
          <img alt="Azure user mappings with only the correct 3 attributes listed" />
        </Frame>

        14. Click **Test connection** in the Microsoft Entra ID application management menu to confirm your connection to the SCIM endpoint.
      </Tab>
    </Tabs>
  </Tab>
</Tabs>

## Frequently asked questions

<AccordionGroup>
  <Accordion title="What if an Okta group is out of sync with an Astro Team?">
    1. In the Okta dashboard, open the Astro application and click **Push Groups**.
    2. Click the value in **Push Status** for the group that's out of sync, then click **Push now**.
  </Accordion>

  <Accordion title="What if an Okta user is out of sync with their Astro user account?">
    If you removed an Okta user but their Astro account remains, [delete the account from Astro](/docs/astro/manage-organization-users#update-or-remove-an-organization-user).

    If an Astro user isn't appearing for an Okta user as expected, remove and re-assign the user in Okta.
  </Accordion>
</AccordionGroup>


# Enable Sub-Second Pipelines
Source: https://astronomer.io/docs/astro/sub-second-pipelines

Use Sub-Second Pipelines to dispatch API-triggered Dag runs in under a second and sustain a high rate of concurrent triggers.

<Note>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews) and is only available for Airflow 3.2+ Deployments.
</Note>

Sub-Second Pipelines are an Astro feature that improves the latency and throughput of triggered Dag runs. They pair the [Astro executor](/docs/astro/astro-executor) with a dedicated event-driven scheduler, so that Astro picks up and dispatches Dag runs in under a second and sustains a higher rate of concurrent triggers than the standard Airflow scheduler. Use them for any workload that fires many API-triggered runs in a short window, not only workloads where individual run latency matters.

This document explains how to enable Sub-Second Pipelines on a Deployment, configure a sub-second worker queue, and route Dags to it.

<Warning>
  Sub-Second Pipelines support API-triggered Dag runs only. Runs triggered by the Airflow standard scheduler (cron schedules and timetables), asset and data-aware scheduling, and message queue triggers continue to use the standard scheduler and don't benefit from sub-second dispatch. Astronomer plans to support additional trigger types in a future release.
</Warning>

## When to use Sub-Second Pipelines

Sub-Second Pipelines are designed for API-triggered workloads where throughput, startup latency, or both are critical. Common use cases include:

* **High-throughput triggered workflows**: Applications, agents, or upstream systems that fire hundreds of Dag runs per second through the Airflow REST API. Sub-Second Pipelines sustain approximately 1,000 Dag runs per minute, where the Celery executor queues and falls behind before that point.
* **On-demand inference**: Machine learning pipelines triggered by an application or service through the Airflow REST API, where end-to-end latency directly affects the user experience.
* **Programmatic workflow invocation**: Backend services that call the Airflow REST API to start a pipeline and need it to start immediately.
* **Reverse ETL and operational pipelines**: API-driven workflows where freshness budgets are measured in seconds rather than minutes.

For scheduled batch workloads, such as cron schedules and timetables, or for asset and event-driven runs, the default scheduling behavior applies and you don't need this feature.

## How it works

Sub-Second Pipelines introduce an Event Scheduler that runs alongside the Airflow standard scheduler in your Deployment. When you trigger a Dag run through the Airflow REST API, the Event Scheduler picks up the request from an internal event bus and immediately spawns the Dag run, bypassing the polling interval that the standard scheduler relies on.

Because the Event Scheduler is event-driven rather than poll-driven, it scales linearly with trigger volume instead of being bottlenecked by a fixed scheduling loop. Combined with the Astro executor's centralized task assignment, this is what enables approximately 1,000 Dag runs per minute. Sizing the Event Scheduler with additional replicas lets it sustain higher throughput under bursty load.

A Dag uses the faster path only when both of the following are true:

* The run was triggered through the Airflow REST API, or the **Trigger Dag** button in the Airflow UI.
* The Dag's tasks are routed to a worker queue that has the **Sub-Second** toggle enabled.

Runs triggered any other way, such as a cron schedule, timetable, asset update, or message queue, and Dags routed to non-sub-second queues, continue to use the standard scheduler. You can mix sub-second Dags and standard Dags in the same Deployment without affecting your existing workloads.

### How to tell whether a Dag qualifies

Astro automatically tags each Dag based on whether it qualifies for sub-second scheduling, so that you can confirm a Dag's status from its tags in the Airflow UI:

* A qualifying Dag is tagged `sub_second`.
* A Dag that doesn't qualify is tagged `sub_second_excluded`, plus a second tag that names the reason, such as `sub_second_excluded_mixed_queues`.

Several conditions can disqualify a Dag, such as mixing sub-second and non-sub-second queues or setting `depends_on_past=True`. The Dag warning on the Dag's page in the Airflow UI names the specific reason. A Dag that doesn't qualify still runs; it uses the standard scheduler instead of the sub-second path.

## Prerequisites

* A Deployment running [Astro Runtime](https://www.astronomer.io/docs/astro/runtime-version-lifecycle-policy) 3.2 or later, which is based on Airflow 3.2 or later.
* A Deployment that uses the [Astro executor](/docs/astro/astro-executor). Sub-Second Pipelines don't support the Celery or Kubernetes executor.
* Permission to edit Deployment settings.

## Enable Sub-Second Pipelines on the Deployment

<Steps>
  <Step title="Open the Deployment">
    In the Astro UI, open your Deployment and click the **Details** tab.
  </Step>

  <Step title="Edit the execution settings">
    In the **Execution** section, click **Edit**.
  </Step>

  <Step title="Confirm the executor">
    Confirm that **Executor** is set to **Astro Executor**.
  </Step>

  <Step title="Enable the toggle">
    Set the **Sub-Second Pipelines** toggle to **On**.

    <Frame>
      <img alt="Sub-Second Pipelines toggle in the Deployment Execution settings" />
    </Frame>

    Enabling the toggle provisions the Event Scheduler component for your Deployment but doesn't change the behavior of any Dags. Dags use the fast path only after you route them to a sub-second worker queue, which you create next.
  </Step>
</Steps>

## Create a sub-second worker queue

Sub-second behavior is opt-in for each [worker queue](/docs/astro/configure-worker-queues). Astronomer recommends creating a dedicated queue for your latency-sensitive Dags instead of enabling it on the default queue, so that you can size and scale the queue independently.

<Steps>
  <Step title="Add a worker queue">
    In the **Execution** section, scroll to **Worker Queues** and click **Add Queue**.
  </Step>

  <Step title="Configure the queue">
    Configure the following settings:

    * **Queue Name**: Enter a short, descriptive name, such as `fast-high-priority`. You reference this name from your Dag code.
    * **Worker Type**: Choose a worker size appropriate for your tasks. For low-latency workloads, a smaller worker type with more workers is often a better fit than a large worker type with few workers.
    * **Storage**: Keep the default of 10 GiB unless your tasks need more ephemeral storage.
    * **Concurrency**: Set the number of tasks that a single worker can run in parallel.
    * **Min # Workers**: Set this to at least `1`. Sub-second startup depends on having a worker already warm and ready, so scale-to-zero defeats the purpose of the feature.
    * **Max # Workers**: Size this for your expected peak concurrency.
  </Step>

  <Step title="Enable the Sub-Second toggle">
    Set the **Sub-Second** toggle in the queue row to **On**.

    <Frame>
      <img alt="Sub-Second toggle on a worker queue" />
    </Frame>
  </Step>

  <Step title="Save the configuration">
    Click **Update Deployment**.
  </Step>
</Steps>

## Size the Event Scheduler

The Event Scheduler runs as one or more replicas in your Deployment. For production workloads, run at least two replicas so that the scheduler stays available during restarts and can absorb bursts.

<Steps>
  <Step title="Open the Advanced section">
    In the Deployment edit view, expand the **Advanced** section.
  </Step>

  <Step title="Set the replica count">
    Set **Scheduler Replicas** to `2`, or higher if you expect heavy concurrent trigger volume.

    <Frame>
      <img alt="Scheduler Replicas in the Advanced settings" />
    </Frame>

    For high trigger rates, also turn on **API Server Autoscaling** in the same section so that the API server can keep up with incoming triggers. See [API server autoscaling](/docs/astro/api-server-autoscaling).
  </Step>

  <Step title="Save the configuration">
    Click **Update Deployment**.
  </Step>
</Steps>

<Tip>
  Autoscaling for the Event Scheduler is on the near-term roadmap. Until it ships, you set the replica count manually. If you expect highly variable load, choose a replica count that covers your peak rather than your average.
</Tip>

## Route your Dag to the sub-second queue

A Dag runs on the sub-second path only if its tasks are assigned to the sub-second worker queue. Set the queue in `default_args` so that every task in the Dag inherits it. The `queue` value must exactly match the **Queue Name** you set in [Create a sub-second worker queue](#create-a-sub-second-worker-queue). If a task is assigned to a queue that doesn't exist or isn't referenced properly, the task might remain in a `queued` state and fail to execute.

The following example assigns every task in the Dag to the `fast-high-priority` queue:

```python wrap theme={null}
from airflow.sdk import dag, task

@dag(default_args={"queue": "fast-high-priority"})
def payment_risk_check_on_demand():
    @task
    def score_transaction():
        ...

    score_transaction()

payment_risk_check_on_demand()
```

To route specific tasks to different sub-second queues, pass a `queue` argument to the `@task` decorator instead:

```python wrap theme={null}
@task(queue="fast-high-priority")
def score_transaction():
    ...
```

<Warning>
  Sub-Second Pipelines don't support using sub-second and standard worker queues in the same Dag. If you assign tasks from the same Dag to both types of queues, the standard scheduler routes the Dag.
</Warning>

Deploy the Dag to your Deployment as you normally would, using `astro deploy`, a Git-based deploy, or the [Astro IDE](/docs/astro/ide-overview).

## Verify sub-second behavior

After you deploy, trigger a run of your Dag through the [Airflow REST API](/docs/astro/airflow-api) or the **Trigger Dag** button in the Airflow UI:

```sh wrap theme={null}
curl -X POST \
  "https://<your-deployment-url>/api/v2/dags/payment_risk_check_on_demand/dagRuns" \
  -H "Authorization: Bearer $ASTRO_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{}'
```

Replace `<your-deployment-url>` with your Deployment's Airflow API URL and `$ASTRO_API_TOKEN` with a valid [Deployment API token](/docs/astro/deployment-api-tokens).

You can confirm that a run took the sub-second path in several ways. The Airflow UI is the quickest place to check, and the SLA Metrics API is available for programmatic access.

### Check the Dag's tags

In the Airflow UI, open the Dag and check its tags. A Dag on the sub-second path is tagged `sub_second`. If it's tagged `sub_second_excluded` instead, the run uses the standard scheduler. See [How to tell whether a Dag qualifies](#how-to-tell-whether-a-dag-qualifies) for the reason tags.

### Check the Sub-Second Metrics tab in the Airflow UI

Each sub-second Dag has a **Sub-Second Metrics** tab in the Airflow UI that shows time to first task and per-task lag.

Open the tab from a Dag's page for a windowed aggregate across recent runs, which is useful for spotting regressions. Select the time window with the **Window** menu.

<Frame>
  <img alt="Dag-level Sub-Second Metrics tab in the Airflow UI" />
</Frame>

Open the tab from an individual Dag run to see that run's time to first task, average task lag, and a per-task lag breakdown.

<Frame>
  <img alt="Per-run Sub-Second Metrics tab in the Airflow UI" />
</Frame>

### Check the Event Scheduler logs

In the Astro UI, open the Deployment's **Logs** tab and set the **Source** filter to **Event Scheduler**. After you trigger a run, watch for log lines from the `astronomer.event_scheduler` component that confirm the run was received and scheduled, with timestamps milliseconds apart.

<Frame>
  <img alt="Event Scheduler logs in the Astro UI Logs tab" />
</Frame>

A successful sub-second run produces a sequence similar to the following:

```text wrap theme={null}
Event received from Redis ... event_type: schedule_dagruns
Discovered DagRuns ... count: 1
DagRun found via Redis event ... dag_id: payment_risk_check_on_demand
DagRun scheduler added ... active_count: 1
DagRun scheduler started
DagRun transitioned to running
DagRun completed ... state: success
DagRun scheduler stopped ... duration_seconds: ~2
```

The `duration_seconds` value on the final line shows the total run time.

### Query the SLA Metrics API

For programmatic access, such as monitoring dashboards or latency alerts, the Event Scheduler exposes two endpoints with the same metrics shown in the **Sub-Second Metrics** tab.

Use the per-run endpoint to retrieve metrics for a specific Dag run:

```sh wrap theme={null}
curl -sS -X GET --location \
  "https://<your-deployment-url>/astro-event-scheduler/sla_metrics/dag_runs/<dag_id>/<run_id>" \
  --header "Authorization: Bearer $ASTRO_API_TOKEN" \
  --header "Accept: application/json"
```

The endpoint returns a response similar to the following:

```json wrap theme={null}
{
  "dag_id": "payment_risk_check_on_demand",
  "run_id": "manual__2026-05-22T17:36:50.487342+00:00",
  "time_to_first_task": 0.691809,
  "task_lag": [
    {
      "task_id": "load_payment_attempt",
      "map_index": -1,
      "task_start_lag": 0.691809
    },
    {
      "task_id": "lookup_customer_history",
      "map_index": -1,
      "task_start_lag": 0.748826
    },
    {
      "task_id": "score_transaction",
      "map_index": -1,
      "task_start_lag": 0.743206
    }
  ]
}
```

The per-run endpoint returns scalar measurements, in seconds, for a single Dag run: one `time_to_first_task` value for the run, and one `task_lag` entry for each task instance (except for the root task), including each mapped task instance. In each `task_lag` entry, `map_index` is the dynamic map index, which is `-1` for non-mapped tasks, and `task_start_lag` is the per-task scheduling lag.

Use the aggregate endpoint to retrieve metrics across a time window:

```sh wrap theme={null}
curl -sS -X GET --location \
  "https://<your-deployment-url>/astro-event-scheduler/sla_metrics/aggregate?since=2026-05-21T00:00:00Z" \
  --header "Authorization: Bearer $ASTRO_API_TOKEN" \
  --header "Accept: application/json"
```

The endpoint returns a response similar to the following:

```json wrap theme={null}
{
  "since": "2026-05-21T00:00:00Z",
  "until": null,
  "dag_id": "payment_risk_check_on_demand",
  "is_sub_second": true,
  "dag_runs_considered": 5,
  "time_to_first_task": {
    "count": 5,
    "min": 0.678138,
    "max": 0.714617,
    "mean": 0.688747,
    "p50": 0.680133,
    "p95": 0.710055,
    "p99": 0.713705
  },
  "task_lag": {
    "count": 15,
    "min": 0.688975,
    "max": 0.748826,
    "mean": 0.720769,
    "p50": 0.722849,
    "p95": 0.744892,
    "p99": 0.748039
  }
}
```

The response echoes the filter parameters and returns aggregated distributions, in seconds, across all Dag runs in the window:

* `time_to_first_task`: The distribution of `time_to_first_task` values across all Dag runs considered.
* `task_lag`: The distribution of `task_start_lag` values across all task instances in those runs.

The aggregate endpoint accepts the following query parameters:

| Parameter       | Required | Description                                                                                    |
| --------------- | -------- | ---------------------------------------------------------------------------------------------- |
| `since`         | Yes      | A UTC ISO-8601 timestamp. Includes only Dag runs with `queued_at >= since`.                    |
| `until`         | No       | An exclusive upper bound as a UTC ISO-8601 timestamp.                                          |
| `dag_id`        | No       | Restricts results to a single Dag.                                                             |
| `is_sub_second` | No       | Filters by sub-second-tagged Dags. Defaults to `true`. Omit the parameter to include all runs. |

## Troubleshoot Sub-Second Pipelines

If the run doesn't appear in the **Sub-Second Metrics** tab, the Event Scheduler logs, or the SLA Metrics API, confirm the following:

* You triggered the run through the Airflow REST API or the Airflow UI **Trigger Dag** button, not through a cron schedule, timetable, or asset update.
* The Dag's `queue` value matches a worker queue that has **Sub-Second** enabled.
* The Deployment-level **Sub-Second Pipelines** toggle is on.
* The Dag isn't tagged `sub_second_excluded`. Check the tag and the Dag warning for the reason.
* The Deployment uses the Astro executor on Astro Runtime 3.2 or later.

## Best practices

* **Don't enable sub-second on every queue.** The Event Scheduler is most efficient when it focuses on the workloads that need it. Keep batch and scheduled Dags on the default queue.
* **Size workers for throughput, not size.** Many smaller workers usually outperform a few large workers for latency-sensitive workloads. Choose a worker type that matches the per-task resource needs of your Dag, not the largest type available.
* **Test under realistic load.** Trigger your Dag through the Airflow REST API at the rate you expect in production and watch the Event Scheduler metrics. If you see queue buildup, increase **Max # Workers** on the queue or add more Event Scheduler replicas.

## See also

* [Astro executor](/docs/astro/astro-executor)
* [Worker queues](/docs/astro/configure-worker-queues)
* [API server autoscaling](/docs/astro/api-server-autoscaling)
* [Airflow REST API](/docs/astro/airflow-api)
* [Astronomer feature lifecycle](/docs/astro/feature-previews)


# View task-level utilization metrics in the Airflow UI
Source: https://astronomer.io/docs/astro/task-level-metrics

See CPU and memory utilization for individual Dags, tasks, and task instances from a new Resource Metrics tab in the Airflow UI.

Astro can show CPU and memory utilization for individual Dags, tasks, and task instances directly in the Airflow UI. This is different from the Deployment-wide worker, scheduler, and triggerer metrics available on the Astro UI's **Analytics** page. See [View metrics for Astro Deployments](/docs/astro/deployment-metrics).

## Prerequisites

* An Astro Deployment running the [Astro Executor](/docs/astro/astro-executor) on:
  * Hosted execution mode with Astro Runtime 3.1+, or
  * Remote Execution mode on Astro Runtime 3.1+, using a Remote Execution Agent 1.5.0+ and Helm chart 2.0.0+. See [Helm chart versioning](/docs/astro/agent-maintenance-policy#helm-chart-versioning) for the Agent-to-chart compatibility matrix.

## What you'll see

A **Resource Metrics** tab appears in the Airflow UI with three views:

* **Dag**: Aggregated CPU and memory usage across all tasks in a Dag.
* **Task**: CPU and memory usage for a specific task across its recent runs.
* **Task Instance**: CPU and memory usage for a single task run, mapped to the worker that ran it.

The **Resource Metrics** tab supports both light and dark mode. Use it to answer questions like:

* Which tasks are consuming the most CPU or memory?
* Which tasks are likely causing out-of-memory (OOM) errors?
* Which Dags are driving infrastructure cost?
* Which tasks are good candidates for right-sizing or optimization?

## How Astro collects task-level metrics

Astro samples task-level metrics on their own interval, separate from the interval used for Deployment-wide worker metrics. For long-running tasks, samples can lag up to five seconds behind the task.

This sampling shapes what each metric represents:

* CPU values account for all work the task performs, but averaging across the sampling interval smooths very short-lived peaks.
* Memory values reflect usage at each sample, so a spike shorter than the sampling interval on an otherwise idle task doesn't appear.

Astro collects metrics for work that runs in the task slot of an Astro Executor worker, and the **Resource Metrics** tab shows the most recent seven days of data.

## See also

* [View metrics for Astro Deployments](/docs/astro/deployment-metrics)
* [Astro Executor](/docs/astro/astro-executor)
* [Remote Execution Agent maintenance policy](/docs/astro/agent-maintenance-policy)


# Astro Terraform Provider
Source: https://astronomer.io/docs/astro/terraform-provider

Learn about the Astro Terraform Provider and how to work with it.

[Terraform by HashiCorp](https://www.terraform.io/) allows you to manage your infrastructure as code. This means you can programmatically manage resources, user access, and networking using Terraform and Terraform Providers. This provides you with an alternative to the Astro CLI, Astro UI, or Astro API to manage resources.

Terraform providers act as an interface between Terraform and different platforms or services that you might use in your infrastructure implementation. You can use the [Astro Terraform Provider](https://github.com/astronomer/terraform-provider-astro) to programmatically manage your Astro resources.

The Astro Terraform Provider is open source and available through the [Terraform registry](https://registry.terraform.io/providers/astronomer/astro/latest). You can also clone the provider from its [GitHub Repo](https://github.com/astronomer/terraform-provider-astro) if you also want to adapt it for your custom use case, or submit a suggestion with a pull request.

You can read more about the Astro Terraform Provider in the [Terraform Registry](https://registry.terraform.io/providers/astronomer/astro/latest/docs) docs and see example code in the [GitHub Repo](https://github.com/astronomer/terraform-provider-astro/tree/main/examples).

## Terraform Registry Guides

<CardGroup>
  <Card title="Get started with Astro Terraform Provider" icon="terraform" href="https://registry.terraform.io/providers/astronomer/astro/latest/docs/guides/get-started">
    Automate the onboarding of a new team onto Astro by creating and managing a Workspace and Deployment.
  </Card>

  <Card title="Use Import Script to migrate existing resources" icon="astro" href="https://registry.terraform.io/providers/astronomer/astro/latest/docs/guides/import-script">
    Migrate an existing Workspace, API token, and Team into Terraform using the Terraform Import Script.
  </Card>
</CardGroup>


# Transfer a Deployment to a different Workspace
Source: https://astronomer.io/docs/astro/transfer-a-deployment

Transfer a Deployment to another Workspace in the same cluster.

Transferring a Deployment can be helpful when your team needs to change user access to a Deployment. Transferring a Deployment moves all Dags, task history, connections, and other Astro configurations to another Workspace. This process doesn't affect your task scheduling or any currently running tasks.

## Prerequisites

* You must be a Workspace Owner or Operator in both the original Workspace and the target Workspace.
* The Workspaces must be in the same Organization.
* Deployments can't be transferred to a different cluster from the one in which they were created.
* Only the users who are members of the target Workspace can access the Deployment after it is transferred.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Transfer a Deployment to another Workspace

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Deployments**, then select a Deployment.
    2. Click the Deployment's **More actions** menu (⋯), then select **Transfer Deployment**.
    3. Select the target Workspace where you want to transfer the Deployment.
    4. Click **Transfer Deployment**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
    2. Click the **Options** menu and select **Transfer Deployment**.

       <Frame>
         <img alt="Transfer Deployment in options menu" />
       </Frame>
    3. Select the target Workspace where you want to transfer the Deployment.
    4. Click **Transfer Deployment**.
  </Tab>
</Tabs>


# Astro user permissions reference
Source: https://astronomer.io/docs/astro/user-permissions

Learn about Astronomer's RBAC system and how to assign roles to users.

To better protect your data pipelines and cloud infrastructure, Astro provides role-based access control (RBAC) for Organizations and Workspaces. Each Astro user has a Workspace role in each Workspace they belong to, plus a single Organization role. Users can also belong to [Teams](/docs/astro/manage-teams), which apply the same Workspace role to a group of users. RBAC is also available for Deployments through Deployment Admin and custom Deployment roles. For Deployments running Astro Runtime 3.1-12 or later, Dag-level roles provide fine-grained access control for individual Dags within a Deployment. See [Dag-level access control](/docs/astro/dag-level-access-control).

You can also apply roles to API tokens to limit the scope of their actions in CI/CD and automation pipelines. See [Manage Deployments as code](/docs/astro/manage-deployments-as-code).

Astro has hierarchical RBAC. Within a given Workspace or Organization, senior roles have their own permissions in addition to the permissions granted to lower roles. For example, a user or API token with Organization Owner permissions inherits Organization Billing Admin and Organization Member permissions because those roles are lower in the hierarchy.

The Astro role hierarchies in order of inheritance are:

* Organization Owner > Organization Billing Admin > Organization Observe Admin > Organization Observe Member > Organization Member
* Workspace Owner > Workspace Operator > Workspace Author > Workspace Member > Workspace Accessor (Preview)

Additionally, Organization Owners inherit Workspace Owner permissions for all Workspaces in the Organization.

## Organization roles

An Organization role grants a user or API token some level of access to an Astro Organization. The Organization Owner role includes access to all of the Workspaces within that Organization. All users have at least an Organization Member role regardless of whether they belong to a Workspace, however, an API token's access is based on the scope you define for it. For example, you must give an API token an `organization owner` role to perform Organization-level actions or to access the list of all Workspaces in the Organization.

<Info>
  Developer plans are limited to two non Organization Owner users. See
  [pricing](https://www.astronomer.io/pricing/).
</Info>

The following table lists the available Organization roles:

| Permission                                                            | **Member** | **Observe Member** | **Observe Admin** | **Billing Admin** | **Owner** |
| --------------------------------------------------------------------- | ---------- | ------------------ | ----------------- | ----------------- | --------- |
| View Organization details and user membership                         | ✔️         | ✔️                 | ✔️                | ✔️                | ✔️        |
| View lineage metadata in the **Lineage** tab                          | ✔️         | ✔️                 | ✔️                | ✔️                | ✔️        |
| View clusters                                                         | ✔️         | ✔️                 | ✔️                | ✔️                | ✔️        |
| View Data Products, SLAs, connections, and monitors in Observe        | ✔️         | ✔️                 | ✔️                | ✔️                | ✔️        |
| View Observe notification channels and alerts for their Workspaces    |            | ✔️                 | ✔️                | ✔️                | ✔️        |
| Create, update, and delete Data Products                              |            | ✔️                 | ✔️                | ✔️                | ✔️        |
| Create, update, and delete Data Product SLAs                          |            | ✔️                 | ✔️                | ✔️                | ✔️        |
| Create, update, and delete notification channels                      |            |                    | ✔️                | ✔️                | ✔️        |
| Update Organization billing information and settings                  |            |                    |                   | ✔️                | ✔️        |
| View usage for all Workspaces in the **Usage** tab                    |            |                    |                   | ✔️                | ✔️        |
| View Organization-level metrics dashboards on the **Dashboards** page |            |                    |                   | ✔️                | ✔️        |
| Create, update, and delete clusters                                   |            |                    |                   |                   | ✔️        |
| Create a new Workspace                                                |            |                    |                   |                   | ✔️        |
| Workspace Owner permissions to all Workspaces                         |            |                    |                   |                   | ✔️        |
| Update roles and permissions of existing Organization users           |            |                    |                   |                   | ✔️        |
| Invite a new user to an Organization                                  |            |                    |                   |                   | ✔️        |
| Remove a user from an Organization                                    |            |                    |                   |                   | ✔️        |
| Create, update, and delete Organization API tokens                    |            |                    |                   |                   | ✔️        |
| Export audit logs                                                     |            |                    |                   |                   | ✔️        |
| Access, regenerate, and delete single sign-on (SSO) bypass links      |            |                    |                   |                   | ✔️        |
| Create, update, and delete a Team                                     |            |                    |                   |                   | ✔️        |
| Configure environment secrets fetching                                |            |                    |                   |                   | ✔️        |
| Configure IP access                                                   |            |                    |                   |                   | ✔️        |
| Enable and disable AI features for the Organization                   |            |                    |                   |                   | ✔️        |

To manage users in an Organization, see [Manage Organization users](/docs/astro/manage-organization-users). To manage the Organization permissions of your API tokens, see [Organization API tokens](/docs/astro/organization-api-tokens).

### Enhanced Support Access

Enhanced Support Access is enabled by default for all Organizations to ensure faster, more effective assistance from the Astronomer support team. It grants *read-only* Admin access to your Organization’s details, allowing the support team to troubleshoot issues in real time and provide premium-level support. Support does not have access to make any changes to your environment.

You can view any activity by the Astronomer support team by viewing your Organization's [Audit Logs](/docs/astro/audit-logs).

If you have [IP Access List](/docs/astro/ip-access-list) enabled, Astronomer support with Enhanced Support Access can still view your Organization's details.

You can disable this feature at any time by going to the **General Settings** page in your [Organization Settings](https://cloud.astronomer.io/settings/general) page. Click **Edit Details** and change the **Enhanced Support Access** from **Allowed** to **Disallowed**.

## Workspace roles

A Workspace role grants a user or API token some level of access to a specific Workspace. If a user or API token has some level of access to a Workspace, that access applies to all Deployments in the Workspace.

* A **Workspace Accessor** has the fewest permissions in a Workspace. A Workspace Accessor can only see basic information about the Workspace in which they have the role. They can only be granted further permissions through a Deployment role. If you grant a user a Deployment role without first assigning them a Workspace role, they'll automatically be granted the Workspace Accessor role for the Workspace. For more information, see [Customize Deployment roles](/docs/astro/customize-deployment-roles).
* A **Workspace Member** can view the most basic details about Deployments, Dags, tasks, logs, and alerts. Give a user this role if they need to be able to monitor a Dag run or view Deployment health, but they shouldn't make any changes to a Deployment themselves.
* A **Workspace Author** has all of the same permissions as a Workspace Member, plus the ability to update Dag code and run Dags in the Airflow UI, plus limited permissions to configure Deployment-level observability features such as Astro alerts. Give a user this role if they are primarily Dag developers and don't need to manage the environments their Dags run in.
* A **Workspace Operator** has all the same permissions as a Workspace Author, plus the ability to manage Deployment-level configurations, such as environment variables. Give a user this role if they need to manage the environments that Dags run in.
* A **Workspace Owner** has all the same permissions as a Workspace Operator, plus the ability to manage user membership in the Workspace. Give a user this role if they need to administrate membership to the Workspace.

To manage a user's Workspace permissions, see [Manage Workspace users](/docs/astro/manage-workspace-users#add-a-user-to-a-workspace).

## Deployment roles

There are two types of Deployment roles: the default Deployment Admin role and [custom Deployment roles](/docs/astro/customize-deployment-roles).

*Deployment Admin roles* have the same permissions as the [Workspace Operator](#workspace-roles) role but only Deployment-level operations in a specific Deployment. For example, a Deployment Admin can create a Deployment [environment variable](/docs/astro/environment-variables) but, unlike a Workspace Operator, they can't create an [Astro alert](/docs/astro/alerts) because alerts are configured at the Workspace level.

A *custom Deployment role* is a role that your Organization has configured to have specific Deployment-level permissions. For a complete list of available custom Deployment role permissions, see [Custom role permissions reference](/docs/astro/deployment-role-reference).

## Dag roles

Dag roles provide per-Dag access control within a Deployment. Unlike Deployment roles, which apply to all Dags in a Deployment, Dag roles are scoped to specific Dags by tag or Dag ID. Astronomer recommends binding roles using Dag tags so that new Dags with matching tags are automatically covered. There are two default Dag roles:

* **Dag Viewer**: Read-only access to a specific Dag and its resources.
* **Dag Author**: Read, edit, and delete access to a specific Dag and its resources.

You can also create [custom Dag roles](/docs/astro/dag-level-access-control#create-a-custom-dag-role) with granular permissions. For complete setup instructions, see [Dag-level access control](/docs/astro/dag-level-access-control). For a list of available custom Dag role permissions, see [Custom role permissions reference](/docs/astro/deployment-role-reference#dag-scope-permissions).

## Permissions reference

### Workspace role permissions

| Permission                                                                                                                                   | **Workspace Accessor** | **Workspace Member** | **Workspace Author** | **Workspace Operator** | **Workspace Owner** |
| -------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- | -------------------- | -------------------- | ---------------------- | ------------------- |
| View Deployment and Workspace users and teams                                                                                                | ✔️                     | ✔️                   | ✔️                   | ✔️                     | ✔️                  |
| View all Deployments in the Astro UI                                                                                                         | ✔️                     | ✔️                   | ✔️                   | ✔️                     | ✔️                  |
| View Dag metadata in the Astro UI                                                                                                            | ✔️                     | ✔️                   | ✔️                   | ✔️                     | ✔️                  |
| View Dags in the Airflow UI                                                                                                                  |                        | ✔️                   | ✔️                   | ✔️                     | ✔️                  |
| View Airflow task logs                                                                                                                       |                        | ✔️                   | ✔️                   | ✔️                     | ✔️                  |
| View Airflow datasets                                                                                                                        |                        | ✔️                   | ✔️                   | ✔️                     | ✔️                  |
| Create and delete Airflow datasets                                                                                                           |                        |                      | ✔️                   | ✔️                     | ✔️                  |
| View Astro alerts                                                                                                                            |                        | ✔️                   | ✔️                   | ✔️                     | ✔️                  |
| View the **Cluster Activity** tab in the Airflow UI                                                                                          |                        | ✔️                   | ✔️                   | ✔️                     | ✔️                  |
| Use custom plugins from the Airflow UI menu                                                                                                  |                        | ✔️                   | ✔️                   | ✔️                     | ✔️                  |
| Manually trigger Dag and task runs                                                                                                           |                        |                      | ✔️                   | ✔️                     | ✔️                  |
| Pause or unpause a Dag                                                                                                                       |                        |                      | ✔️                   | ✔️                     | ✔️                  |
| Clear/mark a task run or Dag run                                                                                                             |                        |                      | ✔️                   | ✔️                     | ✔️                  |
| Create, update, and delete Astro alerts                                                                                                      |                        |                      | ✔️                   | ✔️                     | ✔️                  |
| View Airflow connections, variables, plugins, providers, pools, and XComs that were created in the Airflow UI                                |                        |                      | ✔️                   | ✔️                     | ✔️                  |
| Create, update, and delete Airflow connections, variables, plugins, providers, pools, and XComs                                              |                        |                      |                      | ✔️                     | ✔️                  |
| Create, update, delete, and assign connections, Airflow variables, and environment variables to Deployments in the Astro Environment Manager |                        |                      |                      | ✔️                     | ✔️                  |
| Update Deployment configurations                                                                                                             |                        |                      |                      | ✔️                     | ✔️                  |
| Create and delete Deployments                                                                                                                |                        |                      |                      | ✔️                     | ✔️                  |
| Create, update, and delete Deployment environment variables                                                                                  |                        |                      |                      | ✔️                     | ✔️                  |
| Create, update, and delete Workspace API tokens                                                                                              |                        |                      |                      |                        | ✔️                  |
| Create, delete, pause, and unpause [hibernation schedules](/docs/astro/deployment-resources#hibernate-a-development-deployment)                   |                        |                      |                      | ✔️                     | ✔️                  |
| Create, update, and delete Deployment API tokens                                                                                             |                        |                      |                      |                        | ✔️                  |
| Update user roles and permissions                                                                                                            |                        |                      |                      |                        | ✔️                  |
| Invite users to a Workspace                                                                                                                  |                        |                      |                      |                        | ✔️                  |
| Assign Teams to or remove from Workspaces                                                                                                    |                        |                      |                      |                        | ✔️                  |
| List, cordon, uncordon, and delete a Remote Execution Agent                                                                                  |                        |                      |                      |                        | ✔️                  |
| List, create, and delete Remote Execution Agent API tokens                                                                                   |                        |                      |                      |                        | ✔️                  |
| View Astro IDE                                                                                                                               |                        | ✔️                   | ✔️                   | ✔️                     | ✔️                  |
| Edit projects in Astro IDE and use AI features                                                                                               |                        |                      | ✔️                   | ✔️                     | ✔️                  |
| Start ephemeral test Deployments in Astro IDE                                                                                                |                        |                      |                      | ✔️                     | ✔️                  |

### Deployment Admin permissions

A Deployment Admin has permissions equivalent to a [Workspace Operator](#workspace-roles), but scoped to a specific Deployment rather than the entire Workspace. A user can be a Deployment Admin for multiple Deployments.

Because Deployment Admin permissions apply only within the assigned Deployment, a Deployment Admin can't perform Workspace-level actions such as creating Astro alerts, managing Workspace API tokens, inviting users, or using the Astro Environment Manager.

Within their assigned Deployment, a Deployment Admin can:

* View Deployment users and teams
* View Dags in the Airflow UI
* View Airflow task logs, datasets, and alerts
* View the **Cluster Activity** tab in the Airflow UI
* Use custom plugins from the Airflow UI menu
* Create and delete Airflow datasets
* Manually trigger Dag and task runs
* Pause or unpause a Dag
* Clear/mark a task run or Dag run
* View, create, update, and delete Airflow connections, variables, plugins, providers, pools, and XComs
* Update Deployment configurations
* Create, update, and delete Deployment environment variables
* Create, delete, pause, and unpause [hibernation schedules](/docs/astro/deployment-resources#hibernate-a-development-deployment)
* Create, update, and delete Deployment API tokens
* List, cordon, uncordon, and delete a Remote Execution Agent
* List, create, and delete Remote Execution Agent API tokens
* View, edit, and start ephemeral test Deployments in Astro IDE

## Relationship between user roles and Team roles

There are two ways to define a user's role in a Workspace:

* Define the individual user role when you [add a user](/docs/astro/manage-workspace-users#add-a-user-to-a-workspace) to a Workspace.
* Assign a Workspace role to a [Team](/docs/astro/manage-teams#add-a-team-to-a-workspace) and add the user to the Team.

If a user has permissions to a Workspace both as an individual and as a member of a Team, then Astronomer recognizes the more privileged role.

For example, if a user belongs to a Workspace as a **Workspace Member**, but also belongs to a Team in the Workspace with **Workspace Owner** privileges, then the user has **Workspace Owner** privileges in the Workspace.


# View clusters
Source: https://astronomer.io/docs/astro/view-clusters

View details about your clusters and cluster-specific configurations.

## View clusters

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Clusters** to view a list of the clusters that are available to your Organization.
    2. Click a cluster to view its information. See the following table for more information about each available information page.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, under **Organization Settings**, click **Clusters** to view a list of the clusters that are available to your Organization.
    2. Click a cluster to view its information. See the following table for more information about each available information page.
  </Tab>
</Tabs>

| Tab name                | Description                                                                                                                                                                                                                                             |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Details                 | General configurations, including the cluster name, IDs, and connectivity options. You can edit the cluster name after creation.                                                                                                                        |
| Workspace Authorization | List of Workspaces that can create Deployments on this cluster. See [Authorize Workspaces to a cluster](/docs/astro/authorize-workspaces-to-a-cluster).                                                                                                      |
| VPC Peering connections | To set up a private connection, you can create a VPC peering connection. VPC peering ensures private and secure connectivity, reduces network transit costs, and simplifies network layouts. See the [Networking overview](/docs/astro/networking-overview). |
| Routes                  | Connection routes used for your peering connections. See the [Networking overview](/docs/astro/networking-overview).                                                                                                                                         |
| Worker Types            | The available worker types for configuring worker queues on your cluster.                                                                                                                                                                               |


# Create and manage Workspace API tokens
Source: https://astronomer.io/docs/astro/workspace-api-tokens

Create and manage Workspace API tokens to automate key Workspace actions, like adding users and creating Deployments.

Use Workspace API tokens to automate Workspace actions such as creating Deployments and managing users as part of your CI/CD pipelines. You need to be a Workspace Owner to manage Workspace API tokens. For an overview of how Astro authenticates API requests, including JWT validation, token lifetimes, and rotation behavior, see [API authentication and token security](/docs/astro/api-authentication).

Using Workspace API tokens, you can automate:

* Creating and updating Deployments using a [Deployment file](/docs/astro/manage-deployments-as-code)
* Adding batches of users to a Workspace in a CI/CD pipeline. See [Add a group of users to Astro using the Astro CLI](/docs/astro/manage-workspace-users#add-a-group-of-users-to-a-workspace-using-the-astro-cli).
* Creating preview Deployments whenever you create a feature branch in your Astro project Git repository.
* Performing Deployment-level actions on any Deployment in a Workspace, such as deploying code or making calls to the Airflow rest API. Workspace API tokens can complete the same actions as Deployment API tokens for any Deployment in the Workspace.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Workspace API token types

There are two ways to use API tokens to interact with a Workspace:

* Create a new Workspace API token that's scoped only to a single Workspace.
* Add an existing [Organization API token](/docs/astro/organization-api-tokens) to a Workspace and grant it Workspace permissions. You can add an Organization API token to multiple Workspaces.

### Create a Workspace API token

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Settings**.

    2. Click **Workspaces** in the sidebar, then select your Workspace.

    3. In the **Access Management** section, click **API Tokens**.

    4. Click **+ Add API Token** > **New Workspace API Token**.

    5. Configure the new Workspace API token:

       * **Name**: The name for the API token.
       * **Description**: Optional. The Description for the API token.
       * **Workspace Role**: The role that the API token can assume. See [User permissions](/docs/astro/user-permissions#workspace-roles).
       * **Expiration**: The number of days that the API token can be used before it expires.

    6. Click **Create API token**. A confirmation screen showing the token appears.

    7. Copy the token and store it in a safe place. You will not be able to retrieve this value from Astro again.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open your Workspace.

    2. Go to **Workspace Settings** > **Access Management** > **API Tokens**.

    3. Click **+ API Token** > **Create Workspace API Token**

    4. Configure the new Workspace API token:

       * **Name**: The name for the API token.
       * **Description**: Optional. The Description for the API token.
       * **Workspace Role**: The role that the API token can assume. See [User permissions](/docs/astro/user-permissions#workspace-roles).
       * **Expiration**: The number of days that the API token can be used before it expires.

    5. Click **Create API token**. A confirmation screen showing the token appears.

    6. Copy the token and store it in a safe place. You will not be able to retrieve this value from Astro again.
  </Tab>
</Tabs>

### Assign an Organization API token to a Workspace

To centralize API token management, you can add an Organization token to a Workspace instead of creating a dedicated Workspace API token. Workspace-scoped API tokens are useful if you want to manage API tokens from the Organization level on a single screen, or you want to use a single API token for multiple Workspaces.

Note that you must have Organization Owner permissions to manage Workspace API tokens at the Organization level.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Settings**.

    2. Click **Workspaces** in the sidebar, then select your Workspace.

    3. In the **Access Management** section, click **API Tokens**.

    4. Click **+ Add API Token** > **Assign Organization API Token**.

    5. In **Organization API Tokens**, select the Organization API token you want to use. In **Workspace Role**, select the permissions that you want the Organization API token to have in the Workspace.

    6. Click **Update API Token**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open your Workspace.

    2. Go to **Workspace Settings** > **Access Management** > **API Tokens**.

    3. Click **+ API Token** > **Add Organization API Token**

    4. In **Organization API Tokens**, select the Organization API token you want to use. In **Workspace Role**, select the permissions that you want the Organization API token to have in the Workspace.

    5. Click **Add**.
  </Tab>
</Tabs>

## Manage Workspace API token access

You can view and manage the roles for a Workspace API token from its access management page.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Settings**.

    2. Click **Workspaces** in the sidebar, then select your Workspace.

    3. In the **Access Management** section, click **API Tokens**.

    4. Click the row for the API token you want to manage.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open your Workspace.

    2. Go to **Workspace Settings** > **Access Management** > **API Tokens**.

    3. Click the row for the API token you want to manage.
  </Tab>
</Tabs>

The token access management page shows the following information and management options:

* **Workspace Role**: View or update the token's Workspace role. Click **Edit** to change the role, then click **Save changes**.
* **Deployment Roles**: View all Deployment role assignments for the token. Click **+ Deployment** to assign the token to a Deployment with a specific role. To edit or remove a Deployment role, open the **More actions** menu (⋯) next to the Deployment.
* **Dag Roles**: View and manage the token's Dag role assignments. Click **+ Dag** to add a Dag role. To edit or remove a Dag role, open the **More actions** menu (⋯) next to the Dag entry. See [Dag-level access control](/docs/astro/dag-level-access-control) for more information about Dag roles.

## Update a Workspace API token

If you delete a Workspace API token, make sure that no existing CI/CD pipelines are using it. After it's deleted, an API token can't be recovered. If you unintentionally delete an API token, create a new one and update any CI/CD workflows that used the deleted API token.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Settings**.

    2. Click **Workspaces** in the sidebar, then select your Workspace.

    3. In the **Access Management** section, click **API Tokens**.

    4. Open the **More actions** menu (⋯) next to your API token, then click **Edit Token**.

    5. Update the name, description, or Workspace role of your token, then click **Update API Token**.

    6. Optional. To delete a Workspace API token, click **Delete API Token**, enter `Delete`, and then click **Yes, Continue**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open your Workspace.

    2. Go to **Workspace Settings** > **Access Management** > **API Tokens**.

    3. Click **Edit** next to your API token.

    4. Update the name, description, or Workspace role of your token, then click **Save Changes**.

    5. Optional. To delete a Workspace API token, click **Delete API Token**, enter `Delete`, and then click **Yes, Continue**.
  </Tab>
</Tabs>

## Delete or remove a Workspace API token

If you delete a Workspace API token or remove an Organization API token from your Workspace, make sure that no existing CI/CD pipelines are using the token. After you delete a Workspace API token, it can't be recovered. If you unintentionally delete an API token, create a new one and update any CI/CD workflows that used the deleted API token.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Settings**.

    2. Click **Workspaces** in the sidebar, then select your Workspace.

    3. In the **Access Management** section, click **API Tokens**.

    4. Open the **More actions** menu (⋯) next to your API token, then click **Delete Token**. Enter `Delete`, then click **Yes, Continue**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open your Workspace.

    2. Go to **Workspace Settings** > **Access Management** > **API Tokens**.

    3. Click **Edit** next to your API token.

    4. If you're removing an Organization API token, click **Remove API token**. If you're deleting a Workspace API token, click **Delete API Token**, enter `Delete`, then click **Yes, Continue**.
  </Tab>
</Tabs>

## Rotate a Workspace API token

Rotating a Workspace API token lets you renew a token without needing to reconfigure its name, description, and permissions. You can also rotate a token if you lose your current token value and need it for additional workflows.

When you rotate a Workspace API token, you receive a new valid token from Astro that can be used in your existing workflows. The previous token value becomes invalid and any workflows using those previous values stop working.

To rotate an Organization API token with Workspace permissions, see [Organization API tokens](/docs/astro/organization-api-tokens).

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, click **Settings**.

    2. Click **Workspaces** in the sidebar, then select your Workspace.

    3. In the **Access Management** section, click **API Tokens**.

    4. Open the **More actions** menu (⋯) next to your API token, then click **Rotate Token**. Type in `ROTATE` to confirm, and click **Yes, Continue**. The Astro UI rotates the token and shows the new token value.

    5. Copy the new token value and store it in a safe place. You won't be able to retrieve this value from Astro again.

    6. In any workflows using the token, replace the old token value with the new value you copied.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, open your Workspace.

    2. Go to **Workspace Settings** > **Access Management** > **API Tokens**.

    3. Click **Edit** next to your API token.

    4. Click **Rotate token**. The Astro UI rotates the token and shows the new token value.

    5. Copy the new token value and store it in a safe place. You won't be able to retrieve this value from Astro again.

    6. In any workflows using the token, replace the old token value with the new value you copied.
  </Tab>
</Tabs>

## Use a Workspace API token with the Astro CLI

To use a Workspace API token with Astro CLI, specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI.

For example, to automate Astro CLI Workspace commands on a Mac, run the following command to set a temporary value for the environment variable:

```sh wrap theme={null}
export ASTRO_API_TOKEN=<your-token>
```

After you set the variable, you can run `astro deployment` and `astro workspace` commands for your Workspace without authenticating yourself to Astronomer. Astronomer recommends storing `ASTRO_API_TOKEN` as a secret before using it to automate the Astro CLI for production workflows.

<Info>If you have both `ASTRO_API_TOKEN` and `ASTRONOMER_KEY_ID`/`ASTRONOMER_KEY_SECRET` set in an environment, your Astro Workspace token takes precedence and is used for all Deployment actions in that Workspace.</Info>

### Use a Workspace API token for CI/CD

You can use Workspace API tokens and the Astro CLI to automate various Workspace and Deployment management actions in CI/CD.

For all use cases, you must make the following environment variable available to your CI/CD environment:

```text wrap theme={null}
ASTRO_API_TOKEN=<your-token>
```

After you set this environment variable, you can run Astro CLI commands from CI/CD pipelines without needing to manually authenticate to Astro. For more information and examples, see [Automate code deploys with CI/CD](/docs/astro/set-up-ci-cd).


# Access Airflow database
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/access-airflow-database

Connect to and manage Airflow metadata databases in APC.

Each Airflow Deployment in Astro Private Cloud (APC) has its own metadata database. APC supports both PostgreSQL and MySQL as the database backend. You can connect to the database to run queries, troubleshoot issues, or perform maintenance tasks.

## Database architecture

<Tabs>
  <Tab title="PostgreSQL">
    Each Deployment has a single PostgreSQL database containing two schemas:

    | Schema    | Purpose                                                                      |
    | --------- | ---------------------------------------------------------------------------- |
    | `airflow` | Airflow metadata, including Dags, task instances, connections, and variables |
    | `celery`  | Celery task results when using the Celery Executor                           |
  </Tab>

  <Tab title="MySQL">
    Each Deployment has a single MySQL database. All Airflow metadata tables and Celery result tables exist in the same database without schema separation.
  </Tab>
</Tabs>

## Retrieve the connection string

Each Deployment stores its database connection string in a Kubernetes secret named `<release-name>-metadata`. Run the following command to retrieve it:

```bash wrap theme={null}
kubectl get secret -n <deployment-namespace> \
  <release-name>-metadata -o jsonpath='{.data.connection}' | base64 -d
```

This returns a connection URI in the following format:

<Tabs>
  <Tab title="PostgreSQL">
    ```text wrap theme={null}
    postgresql://<username>:<password>@<host>:<port>/<database>?sslmode=prefer
    ```
  </Tab>

  <Tab title="MySQL">
    ```text wrap theme={null}
    mysql://<username>:<password>@<host>:<port>/<database>
    ```
  </Tab>
</Tabs>

## Connect to the database

### Use kubectl exec

Run the following command to open an Airflow metadata database shell from the scheduler pod:

```bash wrap theme={null}
kubectl exec -it -n <deployment-namespace> \
  deployment/<release-name>-scheduler -c scheduler -- \
  airflow db shell
```

This command uses the Airflow metadata database connection that is already configured in the Deployment.

### Connect from your local computer

If the database is accessible as a Kubernetes service, you can use `kubectl port-forward` to connect from your local computer. Identify the database service and namespace from the `<host>` field in the connection string, then forward the port:

<Tabs>
  <Tab title="PostgreSQL">
    ```bash wrap theme={null}
    kubectl port-forward -n <database-namespace> \
      svc/<database-service> <local-port>:<database-port>
    ```

    <Note>
      If PgBouncer is enabled, forward port 6543 from the PgBouncer service in the
      Deployment namespace instead.
    </Note>

    Then connect using the credentials from the connection string:

    ```bash wrap theme={null}
    psql -h localhost -p <local-port> -U <username> -d <database>
    ```
  </Tab>

  <Tab title="MySQL">
    ```bash wrap theme={null}
    kubectl port-forward -n <database-namespace> \
      svc/<database-service> <local-port>:<database-port>
    ```

    Then connect using the credentials from the connection string:

    ```bash wrap theme={null}
    mysql -h 127.0.0.1 -P <local-port> -u <username> -p <database>
    ```
  </Tab>
</Tabs>

If the database is an external service (for example, Amazon RDS or Google Cloud SQL), connect directly using the host and port from the connection string instead of port-forwarding.

## Common queries

After you connect to the database, use the following queries to inspect Airflow metadata.

### Dag information

<Tabs>
  <Tab title="Airflow 2.x">
    ```sql wrap theme={null}
    SELECT dag_id, is_active, is_paused, last_parsed_time
    FROM dag ORDER BY dag_id;

    SELECT dag_id, run_id, state, start_date
    FROM dag_run ORDER BY start_date DESC LIMIT 20;
    ```
  </Tab>

  <Tab title="Airflow 3.x">
    ```sql wrap theme={null}
    SELECT dag_id, is_paused, last_parsed_time
    FROM dag ORDER BY dag_id;

    SELECT dag_id, run_id, state, start_date
    FROM dag_run ORDER BY start_date DESC LIMIT 20;
    ```
  </Tab>
</Tabs>

### Task instance status

```sql wrap theme={null}
SELECT dag_id, task_id, state, start_date, try_number
FROM task_instance
ORDER BY start_date DESC LIMIT 50;
```

### Task failures

<Tabs>
  <Tab title="PostgreSQL">
    ```sql wrap theme={null}
    SELECT dag_id, task_id, state, start_date
    FROM task_instance
    WHERE state = 'failed'
      AND start_date > NOW() - INTERVAL '24 hours';
    ```
  </Tab>

  <Tab title="MySQL">
    ```sql wrap theme={null}
    SELECT dag_id, task_id, state, start_date
    FROM task_instance
    WHERE state = 'failed'
      AND start_date > NOW() - INTERVAL 24 HOUR;
    ```
  </Tab>
</Tabs>

## External database configuration

To use an external database instead of the APC-managed database, create the Deployment using the `upsertDeployment` [APC API](/docs/astro-private-cloud/v-2-x/houston-api) mutation with the following fields:

* `skipAirflowDatabaseProvisioning`: Set to `true` so the deployment orchestrator doesn't provision a database for this Deployment.
* `metadataConnection` or `metadataConnectionJson`: The connection string or JSON object pointing to your external database.
* `resultBackendConnection` or `resultBackendConnectionJson`: The connection string or JSON object for the Celery result backend.

For a complete example of the `upsertDeployment` mutation payload with external database configuration, see [Bring your own Airflow database](/docs/astro-private-cloud/v-2-x/multi-db).

## Back up and restore

<Warning>
  For production environments, run backup and restore commands from your local
  computer or a dedicated admin computer. Don't run backup or restore commands
  from inside Airflow containers.
</Warning>

Before you run a backup, estimate the database size and confirm that your local computer has enough free disk space.

<Tabs>
  <Tab title="PostgreSQL">
    ```sql wrap theme={null}
    SELECT
      schemaname,
      pg_size_pretty(SUM(pg_total_relation_size(format('%I.%I', schemaname, tablename)::regclass))) AS total_size
    FROM pg_tables
    GROUP BY schemaname
    ORDER BY SUM(pg_total_relation_size(format('%I.%I', schemaname, tablename)::regclass)) DESC;
    ```
  </Tab>

  <Tab title="MySQL">
    ```sql wrap theme={null}
    SELECT
      table_schema,
      ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS size_mb
    FROM information_schema.tables
    GROUP BY table_schema
    ORDER BY size_mb DESC;
    ```
  </Tab>
</Tabs>

### Back up the database

<Tabs>
  <Tab title="PostgreSQL">
    ```bash wrap theme={null}
    pg_dump -h <host> -p <port> -U <username> -d <database> -n airflow > backup.sql
    ```
  </Tab>

  <Tab title="MySQL">
    To back up a MySQL database from your local computer, use port-forwarding or a direct connection to the database host:

    ```bash wrap theme={null}
    mysqldump -h <host> -P <port> -u <username> -p <database> > backup.sql
    ```
  </Tab>
</Tabs>

### Restore the database

<Tabs>
  <Tab title="PostgreSQL">
    ```bash wrap theme={null}
    psql -h <host> -p <port> -U <username> -d <database> < backup.sql
    ```
  </Tab>

  <Tab title="MySQL">
    To restore a MySQL database from your local computer, use port-forwarding or a direct connection to the database host:

    ```bash wrap theme={null}
    mysql -h <host> -P <port> -u <username> -p <database> < backup.sql
    ```
  </Tab>
</Tabs>

Replace the placeholders with values from the connection string. See [Retrieve the connection string](#retrieve-the-connection-string).

## Security best practices

<Warning>
  Direct database access bypasses Airflow's security model. Use with caution and
  only for troubleshooting or maintenance tasks.
</Warning>

* Use read-only access for monitoring.
* Never expose database ports publicly.
* Use SSL for all connections.
* Rotate credentials regularly.


# Airflow 3 new features
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/af3-features

Learn which Apache Airflow 3 features are supported in Astro Private Cloud 2.0.

Astro Private Cloud can be used with all [supported Airflow versions](https://airflow.apache.org/docs/apache-airflow/stable/installation/supported-versions.html), including Airflow 3. Apache Airflow 3 introduces a suite of new features such as event-driven scheduling, advanced inference execution, a redesigned UI, and high-performance backfills. See the [complete documentation for Airflow](https://airflow.apache.org/docs/apache-airflow/stable/index.html).

## Airflow 3 new features overview

**Backfills**: Backfills solve one of the most common and time-consuming challenges in data orchestration: reliably reprocessing historical or newly available data. Previously, backfills in Airflow had to be triggered from a command-line process that could easily terminate if the session was lost, leaving longer reruns vulnerable to interruption and without robust monitoring. In Airflow 3, backfills become first-class citizens managed by the scheduler itself, enabling asynchronous API triggers, real-time monitoring through the UI, and the ability to pause or cancel jobs mid-run.

**UI Modernization**: Airflow 3 introduces a modern, React-based UI that unifies logs, task details, and dynamic Dag updates.

**Event-driven Scheduling**: Event-driven scheduling in Airflow 3 lets pipelines react to near real-time data changes or external triggers, rather than relying solely on fixed time-based schedules. This means a Dag can automatically start running as soon as a message arrives in the message queue of a supported service.

**Inference Execution**: Airflow 3.0 introduces several enhancements to support AI Inference Execution:

* Ad-hoc scheduling: Airflow 3.0 allows Dags to be run independently of any data interval, which is crucial for supporting inference execution. This feature enables on-demand execution of inference tasks without being constrained by predefined schedules.
* Synchronous Dag execution: The new version supports simultaneous execution of the same Dag, allowing for synchronous inference runs. This is particularly useful for scenarios where multiple inference requests need to be processed concurrently.
* API-triggered execution: Airflow 3.0 introduces the ability to trigger Dags via API calls, enabling multiple instances to be initiated simultaneously for inference tasks. This feature facilitates experimentation and allows for dynamic, near real-time inference processing.
* Event-driven scheduling: The new version supports automatic triggering of Dags based on external events or data availability. This can be particularly useful for inference pipelines that need to react to new data or model updates in near real-time.
* Language-agnostic Task Execution Interface: Airflow 3.x lays the groundwork to run tasks in any language. This enables users to implement inference tasks in the most suitable language for their models, without expensive code refactoring such as using C++, Golang, Java, etc. for more efficient execution.

## Supported Airflow 3 features

The following Airflow 3 features have been tested and are supported with Astro Private Cloud 2.0:

### Core platform

* **Deployment CRUD**: Create, update, and delete Airflow deployments through Astro Private Cloud.
* **All Deployment Types**: Airflow 3 is supported in both unified and split control plane - data plane modes.
* **All executors**: Airflow 3 is supported with Celery Executor and Kubernetes executor.

### Observability and operations

* **Human-in-the-loop (HITL)**: Supported for manual approvals and task-level interventions.
* **Backfills**: Reliably reprocess historical or newly available data with improved performance and visibility.
* **Deadline Alerts / SLA Enhancements**: Improved SLA monitoring and alerting within Airflow 3.
* **Remote Logging**: Integrated support for remote log streaming and storage via the platform.

### User experience and extensibility

* **New UI and Plugins**: New Airflow 3 UI and compatible Astronomer plugins supported.
* **Assets and Asset Decorators**: Support for Airflow 3’s asset-based Dag authoring and tracking model.
* **Event-driven scheduling**: React to near real-time data changes or external triggers.
* **Language-agnostic Task Execution Interface**: Implement tasks in Golang.

### Security and access

* **Airflow 3 RBAC**: Full support for Airflow’s built-in role-based access control model.

### Migration and compatibility

* **Connections, XComs, and Variables Migration**: Migration tooling available to transition from Airflow 2.x to Airflow 3.

## Unsupported Airflow 3 features

The following Airflow 3 features aren't supported with Astro Private Cloud 2.0:

* **Dag versioning**: Tracking and managing versions of Dags across deployments isn't supported.
* **Remote workers**: Triggering or running tasks remotely isn't supported through Astro Private Cloud.


# Astro Private Cloud features
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/apc-features

A summary of key Astro Private Cloud features.

Astro Private Cloud (APC) offers enterprise capabilities to help build, secure, scale, and monitor your self-hosted data platform built around Apache Airflow.

## Run, scale, and optimize Airflow across platforms

* **Kubernetes-Native Deployment**: Astro Private Cloud runs on your own Kubernetes cluster (EKS, GKE, AKS, OpenShift or other). Use Kubernetes for service coordination, communication, and fault tolerance.
* **Run Cross-Cloud and Cross-Region**: Manage Airflow Deployments across multiple Kubernetes clusters, regardless of cloud provider or region.
* **Airflow Deployment Lifecycle Management**: Create, update, and delete Airflow Deployments.
* **In-Place Upgrades**: Update to the latest Airflow version without costly downtime or lengthy migration processes. Manage upgrades independently from platform updates.
* **Airflow Rollbacks**: Roll back to earlier Airflow versions, and track update and rollback history.
* **Multiple Executor Support**: Choose between Kubernetes Executor for dynamic task isolation or Celery Executor for distributed task processing. Automatically scale workers based on workload demands.
* **Resource Management**: Dynamically scale resources per Airflow Deployment. Adjust CPU, memory, and worker counts through the UI to match your workload requirements.
* **Private Docker Registry**: Each Airflow Deployment maintains its own Docker image with custom libraries and environment settings. Images are automatically built and pushed to your private registry.
* **Deployment Isolation**: Each Airflow Deployment runs in its own Kubernetes namespace, providing data isolation and protecting against noisy neighbors.
* **Flexible Dag Deployment Options**: Deploy Airflow Dags using the following options:
  * Image-based deploys (Dags baked into container image)
  * Dag-only deploys (Dag bundles pushed to running Airflow Deployments), git-sync per Pod (no shared volume)
  * Git-sync relay (local repository clones reduce external git load)
  * Object storage (for example S3/GCS/Azure Blob)
  * RWX storage classes
  * NFS shared volume (enables a single shared clone of your Dag repository per namespace, giving all Airflow components (scheduler, webserver, workers, triggerer) a consistent Dag view, reducing Pod cold-start time, and minimizing network traffic, disk usage, and credential copies)
* **Environment Deletion Cleanup**: Automated infrastructure and database cleanup when Airflow Deployments are deleted.
* **Extensible Platform**: Bring your own Postgres or MySQL database; Bring your own container registry, ingress controller, Elasticsearch; export logs and metrics to tools of your choice.

## Comprehensive monitoring and observability

* **Centralized Airflow and Platform Performance Dashboards**: Pre-built Grafana dashboards visualize Airflow and platform metrics. Create custom dashboards to meet specific monitoring needs.
* **Centralized Metrics**: Track scheduler performance, task success rates, and resource utilization. Centralized time-series metrics collection in Prometheus for both platform and Deployment-level monitoring.
* **Alert Manager**: Configure email alerts based on platform and infrastructure health metrics. Get notified of issues such as slow schedulers or resource constraints.
* **Centralized Logging**: Elasticsearch provides powerful log search across all Airflow Deployments. Vector automatically collects and indexes Airflow logs.

## Security and governance for highly-sensitive workloads

* **Deploy in Air-Gapped or Restricted Network Environments**: Run Astro Private Cloud entirely within your own environment. Maintain full control over data location and network security boundaries.
* **Tenant Isolation**: Run each Airflow Deployment in its own Kubernetes namespace or cluster with:
  * Resource isolation: CPU, memory, and storage limits per Deployment
  * Network isolation: Network policies to control traffic between Deployments
  * RBAC isolation: Service accounts and roles scoped to specific namespaces
* **Role-Based Access Control (RBAC)**: Granular access control at Platform, Workspace, and Airflow Deployment levels. Three role types (Admin, Editor, Viewer) map directly to Airflow RBAC permissions.
* **Enterprise SSO Integration**: Integrate with major identity providers including Okta, Auth0, Microsoft Entra ID (Azure AD), Google OAuth, and AWS Cognito. Support for OpenID Connect (OIDC) and custom OAuth flows.
* **SCIM Provisioning**: Automatically provision and deprovision users and teams based on your identity provider. Maintain centralized user management and access control.
* **Service Accounts**: Create Deployment-level or Workspace-level service accounts for CI/CD pipelines and API automation. Generate API keys with specific permission scopes.
* **Network Security**: NGINX ingress controller enforces authentication and manages traffic routing out-of-the-box; option to bring-your-own ingress controller. TLS encryption for all communications between components.
* **Secrets Management**: Securely store identity provider credentials and API secrets as encrypted Kubernetes secrets.
* **CVE SLAs**: All Astro Private Cloud container images are security hardened and come with CVE mitigation and remediation SLAs.
* **Run without Cluster Permissions**: Install the Astro Private Cloud platform and Airflow Deployments with namespace permissions only.

## Developer productivity and platform automation

* **Newest Airflow Features**: Astro Private Cloud supports Airflow 2 and Airflow 3.
* **APC API**: Automate all platform operations with a GraphQL API.
* **Astro CLI**: Install, run, and test Airflow from your command line. Launch a local Airflow stack using Docker for development and testing of Dags, hooks, and operators.
* **Astro Private Cloud UI**: Modern web-based interface to create and manage Workspaces and Airflow Deployments. Scale resources up or down per Airflow Deployment, invite users, and monitor Airflow logs from a centralized dashboard.
* **CI/CD Integration**: Seamlessly integrate with popular CI/CD tools including GitHub Actions, GitLab, Jenkins, CircleCI, and AWS CodeBuild. Use service accounts to authenticate and automate Deployments.
* **Dag-Only Deploys**: A push-based service to update Dags in running Airflow Deployments without rebuilding container images or requiring shared volumes, enabling rapid iteration and Deployment of Dags.
* **Airflow Registry**: Discover over 1,500 integrations to accelerate workflow development. See [Airflow Registry](https://airflow.apache.org/registry/).

## Committer-led support

* **24x7x365 Support**: Access to the world’s leading Airflow experts and committers.
* **Education, Enablement, and Certification**: Build Airflow expertise across your organization with diverse training and certification options.


# Astro Private Cloud documentation
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/astro-private-cloud-overview

Documentation for how to run Apache Airflow® at enterprise scale with Astro Private Cloud.

<Info>Astro Private Cloud is a commercial product for running [Apache Airflow®](https://airflow.apache.org/) on your own Kubernetes clusters. Using Astronomer tooling, you get enterprise-grade security, scalability, and control over your Airflow experience.</Info>

## Astro Private Cloud is your self-hosted enterprise data platform

Astro Private Cloud is a self-hosted data orchestration and scheduling platform built around Apache Airflow, designed to run on your own Kubernetes clusters.

Astro Private Cloud gives you a hardened Airflow distribution plus features that enable reliable operations at scale:

* Centralized monitoring, logging and optimization of Airflow performance and infrastructure consumption
* Airflow deployment rollbacks
* Management of multiple Airflow deployments across clusters and cloud providers
* CI/CD integrations
* Role-based access control
* Support with defined CVE SLAs

With Astro Private Cloud, your data engineering teams can focus on building Dags and not worry about platform performance or availability, while platform teams get a repeatable, reliable, Kubernetes-native way to deploy and scale Airflow across environments.

The following diagram illustrates how you can run Astro Private Cloud in your environment:

<Frame>
  <img alt="Astro Private Cloud Overview" />
</Frame>


# Backfill permissions
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/backfill-permissions

Configure user permissions for Apache Airflow backfill operations on Astro Private Cloud.

A backfill reruns a Dag for a historical date range. On Astro Private Cloud (APC), the APC API issues a JWT for each user when they access the Airflow UI or API. The JWT carries the user's mapped Airflow role and the corresponding permissions. Airflow enforces backfill access against those permissions.

This page describes the APC-specific role mapping and how to grant, revoke, and audit backfill access. For general Airflow concepts, CLI flags, and UI behavior, see the related Astronomer Learn guides:

* [Rerun Airflow Dags and tasks (Airflow 2.x)](/docs/learn/2.x/rerunning-dags#backfill)
* [Rerun Airflow Dags and tasks (Airflow 3.x)](/docs/learn/rerunning-dags#backfill)

For general user role assignment outside the backfill scope, see [Manage users on Astro Private Cloud](/docs/astro-private-cloud/v-2-x/manage-platform-users) and the [User roles and permissions reference](/docs/astro-private-cloud/v-2-x/role-permission-reference).

## Permission model

### APC API deployment role to Airflow role

When a user opens the Airflow UI or API for a Deployment, the APC API maps the user's deployment-level role to an Airflow role and encodes the matching permission set in the JWT.

| APC API role        | Airflow role | Backfill access                    |
| ------------------- | ------------ | ---------------------------------- |
| `DEPLOYMENT_VIEWER` | Viewer       | `read`                             |
| `DEPLOYMENT_EDITOR` | User         | `read`, `create`, `edit`, `delete` |
| `DEPLOYMENT_ADMIN`  | Admin        | `read`, `create`, `edit`, `delete` |

<Note>
  A user with `WORKSPACE_ADMIN` on the parent Workspace, or `SYSTEM_ADMIN` at the system level, inherits Admin-equivalent backfill access on every Deployment in scope. You don't need to grant a deployment-level role on top.
</Note>

### Backfill actions

The `backfill` resource supports four actions.

| Action   | Description                                                 |
| -------- | ----------------------------------------------------------- |
| `read`   | View backfill status and history                            |
| `create` | Create new backfills                                        |
| `edit`   | Modify an existing backfill, such as pausing or resuming it |
| `delete` | Cancel a backfill                                           |

### How permission changes propagate

The APC API issues a JWT when the user accesses the Airflow UI. The JWT lifetime is controlled by the `jwt.authDuration` APC config and defaults to 24 hours. A role change takes effect when the APC API issues a new JWT, typically on the user's next sign-in or token refresh. Until then, the existing JWT continues to grant the previous permissions, so a role change can take up to 24 hours to reach an active session. To force the change immediately, have the user sign out and sign back in.

## Prerequisites

* The user has `DEPLOYMENT_EDITOR` or `DEPLOYMENT_ADMIN` on the target Deployment, or an inherited role from the Workspace or system level.
* The Dag exists and is unpaused.
* The target date range is valid for the Dag's schedule.

## Trigger a backfill

The trigger mechanism depends on the Airflow version running in your Deployment.

<Tabs>
  <Tab title="Airflow 2.x">
    Airflow 2 doesn't expose a backfill action in the UI. Use the Airflow CLI from a machine that can reach the Deployment.

    ```bash wrap theme={null}
    airflow dags backfill \
      --start-date 2026-01-01 \
      --end-date 2026-01-31 \
      --reset-dagruns \
      <dag-id>
    ```

    The `--reset-dagruns` flag deletes existing backfill-related Dag runs in the range and starts a fresh set. To rerun only failed tasks instead, use `--rerun-failed-tasks`. For a walkthrough with examples, see [Rerun Airflow Dags and tasks](/docs/learn/2.x/rerunning-dags#backfill).
  </Tab>

  <Tab title="Airflow 3.x">
    You can trigger an Airflow 3 backfill from the UI, the CLI, or the REST API.

    **Use the Airflow UI**

    <Steps>
      <Step title="Open the Dag's details">
        Navigate to the Dag in the Airflow UI and open its details page.
      </Step>

      <Step title="Trigger a backfill">
        Click **Trigger**, then select **Backfill** in the dialog.
      </Step>

      <Step title="Configure the run">
        Set the start date, end date, and reprocessing behavior, then submit.
      </Step>
    </Steps>

    **Use the Airflow CLI**

    ```bash wrap theme={null}
    airflow backfill create \
      --dag-id <dag-id> \
      --from-date 2026-01-01 \
      --to-date 2026-01-31 \
      --reprocess-behavior failed
    ```

    The `--reprocess-behavior` flag accepts `none`, `failed`, or `completed`. For a walkthrough with examples, see [Rerun Airflow Dags and tasks](/docs/learn/rerunning-dags#backfill).

    **Use the Airflow REST API**

    ```bash wrap theme={null}
    curl -X POST \
      "https://<deployment-url>/api/v2/backfills" \
      -H "Authorization: Bearer <token>" \
      -H "Content-Type: application/json" \
      -d '{
        "dag_id": "<dag-id>",
        "from_date": "2026-01-01T00:00:00Z",
        "to_date": "2026-01-31T00:00:00Z",
        "reprocess_behavior": "failed"
      }'
    ```

    This endpoint requires `backfill.create`, which `DEPLOYMENT_EDITOR` and `DEPLOYMENT_ADMIN` provide. For automated backfills triggered by CI or scheduled jobs, use a Deployment service account with `DEPLOYMENT_EDITOR` rather than a personal user token.
  </Tab>
</Tabs>

## Manage backfill access

Backfill access uses the standard APC role-assignment flow. For the full reference, see [Manage users on Astro Private Cloud](/docs/astro-private-cloud/v-2-x/manage-platform-users). The examples in this section show the calls scoped to backfill use cases.

### Grant or change a role

<Tabs>
  <Tab title="APC API">
    Use the `deploymentAddUserRole` GraphQL mutation to add a user, or `deploymentUpdateUserRole` to change an existing user's role.

    ```graphql wrap theme={null}
    mutation {
      deploymentAddUserRole(
        deploymentId: "<deployment-id>"
        email: "<user-email>"
        role: DEPLOYMENT_EDITOR
      ) {
        id
      }
    }
    ```

    Use `role: DEPLOYMENT_VIEWER` to restrict a user to read-only access instead.
  </Tab>

  <Tab title="Astro CLI">
    ```bash wrap theme={null}
    astro deployment user add \
      --deployment-id=<deployment-id> \
      --email=<user-email> \
      --role=DEPLOYMENT_EDITOR
    ```

    To change an existing user's role, use `astro deployment user update` with `--deployment-id` and `--role`.
  </Tab>
</Tabs>

### Restrict to read-only access

Use the same APC API or Astro CLI commands with `role: DEPLOYMENT_VIEWER` (or `--role=DEPLOYMENT_VIEWER`). The user can view backfill status and history but can't create, modify, or cancel backfills.

### Revoke access

To remove a user from the Deployment entirely, use the APC `deploymentRemoveUserRole` mutation or `astro deployment user remove`.

<Tabs>
  <Tab title="APC API">
    Use the `deploymentRemoveUserRole` mutation to remove a user from a Deployment entirely:

    ```graphql wrap theme={null}
    mutation {
      deploymentRemoveUserRole(
        deploymentId: "<deployment-id>"
        email: "<user-email>"
      ) {
        id
      }
    }
    ```

    After the role binding is removed, the user retains their previous access until their JWT expires (up to 24 hours by default).
  </Tab>

  <Tab title="Astro CLI">
    ```bash wrap theme={null}
    astro deployment user remove \
      --deployment-id=<deployment-id> \
      --email=<user-email>
    ```
  </Tab>
</Tabs>

After the role binding is removed, the user retains their previous backfill capabilities until their JWT expires. See [How permission changes propagate](#how-permission-changes-propagate).

## Monitor backfills

### Use the Airflow UI

Backfill runs appear in the Dag's run list with a `backfill` run-type label, alongside scheduled and manually triggered runs.

### Query the metadata database

Backfill Dag runs are stored in the `dag_run` table with `run_type = 'backfill'`.

```sql wrap theme={null}
SELECT dag_id, run_type, state, start_date, end_date
FROM dag_run
WHERE run_type = 'backfill'
ORDER BY start_date DESC;
```

<Note>
  Direct access to the metadata database is typically restricted to platform operators on APC. If you don't have direct access, use the Airflow UI or the Airflow REST API instead.
</Note>

## Troubleshoot

### Access denied when creating a backfill

The user has `DEPLOYMENT_VIEWER`, which only grants `backfill.read`. Promote them with `deploymentUpdateUserRole`:

Use the `deploymentUpdateUserRole` mutation to change an existing user's role on a Deployment, for example to promote a `DEPLOYMENT_VIEWER` to `DEPLOYMENT_EDITOR`:

```graphql wrap theme={null}
mutation {
  deploymentUpdateUserRole(
    deploymentId: "<deployment-id>"
    email: "<user-email>"
    role: DEPLOYMENT_EDITOR
  ) {
    id
  }
}
```

### Backfills aren't visible to a user

The user has no role on the Deployment. Add them with at least `DEPLOYMENT_VIEWER`. If the user holds `WORKSPACE_VIEWER` and still can't see the Deployment, confirm the Deployment is in the same Workspace.

### A role change hasn't taken effect

The user is still presenting their existing JWT, which keeps the previous permissions until it expires. The default JWT lifetime is 24 hours. Have the user sign out and sign back in to force APC to issue a fresh token. If they authenticate through SSO, ensure the IdP session is also refreshed.

## Best practices

* Assign `DEPLOYMENT_EDITOR` to operators who run backfills regularly. Reserve `DEPLOYMENT_ADMIN` for users who also manage Deployment configuration.
* Assign `DEPLOYMENT_VIEWER` to stakeholders who only need to monitor Dag runs.
* Use a Deployment service account with `DEPLOYMENT_EDITOR` for automated backfills triggered from CI or scheduled jobs, rather than a personal user token.
* Audit backfill activity by querying `dag_run.run_type = 'backfill'` or by reviewing the Dag's run list.
* Test large historical backfills in a non-production Deployment first.
* Set worker concurrency and Dag-level `max_active_runs` to limit the load a backfill places on the Deployment.


# Breaking changes and removals
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/breaking-changes-removals

Breaking changes and removals in Astro Private Cloud 2.0, their user impact, and migration steps.

Astro Private Cloud (APC) 2.0 restructures, renames, and removes several Helm values. This page outlines each change, its user impact, and migration steps.

<Note>
  If you are upgrading from APC 1.x, the changes below represent the incremental differences from 1.x to 2.0. If you are upgrading from 0.37.x directly to 2.0, the [1.0 breaking changes](/docs/astro-private-cloud/v-1-x/breaking-changes-removals) also apply.
</Note>

## Feature flag restructuring (boolean → nested)

### Background

In APC 2.0, scattered `global.*` boolean flags are reorganized into domain-grouped structures with a consistent `.enabled` pattern. This improves consistency and makes the configuration schema easier to extend.

### Affected keys

| Old Path                                     | New Path                                                                         |
| -------------------------------------------- | -------------------------------------------------------------------------------- |
| `global.rbacEnabled`                         | `global.rbac.enabled`                                                            |
| `global.sccEnabled`                          | `global.scc.enabled`                                                             |
| `global.openshiftEnabled`                    | `global.openshift.enabled`                                                       |
| `global.networkNSLabels`                     | `global.networkNSLabels.enabled`                                                 |
| `global.namespaceFreeFormEntry`              | `global.namespaceManagement.namespaceFreeFormEntry.enabled`                      |
| `global.taskUsageMetricsEnabled`             | `global.metricsReporting.taskUsageMetrics.enabled`                               |
| `global.deployRollbackEnabled`               | `global.deploymentLifecycle.deployRollback.enabled`                              |
| `global.podDisruptionBudgetsEnabled`         | `global.podDisruptionBudgets.enabled`                                            |
| `global.postgresqlEnabled`                   | `global.postgresql.enabled`                                                      |
| `global.prometheusPostgresExporterEnabled`   | `global.prometheusPostgresExporter.enabled`                                      |
| `global.manualNamespaceNamesEnabled`         | `global.namespaceManagement.manualNamespaceNames.enabled`                        |
| `global.enablePerHostIngress`                | `global.perHostIngress.enabled`                                                  |
| `global.enableArgoCDAnnotation`              | `global.argoCD.annotation.enabled`                                               |
| `global.disableManageClusterScopedResources` | `global.manageClusterScopedResources.enabled` (inverted: `false` becomes `true`) |
| `global.astronomerEnabled`                   | `global.astronomer.enabled`                                                      |
| `global.nginxEnabled`                        | `global.nginx.enabled`                                                           |
| `global.alertmanagerEnabled`                 | `global.alertmanager.enabled`                                                    |
| `global.grafanaEnabled`                      | `global.grafana.enabled`                                                         |
| `global.kubeStateEnabled`                    | `global.kubeState.enabled`                                                       |
| `global.prometheusEnabled`                   | `global.prometheus.enabled`                                                      |
| `global.elasticsearchEnabled`                | `global.elasticsearch.enabled`                                                   |
| `global.vectorEnabled`                       | `global.daemonsetLogging.enabled`                                                |
| `global.fluentdEnabled`                      | `global.daemonsetLogging.enabled`                                                |

### Subtree moves

| Old Path                           | New Path                                      |
| ---------------------------------- | --------------------------------------------- |
| `global.features.namespacePools.*` | `global.namespaceManagement.namespacePools.*` |
| `global.dagOnlyDeployment.*`       | `global.deployMechanisms.dagOnlyDeployment.*` |
| `global.loggingSidecar.*`          | `global.logging.loggingSidecar.*`             |

### Impact

* APC 2.0 silently ignores any `values.yaml` overrides that use the old key paths, causing values to fall back to chart defaults.

### Required action

Run the appropriate migration script before upgrading:

* From 1.x: [`bin/migrate-helm-chart-values-1x-to-2x.py`](https://github.com/astronomer/astronomer/blob/release-2.0/bin/migrate-helm-chart-values-1x-to-2x.py)
* From 0.37.x: [`bin/migrate-helm-chart-values-037x-to-2x.py`](https://github.com/astronomer/astronomer/blob/release-2.0/bin/migrate-helm-chart-values-037x-to-2x.py)

## APC API config deployment flag restructuring

### Background

In addition to the `global.*` feature flags, the APC API configuration flags under `astronomer.houston.config.deployments` have also been restructured into domain-grouped nested paths. The migration scripts handle these automatically.

### Restructured keys (boolean → nested)

| Old key                               | New path                                                         |
| ------------------------------------- | ---------------------------------------------------------------- |
| `dagProcessorEnabled`                 | `airflowComponents.dagProcessor.enabled`                         |
| `triggererEnabled`                    | `airflowComponents.triggerer.enabled`                            |
| `configureDagDeployment`              | `deployMechanisms.configureDagDeployment.enabled`                |
| `gitSyncDagDeployment`                | `deployMechanisms.gitSyncDagDeployment.enabled`                  |
| `nfsMountDagDeployment`               | `deployMechanisms.nfsMountDagDeployment.enabled`                 |
| `enableListAllRuntimeVersions`        | `runtimeManagement.listAllRuntimeVersions.enabled`               |
| `enableUpdateDeploymentImageEndpoint` | `deploymentImagesRegistry.updateDeploymentImageEndpoint.enabled` |
| `grafanaUIEnabled`                    | `metricsReporting.grafana.enabled`                               |
| `hardDeleteDeployment`                | `deploymentLifecycle.hardDeleteDeployment.enabled`               |
| `logHelmValues`                       | `logHelmValues.enabled`                                          |
| `manualReleaseNames`                  | `namespaceManagement.manualReleaseNames.enabled`                 |

All paths above are relative to `astronomer.houston.config.deployments`.

### Relocated keys

| Old key                                | New path                                                  |
| -------------------------------------- | --------------------------------------------------------- |
| `pgBouncerResourceCalculationStrategy` | `databaseManagement.pgBouncerResourceCalculationStrategy` |
| `serviceAccountAnnotationKey`          | `deploymentImagesRegistry.serviceAccountAnnotationKey`    |

### Deleted keys

| Deleted key                                    | Reason                                                                                                                        |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `astroUnitsEnabled`                            | Astro Units resource strategy replaced by [config governance](/docs/astro-private-cloud/v-2-x/config-governance) for `deployments` |
| `resourceProvisioningStrategy`                 | Astro Units resource provisioning strategy removed                                                                            |
| `maxPodAu`                                     | Obsolete resource capacity setting                                                                                            |
| `upsertDeploymentEnabled`                      | Deprecated deployment upsert flag                                                                                             |
| `canUpsertDeploymentFromUI`                    | Deprecated UI upsert flag                                                                                                     |
| `enableSystemAdminCanCreateDeprecatedAirflows` | Legacy Airflow version creation flag removed                                                                                  |

### Impact

* APC 2.0 silently ignores any `values.yaml` overrides that use the old key paths under `astronomer.houston.config.deployments`, causing values to fall back to APC API defaults.

### Required action

Run the appropriate migration script before upgrading. The scripts handle both `global.*` and `astronomer.houston.config.deployments.*` key restructuring.

## Fluentd replaced by Vector (0.37.x → 2.0 only)

<Note>
  This change was introduced in APC 1.0. If you are upgrading from 1.x, Fluentd was already replaced and no action is needed.
</Note>

### Background

APC 2.0 uses *Vector* for log collection instead of Fluentd. The migration script renames the top-level `fluentd` key to `vector`, preserving resource requests and limits.

### Impact

* Resource values (CPU, memory) carry over automatically.
* Custom Fluentd configuration — such as custom pipelines, filters, output plugins, or parser definitions — doesn't translate to Vector format.

### Required action

If you have custom Fluentd configuration:

<Steps>
  <Step title="Document your Fluentd customizations" />

  <Step title="Recreate your customizations in Vector format">
    Refer to the [Vector documentation](https://vector.dev/docs/) for guidance.
  </Step>

  <Step title="Update the Vector configuration">
    Update the `vector` section in your migrated values file with the new Vector-compatible configuration.
  </Step>
</Steps>

## Kibana removed (0.37.x → 2.0 only)

<Note>
  This change was introduced in APC 1.0. If you are upgrading from 1.x, Kibana was already removed.
</Note>

### Background

APC 2.0 no longer includes Kibana. The migration script deletes the top-level `kibana` section.

### Impact

* The `kibana.<BASEDOMAIN>` endpoint is no longer available after upgrade.
* Elasticsearch remains available for log storage and can be queried directly or via external tools.

### Required action

If you rely on Kibana for log viewing, set up an alternative log viewing solution (such as Grafana with Loki, or direct Elasticsearch queries) before upgrading.

## Prometheus blackbox exporter removed (0.37.x → 2.0 only)

<Note>
  This change was introduced in APC 1.0. If you are upgrading from 1.x, the blackbox exporter was already removed.
</Note>

### Background

APC 2.0 no longer includes the Prometheus blackbox exporter. The migration script deletes the top-level `prometheus-blackbox-exporter` section.

### Impact

If you rely on blackbox probing for uptime monitoring of platform services, that monitoring will stop after the upgrade.

### Required action

Set up alternative uptime monitoring before upgrading if you depend on blackbox exporter probes.

## PgBouncer port changed: 5432 → 6543 (0.37.x → 2.0 only)

<Note>
  This change was introduced in APC 1.0. If you are upgrading from 1.x, the port was already changed.
</Note>

### Background

The default PgBouncer service port changes from `5432` to `6543` to avoid conflict with the PostgreSQL default port.

### Impact

External services or scripts connecting to PgBouncer on port 5432 will fail to connect after the upgrade.

### Required action

Update any external services, connection strings, or scripts that reference the PgBouncer port. If you need to keep port 5432, set `global.pgbouncer.servicePort: "5432"` in your override file after migration.

## PgBouncer secret key renamed (0.37.x → 2.0 only)

<Note>
  This change was introduced in APC 1.0. If you are upgrading from 1.x, this key was already renamed.
</Note>

### Background

The PgBouncer configuration key `global.pgbouncer.krb5ConfSecretName` is renamed to `global.pgbouncer.secretName`. The migration script handles this automatically.

### Impact

The underlying Kubernetes Secret is unchanged — only the Helm values key name changes.

### Required action

Verify that the value carried over correctly after migration.

## NATS JetStream enabled by default (0.37.x → 2.0 only)

<Note>
  This change was introduced in APC 1.0. If you are upgrading from 1.x, JetStream is already enabled.
</Note>

### Background

In 0.37.x, `global.nats.jetStream.enabled` defaults to `false`. In 2.x, it defaults to `true`.

### Impact

JetStream requires persistent storage. Ensure your cluster has a storage class available for JetStream volumes.

### Required action

Verify that your cluster supports persistent volumes for NATS JetStream. If you need to keep JetStream disabled, set `global.nats.jetStream.enabled: false` in your override file after migration.

## Deleted keys (0.37.x → 2.0 only)

APC 2.0 removes the following keys. If you are upgrading from 1.x, 1.0 already removed them.

| Deleted Key                                 | Reason                                               |
| ------------------------------------------- | ---------------------------------------------------- |
| `global.singleNamespace`                    | Single-namespace mode no longer supported            |
| `global.veleroEnabled`                      | Velero integration removed from chart                |
| `global.enableHoustonInternalAuthorization` | Internal authorization replaced by mode-based gating |
| `global.nodeExporterSccEnabled`             | Node exporter SCC no longer needed                   |
| `global.stan`                               | NATS Streaming replaced by NATS JetStream            |
| `tags.stan`                                 | NATS Streaming tag removed                           |
| `stan` (top-level)                          | NATS Streaming deployment removed                    |

## New keys added with defaults

The migration script adds the following keys with default values if they aren't already present. Review each key and override the default if it doesn't match your environment.

| Key Path                              | Default Value | Description                                                                           |
| ------------------------------------- | ------------- | ------------------------------------------------------------------------------------- |
| `global.authHeaderSecretName`         | `~` (null)    | Kubernetes secret for cross-plane authentication — set if running in multi-plane mode |
| `global.plane.mode`                   | `"unified"`   | Platform operating mode: `control`, `data`, or `unified`                              |
| `global.plane.domainPrefix`           | `""`          | Cluster identifier prefix for multi-plane DNS                                         |
| `global.podLabels`                    | `{}`          | Labels applied to every Pod                                                           |
| `global.logging.provider`             | `~` (null)    | Logging provider identifier                                                           |
| `nats.init.resources.requests.cpu`    | `"75m"`       | NATS init container CPU request                                                       |
| `nats.init.resources.requests.memory` | `"30Mi"`      | NATS init container memory request                                                    |
| `nats.init.resources.limits.cpu`      | `"250m"`      | NATS init container CPU limit                                                         |
| `nats.init.resources.limits.memory`   | `"100Mi"`     | NATS init container memory limit                                                      |


# Configure CI/CD on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/ci-cd

Deploy Apache Airflow images and Dags using CI/CD pipelines.

Deploy Airflow images and Dags to Astro Private Cloud using CI/CD pipelines. This guide covers deployment options via CLI, API, and common CI/CD platforms.

## Benefits of CI/CD for Airflow deployments

Deploying Dags and other changes via CI/CD workflows provides:

* **Streamlined development**: Deploy new and updated Dags efficiently across team members.
* **Faster error response**: Decrease maintenance costs and respond quickly to failures.
* **Improved code quality**: Enforce continuous automated testing to protect production Dags.

## Deployment methods

Astro Private Cloud supports multiple deployment methods. The Astro CLI approach is recommended for most use cases due to its simplicity.

### CLI deployment

The Astro CLI provides the simplest way to deploy to Astro Private Cloud from CI/CD pipelines.

Build and deploy an image:

```bash wrap theme={null}
astro deploy <DEPLOYMENT-ID>
```

Deploy Dags only:

```bash wrap theme={null}
astro deploy <DEPLOYMENT-ID> --dags
```

Deploy a pre-built image:

```bash wrap theme={null}
astro deploy <DEPLOYMENT-ID> \
  --image-name quay.io/myorg/airflow:v1.2.3 \
  --remote \
  --runtime-version 12.1.0
```

<Note>
  This command calls the `updateDeploymentImage` mutation of the APC API, which is disabled by default. See [Enable the update deployment image endpoint](/docs/astro-private-cloud/v-2-x/deploy-git-sync#enable-the-update-deployment-image-endpoint).
</Note>

The following optional flags are available for `astro deploy`:

* `--dags`: Deploy only your `dags` folder. Works only if Dag-only deploys are enabled for the Deployment.
* `--image-name <custom-image>`: The name of a pre-built custom Docker image to use with your project. The image must be available on your local machine. If specified, building the image is skipped.
* `--remote`: Directly point the Deployment to the remote image and skip pushing the image. Use with `--image-name`.
* `--runtime-version <version>`: Specify the Runtime version of your image. Use with `--image-name`.
* `--force`: Force deploy even if your project contains errors or uncommitted changes. Use with caution in CI/CD pipelines, as it bypasses the safeguard that ensures only committed code is deployed.
* `--description "<text>"`: Attach a description to a code deploy for traceability. If not provided, the system automatically assigns a default description based on deploy type.

### API deployment

For advanced automation scenarios, you can use the APC API `upsertDeployment` mutation to deploy a pre-built image to a Deployment. This approach is useful when you need to integrate with systems that can't use the Astro CLI directly.

This approach is useful when you need to integrate with systems that can't use the Astro CLI directly.

```graphql wrap theme={null}
mutation {
  upsertDeployment(
    workspaceUuid: "<workspace-uuid>"
    clusterId: "<cluster-id>"
    releaseName: "my-deployment"
    image: "quay.io/myorg/airflow:v1.2.3"
    runtimeVersion: "12.1.0"
    deployRevisionDescription: "CI/CD Pipeline Deploy"
  ) {
    id
    status
  }
}
```

The mutation accepts the following fields:

* `workspaceUuid`: The ID of the Workspace that contains the Deployment. You can provide `workspaceLabel` instead. One of the two is required.
* `clusterId`: The ID of the cluster that hosts the Deployment.
* `releaseName`: The release name of your Deployment, following the pattern `spaceyword-spaceyword-4digits`. For example, `infrared-photon-7780`.
* `image`: The full image path including registry, repository, and tag. The image must be accessible from your Astro Private Cloud data plane.
* `runtimeVersion`: The Astro Runtime version that the image is based on. For example, `12.1.0`.
* `deployRevisionDescription`: An optional description for the deploy revision, useful for tracking deploys in the APC UI.

For more information about deploying custom images with the APC API, see [Configure a custom image registry](/docs/astro-private-cloud/v-2-x/custom-image-registry).

To explore the full APC API schema and test mutations interactively, use the [GraphQL playground](/docs/astro-private-cloud/v-2-x/houston-api-develop-test).

## CI/CD platform examples

The following examples show how to implement CI/CD pipelines using the Astro CLI with popular CI/CD platforms. For advanced Docker registry-based deployment examples, see [Advanced: Docker registry deployment](#advanced-docker-registry-deployment).

### GitHub Actions

```yaml wrap theme={null}
name: Deploy to Astro Private Cloud

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Astro CLI
        run: curl -sSL https://install.astronomer.io | sudo bash -s

      - name: Authenticate
        run: astro auth login <platform-domain> --token-login
        env:
          ASTRONOMER_KEY_ID: ${{ secrets.ASTRONOMER_KEY_ID }}
          ASTRONOMER_KEY_SECRET: ${{ secrets.ASTRONOMER_KEY_SECRET }}

      - name: Deploy
        run: astro deploy ${{ vars.DEPLOYMENT_ID }}
```

### GitLab CI

```yaml wrap theme={null}
deploy-airflow:
  stage: deploy
  image: ubuntu:latest
  script:
    - curl -sSL https://install.astronomer.io | bash -s
    - astro auth login ${PLATFORM_DOMAIN} --token-login
    - astro deploy ${DEPLOYMENT_ID}
  variables:
    ASTRONOMER_KEY_ID: ${ASTRONOMER_KEY_ID}
    ASTRONOMER_KEY_SECRET: ${ASTRONOMER_KEY_SECRET}
  only:
    - main
```

### CircleCI

```yaml wrap theme={null}
version: 2.1

jobs:
  deploy:
    docker:
      - image: cimg/base:current
    steps:
      - checkout
      - run:
          name: Install and Deploy
          command: |
            curl -sSL https://install.astronomer.io | sudo bash -s
            astro auth login ${PLATFORM_DOMAIN} --token-login
            astro deploy ${DEPLOYMENT_ID}

workflows:
  deploy-workflow:
    jobs:
      - deploy:
          filters:
            branches:
              only: main
```

### Example CI/CD workflow

Consider an Astro project hosted on GitHub and deployed to Astro Private Cloud. In this scenario, `dev` and `main` branches of an Astro project are hosted on a single GitHub repository, and `dev` and `prod` Airflow Deployments are hosted on an Astronomer Workspace.

Using CI/CD, you can automatically deploy Dags to your Airflow Deployment by pushing or merging code to a corresponding branch in GitHub. The general setup:

1. Create two Airflow Deployments within your Astronomer Workspace, one for `dev` and one for `prod`.
2. Create a repository in GitHub that hosts project code for all Airflow Deployments within your Astronomer Workspace.
3. In your GitHub code repository, create a `dev` branch off of your `main` branch.
4. Configure your CI/CD tool to deploy to your `dev` Airflow Deployment whenever you push to your `dev` branch, and to deploy to your `prod` Airflow Deployment whenever you merge your `dev` branch into `main`.

That would look something like this:

<Frame>
  <img alt="CI/CD Workflow Diagram" />
</Frame>

## Service account authentication

<Tip>
  Service accounts provide secure, non-interactive authentication for CI/CD pipelines without requiring user credentials.
</Tip>

### Prerequisites

Before completing this setup, ensure you:

* Have access to a running Astro Deployment.
* Installed the [Astro CLI](https://github.com/astronomer/astro-cli).
* Are familiar with your CI/CD tool of choice.

### Create a service account

To authenticate your CI/CD pipeline to the Astronomer private Docker registry, create a service account and grant it an appropriate set of permissions. You can do so using the Astro Private Cloud UI or CLI. After creation, you can delete this service account at any time. In both cases, creating a service account generates an API key for the CI/CD process.

You can create service accounts at the:

* Workspace level: Allows you to deploy to multiple Airflow Deployments with one code push.
* Deployment level: Ensures that your CI/CD pipeline only deploys to one particular Deployment.

#### Create a service account using the CLI

Deployment level service account:

First, get your Deployment ID:

```bash wrap theme={null}
astro deployment list
```

This outputs the list of running Deployments you have access to and their corresponding UUIDs.

With that UUID, run:

```bash wrap theme={null}
astro deployment service-account create -d <deployment-id> --label <service-account-label> --role <deployment-role>
```

Workspace level service account:

First, get your Workspace ID:

```bash wrap theme={null}
astro workspace list
```

Then create the service account:

```bash wrap theme={null}
astro workspace service-account create -w <workspace-id> --label <service-account-label> --role <workspace-role>
```

#### Create a service account using the API

You can also create a service account using the GraphQL API. The `deploymentUuid` field is the same Deployment ID (UUID) returned by `astro deployment list`.

```graphql wrap theme={null}
mutation {
  createDeploymentServiceAccount(
    deploymentUuid: "<deployment-id>"
    label: "CI/CD Pipeline"
    role: DEPLOYMENT_ADMIN
  ) {
    id
    apiKey
  }
}
```

Set in CI/CD environment:

```bash wrap theme={null}
export ASTRONOMER_KEY_ID=<service-account-id>
export ASTRONOMER_KEY_SECRET=<api-key>
```

#### Create a service account using the Astro Private Cloud UI

If you prefer to provision a service account through the Astro Private Cloud UI:

1. Log into Astronomer and navigate to: `Deployment` > `Service Accounts`
2. Configure your service account:
   * Give it a Name
   * Give it a Category (optional)
   * Grant it a User Role (must be "Editor" or "Admin" to deploy code)
3. Copy the API key that is generated

<Note>The API key is only visible during the session. Store it securely in an environment variable or secret management tool.</Note>

<Note>For more information on Workspace roles, see [Roles and Permissions](/docs/astro-private-cloud/v-2-x/role-permission-reference).</Note>

### Set credentials in CI/CD environment

After creating a service account, set the credentials in your CI/CD environment:

```bash wrap theme={null}
export ASTRONOMER_KEY_ID=<service-account-id>
export ASTRONOMER_KEY_SECRET=<api-key>
```

The Astro CLI automatically uses these environment variables for authentication.

## Best practices

* Use service accounts for CI/CD authentication instead of personal credentials.
* Store credentials securely in CI/CD secrets or environment variables.
* Deploy only committed code in CI/CD pipelines to ensure reproducibility. Avoid using `--force` unless you have a specific reason to bypass the git commit check.
* Add deployment descriptions with `--description` for audit trail and version tracking.
* Test in staging before production Deployment to catch issues early. For guidance on writing Dags that work across environments, see [Manage Airflow code](/docs/learn/managing-airflow-code) and [Dag writing best practices](/docs/learn/dag-best-practices).
* Use Dag-only deploys when you only need to update Dag files without rebuilding images.

## Advanced: Docker registry deployment

For advanced use cases, legacy systems, or when you need more control over the Docker build and push process, you can deploy directly to the Astronomer Docker registry. Most users should use the [CLI deployment method](#cli-deployment) instead.

When to use Docker registry deployment:

* You need custom Docker build processes or multi-stage builds.
* You're integrating with existing Docker-based CI/CD workflows.
* You require fine-grained control over image tagging and versioning.
* You're working with legacy CI/CD systems that don't support the Astro CLI.

<Info>If you're using BuildKit with the [Buildx plugin](https://github.com/docker/buildx), you need to add the `--provenance=false` flag to your `docker buildx build` commands.</Info>

<Note>The Docker registry examples use `RELEASE_NAME` (for example, `infrared-photon-7780`) instead of `DEPLOYMENT_ID`. Both refer to your Astro Deployment, but the Astro CLI uses `DEPLOYMENT_ID` while the Docker registry approach uses the release name.</Note>

### Authenticate and push to Docker

The first step of this pipeline authenticates against the Docker registry that stores an individual Docker image for every code push or configuration change:

```bash wrap theme={null}
docker login registry.${BASE_DOMAIN} -u _ -p $${API_KEY_SECRET}
```

In this example:

* `BASE_DOMAIN` = The domain at which your Astro Private Cloud instance is running
* `API_KEY_SECRET` = The API key that you got from the CLI or the UI and stored in your secret manager

### Build and push an image

After you are authenticated, you can build, tag, and push your Airflow image to the private registry, where a webhook triggers an update to your Astro Deployment.

<Note>To deploy successfully to Astro Private Cloud, the version in the `FROM` statement of your project's Dockerfile must be *the same as or newer than* the Runtime version of your Astro Deployment. For more information on upgrading, see [Upgrade Airflow](/docs/runtime/manage-airflow-versions).</Note>

Image naming components:

* **Registry Address**: Tells Docker where to push images. On Astro Private Cloud, your private registry is located at `registry.${BASE_DOMAIN}`.
* **Release Name**: The release name of your Astro Deployment, following the pattern `spaceyword-spaceyword-4digits` (for example, `infrared-photon-7780`).
* **Tag Name**: Each deploy generates a Docker image with a corresponding tag. If you deploy via the CLI, the tag defaults to `deploy-n`, with `n` representing the number of deploys. For CI/CD, customize this tag to include the source and build number.

Example with custom tag:

```bash wrap theme={null}
docker build -t registry.${BASE_DOMAIN}/${RELEASE_NAME}/airflow:ci-${BUILD_NUMBER} .
```

### Run unit tests

For CI/CD pipelines that push code to a production Deployment, Astronomer recommends adding a unit test after the image build step to ensure that you don't push a Docker image with breaking changes. To run a basic unit test, add a step in your CI/CD pipeline that executes `docker run` and then runs `pytest tests` in a container based on your newly built image before it's pushed to your registry. For guidance on writing pytest tests for Airflow, including Dag validation tests and unit tests for custom operators, see [Test Airflow Dags](/docs/learn/testing-airflow).

For example, you can add the following command as a step in your CI/CD pipeline:

<Note>`BASE_DOMAIN`, `RELEASE_NAME`, and `BUILD_NUMBER` should be set as environment variables in your CI/CD tool.</Note>

```bash wrap theme={null}
docker run --rm registry.${BASE_DOMAIN}/${RELEASE_NAME}/airflow:ci-${BUILD_NUMBER} /bin/bash -c "pytest tests"
```

### Configure your CI/CD pipeline

Depending on your CI/CD tool, configuration varies slightly. This section focuses on outlining what needs to be accomplished, not the specifics of how.

At its core, your CI/CD pipeline first authenticates to the Astronomer private registry, then builds, tags, and pushes your Docker image to that registry.

### Docker registry example: GitHub Actions

This example shows how to implement CI/CD using GitHub Actions with Docker registry deployment for both development and production environments.

Setup steps:

1. Create a GitHub repository for your Astro project with `dev` and `main` branches.
2. Create two [Deployment-level service accounts](#create-a-service-account): one for Dev and one for Production.
3. Add service accounts as [GitHub secrets](https://docs.github.com/en/actions/reference/encrypted-secrets) named `SERVICE_ACCOUNT_KEY` and `SERVICE_ACCOUNT_KEY_DEV`.
4. Create a GitHub Action with the following workflow:

   ```yaml expandable wrap theme={null}
   name: Astronomer CI - Deploy code
   on:
     push:
       branches: [dev]
     pull_request:
       types:
         - closed
       branches: [main]
   jobs:
     dev-push:
       if: github.ref == 'refs/heads/dev'
       runs-on: ubuntu-latest
       steps:
       - name: Check out the repo
         uses: actions/checkout@v3
       - name: Log in to registry
         uses: docker/login-action@v1
         with:
           registry: registry.${BASE_DOMAIN}
           username: _
           password: ${{ secrets.SERVICE_ACCOUNT_KEY_DEV }}
       - name: Build image
         run: docker build -t registry.${BASE_DOMAIN}/<dev-release-name>/airflow:ci-${{ github.sha }} .
       - name: Run tests
         run: docker run --rm registry.${BASE_DOMAIN}/<dev-release-name>/airflow:ci-${{ github.sha }} /bin/bash -c "pytest tests"
       - name: Push image
         run: docker push registry.${BASE_DOMAIN}/<dev-release-name>/airflow:ci-${{ github.sha }}
     prod-push:
       if: github.event.action == 'closed' && github.event.pull_request.merged == true
       runs-on: ubuntu-latest
       steps:
       - name: Check out the repo
         uses: actions/checkout@v3
       - name: Log in to registry
         uses: docker/login-action@v1
         with:
           registry: registry.${BASE_DOMAIN}
           username: _
           password: ${{ secrets.SERVICE_ACCOUNT_KEY }}
       - name: Build image
         run: docker build -t registry.${BASE_DOMAIN}/<prod-release-name>/airflow:ci-${{ github.sha }} .
       - name: Run tests
         run: docker run --rm registry.${BASE_DOMAIN}/<prod-release-name>/airflow:ci-${{ github.sha }} /bin/bash -c "pytest tests"
       - name: Push image
         run: docker push registry.${BASE_DOMAIN}/<prod-release-name>/airflow:ci-${{ github.sha }}
   ```

Replace `<dev-release-name>` and `<prod-release-name>` with your Deployment release names.

5. Test the workflow by committing changes to `dev` to update your development Deployment, then merge `dev` into `main` via pull request to update production.

<Note>
  The prod-push action only runs after merging a pull request. To further restrict this pipeline, add branch protection settings in GitHub to prevent direct pushes to `main`.
</Note>

### Additional Docker registry examples

The following sections provide templates for configuring CI/CD pipelines using popular CI/CD tools with Docker registry deployment. Each template can be customized to manage multiple branches or Deployments based on your needs.

#### DroneCI

```yaml expandable wrap theme={null}
pipeline:
  build:
    image: quay.io/astronomer/ap-build:latest
    commands:
      - docker build -t registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${DRONE_BUILD_NUMBER} .
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    when:
      event: push
      branch: [ master, release-* ]

  test:
    image: quay.io/astronomer/ap-build:latest
    commands:
      - docker run --rm registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${DRONE_BUILD_NUMBER} /bin/bash -c "pytest tests"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    when:
      event: push
      branch: [ master, release-* ]

  push:
    image: quay.io/astronomer/ap-build:latest
    commands:
      - echo $${SERVICE_ACCOUNT_KEY}
      - docker login registry.$BASE_DOMAIN -u _ -p $SERVICE_ACCOUNT_KEY
      - docker push registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${DRONE_BUILD_NUMBER}
    secrets: [ SERVICE_ACCOUNT_KEY ]
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    when:
      event: push
      branch: [ master, release-* ]
```

#### CircleCI

```yaml expandable wrap theme={null}
# Python CircleCI configuration file
#
# Check https://circleci.com/docs/language-python/ for more details
#
version: 2
jobs:
  build:
    machine: ubuntu-2204:202509-01
    steps:
      - checkout
      - restore_cache:
          keys:
          - v1-dependencies-{{ checksum "requirements.txt" }}
          # fallback to using the latest cache if no exact match is found
          - v1-dependencies-
      - run:
          name: Install test deps
          command: |
            # Use a virtual env to encapsulate everything in one folder for
            # caching. And make sure it lives outside the checkout, so that any
            # style checkers don't run on all the installed modules
            python -m venv ~/.venv
            . ~/.venv/bin/activate
            pip install -r requirements.txt
      - save_cache:
          paths:
            - ~/.venv
          key: v1-dependencies-{{ checksum "requirements.txt" }}
      - run:
          name: run linter
          command: |
            . ~/.venv/bin/activate
            pycodestyle .
  deploy:
    docker:
      - image: docker:latest
    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true
      - run:
          name: Push to Docker Hub
          command: |
            TAG=0.1.$CIRCLE_BUILD_NUM
            docker build -t registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-$TAG .
            docker run --rm registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-$TAG /bin/bash -c "pytest tests"
            docker login registry.$BASE_DOMAIN -u _ -p $SERVICE_ACCOUNT_KEY
            docker push registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-$TAG

workflows:
  version: 2
  build-deploy:
    jobs:
      - build
      - deploy:
          requires:
            - build
          filters:
            branches:
              only:
                - master
```

#### Jenkins

```yaml wrap theme={null}
pipeline {
    agent any
    stages {
        stage('Deploy to astronomer') {
            when { branch 'master' }
            steps {
                script {
                    sh 'docker build -t registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${BUILD_NUMBER} .'
                    sh 'docker run --rm registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${BUILD_NUMBER} /bin/bash -c "pytest tests"'
                    sh 'docker login registry.$BASE_DOMAIN -u _ -p $SERVICE_ACCOUNT_KEY'
                    sh 'docker push registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${BUILD_NUMBER}'
                }
            }
        }
    }
    post {
        always {
            cleanWs()
        }
    }
}
```

#### Bitbucket

If you are using [Bitbucket](https://bitbucket.org/), this script should work (courtesy of our friends at [Das42](https://www.das42.com/))

```yaml wrap theme={null}
image: quay.io/astronomer/ap-build:latest

pipelines:
  branches:
    master:
      - step:
          name: Deploy to production
          deployment: production
          script:
            - echo ${SERVICE_ACCOUNT_KEY}
            - docker build -t registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${BITBUCKET_BUILD_NUMBER} .
            - docker run --rm registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${BITBUCKET_BUILD_NUMBER} /bin/bash -c "pytest tests"
            - docker login registry.$BASE_DOMAIN -u _ -p $SERVICE_ACCOUNT_KEY
            - docker push registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-${BITBUCKET_BUILD_NUMBER}
          services:
            - docker
          caches:
            - docker
```

#### GitLab

```yaml wrap theme={null}
astro_deploy:
  stage: deploy
  image: docker:latest
  services:
    - docker:dind
  script:
    - echo "Building container.."
    - docker build -t registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:CI-$CI_PIPELINE_IID .
    - docker run --rm registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:CI-$CI_PIPELINE_IID /bin/bash -c "pytest tests"
    - docker login registry.$BASE_DOMAIN -u _ -p $SERVICE_ACCOUNT_KEY
    - docker push registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:CI-$CI_PIPELINE_IID
  only:
    - master
```

#### AWS CodeBuild

```yaml wrap theme={null}
version: 0.2
phases:
  install:
    runtime-versions:
      python: latest

  pre_build:
    commands:
      - echo Logging in to dockerhub ...
      - docker login "registry.$BASE_DOMAIN" -u _ -p "$API_KEY_SECRET"
      - export GIT_VERSION="$(git rev-parse --short HEAD)"
      - echo "GIT_VERSION = $GIT_VERSION"
      - pip install -r requirements.txt

  build:
    commands:
      - docker build -t "registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-$GIT_VERSION" .
      - docker run --rm "registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-$GIT_VERSION" /bin/bash -c "pytest tests"
      - docker push "registry.$BASE_DOMAIN/$RELEASE_NAME/airflow:ci-$GIT_VERSION"
```

#### Azure DevOps

This example shows how to automatically deploy your Astro project from a GitHub repository using an [Azure DevOps](https://azure.microsoft.com/en-us/services/devops/) pipeline.

<Tip>To see an example GitHub project that uses this configuration, see [cs-tutorial-azuredevops](https://github.com/astronomer/cs-tutorial-azuredevops) on GitHub.</Tip>

Prerequisites:

* A GitHub repository hosting your Astro project.
* An Azure DevOps account with permissions to create new pipelines.

Setup steps:

1. Create a file called `astro-devops-cicd.yaml` in your Astro project repository:

   ```yaml wrap theme={null}
   # Control which branches have CI triggers:
   trigger:
   - main

   # To trigger the build/deploy only after a PR has been merged:
   pr: none

   # Optionally use Variable Groups & Azure Key Vault:
   #variables:
   #- group: Variable-Group
   #- group: Key-Vault-Group

   stages:
   - stage: build
     jobs:
     - job: run_build
       pool:
         vmImage: 'Ubuntu-latest'
       steps:
       - script: |
           echo "Building container.."
           docker build -t registry.$(BASE-DOMAIN)/$(RELEASE-NAME)/airflow:$(Build.SourceVersion) .
           docker run --rm registry.$(BASE-DOMAIN)/$(RELEASE-NAME)/airflow:$(Build.SourceVersion) /bin/bash -c "pytest tests"
           docker login registry.$(BASE-DOMAIN) -u _ -p $(SVC-ACCT-KEY)
           docker push registry.$(BASE-DOMAIN)/$(RELEASE-NAME)/airflow:$(Build.SourceVersion)
   ```

2. Follow the steps in [Azure documentation](https://docs.microsoft.com/en-us/azure/devops-project/azure-devops-project-github#configure-access-to-your-github-repo-and-select-a-framework) to link your GitHub repository to an Azure pipeline. When prompted for the source code for your pipeline, specify that you have an existing Azure Pipelines YAML file and provide the file path: `astro-devops-cicd.yaml`.

3. Finish and save your Azure pipeline setup.

4. In Azure, [add environment variables](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops\&tabs=yaml%2Cbatch) for the following values:
   * `BASE-DOMAIN`: Your base domain for Astro Private Cloud
   * `RELEASE-NAME`: The release name for your Deployment
   * `SVC-ACCT-KEY`: The service account key you created for CI/CD (mark as secret)

After completing this setup, any merges to the main branch of your GitHub repository trigger the pipeline and deploy your changes to Astro Private Cloud.


# Add certificate authorities (CAs) to Docker Desktop
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-desktop-container-solution-extra-cas

Add trusted certificate authorities (CAs) to Docker Desktop

If Astro Private Cloud (APC) users deploy images to a container registry, including the integrated container registry, that uses a TLS certificate signed by a private certificate authority (CA), you need to configure Docker Desktop to trust the CA's public certificate.

Obtain a copy of the CA's public certificate in pem format and place it in `/etc/docker/certs.d`:

```bash wrap theme={null}
mkdir -p /etc/docker/certs.d
cp privateCA.pem /etc/docker/certs.d/
```


# Trust private certificate authorities (CAs)
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-private-cas

Configure Astro Private Cloud to trust private certificate authorities (CAs)

This guide explains how to install Astro Private Cloud in an environment that uses Private Certificate Authority (Private CA)–issued TLS certificates. In this setup, all platform components, including the control plane, data plane, and internal services, communicate over secure HTTPS connections that are validated against your organization’s internal CA. This procedure covers configuring trusted roots, deploying certificates to the appropriate namespaces, and ensures the Astro installation properly trusts and uses the Private CA during installation and runtime.

1. Store the CA's root public certificate to an [Opaque Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret/#secret-types) in the Astro Private Cloud namespace with a descriptive name, such as `private-root-ca`, by running the following command.

<Tip>
  Before you run this command, keep the following in mind:

  * The root certificate you specify should be the certificate of the authority that signed the Astro Private Cloud certificate. This isn't the certificate associated with Astro Private Cloud or any other service.
  * The name of the secret file must be `cert.pem` for your certificate to be trusted properly.
  * The file must contain only a single certificate, it can't be a certificate bundle.
</Tip>

```bash wrap theme={null}
kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem
```

2. Add `<secret name>` to the list of secret names contained in `global.privateCaCerts` in `values.yaml`:

```yaml wrap theme={null}
global:
  privateCaCerts:
  - private-root-ca
```

<Warning>Step 3 and 4 are additional steps for Private CA on the control plane only</Warning>

3. Create a database secret and add `<secret name>` to the list of secret names.

```sh wrap theme={null}
kubectl -n astronomer create secret generic db-private-ca  --from-file=cert.pem=./private-root-ca.pem
```

```yaml wrap theme={null}
global:
  privateCaCerts:
  - private-root-ca
  - db-private-ca
```

4. Add your Private CA to your Helm `values.yaml`.

```yaml wrap theme={null}
database:
  connection:
    ssl:
      ca: /etc/ssl/certs/ca-certificates.pem
      rejectUnauthorized: true
```

## LDAP over LDAPS with a private CA

<Note>
  **Astro Private Cloud 2.1**

  This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
</Note>

When your LDAP directory presents a server certificate signed by a Private CA, this configuration is what makes Houston trust the certificate at the LDAP layer. After completing the preceding steps, keep `auth.ldap.tls.mode: ldaps` and `auth.ldap.tls.verifyServerCert: true` in Houston's configuration. For the LDAP-side configuration, see [Configure LDAP authentication](/docs/astro-private-cloud/v-2-x/configure-ldap-authentication#configure-tls).


# Astro Private Cloud control plane architecture
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/control-plane-architecture

Understand how Astro Private Cloud components behave when a cluster runs in control plane mode.

Astro Private Cloud (APC) separates responsibilities between a control plane cluster and one or more data plane clusters. The control plane hosts the shared management experience while delegating Airflow execution to attached data planes. This document summarizes the components that run inside a control plane, how they interact with data planes, and what infrastructure operators must provide. For the runtime side of the platform, see [Data Plane Architecture](/docs/astro-private-cloud/v-2-x/data-plane-architecture), or review [Unified Architecture](/docs/astro-private-cloud/v-2-x/unified-architecture) if you run both roles in a single cluster.

## Responsibilities

A control plane cluster focuses on:

* **Platform management**: The APC control service (APC API), the web interface (Astro UI), the event streaming broker (NATS JetStream), and supporting cronjobs store platform configuration, authenticate users, and orchestrate deployments across data planes.
* **Tenant management**: Admins create workspaces, manage users and teams, integrate IdP, control registry access tokens, and publish runtime images from this cluster.
* **Centralized telemetry**: The metrics store (Prometheus) scrapes both local platform service metrics and federates metrics from attached data planes. The alert routing service (Alertmanager) raises notifications for the entire platform.
* **Coordinate with data planes**: The control plane exposes TLS-secured ingress endpoints (`app.<base-domain>`, `houston.<base-domain>`, etc.) for users and the Astro CLI, and it keeps secure connections open to each data plane’s deployment orchestrator and metrics gateway so configuration and telemetry stay in sync.

The control plane never hosts user Airflow workloads. Instead, it maintains references to external Kubernetes clusters (data planes) and coordinates their lifecycle.

## Core components in control mode

APC enables the following components when `global.plane.mode` is set to `control` (or `unified`):

* **Astro Private Cloud web interface (UI) (`charts/astronomer/templates/astro-ui/*`)**: Provides the web application for administrators and users. Runs only on the control plane.
* **Astro Private Cloud control service (APC API) and cronjobs (`charts/astronomer/templates/houston/**/*`)**: Manages platform metadata, user auth, workspace creation, registry tokens, and periodic tasks (cleanup jobs, metrics aggregation).
* **Internal event streaming service (NATS JetStream) (`charts/nats/templates/*`)**: Event bus used by the APC API and deployment orchestrator to coordinate deployments.
* **Alert routing service (Alertmanager) (`charts/alertmanager/templates/*`)**: Consolidates alerts sent by Prometheus.
* **Central metrics store (Prometheus)**: The shared Prometheus StatefulSet (`charts/prometheus/templates/prometheus-statefulset.yaml`) runs in all modes, but its federation jobs, ingress, and auth proxy are tailored to aggregate data plane metrics when running as a control plane.
* **Control plane NGINX ingress (`charts/nginx/templates/controlplane/*`)**: Exposes the Control Plane UI and API endpoints for users and the Astro CLI.
* **Optional Postgres/PgBouncer (`charts/postgresql`, `charts/pgbouncer`)**: Most Deployments use an external database, but if `global.postgresql.enabled=true`, the embedded database is deployed regardless of plane mode. In a split Deployment this chart is typically disabled. Astronomer recommends only using embedded Postgres for testing or development environments.

These services rely on a handful of Kubernetes constructs such as ClusterRoles, service accounts, network policies, and ingress controllers that the chart generates automatically when `mode=control` or `mode=unified`.

## Network surface

Control plane ingress endpoints typically include:

* `app.<base-domain>`: APC web interface (UI) for administrators and workspace users.
* `houston.<base-domain>`: API traffic for the UI, Astro CLI, and deployment orchestrator callbacks.
* `alertmanager.<base-domain>` and `prometheus.<base-domain>`: Optional if those dashboards are exposed.
* `registry.<base-domain>`: When hosting the integrated container registry on the control plane. In a split deployment the registry generally lives on data planes; expose or disable depending on your design.

Control plane pods also connect to data planes using deployment orchestrator and Prometheus federation endpoints sharing tokens that the APC API issues.

## Connection to data planes

Each data plane is registered with the control plane through the APC API. After registration:

1. `deployment orchestrator (Commander)` in data planes receives payloads from the Control Plane the APC API for desired state, applies Helm releases for each Airflow deployment, and posts status updates.
2. `Secret distribution job (Config Syncer)` (data plane cronjob) pushes tokens into Airflow namespaces; the APC API maintains them.
3. Data plane Prometheus remote-writes into control plane Prometheus or exposes a federate endpoint that control Prometheus scrapes.
4. Optional logging plane components such as the log forwarder (Vector) and log store (Elasticsearch) ship logs upward or to an external destination.

## Unified mode comparison

Setting `global.plane.mode` to `unified` deploys both control plane and data plane components into the same cluster. That configuration is helpful for testing or small environments but loses the strict separation and isolation you gain with split deployments. For a detailed comparison, see the accompanying [Unified Architecture](/docs/astro-private-cloud/v-2-x/unified-architecture) page.

## Next steps

* Deploy the control plane by following the [Install Control Plane](/docs/astro-private-cloud/v-2-x/install-control-plane) guide.
* Provision data plane clusters and register them via the Astro UI.
* Review network policies and firewall rules to secure traffic between planes.
* Configure monitoring and alert policies using Prometheus and the alert routing service (Alertmanager).


# Astro Private Cloud data plane architecture
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/data-plane-architecture

Understand how Astro Private Cloud components behave when a cluster runs in data plane mode.

The Astro Private Cloud (APC) data plane hosts the execution layer for your Airflow deployments. When you set the `global.plane.mode: data` in your `values.yaml` file, the Helm chart deploys only the runtime-facing components while relying on a separate control plane for user management, configuration, registry orchestration, and token orchestration. This document summarizes the data plane’s responsibilities, the services that run in this mode, and how the services integrate with the control plane. For the management plane, see [Control Plane Architecture](/docs/astro-private-cloud/v-2-x/control-plane-architecture), or review [Unified Architecture](/docs/astro-private-cloud/v-2-x/unified-architecture) if you want to run both in a single cluster.

## Responsibilities

A data plane cluster focuses on:

* **Running customer Airflow Deployments**: The deployment orchestrator installs and upgrades each Deployment’s runtime chart using configuration synced from the APC API.
* **Serving Airflow ingress**: The ingress controller (Data plane NGINX) exposes `deployments.<domain-prefix>.<base-domain>` (and any per-Deployment vanity hostnames) to route user traffic into the correct Airflow namespace.
* **Collecting telemetry**: The metrics collector (Prometheus) scrapes Deployment namespaces and either exposes a federate endpoint or remote-writes metrics to the control plane. Optional logging stacks (log forwarder Vector and log store Elasticsearch) gather task logs.
* **Handling image distribution**: The platform registry (Registry) stores runtime and Dag images local to the data plane and syncs credentials using the secret distribution job (Config Syncer).
* **Maintaining secure connectivity**: Config Syncer distributes the APC API-issued tokens and certificates so Deployments can authenticate back to the APC API, the registry, and other platform services.

## Core components in data plane mode

When `global.plane.mode` is set to `data` or `unified`, APC enables the following charts:

* **The deployment orchestrator (`charts/astronomer/templates/commander/*`)**: Polls the APC API for desired state and applies/rolls back Helm releases for each Deployment.
* **Config Syncer (`charts/astronomer/templates/config-syncer/*`)**: Periodically mirrors platform secrets (registry credentials, the APC API tokens, runtime settings) into each Airflow namespace.
* **Data plane NGINX (`charts/nginx/templates/dataplane/*`)**: Provides ingress for the per-Deployment Airflow UI and API as well as for platform components like the Registry, Prometheus, and Elasticsearch. This component isn't installed when OpenShift is enabled.
* **Platform registry (Registry) (`charts/astronomer/templates/registry/*`)**: Optional container registry for runtime/Dag images when the bundled registry is enabled.
* **Vector (`charts/vector/templates/*`)**: Runs as a DaemonSet when `global.daemonsetLogging.enabled` is set, or as a sidecar container when `global.logging.loggingSidecar` is enabled, tailing task logs and shipping them to Elasticsearch or an external destination.
* **Elasticsearch (`charts/elasticsearch/templates/*`)**: Optional log storage for Deployments. Only deployed in data plane or unified modes when enabled.
* **Cluster state exporter (kube-state-metrics) (`charts/kube-state/templates/*`)**: Scrapes namespace-level object metadata for Prometheus.
* **Metrics gateway (Prometheus federation/auth) (`charts/prometheus/templates/prometheus-federation-*`)**: Adds the auth proxy, Service, and federation jobs necessary for the control plane to scrape metrics securely.
* **Auxiliary services**: External-es proxy, namespace pool RBAC, and other helper charts that only make sense near the workloads.

Prometheus, Postgres, and other shared charts still exist. In data mode, Prometheus pushes or exposes metrics back to the control plane rather than aggregating globally.

## Network endpoints

Data plane ingress typically includes:

* `deployments.<domain-prefix>.<base-domain>`: Airflow web UIs and APIs for each Deployment. the deployment orchestrator configures path-based routing for these Deployments.
* `registry.<domain-prefix>.<base-domain>`: When hosting the registry in the data plane.
* deployment orchestrator endpoints for registering clusters and creating and updating Deployments.
* Optional vanity hostnames per Deployment managed by the deployment orchestrator’s Helm releases.
* Data plane metrics collector (Prometheus) serves a federate endpoint scraped by the control plane.

Outbound connections include:

* **To the APC API**: the deployment orchestrator, Config Syncer, and cronjobs call the control plane API over TLS.
* **To external registries/log stores**: Depending on how runtime images and logs are hosted.

## Control plane integration

Data planes authenticate against the APC API using service accounts and tokens that the control plane provisions. The typical workflow is:

1. **Registration**: A platform admin registers a data plane entry in the APC API, which generates unique the deployment orchestrator and Config Syncer tokens.
2. **Secret distribution**: During install you provide the APC API tokens via `astronomer.houston.config.dataplane.*` values. The secret distribution job (Config Syncer) keeps runtime secrets fresh.
3. **Deployment lifecycle**: APC API pushes requests to the deployment orchestrator, which reconciles Deployments. If the control plane issues upgrades or scale instructions, the deployment orchestrator applies them locally.
4. **Telemetry forwarding**: The metrics collector (Prometheus) and log forwarder (Vector) transport metrics and logs to the control plane (or third-party sinks) so administrators have a single-pane view.

## Behavior during a control plane outage

In a split Deployment, data plane clusters operate independently from the control plane at runtime. If the APC API becomes unavailable, your existing Airflow workloads continue without interruption, but all management operations and external access to Airflow stop.

**What stops working**:

* **Authentication**: Sign-in to both the Astro Private Cloud UI and Airflow UIs fails because authentication flows depend on the APC API.
* **Astro Private Cloud UI, APC API, and Astro CLI**: All management interfaces become unavailable, including deployment creation, configuration changes, and user management.
* **New data plane registration**: Registration requires the APC API.
* **Deployment changes**: the deployment orchestrator can't receive new instructions from the APC API, so you can't create, update, or delete Deployments.
* **External Airflow API access**: Requests to the Airflow REST API from outside the data plane cluster fail because they route through the APC API-authenticated ingress.

**What keeps running**:

* **Existing Airflow Deployments**: All Airflow components (scheduler, webserver, workers, triggerer) remain running. the APC API isn't in the runtime execution path for Dags.
* **Scheduled Dag runs**: The Airflow scheduler continues to trigger Dags on schedule, and in-flight tasks complete normally.
* **Internal Airflow communication**: Task execution, XCom, and connections to external data sources continue to function.
* **Data plane infrastructure**: the deployment orchestrator retains the last-known desired state and continues to reconcile existing Helm releases. Config Syncer, NGINX ingress, and other data plane components remain running.
* **Local metrics collection**: Prometheus on the data plane continues scraping Airflow and platform metrics locally.

<Warning>
  While Airflow keeps running, you can't make changes or access Airflow UIs through the standard ingress during a control plane outage. If you need emergency access to a running Deployment, use `kubectl` to interact directly with the data plane cluster.
</Warning>

When the control plane recovers, the deployment orchestrator and Config Syncer reconnect to the APC API automatically. Prometheus federation or remote-write resumes, and any metrics gap during the outage appears in your monitoring dashboards. No manual intervention is required on the data plane side unless tokens expired during the outage. See [Register a data plane](/docs/astro-private-cloud/v-2-x/register-data-plane) for token management details.

## Monitoring and alerting

Data plane Prometheus scrapes:

* Airflow Deployments, including scheduler, webserver/API server, workers.
* Platform Pods, like the deployment orchestrator, Config Syncer, NGINX, and Vector.
* Kube-state-metrics for namespace-wide object counts.

Use Alertmanager rules provided by the chart or integrate with the control plane Alertmanager to drive notifications.

<Tip>
  Watch for deployment orchestrator heartbeat failures, Config Syncer errors, and Prometheus remote-write issues—these often indicate connectivity problems back to the APC API.
</Tip>

## Comparison to other modes

Split deployments keep the affected area confined to workload execution, letting you scale your data planes independently and enforce network boundaries to sensitive data resources.

* For the management plane overview see [Control Plane Architecture](/docs/astro-private-cloud/v-2-x/control-plane-architecture).
* For a single-cluster footprint see [Unified Architecture](/docs/astro-private-cloud/v-2-x/unified-architecture).

## Next steps

* Deploy a data plane using the [Install data plane](/docs/astro-private-cloud/v-2-x/install-data-plane) guide.
* [Register the data plane](/docs/astro-private-cloud/v-2-x/register-data-plane) with your control plane and verify deployment orchestrator heartbeat.
* Configure DNS, TLS certificates, and networking policies based on the ingress endpoints.
* Integrate telemetry (metrics, logs) with your central observability tools.


# Database architecture
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/database-architecture

Understand the database architecture in Astro Private Cloud including the APC API and Apache Airflow metadata databases.

Astro Private Cloud (APC) uses PostgreSQL as the primary database system for both platform management (the APC API) and Apache Airflow deployments. The following sections describe the database architecture, provisioning, connection pooling, and high availability configuration.

## Database components

### APC database

The APC API uses a PostgreSQL database to store platform configuration and state.

Key tables:

* **User** - System users with authentication credentials
* **Workspace** - Organizational units for grouping deployments
* **Deployment** - Airflow deployment configurations
* **Cluster** - Data plane cluster properties including cloud provider, region, and configuration
* **RoleBinding** - User and service account role assignments
* **ServiceAccount** - API keys for programmatic access
* **DeployRevision** - Deployment version history
* **TaskUsage** - Task execution metrics

### Airflow metadata database

Each Airflow Deployment gets its own isolated PostgreSQL database containing two schemas:

| Schema    | Purpose                                  |
| --------- | ---------------------------------------- |
| `airflow` | Airflow scheduler and webserver metadata |
| `celery`  | Celery task results backend              |

The database name is derived from the Deployment release name:

```text wrap theme={null}
deployment-name → deployment_name_airflow
```

Each schema has a separate database user with permissions scoped to that schema. the deployment orchestrator provisions the database and both schemas automatically when you create a Deployment. For more information about what Airflow stores in the metadata database, see [Understanding the Airflow metadata database](/docs/learn/airflow-database).

## Database provisioning

### Automatic provisioning

By default, the deployment orchestrator automatically provisions databases for new Deployments. No additional configuration is required.

### External database

To use pre-existing or managed databases, set `skipAirflowDatabaseProvisioning` to `true` in the `upsertDeployment` mutation:

```graphql wrap theme={null}
mutation {
  upsertDeployment(
    workspaceUuid: "<workspace-uuid>"
    label: "<my-deployment-label>"
    skipAirflowDatabaseProvisioning: true
  ) {
    id
  }
}
```

When using external databases, provide the connection string in your Deployment configuration. For complete setup steps with connection string examples, see [Bring your own Airflow database](/docs/astro-private-cloud/v-2-x/multi-db).

For supported PostgreSQL versions, see the [Apache Airflow database backend documentation](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html).

## Connection pooling (PgBouncer)

APC uses PgBouncer for connection pooling to reduce database connection overhead. Airflow can open many database connections due to its distributed nature, and each PostgreSQL connection creates a dedicated OS process. PgBouncer reduces this overhead by maintaining a pool of reusable server connections. For more detail on why PgBouncer is recommended, see the [Apache Airflow Helm chart production guide](https://airflow.apache.org/docs/helm-chart/stable/production-guide.html).

<Note>
  PgBouncer is only used with the embedded PostgreSQL database. When you provide a `metadataConnectionString` URI through the `upsertDeployment` mutation, Airflow connects directly to the external database and bypasses PgBouncer. If you use an external database and need connection pooling, configure it through your managed database service or deploy a separate PgBouncer instance.
</Note>

PgBouncer operates in `transaction` pool mode by default, which means server connections are returned to the pool after each transaction completes. This mode doesn't support session-level features such as prepared statements or `SET` commands that persist across transactions.

### Configuration

```yaml wrap theme={null}
pgbouncer:
  enabled: true
  port: 6543
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m
```

### Pool sizes

Pool sizes are configured per Deployment in the Airflow chart values:

```yaml wrap theme={null}
pgbouncer:
  metadataPoolSize: 10       # Airflow metadata connections per pool
  resultBackendPoolSize: 5   # Celery result backend connections per pool
  maxClientConn: 100         # Maximum client connections to PgBouncer
```

These defaults match the [Apache Airflow Helm chart defaults](https://airflow.apache.org/docs/helm-chart/stable/parameters-ref.html) and work for most Deployments. To determine whether you need to adjust pool sizes, monitor PgBouncer's `cl_waiting` metric. If clients consistently wait for connections, increase pool sizes or scale PgBouncer replicas.

<Note>
  Airflow also maintains its own [SQLAlchemy connection pool](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#sql-alchemy-pool-size) (default `pool_size: 5`, `max_overflow: 10`) between each Airflow component and PgBouncer. With these defaults, each Airflow process can open up to 15 simultaneous database connections. The full connection chain is: Airflow component → SQLAlchemy pool → PgBouncer → PostgreSQL.
</Note>

### Kerberos support

PgBouncer in APC includes Kerberos support:

```yaml wrap theme={null}
pgbouncer:
  image:
    repository: quay.io/astronomer/ap-pgbouncer-krb
    tag: 1.25.0-3
```

For complete Kerberos database setup instructions, see [Configure Kerberos authentication for Airflow databases](/docs/astro-private-cloud/v-2-x/kerberos-database-setup).

## High availability

### PostgreSQL replication

<Warning>
  Astronomer doesn't recommend using internal PostgreSQL instance. If you need production database resiliency, use an externally managed database service such as Amazon RDS, Google Cloud SQL, or Azure Database for PostgreSQL. Managed services provide built-in replication, automated failover, and backup capabilities that are more reliable than running replication within the Helm chart.
</Warning>

Replication is disabled by default. The following example shows a recommended production configuration that enables streaming replication with synchronous commit:

```yaml wrap theme={null}
postgresql:
  replication:
    enabled: true
    slaveReplicas: 2
    synchronousCommit: "on"
    numSynchronousReplicas: 1
    user: repl_user
```

### Persistence

```yaml wrap theme={null}
postgresql:
  persistence:
    enabled: true
    size: 8Gi
    storageClass: ""  # Use default storage class
```

## Default and recommended for production

The following table compares the chart default values with starting points that Astronomer recommends for production environments. The **Default** column reflects the values shipped in the Helm chart. The **Recommended for production** column reflects general guidance based on Astronomer operational experience. Adjust these values based on the number of Deployments, workload scale, and observed resource utilization in your environment.

For additional production guidance, see the [Apache Airflow Helm chart production guide](https://airflow.apache.org/docs/helm-chart/stable/production-guide.html). Astronomer also recommends using a managed database service such as Amazon RDS, Google Cloud SQL, or Azure Database for PostgreSQL rather than the embedded PostgreSQL container for production workloads.

### PostgreSQL

| Setting                                         | Default | Recommended for production                                                  |
| ----------------------------------------------- | ------- | --------------------------------------------------------------------------- |
| `postgresql.resources.requests.cpu`             | 250m    | 1000m. Adjust based on query load                                           |
| `postgresql.resources.requests.memory`          | 256Mi   | 2Gi. Adjust based on active connection count                                |
| `postgresql.resources.limits.cpu`               | 1000m   | 2000m. Adjust based on query load                                           |
| `postgresql.resources.limits.memory`            | 2Gi     | 4Gi. Adjust based on active connection count                                |
| `postgresql.persistence.enabled`                | `true`  | `true`                                                                      |
| `postgresql.persistence.size`                   | 8Gi     | 50Gi or more. Scale based on the number of Deployments and retention policy |
| `postgresql.replication.enabled`                | `false` | `true`                                                                      |
| `postgresql.replication.slaveReplicas`          | 1       | 2                                                                           |
| `postgresql.replication.synchronousCommit`      | `"off"` | `"on"`                                                                      |
| `postgresql.replication.numSynchronousReplicas` | 0       | 1                                                                           |

### PgBouncer (platform level)

| Setting                               | Default | Recommended for production                  |
| ------------------------------------- | ------- | ------------------------------------------- |
| `pgbouncer.resources.requests.cpu`    | 250m    | 250m                                        |
| `pgbouncer.resources.requests.memory` | 256Mi   | 256Mi                                       |
| `pgbouncer.resources.limits.cpu`      | 250m    | 500m. Adjust based on connection throughput |
| `pgbouncer.resources.limits.memory`   | 256Mi   | 512Mi. Adjust based on connection count     |

### PgBouncer (per Deployment)

These values match the [Apache Airflow Helm chart defaults](https://airflow.apache.org/docs/helm-chart/stable/parameters-ref.html). No official formula ties pool sizes to a specific workload metric such as Dag count. Instead, adjust pool sizes based on observed connection utilization.

| Setting                           | Default | Recommended for production                                          |
| --------------------------------- | ------- | ------------------------------------------------------------------- |
| `pgbouncer.metadataPoolSize`      | 10      | 10. Increase if `cl_waiting` is consistently non-zero               |
| `pgbouncer.resultBackendPoolSize` | 5       | 5. Increase for Deployments with high Celery task throughput        |
| `pgbouncer.maxClientConn`         | 100     | 100. Increase if Deployments run many concurrent Airflow components |

## Backup and recovery

For database backup and restore procedures, including size estimation, `pg_dump`/`mysqldump` commands, and restore steps, see [Access Airflow database](/docs/astro-private-cloud/v-2-x/access-airflow-database#back-up-and-restore).

## Connection strings

### APC database

The APC database connection depends on whether you use an external database or the in-cluster PostgreSQL. You configure this during [control plane installation](/docs/astro-private-cloud/v-2-x/install-control-plane).

For external databases, provide the connection through `houston.backendConnection` in your Helm values:

```yaml wrap theme={null}
houston:
  backendConnection:
    user: <username>
    pass: <password>
    host: <external-database-host>
    port: 5432
    db: <database-name>
```

Alternatively, provide a pre-existing Kubernetes secret name through `houston.backendSecretName`. The secret must contain a `connection` key with the full connection URI.

To retrieve the active APC database connection string, read the `astronomer-bootstrap` secret:

```bash wrap theme={null}
kubectl get secret -n astronomer astronomer-bootstrap \
  -o jsonpath='{.data.connection}' | base64 -d
```

<Note>
  The in-cluster PostgreSQL option (`global.postgresql.enabled: true`) is only for development or proof-of-concept environments and isn't supported in production.
</Note>

### Airflow database

Both the metadata and result backend connections use the same database with different schemas and credentials:

```text wrap theme={null}
# Metadata connection (airflow schema)
postgresql://<metadata-user>:<password>@<host>:5432/<deployment>_airflow

# Result backend connection (celery schema)
postgresql://<celery-user>:<password>@<host>:5432/<deployment>_airflow
```

When PgBouncer is enabled, Airflow components connect through PgBouncer on port 6543 instead of directly to PostgreSQL on port 5432.

To retrieve the connection string for a specific Deployment, see [Access Airflow database](/docs/astro-private-cloud/v-2-x/access-airflow-database#retrieve-the-connection-string).

## Monitoring

Monitor the following metrics to track database and connection pool health:

| Metric              | Description                                                  |
| ------------------- | ------------------------------------------------------------ |
| `pg_stat_activity`  | Active database connections                                  |
| `pg_database_size`  | Database size on disk                                        |
| `pgbouncer_pools_*` | PgBouncer connection pool statistics, including `cl_waiting` |

## Best practices

* Enable replication for production Deployments.
* Use PgBouncer to manage connection overhead.
* Monitor connection pools for `cl_waiting` and database size growth.
* Configure regular backups. See [Access Airflow database](/docs/astro-private-cloud/v-2-x/access-airflow-database#back-up-and-restore).
* Size pools based on observed connection utilization, not Dag count.
* Use separate credentials for each Deployment.
* Enable SSL for database connections in production.
* Use a managed database service for production workloads instead of the in-cluster PostgreSQL container.


# Database connection behavior reference
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/database-behavior-reference

Reference how Astro Private Cloud configures deployment metadata and result backend database connections.

This reference documents database connection behavior that the control plane API enforces during Deployment upsert.

## Scope

This document focuses on how Astro Private Cloud configures Deployment metadata and result backend connections.

It doesn't cover infrastructure-level replication, backup, or recovery procedures.

For complete setup steps, see the following documents:

* [Database architecture](/docs/astro-private-cloud/v-2-x/database-architecture) for database components, connection pooling, high availability, and production recommendations.
* [Access Airflow database](/docs/astro-private-cloud/v-2-x/access-airflow-database) for connecting to the database, running queries, and backup and restore procedures.
* [Bring your own Airflow database](/docs/astro-private-cloud/v-2-x/multi-db) for manual metadata and result backend connections.
* [Configure Kerberos authentication for Airflow databases](/docs/astro-private-cloud/v-2-x/kerberos-database-setup) for Kerberos-specific database setup.
* [Programmatically create or update Deployments on Astro Private Cloud](/docs/astro-private-cloud/v-2-x/create-deployment-programmatic) for upsert workflow details.

## Default deployment database behavior

By default, Astro Private Cloud generates deployment-specific database connection details from:

* Global deployment configuration.
* Deployment release name.
* Cluster database type.

Generated connection details use:

* One deployment database name derived from release name.
* `airflow` schema for metadata.
* `celery` schema for result backend.
* Separate generated credentials for metadata and result backend access.

## Configure provisioning behavior

Set global database behavior in deployment config:

```yaml wrap theme={null}
deployments:
  databaseManagement:
    database:
      enabled: true
      retainOnDelete: false
      allowRootAccess: false
```

* `enabled: false` skips automatic generation of deployment connection details.
* `retainOnDelete: true` keeps deployment database resources after deployment deletion.
* `allowRootAccess: true` leaves root grants in place.

You can also set per-deployment behavior on upsert:

```graphql wrap theme={null}
skipAirflowDatabaseProvisioning: Boolean
```

## Manual connection input

Manual connection strings are disabled by default. To pass manual connection values in upsert payloads, set `deployments.databaseManagement.manualConnectionStrings.enabled` to `true`. In APC 2.x this is a `deployments.*` setting, so cluster, Workspace, and Deployment overrides take precedence over `values.yaml`. The recommended path is to add the following to the data plane cluster's **Configuration Override** (cluster overrides apply to `deployments.*`, so don't include the `deployments.` prefix):

```yaml wrap theme={null}
databaseManagement:
  manualConnectionStrings:
    enabled: true
```

If disabled (the default), upsert rejects manual connection fields.

For manual connection setup and examples, see [Bring your own Airflow database](/docs/astro-private-cloud/v-2-x/multi-db). For precedence between platform config and cluster, Workspace, and Deployment overrides, see [Configure Astro Private Cloud](/docs/astro-private-cloud/v-2-x/configure-astro-private-cloud).

## PgBouncer input behavior

PgBouncer behavior applies through deployment chart config:

```yaml wrap theme={null}
deployments:
  helm:
    airflow:
      pgbouncer:
        enabled: true
```

When PgBouncer is enabled for PostgreSQL-based Deployments:

* URI-style manual connection input is rejected.
* JSON-style manual connection input is required.

Expected JSON fields:

* `metadataConnectionJson`
* `resultBackendConnectionJson`

For Kerberos-driven PgBouncer usage, see [Configure Kerberos authentication for Airflow databases](/docs/astro-private-cloud/v-2-x/kerberos-database-setup).

## Kerberos validation behavior

When `kerberosEnabled` is `true` in an upsert payload:

* Provide both metadata and result backend connections, either URI pair or JSON pair.
* Provide `pgbouncerConfig`.
* Set `pgbouncerConfig.extraIniMetadata` with `user=`.
* Set `pgbouncerConfig.extraIniResultBackend` with `user=`.
* Set `pgbouncerConfig.sslmode`.
* Set `pgbouncerConfig.extraIni` with:
  * `server_gssauth_negotiate = allow`
  * `server_krb_spn`

For payload examples and prerequisites, see [Configure Kerberos authentication for Airflow databases](/docs/astro-private-cloud/v-2-x/kerberos-database-setup) and [Bring your own Airflow database](/docs/astro-private-cloud/v-2-x/multi-db).


# Debug an Astro Private Cloud installation
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/debug-install

Troubleshoot Astro Private Cloud Pods that aren't running after installation.

Use this guide when your Astro Private Cloud (APC) control plane or data plane Pods aren't progressing to a healthy state after installation.

## Ensure platform components are reaching full availability

Work through the following checks to from controllers to individual containers to isolate possible causes when Pods don't reach the `READY` state.

### 1. Verify controllers and ReplicaSets

1. List Deployments, StatefulSets, and ReplicaSets in your namespace and confirm the latest ReplicaSet or StatefulSet shows the expected number of available replicas:

   ```bash wrap theme={null}
   kubectl get deployment,statefulset,replicaset -n <astronomer namespace>
   ```

2. Identify the most recent ReplicaSet for the component that is failing, with results sorted by creation timestamp:

   ```bash wrap theme={null}
   kubectl get replicaset -n <astronomer namespace> --sort-by=.metadata.creationTimestamp
   ```

3. Inspect the returned ReplicaSet for status and events that may be preventing Pods from launching:

   ```bash wrap theme={null}
   kubectl describe replicaset <replicaset-name> -n <astronomer namespace>
   ```

   Resolve issues such as insufficient resources, pull errors, or missing secrets, then re-check the ReplicaSet until `.status.availableReplicas` matches `.spec.replicas`.

### 2. Examine Pods and namespace events

1. List Pod status:

   ```bash wrap theme={null}
   kubectl get pods -n <astronomer namespace>
   ```

2. `describe` a failing Pod to view events, container status, and scheduling details:

   ```bash wrap theme={null}
   kubectl describe pod <pod-name> -n <astronomer namespace>
   ```

3. Review recent events in the namespace for additional context:

   ```bash wrap theme={null}
   kubectl get events -n <astronomer namespace> --sort-by=.lastTimestamp
   ```

### 3. Inspect container logs

If a Pod continues to restart or stuck in `CrashLoopBackOff`, gather logs for each container:

```bash wrap theme={null}
kubectl logs <pod-name> -c <container-name> -n <astronomer namespace>
```

If the container restarts quickly, use `--previous` to view logs from the last attempt:

```bash wrap theme={null}
kubectl logs <pod-name> -c <container-name> -n <astronomer namespace> --previous
```

Use the collected errors to adjust your configuration, for example, by fixing database credentials or registry access. After remediation, re-run `kubectl get pods` to confirm all Pods report `READY` status. If problems persist, collect the relevant logs and events and contact [Astronomer support](https://support.astronomer.io).

## APC Pods stuck in CrashLoopBackOff

The APC API (API) connects directly to the control-plane database during startup. If the Pods restart repeatedly:

1. List Pods to verify their status:

   ```bash wrap theme={null}
   kubectl get pods -n <astronomer namespace>
   ```

2. Test connectivity to the database from inside the cluster:

   ```bash wrap theme={null}
   kubectl run psql --rm -it --restart=Never --namespace <astronomer namespace> \
     --image bitnami/postgresql --command -- \
     psql $(kubectl get secret -n <astronomer namespace> <platform-release-name>-houston-backend \
       --template='{{.data.connection | base64decode }}' | sed 's/?.*//g')
   ```

   If the connection times out, investigate networking or firewall rules between Kubernetes nodes and the Postgres host.

3. Confirm the `astronomer-bootstrap` secret contains the correct connection string:

   ```bash wrap theme={null}
   kubectl get secret astronomer-bootstrap -n <astronomer namespace> -o yaml
   ```

   Decode the `connection` value and fix any typos. After updating the secret, delete the APC API and Grafana Pods so they pick up the change.

## X.509 "certificate signed by unknown authority" while pulling images

If image pulls fail with a certificate error, such as when syncing registry certificates, restart the APC Pods followed by the platform registry Pod. Ensure any custom certificate authorities are configured under `global.privateCaCerts` and applied via `helm upgrade`.

## APC Worker showing NATS timeout errors after installation

After installing or upgrading APC, you might encounter issues where Deployments appear in the Astro CLI and database, but their Kubernetes namespaces aren't created. APC API logs might show `UnhandledPromiseRejectionWarning: NatsError: TIMEOUT`.

This occurs when the NATS JetStream cluster hasn't yet elected a metadata leader before the APC Worker Pods attempt to set up streams and consumers.

To resolve:

1. Verify APC Worker Pods are showing NATS timeout errors:

```bash wrap theme={null}
kubectl logs -l component=houston-worker -n <astronomer namespace>
```

2. Restart the APC Worker Pods to allow them to reconnect after the NATS leader election completes:

```bash wrap theme={null}
kubectl rollout restart deployment <platform-release-name>-houston-worker -n <astronomer namespace>
```

3. Confirm Deployment namespaces are created:

```bash wrap theme={null}
kubectl get namespaces
```

After the APC Worker Pods restart, they successfully create the necessary Kubernetes resources for your deployments.

## APC Worker showing NatsError: 503 after installation

After installing or upgrading APC, the APC API and APC Worker Pods may start successfully but silently fail to connect to NATS JetStream. APC API logs might show repeated `UnhandledPromiseRejectionWarning: NatsError: 503` entries shortly after startup.

This occurs when the APC API and APC Worker start and attempt to initialize JetStream connections before the NATS JetStream subsystem has finished initializing. Both components call `jetstreamManager()` during startup, which requires JetStream to be fully ready — not just the NATS TCP port. When you make this API call during the JetStream initialization window, NATS returns a 503 "No Responders" error. Because neither component retries on 503, they continue running with broken or missing JetStream connections and silently drop all deployment events. As a result, Deployments may appear in the Astro CLI and database but their Kubernetes namespaces are never created.

To resolve:

1. Verify that all NATS Pods are healthy and JetStream has finished initializing:

   ```bash wrap theme={null}
   kubectl get pods -l app=<platform-release-name>-nats -n <astronomer namespace>
   ```

2. Wait until all NATS Pods report `READY` status before continuing. You can also inspect the NATS monitoring endpoint directly from inside the Pod to confirm JetStream is responding:

   ```bash wrap theme={null}
   kubectl exec -n <astronomer namespace> <platform-release-name>-nats-0 -- \
     curl -s http://localhost:8222/jsz
   ```

   The response should contain JetStream stream and consumer statistics. If the request fails or returns an error, wait and retry before proceeding.

3. Restart both the APC API and APC Worker Pods:

   ```bash wrap theme={null}
   kubectl rollout restart deployment \
     <platform-release-name>-houston \
     <platform-release-name>-houston-worker \
     -n <astronomer namespace>
   ```

4. Confirm that APC Worker has established active JetStream subscriptions:

   ```bash wrap theme={null}
   kubectl logs -l component=houston-worker -n <astronomer namespace> | grep -i "Running"
   ```

   You should see a `Running` log line for each of the eight JetStream worker subjects, for example, `NATS houston-upsert-deployment-for-create Running...`.

After both Pods restart and JetStream subscriptions are established, deployment operations resume normally.


# Debug an Astro Private Cloud 2.0 upgrade
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/debug-upgrade

Troubleshoot Astro Private Cloud Pods that aren't running after upgrading to 2.0.

Use this guide when your Astro Private Cloud (APC) control plane or data plane Pods aren't progressing to a healthy state after upgrading to 2.0.

## Helm upgrade fails with patch conflict

* **Cause**: Environment variable ordering in the APC Deployment changed between versions, causing Helm's strategic merge patch to fail.
* **Symptoms**: The upgrade fails with an error containing `cannot patch "astronomer-houston" with kind Deployment` and references to environment variable ordering.
* **Solution**: Delete the APC Deployment with `--cascade=orphan` to preserve running Pods, then retry the Helm upgrade:

```bash wrap theme={null}
kubectl delete deployment/<release-name>-houston -n astronomer --cascade=orphan
helm upgrade -f migrated-values.yaml -n astronomer astronomer astronomer/astronomer --version 2.0.x
```

## APC API crashes with "Configuration property isn't defined"

* **Cause**: The APC Docker image contains an older `default.yaml` configuration file that doesn't include keys added in 2.0. If the Helm ConfigMap doesn't explicitly set these new keys, the APC API fails at startup when it tries to read them.
* **Symptoms**: APC Pods enter `CrashLoopBackOff` with errors like:

```text wrap theme={null}
Error: Configuration property "deployments.runtimeManagement.astroRuntimeReleasesFile" is not defined
```

* **Solution**: Ensure the APC Docker image matches the chart version you are deploying. If you built a custom APC Docker image, rebuild it with the 2.0 codebase that includes the updated `config/default.yaml`. Then restart the APC Pods:

```bash wrap theme={null}
kubectl rollout restart deploy/<release-name>-houston
```

## JetStream Pods stuck in Pending or CrashLoopBackOff

* **Cause**: STAN volume claims or PVCs weren't deleted before the upgrade (applies when upgrading from 0.37.x).
* **Solution**: Delete existing PVCs for STAN:

```bash wrap theme={null}
kubectl delete pvc -l app.kubernetes.io/name=stan
```

## Prisma migrate deploy or database migration job times out

* **Cause**: Database locks or concurrent transactions, often because of Prometheus or monitoring jobs.
* **Solution**:
  * Check for locks:

    ```sql wrap theme={null}
    SELECT * FROM pg_locks pl JOIN pg_stat_activity psa ON pl.pid = psa.pid;
    ```

  * Terminate long-running transactions:

    ```sql wrap theme={null}
    SELECT pg_terminate_backend(pid)
    FROM pg_stat_activity
    WHERE state = 'active' AND now() - query_start > interval '5 minutes';
    ```

## APC Worker logs show NATS: connection timeout

* **Cause**: The APC API tries to connect to STAN before JetStream is ready (applies when upgrading from 0.37.x).
* **Solution**: Restart the APC Worker Pod to reconnect to JetStream:

```bash wrap theme={null}
kubectl rollout restart deploy/<release-name>-houston-worker
```

## Airflow 3 Pods fail with ModuleNotFoundError: `airflow.providers.cncf`

* **Cause**: Cluster configuration override missing or not applied.
* **Solution**:
  * Reapply the override function in the [Airflow 3 migration guide](/docs/astro-private-cloud/v-2-x/migrate-to-airflow-3).
  * Ensure `minimumAstroRuntimeVersion` is set to `3.1-2` or higher in the override config.

## Airflow Deployments fail with Postgres connection errors after upgrade due to missing database name

* **Cause**: The `astronomer-bootstrap` secret `connection` string doesn't include a database name suffix (for example, `/main` on AWS RDS or `/postgres` on AKS), causing connection failures. Verify the correct database name by logging in to your database.
* **Solution**:
  * Patch the `astronomer-bootstrap` secret so `connection` ends with `/<database_name>`, then run a Helm upgrade.
  * The secret is in the `astronomer` namespace (or the namespace where you install APC).
  * In control plane/data plane (CP/DP) mode, patch the secret in both the control plane and data plane clusters.

```bash wrap theme={null}
# Namespace where APC is installed
NAMESPACE=astronomer
# Database name suffix; for AWS the default database is usually "main", for AKS "postgres"
DB_NAME=main

# Read current connection string, append database name, and update the secret
CURRENT=$(kubectl -n "$NAMESPACE" get secret astronomer-bootstrap -o jsonpath='{.data.connection}' | base64 -d)
NEW="${CURRENT%/}/$DB_NAME"
kubectl -n "$NAMESPACE" patch secret astronomer-bootstrap --type=merge -p "{\"data\":{\"connection\":\"$(printf '%s' "$NEW" | base64 -w0)\"}}"

# Apply the update with a Helm upgrade
helm upgrade -f values.yaml -n "$NAMESPACE" astronomer astronomer/astronomer
```

## APC API migration fails due to duplicate workspace labels

* **Cause**: The upgrade includes a migration that adds a unique constraint to the `Workspace.label` column. If your database contains workspaces with duplicate labels, the migration fails with Prisma error `P3009`. Once the migration is marked as failed, all subsequent migrations are blocked, preventing the APC API from starting.
* **Symptoms**: APC database migration Pods fail with `Error: P3009` and the message `migrate found failed migrations in the target database`. APC API and worker Pods enter `CrashLoopBackOff`.
* **Solution**:

<Warning>
  Back up the APC database before you run `DELETE` on `_prisma_migrations` or rely on a retry of this migration.
</Warning>

<Steps>
  <Step title="Check for duplicate workspace labels">
    Connect to your APC database and check for duplicate workspace labels:

    ```sql wrap theme={null}
    SET search_path TO "houston$default";
    SELECT label, COUNT(*) FROM "Workspace" GROUP BY label HAVING COUNT(*) > 1;
    ```
  </Step>

  <Step title="Resolve the duplicate labels">
    Rename one of the duplicate workspace labels, using the workspace ID as a suffix to guarantee uniqueness:

    ```sql wrap theme={null}
    UPDATE "Workspace" SET label = label || '-' || id
    WHERE id = '<workspace-id-to-rename>';
    ```
  </Step>

  <Step title="Check whether the migration partially applied">
    Before clearing the failed Prisma record, check whether the unique constraint was applied:

    ```sql wrap theme={null}
    SELECT constraint_name FROM information_schema.table_constraints
    WHERE table_schema = 'houston$default'
      AND table_name = 'Workspace'
      AND constraint_type = 'UNIQUE';
    ```

    * If the unique constraint on `label` isn't listed, the migration didn't complete. After fixing duplicate labels, you can clear the failed migration and retry. Back up your database first, then delete only that failed migration row:

    ```sql wrap theme={null}
    DELETE FROM "houston$default"."_prisma_migrations"
      WHERE migration_name = '20250918072802_added_unique_constraint_to_ws_label';
    ```

    * If a unique constraint on `label` is present, the schema change may have been applied even though Prisma recorded a failure. Don't delete the `_prisma_migrations` row or re-run the migration without DBA or Astronomer support — you can create conflicting schema or migration state.
  </Step>

  <Step title="Re-run the Helm upgrade">
    ```bash wrap theme={null}
    helm upgrade -f migrated-values.yaml -n astronomer astronomer astronomer/astronomer --version 2.0.x
    ```
  </Step>
</Steps>

## Values migration script reports unexpected errors

* **Cause**: The Python migration script requires Python 3.10+ and the `ruamel.yaml` package.
* **Solution**:
  * Verify your Python version: `python3 --version`
  * Install or upgrade the dependency: `pip install ruamel.yaml`
  * Run the script in dry-run mode first to preview changes: `./bin/migrate-helm-chart-values-1x-to-2x.py --dry-run my-values.yaml`

## Airflow Deployments fail with Postgres connection errors after upgrade

* **Cause**: The `astronomer-bootstrap` secret `connection` string doesn't include a database name suffix (for example, `/postgres`), causing connection failures.
* **Solution**:
  * Patch the `astronomer-bootstrap` secret so `connection` ends with `/<database_name>`, then run a Helm upgrade.
  * In CP/DP mode, patch the secret in both the control plane and data plane clusters.

```bash wrap theme={null}
NAMESPACE=astronomer
DB_NAME=postgres

CURRENT=$(kubectl -n "$NAMESPACE" get secret astronomer-bootstrap -o jsonpath='{.data.connection}' | base64 -d)
NEW="${CURRENT%/}/$DB_NAME"
kubectl -n "$NAMESPACE" patch secret astronomer-bootstrap --type=merge -p "{\"data\":{\"connection\":\"$(printf '%s' "$NEW" | base64 -w0)\"}}"

helm upgrade -f migrated-values.yaml -n "$NAMESPACE" astronomer astronomer/astronomer
```

## Partially applied database migration

* **Cause**: Prisma migration interrupted during the upgrade.
* **Solution**:
  * Run migration manually:

    ```bash wrap theme={null}
    npx prisma migrate deploy
    ```

    * Verify the `Cluster` and `Deployment` tables exist.

## Vector Pods not running (upgrading from 0.37.x)

* **Cause**: The `fluentd` key wasn't renamed to `vector` in your migrated values file, or custom Fluentd resources were incompatible.
* **Solution**: Verify that your migrated values file uses `vector` (not `fluentd`) as the top-level key. Re-run the migration script if needed, or manually rename the key.


# Deploy code overview
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/deploy-code-overview

Learn about the available options for deploying code to APC.

Deploying code is the process of pushing code to an Astro Private Cloud (APC) Deployment. A code deploy can include an entire Astro project as a Docker image, or just the Dags in your project. APC supports several methods for deploying code to a Deployment:

* Deploy [project images](#full-image-deploy) or [Dags only](/docs/astro-private-cloud/v-2-x/deploy-dags) using the Astro CLI
* Deploy Dags using an [NFS volume](/docs/astro-private-cloud/v-2-x/deploy-nfs)
* Deploy Dags using [Git sync](/docs/astro-private-cloud/v-2-x/deploy-git-sync)

Use this document to understand each method and choose the right approach for your use case.

## Deployment methods comparison

| Method           | Use Case                                    | Downtime      | Dependencies | Requirements   |
| ---------------- | ------------------------------------------- | ------------- | ------------ | -------------- |
| CLI - Full Image | Production deployments with OS dependencies | Brief restart | Supports all | Docker, CLI    |
| CLI - Dag Only   | Quick Dag updates                           | None          | Dags only    | Docker, CLI    |
| NFS Volume       | Shared filesystem environments              | None          | Dags only    | NFS server     |
| Git Sync         | GitOps workflows                            | None          | Dags only    | Git repository |

## Astro CLI deploys

### Full image deploy

By default, deploy code by building it into a Docker image and pushing to the Astronomer Registry using the CLI.

This mechanism builds your Dags into a Docker image alongside all other files in your Astro project directory:

* Python packages (`requirements.txt`)
* OS-level packages (`packages.txt`)
* Dockerfile customizations
* Plugins and include files

The resulting image generates Docker containers for each Airflow component. Every time you run `astro deploy`, your project is rebuilt into a new image and containers are restarted.

```bash wrap theme={null}
# Deploy full project image
astro deploy <deployment-id>
```

#### When to use

* Python or OS dependency updates
* Custom Dockerfile modifications
* Initial deployment setup
* Major version changes

### Dag-only deploy

For faster iteration, enable [Dag-only deploys](/docs/astro-private-cloud/v-2-x/deploy-dags) to deploy just your `dags` directory without rebuilding the Docker image.

```bash wrap theme={null}
# Deploy Dags only (no image rebuild)
astro deploy <deployment-id> --dags
```

#### When to use

* Frequent Dag changes
* No dependency changes
* Development workflows
* Quick fixes

<Note>
  You still need Docker access to authenticate to APC before deploying Dags.
</Note>

## NFS volume-based Dag deploys

For teams deploying Dag changes frequently, APC supports [NFS volume-based](https://kubernetes.io/docs/concepts/storage/volumes/#nfs) Dag deploys.

Using this mechanism, deploy Dags by adding Python files to a shared filesystem on your network. Compared to image-based deploys, NFS enables:

* Zero downtime deployments
* Continuous deployment workflows
* Shared Dag storage across environments

```mermaid actions={true} theme={null}
flowchart TD
    A["NFS Server (/dags)"] --> B["Airflow Pods (read-only mount)"]
```

### When to use

* Enterprise environments with existing NFS infrastructure
* High-frequency Dag updates
* Shared Dag development across teams

#### Requirements

* NFS server accessible from Kubernetes cluster
* Platform-level configuration enabled
* Read access for UID/GID 50000

For configuration details, see [Deploy Dags via NFS volume](/docs/astro-private-cloud/v-2-x/deploy-nfs).

## Git-sync Dag deploys

For Git-based workflows, APC supports [git-sync](https://github.com/kubernetes/git-sync) deployments.

Configure a Git repository to sync with your Deployment. When you push changes to the repository, Dags automatically sync with no downtime.

```mermaid actions={true} theme={null}
flowchart TD
    A["Git Repository"] -->|poll or webhook| B["Git-Sync Relay"]
    B --> C["Airflow Pods"]
```

### Sync modes

* **Polling**: Periodically checks for changes (default: every 60 seconds)
* **Webhook**: Triggers sync on push events

#### When to use

* GitOps deployment workflows
* Version-controlled Dag management
* CI/CD pipeline integration
* Branch-based deployment strategies

For configuration details, see [Deploy Dags via git sync](/docs/astro-private-cloud/v-2-x/deploy-git-sync).

## Choose a deployment method

### Decision tree

```mermaid actions={true} theme={null}
flowchart TD
    A([Start]) --> B{Do you need to deploy\nPython/OS dependencies?}
    B -- Yes --> C([CLI Full Image Deploy])
    B -- No --> D{Do you have existing\nNFS infrastructure?}
    D -- Yes --> E([NFS Volume Deploy])
    D -- No --> F{Do you use\nGitOps workflows?}
    F -- Yes --> G([Git Sync Deploy])
    F -- No --> H([CLI Dag-Only Deploy])
```

### Combine methods

You can combine deployment methods:

* Use **CLI Full Image** for dependency updates
* Use **CLI Dag-Only** for quick Dag iterations
* Use **Git Sync** for automated production deployments

<Note>
  A Deployment uses one Dag deploy mechanism. When you configure NFS or Git Sync, Dags come only from that mechanism, and `astro deploy --dags` no longer applies. You can still deploy a new project image to the Deployment with `astro deploy --image`. See [Deploy a project image to a git-sync Deployment](/docs/astro-private-cloud/v-2-x/deploy-git-sync#deploy-a-project-image-to-a-git-sync-deployment).
</Note>

## CI/CD integration

All deployment methods integrate with CI/CD pipelines:

| Method         | CI/CD Approach                    |
| -------------- | --------------------------------- |
| CLI Full Image | `astro deploy` in pipeline        |
| CLI Dag-Only   | `astro deploy --dags` in pipeline |
| NFS Volume     | Copy files to NFS mount           |
| Git Sync       | Push to configured branch         |

## Related documentation

* [Deploy code via the CLI](#astro-cli-deploys)
* [Deploy Dags only](/docs/astro-private-cloud/v-2-x/deploy-dags)
* [Deploy Dags via NFS volume](/docs/astro-private-cloud/v-2-x/deploy-nfs)
* [Deploy Dags via git sync](/docs/astro-private-cloud/v-2-x/deploy-git-sync)


# Deploy Dags to Astro Private Cloud using the Astro CLI
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/deploy-dags

Learn how to enable and trigger Dag-only deploys on Astro Private Cloud.

Dag-only deploys are the fastest way to deploy code to Astro Private Cloud. They are recommended if you only need to deploy changes made to the `dags` directory of your Astro project.

When you configure this feature for a Deployment, you must still do a full project deploy when you make a change to any file in your Astro project that isn't in the `dags` directory, or when you [upgrade Astro Runtime](/docs/runtime/manage-airflow-versions).

Dag-only deploys have the following benefits:

* Dag-only deploys are faster than project deploys.
* Deployments pick up Dag-only deploys without restarting. This results in a more efficient use of workers and no downtime for your Deployments.
* If you have a CI/CD process that includes both Dag and image-based deploys, you can use your repository's permissions to control which users can perform which kinds of deploys.
* Dag deploys transmit significantly less data in most cases, which makes them quicker than image deploys when upload speeds to the Deployment are slow.

You can also configure user permissions so that they can roll back just Dags for Dag-only deploys. See [Granular rollback permissions](/docs/astro-private-cloud/v-2-x/deploy-rollbacks#granular-rollback-permissions).

<Danger>When you [update a Deployment](#configure-dag-only-deploys-on-a-deployment) to support Dag-only deploys, all Dags in your Deployment will be removed. To continue running your Dags, you must redeploy them using `astro deploy --dags`.</Danger>

## Prerequisites

* Astro CLI version 1.23 or later

## Enable on an Astronomer cluster

By default, Dag-only deploys are disabled for all Deployments on Astro Private Cloud. Enabling the feature requires two separate configuration changes:

1. Deploy the Dag deploy server infrastructure by adding `global.deployMechanisms.dagOnlyDeployment.enabled: true` to your `values.yaml` file:

   ```yaml wrap theme={null}
   global:
     deployMechanisms:
       dagOnlyDeployment:
         enabled: true
         serviceAccount:
           create: true
   ```

   Then, push the configuration change. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

2. Allow Dag-only deploys at the cluster level by setting `astronomer.houston.config.deployments.deployMechanisms.configureDagDeployment.enabled` to `true` in the **Configuration Override** section of the Astro UI, or through the `updateCluster` API:

   * In the APC UI, go to your **Clusters** page and select your cluster.

   * In the cluster details page, click **Edit**. In the **Cluster Deployments Configuration** YAML editor, click **Find** and search for the feature you want to override. Then add the following override in the appropriate YAML field:

     ```yaml wrap theme={null}
     astronomer:
       houston:
         config:
           deployments:
             deployMechanisms:
               configureDagDeployment:
                 enabled: true
     ```

     For details on using the UI for configuration, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).

   * Save and apply your changes in the UI.

<Warning>
  If you use a [third-party ingress controller](/docs/astro-private-cloud/v-2-x/third-party-ingress-controllers), you can't upload more than 8 MB of compressed Dags regardless of your Dag server size.
</Warning>

### Customize Dag deploy resources

When you enable Dag-only deploys on a given Deployment, Astro Private Cloud spins up a component in the Deployment called the *Dag deploy server*.
The default resources for the Dag deploy server are 100m CPU and 384 Mi of memory which allows you to push up to 15 MB of compressed Dags per deploy. To deploy more than 15 MB of compressed Dags at a time, increase the CPU and memory in the `resources` configuration by 1 CPU and 1.5 MB for each additional 15 MB of Dags you want to upload. For more information, see [How Dag-only deploys work](#trigger-a-dag-only-deploy).

```yaml wrap theme={null}
global:
  deployMechanisms:
    dagOnlyDeployment:
      enabled: true
      serviceAccount:
        create: true
      resources:
        requests:
          # Update values as required for your cluster
          cpu: "100m"
          memory: "256Mi"
        limits:
          # Update values as required for your cluster
          cpu: "500m"
          memory: "1024Mi"
```

### Dag-deploy service account

Astro Private Cloud automatically creates the necessary service account, role, and rolebindings using the [Dag-deploy role templates](https://github.com/astronomer/airflow-chart/tree/v1.15.5/templates/dag-deploy) in the airflow-chart when using the default configuration settings:

```yaml wrap theme={null}
global:
  deployMechanisms:
    dagOnlyDeployment:
      enabled: true
      serviceAccount:
        create: true
```

If `serviceAccount.create` isn't set to `true`, you must provide your own custom service account templates. Otherwise, the Dag-server Pod fails to create. See [Bring your own Service Account](/docs/astro-private-cloud/v-2-x/byo-service-accounts) for steps about how to configure custom service account templates.

## Configure Dag-only deploys on a deployment

By default, Deployments are configured only for complete project image deploys. To enable Dag-only deploys:

1. Open your Deployment in the Astro Private Cloud UI.
2. In the **Settings** tab under the **Dag Deployment** section, change the **Mechanism** to **Dag Only Deployment**.
3. Click **Deploy changes**
4. Redeploy your Dags using `astro deploy --dags`. See [Trigger a Dag-only deploy](#trigger-a-dag-only-deploy). This step is required because all Dags in your deployed image will be deleted from your Deployment when you enable the feature.

## Trigger a Dag-only deploy

Run the following command to trigger a Dag-only deploy:

```sh wrap theme={null}
astro deploy --dags <deployment-id>
```

<Info>You can still run `astro deploy` to trigger a complete project deploy. When you do this, the Astro CLI builds all project files excluding Dags into a Docker image and deploys the image. It then deploys your Dags separately using the Dag deploy mechanism.</Info>

## Deploy Dag-only deploys programmatically

Astro Private Cloud Deployment includes a REST API endpoint that you can use to upload Dags programmatically.

### Prerequisites

Create a service account that has access to your Deployment and copy its associated API key. See \[Create a service account using the Astro Private Cloud UI]\(ci-cd#create-a-service-account-using-the-Astro Private Cloud-ui). Alternatively, go to `https://app.BASEDOMAIN/token` and copy the generated token to authenticate with your own user credentials.

### Setup

Your automated workflow must include the following two steps:

1. Create a `.tgz` file that contains only the `dags` folder of your Astro project. This file should be accessible from the rest of your automated process. For example, you can do this using the following command:

   ```sh wrap theme={null}
   tar -czf dags.tgz dags
   ```

2. Run a `POST` request to the endpoint `https://deployments.basedomain/<deployment-release-name>/dags/upload` to upload your `.tgz` file to your Deployment. For example, making the request using cURL would look similar to the following:

   ```sh wrap theme={null}
   curl --location 'https://deployments.basedomain/<deployment-release-name>/dags/upload' \
   --header 'Authorization: Bearer <your-service-account-token>' \
   --form 'dags.tar.gz=@<your-dags-tgz-location>'
   ```

## How Deployments handle code deploys

If you deploy Dags to a Deployment that is running a previous version of your code, then tasks that are `running` continue to run on existing workers and will only be terminated if they exceed the 24-hour timeout after the deploy of the Dag deploy.

These new workers execute downstream tasks of Dag runs that are in progress. For example, if you deploy to Astronomer when `Task A` of your Dag is running, `Task A` continues to run on an old Celery worker. If `Task B` and `Task C` are downstream of `Task A`, they are both scheduled on new Celery workers running your latest code.

This means that Dag runs could fail due to downstream tasks running code from a different source than their upstream tasks. Dag runs that fail this way need to be fully restarted from the Airflow UI so that all tasks are executed based on the same source code.


# Configure git-sync code deploys
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/deploy-git-sync

Push Dags to your Airflow Deployment on Astro Private Cloud using git-sync.

You can deploy Dags to an Astro Private Cloud Deployment using [git-sync](https://github.com/kubernetes/git-sync). After setting up this feature, you can deploy Dags from a Git repository without any additional CI/CD. Dags deployed with git-sync automatically appear in the Airflow UI without requiring additional action or causing downtime. You can also roll back images with the Astro Private Cloud UI and APC API.

<Note>`git-sync-relay` RWX volume doesn't work with `azurefile-csi`.</Note>

This guide provides details about setup options and the steps for configuring git-sync as a Dag deploy option.

## Choose a git-sync strategy

When you configure git-sync, you must choose both a **repo fetch mode** and a **repo share mode**:

* [**Repo fetch mode**](#repo-fetch-mode): How the git-sync relay retrieves changes from your Git repository. Choose between [poll mode](#poll-mode) or [webhook](#webhook).
* [**Repo share mode**](#repo-share-mode): How the git-sync relay distributes changes to Airflow Pods in the Deployment. Choose between [git-daemon](#git-daemon) or [shared volume](#shared-volume).

### Repo fetch mode

Repo fetch mode determines how the git-sync relay retrieves changes from your Git repository. Choose one of the following options:

#### Poll mode

The git-sync relay checks the remote Git repository for changes at regular intervals. Use poll mode for repositories with frequent changes across branches.

**Tradeoff**: Frequent polling generates unnecessary network traffic between your Deployment and the repository when changes are infrequent.

#### Webhook

The git-sync relay fetches changes only when a push event fires from the Git repository. Use Webhook mode for repositories that don't change frequently, to avoid unnecessary network traffic between your Deployment and the repository.

**Tradeoff**: If configured for a specific branch, the git-sync relay only downloads changes for that branch regardless of fetch mode. However, in Webhook mode, the webhook fires for every push event in the repository — not just pushes to the configured branch. This means a busy repository can still generate frequent webhook calls even when branch filtering is in place.

### Repo share mode

Repo share mode determines how the git-sync relay distributes the synced repository to Airflow Pods in the Deployment. Choose one of the following options:

#### `git-daemon`

A git-daemon container serves the repository within the namespace using the Git protocol on port 9418. The Airflow Deployment contains a git-sync relay Pod with both a git-sync container that stores the Git repository and a git-daemon container that serves the repository to the namespace.

**Tradeoff**: All Airflow containers must clone the repository at startup, which can cause significant network use with large repositories and increase startup time.

#### Shared volume

The git repository contents are stored on a ReadWriteMany (RWX) storage volume mounted into each Airflow Pod, which eliminates git clone activity between Pods. The git-sync relay Pod pulls from the external Git repository and writes to the RWX volume.

**Requirement**: An RWX-compatible StorageClass volume. RWX-compatible StorageClasses aren't included in standard Kubernetes. You must provision additional cloud infrastructure to support RWX volumes, and the configuration steps differ between cloud providers. See your cloud provider's documentation for details.

## Prerequisites

To enable the git-sync deploy feature, you need:

* An Astro Private Cloud installation that runs Deployments with the [Astronomer Runtime Helm chart](/docs/astro-private-cloud/v-2-x/airflow-chart-compatibility). This is the default for most installations.
* Permission to push new configuration changes to your Astro Private Cloud installation.
* (Shared volume mode) A ReadWriteMany (RWX) compatible StorageClass volume. RWX-compatible StorageClasses require additional cloud infrastructure that varies between providers. See your cloud provider's documentation for configuration steps.

To configure a git-sync deploy mechanism for a Deployment on APC, you need Workspace Editor permissions.

To deploy Dags to a Deployment using a git-sync deploy mechanism, you need permission to push code to a Git repository configured for git-sync deploys.

## Enable git-sync

Git-sync deploys must be explicitly selected in the UI for each Airflow Deployment, for both `git-daemon` and `shared-volume` repo share modes.

However, for the `shared-volume` mode, an APC Admin must configure the RWX shared volume storage class name, `storageClassName`, in the cluster configuration.

To do so:

1. In the APC UI, go to your **Clusters** page and select your cluster.

2. In the cluster details, click **Edit** and add the following override to the **Cluster Deployments Configuration** field, including the path to your RWX compatible storage:

   ```yaml wrap theme={null}
   deployMechanisms:
     gitSyncRelay:
       storageClassName: <your-RWX-storage>
     gitSyncDagDeployment:
       enabled: true
     configureDagDeployment:
       enabled: true
     repoShareMode: "shared-volume"
   ```

   For details on using the UI for configuration, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).

3. Save and apply your changes in the UI.

For `shared-volume` repo share mode, an APC Admin must also set the RWX shared-volume storage class name, `storageClassName`, under `deployMechanisms.gitSyncRelay`. This is a global default; the repo share mode itself is still chosen per Deployment in the UI.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        deployMechanisms:
          configureDagDeployment:
            enabled: true
          gitSyncDagDeployment:
            enabled: true
          gitSyncRelay:
            storageClassName: <your-RWX-storage>
```

## Configure your APC Deployment

Workspace editors can configure a new or existing Airflow Deployment to use a git-sync mechanism for Dag deploys. From there, any member of your organization with write permissions to the Git repository can deploy Dags to the Deployment. To configure a Deployment for git-sync deploys:

1. In the Astro Private Cloud UI, create a new Airflow Deployment or open an existing one.

2. Go to the **Dag Deployment** section of the Deployment's Settings page.

3. For your **Mechanism**, select **Git Sync.**

4. Configure the following values:

   * **Repository URL**: The URL for the Git repository that hosts your Astro project. Use an SSH URL (for example, `git@github.com:org/repo.git`).
   * **Branch Name**: The name of the Git branch that you want to sync with your Deployment
   * **Ssh Key**: The SSH private key for your Git repository
   * **Known Hosts**: The public key for your Git provider, which can be retrieved using `ssh-keyscan -t rsa <provider-domain>`. For an example of how to retrieve GitHub's public key, refer to [Apache Airflow documentation](https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#production-guide-knownhosts).
   * **Authentication Method**: (*APC 2.1 and later*) How git-sync authenticates to the repository. Choose one of:
     * **SSH**: Clone over SSH using a private key — the same **Ssh Key** and **Known Hosts** fields described previously.
     * **HTTPS + Personal Access Token**: Clone a private repository over HTTPS using a personal access token. Requires **Personal Access Token** and, optionally, **HTTPS Username**. Set **Repository URL** to an `https://` URL instead of an SSH URL. See [Authenticate to a private repository over HTTPS](#authenticate-to-a-private-repository-over-https).
     * **HTTPS (public repository)**: Clone a public repository over HTTPS with no credentials. Set **Repository URL** to an `https://` URL.
   * **HTTPS Username**: (*APC 2.1 and later; HTTPS + Personal Access Token only*) Optional. The username paired with your personal access token. Most providers (GitHub, GitLab) accept any non-empty username when a token is supplied; some (for example, Azure DevOps) require a specific username.
   * **Personal Access Token**: (*APC 2.1 and later; HTTPS + Personal Access Token only*) The token used to clone your repository. The token is stored write-only and is never displayed after saving.
   * **Sync Interval**: The time interval between checks for updates in your Git repository, in seconds. A sync is only performed when an update is detected. Astronomer recommends a minimum interval of 60 seconds.
   * **Dags Directory**: The directory in your Git repository that hosts your Dags. Specify the directory's path as relative to the repository's root directory. To use your root directory as your Dags directory, specify this value as `./`. Other changes outside the Dags directory in your Git repository must be deployed using `astro deploy`.
   * **Rev**: The commit reference of the branch that you want to sync with your Deployment
   * **Repo Fetch Mode**: Choose **Poll** or **WebHook**. If you select **WebHook**, you need the **Webhook URL** and **Webhook Secret Key** for your [GitHub Configuration](#configure-a-git-repository-for-git-sync-deploys).
   * **Webhook URL**: (*Webhook mode only*)
   * **Webhook Secret Key**: (*Webhook mode only*)
   * **Ephemeral Storage Overwrite Gigabytes**: The storage limit for your Git repository. If your Git repository is larger than 2 GB, Astronomer recommends setting this slider to your repository size + 1 Gi
   * **Sync Timeout**: The maximum amount of seconds allowed for a sync. Astronomer recommends increasing this value if your repository is larger than 1 GB

5. (Webhook Only) You can now open your GitHub repository and set up a [Repository Webhook](https://docs.github.com/en/webhooks/using-webhooks/creating-webhooks#creating-a-repository-webhook), or you can return to your Deployment details page to configure this later. Be sure to set the following configurations:
   * **Payload URL**: Paste the **Webhook URL** from the Astro Private Cloud UI
   * **Content Type**: Select **JSON**.
   * **Secret**: Paste the **Webhook Secret Key** from the Astro Private Cloud UI
   * **Enable SSL verification**
   * Choose **Just the push event** for the event trigger

6. Save your changes.

<Warning>
  **Repo Share Mode - First deploy error**

  If you complete your Deployment configuration for git-sync and encounter an error during the first Deployment, you might need to force restart the Airflow Deployment at least once, several minutes after you initially create it. For example, you can add any new environment variable to your Deployment, like `FOO=foo`, to force the Deployment containers to restart.

  After you see your Dags update in the Airflow UI, you can remove the environment variable.
</Warning>

After you configure your Deployment, any code pushes to your Dag directory of the specified Git repository and branch will appear in your Deployment with zero downtime.

<Tip>
  Newly created Dag files can take up to five minutes (default configuration) from syncing to appear in the Airflow UI. To shorten this delay, we recommend tuning `AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL` in your Airflow deployment.
</Tip>

Starting in APC 2.1, the Deployment's status indicator in the Astro Private Cloud UI reflects git-sync health alongside overall Deployment health, so you can tell at a glance whether git-sync is syncing successfully without opening the **Logs** tab.

## Configure a Git repository for git-sync deploys

The Git repository you want to sync should contain a directory of Dags that you want to deploy to APC. You can include additional files in the repository, such as your other Astro project files, but note that this might affect performance when deploying new changes to Dags.

To deploy Dags from a private Git repository, add a deploy key to your Git repository and provide the matching private key as the **Ssh Key** so your APC Deployment can access the repository. This process varies slightly between Git repository management tools. For an example, read GitLab's [SSH Key](https://docs.gitlab.com/ee/user/ssh.html) documentation.

Starting in APC 2.1, you can authenticate to a private repository over HTTPS with a personal access token instead of SSH. See [Authenticate to a private repository over HTTPS](#authenticate-to-a-private-repository-over-https).

## Authenticate to a private repository over HTTPS

<Note>
  **Astro Private Cloud 2.1**

  This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
</Note>

Use the **HTTPS + Personal Access Token** authentication method to clone a private repository over HTTPS — useful for enterprise Git hosts such as GitHub Enterprise, self-managed GitLab, or Bitbucket, or where SSH (port 22) isn't available.

1. In your Git provider, create a personal access token (PAT) with read access to the repository (for example, GitHub's `repo` / `read:repository` scope).
2. In the Deployment's **Dag Deployment** settings, set **Authentication Method** to **HTTPS + Personal Access Token**.
3. Set **Repository URL** to the repository's `https://` clone URL.
4. Enter the **Personal Access Token**. Optionally set **HTTPS Username** if your provider requires a specific username; otherwise leave it blank.
5. Save your changes.

The token is stored securely and is never returned by the APC API or shown in the UI after saving. To rotate the token, open the Deployment settings, replace it, and save again.

<Tip>
  You can validate the entered credentials before saving with the **Validate credentials** button in the Deployment settings. The check runs from the target data plane, so it reflects the network path git-sync actually uses.
</Tip>

## Trust a private certificate authority (CA)

<Note>
  **Astro Private Cloud 2.1**

  This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
</Note>

If your Git host presents a TLS certificate signed by a private or internal certificate authority — common for self-managed GitHub Enterprise, GitLab, or Bitbucket — git-sync must trust that CA, or HTTPS clones fail TLS verification. Astro Private Cloud never disables certificate verification.

To make git-sync trust your private CA:

1. Create a Kubernetes Secret in the platform namespace (for example, `astronomer`) containing the CA certificate under the key `cert.pem`, and annotate it so `commander` replicates it into your Deployment namespaces:

   ```bash wrap theme={null}
   kubectl create secret generic my-private-ca \
     --from-file=cert.pem=/path/to/ca.crt \
     --namespace astronomer
   kubectl annotate secret my-private-ca \
     --namespace astronomer \
     astronomer.io/commander-sync=""
   ```

2. Reference the Secret in your APC API configuration. Houston emits `deployments.privateCaCertSecretNames` as `global.privateCaCerts` in the Airflow chart values, and the git-sync container trusts the listed CAs via `update-ca-certificates`. Setting the platform-level `global.privateCaCerts` to the same Secret lets the control-plane credential-validation check trust the host as well:

   ```yaml wrap theme={null}
   global:
     # Platform components (including the commander credential-validation check) trust this CA.
     privateCaCerts:
       - my-private-ca
   astronomer:
     houston:
       config:
         deployments:
           # git-sync Deployments trust this CA when cloning over HTTPS.
           privateCaCertSecretNames:
             - my-private-ca
   ```

<Note>
  The Secret name must be added to `privateCaCertSecretNames` for each cluster that runs git-sync Deployments. Add a git-sync CA to the existing `global.privateCaCerts` list rather than replacing it — Helm replaces list values across `-f` files, so dropping an existing platform CA can break other platform TLS.
</Note>

## Deploy a project image to a git-sync Deployment

git-sync updates only the Dags in your Deployment. Python packages, OS-level packages, and the other files in your Astro project stay in the Deployment image. To update them, deploy a new image. The Deployment keeps git-sync as its Dag deploy mechanism.

Use one of the following methods:

* Build a new image and push it with the Astro CLI:

  ```bash wrap theme={null}
  astro deploy <deployment-id> --image
  ```

  The `--image` flag deploys only the image. The CLI doesn't deploy the Dags from your Astro project, because git-sync provides them.

* Point the Deployment at an image that is already in a registry:

  ```bash wrap theme={null}
  astro deploy <deployment-id> --image-name <host>/<image-name>:<tag> --remote --runtime-version <runtime-version>
  ```

  This command calls the `updateDeploymentImage` mutation of the APC API. Deploys to a [custom registry](/docs/astro-private-cloud/v-2-x/custom-image-registry) and direct API calls use the same mutation. See [Update a Deployment image with the APC API](/docs/astro-private-cloud/v-2-x/houston-update-deployment-image).

### Enable the update deployment image endpoint

APC disables the `updateDeploymentImage` endpoint by default. When the endpoint is disabled, an image update returns the following error:

```text theme={null}
Error: Update deployment image endpoint is disabled. It can be enabled via configuration change.
```

To enable the endpoint, add the following values to your `values.yaml` file:

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        deploymentImagesRegistry:
          updateDeploymentImageEndpoint:
            enabled: true
```

Then push the configuration change. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

This setting doesn't change the Dag deploy mechanism of a Deployment. The Deployment continues to sync Dags from your Git repository.

## Add Kubernetes scheduling configurations for git-sync relay

You can add Kubernetes scheduling configurations — `tolerations`, `nodeSelector`, and `affinity` — to your global git-sync relay configuration. These configurations allow you to:

* Specify node selection criteria with `nodeSelector`
* Configure Pod affinity and anti-affinity rules
* Set tolerations for tainted nodes

These settings can help you comply with security or compliance requirements for workload isolation, optimize resource utilization by co-locating related components, and handle tainted nodes in mixed-use Kubernetes clusters. They aren't required for git-sync relay functionality, so add `nodeSelector`, `affinity`, or `tolerations` only if you need specific node placement for your git-sync-relay components.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        helm:
          gitSyncRelay:
            nodeSelector:
              # your node selection criteria, for example:
              # disktype: ssd
            tolerations:
              # your toleration entries
            affinity:
              nodeAffinity:
                # your node affinity rules
```


# Deploy Dags with NFS
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/deploy-nfs

Configure NFS volume mounts to deploy Dags to Airflow Deployments on APC.

You can use an external [Network File System (NFS) Volume](https://kubernetes.io/docs/concepts/storage/volumes/#nfs) to deploy Dags to an Airflow Deployment on Astro Private Cloud (APC).

Unlike deploying Dags with the Astro CLI, deploying Dags to an NFS volume doesn't require rebuilding a Docker image and restarting your underlying Airflow service. When a Dag is added to an NFS volume, it automatically appears in the Airflow UI without requiring additional action or causing downtime.

## How NFS deploys work

When you configure an NFS volume for a Deployment:

1. APC validates the NFS location format (`SERVER_IP:PATH`).
2. APC creates a Kubernetes PersistentVolume (PV) pointing to your NFS server.
3. APC creates a PersistentVolumeClaim (PVC) bound to the PV.
4. The NFS volume is mounted read-only to the scheduler and workers at `/usr/local/airflow/dags`.
5. Dags are synced by writing files directly to your NFS server.

```text wrap theme={null}
NFS Server (/dags)
       │
       ▼
┌──────────────┐
│ Kubernetes   │
│ PV + PVC     │
└──────┬───────┘
       │
       ├──► Scheduler (/usr/local/airflow/dags)
       │
       └──► Workers (/usr/local/airflow/dags)
```

## Implementation considerations

<Warning>
  If you configure NFS for a Deployment, you can't use the Astro CLI or service accounts to deploy Dags to that Deployment. NFS becomes the exclusive deployment mechanism.
</Warning>

Before configuring NFS deploys:

* **Namespace pools limitation**: NFS deploys won't work if you use [namespace pools](/docs/astro-private-cloud/v-2-x/namespace-pools) and set `global.clusterRoles` to `false`. The NFS deploy feature requires creating PersistentVolumes, which are cluster-scoped resources.
* **Dags only**: NFS volumes deploy only Dags. To add Python dependencies or system packages, update your `requirements.txt` and `packages.txt` files and deploy using the CLI or CI/CD.
* **Airflow version**: NFS volumes require Airflow 2.0 or later.
* **Read-only mount**: The NFS volume is mounted read-only to Airflow components. Write operations must happen directly on the NFS server.

## Prerequisites

* APC 2.0 or later installed
* An NFS server accessible from your Kubernetes cluster
* Network connectivity between cluster nodes and the NFS server
* Read access configured for UID/GID `50000` on the NFS share

## Enable NFS volume storage

A System Admin must enable NFS deploys at the cluster level:

1. In the APC UI, go to your **Clusters** page and select your cluster.

2. In the cluster details page, click **Edit**. In the **Cluster Deployments Configuration** YAML editor, click **Find** and search for the feature you want to override. Then add the following override in the appropriate YAML field:

   ```yaml wrap theme={null}
   deployMechanisms:
     nfsMountDagDeployment:
       enabled: true
     configureDagDeployment:
       enabled: true
   ```

   For details on using the UI for configuration, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).

3. Save and apply your changes in the UI.

## Provision an NFS volume

<Tabs>
  <Tab title="AWS (EFS)">
    1. Create an [EFS file system](https://docs.aws.amazon.com/efs/latest/ug/getting-started.html).
    2. Configure security groups to allow NFS traffic (port 2049) from your EKS nodes.
    3. Create an access point or use the root directory.
    4. Note the file system DNS name: `fs-xxxxxxxx.efs.region.amazonaws.com`.
  </Tab>

  <Tab title="GCP (Filestore)">
    1. Create a [Filestore instance](https://cloud.google.com/filestore/docs/creating-instances).
    2. Configure firewall rules to allow traffic from your GKE nodes.
    3. Create a file share directory for Dags.
    4. Configure [IP-based access control](https://cloud.google.com/filestore/docs/creating-instances#configuring_ip-based_access_control) for UID/GID 50000.
    5. Note the instance IP address.
  </Tab>

  <Tab title="Azure (Azure Files)">
    1. Create a [Premium File Storage account](https://docs.microsoft.com/en-us/azure/storage/files/storage-files-how-to-create-nfs-shares).
    2. Create an NFS file share.
    3. Configure network access from your AKS cluster.
    4. Note the mount address: `<storage-account>.file.core.windows.net:/<storage-account>/<share-name>`.
  </Tab>

  <Tab title="Self-managed">
    For on-premises or self-managed NFS servers:

    1. Ensure the NFS server exports a directory with appropriate permissions.
    2. Configure `/etc/exports` to allow access from Kubernetes node IPs.
    3. Set ownership to UID/GID 50000 or configure `no_root_squash` appropriately.

    Example `/etc/exports`:

    ```text wrap theme={null}
    /srv/airflow-dags 10.0.0.0/8(rw,sync,no_subtree_check,all_squash,anonuid=50000,anongid=50000)
    ```
  </Tab>
</Tabs>

## Configure NFS for a Deployment

<Tabs>
  <Tab title="UI">
    1. In the APC UI, create a new Deployment or open an existing one.
    2. Go to **DAG Deployment** in the Deployment settings.
    3. Select **NFS Volume Mount** as the mechanism.
    4. Enter the NFS location in `IP:PATH` format:
       * AWS EFS: `10.0.0.1:/`
       * GCP Filestore: `10.0.0.1:/dags`
       * Azure Files: `storage-account.file.core.windows.net:/storage-account/share-name`
    5. Click **Save** or **Deploy Changes**.
  </Tab>

  <Tab title="CLI">
    Create a new Deployment with NFS:

    ```bash wrap theme={null}
    astro deployment create my-deployment \
      --nfs-location "192.168.0.1:/dags"
    ```

    Update an existing Deployment to use NFS:

    ```bash wrap theme={null}
    astro deployment update <deployment-id> \
      --nfs-location "192.168.0.1:/dags"
    ```
  </Tab>

  <Tab title="API">
    Use `upsertDeployment` to configure a Deployment's Dag deployment mechanism as an NFS volume mount:

    ```graphql wrap theme={null}
    mutation {
      upsertDeployment(
        workspaceUuid: "<workspace-uuid>"
        label: "my-deployment"
        dagDeployment: {
          type: volume
          nfsLocation: "192.168.0.1:/dags"
        }
      ) {
        id
        releaseName
      }
    }
    ```
  </Tab>
</Tabs>

## Deploy Dags to NFS volume

Once configured, deploy Dags by copying files to your NFS server. The method depends on your infrastructure:

### Direct copy

```bash wrap theme={null}
# Copy DAGs to NFS mount point
cp -r dags/* /mnt/nfs/dags/
```

### Use kubectl

If you have a pod with NFS access:

```bash wrap theme={null}
kubectl cp dags/ deployment-namespace/nfs-sync-pod:/dags/
```

### CI/CD integration

Example GitHub Actions workflow:

```yaml wrap theme={null}
name: Deploy DAGs to NFS
on:
  push:
    branches: [main]
    paths: ['dags/**']

jobs:
  deploy:
    runs-on: self-hosted  # Runner with NFS access
    steps:
      - uses: actions/checkout@v4
      - name: Sync DAGs
        run: |
          rsync -av --delete dags/ /mnt/nfs/airflow-dags/
```

### Sync from cloud storage

For cloud-native workflows, sync from object storage:

```bash wrap theme={null}
# AWS S3 to EFS
aws s3 sync s3://my-bucket/dags/ /mnt/efs/dags/

# GCS to Filestore  
gsutil -m rsync -r gs://my-bucket/dags/ /mnt/filestore/dags/

# Azure Blob to Azure Files
azcopy sync "https://account.blob.core.windows.net/dags" "/mnt/azure/dags"
```

## Verify NFS configuration

Check that the PV and PVC were created:

```bash wrap theme={null}
# List PersistentVolumes
kubectl get pv | grep dags

# List PersistentVolumeClaims in the deployment namespace
kubectl get pvc -n <deployment-namespace> | grep dags
```

Verify the volume is mounted in Airflow Pods:

```bash wrap theme={null}
kubectl exec -n <deployment-namespace> <scheduler-pod> -- \
  ls -la /usr/local/airflow/dags
```

## Troubleshooting

### Dags aren't appearing

1. **Check NFS connectivity**:

   ```bash wrap theme={null}
   kubectl exec -n <namespace> <pod> -- \
     showmount -e <nfs-server-ip>
   ```

2. **Verify mount permissions**:

   ```bash wrap theme={null}
   kubectl exec -n <namespace> <scheduler-pod> -- \
     ls -la /usr/local/airflow/dags
   ```

3. **Check PV/PVC status**:

   ```bash wrap theme={null}
   kubectl describe pv <deployment>-dags-<hash>
   kubectl describe pvc -n <namespace> <deployment>-dags-<hash>
   ```

### Permission denied errors

Ensure your NFS export allows access for UID/GID 50000:

```bash wrap theme={null}
# On NFS server
chown -R 50000:50000 /srv/airflow-dags
chmod -R 755 /srv/airflow-dags
```

### Stale file handle

If pods report stale NFS handles after server restart:

```bash wrap theme={null}
# Restart affected pods
kubectl rollout restart deployment -n <namespace> <scheduler>
kubectl rollout restart statefulset -n <namespace> <workers>
```

### Network connectivity issues

Verify NFS port (2049) is accessible:

```bash wrap theme={null}
kubectl run nfs-test --rm -it --image=busybox -- \
  nc -zv <nfs-server-ip> 2049
```

## Security considerations

* **Network isolation**: Use network policies to restrict which pods can access the NFS server.
* **Access control**: Configure NFS exports to allow only Kubernetes node IPs.
* **Read-only mounts**: APC mounts NFS volumes read-only to prevent accidental modifications from Airflow.
* **Audit logging**: Enable NFS server audit logging for compliance requirements.

## Alternative: Git-sync deploys

If NFS infrastructure isn't available, consider [git-sync deploys](/docs/astro-private-cloud/v-2-x/deploy-git-sync) which pull Dags from a Git repository. Git-sync provides:

* Version control for Dags.
* No external storage infrastructure required.
* Webhook-based or polling synchronization.
* Branch-based deployment strategies.


# Roll back an image deploy on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/deploy-rollbacks

Learn how to roll back to a deploy, which lets you run earlier version of your project code.

Deploy rollbacks are an emergency option if a Deployment unexpectedly stops working after a recent deploy. For example, if one of your Dags worked in development but suddenly fails in a mission-critical production Deployment, you can roll back to your previous deploy to quickly get your pipeline running again. This allows you to revert your deployment to a known good state while you investigate the cause of the failure. You can roll back to any deploy in the last three months regardless of your Dag code or Deployment settings. However, rollbacks are only supported for Runtime 5.0.0 and above or its equivalent Airflow version, 2.3.0 and above.

## Rollback version compatibility

Starting in Astro Private Cloud 1.1, rollbacks are supported across Airflow 3 and Airflow 2 Runtime versions.

The following rollback paths are now available:

| Rollback path          | Supported versions                                                       |
| ---------------------- | ------------------------------------------------------------------------ |
| Airflow 3 to Airflow 3 | Both source and target must be Astro Runtime 3.1-2 or later              |
| Airflow 3 to Airflow 2 | From Astro Runtime 3.1-2 or later to Runtime 12.x or 13.x (Airflow 2.10) |
| Airflow 2 to Airflow 2 | Astro Runtime 2.3.0 and later                                            |

<Warning>
  Astro Runtime versions 3.0-1 through 3.1-1 aren't supported for automated rollbacks. If either the source or target version is in this range, automated rollback isn't available.
</Warning>

### Airflow 3 to Airflow 2 rollback considerations

When rolling back from Airflow 3 to Airflow 2:

* **Dag compatibility**: Dags written for Airflow 3 aren't compatible with Airflow 2. Ensure you roll back your Dags as well, or revert any Airflow 3-specific code changes before rolling back.
* **Longer rollback time**: Cross-major version rollbacks take longer than same-version rollbacks due to database schema migrations.
* **UI warnings**: The UI displays prominent warnings when attempting cross-major version rollbacks to ensure you understand the implications.

## Prerequisites

* You must be at least a Deployment Editor or otherwise have the permission `deployment.images.push` or `deployment.dags.push` or both to roll back images and/or Dags, respectively, for a Deployment.
  * If you want to limit users to either rollback images or Dags, you can configure [granular rollback permissions](/docs/astro-private-cloud/v-2-x/apply-platform-config) for more information on how to modify these values.

## Enable deploy rollbacks

If deploy rollbacks are configured for your installation, you can enable them in individual Deployments through the Deployment Settings page.

1. In the Astro Private Cloud UI, open your Deployment.
2. On the **Settings** page, toggle **Rollback Deploy** to **Enable**.

## Roll back to a deploy

<Danger>Astronomer recommends triggering Deployment rollbacks only as a last resort for recent deploys that aren't working as expected. Deployment rollbacks can be disruptive, especially if you triggered multiple deploys between your current version and the rollback version. See [What happens during a deploy rollback](#what-happens-during-a-deploy-rollback) before you trigger a rollback to anticipate any unexpected effects.</Danger>

1. In the Astro Private Cloud UI, open your Deployment, then go to **Deploy History**.
2. Find the deploy you want to roll back to in the **Deploys** table, then click **Deploy**.
3. Write a description for why you're triggering the rollback, then confirm the rollback.

### What happens during a deploy rollback

A deploy rollback is a new deploy of a previous version of your code. This means that the rollback deploy appears as a new deploy in **Deploy History**, and the records for any deploys between your current version and rollback version are still preserved. In Git terms, this is equivalent to `git revert`.

When you trigger a rollback, the following information is rolled back:

* All project code, including Dags.
* Your Astro Runtime version. Note that if you roll back to a restricted Runtime version, it might include major bugs and performance issues.
* Your Deployment's Dag deploy setting.

The following information isn't rolled back:

* Your Deployment's resource configurations, such as executor and scheduler configurations.
* Your Deployment's environment variable values.
* Any other Deployment settings that you configure through the Astro UI, such as your Deployment name and description.
* For Runtime version downgrades, any data related to features that aren't available in the rollback version are erased from the metadata database and not recoverable.

Logs related to the rollback are exported to Elasticsearch.

## Granular rollback permissions

You might want to enable your users to roll back Dag-only or image-only deploys, without granting them the ability to roll back complete project deploys. This allows you to maintain higher security measures, by limiting who has permission to roll back both image and Dag deploys, or only one of the two.

* `deployment.images.push`: Grants permission to roll back just the image of a revision in an imaged-based, Dag-only, and gitSync deployment.
* `deployment.dags.push`: Grants permission to roll back only Dags for Dag-only deploys.

Users with both permissions can roll back both images and Dags. If users attempt to roll back a deploy and have limited permissions, they can only choose the type of deploy to roll back that they have permission for.

Follow the procedure in [Customize role permissions](/docs/astro-private-cloud/v-2-x/manage-platform-users#customize-role-permissions) to configure these user permissions.


# Disable outbound email
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/disable-outbound-email

Configure Astro Private Cloud to disable outbound email

You can configure Astro Private Cloud (APC) to not send outbound email.

<Info>Setting `astronomer.houston.config.publicSignups: true` with `astronomer.houston.config.email.enabled: false` is only secure when all non-OIDC authentication backends are explicitly disabled and the OIDC provider provides sufficient user validation to prevent untrusted users from accessing APC.</Info>

To disable email transmission and email verification of users attempting to access the platform:

1. In your `values.yaml` file, set `astronomer.houston.config.email.enabled` to `false`.
2. Set `astronomer.houston.config.publicSignups` to `true`.
3. Remove the `EMAIL__SMTP_URL` list-item from `astronomer.houston.secret`.


# Disable management of quotas and limit ranges
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/disable-quotas-limitranges

Prevent Astro Private Cloud from managing Kubernetes resource quotas and limit ranges.

If you install Astro Private Cloud into an environment where administrators have enforced strict resource allocations, you might be restricted to a set of namespaces and can't adjust quotas or limit ranges. If the deployment orchestrator runs with roles that don't include permissions to access cluster limit ranges or quotas, it can result in Airflow Deployment failures when the APC API tries to pass `extra_capacity` requirements. To avoid this, you can use the APC API config, `disableManageResourceQuotasAndLimitRanges` set to `true` to disable passing `extra_capacity`.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        disableManageResourceQuotasAndLimitRanges: true
```

<Warning>When setting `disableManageResourceQuotasAndLimitRanges: true`, all Pods running in the APC application will be able to consume unlimited resources on the nodes that they are scheduled on, which can lead to cluster-wide instability. It is strongly advised that users manage their own `resourceQuotas` and `limitRanges` when setting `disableManageResourceQuotasAndLimitRanges: true`.</Warning>


# Set environment variables on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/environment-variables

Manage environment variables on Astro Private Cloud.

You can use environment variables on Astronomer to set both [Airflow configurations](https://airflow.apache.org/docs/stable/configurations-ref.html) or custom values, which Astro then applies to your Airflow Deployment either locally or in your Astro environment.

Environment variables can be used to set any of the following:

* SMTP to enable email alerts
* Airflow parallelism and Dag concurrency
* [A secrets backend](/docs/astro-private-cloud/v-2-x/secrets-backend) to manage your Airflow connections and variables
* Store Airflow connections and variables
* Customize your default Dag view in the Airflow UI (Tree, Graph, Gantt etc.)

This guide covers the following:

* How to set environment variables on Astronomer
* How environment variables are stored on Astronomer
* How to store Airflow connections and variables as environment variables

## Set environment variables on Astronomer

On Astronomer, there are 3 ways to set environment variables:

* via your `.env` file (*Local Only*)
* via your `Dockerfile`
* via the Astro Private Cloud UI

Read the following instructions on how to configure them.

<Note> While environment variables on Astronomer are the equivalent of updating your `airflow.cfg`, you can't bring your own `airflow.cfg` file on Astronomer and configure it directly.</Note>

### Use `.env` (*Local Only*)

You can use the [Astro CLI](/docs/cli/v1.43/overview) to import environment variables from the `.env` file that `astro dev init` automatically generates when you initiate an Astro project.

To add environment variables locally,

1. Find your `.env` file in your Astro project directory
2. Add your environment variables of choice to that `.env` file
3. Rebuild your image to apply those changes by running `astro dev start --env .env`

In your `.env` file, insert the environment variable `value` and `key`, ensuring all-caps for all characters. For example:

```text wrap theme={null}
AIRFLOW__CORE__DAG_CONCURRENCY=5
```

<Note> If your environment variables contain secrets you don't want to expose in plain-text, you can add your `.env` file to `.gitignore` if and when you deploy these changes to your version control tool.</Note>

#### Confirm your environment variables were applied

By default, Airflow environment variables are hidden in the Airflow UI for local environments. To confirm your environment variables in the Airflow UI for a local environment, set `AIRFLOW__WEBSERVER__EXPOSE_CONFIG=True` in either your Dockerfile or `.env` file (local only).

Alternatively, you can run:

```text wrap theme={null}
docker ps
```

This outputs three Docker containers that run Airflow's primary components on your machine: The Airflow scheduler, API server, and Postgres metadata database.

Now, create a [Bash session](https://docs.docker.com/engine/reference/commandline/exec/#examples) in your scheduler container by running:

```sh wrap theme={null}
docker exec -it <scheduler-container-name> /bin/bash
```

If you run `ls -1` following this command, it returns a list of running files:

```sh wrap theme={null}
bash-5.0$ ls -1
Dockerfile             airflow.cfg            airflow_settings.yaml  dags                   include                logs                   packages.txt           plugins                requirements.txt       unittests.cfg
```

Now, run:

```sh wrap theme={null}
env
```

This prints all environment variables that are running locally, some of which are set by you and some of which are set by Astronomer by default.

<Tip> You can also run `cat airflow.cfg` to output *all* contents in that file. </Tip>

#### Multiple .env files

The CLI looks for `.env` by default. But, if you want to specify multiple files, make `.env` a top-level directory and create sub-files within that folder.

Your project might look like the following:

```text wrap theme={null}
my_project
├── Dockerfile
├── dags
│   └── my_dag
├── plugins
│   └── my_plugin
├── airflow_settings.yaml
└── .env
    ├── dev.env
    └── prod.env
```

### Use your `Dockerfile`

If you work on an Astro project locally, but intend to deploy to Astronomer and want to commit your environment variables to your source control tool, you can set environment variables in your `Dockerfile`. This file was automatically created when you first initialized your Astro project on Astronomer (via `astro dev init`).

Because you commit the Dockerfile upstream, Astronomer strongly recommends withholding environment variables that contain sensitive credentials, and instead inserting them via your `.env` file locally, while adding the `Dockerfile` to your `.gitignore`. Or, you can set environment variables as 'secret' via the Astro Private Cloud UI, as described in [Using the Astro Private Cloud UI](#use-the-astro-private-cloud-ui).

To add environment variables, insert the value and key in your `Dockerfile` beginning with `ENV`, ensuring all-caps for all characters. With your Airflow image commonly referenced as a "FROM" statement at the top, your Dockerfile might look like this:

```docker wrap theme={null}
FROM quay.io/astronomer/astro-runtime:7.1.0
ENV AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG=1
ENV AIRFLOW__CORE__DAG_CONCURRENCY=5
ENV AIRFLOW__CORE__PARALLELISM=25
```

After you add your environment variables:,

* Run `$ astro dev stop` and `$ astro dev start` to rebuild your image and apply your changes locally OR
* Run `$ astro deploy` to apply your changes to your running Airflow Deployment on Astronomer

<Note>
  Environment variables injected via the `Dockerfile` are mounted at build time and can be referenced in any other processes run during the docker build process that immediately follows `astro deploy` or `astro dev start`.

  Environment variables applied via the Astro Private Cloud UI only become available after the docker build process completes.
</Note>

### Use the Astro Private Cloud UI

You can also add environment variables via the Astro Private Cloud UI. For environment variables that you *only* need on Astronomer and not locally, Astronomer recommends using this method.

1. Navigate to the Astro Private Cloud UI
2. Go to **Deployment** > **Variables**
3. Add your environment variables.

Input for all configurations officially supported by Airflow are pre-templated, but you can also specify your own values.

#### Mark environment variables as secret

On Astronomer, you can mark any environment variable as **Secret** in the UI. If your environment variables contain potentially sensitive information, like SMTP password or your S3 bucket information, Astronomer recommends leveraging this feature.

In the **Variables** Tab:

1. Enter a **Key**
2. Enter a **Value**
3. Check **Secret?**
4. Click **Add**
5. Select **Deploy Changes**

After Astro deploys changes, environment variables marked as **Secret** aren't available in plain-text to any user in the Workspace.

A few additional notes:

* Workspace Editors and Admins can set an existing non-secret environment variable to **Secret** at any time
* To convert a **Secret** environment variable to a non-secret environment variable, Astro prompts you to enter a new value
* If you export environment variables through JSON, **Secret** values won't render in plain-text
* You can't add a new variable that has the same key as an existing variable

The following sections provide more detail about how environment variables are encrypted on Astronomer.

<Tip> Workspace roles and permissions apply to actions in the **Variables** tab. For a full breakdown of permissions for each role, reference Astronomer's [Roles and Permissions](/docs/astro-private-cloud/v-2-x/role-permission-reference).</Tip>

### Precedence between methods

Because you can set environment variables using multiple different methods, potentially simultaneously, Astro applies a precedence to each.

Astronomer applies and overrides environment variables in the following order:

1. Astro Private Cloud UI
2. .env (*Local Only*)
3. Dockerfile
4. Default Airflow Values (`airflow.cfg`)

If you set `AIRFLOW__CORE__PARALLELISM` with one value via the Astro Private Cloud UI, and you set the same Environment Variable with another value in your `Dockerfile`, the value set in the Astro Private Cloud UI takes precedence.

### How Astronomer stores environment variables

APC stores all values for environment variables that you add via the UI as a [Kubernetes Secret](https://kubernetes.io/docs/concepts/configuration/secret/), which is encrypted at rest and mounted to your Deployment's Airflow Pods (scheduler, API server or webserver, and workers) as soon as they're set or changed.

Environment variables are *not* stored in Airflow's metadata Database and are *not* stored in Astronomer's platform database. Unlike other components, the Astronomer APC API fetches environment variables from the Kubernetes Secret instead of the platform's database to render them in the Astro Private Cloud UI.

## Add Airflow connections and variables as environment variables

If you regularly use Airflow connections and variables, Astronomer recommends storing and fetching them using environment variables.

Airflow connections and variables are stored in Airflow's metadata database. Adding them *outside* of task definitions and operators requires an additional connection to Airflow's Postgres database, which is called every time the scheduler parses a Dag (as defined by `processor_poll_interval`, which is set to 1 second by default). By adding connections and variables as environment variables, you can refer to them more easily in your code and lower the amount of open connections, preventing a strain on your database and resources.

### Airflow connections

The Environment Variable naming convention for Airflow connections is:

```text wrap theme={null}
ENV AIRFLOW_CONN_<CONN_ID>=<connection-uri>
```

Using the following Airflow connection example:

* Connection ID: `MY_PROD_DB`
* Connection URI: `my-conn-type://login:password@host:5432/schema`

The full environment variable reads:

```text wrap theme={null}
ENV AIRFLOW_CONN_MY_PROD_DB=my-conn-type://login:password@host:5432/schema
```

You can set this environment variable via an `.env` file locally, via your Dockerfile, or via the Astro Private Cloud UI. For more information on how to generate your Connection URI, refer to [Airflow's documentation](https://airflow.apache.org/docs/stable/howto/connection/index.html#generating-connection-uri).

### Airflow variables

The environment variable naming convention for Airflow variables is:

```text wrap theme={null}
ENV AIRFLOW_VAR_<VAR_NAME>=Value
```

For the following Airflow variable example:

* Variable Name: `My_Var`
* Value: `2`

This environment variable reads:

```text wrap theme={null}
ENV AIRFLOW_VAR_MY_VAR=2
```


# Install the Astro Private Cloud control plane
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/install-control-plane

Deploy the Astro Private Cloud control plane on a dedicated Kubernetes cluster.

Use this guide to deploy the Astro Private Cloud (APC) control plane with the Helm-based Astronomer platform charts. The control plane hosts central management services such as the APC UI, APC API, monitoring coordination, and authentication.

<Info>If your organization needs time to issue TLS certificates, configure DNS, or approve firewall changes, review the data plane installation guide in parallel. You can request the control plane and data plane prerequisites at the same time so the clusters are ready when the infrastructure tickets close.</Info>

## Overview

Astro Private Cloud supports two deployment patterns:

* **Split plane**: Deploy a dedicated control plane using this guide, then provision one or more data planes with [Install a Data Plane](/docs/astro-private-cloud/v-2-x/install-data-plane). This separation keeps management services isolated from workload execution and lets you scale each plane independently.
* **Unified mode**: Run both control plane and data plane services inside a single cluster using [Install in Unified Mode](/docs/astro-private-cloud/v-2-x/install-unified). Unified mode is useful for labs or proofs-of-concept but concentrates failures and resource usage.

Choose the pattern that matches your reliability and compliance requirements. After selecting a pattern, determine how many APC environments you need. An environment refers to the pairing of control plane and its associated data planes. In unified mode, this maps to a single Kubernetes cluster, whereas in split mode, the environment is the combination of the control plane and all the registered data plane Kubernetes clusters.

Each APC environment can host multiple Airflow Deployments, potentially on multiple data planes. Common types include:

* **Sandbox**: The lowest environment that contains no sensitive data, used only by system-administrators to experiment, and not subject to change control.
* **Development**: User-accessible environment that is subject to most of the same restrictions of higher environments, with relaxed change control rules.
* **Staging**: All network, security, and patch versions are maintained at the same level as in the production environment. However, it provides no availability guarantees and includes relaxed change control rules.
* **Production**: The production instance hosts your production Airflow environments. You can choose to host development Airflow environments here or in environments with lower levels of support and restrictions.

Create a project folder for every environment you plan to host to contain its configuration files. For example, if you want to install a development environment, create a folder named `~/astronomer-dev/control-plane`.

<Info>Certain files in the project directory might contain secrets when you set up your sandbox or development environments. For your first install, keep these secrets in a secure place on a suitable machine. As you progress to higher environments, such as staging or production, secure these files separately in a vault and use the remaining project files in your directory to serve as the basis for your CI/CD deployment.</Info>

## Prerequisites

<Tabs>
  <Tab title="EKS on AWS">
    <Info>The following prerequisites apply when running Astro Private Cloud on Amazon EKS. See the **Other** tab if you run a different version of Kubernetes on AWS.</Info>

    * An EKS Kubernetes cluster, running a version of Kubernetes certified as compatible on the [Kubernetes Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/kubernetes-version-support) that provides the following components:
      * The [Amazon EBS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) (or an alternative CSI) must be installed on the Kubernetes cluster.
      * An AWS Load Balancer Controller for the IP target type is required for all private Network Load Balancers (NLBs). See [Installing the AWS Load Balancer Controller add-on](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html).
    * A PostgreSQL instance, accessible from your Kubernetes cluster, and running a version of Postgres certified as compatible on the [Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
    * PostgreSQL superuser permissions.
    * Permission to create and modify resources on AWS.
    * Permission to generate a certificate that covers a defined set of subdomains.
    * An SMTP service and credentials. For example, Mailgun or Sendgrid.
    * A machine meeting the following criteria with access to the Kubernetes API Server:
      * The [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html).
      * (Optional) [`eksctl`](https://eksctl.io/) for creating and managing your Astronomer cluster on EKS.
      * Network access to the Kubernetes API Server - either direct access or VPN.
      * Network access to load-balancer resources that are created when Astro Private Cloud is installed later in the procedure - either direct access or VPN.
      * Configured to use the DNS servers where Astro Private Cloud DNS records can be created.
      * [Helm (minimum v3.6)](https://helm.sh/docs/intro/install).
      * The [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
    * (Situational) The [OpenSSL CLI](https://www.openssl.org/docs/man1.0.2/man1/openssl.html) might be required to troubleshoot certain certificate-related conditions.
  </Tab>

  <Tab title="GKE on GCP">
    <Info>The following prerequisites apply when running Astro Private Cloud on Google GKE. See the **Other** tab if you run a different version of Kubernetes on GCP.</Info>

    * A GKE Kubernetes cluster, running a version of Kubernetes listed as compatible on the [Kubernetes Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/kubernetes-version-support).
    * A PostgreSQL instance, accessible from your Kubernetes cluster, and running a version of Postgres certified as compatible on the [Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
    * PostgreSQL superuser permissions.
    * Permission to create and modify resources on Google Cloud Platform.
    * Permission to generate a certificate that covers a defined set of subdomains.
    * An SMTP service and credentials. For example, Mailgun or Sendgrid.
    * A machine that meets the following criteria with access to the Kubernetes API Server:
      * [Google Cloud SDK](https://cloud.google.com/sdk/install).
      * Network access to the Kubernetes API Server - either direct access or VPN.
      * Network access to load-balancer resources that are created when Astro Private Cloud is installed later in the procedure - either direct access or VPN.
      * Configured to use the DNS servers where Astro Private Cloud DNS records can be created.
      * [Helm with minimum version 3.6](https://helm.sh/docs/intro/install).
      * The [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
    * (Situational) The [OpenSSL CLI](https://www.openssl.org/docs/man1.0.2/man1/openssl.html) might be required to troubleshoot certain certificate-related conditions.
  </Tab>

  <Tab title="AKS on Azure">
    <Info>The following prerequisites apply when running Astro Private Cloud on Azure AKS. See the **Other** tab if you run a different version of Kubernetes on Azure.</Info>

    * A Kubernetes cluster, running a version of Kubernetes listed as compatible on the [Kubernetes Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/kubernetes-version-support).
    * A PostgreSQL instance, accessible from your Kubernetes cluster, and running a version of Postgres certified as compatible on the [Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
      * If your organization uses Azure Database for PostgreSQL as the database backend, you need to enable the `pg_trgm` extension using the Azure portal or the Azure CLI before you install Astro Private Cloud. If you don't enable the `pg_trgm` extension, the install fails. For more information about enabling the `pg_trgm` extension, see [PostgreSQL extensions in Azure Database for PostgreSQL - Flexible Server](https://docs.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions).
    * PostgreSQL superuser permissions.
    * Permission to create and modify resources on Azure.
    * Permission to generate a certificate that covers a defined set of subdomains.
    * An SMTP service and credentials. For example, Mailgun or Sendgrid.
    * A machine meeting the following criteria with access to the Kubernetes API Server:
      * The [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest).
      * Network access to the Kubernetes API Server - either direct access or VPN.
      * Network access to load-balancer resources created when Astro Private Cloud is installed later in the procedure - either direct access or VPN.
      * Configured to use the DNS servers where Astro Private Cloud DNS records are created.
      * [Helm (minimum v3.6)](https://helm.sh/docs/intro/install).
      * The [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
    * (Situational) The [OpenSSL CLI](https://www.openssl.org/docs/man1.0.2/man1/openssl.html) might be required to troubleshoot certain certificate-related conditions.
  </Tab>

  <Tab title="Other">
    The following prerequisites apply when running Astro Private Cloud on Kubernetes.

    * A Kubernetes cluster. For versioning considerations, see [Kubernetes Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/kubernetes-version-support).
    * A PostgreSQL instance accessible from your Kubernetes cluster. For versioning considerations, see [Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
    * PostgreSQL superuser permissions.
    * An SMTP service and credentials. For example, Mailgun or Sendgrid.
    * Permission to generate a certificate that covers a defined set of subdomains.
    * The ability to create DNS records.
    * A machine with access to the Kubernetes API Server meeting the following criteria:
      * Network access to the Kubernetes API Server - either direct access or VPN.
      * Network access to load-balancer resources created when Astro Private Cloud is installed later in the procedure - either direct access or VPN.
      * Configured to use the DNS servers where Astro Private Cloud DNS records are created.
      * [Helm (minimum v3.6)](https://helm.sh/docs/intro/install).
      * The [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
    * (Situational) The [OpenSSL CLI](https://www.openssl.org/docs/man1.0.2/man1/openssl.html) might be required to troubleshoot certain certificate-related conditions.
  </Tab>
</Tabs>

<a />

### Ingress controller considerations

Astro Private Cloud requires a Kubernetes Ingress controller to function and provides an integrated Ingress controller by default.
Before installing, decide whether to use a third-party ingress controller or the Astronomer integrated ingress controller.

Astronomer generally recommends you use the integrated Ingress controller, but Astro Private Cloud also supports certain third-party [ingress-controllers](/docs/astro-private-cloud/v-2-x/third-party-ingress-controllers).

Ingress controllers typically need elevated permissions, including a `ClusterRole`, to function. Specifically, the Astro Private Cloud Ingress controller requires the ability to:

* List all namespaces in the cluster.
* View ingresses in the namespaces.
* Retrieve secrets in the namespaces to locate and use private TLS certificates that service the ingresses.

If you have complex regulatory requirements, you might need to use an Ingress controller that's approved by your organization and disable the Astronomer integrated controller. You configure the Ingress controller during the installation.

<a />

## Step 1: Create `values.yaml` from a template

Astro Private Cloud uses Helm to apply platform-level configurations. Choose your cloud provider tab below to copy a ready-to-use `values.yaml`, then customize the template to your requirements.

<Warning>
  As you work with the template configuration, keep the following in mind.

  * Don't make any changes to this file until instructed to do so in later steps.
  * **Don't** run `helm upgrade` or `upgrade.sh` until instructed to do so in later steps.
  * Fully complete the installation in this guide before following any configuration instructions on other Astronomer documentation pages.
</Warning>

<Tabs>
  <Tab title="EKS on AWS">
    ```yaml expandable wrap theme={null}
    ###########################################
    ### Astronomer global configuration for EKS
    ###########################################
    global:
      # Installation mode for the control plane
      plane:
        mode: control

      # Base domain for all control plane subdomains exposed through ingress
      baseDomain: env.astronomer.your.domain

      # For development or proof-of-concept, you can use an in-cluster database.
      # This NOT supported in production.
      postgresql:
        enabled: false

      # Name of secret containing TLS certificate, change if not using "astronomer-tls"
      # tlsSecret: astronomer-tls

      # List of secrets containing the cert.pem of trusted private certification authorities
      # Example command: `kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
      # privateCaCerts:
      # - private-root-ca

      # Expose Postgres metrics for Prometheus to scrape
      # prometheusPostgresExporter:
      #   enabled: true

      # Enable sidecar logging by default
      logging:
        loggingSidecar:
          enabled: true

      # Database SSL configuration
      ssl:
        # Enable SSL connection to Postgres -- must be false if using in-cluster database
        enabled: true

    #########################
    ### Ingress configuration
    #########################
    # nginx:
      # Static IP address the nginx ingress should bind to
      # loadBalancerIP: ~

      # Set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
      # privateLoadBalancer: true

      # Dictionary of arbitrary annotations to add to the nginx ingress.
      # For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
      # Change to 'elb' if your node group is private and doesn't utilize a NAT gateway
      # ingressAnnotations: {service.beta.kubernetes.io/aws-load-balancer-type: nlb}

      # If all subnets are private, auto-discovery may fail.
      # You must enter the subnet IDs manually in the annotation below.
      # service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-id-1,subnet-id-2

    ################################
    ### Astronomer app configuration
    ################################
    astronomer:
      houston:
        upgradeDeployments:
          enabled: false

        # secret:
        # - envName: "EMAIL__SMTP_URL"  # Reference to the Kubernetes secret for SMTP credentials. Only required if email is used.
        #   secretName: "astronomer-smtp"
        #   secretKey: "connection"

        # Application configuration for Houston
        config:
          publicSignups: true  ## set to false immediately after initial system admin user created

          # Allowed user email domains for system level roles
          # allowedSystemLevelDomains: []

          # Default configuration for deployments.
          # Can be overridden on a per-data-plane basis.
          deployments:
            # Enable Airflow 3 deployments for clusters
            runtimeManagement:
              airflowV3:
                enabled: true

            # Allow deletions to immediately remove the database and namespace
            # deploymentLifecycle:
            #   hardDeleteDeployment:
            #     enabled: true

            # Allows you to set your release names
            # namespaceManagement:
            #   manualReleaseNames:
            #     enabled: true

            # Flag to enable using IAM roles (don't enter a specific role)
            # Enables the API for updating deployments
            # deploymentImagesRegistry:
            #   serviceAccountAnnotationKey: eks.amazonaws.com/role-arn
            #   updateDeploymentImageEndpoint:
            #     enabled: true

            # Required if dagOnlyDeployment is enabled
            # deployMechanisms:
            #   configureDagDeployment:
            #     enabled: true
            # upsertDeploymentEnabled: true

          # email:
          #   enabled: false
          #   reply: noreply@your.domain

          # User authentication mechanism. One of the following should be enabled.
          auth:
            github:
              # Allow users authenticate with Github, enabled by default
              enabled: false
            # local:
            #   # Allow users and passwords in the Houston database, disabled by default
            #   enabled: false
            openidConnect:
              # okta:
              #   enabled: false
              # microsoft:
              #   enabled: false
              # adfs:
              #   enabled: false
              # custom:
              #   enabled: false
              google:
                # Allow users to authenticate with Google, enabled by default
                enabled: false
    ```
  </Tab>

  <Tab title="GKE on GCP">
    ```yaml expandable wrap theme={null}
    ###########################################
    ### Astronomer global configuration for GKE
    ###########################################
    global:
      # Installation mode for the control plane
      plane:
        mode: control

      # Base domain for all control plane subdomains exposed through ingress
      baseDomain: env.astronomer.your.domain

      # For development or proof-of-concept, you can use an in-cluster database.
      # This NOT supported in production.
      postgresql:
        enabled: false

      # Name of secret containing TLS certificate, change if not using "astronomer-tls"
      # tlsSecret: astronomer-tls

      # List of secrets containing the cert.pem of trusted private certification authorities
      # Example command: `kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
      # privateCaCerts:
      # - private-root-ca

      # Expose Postgres metrics for Prometheus to scrape
      # prometheusPostgresExporter:
      #   enabled: true

      # Enable sidecar logging by default
      logging:
        loggingSidecar:
          enabled: true

      # Database SSL configuration
      ssl:
        # Enable SSL connection to Postgres -- must be false if using in-cluster database
        enabled: true

    #########################
    ### Ingress configuration
    #########################
    nginx:
      # Static IP address the nginx ingress should bind to
      # loadBalancerIP: ~

      # Set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
      # privateLoadBalancer: true

      # Dictionary of arbitrary annotations to add to the nginx ingress.
      # For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
      # required for azure load balancer post Kubernetes 1.24
      ingressAnnotations:
          service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/healthz"

    ################################
    ### Astronomer app configuration
    ################################
    astronomer:
      houston:
        upgradeDeployments:
          enabled: false

        # Application configuration for Houston
        config:
          publicSignups: true  ## set to false immediately after initial system admin user created

          # Allowed user email domains for system level roles
          # allowedSystemLevelDomains: []

          # Default configuration for deployments.
          deployments:
            # Enable Airflow 3 deployments for clusters
            runtimeManagement:
              airflowV3:
                enabled: true

            # Allow deletions to immediately remove the database and namespace
            # deploymentLifecycle:
            #   hardDeleteDeployment:
            #     enabled: true

            # Allows you to set your release names
            # namespaceManagement:
            #   manualReleaseNames:
            #     enabled: true

            # Flag to enable using IAM roles (don't enter a specific role)
            # Enables the API for updating deployments
            # deploymentImagesRegistry:
            #   serviceAccountAnnotationKey: iam.gke.io/gcp-service-account
            #   updateDeploymentImageEndpoint:
            #     enabled: true

            # Required if dagOnlyDeployment is enabled
            # deployMechanisms:
            #   configureDagDeployment:
            #     enabled: true
            # upsertDeploymentEnabled: true

          # email:
          #   enabled: false
          #   reply: noreply@your.domain

          # secret:
          # - envName: "EMAIL__SMTP_URL"  # Reference to the Kubernetes secret for SMTP credentials. Only required if email is used.
          #   secretName: "astronomer-smtp"
          #   secretKey: "connection"

          # User authentication mechanism. One of the following should be enabled.
          auth:
            github:
              # Allow users authenticate with Github, enabled by default
              enabled: false
            # local:
            #   # Allow users and passwords in the Houston database, disabled by default
            #   enabled: false
            openidConnect:
              # okta:
              #   enabled: false
              # microsoft:
              #   enabled: false
              # adfs:
              #   enabled: false
              # custom:
              #   enabled: false
              google:
                # Allow users to authenticate with Google, enabled by default
                enabled: false
    ```
  </Tab>

  <Tab title="AKS on Azure">
    ```yaml expandable wrap theme={null}
    ###########################################
    ### Astronomer global configuration for AKS
    ###########################################
    global:
      # Installation mode for the control plane
      plane:
        mode: control

      # Base domain for all control plane subdomains exposed through ingress
      baseDomain: env.astronomer.your.domain

      # For development or proof-of-concept, you can use an in-cluster database.
      # This NOT supported in production.
      postgresql:
        enabled: false

      # Name of secret containing TLS certificate, change if not using "astronomer-tls"
      # tlsSecret: astronomer-tls

      # List of secrets containing the cert.pem of trusted private certification authorities
      # Example command: `kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
      # privateCaCerts:
      # - private-root-ca

      # Expose Postgres metrics for Prometheus to scrape
      # prometheusPostgresExporter:
      #   enabled: true

      # Enable sidecar logging by default
      logging:
        loggingSidecar:
          enabled: true

      # Database SSL configuration
      ssl:
        # Enable SSL connection to Postgres -- must be false if using in-cluster database
        enabled: true

    #########################
    ### Ingress configuration
    #########################
    # nginx:
      # Static IP address the nginx ingress should bind to
      # loadBalancerIP: ~

      # Set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
      # privateLoadBalancer: true

      # Dictionary of arbitrary annotations to add to the nginx ingress.
      # For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
      # ingressAnnotations: {}

    ################################
    ### Astronomer app configuration
    ################################
    astronomer:
      houston:
        upgradeDeployments:
          enabled: false

        # Application configuration for Houston
        config:
          publicSignups: true  ## set to false immediately after initial system admin user created

          # Allowed user email domains for system level roles
          # allowedSystemLevelDomains: []

          # Default configuration for deployments.
          deployments:
            # Enable Airflow 3 deployments for clusters
            runtimeManagement:
              airflowV3:
                enabled: true

            # Allow deletions to immediately remove the database and namespace
            # deploymentLifecycle:
            #   hardDeleteDeployment:
            #     enabled: true

            # Allows you to set your release names
            # namespaceManagement:
            #   manualReleaseNames:
            #     enabled: true

            # Flag to enable using IAM roles (don't enter a specific role)
            # Enables the API for updating deployments
            # deploymentImagesRegistry:
            #   serviceAccountAnnotationKey: iam.gke.io/gcp-service-account
            #   updateDeploymentImageEndpoint:
            #     enabled: true

            # Required if dagOnlyDeployment is enabled
            # deployMechanisms:
            #   configureDagDeployment:
            #     enabled: true
            # upsertDeploymentEnabled: true

          # email:
          #   enabled: false
          #   reply: noreply@your.domain

          # secret:
          # - envName: "EMAIL__SMTP_URL"  # Reference to the Kubernetes secret for SMTP credentials. Only required if email is used.
          #   secretName: "astronomer-smtp"
          #   secretKey: "connection"

          # User authentication mechanism. One of the following should be enabled.
          auth:
            github:
              # Allow users authenticate with Github, enabled by default
              enabled: false
            # local:
            #   # Allow users and passwords in the Houston database, disabled by default
            #   enabled: false
            openidConnect:
              # okta:
              #   enabled: false
              # microsoft:
              #   enabled: false
              # adfs:
              #   enabled: false
              # custom:
              #   enabled: false
              google:
                # Allow users to authenticate with Google, enabled by default
                enabled: false
    ```
  </Tab>

  <Tab title="Other">
    ```yaml expandable wrap theme={null}
    #################################################################
    ### Astronomer global configuration for other types of Kubernetes
    #################################################################
    global:
      # Installation mode for the control plane
      plane:
        mode: control

      # Base domain for all control plane subdomains exposed through ingress
      baseDomain: env.astronomer.your.domain

      # For development or proof-of-concept, you can use an in-cluster database.
      # This NOT supported in production.
      postgresql:
        enabled: false

      # Name of secret containing TLS certificate, change if not using "astronomer-tls"
      # tlsSecret: astronomer-tls

      # List of secrets containing the cert.pem of trusted private certification authorities
      # Example command: `kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
      # privateCaCerts:
      # - private-root-ca

      # Expose Postgres metrics for Prometheus to scrape
      # prometheusPostgresExporter:
      #   enabled: true

      # Enable sidecar logging by default
      logging:
        loggingSidecar:
          enabled: true

      # Database SSL configuration
      ssl:
        # Enable SSL connection to Postgres -- must be false if using in-cluster database
        enabled: true

    #########################
    ### Ingress configuration
    #########################
    # nginx:
      # Static IP address the nginx ingress should bind to
      # loadBalancerIP: ~

      # Set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
      # privateLoadBalancer: true

      # Dictionary of arbitrary annotations to add to the nginx ingress.
      # For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
      # ingressAnnotations: {}

    ################################
    ### Astronomer app configuration
    ################################
    astronomer:
      houston:
        upgradeDeployments:
          enabled: false

        # Application configuration for Houston
        config:
          publicSignups: true  ## set to false immediately after initial system admin user created

          # Allowed user email domains for system level roles
          # allowedSystemLevelDomains: []

          # Default configuration for deployments.
          deployments:
            # Enable Airflow 3 deployments for clusters
            runtimeManagement:
              airflowV3:
                enabled: true

            # Allow deletions to immediately remove the database and namespace
            # deploymentLifecycle:
            #   hardDeleteDeployment:
            #     enabled: true

            # Allows you to set your release names
            # namespaceManagement:
            #   manualReleaseNames:
            #     enabled: true

            # Flag to enable using IAM roles (don't enter a specific role)
            # Enables the API for updating deployments
            # deploymentImagesRegistry:
            #   serviceAccountAnnotationKey: iam.gke.io/gcp-service-account
            #   updateDeploymentImageEndpoint:
            #     enabled: true

            # Required if dagOnlyDeployment is enabled
            # deployMechanisms:
            #   configureDagDeployment:
            #     enabled: true
            # upsertDeploymentEnabled: true

          # email:
          #   enabled: false
          #   reply: noreply@your.domain

          # secret:
          # - envName: "EMAIL__SMTP_URL"  # Reference to the Kubernetes secret for SMTP credentials. Only required if email is used.
          #   secretName: "astronomer-smtp"
          #   secretKey: "connection"

          # User authentication mechanism. One of the following should be enabled.
          auth:
            github:
              # Allow users authenticate with Github, enabled by default
              enabled: false
            # local:
            #   # Allow users and passwords in the Houston database, disabled by default
            #   enabled: false
            openidConnect:
              # okta:
              #   enabled: false
              # microsoft:
              #   enabled: false
              # adfs:
              #   enabled: false
              # custom:
              #   enabled: false
              google:
                # Allow users to authenticate with Google, enabled by default
                enabled: false
    ```
  </Tab>
</Tabs>

<Info>Email delivery is disabled by default. If you want to enable it, you can configure it in a later step: [Configure outbound SMTP email](#configure-outbound-smtp-email).</Info>

<Info>The snippets in this section leave `astronomer.houston.config.publicSignups: true` so you can create the initial administrator account. You will lock down account creation in [Disable anonymous account creation](#disable-anonymous-account-creation).</Info>
<Warning>The snippets in this section don't enable any authentication mechanisms. You need to enable at least one mechanism to sign in as the first admin user.</Warning>

<a />

## Step 2: Choose and configure a base domain

When you install Astro Private Cloud, it creates a variety of services that your users access to manage, monitor, and run Airflow.

Choose a base domain such as `astronomer.example.com`, `astro-sandbox.example.com`, `astro-prod.example.internal` for which:

* You have the ability to create and edit DNS records
* You have the ability to issue TLS certificates
* The following hostnames are used by the Control Plane components:
  * `app.<base-domain>`
  * `houston.<base-domain>`
  * `alertmanager.<base-domain>`
  * `prometheus.<base-domain>`

The base domain itself doesn't need to be available and can point to another service not associated with Astronomer or Airflow.

When choosing a base domain, consider the following:

* The name you choose must be resolvable by both your users and Kubernetes itself.
* All data planes in the environment must be hosted as a sub-domain under this common base domain, for example `dp-01.<base-domain>`, so ensure you can create DNS records and issue TLS certificates for subdomains of this base domain.
* You need to have or obtain a TLS certificate that is recognized as valid by your users. If your organization hosts a registry for APC images, ensure the TLS certificate is trusted by Kubernetes as well.
* Wildcard certificates are only valid one level deep. For example, an ingress controller that uses a certificate called `*.example.com` can provide service for `app.example.com` but not `app.astronomer-dev.example.com`.
* The bottom-level sub-domains, such as `app` and `prometheus`, are fixed and can't be changed.

The base domain is visible to end users. They can view the base domain in the following scenarios:

* When users access the Astro Private Cloud UI. For example, `https://app.sandbox-astro.example.com`.
* When users authenticate to the Astro CLI. For example, `astro login sandbox-astro.example.com`.

<Info>If you install Astro Private Cloud on OpenShift and also want to use OpenShift's integrated ingress controller, you can use the hostname of the default OpenShift ingress controller as your base domain, such as `app.apps.<OpenShift-domain>`. Doing this requires permission to reconfigure the route admission policy for the standard ingress controller to `InterNamespaceAllowed`. See [Third Party Ingress Controller - Configuration notes for OpenShift](/docs/astro-private-cloud/v-2-x/third-party-ingress-controllers#required-environment-configuration-openshift) for additional information and options.</Info>

### Configure the base domain

Locate the `global.baseDomain` in your `values.yaml` file and change it to your base domain as shown in the following example:

```yaml wrap theme={null}
global:
  # Base domain for all subdomains exposed through ingress
  baseDomain: sandbox-astro.example.com
```

<a />

## Step 3: Create the Astro Private Cloud platform namespace

In your Kubernetes cluster, create a [Kubernetes namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) to contain the Astro Private Cloud platform. This guide refers to this namespace as `<apc platform namespace>` below. For example, if you chose `apc-cp` you would create the namespace as follows:

```bash wrap theme={null}
kubectl create namespace apc-cp
```

<a />

## Step 4: Request and validate an Astronomer TLS certificate

To install Astro Private Cloud, you need a TLS certificate that is valid for several domains. One of the domains is the primary name on the certificate, also known as the common name (CN). The additional domains are equally valid, supplementary domains known as Subject Alternative Names (SANs).

Astro Private Cloud requires a private certificate to be present in the Astro Private Cloud platform namespace, even if you use a third-party ingress controller that doesn't otherwise require it.

<a />

### Request an ingress controller TLS certificate

Request a TLS certificate for the control plane from your security team for Astro Private Cloud. In your request, include the following:

* Your chosen base domain as the Common Name (CN). If your certificate authority won't issue certificates for the bare base domain, use `app.<base-domain>` as the CN instead.
* *Either* request a wildcard SAN of `*.<base-domain>` (plus an explicit SAN for `<base-domain>`) *or* list each hostname individually:
  * `app.<base-domain>` (omit if already used as the Common Name)
  * `houston.<base-domain>`
  * `prometheus.<base-domain>`
  * `alertmanager.<base-domain>` (required if you keep the integrated Alertmanager enabled)

<Warning>Wildcards only cover a single DNS segment. You can't reuse a data plane wildcard such as `*.<domainPrefix>.<base-domain>` for the control-plane hosts (`app.<base-domain>`, `houston.<base-domain>`, and so on); request a certificate that explicitly matches the control-plane names listed earlier.</Warning>

* Request the following return format:
  * A `key.pem` containing the private key in pem format
  * **Either** a `full-chain.pem` (containing the public certificate and additional certificates required to validate it, in pem format) **or** a bare `cert.pem` and explicit affirmation that there are no intermediate certificates and that the public certificate is the full chain.
  * **Either** the `private-root-ca.pem` in pem format of the private Certificate Authority used to create your certificate or a statement that the certificate is signed by a public Certificate Authority.

### Validate the received certificate and associated items

Ensure that you received each of the following three items:

* A `key.pem` containing the private key in pem format.
* **Either** a `full-chain.pem`, in pem format, that contains the public certificate and additional certificates required to validate it **or** a bare `cert.pem` and explicit affirmation that there are no intermediate certificates and that the public certificate is the full chain.
* **Either** the `private-root-ca.pem` in pem format of the private Certificate Authority used to create your certificate **or** a statement that the certificate is signed by public Certificate Authority.

To validate that your security team generated the correct certificate, run the following command using the `openssl` CLI:

```bash wrap theme={null}
openssl x509 -in  <your-certificate-filepath> -text -noout
```

This command generates a report. If the `X509v3 Subject Alternative Name` section of this report includes either a single `*.<base-domain>` wildcard domain or all subdomains, then the certificate creation was successful.

Confirm that your full-chain certificate chain is ordered correctly. To determine your certificate chain order, run the following command using the `openssl` CLI:

```bash wrap theme={null}
openssl crl2pkcs7 -nocrl -certfile <your-full-chain-certificate-filepath> | openssl pkcs7 -print_certs -noout
```

The command generates a report of all certificates. Verify that the certificates are in the following order:

* Domain
* Intermediate (optional)
* Root

<a />

## Step 5: Store and configure the ingress controller TLS certificate

Determine whether or not your certificate was issued by an intermediate certificate-authority. If you don't know, assume you use an intermediate certificate and attempt to obtain a `full-chain.pem` bundle from your certificate authority.

Certificates issued by operators of root certificate authorities, including but not limited to LetsEncrypt, are frequently issued from intermediate certificate authorities associated with a trusted root CA.

<Warning>Astro Private Cloud backend services have stricter trust requirements than most web browsers. Web browsers might auto-complete the chain and consider your certificate valid, even if you don't provide the intermediate certificate-authority's public certificate. Astro Private Cloud backend services can reject the same certificate, and cause Dag and image deploys to fail.</Warning>

If, and only if, your certificate was issued directly by the root Certificate Authority of a universally trusted certificate authority, and not from one of their intermediaries, then the `server.crt` is also the full-chain certificate bundle.

Identify your full-chain public certificate `.pem` file and use it while storing and configuring the ingress controller TLS certificate.

Run the following command to store the public full-chain certificate in the Astro Private Cloud Platform Namespace in a `tls`-type Kubernetes secret. You can create a custom name for this secret. The following example uses the default name, `astronomer-tls`.

<Warning>The `--cert` parameter must reference your `full-chain.pem`, which includes the server certificate *and* any intermediate certificates, if any. Using the server cert directly causes Dag and image deploys to fail.</Warning>

```bash wrap theme={null}
kubectl -n <apc platform namespace> create secret tls astronomer-tls --cert <fullchain-pem-filepath> --key <your-private-key-filepath>
```

Astronomer recommends naming the secret `astronomer-tls` with no substitutions when using a third-party ingress controller. If you use another name for the secret, you must uncomment and update the `tlsSecret` in your `values.yaml` file.

<a />

## Step 6: (Optional) Configure a third-party ingress controller

Skip this step if the control plane will keep using Astronomer’s built-in ingress controller. Configure a custom ingress only when this cluster must integrate with your organization’s ingress stack. The data plane guide includes its own instructions for data plane ingress changes.

If you need a third-party controller, follow the provider-specific guidance in [Third-party Ingress-Controllers](/docs/astro-private-cloud/v-2-x/third-party-ingress-controllers) for the control plane cluster, then return here before continuing.

<a />

## Step 7: (Optional) Configure a private certificate authority

Skip this step if you don't use a private Certificate Authority (private CA) to sign the certificate used by your ingress-controller. Or, if you don't use a private CA for any of the following services that the Astro Private Cloud platform interacts with.

Astro Private Cloud trusts public Certificate Authorities automatically.

Astro Private Cloud must be configured to trust any private Certificate Authorities issuing certificates for systems Astro Private Cloud interacts with, including but not limited to:

* ingress controller
* email server, unless disabled
* any container registries that Kubernetes pulls from
* if using OAuth, the OAuth provider
* if using external elasticsearch, any external elasticsearch instances
* if using external Elasticsearch, any external Elasticsearch instances
* if using external Prometheus, any external Prometheus instances

Perform the procedure described in [Configuring private CAs](/docs/astro-private-cloud/v-2-x/configure-private-cas) for each certificate authority used to sign TLS certificates. After creating the trust secret (for example `astronomer-ca`), add it to `global.privateCaCerts` in `values.yaml` so platform components trust the issuer.

<Info>Astro CLI users must also configure both their operating system and container solution, [Docker Desktop or Podman](/docs/astro-private-cloud/v-2-x/configure-desktop-container-solution-extra-cas), to trust the private certificate Authority that was used to create the certificate used by the Astro Private Cloud ingress controller and any third-party container registries.</Info>

<a />

## Step 8: (Optional) Confirm your Kubernetes cluster trusts required CAs

Skip this step unless the Astro Private Cloud control plane will pull [platform container images](#configure-a-private-docker-registry-platform) from an external container registry that uses a certificate signed by a private CA.

Kubernetes must be able to pull images from one or more container registries for Astro Private Cloud to function. By default, Kubernetes only trusts publicly signed certificates. This means that by default, Kubernetes doesn't honor the list of certificates [trusted by the Astro Private Cloud platform](/docs/astro-private-cloud/v-2-x/configure-private-cas).

Many enterprises configure Kubernetes to trust additional certificate authorities as part of their standard cluster creation procedure. Contact your Kubernetes administrator to find out what, if any, private certificates are currently trusted by your Kubernetes cluster. Then, consult your Kubernetes administrator and Kubernetes provider's documentation for instructions on configuring Kubernetes to trust additional CAs.

Follow procedures for your Kubernetes provider to configure Kubernetes to trust each CA associated with your container registries.

Certain clusters don't provide a mechanism to configure the list of certificates trusted by Kubernetes.

While configuring the Kubernetes list of cluster certificates is a customer responsibility, Astro Private Cloud includes an optional component that can, for certain Kubernetes cluster configurations, add certificates defined in `global.privateCaCerts` to the list of certificates trusted by Kubernetes. Enable this by setting `global.privateCaCertsAddToHost.enabled` and `global.privateCaCertsAddToHost.addToContainerd` to `true` in your `values.yaml` file and setting `global.privateCaCertsAddToHost.containerdConfigToml` to:

```text wrap theme={null}
[host."https://<image registry hostname>"]
  ca = "/etc/containerd/certs.d/<image registry hostname>/<secret name>.pem"
```

For example, if your registry lives at `my-registry.example.com` and you store the CA certificate in a secret named `my-private-ca`, the `global.privateCaCertsAddToHost` section would be:

```yaml wrap theme={null}
  global:
    privateCaCertsAddToHost:
      enabled: true
      addToContainerd: true
      hostDirectory: /etc/containerd/certs.d
      containerdConfigToml: |-
        [host."https://my-registry.example.com"]
          ca = "/etc/containerd/certs.d/my-registry.example.com/my-private-ca.pem"
```

<a />

## Step 9: Configure outbound SMTP email

Astro Private Cloud requires the ability to send email to:

* Notify users of errors with their Airflow Deployments.
* Send emails to invite new users to Astro Private Cloud.
* Send certain platform alerts, enabled by default but can be configured.

Astro Private Cloud sends all outbound email using SMTP.

<Info>If SMTP isn't available in the environment where you're installing Astro Private Cloud, follow instructions in [configure Astro Private Cloud to not send outbound email](/docs/astro-private-cloud/v-2-x/disable-outbound-email), and then skip the rest of this section.</Info>

1. Obtain a set of SMTP credentials from your email administrator for you to use to send email from Astro Private Cloud. When you request an email address and display name, remember that these emails aren't designed for users to reply directly to them. Request all the following information:
   * Email address
   * Email display name requirements. Some email servers require a **From** line of: `Do Not Reply <donotreply@example.com>`.
   * SMTP username. This is usually the same as the email address.
   * SMTP password
   * SMTP hostname
   * SMTP port
   * Whether or not the connection supports TLS

<Info>If there is a `/` or any other escape character in your username or password, you may need to [URL encode](https://www.urlencoder.org/) those characters.</Info>

2. Ensure that your Kubernetes cluster has access to send outbound email to the SMTP server.

3. Change the configuration in `values.yaml` from `noreply@your.domain` to an email address that is valid to use with your SMTP credentials.

4. Construct an email connection string and store it in a secret in the Astro Private Cloud platform namespace. The following example shows how to store the connection in a secret called `astronomer-smtp` for a user `my@user` with a password `my@pass`. Make sure to *url-encode* the username and password if they contain special characters.

   ```sh wrap theme={null}
   kubectl -n <apc platform namespace> create secret generic astronomer-smtp --from-literal connection="smtp://my%40user:my%40pass@smtp.email.internal/?requireTLS=true"
   ```

   In general, an SMTP URI is formatted as `smtps://USERNAME:PASSWORD@HOST/?pool=true`. The following table contains examples of the URI for some of the most popular SMTP services:

   | Provider          | Example SMTP URL                                                                                 |
   | ----------------- | ------------------------------------------------------------------------------------------------ |
   | AWS SES           | `smtp://AWS_SMTP_Username:AWS_SMTP_Password@email-smtp.us-east-1.amazonaws.com/?requireTLS=true` |
   | SendGrid          | `smtps://apikey:SG.sometoken@smtp.sendgrid.net:465/?pool=true`                                   |
   | Mailgun           | `smtps://xyz%40example.com:password@smtp.mailgun.org/?pool=true`                                 |
   | Office365         | `smtp://xyz%40example.com:password@smtp.office365.com:587/?requireTLS=true`                      |
   | Custom SMTP-relay | `smtp://smtp-relay.example.com:25/?ignoreTLS=true`                                               |

   If your SMTP provider isn't listed, refer to the provider's documentation for information on creating an SMTP URI.

5. Ensure this secret is referenced in the `values.yaml` file via an entry in the `astronomer.houston.secret` list. For example:

   ```yaml wrap theme={null}
   astronomer:
     houston:
       secret:
       - envName: "EMAIL__SMTP_URL"  # Reference to the Kubernetes secret for SMTP credentials. Only required if email is used.
         secretName: "astronomer-smtp"
         secretKey: "connection"
   ```

## Step 10: Configure volume storage classes

Skip this step if a single default storage class is sufficient for every control plane component. Otherwise, set the fields below to point at the storage classes you want to use. Astronomer recommends solid-state storage for all volumes.

Key fields to review in `values.yaml`:

* `global.storageClass`: Fallback storage class for control plane components.
* `postgresql.persistence.storageClass`: Only required if you enable the bundled Postgres database (not recommended outside of testing environments).
* `prometheus.persistence.storageClassName`: Used by the control plane Prometheus when retaining metrics locally.
* `alertmanager.persistence.storageClassName`: Required if Alertmanager should keep state on disk.
* `nats.jetstream.fileStorage.storageClassName`: Only relevant if you enable JetStream persistence; most control plane deployments leave JetStream stateless.

Example: to point Prometheus at a custom storage class called `fast-storage`, add:

```yaml wrap theme={null}
prometheus:
  persistence:
    storageClassName: fast-storage
```

When you have the desired values, merge them into `values.yaml` manually or by using a YAML merge tool of your choosing.

<a />

## Step 11: Configure the database

Astro Private Cloud requires a central Postgres database that acts as the backend for Astro Private Cloud's APC API.

<Check>
  If, while evaluating Astro Private Cloud, you need to create a temporary environment where Postgres isn't available, locate the `global.postgresql.enabled` option already present in your `values.yaml` and set it to `true`, then skip the remainder of this step.

  Note that `global.postgresql.enabled` to `true` is an unsupported configuration, and should never be used on any development, staging, or production environment.
</Check>

<Info>
  If you use Azure Database for either PostgreSQL or another Postgres instance that doesn't enable the `pg_trgm` by default, you must enable the `pg_trgm` extension prior to installing Astro Private Cloud. If `pg_trgm` isn't enabled, the installation fails. `pg_trgm` is enabled by default on Amazon RDS and Google Cloud SQL for PostgresQL.

  For instructions on enabling the `pg_trgm` extension for Azure Flexible Server, see [PostgreSQL extensions in Azure Database for PostgreSQL - Flexible Server](https://docs.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions).
</Info>

Additional requirements apply to the following databases:

* AWS RDS:
  * [t2 medium](https://aws.amazon.com/rds/instance-types/) is the minimum RDS instance size you can use.
* Azure Flexible Server:
  * You must enable the `pg_trgm` extension as per the advisory earlier in this section.
  * Set `global.ssl.mode` to `prefer` in your `values.yaml` file.

Create a Kubernetes Secret in the namespace chosen for the install, named `astronomer-bootstrap`, that points to your database. You must URL encode any special characters in your Postgres password.

<Warning>The in-cluster Postgres option (`global.postgresql.enabled: true`) should only be used for short-lived testing. Always rely on an external Postgres instance for any persistent environment.</Warning>

<Warning>PostgreSQL usernames must be lowercase.</Warning>

To create this secret, run the following command replacing the APC platform namespace, username, password, database hostname, and database port with their respective values:

```bash wrap theme={null}
kubectl -n <apc platform namespace> create secret generic astronomer-bootstrap \
  --from-literal connection="postgres://<url-encoded username>:<url-encoded password>@<database hostname>:<database port>"
```

For example, for a username named `bob` with password `abc@abc` for the database `dbname` at hostname `some.host.internal`, you would run:

```bash wrap theme={null}
kubectl -n astronomer create secret generic astronomer-bootstrap \
  --from-literal connection="postgres://bob:abc%40abc@some.host.internal:5432/dbname"
```

<Warning>This secret must be named `astronomer-bootstrap` and must be present in the APC platform namespace before you install Astro Private Cloud.</Warning>

<a />

## Step 12: Configure the Docker registry used for platform images

Skip this step if you are installing Astro Private Cloud onto a Kubernetes cluster that can pull container images from public image repositories and you don't want to mirror these images locally.

<Info>
  Docker registry secrets will also need to be created in any data planes you register with this environment, which will be covered in the [data plane installation guide](/docs/astro-private-cloud/v-2-x/install-data-plane).
</Info>

<Tabs>
  <Tab title="Anonymous">
    If your registry can be reached without credentials, ensure the endpoint is restricted to trusted networks (for example private subnets or VPN access). Avoid exposing the platform image registry directly to the public internet. No additional APC configuration is required beyond setting the repository locations later in this step.
  </Tab>

  <Tab title="Amazon ECR">
    1. Grant your worker nodes or IRSA service accounts the IAM permissions required to pull images from the target ECR repository. At minimum, allow `ecr:GetAuthorizationToken`, `ecr:BatchCheckLayerAvailability`, `ecr:GetDownloadUrlForLayer`, and `ecr:BatchGetImage`.
    2. Ensure network access from the cluster to the appropriate ECR endpoints (for example, VPC endpoints or public ECR endpoints).
    3. Set the platform repository prefix in `values.yaml`. For example:

       ```yaml wrap theme={null}
       global:
         privateRegistry:
           enabled: true
           repository: <account-id>.dkr.ecr.<region>.amazonaws.com/<platform-prefix>
       astronomer:
         houston:
           config:
             deployments:
               helm:
                 runtimeImages:
                   airflow:
                     repository: <account-id>.dkr.ecr.<region>.amazonaws.com/<platform-prefix>/astro-runtime
                 runtimeImagesV3:
                   airflow:
                     repository: <account-id>.dkr.ecr.<region>.amazonaws.com/<platform-prefix>/runtime
                 airflow:
                   defaultAirflowRepository: <account-id>.dkr.ecr.<region>.amazonaws.com/<platform-prefix>/ap-airflow
                   defaultRuntimeRepository: <account-id>.dkr.ecr.<region>.amazonaws.com/<platform-prefix>/astro-runtime
       ```

       When you rely on IAM-based authentication, `global.privateRegistry.secretName` isn't required. If you use static credentials, create the matching Docker registry secret following the AWS ECR documentation and set `secretName` accordingly.
  </Tab>

  <Tab title="Other registries">
    1. Create a Docker registry secret in the APC platform namespace:

       ```bash wrap theme={null}
       kubectl -n <apc platform namespace> create secret docker-registry <secret-name> \
         --docker-server=<registry-host> \
         --docker-username=<username> \
         --docker-password=<password> \
         --docker-email=<email>
       ```

    2. Update `values.yaml` so the platform charts reference your registry and credentials:

       ```yaml wrap theme={null}
       global:
         privateRegistry:
           enabled: true
           repository: <custom-platform-repo-prefix>
           secretName: <secret-name>
       astronomer:
         houston:
           config:
             deployments:
               helm:
                 runtimeImages:
                   airflow:
                     repository: <custom-platform-repo-prefix>/astro-runtime
                 runtimeImagesV3:
                   airflow:
                     repository: <account-id>.dkr.ecr.<region>.amazonaws.com/<platform-prefix>/runtime
                 airflow:
                   defaultAirflowRepository: <custom-platform-repo-prefix>/ap-airflow
                   defaultRuntimeRepository: <custom-platform-repo-prefix>/astro-runtime
       ```
  </Tab>
</Tabs>

For additional examples (including per-deployment registry settings and air gapped workflows), see [Configure a custom registry for Deployment images](/docs/astro-private-cloud/v-2-x/custom-image-registry).

<a />

## Step 13: Determine which version of Astro Private Cloud to install

Astronomer recommends that new Astro Private Cloud installations use the most recent APC version available. Keep this version number available for the following steps. For a separate control plane and data plane topology, at least version 1.0.0 of Astro Private Cloud is required.

See Astro Private Cloud's [lifecycle policy](/docs/astro-private-cloud/v-2-x/release-lifecycle-policy) and [version compatibility reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference) for more information.

<a />

## Step 14: Fetch Airflow Helm charts

If you have internet access to `https://helm.astronomer.io`, run the following command on the machine where you want to install Astro Private Cloud:

```bash wrap theme={null}
helm repo add astronomer https://helm.astronomer.io/
helm repo update
```

If you don't have internet access to `https://helm.astronomer.io`, download the Astro Private Cloud Platform Helm chart file corresponding to the version of Astro Private Cloud you are installing or upgrading to from `https://helm.astronomer.io/astronomer-<version number>.tgz`. For example, for Astro Private Cloud v1.0.0 you would download `https://helm.astronomer.io/astronomer-1.0.0.tgz`. This file doesn't need to be uploaded to an internal chart repository.

<a />

## Step 15: Create and customize `upgrade.sh`

Create a file named `upgrade.sh` in your platform deployment project directory containing the following script. Specify the following values at the beginning of the script:

* `CHART_VERSION`: Your Astro Private Cloud version, including patch and a `v` prefix. For example, `v1.0.0`.
* `RELEASE_NAME`: Your Helm release name. `astronomer` is strongly recommended.
* `NAMESPACE`: The namespace to install platform components into. `astronomer` is strongly recommended.
* `CHART_NAME`: Set to `astronomer/astronomer` if fetching platform images from the internet. Otherwise, specify the filename if you're installing from a file (for example `astronomer-1.0.0.tgz`).

<Warning> Don't run this script after you create it. Your installation uses this script later, when you run your final upgrades and install processes.</Warning>

```bash wrap theme={null}
#!/bin/bash
set -xe

# typically astronomer
RELEASE_NAME=<astronomer-platform-release-name>
# typically astronomer
NAMESPACE=<astronomer-platform-namespace>
# typically astronomer/astronomer
CHART_NAME=<chart name>
# format is v<major>.<minor>.<path> e.g. v1.0.0
CHART_VERSION=<v-prefixed version of the Astro Private Cloud platform chart>
# ensure all the above environment variables have been set

helm repo add --force-update astronomer https://helm.astronomer.io
helm repo update

# upgradeDeployments false ensures that Airflow charts aren't upgraded when this script is run
# If you deployed a config change that is intended to reconfigure something inside Airflow,
# then you may set this value to "true" instead. When it is "true", then each Airflow chart will
# restart. Note that some stable version upgrades require setting this value to true regardless of your own configuration.
helm upgrade --install --namespace $NAMESPACE \
            -f ./values.yaml \
            --reset-values \
            --version $CHART_VERSION \
            --debug \
            --set astronomer.houston.upgradeDeployments.enabled=false \
            $RELEASE_NAME \
            $CHART_NAME $@
```

<a />

## Step 16: Mirror platform images

<Info>This step is optional but strongly recommended for production environments so your cluster can pull platform images from a registry you control.</Info>

1. Gather the list of required platform images using one of the following methods:

<Tabs>
  <Tab title="Shell">
    Mac and Linux users with `jq` installed can set `CHART_VERSION` in the following snippet and run it to produce a list of images.

    ```bash wrap theme={null}
    CHART_VERSION=<v-prefixed version of the Astro Private Cloud platform chart>
    UNPREFIXED_CHART_VERSION=${CHART_VERSION#v}
    curl -s https://updates.astronomer.io/astronomer-software/releases/astronomer-${UNPREFIXED_CHART_VERSION}.json | jq -r '(.astronomer.images, .airflow.images) | to_entries[] | "\(.value.repository):\(.value.tag)"'| sort
    ```
  </Tab>

  <Tab title="Windows Powershell">
    Windows PowerShell users can set `CHART_VERSION` in the following snippet and run it to produce a list of images.

    ```powershell wrap theme={null}
    $CHART_VERSION = "<v-prefixed version>"
    $UNPREFIXED_CHART_VERSION = $CHART_VERSION.TrimStart('v')
    $jsonUrl     = "https://updates.astronomer.io/astronomer-software/releases/astronomer-$UNPREFIXED_CHART_VERSION.json"
    $jsonContent = Invoke-WebRequest $jsonUrl -UseBasicParsing
    $json        = $jsonContent.Content | ConvertFrom-Json
    $astronomerImages = $json.astronomer.images.PSObject.Properties.Value
    $airflowImages    = $json.airflow.images.PSObject.Properties.Value

    $images = $astronomerImages + $airflowImages

    $images |
        ForEach-Object { "$($_.repository):$($_.tag)" } |
        Sort-Object
    ```
  </Tab>

  <Tab title="Other">
    Visit the [release metadata](https://updates.astronomer.io/astronomer-software/releases/index.html) page and download the json-formatted release metadata corresponding to the version of Astro Private Cloud you are installing and use another method of your choice to extract the list of images from beneath the `astronomer.images` and `airflow.images` keys.
  </Tab>
</Tabs>

2. Copy these images to the container registry using the naming scheme you configured [when you set up a custom image registry](#configure-a-private-docker-registry-platform).

<a />

## Step 17: Fetch Astro Runtime updates

If you are installing Astro Private Cloud into an egress-controlled or air gapped environment, perform the following steps.

By default, Astro Private Cloud checks for Airflow updates, which are included in the Astro Runtime, once per day at midnight by querying `https://updates.astronomer.io/astronomer-runtime`. This returns a JSON file with details about the latest available Astro Runtime versions.

In an egress-controlled or air gapped environment, you need to store the JSON file in the cluster itself, avoiding the external check. To store the JSON file in the cluster, complete the following steps:

1. Download the JSON files and store them in a Kubernetes configmap by running the following commands:

```bash wrap theme={null}
curl -XGET https://updates.astronomer.io/astronomer-runtime -o astro_runtime_releases.json

kubectl -n <apc platform namespace> create configmap astro-runtime-base-images --from-file=astro_runtime_releases.json
```

2. Add your configmap name, `astro-runtime-base-images` to your APC API configuration using the `runtimeReleasesConfigMapName` configuration:

```yaml wrap theme={null}
astronomer:
  houston:
    runtimeReleasesConfigMapName: astro-runtime-base-images
    config:
      airgapped:
        enabled: true
```

<a />

## Step 18: (OpenShift only) Apply OpenShift-specific configuration

If you're not installing Astro Private Cloud into an OpenShift Kubernetes cluster, skip this step.

Add the following values into `values.yaml`. You can do this manually or by using a YAML merge tool of your choosing.

```yaml wrap theme={null}
global:
  openshift:
    enabled: true
  scc:
    enabled: false
  extraAnnotations:
    kubernetes.io/ingress.class: openshift-default
    route.openshift.io/termination: "edge"
  authSidecar:
    enabled: true
  deployMechanisms:
    dagOnlyDeployment:
      securityContext:
        fsGroup: ""
  daemonsetLogging:
    enabled: false
  logging:
    loggingSidecar:
      enabled: true
      name: sidecar-log-consumer
elasticsearch:
  sysctlInitContainer:
    enabled: false

# bundled postgresql not a supported option, only for use in proof-of-concepts
postgresql:
  securityContext:
    enabled: false
  volumePermissions:
    enabled: false
```

<Info>
  Only Ingress objects with the annotation `route.openshift.io/termination: "edge"` are supported for generating routes in OpenShift 4.11 and later.
  Other termination types are no longer supported for automatic route generation.

  If you're on an older version of OpenShift, route creation should be done manually.
</Info>

Astro Private Cloud on OpenShift is only supported when using [a third-party ingress controller](#configure-third-party-ingress-controller) and the logging sidecar feature of Astro Private Cloud. The preceding configuration enables both of these items.

<a />

## Step 19: (Optional) Integrate an external identity provider

Astro Private Cloud includes integrations for several of the most popular OAUTH2 identity providers (IdPs), such as Okta and Microsoft Entra ID. Configuring an external IdP allows you to automatically provision and manage users in accordance with your organization's security requirements. See [Integrate an auth system](/docs/astro-private-cloud/v-2-x/integrate-auth-system) to configure the identity provider of your choice in your `values.yaml` file.

<a />

## Step 20: Install Astro Private Cloud using Helm

Deploy the control plane using the `upgrade.sh` script you created earlier. Confirm `RELEASE_NAME`, `NAMESPACE`, and `CHART_VERSION` reflect your environment, then execute:

```bash wrap theme={null}
./upgrade.sh
```

To review manifests before applying them, run `./upgrade.sh --dry-run` or use `helm template` with the same flags defined in the script.

<a />

## Step 21: Configure DNS to point to the ingress controller

Whether you use the Astronomer integrated ingress controller or a third-party controller, publish the same set of DNS records so users can reach control plane services.

* If you use the integrated controller, get the load balancer address directly:

  ```sh wrap theme={null}
  kubectl -n <apc platform namespace> get svc astronomer-cp-nginx
  ```

* If you use a third-party controller, ask your ingress administrator for the hostname or IP address that should front the Astronomer routes (refer back to [Configure a third-party ingress controller](#configure-third-party-ingress-controller)).

Create either a wildcard record such as `*.sandbox-astro.example.com` or individual CNAME records for the following hostnames so that traffic routes through the chosen load balancer:

* `app.<base-domain>` (required)
* `houston.<base-domain>` (required)
* `prometheus.<base-domain>` (required)
* `alertmanager.<base-domain>` (required if you keep the integrated Alertmanager enabled)
* `<base-domain>` (optional but recommended, provides a vanity redirect to `app.<base-domain>`)

Astronomer generally recommends pointing the zone apex (`@`) directly to the load balancer address and mapping the remaining hostnames as CNAMEs to that apex. In lower environments, you can safely use a low TTL (for example 60 seconds) to speed up troubleshooting during the initial rollout.

After your DNS provider propagates the records, verify them with tools like `dig <hostname>` or `getent hosts <hostname>`. You can complete this DNS work after verifying the platform pods—Astronomer services stay healthy without external DNS, but end users need these records to sign in.

<Info>
  Upgrades from 0.x to 2.0 rename the control plane ingress Service from `astronomer-nginx` to `astronomer-cp-nginx`. This provisions a new LoadBalancer with a new public IP/hostname. If you're upgrading, update DNS and firewall/allowlists and re-issue TLS/SSL certificates if they reference the previous LoadBalancer hostname. See the [upgrade guide](/docs/astro-private-cloud/v-2-x/upgrade-037-to-2).
</Info>

<a />

## Step 22: Verify you can access the UI

Visit `https://app.<base-domain>` in your web browser to view the Astro Private Cloud web interface. If any components aren't ready, consult the [debugging guide](/docs/astro-private-cloud/v-2-x/debug-install) or contact [Astronomer support](https://support.astronomer.io) with the relevant logs and events.

Congratulations, you have configured and installed an Astro Private Cloud platform instance - your new Airflow control plane!

From the Astro Private Cloud UI, you'll be able to both invite and manage users as well as create and monitor Airflow Deployments on the platform.

<a />

## Step 23: Disable anonymous account creation

Leave `astronomer.houston.config.publicSignups: true` only long enough to create your first administrator. Afterwards, secure the platform as follows:

1. If you keep public signups enabled, turn on outbound email (`astronomer.houston.config.email.enabled: true`), specify a trusted domain list under `astronomer.houston.config.allowedSystemLevelDomains`, and verify that users can only join through an approved identity provider.
2. Otherwise, set `astronomer.houston.config.publicSignups: false` so new accounts require an invitation.
3. Apply the updated configuration with `helm upgrade` targeting the control plane release.

## Additional customization

The following topics include optional information about one or multiple topics in the installation guide:

* [Configure a private Certificate Authority](/docs/astro-private-cloud/v-2-x/configure-private-cas)
* [Disable outbound emails](/docs/astro-private-cloud/v-2-x/disable-outbound-email)
* Add trusted CAs to [Docker Desktop](/docs/astro-private-cloud/v-2-x/configure-desktop-container-solution-extra-cas)

## Next steps

<a />

### Register the data planes with the control plane

Add the data planes to the control plane to begin creating Airflow Deployments. See [Register a data plane with the APC control plane](/docs/astro-private-cloud/v-2-x/register-data-plane) for instructions on exchanging tokens, approving connectivity, and assigning deployments.


# Install the Astro Private Cloud data plane
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/install-data-plane

Deploy an Astro Private Cloud (APC) data plane on a dedicated Kubernetes cluster to run Airflow workloads.

Use this guide to deploy an Astro Private Cloud (APC) data plane with the Helm-based Astronomer platform charts. Data planes host Airflow runtimes and execute Dag workloads while relying on the APC control plane for shared services such as the UI, APC API, monitoring coordination, and authentication.

## Prerequisites

Before you begin, deploy the control plane and verify network connectivity between the clusters. See [Install the APC Control Plane](/docs/astro-private-cloud/v-2-x/install-control-plane) for setup steps.

<Warning>
  While it is possible to register a data plane to multiple control planes, Astronomer doesn't recommend or support this configuration.
</Warning>

<Tabs>
  <Tab title="EKS on AWS">
    <Info>The following prerequisites apply when running Astro Private Cloud on Amazon EKS. See the **Other** tab if you run a different version of Kubernetes on AWS.</Info>

    * An EKS Kubernetes cluster, running a version of Kubernetes certified as compatible on the [Kubernetes Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/kubernetes-version-support) that provides the following components:
      * The [Amazon EBS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) (or an alternative CSI) must be installed on the Kubernetes Cluster.
      * An AWS Load Balancer Controller for the IP target type is required for all private Network Load Balancers (NLBs). See [Installing the AWS Load Balancer Controller add-on](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html).
    * A PostgreSQL instance, accessible from your Kubernetes cluster, and running a version of Postgres certified as compatible on the [Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
    * PostgreSQL superuser permissions.
    * Permission to create and modify resources on AWS.
    * Permission to generate a certificate that covers a defined set of subdomains.
    * An SMTP service and credentials. For example, Mailgun or Sendgrid.
    * The [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html).
    * (Optional) [`eksctl`](https://eksctl.io/) for creating and managing your Astronomer cluster on EKS.
    * A machine meeting the following criteria with access to the Kubernetes API Server:
      * Network access to the Kubernetes API Server - either direct access or VPN.
      * Network access to load-balancer resources that are created when Astro Private Cloud is installed later in the procedure - either direct access or VPN.
      * Configured to use the DNS servers where Astro Private Cloud DNS records can be created.
      * [Helm (minimum v3.6)](https://helm.sh/docs/intro/install).
      * The [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
    * (Situational) The [OpenSSL CLI](https://www.openssl.org/docs/man1.0.2/man1/openssl.html) might be required to troubleshoot certain certificate-related conditions.
  </Tab>

  <Tab title="GKE on GCP">
    <Info>The following prerequisites apply when running Astro Private Cloud on Google GKE. See the **Other** tab if you run a different version of Kubernetes on GCP.</Info>

    * A GKE Kubernetes cluster, running a version of Kubernetes listed as compatible on the [Kubernetes Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/kubernetes-version-support).
    * A PostgreSQL instance, accessible from your Kubernetes cluster, and running a version of Postgres certified as compatible on the [Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
    * PostgreSQL superuser permissions.
    * Permission to create and modify resources on Google Cloud Platform.
    * Permission to generate a certificate that covers a defined set of subdomains.
    * An SMTP service and credentials. For example, Mailgun or Sendgrid.
    * [Google Cloud SDK](https://cloud.google.com/sdk/install).
    * A machine that meets the following criteria with access to the Kubernetes API Server:
      * Network access to the Kubernetes API Server - either direct access or VPN.
      * Network access to load-balancer resources that are created when Astro Private Cloud is installed later in the procedure - either direct access or VPN.
      * Configured to use the DNS servers where Astro Private Cloud DNS records can be created.
      * [Helm with minimum version 3.6](https://helm.sh/docs/intro/install).
      * The [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
    * (Situational) The [OpenSSL CLI](https://www.openssl.org/docs/man1.0.2/man1/openssl.html) might be required to troubleshoot certain certificate-related conditions.
  </Tab>

  <Tab title="AKS on Azure">
    <Info>The following prerequisites apply when running Astro Private Cloud on Azure AKS. See the **Other** tab if you run a different version of Kubernetes on Azure.</Info>

    * A Kubernetes cluster, running a version of Kubernetes listed as compatible on the [Kubernetes Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/kubernetes-version-support).
    * A PostgreSQL instance, accessible from your Kubernetes cluster, and running a version of Postgres certified as compatible on the [Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
      * If your organization uses Azure Database for PostgreSQL as the database backend, you need to enable the `pg_trgm` extension using the Azure portal or the Azure CLI before you install Astro Private Cloud. If you don't enable the `pg_trgm` extension, the install fails. For more information about enabling the `pg_trgm` extension, see [PostgreSQL extensions in Azure Database for PostgreSQL - Flexible Server](https://docs.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions).
    * PostgreSQL superuser permissions.
    * Permission to create and modify resources on Azure.
    * Permission to generate a certificate that covers a defined set of subdomains.
    * An SMTP service and credentials. For example, Mailgun or Sendgrid.
    * The [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest).
    * A machine meeting the following criteria with access to the Kubernetes API Server:
      * Network access to the Kubernetes API Server - either direct access or VPN.
      * Network access to load-balancer resources created when Astro Private Cloud is installed later in the procedure - either direct access or VPN.
      * Configured to use the DNS servers where Astro Private Cloud DNS records will be created.
      * [Helm (minimum v3.6)](https://helm.sh/docs/intro/install).
      * The [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
    * (Situational) The [OpenSSL CLI](https://www.openssl.org/docs/man1.0.2/man1/openssl.html) might be required to trouble-shoot certain certificate-related conditions.
  </Tab>

  <Tab title="Other">
    The following prerequisites apply when running Astro Private Cloud on Kubernetes.

    * A Kubernetes cluster. For versioning considerations, see [Kubernetes Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/kubernetes-version-support).
    * A PostgreSQL instance accessible from your Kubernetes cluster. For versioning considerations, see [Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
    * PostgreSQL superuser permissions.
    * An SMTP service and credentials. For example, Mailgun or Sendgrid.
    * Permission to generate a certificate that covers a defined set of subdomains.
    * PostgreSQL superuser permissions.
    * The ability to create DNS records.
    * A machine with access to the Kubernetes API Server meeting the following criteria:
      * Network access to the Kubernetes API Server - either direct access or VPN.
      * Network access to load-balancer resources created when Astro Private Cloud is installed later in the procedure - either direct access or VPN.
      * Configured to use the DNS servers where Astro Private Cloud DNS records will be created.
      * [Helm (minimum v3.6)](https://helm.sh/docs/intro/install).
      * The [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
    * (Situational) The [OpenSSL CLI](https://www.openssl.org/docs/man1.0.2/man1/openssl.html) might be required to trouble-shoot certain certificate-related conditions.
  </Tab>
</Tabs>

<a />

### Ingress controller considerations

Astro Private Cloud requires a Kubernetes Ingress controller in the data plane to function and provides an integrated Ingress controller by default.
Before installing, you need to decide whether to use a third-party ingress controller or use Astronomer's integrated ingress controller.

Astronomer generally recommends you use the integrated Ingress controller, but Astro Private Cloud also supports certain third-party [ingress-controllers](/docs/astro-private-cloud/v-2-x/third-party-ingress-controllers).

Ingress controllers typically need elevated permissions, including a `ClusterRole`, to function. Specifically, the Astro Private Cloud Ingress controller requires the ability to:

* List all namespaces in the cluster.
* View ingresses in the namespaces.
* Retrieve secrets in the namespaces to locate and use private TLS certificates that service the ingresses.

If you have complex regulatory requirements, you might need to use an Ingress controller that's approved by your organization and disable Astronomer's integrated controller. You configure the Ingress controller during the installation.

<a />

## Step 1: Create a directory to hold files used when provisioning the data plane

Reuse the same top-level directory you created during the control plane install (for example, `~/astronomer-dev`) and create a subdirectory for each data plane. Using names such as `~/astronomer-dev/dp-dev-01a`, `~/astronomer-dev/dp-dev-01b`, or `~/astronomer-dev/dp-prod-01a` keeps the control plane environment obvious in the path and scales cleanly as you add more data planes.

<Info>Certain files in the project directory might contain secrets when you set up your sandbox or development environments. For your first install, keep these secrets in a secure place on a suitable machine. As you progress to higher environments, such as staging or production, secure these files separately in a vault and use the remaining project files in your directory to serve as the basis for your CI/CD deployment.</Info>

<a />

## Step 2: Create `values.yaml` from a template

Astro Private Cloud uses Helm to apply platform-level configurations. Choose your cloud provider tab below to copy a ready-to-use `values.yaml`, then update image tags, domains, and secrets before deploying.

<Warning>
  As you work with the template configuration, keep the following in mind.

  * Don't make any changes to this file until instructed to do so in later steps.
  * Don't run `helm upgrade` or `upgrade.sh` until instructed to do so in later steps.
  * Ignore any instructions to run `helm upgrade` from other Astronomer documentation until you've completed this installation.
</Warning>

<Tabs>
  <Tab title="EKS on AWS">
    ```yaml expandable wrap theme={null}
    ###########################################
    ### Astronomer global configuration for EKS
    ###########################################
    global:
      # Installation mode for the control plane
      plane:
        mode: data
        # Unique prefix for all data plane subdomains exposed through ingress
        domainPrefix: dp-01

      # Base domain shared with the control plane
      baseDomain: env.astronomer.your.domain

      # For development or proof-of-concept, you can use an in-cluster database.
      # This NOT supported in production.
      postgresql:
        enabled: false

      # Name of secret containing TLS certificate, change if not using "astronomer-tls"
      # tlsSecret: astronomer-tls

      # List of secrets containing the cert.pem of trusted private certification authorities
      # Example command: `kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
      # privateCaCerts:
      # - private-root-ca

      # Expose Postgres metrics for Prometheus to scrape
      # prometheusPostgresExporter:
      #   enabled: true

      # Database SSL configuration
      ssl:
        # Enable SSL connection to Postgres -- must be false if using in-cluster database
        enabled: true

    #########################
    ### Ingress configuration
    #########################
    # nginx:
      # Static IP address the nginx ingress should bind to
      # loadBalancerIP: ~

      # Set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
      # privateLoadBalancer: true

      # Dictionary of arbitrary annotations to add to the nginx ingress.
      # For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
      # Change to 'elb' if your node group is private and doesn't utilize a NAT gateway
      # ingressAnnotations: {service.beta.kubernetes.io/aws-load-balancer-type: nlb}

      # If all subnets are private, auto-discovery may fail.
      # You must enter the subnet IDs manually in the annotation below.
      # service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-id-1,subnet-id-2

    #####################################
    ### Astronomer platform configuration
    #####################################

    # tags:
        # platform: true
        # monitoring: true
        # logging: true
    ```
  </Tab>

  <Tab title="GKE on GCP">
    ```yaml expandable wrap theme={null}
    ###########################################
    ### Astronomer global configuration for GKE
    ###########################################
    global:
      # Installation mode for the control plane
      plane:
        mode: data
        # Unique prefix for all data plane subdomains exposed through ingress
        domainPrefix: dp-01

      # Base domain shared with the control plane
      baseDomain: env.astronomer.your.domain

      # For development or proof-of-concept, you can use an in-cluster database.
      # This NOT supported in production.
      postgresql:
        enabled: false

      # Name of secret containing TLS certificate, change if not using "astronomer-tls"
      # tlsSecret: astronomer-tls

      # List of secrets containing the cert.pem of trusted private certification authorities
      # Example command: `kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
      # privateCaCerts:
      # - private-root-ca

      # Expose Postgres metrics for Prometheus to scrape
      # prometheusPostgresExporter:
      #   enabled: true

      # Database SSL configuration
      ssl:
        # Enable SSL connection to Postgres -- must be false if using in-cluster database
        enabled: true

    #########################
    ### Ingress configuration
    #########################
    # nginx:
      # Static IP address the nginx ingress should bind to
      # loadBalancerIP: ~

      # Set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
      # privateLoadBalancer: true

      # Dictionary of arbitrary annotations to add to the nginx ingress.
      # For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
      # Change to 'elb' if your node group is private and doesn't utilize a NAT gateway
      # ingressAnnotations: {}

    #####################################
    ### Astronomer platform configuration
    #####################################

    # tags:
        # platform: true
        # monitoring: true
        # logging: true
    ```
  </Tab>

  <Tab title="AKS on Azure">
    ```yaml expandable wrap theme={null}
    ###########################################
    ### Astronomer global configuration for AKS
    ###########################################
    global:
      # Installation mode for the control plane
      plane:
        mode: data
        # Unique prefix for all data plane subdomains exposed through ingress
        domainPrefix: dp-01

      # Base domain shared with the control plane
      baseDomain: env.astronomer.your.domain

      # For development or proof-of-concept, you can use an in-cluster database.
      # This NOT supported in production.
      postgresql:
        enabled: false

      # Name of secret containing TLS certificate, change if not using "astronomer-tls"
      # tlsSecret: astronomer-tls

      # List of secrets containing the cert.pem of trusted private certification authorities
      # Example command: `kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
      # privateCaCerts:
      # - private-root-ca

      # Expose Postgres metrics for Prometheus to scrape
      # prometheusPostgresExporter:
      #   enabled: true

      # Database SSL configuration
      ssl:
        # Enable SSL connection to Postgres -- must be false if using in-cluster database
        enabled: true

    #########################
    ### Ingress configuration
    #########################
    nginx:
      # Static IP address the nginx ingress should bind to
      # loadBalancerIP: ~

      # Set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
      # privateLoadBalancer: true

      # Dictionary of arbitrary annotations to add to the nginx ingress.
      # For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
      # required for azure load balancer post Kubernetes 1.24
      ingressAnnotations:
          service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/healthz"

    #####################################
    ### Astronomer platform configuration
    #####################################

    # tags:
        # platform: true
        # monitoring: true
        # logging: true
    ```
  </Tab>

  <Tab title="Other">
    ```yaml expandable wrap theme={null}
    #################################################################
    ### Astronomer global configuration for other types of Kubernetes
    #################################################################
    global:
      # Installation mode for the control plane
      plane:
        mode: data
        # Unique prefix for all data plane subdomains exposed through ingress
        domainPrefix: dp-01

      # Base domain shared with the control plane
      baseDomain: env.astronomer.your.domain

      # For development or proof-of-concept, you can use an in-cluster database.
      # This NOT supported in production.
      postgresql:
        enabled: false

      # Name of secret containing TLS certificate, change if not using "astronomer-tls"
      # tlsSecret: astronomer-tls

      # List of secrets containing the cert.pem of trusted private certification authorities
      # Example command: `kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
      # privateCaCerts:
      # - private-root-ca

      # Expose Postgres metrics for Prometheus to scrape
      # prometheusPostgresExporter:
      #   enabled: true

      # Database SSL configuration
      ssl:
        # Enable SSL connection to Postgres -- must be false if using in-cluster database
        enabled: true

    #########################
    ### Ingress configuration
    #########################
    # nginx:
      # Static IP address the nginx ingress should bind to
      # loadBalancerIP: ~

      # Set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
      # privateLoadBalancer: true

      # Dictionary of arbitrary annotations to add to the nginx ingress.
      # For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
      # required for azure load balancer post Kubernetes 1.24
      # ingressAnnotations: {}

    #####################################
    ### Astronomer platform configuration
    #####################################

    # tags:
        # platform: true
        # monitoring: true
        # logging: true
    ```
  </Tab>
</Tabs>

<a />

## Step 3: Choose and configure the data plane domain prefix

Assign a unique value to `global.plane.domainPrefix`. Astronomer uses this prefix as the leftmost label for every data plane hostname (for example, `commander.<domainPrefix>.<baseDomain>` and `prometheus.<domainPrefix>.<baseDomain>`) and includes it in monitoring metadata.

* Use a DNS-compliant label: lowercase letters, numbers, and hyphens only; 1–63 characters; and no leading or trailing hyphen.
* Confirm you can create DNS records and issue TLS certificates for the resulting hostnames. A [later step](#astronomer-tls-certificate) lists the exact FQDNs that require coverage.

Update your values file with the chosen suffix. For example:

```yaml wrap theme={null}
global:
  plane:
    domainPrefix: apc-dp-01a
```

<a />

## Step 4: Configure a base domain

Set `global.baseDomain` in this data plane's `values.yaml` to the same value used by the control plane. All planes must share the exact base domain so HTTPS certificates and DNS records align.

Update your values file with the base domain. For example:

```yaml wrap theme={null}
global:
  baseDomain: sandbox-astro.example.com
```

<a />

## Step 5: Create the Astro Private Cloud platform namespace

In your Kubernetes cluster, create a [Kubernetes namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) to contain the Astro Private Cloud platform, for example `apc-dp-01`:

```sh wrap theme={null}
kubectl create namespace apc-dp-01
```

Astro Private Cloud installs components into this namespace to provision and manage Airflow Deployments running in other namespaces. Each Airflow Deployment has its own isolated namespace.

<a />

## Step 6: Request and validate an Astronomer TLS certificate

To install Astro Private Cloud, you need a TLS certificate that is valid for several domains. One of the domains is the primary name on the certificate, also known as the common name (CN). The additional domains are equally valid, supplementary domains known as Subject Alternative Names (SANs).

Astronomer requires a private certificate to be present in the Astro Private Cloud platform namespace, even if you use a third-party ingress controller that doesn't otherwise require it.

<a />

### Request an ingress controller TLS certificate

Request a TLS certificate from your security team for Astro Private Cloud. In your request, include the following:

* Use `<domainPrefix>.<baseDomain>` as the Common Name (CN). If your certificate authority won't issue certificates for the bare base domain, use `commander.<domainPrefix>.<baseDomain>` as the CN instead.
* Add Subject Alternative Names (SANs) for either of the following options:
  * Option 1: request a wildcard SAN of `*.<domainPrefix>.<baseDomain>` plus an explicit SAN for `<domainPrefix>.<baseDomain>`.
  * Option 2: list each hostname individually:
    * `<domainPrefix>.<baseDomain>` (data plane metadata service)
    * `commander.<domainPrefix>.<baseDomain>`
    * `deployments.<domainPrefix>.<baseDomain>` (required for Airflow UIs and APIs)
    * `prometheus.<domainPrefix>.<baseDomain>`
    * `prom-proxy.<domainPrefix>.<baseDomain>`
    * `registry.<domainPrefix>.<baseDomain>` (only if you keep the integrated registry enabled)
    * `es-proxy.<domainPrefix>.<baseDomain>` and `elasticsearch.<domainPrefix>.<baseDomain>` (only when logging tags are enabled)

<Warning>Wildcards only cover a single label. A control plane certificate like `*.<baseDomain>` won't secure data plane hosts such as `commander.<domainPrefix>.<baseDomain>`. Request a dedicated wildcard `*.<domainPrefix>.<baseDomain>` or list each hostname explicitly.</Warning>

* If you use the Astro Private Cloud integrated container registry, specify that the encryption type of the certificate must be RSA.
* Request the following return format:
  * A `key.pem` containing the private key in pem format
  * **Either** a `full-chain.pem` (containing the public certificate and additional certificates required to validate it, in pem format) **or** a bare `cert.pem` and explicit affirmation that there are no intermediate certificates and that the public certificate is the full chain.
  * **Either** the `private-root-ca.pem` in pem format of the private Certificate Authority used to create your certificate or a statement that the certificate is signed by a public Certificate Authority.

<Warning>If you're using the Astro Private Cloud integrated container registry, the encryption type used on your TLS certificate must be **RSA**. Certbot users must include `--key-type rsa` when requesting certificates. Most other solutions generate RSA keys by default.</Warning>

### Validate the received certificate and associated items

Ensure that you received each of the following three items:

* A `key.pem` containing the private key in pem format.
* **Either** a `full-chain.pem`, in pem format, that contains the public certificate and additional certificates required to validate it **or** a bare `cert.pem` and explicit affirmation that there are no intermediate certificates and that the public certificate is the full chain.
* **Either** the `private-root-ca.pem` in pem format of the private Certificate Authority used to create your certificate **or** a statement that the certificate is signed by public Certificate Authority.

To validate that your security team generated the correct certificate, run the following command using the `openssl` CLI:

```sh wrap theme={null}
openssl x509 -in  <your-certificate-filepath> -text -noout
```

This command will generate a report. If the `X509v3 Subject Alternative Name` section of this report includes either a single `*.<baseDomain>` wildcard domain or all subdomains, then the certificate creation was successful.

Confirm that your full-chain certificate chain is ordered correctly. To determine your certificate chain order, run the following command using the `openssl` CLI:

```sh wrap theme={null}
openssl crl2pkcs7 -nocrl -certfile <your-full-chain-certificate-filepath> | openssl pkcs7 -print_certs -noout
```

The command generates a report of all certificates. Verify that the certificates are in the following order:

* Domain
* Intermediate (optional)
* Root

<a />

### (Optional) Additional validation for the Astronomer integrated container registry

<Info>If you don't plan to store images in Astronomer's integrated container registry and instead plan to store all container images using an [external container registry](#configure-a-private-docker-registry-airflow), you can skip this step.</Info>

The Astro Private Cloud integrated container registry requires that your private key signs traffic originating from the Astro Private Cloud platform using the RSA encryption method. Confirm that the key is signing traffic correctly before proceeding.

Run the following command to extract the bare public cert, if it wasn't already included in the files provided by your certificate authority, from the full-chain certificate file:

```sh wrap theme={null}
openssl crl2pkcs7 -nocrl -certfile full-chain.pem | openssl pkcs7 -print_certs -noout > cert.pem
```

Examine the public certificate and ensure all Signature Algorithms are listed as `sha1WithRSAEncryption`.

```sh wrap theme={null}
openssl x509 -in cert.pem -text|grep Algorithm
        Signature Algorithm: sha1WithRSAEncryption
            Public Key Algorithm: rsaEncryption
    Signature Algorithm: sha1WithRSAEncryption
```

If your key isn't compatible with the Astro Private Cloud integrated container registry, ask your Certificate Authority to [re-issue the credentials](#request-a-certificate-bundle) and emphasize the need for an RSA cert, or [use an external container registry](#configure-a-private-docker-registry-airflow).

<a />

## Step 7: Store and configure the ingress controller TLS certificate

Determine whether or not your certificate was issued by an intermediate certificate-authority. If you don't know, assume you use an intermediate certificate and attempt to obtain a `full-chain.pem` bundle from your certificate authority.

Certificates issued by operators of root certificate authorities, including but not limited to LetsEncrypt, are frequently issued from intermediate certificate authorities associated with a trusted root CA.

<Warning>Astro Private Cloud backend services have stricter trust requirements than most web-browsers. Web Browsers might auto-complete the chain and consider your certificate valid, even if you don't provide the intermediate certificate-authority's public certificate. Astro Private Cloud backend services can reject the same certificate, and cause Dag and image deploys to fail.</Warning>

If, and only if, your certificate was issued directly by the root Certificate Authority of a universally trusted certificate authority, and not from one of their intermediaries, then the `server.crt` is also the full-chain certificate bundle.

Identify your full-chain public certificate `.pem` file and use it while storing and configuring the ingress controller TLS certificate.

<Warning>The `--cert` parameter must reference your `full-chain.pem`, which includes the server certificate *and* any intermediate certificates, if any. Using the server cert directly causes Dag and image deploys to fail.</Warning>

Run the following command to store the public full-chain certificate in the Astro Private Cloud Platform Namespace in a `tls`-type Kubernetes secret. You can create a custom name for this secret. The following example uses the name, `astronomer-tls`.

```sh wrap theme={null}
kubectl -n <astronomer platform namespace> create secret tls astronomer-tls --cert <fullchain-pem-filepath> --key <your-private-key-filepath>
```

Naming the secret `astronomer-tls` with no substitutions is recommended when using a third-party ingress controller.

<a />

## Step 8: (Optional) Configure a third-party ingress controller

If you use Astro Private Cloud's integrated ingress controller, you can skip this step.

Complete the full setup as described in [Third-party Ingress-Controllers](/docs/astro-private-cloud/v-2-x/third-party-ingress-controllers), which includes steps to configure ingress controllers in specific environment types. When you're done, return to this page and continue to the next step.

<a />

## Step 9: Configure a private certificate authority

Skip this step if you don't use a private Certificate Authority (private CA) to sign the certificate used by your ingress-controller. Or, if you don't use a private CA for any of the following services that the Astro Private Cloud platform interacts with.

Astro Private Cloud trusts public Certificate Authorities automatically.

Astro Private Cloud must be configured to trust any private Certificate Authorities issuing certificates for systems Astro Private Cloud interacts with, including but not limited-to:

* ingress controller
* any container registries that Kubernetes pulls from
* if using OAuth, the OAuth provider
* if using external Elasticsearch, any external Elasticsearch instances
* if using external Prometheus, any external Prometheus instances

Perform the procedure described in [Configuring private CAs](/docs/astro-private-cloud/v-2-x/configure-private-cas) for each certificate authority used to sign TLS certificates. After creating the trust secret (for example `astronomer-ca`), add it to `global.privateCaCerts` in `values.yaml` so platform components trust the issuer.

<Info>Astro CLI users must also configure both their operating system and container solution, [Docker Desktop or Podman](/docs/astro-private-cloud/v-2-x/configure-desktop-container-solution-extra-cas), to trust the private certificate Authority that was used to create the certificate used by the Astro Private Cloud ingress controller and any third-party container registries.</Info>

<a />

## Step 10: Confirm your Kubernetes cluster trusts required CAs

If at least one of the following circumstances apply to your installation, complete this step:

* Users will deploy images to an external container registry *and* that registry is using a TLS certificate issued by a private CA.
* You plan for your users to deploy Airflow images to Astro Private Cloud's integrated container registry *and* Astronomer is using a TLS certificate issued by a private CA.

Kubernetes must be able to pull images from one or more container registries for Astro Private Cloud to function. By default, Kubernetes only trusts publicly signed certificates. This means that by default, Kubernetes doesn't honor the list of certificates [trusted by the Astro Private Cloud platform](/docs/astro-private-cloud/v-2-x/configure-private-cas).

Many enterprises configure Kubernetes to trust additional certificate authorities as part of their standard cluster creation procedure. Contact your Kubernetes Administrator to find out what, if any, private certificates are currently trusted by your Kubernetes Cluster. Then, consult your Kubernetes administrator and Kubernetes provider's documentation for instructions on configuring Kubernetes to trust additional CAs.

Follow procedures for your Kubernetes provider to configure Kubernetes to trust each CA associated with your container registries, including the integrated container registry, if applicable.

Certain clusters don't provide a mechanism to configure the list of certificates trusted by Kubernetes.

While configuring the Kubernetes list of cluster certificates is a customer responsibility, Astro Private Cloud includes an optional component that can, for certain Kubernetes cluster configurations, add certificates defined in `global.privateCaCerts` to the list of certificates trusted by Kubernetes. This can be enabled by setting `global.privateCaCertsAddToHost.enabled` and `global.privateCaCertsAddToHost.addToContainerd` to `true` in your `values.yaml` file and setting `global.privateCaCertsAddToHost.containerdConfigToml` to:

```text wrap theme={null}
[host."https://registry.<domainPrefix>.<baseDomain>"]
  ca = "/etc/containerd/certs.d/<registry hostname>/<secret name>.pem"
```

For example, if your base domain is `apc-01.mydomain.internal`, the domain prefix is `apc-dp-01a`, and the CA public certificate is stored in the namespace in a secret named `my-private-ca`, the `global.privateCaCertsAddToHost` section would be:

```yaml wrap theme={null}
  global:
    privateCaCertsAddToHost:
      enabled: true
      addToContainerd: true
      hostDirectory: /etc/containerd/certs.d
      containerdConfigToml: |-
        [host."https://registry.apc-dp-01a.apc-01.mydomain.internal"]
          ca = "/etc/containerd/certs.d/registry.apc-dp-01a.apc-01.mydomain.internal/my-private-ca.pem"
```

## Step 11: Configure volume storage classes

Skip this step if your cluster defines a volume storage class, and you want to use it for all volumes associated with Astro Private Cloud and its Airflow Deployments.

Astronomer strongly recommends that you don't back any volumes used for Astro Private Cloud with mechanical hard drives.

Create `storage-class-config.yaml` in your project directory and update the configuration to match your environment:

```yaml wrap theme={null}
global:
  prometheus:
    persistence:
      storageClassName: "<desired-storage-class>"

astronomer:
  registry:
    persistence:
      storageClassName: "<desired-storage-class>"
  elasticsearch:
    common:
      persistence:
        storageClassName: "<desired-storage-class>"  # Required only if logging is enabled
```

Remove the `elasticsearch` section unless you plan to enable logging (`tags.logging: true`).

Merge these values into `values.yaml` manually or by using a YAML merge tool of your choosing.

<a />

## Step 12: Configure the database

The data plane needs access to a database to create and manage the Airflow Deployment databases. To do this an admin user with the ability to create databases and users should be configured and placed into the `astronomer-bootstrap` secret in the data plane namespace.

<Warning>PostgreSQL usernames must be lowercase. </Warning>

1. Ensure firewalls, network policies, and routing rules allow pods in this data plane cluster to reach the database host/port.
2. Create a Kubernetes secret with the admin user connection string in the data plane namespace:

   ```bash wrap theme={null}
   kubectl -n <dataplane namespace> create secret generic astronomer-bootstrap \
     --from-literal connection="postgres://<url-encoded username>:<url-encoded password>@<database hostname>:<database port>"
   ```

   If the secret already exists, use `kubectl apply` to update it instead of recreating it.

If your organization rotates database credentials automatically, include the data plane namespace in the same rotation workflow so the secret stays in sync.

<a />

## Step 13: Configure an external Docker registry for Airflow images

Astro Private Cloud users create customized Airflow container images when they deploy project code to the platform. These images frequently contain sensitive information and must be stored in a secure location accessible to Kubernetes.

<Tabs>
  <Tab title="Anonymous">
    Ensure network access from the cluster to your registry endpoint and limit visibility to trusted networks (for example private subnets or VPN access). No additional Astronomer configuration is required.
  </Tab>

  <Tab title="Authenticated">
    1. Create a Docker registry secret in the Astronomer namespace and annotate it so the deployment orchestrator syncs credentials to Deployment namespaces:

       ```bash wrap theme={null}
       kubectl -n <astronomer platform namespace> create secret docker-registry <secret-name> \
         --docker-server=<registry-host> \
         --docker-username=<username> \
         --docker-password=<password> \
         --docker-email=<email>

       kubectl -n <astronomer platform namespace> annotate secret <secret-name> \
         "astronomer.io/commander-sync"="platform=astronomer"
       ```

    2. Update `values.yaml` so the APC API directs Deployments to your registry:

       ```yaml wrap theme={null}
       astronomer:
         houston:
           config:
             deployments:
               registry:
                 protectedCustomRegistry:
                   enabled: true
                   updateRegistry:
                     enabled: true
                     host: <registry-host>/<repository>
                     secretName: <secret-name>
       ```

    3. After applying the change, run `kubectl create job -n <astronomer platform namespace> --from=cronjob/<platform-release-name>-config-syncer upgrade-config-synchronization` to push credentials to existing Deployment namespaces.
  </Tab>
</Tabs>

See [Configure a custom registry for Deployment images](/docs/astro-private-cloud/v-2-x/custom-image-registry) for full details, including per-deployment registries and air gapped workflows.

<a />

## Step 14: Determine which version of Astro Private Cloud to install

Astronomer recommends that new Astro Private Cloud installations use the most recent APC version available. Keep this version number available for the following steps. For a separate control plane and data plane topology, at least version 1.0.0 of Astro Private Cloud is required.

See Astro Private Cloud's [lifecycle policy](/docs/astro-private-cloud/v-2-x/release-lifecycle-policy) and [version compatibility reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference) for more information.

<a />

## Step 15: Fetch Airflow Helm charts

If you have internet access to `https://helm.astronomer.io`, run the following command on the machine where you want to install Astro Private Cloud:

```sh wrap theme={null}
helm repo add astronomer https://helm.astronomer.io/
helm repo update
```

If you don't have internet access to `https://helm.astronomer.io`, download the Astro Private Cloud Platform Helm chart file corresponding to the version of Astro Private Cloud you are installing or upgrading to from `https://helm.astronomer.io/astronomer-<version number>.tgz`. For example, for Astro Private Cloud v1.0.0 you would download `https://helm.astronomer.io/astronomer-1.0.0.tgz`. This file doesn't need to be uploaded to an internal chart repository.

<a />

## Step 16: Create and customize `upgrade.sh`

Create a file named `upgrade.sh` in your platform deployment project directory containing the following script. Specify the following values at the beginning of the script:

* `CHART_VERSION`: Your Astro Private Cloud version, including patch and a `v` prefix. For example, `v1.0.0`.
* `RELEASE_NAME`: Your Helm release name. `astronomer` is strongly recommended.
* `NAMESPACE`: The namespace to install platform components into. `astronomer` is strongly recommended.
* `CHART_NAME`: Set to `astronomer/astronomer` if fetching platform images from the internet. Otherwise, specify the filename if you're installing from a file (for example `astronomer-1.0.0.tgz`).

```sh wrap theme={null}
#!/bin/bash
set -xe

# typically astronomer
RELEASE_NAME=<astronomer-platform-release-name>
# typically astronomer
NAMESPACE=<astronomer-platform-namespace>
# typically astronomer/astronomer
CHART_NAME=<chart name>
# format is v<major>.<minor>.<path> e.g. v0.32.9
CHART_VERSION=<v-prefixed version of the Astro Private Cloud platform chart>
# ensure all the above environment variables have been set

helm repo add --force-update astronomer https://helm.astronomer.io
helm repo update

# upgradeDeployments false ensures that Airflow charts aren't upgraded when this script is run
# If you deployed a config change that is intended to reconfigure something inside Airflow,
# then you may set this value to "true" instead. When it is "true", then each Airflow chart will
# restart. Note that some stable version upgrades require setting this value to true regardless of your own configuration.
# If you are currently on Astro Private Cloud 0.25, 0.26, or 0.27, you must upgrade to version 0.28 before upgrading to 0.29. A direct upgrade to 0.29 from a version lower than 0.28 isn't possible.
helm upgrade --install --namespace $NAMESPACE \
            -f ./values.yaml \
            --reset-values \
            --version $CHART_VERSION \
            --debug \
            $RELEASE_NAME \
            $CHART_NAME $@
```

<a />

## Step 18: (OpenShift only) Apply OpenShift-specific configuration

If you're not installing Astro Private Cloud into an OpenShift Kubernetes cluster, skip this step.

Add the following values into `values.yaml`. You can do this manually or by using a YAML merge tool of your choosing.

```yaml wrap theme={null}
global:
  nginx:
    enabled: false
  openshift:
    enabled: true
  scc:
    enabled: false
  extraAnnotations:
    kubernetes.io/ingress.class: openshift-default
    route.openshift.io/termination: "edge"
  authSidecar:
    enabled: true
  deployMechanisms:
    dagOnlyDeployment:
      securityContext:
        fsGroup: ""
  daemonsetLogging:
    enabled: false
  logging:
    loggingSidecar:
      enabled: true
      name: sidecar-log-consumer
elasticsearch:
  sysctlInitContainer:
    enabled: false

# bundled postgresql not a supported option, only for use in proof-of-concepts
postgresql:
  securityContext:
    enabled: false
  volumePermissions:
    enabled: false
```

<Info>
  Only Ingress objects with the annotation `route.openshift.io/termination: "edge"` are supported for generating routes in OpenShift 4.11 and later.
  Other termination types are no longer supported for automatic route generation.

  If you're on an older version of OpenShift, route creation should be done manually.
</Info>

Astro Private Cloud on OpenShift is only supported when using [a third-party ingress-controller](#configure-third-party-ingress-controller) and using the [logging sidecar](#configure-sidecar-logging) feature of Astro Private Cloud. The above configuration enables both of these items.

<a />

## Step 19: (Optional) Limit Astronomer to a namespace pool

By default, Astro Private Cloud automatically creates namespaces for each new Airflow Deployment.

You can restrict the Airflow management components of Astro Private Cloud to a list of predefined namespaces and configure it to operate without a ClusterRole by following the instructions in [Configure a Kubernetes namespace pool for Astro Private Cloud](/docs/astro-private-cloud/v-2-x/namespace-pools). If you want to disable creation of role and rolebindings for the deployment orchestrator, `config-syncer`, and kubestate metrics, you can set `global.namespaceManagement.namespacePools.createRbac` to `false`.

If `global.rbac.enabled` is `false`, the platform no longer creates any role, rolebindings, or service accounts. The user must define default roles to the k8s default service account to continue with the platform install. See [Bring your own Kubernetes service accounts](/docs/astro-private-cloud/v-2-x/byo-service-accounts) for setup steps.

<a />

## Step 20: (Optional) Enable sidecar logging

Running a logging sidecar to export Airflow task logs is essential for running Astro Private Cloud in a multi-tenant cluster.

By default, Astro Private Cloud creates a privileged DaemonSet to aggregate logs from Airflow components for viewing from within Airflow and the Astro Private Cloud UI.

You can replace this privileged Daemonset with unprivileged logging sidecars by following instructions in [Export logs using container sidecars](/docs/astro-private-cloud/v-2-x/export-task-logs#export-logs-using-container-sidecars).

<a />

## Step 21: Install the data plane using Helm

Deploy the data plane using the `upgrade.sh` script you created earlier. Confirm `RELEASE_NAME`, `NAMESPACE`, and `CHART_VERSION` reflect your environment, then execute:

```bash wrap theme={null}
./upgrade.sh
```

To review manifests before applying them, run `./upgrade.sh --dry-run` or use `helm template` with the same flags defined in the script.

<a />

## Step 22: Configure DNS to point to the ingress controller

Whether you use Astronomer's integrated ingress controller or a third-party controller, publish the same set of DNS records so users can reach data plane services.

* If you use the integrated controller, get the load balancer address directly:

  ```sh wrap theme={null}
  kubectl -n <astronomer platform namespace> get svc astronomer-dp-nginx
  ```

* If you use a third-party controller, ask your ingress administrator for the hostname or IP address that should front the Astronomer routes (refer back to the details you gathered in [Step 9](#configure-third-party-ingress-controller)).

Create either a wildcard record for `*.<domainPrefix>.<baseDomain>`, such as `*.apc-dp-01a.apc-01.example.com` or individual CNAME records for the following data plane hostnames so that traffic routes through the chosen load balancer:

* `<domainPrefix>.<baseDomain>`
* `commander.<domainPrefix>.<baseDomain>`
* `prometheus.<domainPrefix>.<baseDomain>`
* `prom-proxy.<domainPrefix>.<baseDomain>`
* `registry.<domainPrefix>.<baseDomain>` (only if you keep the integrated registry enabled)
* `prometheus.<domainPrefix>.<baseDomain>`
* `prom-proxy.<domainPrefix>.<baseDomain>`
* `registry.<domainPrefix>.<baseDomain>` (only if you keep the integrated registry enabled)
* `es-proxy.<domainPrefix>.<baseDomain>` and `elasticsearch.<domainPrefix>.<baseDomain>` (only when logging tags are enabled)

Astronomer generally recommends pointing the zone apex (`@`) directly to the load balancer address and mapping the remaining hostnames as CNAMEs to that apex. In lower environments, you can safely use a low TTL (for example 60 seconds) to speed up troubleshooting during the initial rollout.

After your DNS provider propagates the records, verify them with tools like `dig commander.<domainPrefix>.<baseDomain>` or `getent hosts commander.<domainPrefix>.<baseDomain>`. You can complete this DNS work after verifying the platform pods—Astronomer services stay healthy without external DNS, but end users need these records to sign in.

<a />

## Step 23: Verify Pods creation

To verify all pods are up and running, run:

```sh wrap theme={null}
kubectl -n <astronomer platform namespace> get pods
```

All pods should be in Running status. For example,

```command wrap theme={null}
$ kubectl -n astronomer get pods

NAME                                                   READY   STATUS    RESTARTS        AGE
astronomer-commander-6bd95b6f9b-t2dg7                  1/1     Running   0               6m6s
astronomer-commander-6bd95b6f9b-vz8kn                  1/1     Running   0               5m50s
astronomer-dp-nginx-5657d4869b-lsczb                   1/1     Running   0               20m
astronomer-dp-nginx-5657d4869b-n4dhz                   1/1     Running   0               20m
<snip>
```

If all pods aren't in running status, check the [guide on debugging your installation](/docs/astro-private-cloud/v-2-x/debug-install) or contact [Astronomer support](https://support.astronomer.io) for additional configuration assistance.

<Info>If you added the `podLabels` configuration, you can also search for Pods created by Astro Private Cloud by searching for the key-value pair in the label you created. See [Add Pod labels](/docs/astro-private-cloud/v-2-x/add-podlabels).</Info>

Congratulations, you have configured and installed an Astro Private Cloud platform instance—your new data plane!

## Additional customization

The following topics include optional information about one or multiple topics in the installation guide:

* [Configure a private Certificate Authority](/docs/astro-private-cloud/v-2-x/configure-private-cas)
* Add trusted CAs to [Docker Desktop](/docs/astro-private-cloud/v-2-x/configure-desktop-container-solution-extra-cas)

## Next steps

<a />

### Register the data plane with the control plane

Add the data plane to the control plane UI so the APC API can schedule workloads onto it. See [Register a data plane with the APC control plane](/docs/astro-private-cloud/v-2-x/register-data-plane) for instructions on exchanging tokens, approving connectivity, and assigning deployments.


# Plan your Astro Private Cloud installation
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/install-overview

Decide whether to run split or unified APC deployments and map the follow-up guides you need.

## Choose an installation topology

Astro Private Cloud (APC) supports two primary layouts:

* **Split planes**: Deploy the [control plane](/docs/astro-private-cloud/v-2-x/install-control-plane) on a dedicated cluster, then add one or more [data planes](/docs/astro-private-cloud/v-2-x/install-data-plane). This separation isolates management services from Airflow workloads, scales independently, and provides superior fault isolation between clusters.
* **Unified mode**: Run control plane and data plane components in one cluster using [Unified Mode](/docs/astro-private-cloud/v-2-x/install-unified). Unified mode simplifies evaluation environments, but concentrates failures and resource usage.

<Info>If you want to run both the Control Plane and Data Plane Helm releases in the same cluster while keeping responsibilities separate, install the control plane first and then the data plane. This approach consumes more resources than unified mode, but preserves isolation.</Info>

## Installation flow

1. **Review prerequisites**: Ensure the target cluster meets the requirements listed in each install guide. These requirements include cloud-specific Kubernetes versions, Postgres access, TLS, and ingress controller.
2. **Plan DNS and certificates**: Reserve hostnames, such as `app.<base-domain>`, `houston.<base-domain>`, `deployments.<base-domain>`, etc, and request certificates that cover the necessary SANs. Split Deployments use separate certificates per plane.
3. **Prepare storage and secrets**: Decide whether to use external Postgres, registry storage, and log destinations. Collect any APC API tokens or registry credentials required during installation.
4. **Install control plane services**: [Install the Control Plane](/docs/astro-private-cloud/v-2-x/install-control-plane) to deploy the APC API, Astro UI, and supporting components.
5. **Install runtime services**: For split Deployments, deploy data planes with [Install data plane](/docs/astro-private-cloud/v-2-x/install-data-plane) and register them in the APC API. For unified mode, follow the unified guide to add runtime components within the same cluster.
6. **Verify and harden**: Confirm deployment orchestrator heartbeat, Prometheus federation, and ingress connectivity. Configure RBAC, network policies, and backups appropriate for your topology.

## Related references

* [Control plane architecture](/docs/astro-private-cloud/v-2-x/control-plane-architecture)
* [Data plane architecture](/docs/astro-private-cloud/v-2-x/data-plane-architecture)
* [Unified architecture](/docs/astro-private-cloud/v-2-x/unified-architecture)

## Need help?

If you are unsure which path to pick, start with the split-plane installation. You can always evaluate unified mode later or run both Helm releases in the same cluster if you need a stepping stone toward a dedicated control plane.


# Install Astro Private Cloud in unified mode
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/install-unified

Deploy Astro Private Cloud in unified mode on a single Kubernetes cluster.

Use this guide to deploy an Astro Private Cloud (APC) unified cluster, where control plane and data plane components run together in a single Kubernetes cluster. Unified mode combines management services, such as Astro UI, the APC API, and NATS, with runtime services like the deployment orchestrator, Config Syncer, and data plane ingress so that platform operators can evaluate APC without maintaining separate clusters.

If you prefer to keep control plane and data plane Helm releases separate but run them in the *same* Kubernetes cluster, follow the dedicated [control plane](/docs/astro-private-cloud/v-2-x/install-control-plane) and [data plane](/docs/astro-private-cloud/v-2-x/install-data-plane) install guides sequentially. That approach consumes slightly more resources than unified mode but keeps responsibilities isolated.

<Note>
  At some points in this installation procedure, your particular environment configurations might require you to take additional steps, or, enable you to skip certain steps.

  When you see a callout like this, read it carefully and follow the instructions if they apply to your installation.
</Note>

## Prerequisites

<Tip>
  Many sections in this document reuse the [control plane installation workflow](/docs/astro-private-cloud/v-2-x/install-control-plane). Integrate the data plane-specific configuration from [Install data plane](/docs/astro-private-cloud/v-2-x/install-data-plane) where called out to ensure unified clusters contain all runtime functionality.
</Tip>

<Tabs>
  <Tab title="EKS on AWS">
    <Info>The following prerequisites apply when running APC on Amazon EKS. See the **Other** tab if you run a different version of Kubernetes on AWS.</Info>

    * An EKS Kubernetes cluster, running a version of Kubernetes certified as compatible on the [Kubernetes Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/kubernetes-version-support) that provides the following components:
      * The [Amazon EBS CSI driver](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html) (or an alternative CSI) must be installed on the Kubernetes Cluster.
      * An AWS Load Balancer Controller for the IP target type is required for all private Network Load Balancers (NLBs). See [Installing the AWS Load Balancer Controller add-on](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html).
    * A PostgreSQL instance, accessible from your Kubernetes cluster, and running a version of Postgres certified as compatible on the [Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
    * PostgreSQL superuser permissions.
    * Permission to create and modify resources on AWS.
    * Permission to generate a certificate that covers a defined set of subdomains.
    * An SMTP service and credentials. For example, Mailgun or SendGrid.
    * The [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html).
    * (Optional) [`eksctl`](https://eksctl.io/) for creating and managing your Astronomer cluster on EKS.
    * A machine meeting the following criteria with access to the Kubernetes API Server:
      * Network access to the Kubernetes API Server - either direct access or VPN.
      * Network access to load-balancer resources that are created when APC is installed later in the procedure - either direct access or VPN.
      * Configured to use the DNS servers where APC DNS records can be created.
      * [Helm (minimum v3.6)](https://helm.sh/docs/intro/install).
      * The [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
    * (Situational) The [OpenSSL CLI](https://www.openssl.org/docs/man1.0.2/man1/openssl.html) might be required to troubleshoot certain certificate-related conditions.
  </Tab>

  <Tab title="GKE on GCP">
    <Info>The following prerequisites apply when running APC on Google GKE. See the **Other** tab if you run a different version of Kubernetes on GCP.</Info>

    * A GKE Kubernetes cluster, running a version of Kubernetes listed as compatible on the [Kubernetes Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/kubernetes-version-support).
    * A PostgreSQL instance, accessible from your Kubernetes cluster, and running a version of Postgres certified as compatible on the [Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
    * PostgreSQL superuser permissions.
    * Permission to create and modify resources on Google Cloud Platform.
    * Permission to generate a certificate that covers a defined set of subdomains.
    * An SMTP service and credentials. For example, Mailgun or SendGrid.
    * [Google Cloud SDK](https://cloud.google.com/sdk/install).
    * A machine that meets the following criteria with access to the Kubernetes API Server:
      * Network access to the Kubernetes API Server - either direct access or VPN.
      * Network access to load-balancer resources that are created when APC is installed later in the procedure - either direct access or VPN.
      * Configured to use the DNS servers where APC DNS records can be created.
      * [Helm with minimum version 3.6](https://helm.sh/docs/intro/install).
      * The [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
    * (Optional) The [OpenSSL CLI](https://www.openssl.org/docs/man1.0.2/man1/openssl.html) might be required to troubleshoot certain certificate-related conditions.
  </Tab>

  <Tab title="AKS on Azure">
    <Info>The following prerequisites apply when running APC on Azure AKS. See the **Other** tab if you run a different version of Kubernetes on Azure.</Info>

    * A Kubernetes cluster, running a version of Kubernetes listed as compatible on the [Kubernetes Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/kubernetes-version-support).
    * A PostgreSQL instance, accessible from your Kubernetes cluster, and running a version of Postgres certified as compatible on the [Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
      * If your organization uses Azure Database for PostgreSQL as the database backend, you need to enable the `pg_trgm` extension using the Azure portal or the Azure CLI before you install APC. If you don't enable the `pg_trgm` extension, the install fails. For more information about enabling the `pg_trgm` extension, see [PostgreSQL extensions in Azure Database for PostgreSQL - Flexible Server](https://docs.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions).
    * PostgreSQL superuser permissions.
    * Permission to create and modify resources on Azure.
    * Permission to generate a certificate that covers a defined set of subdomains.
    * An SMTP service and credentials. For example, Mailgun or SendGrid.
    * The [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest).
    * A machine meeting the following criteria with access to the Kubernetes API Server:
      * Network access to the Kubernetes API Server - either direct access or VPN.
      * Network access to load-balancer resources created when APC is installed later in the procedure - either direct access or VPN.
      * Configured to use the DNS servers where APC DNS records are created.
      * [Helm (minimum v3.6)](https://helm.sh/docs/intro/install).
      * The [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
    * (Optional) The [OpenSSL CLI](https://www.openssl.org/docs/man1.0.2/man1/openssl.html) might be required to troubleshoot certain certificate-related conditions.
  </Tab>

  <Tab title="Other">
    The following prerequisites apply when running APC on Kubernetes.

    * A Kubernetes cluster. For versioning considerations, see [Kubernetes Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/kubernetes-version-support).
    * A PostgreSQL instance accessible from your Kubernetes cluster. For versioning considerations, see [Version Compatibility Reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
    * PostgreSQL superuser permissions.
    * An SMTP service and credentials. For example, Mailgun or SendGrid.
    * Permission to generate a certificate that covers a defined set of subdomains.
    * The ability to create DNS records.
    * A machine with access to the Kubernetes API Server meeting the following criteria:
      * Network access to the Kubernetes API Server - either direct access or VPN.
      * Network access to load-balancer resources created when APC is installed later in the procedure - either direct access or VPN.
      * Configured to use the DNS servers where APC DNS records are created.
      * [Helm (minimum v3.6)](https://helm.sh/docs/intro/install).
      * The [Kubernetes CLI (kubectl)](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
    * (Optional) The [OpenSSL CLI](https://www.openssl.org/docs/man1.0.2/man1/openssl.html) might be required to troubleshoot certain certificate-related conditions.
  </Tab>
</Tabs>

<Info>Ensure your cluster meets both the [control plane](/docs/astro-private-cloud/v-2-x/install-control-plane) and [data plane](/docs/astro-private-cloud/v-2-x/install-data-plane) prerequisites, because unified mode deploys services from each.</Info>

<a />

### Ingress controller considerations

Astro Private Cloud requires a Kubernetes Ingress controller to function and provides an integrated Ingress controller by default. Before installing, decide whether to use a third-party ingress controller or use the integrated ingress controller.

Astronomer generally recommends you use the integrated Ingress controller, but Astro Private Cloud also supports certain third-party [ingress-controllers](/docs/astro-private-cloud/v-2-x/third-party-ingress-controllers).

Ingress controllers typically need elevated permissions, including a `ClusterRole`, to function. Specifically, the Astro Private Cloud Ingress controller requires the ability to:

* List all namespaces in the cluster.
* View ingresses in the namespaces.
* Retrieve secrets in the namespaces to locate and use private TLS certificates that service the ingresses.

If you have complex regulatory requirements, you might need to use an Ingress controller that's approved by your organization and disable the integrated controller. You configure the Ingress controller during the installation.

<a />

## Step 1: Plan the structure of your Astro Private Cloud environments

Before installing APC, consider how many instances of the platform you want to host because you install each of these instances on separate Kubernetes clusters, following the instructions in this document.

Each instance of APC can host multiple Airflow environments, or Deployments. Some common types of APC instances you might consider hosting are:

* Sandbox: The lowest environment that contains no sensitive data, used only by system-administrators to experiment, and not subject to change control.
* Development: User-accessible environment that is subject to most of the same restrictions of higher environments, with relaxed change control rules.
* Staging: All network, security, and patch versions are maintained at the same level as in the production environment. However, it provides no availability guarantees and includes relaxed change control rules.
* Production: The production instance hosts your production Airflow environments. You can choose to host development Airflow environments here or in environments with lower levels of support and restrictions.

Plan each environment as a pairing of one control plane with one or more data planes. Create a project folder for every environment you plan to host to contain its configuration files. For example, if you want to install a development environment, create a folder named `~/astronomer-dev/unified`.

In addition to the default cluster in unified mode, additional data plane clusters can be registered by following the [Install a Data Plane](/docs/astro-private-cloud/v-2-x/install-data-plane) guide.

<Info>Certain files in the project directory might contain secrets when you set up your sandbox or development environments. For your first install, keep these secrets in a secure place on a suitable machine. As you progress to higher environments, such as staging or production, secure these files separately in a vault and use the remaining project files in your directory to serve as the basis for your CI/CD deployment.</Info>

<a />

## Step 2: Create `values.yaml` from a template

APC uses Helm to apply platform-level configurations. Choose your cloud provider tab below to copy a ready-to-use `values.yaml`. Then, in the following steps, update image tags, domains, and secrets before deploying.

<Warning>
  As you work with the template configuration, use the following guidelines to avoid installation issues:

  * Don't make any changes to the `values.yaml` file until instructed to do.
  * Don't run `helm upgrade` or `upgrade.sh` until instructed to do so.
  * Ignore any instructions to run `helm upgrade` from other Astronomer documentation until after you complete this unified mode installation procedure.
</Warning>

<Tabs>
  <Tab title="EKS on AWS">
    ```yaml expandable wrap theme={null}
    ###########################################
    ### Astronomer global configuration for EKS
    ###########################################
    global:
      # Installation mode for the control plane
      plane:
        mode: unified

      # Base domain for all control plane subdomains exposed through ingress
      baseDomain: env.astronomer.your.domain

      # For development or proof-of-concept, you can use an in-cluster database.
      # This NOT supported in production.
      postgresql:
        enabled: false

      # Name of secret containing TLS certificate, change if not using "astronomer-tls"
      # tlsSecret: astronomer-tls

      # List of secrets containing the cert.pem of trusted private certification authorities
      # Example command: `kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
      # privateCaCerts:
      # - private-root-ca

      # Expose Postgres metrics for Prometheus to scrape
      # prometheusPostgresExporter:
      #   enabled: true

      # Use a sidecar for exporting task logs
      logging:
        loggingSidecar:
          enabled: true
          name: sidecar-log-consumer

      # Enable dag-only deployments
      deployMechanisms:
        dagOnlyDeployment:
          enabled: true

      # Database SSL configuration
      ssl:
        # Enable SSL connection to Postgres -- must be false if using in-cluster database
        enabled: true

    #########################
    ### Ingress configuration
    #########################
    # nginx:
      # Static IP address the nginx ingress should bind to
      # loadBalancerIP: ~

      # Set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
      # privateLoadBalancer: true

      # Dictionary of arbitrary annotations to add to the nginx ingress.
      # For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
      # Change to 'elb' if your node group is private and doesn't utilize a NAT gateway
      # ingressAnnotations: {service.beta.kubernetes.io/aws-load-balancer-type: nlb}

      # If all subnets are private, auto-discovery may fail.
      # You must enter the subnet IDs manually in the annotation below.
      # service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-id-1,subnet-id-2

    ################################
    ### Astronomer app configuration
    ################################
    astronomer:
      houston:
        upgradeDeployments:
          enabled: false

        # Application configuration for Houston
        config:
          publicSignups: true  ## set to false immediately after initial system admin user created

          # Allowed user email domains for system level roles
          # allowedSystemLevelDomains: []

          # Default configuration for deployments.
          # Can be overridden on a per-data-plane basis.
          deployments:
            # Enable Airflow 3 deployments for clusters
            runtimeManagement:
              airflowV3:
                enabled: true

            # Allow deletions to immediately remove the database and namespace
            # deploymentLifecycle:
            #   hardDeleteDeployment:
            #     enabled: true

            # Allows you to set your release names
            # namespaceManagement:
            #   manualReleaseNames:
            #     enabled: true

            # Flag to enable using IAM roles (don't enter a specific role)
            # Enables the API for updating deployments
            # deploymentImagesRegistry:
            #   serviceAccountAnnotationKey: eks.amazonaws.com/role-arn
            #   updateDeploymentImageEndpoint:
            #     enabled: true

            # Required if dagOnlyDeployment is enabled
            deployMechanisms:
              configureDagDeployment:
                enabled: true
            # upsertDeploymentEnabled: true

          # email:
          #   enabled: false
          #   reply: noreply@your.domain

          # secret:
          # - envName: "EMAIL__SMTP_URL"  # Reference to the Kubernetes secret for SMTP credentials. Only required if email is used.
          #   secretName: "astronomer-smtp"
          #   secretKey: "connection"

          # User authentication mechanism. One of the following should be enabled.
          auth:
            github:
              # Allow users authenticate with Github, enabled by default
              enabled: false
            # local:
            #   # Allow users and passwords in the Houston database, disabled by default
            #   enabled: false
            openidConnect:
              # okta:
              #   enabled: false
              # microsoft:
              #   enabled: false
              # adfs:
              #   enabled: false
              # custom:
              #   enabled: false
              google:
                # Allow users to authenticate with Google, enabled by default
                enabled: false

    #################################
    ## Default tagged groups enabled
    #################################
    # tags:
      # Enable platform components by default (nginx, astronomer)
      # platform: true

      # Enable monitoring stack (prometheus, kube-state)
      # monitoring: true

      # Enable logging stack (elasticsearch, vector)
      # logging: true
    ```
  </Tab>

  <Tab title="GKE on GCP">
    ```yaml expandable wrap theme={null}
    ###########################################
    ### Astronomer global configuration for GKE
    ###########################################
    global:
      # Installation mode for the control plane
      plane:
        mode: unified

      # Base domain for all control plane subdomains exposed through ingress
      baseDomain: env.astronomer.your.domain

      # For development or proof-of-concept, you can use an in-cluster database.
      # This NOT supported in production.
      postgresql:
        enabled: false

      # Name of secret containing TLS certificate, change if not using "astronomer-tls"
      # tlsSecret: astronomer-tls

      # List of secrets containing the cert.pem of trusted private certification authorities
      # Example command: `kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
      # privateCaCerts:
      # - private-root-ca

      # Expose Postgres metrics for Prometheus to scrape
      # prometheusPostgresExporter:
      #   enabled: true

      # Use a sidecar for exporting task logs
      logging:
        loggingSidecar:
          enabled: true
          name: sidecar-log-consumer

      # Enable dag-only deployments
      deployMechanisms:
        dagOnlyDeployment:
          enabled: true

      # Database SSL configuration
      ssl:
        # Enable SSL connection to Postgres -- must be false if using in-cluster database
        enabled: true

    #########################
    ### Ingress configuration
    #########################

    # nginx:
      # Static IP address the nginx ingress should bind to
      # loadBalancerIP: ~

      # Set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
      # privateLoadBalancer: true

      # Dictionary of arbitrary annotations to add to the nginx ingress.
      # For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
      # ingressAnnotations: {}

    ################################
    ### Astronomer app configuration
    ################################
    astronomer:
      houston:
        upgradeDeployments:
          enabled: false

        # Application configuration for Houston
        config:
          publicSignups: true  ## set to false immediately after initial system admin user created

          # Allowed user email domains for system level roles
          # allowedSystemLevelDomains: []

          # Default configuration for deployments.
          # Can be overridden on a per-data-plane basis.
          deployments:
            # Enable Airflow 3 deployments for clusters
            runtimeManagement:
              airflowV3:
                enabled: true

            # Allow deletions to immediately remove the database and namespace
            # deploymentLifecycle:
            #   hardDeleteDeployment:
            #     enabled: true

            # Allows you to set your release names
            # namespaceManagement:
            #   manualReleaseNames:
            #     enabled: true

            # Flag to enable using IAM roles (don't enter a specific role)
            # Enables the API for updating deployments
            # deploymentImagesRegistry:
            #   serviceAccountAnnotationKey: iam.gke.io/gcp-service-account
            #   updateDeploymentImageEndpoint:
            #     enabled: true

            # Required if dagOnlyDeployment is enabled
            deployMechanisms:
              configureDagDeployment:
                enabled: true
            # upsertDeploymentEnabled: true

          # email:
          #   enabled: false
          #   reply: noreply@your.domain

          # secret:
          # - envName: "EMAIL__SMTP_URL"  # Reference to the Kubernetes secret for SMTP credentials. Only required if email is used.
          #   secretName: "astronomer-smtp"
          #   secretKey: "connection"

          # User authentication mechanism. One of the following should be enabled.
          auth:
            github:
              # Allow users authenticate with Github, enabled by default
              enabled: false
            # local:
            #   # Allow users and passwords in the Houston database, disabled by default
            #   enabled: false
            openidConnect:
              # okta:
              #   enabled: false
              # microsoft:
              #   enabled: false
              # adfs:
              #   enabled: false
              # custom:
              #   enabled: false
              google:
                # Allow users to authenticate with Google, enabled by default
                enabled: false

    #################################
    ## Default tagged groups enabled
    #################################
    # tags:
      # Enable platform components by default (nginx, astronomer)
      # platform: true

      # Enable monitoring stack (prometheus, kube-state)
      # monitoring: true

      # Enable logging stack (elasticsearch, vector)
      # logging: true
    ```
  </Tab>

  <Tab title="AKS on Azure">
    ```yaml expandable wrap theme={null}
    ###########################################
    ### Astronomer global configuration for AKS
    ###########################################
    global:
      # Installation mode for the control plane
      plane:
        mode: unified

      # Base domain for all control plane subdomains exposed through ingress
      baseDomain: env.astronomer.your.domain

      # For development or proof-of-concept, you can use an in-cluster database.
      # This NOT supported in production.
      postgresql:
        enabled: false

      # Name of secret containing TLS certificate, change if not using "astronomer-tls"
      # tlsSecret: astronomer-tls

      # List of secrets containing the cert.pem of trusted private certification authorities
      # Example command: `kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
      # privateCaCerts:
      # - private-root-ca

      # Expose Postgres metrics for Prometheus to scrape
      # prometheusPostgresExporter:
      #   enabled: true

      # Use a sidecar for exporting task logs
      logging:
        loggingSidecar:
          enabled: true
          name: sidecar-log-consumer

      # Enable dag-only deployments
      deployMechanisms:
        dagOnlyDeployment:
          enabled: true

      # Database SSL configuration
      ssl:
        # Enable SSL connection to Postgres -- must be false if using in-cluster database
        enabled: true

    #########################
    ### Ingress configuration
    #########################

    # nginx:
      # Static IP address the nginx ingress should bind to
      # loadBalancerIP: ~

      # Set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
      # privateLoadBalancer: true

      # Dictionary of arbitrary annotations to add to the nginx ingress.
      # For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
      # required for azure load balancer post Kubernetes 1.24
      ingressAnnotations:
          service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: "/healthz"

    ################################
    ### Astronomer app configuration
    ################################
    astronomer:
      houston:
        upgradeDeployments:
          enabled: false

        # Application configuration for Houston
        config:
          publicSignups: true  ## set to false immediately after initial system admin user created

          # Allowed user email domains for system level roles
          # allowedSystemLevelDomains: []

          # Default configuration for deployments.
          # Can be overridden on a per-data-plane basis.
          deployments:
            # Enable Airflow 3 deployments for clusters
            runtimeManagement:
              airflowV3:
                enabled: true

            # Allow deletions to immediately remove the database and namespace
            # deploymentLifecycle:
            #   hardDeleteDeployment:
            #     enabled: true

            # Allows you to set your release names
            # namespaceManagement:
            #   manualReleaseNames:
            #     enabled: true

            # Flag to enable using IAM roles (don't enter a specific role)
            # Enables the API for updating deployments
            # deploymentImagesRegistry:
            #   serviceAccountAnnotationKey:
            #   updateDeploymentImageEndpoint:
            #     enabled: true

            # Required if dagOnlyDeployment is enabled
            deployMechanisms:
              configureDagDeployment:
                enabled: true
            # upsertDeploymentEnabled: true

          # email:
          #   enabled: false
          #   reply: noreply@your.domain

          # secret:
          # - envName: "EMAIL__SMTP_URL"  # Reference to the Kubernetes secret for SMTP credentials. Only required if email is used.
          #   secretName: "astronomer-smtp"
          #   secretKey: "connection"

          # User authentication mechanism. One of the following should be enabled.
          auth:
            github:
              # Allow users authenticate with Github, enabled by default
              enabled: false
            # local:
            #   # Allow users and passwords in the Houston database, disabled by default
            #   enabled: false
            openidConnect:
              # okta:
              #   enabled: false
              # microsoft:
              #   enabled: false
              # adfs:
              #   enabled: false
              # custom:
              #   enabled: false
              google:
                # Allow users to authenticate with Google, enabled by default
                enabled: false

    #################################
    ## Default tagged groups enabled
    #################################
    # tags:
      # Enable platform components by default (nginx, astronomer)
      # platform: true

      # Enable monitoring stack (prometheus, kube-state)
      # monitoring: true

      # Enable logging stack (elasticsearch, vector)
      # logging: true
    ```
  </Tab>

  <Tab title="Other">
    ```yaml expandable wrap theme={null}
    #######################################################
    ### Astronomer global configuration for other providers
    #######################################################
    global:
      # Installation mode for the control plane
      plane:
        mode: unified

      # Base domain for all control plane subdomains exposed through ingress
      baseDomain: env.astronomer.your.domain

      # For development or proof-of-concept, you can use an in-cluster database.
      # This NOT supported in production.
      postgresql:
        enabled: false

      # Name of secret containing TLS certificate, change if not using "astronomer-tls"
      # tlsSecret: astronomer-tls

      # List of secrets containing the cert.pem of trusted private certification authorities
      # Example command: `kubectl -n astronomer create secret generic private-root-ca --from-file=cert.pem=./private-root-ca.pem`
      # privateCaCerts:
      # - private-root-ca

      # Expose Postgres metrics for Prometheus to scrape
      # prometheusPostgresExporter:
      #   enabled: true

      # Use a sidecar for exporting task logs
      logging:
        loggingSidecar:
          enabled: true
          name: sidecar-log-consumer

      # Enable dag-only deployments
      deployMechanisms:
        dagOnlyDeployment:
          enabled: true

      # Database SSL configuration
      ssl:
        # Enable SSL connection to Postgres -- must be false if using in-cluster database
        enabled: true

    #########################
    ### Ingress configuration
    #########################

    # nginx:
      # Static IP address the nginx ingress should bind to
      # loadBalancerIP: ~

      # Set privateLoadbalancer to 'false' to make nginx request a LoadBalancer on a public vnet
      # privateLoadBalancer: true

      # Dictionary of arbitrary annotations to add to the nginx ingress.
      # For full configuration options, see https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
      # ingressAnnotations: {}

    ################################
    ### Astronomer app configuration
    ################################
    astronomer:
      houston:
        upgradeDeployments:
          enabled: false

        # Application configuration for Houston
        config:
          publicSignups: true  ## set to false immediately after initial system admin user created

          # Allowed user email domains for system level roles
          # allowedSystemLevelDomains: []

          # Default configuration for deployments.
          # Can be overridden on a per-data-plane basis.
          deployments:
            # Enable Airflow 3 deployments for clusters
            runtimeManagement:
              airflowV3:
                enabled: true

            # Allow deletions to immediately remove the database and namespace
            # deploymentLifecycle:
            #   hardDeleteDeployment:
            #     enabled: true

            # Allows you to set your release names
            # namespaceManagement:
            #   manualReleaseNames:
            #     enabled: true

            # Flag to enable using IAM roles (don't enter a specific role)
            # Enables the API for updating deployments
            # deploymentImagesRegistry:
            #   serviceAccountAnnotationKey:
            #   updateDeploymentImageEndpoint:
            #     enabled: true

            # Required if dagOnlyDeployment is enabled
            deployMechanisms:
              configureDagDeployment:
                enabled: true
            # upsertDeploymentEnabled: true

          # email:
          #   enabled: false
          #   reply: noreply@your.domain

          # secret:
          # - envName: "EMAIL__SMTP_URL"  # Reference to the Kubernetes secret for SMTP credentials. Only required if email is used.
          #   secretName: "astronomer-smtp"
          #   secretKey: "connection"

          # User authentication mechanism. One of the following should be enabled.
          auth:
            github:
              # Allow users authenticate with Github, enabled by default
              enabled: false
            # local:
            #   # Allow users and passwords in the Houston database, disabled by default
            #   enabled: false
            openidConnect:
              # okta:
              #   enabled: false
              # microsoft:
              #   enabled: false
              # adfs:
              #   enabled: false
              # custom:
              #   enabled: false
              google:
                # Allow users to authenticate with Google, enabled by default
                enabled: false

    #################################
    ## Default tagged groups enabled
    #################################
    # tags:
      # Enable platform components by default (nginx, astronomer)
      # platform: true

      # Enable monitoring stack (prometheus, kube-state)
      # monitoring: true

      # Enable logging stack (elasticsearch, vector)
      # logging: true
    ```
  </Tab>
</Tabs>

<Tip>Email delivery is disabled by default. If you want to enable it, you can configure it in a later step: [Configure outbound SMTP email](#configure-outbound-smtp-email).</Tip>

### Configure public signups

The `apc-values.yaml` examples leave `astronomer.houston.config.publicSignups: true`, so you can create the initial administrator account. You can control account creation in [Disable anonymous account creation](#disable-anonymous-account-creation).

<a />

## Step 3: Choose and configure a base domain

When you install APC it creates a variety of services that your users access to manage, monitor, and run Airflow.

Choose a base domain such as `astronomer.example.com`, `astro-sandbox.example.com`, or `astro-prod.example.internal` for which:

* You have the ability to create and edit DNS records
* You have the ability to issue TLS certificates
* The following addresses are used by Astronomer components:
  * `app.<base-domain>`
  * `deployments.<base-domain>`
  * `houston.<base-domain>`
  * `alertmanager.<base-domain>`
  * `prometheus.<base-domain>`
  * `registry.<base-domain>`

<Tip>
  The base domain itself doesn't need to be available and can point to another service not associated with Astronomer or Airflow. If the base domain is available, you can choose to establish a vanity redirect from `<base-domain>` to `app.<base-domain>` later in the installation process.
</Tip>

When choosing a base domain, consider the following:

* The name you choose must be resolvable by both your users and Kubernetes itself.
* All hostnames must remain under the base domain (for example, `app.<base-domain>`), so ensure you can create DNS records and issue TLS certificates for those subdomains.
* You need to have or obtain a TLS certificate that is recognized as valid by your users. If you use the APC integrated container registry, the TLS certification must also be recognized as valid by Kubernetes itself.
* Wildcard certificates are only valid one level deep. For example, an ingress controller that uses a certificate called `*.example.com` can provide service for `app.example.com` but not `app.astronomer-dev.example.com`.
* The bottom-level hostnames, such as `app`, `registry`, or `prometheus`, are fixed and can't be changed.

The base domain is visible to end users. They can view the base domain in the following scenarios:

* When users access the APC UI. For example, `https://app.sandbox-astro.example.com`.
* When users access an Airflow Deployment. For example, `https://deployments.sandbox-astro.example.com/deployment-release-name/airflow`.
* When users authenticate to the Astro CLI. For example, `astro login sandbox-astro.example.com`.

<Info>If you install APC on OpenShift and also want to use OpenShift's integrated ingress controller, you can use the hostname of the default OpenShift ingress controller as your base domain, such as `app.apps.<OpenShift-domain>`. Doing this requires permission to reconfigure the route admission policy for the standard ingress controller to `InterNamespaceAllowed`. See [Third Party Ingress Controller - Configuration notes for OpenShift](/docs/astro-private-cloud/v-2-x/third-party-ingress-controllers#required-environment-configuration-openshift) for additional information and options.</Info>

### Configure the base domain

Locate the `global.baseDomain` in your `values.yaml` file and change it to your base domain as shown in the following example:

```yaml wrap theme={null}
global:
  # Base domain for all subdomains exposed through ingress
  baseDomain: sandbox-astro.example.com
```

<a />

## Step 4: Create the APC platform namespace

In your Kubernetes cluster, create a [Kubernetes namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) to contain the APC platform. The following example uses `apc` as the namespace.

```bash wrap theme={null}
kubectl create namespace apc
```

APC uses the contents of this namespace to provision and manage Airflow instances running in other namespaces. Each Airflow instance has its own isolated namespace.

<a />

## Step 5: Request and validate an Astronomer TLS certificate

To install APC you need a TLS certificate that is valid for several domains. One of the domains is the primary name on the certificate, also known as the common name (CN). The additional domains are equally valid, supplementary domains known as Subject Alternative Names (SANs).

<Warning>
  Astronomer requires a private certificate in the APC platform namespace, even if you use a third-party ingress controller that doesn't otherwise require it.
</Warning>

<a />

### Request an ingress controller TLS certificate

Request a TLS certificate from your security team for APC. In your request, include the following:

* Your chosen base domain as the Common Name (CN). If your certificate authority won't issue certificates for the bare base domain, use `app.<base-domain>` as the CN instead.
* *Either* request a wildcard SAN of `*.<base-domain>` (plus an explicit SAN for `<base-domain>`) *or* list each hostname individually:
  * `app.<base-domain>` (omit if already used as the Common Name)
  * `deployments.<base-domain>` (required for Airflow UIs and APIs)
  * `houston.<base-domain>`
  * `prometheus.<base-domain>`
  * `registry.<base-domain>` (required if you keep the integrated container registry enabled)
  * `alertmanager.<base-domain>` (required if you keep the integrated Alertmanager enabled)
* If you use the APC integrated container registry, specify that the encryption type of the certificate must be RSA.
* Request the following return format:
  * A `key.pem` containing the private key in pem format
  * *Either* a `full-chain.pem` (containing the public certificate and additional certificates required to validate it, in pem format) *or* a bare `cert.pem` and explicit affirmation that there are no intermediate certificates and that the public certificate is the full chain.
  * *Either* the `private-root-ca.pem` in pem format of the private Certificate Authority used to create your certificate or a statement that the certificate is signed by a public Certificate Authority.

<Warning>If you use the APC integrated container registry, the encryption type used on your TLS certificate must be **RSA**. Certbot users must include `--key-type rsa` when requesting certificates. Most other solutions generate RSA keys by default.</Warning>

### Validate the received certificate and associated items

Ensure that you received each of the following three items:

* A `key.pem` containing the private key in pem format.
* *Either* a `full-chain.pem`, in pem format, that contains the public certificate and additional certificates required to validate it *or* a bare `cert.pem` and explicit affirmation that there are no intermediate certificates and that the public certificate is the full chain.
* *Either* the `private-root-ca.pem` in pem format of the private Certificate Authority used to create your certificate *or* a statement that the certificate is signed by public Certificate Authority.

To validate that your security team generated the correct certificate, run the following command using the `openssl` CLI:

```bash wrap theme={null}
openssl x509 -in <your-certificate-filepath> -text -noout
```

This command generates a report. If the `X509v3 Subject Alternative Name` section of this report includes either a single `*.<base-domain>` wildcard domain or all subdomains, then the certificate creation was successful.

Confirm that your full-chain certificate chain is ordered correctly. To determine your certificate chain order, run the following command using the `openssl` CLI:

```bash wrap theme={null}
openssl crl2pkcs7 -nocrl -certfile <your-full-chain-certificate-filepath> | openssl pkcs7 -print_certs -noout
```

The command generates a report of all certificates. Verify that the certificates are in the following order:

* Domain
* (Optional) Intermediate
* Root

<a />

### (Optional) Additional validation for the Astronomer integrated container registry

<Check>If you don't plan to store images in Astronomer's integrated container registry and instead plan to store all container images using an external container registry, you can skip this step.</Check>

The APC integrated container registry requires that your private key signs traffic originating from the APC platform using the RSA encryption method. Confirm that the key is signing traffic correctly before proceeding.

Run the following command to extract the bare public cert, if it wasn't already included in the files provided by your certificate authority, from the full-chain certificate file:

```bash wrap theme={null}
openssl crl2pkcs7 -nocrl -certfile full-chain.pem | openssl pkcs7 -print_certs -noout > cert.pem
```

Examine the public certificate and ensure all Signature Algorithms are listed as `sha1WithRSAEncryption`.

```bash wrap theme={null}
openssl x509 -in cert.pem -text|grep Algorithm
        Signature Algorithm: sha1WithRSAEncryption
            Public Key Algorithm: rsaEncryption
    Signature Algorithm: sha1WithRSAEncryption
```

If your key isn't compatible with the APC integrated container registry, ask your Certificate Authority to [re-issue the credentials](#request-a-certificate-bundle) and emphasize the need for an RSA cert, or plan to use an external container registry instead.

<a />

## Step 6: Store and configure the ingress controller TLS certificate

Determine whether or not your certificate was issued by an intermediate certificate-authority. If you don't know, assume you use an intermediate certificate and attempt to obtain a `full-chain.pem` bundle from your certificate authority.

Certificates issued by operators of root certificate authorities, including but not limited to LetsEncrypt, are frequently issued from intermediate certificate authorities associated with a trusted root CA.

<Warning>APC backend services have stricter trust requirements than most web-browsers. Web Browsers might auto-complete the chain and consider your certificate valid, even if you don't provide the intermediate certificate-authority's public certificate. APC backend services can reject the same certificate, and cause Dag and image deploys to fail.</Warning>

If, and only if, your certificate was issued directly by the root Certificate Authority of a universally trusted certificate authority, and not from one of their intermediaries, then the `server.crt` is also the full-chain certificate bundle.

Identify your full-chain public certificate `.pem` file and use it while storing and configuring the ingress controller TLS certificate.

<Warning>The `--cert` parameter must reference your `full-chain.pem`, which includes the server certificate *and* any intermediate certificates, if any. Using the server cert directly causes Dag and image deploys to fail.</Warning>

Run the following command to store the public full-chain certificate in the APC Platform Namespace in a `tls`-type Kubernetes secret. You can create a custom name for this secret. The following example uses the name `astronomer-tls`.

```bash wrap theme={null}
kubectl -n <astronomer platform namespace> create secret tls astronomer-tls --cert <fullchain-pem-filepath> --key <your-private-key-filepath>
```

However, if your security team has instructed you that there are no intermediate certificates, run the following command.

```bash wrap theme={null}
kubectl -n astronomer create secret tls astronomer-tls --cert full-chain.pem --key server_private_key.pem
```

Astronomer recommends naming the secret `astronomer-tls` when using a third-party ingress controller.

<a />

## Step 7: (Optional) Configure a third-party ingress controller

<Check>
  If you use APC's integrated ingress controller, you can skip this step.
</Check>

Complete the full setup as described in [Third-party Ingress-Controllers](/docs/astro-private-cloud/v-2-x/third-party-ingress-controllers), which includes steps to configure ingress controllers in specific environment types. When you're done, return to this page and continue to the next step.

<a />

## Step 8: Configure a private certificate authority

<Check>
  Skip this step if you don't use a private Certificate Authority (private CA) to sign the certificate used by your ingress-controller. Or, if you don't use a private CA for any of the following services that the APC platform interacts with.
</Check>

APC trusts public Certificate Authorities automatically.

APC must be configured to trust any private Certificate Authorities issuing certificates for systems APC interacts with, including, but not limited to the following:

* Ingress controller
* Email server, unless disabled
* Any container registries that Kubernetes pulls from
* If using OAuth, the OAuth provider
* If using external Elasticsearch, any external Elasticsearch instances
* If using external Prometheus, any external Prometheus instances

Perform the procedure described in [Configuring private CAs](/docs/astro-private-cloud/v-2-x/configure-private-cas) for each certificate authority used to sign TLS certificates. After creating the trust secret (for example `astronomer-ca`), add it to `global.privateCaCerts` in `values.yaml` so platform components trust the issuer.

<Info>Astro CLI users must also configure both their operating system and container solution, [Docker Desktop or Podman](/docs/astro-private-cloud/v-2-x/configure-desktop-container-solution-extra-cas), to trust the private certificate Authority that was used to create the certificate used by the APC ingress controller and any third-party container registries.</Info>

<a />

## Step 9: Confirm your Kubernetes cluster trusts required CAs

<Check>
  If at least one of the following circumstances apply to your installation, you must complete this step:

  * You configured APC to pull [platform container images](#configure-a-private-docker-registry-platform) from an external container registry that uses a certificate signed by a private CA.
  * You plan for your users to deploy Airflow images to APC's integrated container registry *and* Astronomer is using a TLS certificate issued by a private CA.
  * Users will deploy images to an external container registry *and* that registry is using a TLS certificate issued by a private CA.
</Check>

Kubernetes must be able to pull images from one or more container registries for APC to function. By default, Kubernetes only trusts publicly signed certificates. This means that by default, Kubernetes doesn't honor the list of certificates [trusted by the APC platform](/docs/astro-private-cloud/v-2-x/configure-private-cas).

Many enterprises configure Kubernetes to trust additional certificate authorities as part of their standard cluster creation procedure. Contact your Kubernetes Administrator to find out what, if any, private certificates are currently trusted by your Kubernetes cluster. Then, consult your Kubernetes administrator and Kubernetes provider's documentation for instructions on configuring Kubernetes to trust additional CAs.

Follow procedures for your Kubernetes provider to configure Kubernetes to trust each CA associated with your container registries, including the integrated container registry, if applicable.

Certain clusters don't provide a mechanism to configure the list of certificates trusted by Kubernetes.

While configuring the Kubernetes list of cluster certificates is a customer responsibility, APC includes an optional component that can, for certain Kubernetes cluster configurations, add certificates defined in `global.privateCaCerts` to the list of certificates trusted by Kubernetes. This can be enabled by setting `global.privateCaCertsAddToHost.enabled` and `global.privateCaCertsAddToHost.addToContainerd` to `true` in your `values.yaml` file and setting `global.privateCaCertsAddToHost.containerdConfigToml` to:

```yaml wrap theme={null}
[host."https://registry.<baseApp>"]
  ca = "/etc/containerd/certs.d/<registry hostname>/<secret name>.pem"
```

For example, if your base domain is `astro-sandbox.example.com` and the CA public-certificate is stored in the platform namespace in a secret named `my-private-ca`, the `global.privateCaCertsAddToHost` section would be:

```yaml wrap theme={null}
  global:
    privateCaCertsAddToHost:
      enabled: true
      addToContainerd: true
      hostDirectory: /etc/containerd/certs.d
      containerdConfigToml: |-
        [host."https://registry.astro-sandbox.example.com"]
          ca = "/etc/containerd/certs.d/registry.astro-sandbox.example.com/my-private-ca.pem"
```

<a />

## Step 10: Configure outbound SMTP email

APC requires the ability to send email to:

* Notify users of errors with their Airflow Deployments.
* Send emails to invite new users to Astronomer.
* Send certain platform alerts, enabled by default but can be configured.

APC sends all outbound email using SMTP.

<Check>If SMTP isn't available in the environment where you're installing APC follow instructions in [configure APC to not send outbound email](/docs/astro-private-cloud/v-2-x/disable-outbound-email), and then skip the rest of this section.</Check>

1. Obtain a set of SMTP credentials from your email administrator for you to use to send email from APC. When you request an email address and display name, remember that these emails aren't designed for users to reply directly to them. Request all the following information:
   * Email address.
   * Email display name requirements. Some email servers require a **From** line of: `Do Not Reply <donotreply@example.com>`.
   * SMTP username. This is usually the same as the email address.
   * SMTP password.
   * SMTP hostname.
   * SMTP port.
   * Whether or not the connection supports TLS.

<Info>If there is a `/` or any other escape character in your username or password, you might need to [URL encode](https://www.urlencoder.org/) those characters.</Info>

2. Ensure that your Kubernetes cluster has permissions configured to send outbound email to the SMTP server.
3. Change the configuration in `values.yaml` from `noreply@my.email.internal` to an email address that is valid to use with your SMTP credentials.
4. Construct an email connection string and store it in a secret in the Astronomer platform namespace. The following example shows how to store the connection in a secret called `astronomer-smtp`. Make sure to *url-encode* the username and password if they contain special characters.

   ```bash wrap theme={null}
   kubectl -n astronomer create secret generic astronomer-smtp --from-literal connection="smtp://my@40user:my%40pass@smtp.email.internal/?requireTLS=true"
   ```

   In general, an SMTP URI is formatted as `smtps://USERNAME:PASSWORD@HOST/?pool=true`. The following table contains examples of the URI for some of the most popular SMTP services:

   | Provider          | Example SMTP URL                                                                                 |
   | ----------------- | ------------------------------------------------------------------------------------------------ |
   | AWS SES           | `smtp://AWS_SMTP_Username:AWS_SMTP_Password@email-smtp.us-east-1.amazonaws.com/?requireTLS=true` |
   | SendGrid          | `smtps://apikey:SG.sometoken@smtp.sendgrid.net:465/?pool=true`                                   |
   | Mailgun           | `smtps://xyz%40example.com:password@smtp.mailgun.org/?pool=true`                                 |
   | Office365         | `smtp://xyz%40example.com:password@smtp.office365.com:587/?requireTLS=true`                      |
   | Custom SMTP-relay | `smtp://smtp-relay.example.com:25/?ignoreTLS=true`                                               |

   If your SMTP provider isn't listed, refer to the provider's documentation for information on creating an SMTP URI.

<Info>If there is a `/` or any other escape character in your username or password, you might need to [URL encode](https://www.urlencoder.org/) those characters.</Info>

## Step 11: Configure volume storage classes

<Check>
  Skip this step if your cluster defines a volume storage class, and you want to use it for all volumes associated with APC and its Airflow Deployments.
</Check>

Astronomer strongly recommends that you don't back any volumes used for APC with mechanical hard drives.

Create `storage-class-config.yaml` in your project directory and update the configuration to match your environment:

```yaml expandable wrap theme={null}
global:
  prometheus:
    persistence:
      storageClassName: "<desired-storage-class>"
  elasticsearch:
    common:
      persistence:
        storageClassName: "<desired-storage-class>"
astronomer:
  registry:
    persistence:
      storageClassName: "<desired-storage-class>"
  houston:
    config:
      deployments:
        helm:
          dagDeploy:
            persistence:
              storageClass: "<desired-storage-class>"
          airflow:
            redis:
              persistence:
                storageClassName: "<desired-storage-class>"
nats:
  nats:
    jetStream:
      fileStorage:
        storageClassName: "<desired-storage-class>"
# this option doesn't apply when using an external postgres database
# bundled postgresql not a supported option, only for use in proof-of-concepts
postgresql:
  persistence:
    storageClass: "<desired-storage-class>"
```

Merge these values into `values.yaml` manually or by using a YAML merge tool of your choosing.

<a />

## Step 12: Configure the database

Astronomer requires a central Postgres database that acts as the backend for the APC API and hosts individual metadata databases for all Deployments created on the platform.

<Tip>
  If, while evaluating APC you need to create a temporary environment where Postgres isn't available, locate the `global.postgresql.enabled` option already present in your `values.yaml` and set it to `true`, then skip the remainder of this step.

  Note that `global.postgresql.enabled` to `true` is an unsupported configuration, and should never be used on any development, staging, or production environment.
</Tip>

<Info>
  If you use Azure Database for either PostgreSQL or another Postgres instance that doesn't enable the `pg_trgm` by default, you must enable the `pg_trgm` extension prior to installing APC. If `pg_trgm` isn't enabled, the install will fail. `pg_trgm` is enabled by default on Amazon RDS and Google Cloud SQL for PostgreSQL.

  For instructions on enabling the `pg_trgm` extension for Azure Flexible Server, see [PostgreSQL extensions in Azure Database for PostgreSQL - Flexible Server](https://docs.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions).
</Info>

Additional requirements apply to the following databases:

* AWS RDS:
  * [t2 medium](https://aws.amazon.com/rds/instance-types/) is the minimum RDS instance size you can use.
* Azure Flexible Server:
  * You must enable the `pg_trgm` extension as per the advisory earlier in this section.
  * Set `global.ssl.mode`to `prefer` in your `values.yaml` file.

Create a Kubernetes Secret named `astronomer-bootstrap` that points to your database. You must URL encode any special characters in your Postgres password.

<Warning>The in-cluster Postgres option (`global.postgresql.enabled: true`) is deprecated and should only be used for short-lived testing. Always rely on an external Postgres instance for any persistent environment.</Warning>

<Warning>PostgreSQL usernames must be lowercase. </Warning>

To create this secret, run the following command replacing the Astronomer platform namespace, username, password, database hostname, and database port with their respective values. Remember that username and password must be URL-encoded if they contain special-characters:

```bash wrap theme={null}
kubectl -n <astronomer platform namespace> create secret generic astronomer-bootstrap \
  --from-literal connection="postgres://<url-encoded username>:<url-encoded password>@<database hostname>:<database port>"
```

For example, for a username named `bob` with password `abc@abc` at hostname `some.host.internal`, you would run:

```bash wrap theme={null}
kubectl -n astronomer create secret generic astronomer-bootstrap \
  --from-literal connection="postgres://bob:abc%40abc@some.host.internal:5432"
```

<a />

## Step 13: Configure the Docker registry used for platform images

<Check>
  Skip this step if you are installing APC onto a Kubernetes cluster that can pull container images from public image repositories and you don't want to mirror these images locally.
</Check>

<Tabs>
  <Tab title="Anonymous">
    If you can retrieve images from a registry that can be reached without credentials, ensure the endpoint hosting the registry is restricted to trusted networks, for example, private subnets or VPN access. Avoid exposing the platform image registry directly to the public internet. No additional Astronomer configuration is required beyond setting the repository locations later in this step.
  </Tab>

  <Tab title="Amazon ECR">
    1. Grant your worker nodes or IRSA service accounts the IAM permissions required to pull images from the target ECR repository. At minimum, allow `ecr:GetAuthorizationToken`, `ecr:BatchCheckLayerAvailability`, `ecr:GetDownloadUrlForLayer`, and `ecr:BatchGetImage`.
    2. Ensure network access from the cluster to the appropriate ECR endpoints (for example, VPC endpoints or public ECR endpoints).
    3. Set the platform repository prefix in `values.yaml`. For example:

       ```yaml wrap theme={null}
       global:
         privateRegistry:
           enabled: true
           repository: <account-id>.dkr.ecr.<region>.amazonaws.com/<platform-prefix>
       astronomer:
         houston:
           config:
             deployments:
               helm:
                 runtimeImages:
                   airflow:
                     repository: <account-id>.dkr.ecr.<region>.amazonaws.com/<platform-prefix>/astro-runtime
                 runtimeImagesV3:
                   airflow:
                     repository: <account-id>.dkr.ecr.<region>.amazonaws.com/<platform-prefix>/runtime
                 airflow:
                   defaultAirflowRepository: <account-id>.dkr.ecr.<region>.amazonaws.com/<platform-prefix>/astro-runtime
                   defaultRuntimeRepository: <account-id>.dkr.ecr.<region>.amazonaws.com/<platform-prefix>/astro-runtime
       ```

       When you rely on IAM-based authentication, `global.privateRegistry.secretName` isn't required. If you use static credentials, create the matching Docker registry secret following the AWS ECR documentation and set `secretName` accordingly.
  </Tab>

  <Tab title="Other registries">
    1. Create a Docker registry secret in the Astronomer namespace and annotate it so the deployment orchestrator propagates the credentials to Deployment namespaces:

       ```bash wrap theme={null}
       kubectl -n <astronomer platform namespace> create secret docker-registry <secret-name> \
         --docker-server=<registry-host> \
         --docker-username=<username> \
         --docker-password=<password> \
         --docker-email=<email>

       kubectl -n <astronomer platform namespace> annotate secret <secret-name> \
         "astronomer.io/commander-sync"="platform=astronomer"
       ```

    2. Update `values.yaml` so the platform charts reference your registry and credentials:

       ```yaml wrap theme={null}
       global:
         privateRegistry:
           enabled: true
           repository: <custom-platform-repo-prefix>
           secretName: <secret-name>
       astronomer:
         houston:
           config:
             deployments:
               helm:
                 runtimeImages:
                   airflow:
                     repository: <custom-platform-repo-prefix>/astro-runtime
                 airflow:
                   defaultAirflowRepository: <custom-platform-repo-prefix>/ap-airflow
                   defaultRuntimeRepository: <custom-platform-repo-prefix>/astro-runtime
       ```

    3. After applying the configuration change, run `kubectl create job -n <astronomer platform namespace> --from=cronjob/<platform-release-name>-config-syncer upgrade-config-synchronization` to push the updated credentials to existing Deployment namespaces.
  </Tab>
</Tabs>

For additional examples (including per-Deployment registry settings and air-gapped workflows), see [Configure a custom registry for Deployment images](/docs/astro-private-cloud/v-2-x/custom-image-registry).

<a />

## Step 14: Determine which version of APC to install

Astronomer recommends that new APC installations use the most recent APC version available. Keep this version number available for the following steps.

See APC's [lifecycle policy](/docs/astro-private-cloud/v-2-x/release-lifecycle-policy) and [version compatibility reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference) for more information.

<a />

## Step 15: Fetch Airflow Helm charts

If you have internet access to `https://helm.astronomer.io`, run the following command on the machine where you want to install APC:

```bash wrap theme={null}
helm repo add astronomer https://helm.astronomer.io/
helm repo update
```

If you don't have internet access to `https://helm.astronomer.io`, download the APC Platform Helm chart file corresponding to the version of APC you are installing or upgrading to from `https://helm.astronomer.io/astronomer-<version number>.tgz`. For example, for APC v1.0.0 you would download `https://helm.astronomer.io/astronomer-1.0.0.tgz`. This file doesn't need to be uploaded to an internal chart repository.

<a />

## Step 16: Create and customize `upgrade.sh`

Create a file named `upgrade.sh` in your platform deployment project directory containing the following script. Specify the following values at the beginning of the script:

* `CHART_VERSION`: Your APC version, including patch and a `v` prefix. For example, `v1.0.0`.
* `RELEASE_NAME`: Your Helm release name. `astronomer` is strongly recommended.
* `NAMESPACE`: The namespace to install platform components into. `astronomer` is strongly recommended.
* `CHART_NAME`: Set to `astronomer/astronomer` if fetching platform images from the internet. Otherwise, specify the filename if you're installing from a file (for example `astronomer-1.0.0.tgz`).

```bash wrap theme={null}
#!/bin/bash
set -xe

# typically astronomer
RELEASE_NAME=<astronomer-platform-release-name>
# typically astronomer
NAMESPACE=<astronomer-platform-namespace>
# typically astronomer/astronomer
CHART_NAME=<chart name>
# format is v<major>.<minor>.<path> e.g. v1.0.0
CHART_VERSION=<v-prefixed version of the APC platform chart>
# ensure all the above environment variables have been set

helm repo add --force-update astronomer https://helm.astronomer.io
helm repo update

# upgradeDeployments false ensures that Airflow charts aren't upgraded when this script is run
# If you deployed a config change that is intended to reconfigure something inside Airflow,
# then you may set this value to "true" instead. When it is "true", then each Airflow chart will
# restart. Note that some stable version upgrades require setting this value to true regardless of your own configuration.
# If you are currently on APC 0.25, 0.26, or 0.27, you must upgrade to version 0.28 before upgrading to 0.29. A direct upgrade to 0.29 from a version lower than 0.28 isn't possible.
helm upgrade --install --namespace $NAMESPACE \
            -f ./values.yaml \
            --reset-values \
            --version $CHART_VERSION \
            --debug \
            --set astronomer.houston.upgradeDeployments.enabled=false \
            $RELEASE_NAME \
            $CHART_NAME $@
```

<a />

## Step 17: Mirror platform images

<Warning>This step is optional but strongly recommended for production environments so your cluster can pull platform images from a registry you control.</Warning>

1. Gather the list of required platform images using one of the following methods:

<Tabs>
  <Tab title="Shell">
    Mac and Linux users with `jq` installed can set `CHART_VERSION` in the following snippet and run it to produce a list of images.

    ```bash wrap theme={null}
    CHART_VERSION=<v-prefixed version of the APC platform chart>
    UNPREFIXED_CHART_VERSION=${CHART_VERSION#v}
    curl -s https://updates.astronomer.io/astronomer-software/releases/astronomer-${UNPREFIXED_CHART_VERSION}.json | jq -r '(.astronomer.images, .airflow.images) | to_entries[] | "\(.value.repository):\(.value.tag)"'| sort
    ```
  </Tab>

  <Tab title="Windows Powershell">
    Windows PowerShell users can set `CHART_VERSION` in the following snippet and run it to produce a list of images.

    ```powershell wrap theme={null}
    $CHART_VERSION = "<v-prefixed version>"
    $UNPREFIXED_CHART_VERSION = $CHART_VERSION.TrimStart('v')
    $jsonUrl     = "https://updates.astronomer.io/astronomer-software/releases/astronomer-$UNPREFIXED_CHART_VERSION.json"
    $jsonContent = Invoke-WebRequest $jsonUrl -UseBasicParsing
    $json        = $jsonContent.Content | ConvertFrom-Json
    $astronomerImages = $json.astronomer.images.PSObject.Properties.Value
    $airflowImages    = $json.airflow.images.PSObject.Properties.Value

    $images = $astronomerImages + $airflowImages

    $images |
        ForEach-Object { "$($_.repository):$($_.tag)" } |
        Sort-Object
    ```
  </Tab>

  <Tab title="Other">
    Visit the [release metadata](https://updates.astronomer.io/astronomer-software/releases/index.html) page and download the json-formatted release metadata corresponding to the version of APC you are installing and use another method of your choice to extract the list of images from beneath the `astronomer.images` and `airflow.images` keys.
  </Tab>
</Tabs>

2. Copy these images to the container registry using the naming scheme you configured [when you set up a custom image registry](#configure-a-private-docker-registry-platform).

<a />

## Step 18: Fetch Airflow/Astro Runtime updates

If you are installing APC into an egress-controlled or air-gapped environment, perform the following steps.

By default, APC checks for Airflow updates, which are included in the Astro Runtime, once per day at midnight, by querying `https://updates.astronomer.io/astronomer-runtime`. This returns a JSON file with details about the latest available Astro Runtime versions.

In an egress-controlled or air-gapped environment, you need to store the JSON file in the cluster itself, avoiding the external check. To store the JSON file in the cluster, complete the following steps:

1. Download the JSON files and store them in a Kubernetes configmap by running the following commands:

```bash wrap theme={null}
curl -XGET https://updates.astronomer.io/astronomer-runtime -o astro_runtime_releases.json

kubectl -n <astronomer platform namespace> create configmap astro-runtime-base-images --from-file=astro_runtime_releases.json
```

2. Add your configmap name, `astro-runtime-base-images` to your APC API configuration using the `runtimeReleasesConfigMapName` configuration:

```yaml wrap theme={null}
astronomer:
  houston:
    runtimeReleasesConfigMapName: astro-runtime-base-images
    config:
      airgapped:
        enabled: true
```

<a />

## Step 19: (OpenShift only) Apply OpenShift-specific configuration

<Check>
  If you're not installing APC into an OpenShift Kubernetes cluster, skip this step.
</Check>

Add the following values into `values.yaml`. You can do this manually or by using a YAML merge tool of your choosing.

```yaml wrap theme={null}
global:
  openshift:
    enabled: true
  scc:
    enabled: false
  extraAnnotations:
    kubernetes.io/ingress.class: openshift-default
    route.openshift.io/termination: "edge"
  authSidecar:
    enabled: true
  deployMechanisms:
    dagOnlyDeployment:
      securityContext:
        fsGroup: ""
  nodeExporter:
    enabled: false
  daemonsetLogging:
    enabled: false
  logging:
    loggingSidecar:
      enabled: true
      name: sidecar-log-consumer
elasticsearch:
  sysctlInitContainer:
    enabled: false

# bundled postgresql not a supported option, only for use in proof-of-concepts
postgresql:
  securityContext:
    enabled: false
  volumePermissions:
    enabled: false
```

<Info>
  Only Ingress objects with the annotation `route.openshift.io/termination: "edge"` are supported for generating routes in OpenShift 4.11 and later.
  Other termination types are no longer supported for automatic route generation.

  If you're on an older version of OpenShift, route creation should be done manually.
</Info>

APC on OpenShift is only supported when using [a third-party ingress-controller](#configure-third-party-ingress-controller) and using the [logging sidecar](#configure-sidecar-logging) feature of APC. The above configuration enables both of these items.

<a />

## Step 20: (Optional) Limit Astronomer to a namespace pool

By default, APC automatically creates namespaces for each new Airflow Deployment.

You can restrict the Airflow management components of APC to a list of predefined namespaces and configure it to operate without a ClusterRole by following the instructions in [Configure a Kubernetes namespace pool for APC](/docs/astro-private-cloud/v-2-x/namespace-pools). If you want to disable creation of role and rolebindings for the deployment orchestrator, `config-syncer`, and kubestate metrics, you can set `global.namespaceManagement.namespacePools.createRbac` to `false`.

If `global.rbac.enabled` is `false`, the platform no longer creates any role, rolebindings, or service accounts. The user must define default roles to the k8s default service account to continue with the platform install. See [Bring your own Kubernetes service accounts](/docs/astro-private-cloud/v-2-x/byo-service-accounts) for setup steps.

<a />

## Step 21: (Optional) Enable sidecar logging

Running a logging sidecar to export Airflow task logs is essential for running APC in a multi-tenant cluster.

By default, APC creates a privileged DaemonSet to aggregate logs from Airflow components for viewing from within Airflow and the APC UI.

You can replace this privileged Daemonset with unprivileged logging sidecars by following instructions in [Export logs using container sidecars](/docs/astro-private-cloud/v-2-x/export-task-logs#export-logs-using-container-sidecars).

<a />

## Step 22: (Optional) Integrate an external identity provider

APC includes integrations for several of the most popular OAUTH2 identity providers (IdPs), such as Okta and Microsoft Entra ID. Configuring an external IdP allows you to automatically provision and manage users in accordance with your organization's security requirements. See [Integrate an auth system](/docs/astro-private-cloud/v-2-x/integrate-auth-system) to configure the identity provider of your choice in your `values.yaml` file.

<a />

## Step 23: Install APC using Helm

Deploy the control plane using the `upgrade.sh` script you created earlier. Confirm `RELEASE_NAME`, `NAMESPACE`, and `CHART_VERSION` reflect your environment, then execute:

```bash wrap theme={null}
./upgrade.sh
```

To review manifests before applying them, run `./upgrade.sh --dry-run` or use `helm template` with the same flags defined in the script.

<a />

## Step 24: Configure DNS for the integrated ingress controller

Whether you use Astronomer's integrated ingress controller or a third-party controller, publish the same set of DNS records so users can reach control plane services.

* If you use the integrated controller, get the load balancer address directly:

  ```bash wrap theme={null}
  kubectl -n <astronomer platform namespace> get svc astronomer-nginx
  ```

* If you use a third-party controller, ask your ingress administrator for the hostname or IP address that should front the Astronomer routes (refer back to [Configure a third-party ingress controller](#configure-third-party-ingress-controller)).

Create either a wildcard record such as `*.sandbox-astro.example.com` or individual CNAME records for the following hostnames so that traffic routes through the chosen load balancer:

* `app.<base-domain>` (required)
* `deployments.<base-domain>` (required for Airflow UIs and APIs)
* `houston.<base-domain>` (required)
* `prometheus.<base-domain>` (required)
* `registry.<base-domain>` (required if you keep the integrated container registry enabled)
* `alertmanager.<base-domain>` (required if you keep the integrated Alertmanager enabled)
* `<base-domain>` (optional but recommended, provides a vanity redirect to `app.<base-domain>`)

Astronomer generally recommends pointing the zone apex (`@`) directly to the load balancer address and mapping the remaining hostnames as CNAMEs to that apex. In lower environments, you can safely use a low TTL (for example 60 seconds) to speed up troubleshooting during the initial rollout.

After your DNS provider propagates the records, verify them with tools like `dig <hostname>` or `getent hosts <hostname>`. You can complete this DNS work after verifying the platform pods—Astronomer services stay healthy without external DNS, but end users need these records to sign in.

<a />

## Step 25: Verify you can access the UI

Visit `https://app.<base-domain>` in your web-browser to view APC's web interface. If any components aren't ready, consult the [debugging guide](/docs/astro-private-cloud/v-2-x/debug-install) or contact [Astronomer support](https://support.astronomer.io) with the relevant logs and events.

Congratulations, you have configured and installed an APC platform instance — your new Airflow control plane.

From the UI, you can invite and manage users and create and monitor Airflow Deployments on the platform.

<a />

## Step 26: Disable anonymous account creation

Leave `astronomer.houston.config.publicSignups: true` only until you create your first administrator. Afterwards, secure the platform using the following steps:

1. If you keep public sign-ups enabled, turn on outbound email (`astronomer.houston.config.email.enabled: true`), specify a trusted domain list under `astronomer.houston.config.allowedSystemLevelDomains`, and verify that users can only join through an approved identity provider.
2. Otherwise, set `astronomer.houston.config.publicSignups: false` so new accounts require an invitation.
3. Apply the updated configuration with `helm upgrade` targeting the control plane release.

## Additional customization

The following topics include optional information about one or multiple topics in the installation guide:

* [Configure a private Certificate Authority](/docs/astro-private-cloud/v-2-x/configure-private-cas)
* [Disable outbound emails](/docs/astro-private-cloud/v-2-x/disable-outbound-email)
* Add trusted CAs to [Docker Desktop](/docs/astro-private-cloud/v-2-x/configure-desktop-container-solution-extra-cas)

## Next steps

<a />

### Register the data plane with the control plane

Start adding users, workspaces, and deployments in your newly installed or upgraded APC environment at `https://app.<base-domain>`.


# Run the KubernetesPodOperator on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/kube-pod-operator

Run the KubernetesPodOperator on Astro Private Cloud.

The [`KubernetesPodOperator`](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html) is one of the most customizable Apache Airflow operators. A task using the `KubernetesPodOperator` runs in a dedicated, isolated Kubernetes Pod that terminates after the task completes. To learn more about the benefits and usage of the `KubernetesPodOperator`, see the [`KubernetesPodOperator` Learn guide](/docs/learn/kubepod-operator).

This guide explains how to complete specific goals using the `KubernetesPodOperator` on Astro Private Cloud.

## Prerequisites

* A running Airflow Deployment on Astro Private Cloud

## Set up the `KubernetesPodOperator`

### Install the operator

Run the following command to install the `apache-airflow-providers-cncf-kubernetes` package:

```bash wrap theme={null}
pip install apache-airflow-providers-cncf-kubernetes
```

### Specify parameters

Instantiate the operator based on your image and setup:

```python wrap theme={null}
from airflow.configuration import conf
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator

namespace = conf.get("kubernetes", "NAMESPACE")

KubernetesPodOperator(
    namespace=namespace,
    image="ubuntu:24.04",
    cmds=["bash", "-cx"],
    arguments=["echo", "10", "echo pwd"],
    labels={"<pod-label>": "<label-name>"},
    name="airflow-test-pod",
    is_delete_operator_pod=True,
    in_cluster=True,
    task_id="task-two",
    get_logs=True,
)
```

For each instantiation of the `KubernetesPodOperator`, you must specify the following values:

* `namespace = conf.get("kubernetes", "NAMESPACE")`: Every Deployment runs on its own Kubernetes namespace. Information about this namespace can be programmatically imported as long as you set this variable.
* `image`: This is the Docker image that the operator will use to run its defined task, commands, and arguments. The value you specify is assumed to be an image tag that's publicly available on a public registry. To pull an image from a private registry, read [Pull images from a Private Registry](#pull-images-from-a-private-registry).
* `in_cluster=True`: When this value is set, your task will run within the cluster from which it's instantiated. This ensures that the Kubernetes Pod running your task has the correct permissions within the cluster.
* `is_delete_operator_pod=True`: This setting ensures that once a `KubernetesPodOperator` task is complete, the Kubernetes Pod that ran that task is terminated. This ensures that there are no unused pods in your cluster taking up resources.

#### Add resources to your Deployment on Astro Private Cloud

The `KubernetesPodOperator` is entirely powered by the resources allocated to the `Extra Capacity` slider of your deployment's `Configure` page in the [Software UI](/docs/astro-private-cloud/v-2-x/manage-workspaces) in lieu of needing a Celery worker (or scheduler resources for those running the Local Executor). Raising the slider will increase your namespace's [resource quota](https://kubernetes.io/docs/concepts/policy/resource-quotas/) such that Airflow has permissions to successfully launch pods within your deployment's namespace.

> **Note**: Your Airflow scheduler and webserver will remain necessary fixed resources that ensure the rest of your tasks can execute and that your deployment stays up and running.

In terms of resource allocation, Astronomer recommends starting with 1 CPU and 3840 Mi memory in `Extra Capacity` and scaling up from there as needed. If it's set too low, you may get a permissions error:

```text wrap theme={null}
ERROR - Exception when attempting to create namespace Pod.
Reason: Forbidden
"Failure","message":"pods is forbidden: User \"system:serviceaccount:astronomer-cloud-solar-orbit-4143:solar-orbit-4143-airflow-worker\" can't create pods in the namespace \"datarouter\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
```

On Astro Private Cloud, the largest node a single pod can occupy is dependent on the size of your underlying node pool.

> **Note**: If you need to increase your [limit range](https://kubernetes.io/docs/concepts/policy/limit-range/) on Astro Private Cloud, contact your system admin.

#### Define resources per task

A notable advantage of leveraging Airflow's `KubernetesPodOperator` is that you can control compute resources in the task definition.

> **Note**: If you're using the Kubernetes Executor, note that this value is separate from the `executor_config` parameter. In this case, the `executor_config` would only define the Airflow worker that is launching your Kubernetes task.

### Example task definition

```python expandable wrap theme={null}
from datetime import datetime, timedelta

from airflow import DAG
from airflow.configuration import conf
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from kubernetes.client import models as k8s

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2019, 1, 1),
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

namespace = conf.get('kubernetes', 'NAMESPACE')

# This will detect the default namespace locally and read the
# environment namespace when deployed to Astronomer.
if namespace == 'default':
    config_file = '/usr/local/airflow/include/.kube/config'
    in_cluster = False
else:
    in_cluster = True
    config_file = None

dag = DAG("example_kubernetes_pod", schedule="@once", default_args=default_args)

# This is where you define your resource allocation.
compute_resources = k8s.V1ResourceRequirements(
    limits={"cpu": "800m", "memory": "3Gi"},
    requests={"cpu": "800m", "memory": "3Gi"}
)

with dag:
    KubernetesPodOperator(
        namespace=namespace,
        image="hello-world",
        labels={"foo": "bar"},
        name="airflow-test-pod",
        task_id="task-one",
        in_cluster=in_cluster,  # if set to true, will look in the cluster, if false, looks for file
        cluster_context="docker-for-desktop",  # is ignored when in_cluster is set to True
        config_file=config_file,
        container_resources=compute_resources,
        is_delete_operator_pod=True,
        get_logs=True,
    )
```

In the example above, the resources are defined by building the following `V1ResourceRequirements` object:

```python wrap theme={null}
from kubernetes.client import models as k8s

compute_resources = k8s.V1ResourceRequirements(
    limits={"cpu": "800m", "memory": "3Gi"},
    requests={"cpu": "800m", "memory": "3Gi"}
)
```

This object allows you to specify Memory and CPU requests and limits for any given task and its corresponding Kubernetes Pod. For more information, read [Kubernetes Documentation on Requests and Limits](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits).

When this Dag runs, it will launch a Pod that runs the `hello-world` image, which is pulled from Docker Hub, in your Airflow Deployment's namespace with the resource requests defined above. Once the task finishes, the Pod will terminate gracefully.

## Pull images from a private registry

By default, the `KubernetesPodOperator` will look for images hosted publicly on [Docker Hub](https://hub.docker.com/). If you want to pull images from a private registry, you may do so.

To pull images from a private registry on Astro Private Cloud:

1. Retrieve a `config.json` file that contains your Docker credentials by following the [Docker documentation](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials). The generated file should look something like this:

   ```json wrap theme={null}
   {
       "auths": {
           "https://index.docker.io/v1/": {
               "auth": "c3R...zE2"
           }
       }
   }
   ```

2. Follow the [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials) to create a secret based on your credentials.

3. In your Dag code, import `models` from `kubernetes.client` and specify `image_pull_secrets` with your Kubernetes secret. After configuring this value, you can pull an image as you would from a public registry like in the following example.

   ```python {1,5} wrap theme={null}
   from kubernetes.client import models as k8s

   KubernetesPodOperator(
       namespace=namespace,
       image_pull_secrets=[k8s.V1LocalObjectReference("<your-secret-name>")],
       image="<your-docker-image>",
       cmds=["<commands-for-image>"],
       arguments=["<arguments-for-image>"],
       labels={"<pod-label>": "<label-name>"},
       name="<pod-name>",
       is_delete_operator_pod=True,
       in_cluster=True,
       task_id="<task-name>",
       get_logs=True,
   )
   ```

## Local testing

Astronomer recommends testing your Dags locally before pushing them to a Deployment on Astro Private Cloud. For more information, read [How to run the `KubernetesPodOperator` locally](/docs/learn/kubepod-operator). That guide provides information on how to use [MicroK8s](https://microk8s.io/) or [Docker for Kubernetes](https://matthewpalmer.net/kubernetes-app-developer/articles/how-to-run-local-kubernetes-docker-for-mac.html) to run tasks with the `KubernetesPodOperator` in a local environment.

<Note>To pull images from a private registry locally, you'll have to create a secret in your local namespace and similarly call it in your operator following the guidelines above.</Note>


# Run the Kubernetes executor on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/kubernetes-executor

Run and configure the Kubernetes executor on Astro Private Cloud.

The [Kubernetes Executor](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/kubernetes_executor.html) creates individual Pods that dynamically delegate work and resources to individual tasks. For each task that needs to run, the executor works with the Kubernetes API and dynamically launches Pods which terminate when the task is completed.

You can customize your Kubernetes Pods to scale depending on how many Airflow tasks you're running at a given time. It also means you can configure the following for each individual Airflow task:

* Memory allocation
* Service accounts
* Airflow image

To configure these resources for a given task's Pod, you specify a `pod_override` in your Dag code. To specify a Pod template for many or all of your tasks, you can write a helper function to construct a `pod_override` in your Dags or configure a global setting. For more information on configuring Pod template values, reference the [Kubernetes documentation](https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates).

## Prerequisites

You must have an Airflow Deployment on Astro Private Cloud running with the Kubernetes executor. For more information on configuring an executor, see [Configure a Deployment](/docs/astro-private-cloud/v-2-x/configure-deployment). To learn more about different executor types, see [Airflow executors explained](/docs/learn/airflow-executors-explained).

You can disable automatic setting of worker CPU/memory resources from the API or UI using the `workers.resources.enabled` option for the KubernetesExecutor.

For complete details, including YAML examples and guidance for both default and custom configurations, see [Manage Kubernetes worker CPU and memory outside UI/API](/docs/astro-private-cloud/v-2-x/configure-component-size-limits#disable-api/ui-resource-configuration).

## Configure the default worker Pod for all Deployments

By default, the Kubernetes executor launches workers based on a `podTemplate` configuration in the [Astronomer Airflow Helm chart](https://github.com/astronomer/airflow-chart/blob/master/values.yaml).

You can modify the default `podTemplate` to configure the default worker Pods for all Deployments using the Kubernetes executor on your Astro Private Cloud installation. You can then override this default at the task level using a `pod_override` file. See [Configure the worker Pod for a specific task](#configure-the-worker-pod-for-a-specific-task).

1. In your `values.yaml` file, copy the complete `podTemplate` configuration from your version of the [Astronomer Airflow Helm chart](https://github.com/astronomer/airflow-chart/blob/master/values.yaml). Your file should look like the following:

   ```yaml expandable wrap theme={null}
   astronomer:
     houston:
       config:
         deployments:
           helm:
             airflow:
               podTemplate: |
                 # Licensed to the Apache Software Foundation (ASF) under one
                 # or more contributor license agreements.  See the NOTICE file
                 # distributed with this work for additional information
                 # regarding copyright ownership.  The ASF licenses this file
                 # to you under the Apache License, Version 2.0 (the
                 # "License"); you may not use this file except in compliance
                 # with the License.  You may obtain a copy of the License at
                 #
                 #   http://www.apache.org/licenses/LICENSE-2.0
                 #
                 # Unless required by applicable law or agreed to in writing,
                 # software distributed under the License is distributed on an
                 # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
                 # KIND, either express or implied.  See the License for the
                 # specific language governing permissions and limitations
                 # under the License.
                 ---
                 {{- $nodeSelector := or .Values.nodeSelector .Values.workers.nodeSelector }}
                 {{- $affinity := or .Values.affinity .Values.workers.affinity }}
                 {{- $tolerations := or .Values.tolerations .Values.workers.tolerations }}
                 {{- $securityContext := include "airflowPodSecurityContext" (list . .Values.workers) }}
                 {{- $containerSecurityContext := include "containerSecurityContext" (list . .Values.workers) }}
                 apiVersion: v1
                 kind: Pod
                 metadata:
                   name: astronomer-pod-template-file
                   labels:
                     tier: airflow
                     component: worker
                     release: {{ .Release.Name }}
                 {{- if or (.Values.labels) (.Values.workers.labels) }}
                   {{- mustMerge .Values.workers.labels .Values.labels | toYaml | nindent 4 }}
                 {{- end }}
                   {{- if or .Values.airflowPodAnnotations .Values.workers.podAnnotations }}
                   annotations:
                     {{- if .Values.airflowPodAnnotations }}
                       {{- toYaml .Values.airflowPodAnnotations | nindent 4 }}
                     {{- end }}
                     {{- if .Values.workers.podAnnotations }}
                       {{- toYaml .Values.workers.podAnnotations | nindent 4 }}
                     {{- end }}
                   {{- end }}
                 spec:
                   {{- if or (and .Values.dags.gitSync.enabled (not .Values.dags.persistence.enabled)) .Values.workers.extraInitContainers }}
                   initContainers:
                     {{- if and .Values.dags.gitSync.enabled (not .Values.dags.persistence.enabled) }}
                     {{- include "git_sync_container" (dict "Values" .Values "is_init" "true") | nindent 4 }}
                     {{- end }}
                     {{- if .Values.workers.extraInitContainers }}
                     {{- toYaml .Values.workers.extraInitContainers | nindent 4 }}
                     {{- end }}
                   {{- end }}
                   containers:
                     - args: []
                       command: []
                       envFrom:
                       {{- include "custom_airflow_environment_from" . | default "\n  []" | indent 6 }}
                       env:
                         - name: AIRFLOW__CORE__EXECUTOR
                           value: LocalExecutor
                 {{- include "standard_airflow_environment" . | indent 6}}
                 {{- include "custom_airflow_environment" . | indent 6 }}
                 {{- include "container_extra_envs" (list . .Values.workers.env) | indent 6 }}
                       image: {{ template "pod_template_image" . }}
                       imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}
                       securityContext: {{ $containerSecurityContext | nindent 8 }}
                       name: base
                       ports: []
                       resources: {{- toYaml .Values.workers.resources | nindent 8 }}
                       volumeMounts:
                         - mountPath: {{ template "airflow_logs" . }}
                           name: logs
                         - name: config
                           mountPath: {{ template "airflow_config_path" . }}
                           subPath: airflow.cfg
                           readOnly: true
                 {{- if .Values.airflowLocalSettings }}
                         - name: config
                           mountPath: {{ template "airflow_local_setting_path" . }}
                           subPath: airflow_local_settings.py
                           readOnly: true
                 {{- end }}
                 {{- if or .Values.dags.gitSync.enabled .Values.dags.persistence.enabled }}
                         {{- include "airflow_dags_mount" . | nindent 8 }}
                 {{- end }}
                 {{- if .Values.workers.extraVolumeMounts }}
                 {{ toYaml .Values.workers.extraVolumeMounts | indent 8 }}
                 {{- end }}
                 {{- if .Values.workers.extraContainers }}
                 {{- toYaml .Values.workers.extraContainers | nindent 4 }}
                 {{- end }}
                   hostNetwork: false
                   {{- if .Values.workers.priorityClassName }}
                   priorityClassName: {{ .Values.workers.priorityClassName }}
                   {{- end }}
                   {{- if .Values.workers.runtimeClassName }}
                   runtimeClassName: {{ .Values.workers.runtimeClassName }}
                   {{- end }}
                   {{- if .Values.workers.hostAliases }}
                   hostAliases: {{- toYaml .Values.workers.hostAliases | nindent 4 }}
                   {{- end }}
                   {{- if or .Values.registry.secretName .Values.registry.connection }}
                   imagePullSecrets:
                     - name: {{ template "registry_secret" . }}
                   {{- end }}
                   restartPolicy: Never
                   nodeSelector: {{ toYaml $nodeSelector | nindent 4 }}
                   securityContext: {{ $securityContext | nindent 4 }}
                   affinity: {{ toYaml $affinity | nindent 4 }}
                   tolerations: {{ toYaml $tolerations | nindent 4 }}
                   serviceAccountName: {{ include "worker.serviceAccountName" . }}
                   volumes:
                   {{- if .Values.dags.persistence.enabled }}
                   - name: dags
                     persistentVolumeClaim:
                       claimName: {{ template "airflow_dags_volume_claim" . }}
                   {{- else if .Values.dags.gitSync.enabled }}
                   - name: dags
                     emptyDir: {}
                   {{- end }}
                   {{- if .Values.logs.persistence.enabled }}
                   - name: logs
                     persistentVolumeClaim:
                       claimName: {{ template "airflow_logs_volume_claim" . }}
                   {{- else }}
                   - emptyDir: {}
                     name: logs
                   {{- end }}
                   {{- if and .Values.dags.gitSync.enabled .Values.dags.gitSync.sshKeySecret }}
                   {{- include "git_sync_ssh_key_volume" . | nindent 2 }}
                   {{- end }}
                   - configMap:
                       name: {{ include "airflow_config" . }}
                     name: config
                   {{- if .Values.workers.extraVolumes }}
                   {{ toYaml .Values.workers.extraVolumes | nindent 2 }}
                   {{- end }}
   ```

2. Customize the pod template configuration based on your use case, such as by requesting default limits on CPU and memory usage. To configure these resources for each Pod, you configure a Pod template. For more information on configuring Pod template values, see the [Kubernetes documentation](https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates).

3. Push the configuration change to your platform. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

## Configure the worker Pod for a specific task

For each task with the Kubernetes executor, you can customize its individual worker Pod and override the defaults used in Astro Private Cloud by configuring a `pod_override` file.

1. Add the following import to your Dag file:

   ```python wrap theme={null}
   from kubernetes.client import models as k8s
   ```

2. Add a `pod_override` configuration to the Dag file containing the task. See the [`kubernetes-client`](https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Container.md) for a list of all possible settings you can include in the configuration.

3. Specify the `pod_override` in the task's parameters.

### Example: Set CPU or memory limits and requests

One of the most common use cases for customizing a Kubernetes worker Pod is to request a specific amount of resources for a task.

The following example shows how you can use a `pod_override` configuration in your Dag code to request custom resources for a task:

```python expandable wrap theme={null}
import pendulum
import time
from airflow import DAG
from airflow.decorators import task
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
from airflow.example_dags.libs.helper import print_stuff
from kubernetes.client import models as k8s
k8s_exec_config_resource_requirements = {
    "pod_override": k8s.V1Pod(
        spec=k8s.V1PodSpec(
            containers=[
                k8s.V1Container(
                    name="base",
                    resources=k8s.V1ResourceRequirements(
                        requests={"cpu": 0.5, "memory": "1024Mi"},
                        limits={"cpu": 0.5, "memory": "1024Mi"}
                    )
                )
            ]
        )
    )
}
with DAG(
    dag_id="example_kubernetes_executor_pod_override_sources",
    schedule=None,
    start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
    catchup=False
):
    BashOperator(
      task_id="bash_resource_requirements_override_example",
      bash_command="echo hi",
      executor_config=k8s_exec_config_resource_requirements
    )

    @task(executor_config=k8s_exec_config_resource_requirements)
    def resource_requirements_override_example():
        print_stuff()
        time.sleep(60)

    resource_requirements_override_example()
```

When this Dag runs, it launches a Kubernetes Pod with exactly 0.5m of CPU and 1024Mi of memory, as long as that infrastructure is available in your cluster. Once the task finishes, the Pod terminates gracefully.

<Tip>You can also configure the default `request` and `limit` values for CPU and memory for KubernetesExecutor Pods in the Astro Private Cloud UI. See more about [configuring custom resources in the UI](/docs/astro-private-cloud/v-2-x/customize-resource-usage#set-cpu-and-memory-resources-in-the-astro-private-cloud-ui).</Tip>

### Manage default ephemeral storage configurations

You can use the `ephemeralStorage.disabled` configuration to define whether your task Pods assign ephemeral storage amounts from either your default configurations or your `pod_override` configuration.

The following APC API configuration `ephemeralStorage.disabled` is set to `true` by default. When set to `true`, Astro Private Cloud doesn't configure any ephemeral storage. However, you can set ephemeral storage at the Dag level by using Pod Override.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        executors:
          - name: KubernetesExecutor
            workers:
              ephemeralStorage:
                disabled: true
```

You can configure ephemeral storage for all Deployments using the resource configurations set in your APC API configmap by setting `ephemeralstorage.disabled` to `false`.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        executors:
          - name: KubernetesExecutor
            workers:
              ephemeralStorage:
                disabled: false
```

## Mount secret environment variables to worker Pods

[Deployment environment variables](/docs/astro-private-cloud/v-2-x/environment-variables) marked as secrets are stored in a Kubernetes secret called `<release-name>-env` on your Deployment namespace. To use a secret value in a task running on the KubernetesExecutor, mount the secret to the Pod running the task.

1. Run the following command to find the namespace (release name) of your Airflow Deployment:

   ```sh wrap theme={null}
   kubectl get ns
   ```

2. Add the following import to your Dag file:

   ```python wrap theme={null}
   from airflow.kubernetes.secret import Secret
   ```

3. Define a Kubernetes `Secret` in your Dag instantiation using the following format:

   ```python wrap theme={null}
   secret_env = Secret(deploy_type="env", deploy_target="<SECRET_KEY>", secret="<release-name>-env", key="<SECRET_KEY>")
   namespace = conf.get("kubernetes", "<release-name>")
   ```

4. Specify the `Secret` in the `secret_key_ref` section of your `pod_override` configuration.

5. In the task where you want to use the secret value, add the following task-level argument:

   ```python wrap theme={null}
   op_kwargs={
           "env_name": secret_env.deploy_target
   },
   ```

6. In the executable for the task, call the secret value using `os.environ[env_name]`.

In the following example, a secret named `MY_SECRET` is pulled from `infrared-photon-7780-env` and printed to logs.

```python expandable wrap theme={null}
import pendulum
from kubernetes.client import models as k8s

from airflow.configuration import conf
from airflow.kubernetes.secret import Secret
from airflow.models import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from airflow.operators.python import PythonOperator

def print_env(env_name):
    import os
    print(os.environ[env_name])

with DAG(
        dag_id='test-secret',
        start_date=pendulum.datetime(2022, 1, 1, tz="UTC"),
        end_date=pendulum.datetime(2022, 1, 5, tz="UTC"),
        schedule_interval="@once",
) as dag:
    secret_env = Secret(deploy_type="env", deploy_target="MY_SECRET", secret="infrared-photon-7780-env", key="MY_SECRET")
    namespace = conf.get("kubernetes", "infrared-photon-7780")

    p = PythonOperator(
        python_callable=print_env,
        op_kwargs={
            "env_name": secret_env.deploy_target
        },
        task_id='test-py-env',
        executor_config={
            "pod_override": k8s.V1Pod(
                spec=k8s.V1PodSpec(
                    containers=[
                        k8s.V1Container(
                            name="base",
                            env=[
                                k8s.V1EnvVar(
                                    name=secret_env.deploy_target,
                                    value_from=k8s.V1EnvVarSource(
                                        secret_key_ref=k8s.V1SecretKeySelector(name=secret_env.secret,
                                                                               key=secret_env.key)
                                    ),
                                )
                            ],
                        )
                    ]
                )
            ),
        }
    )
```


# Log in to Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/log-in-to-private-cloud

Log in to Astro Private Cloud to access features and functionality.

You can use the Astro Private Cloud (APC) UI and the Astro CLI to view and modify your data planes, Workspaces, Deployments, environment variables, tasks, and users. You need to authenticate your user credentials when you're using the UI or the Astro CLI for development on Astro.

## Prerequisites

* An Astronomer account.
* [Astro CLI](/docs/cli/v1.43/overview) v1.37 or higher .

## Sign in to the Astro Private Cloud UI

1. Go to `app.<basedomain>`.

2. Sign in to the Astro Private Cloud UI using one of the authentication methods that has been configured by your organization. To integrate an identity provider (IdP) with Astro Private Cloud, see [Integrate an auth system](/docs/astro-private-cloud/v-2-x/integrate-auth-system).

## Sign in to the Astro CLI

Developing locally with the Astro CLI doesn't require an Astro account. This includes commands such as `astro dev start` and `astro dev pytest`. If you want to use functionality specific to Astro Private Cloud, including managing users and [deploying code](/docs/astro-private-cloud/v-2-x/deploy-code-overview), you must first sign in to Astro with the Astro CLI.

1. In the Astro CLI, run the following command:

   ```sh wrap theme={null}
   astro login <basedomain>
   ```

2. Enter your username and password or use an OAuth token for authentication:

   * Press **Enter**.
   * Copy the URL in the command prompt, open a browser, paste the URL in the address bar, and then press **Enter**. If you're not taken immediately to the Astronomer Auth Token page, sign in to Astro Private Cloud, paste the URL in the address bar, and press **Enter**.
   * Copy the OAuth token, paste it in the command prompt after **OAuth Token**, and then press **Enter**.

<Info>If you can't enter your password in the command prompt, your organization is using an alternative authentication method. Contact your administrator, or use an OAuth token for authentication.</Info>

## Access a different base domain

When you need to access multiple installations of Astro Private Cloud with the Astro CLI at the same time or you need to use Astro and Astro Private Cloud at the same time, you need to authenticate to each cluster individually by specifying its base domain.

A base domain or URL is the static element of a website address. For example, when you visit the Astronomer website, the address bar always displays `astronomer.io` no matter what page you access on the Astronomer website.

For Astro Private Cloud, every cluster has a base domain that you must authenticate to in order to access it. If your organization has multiple clusters, you can run Astro CLI commands to quickly move from one base domain to another. This can be useful when you need to move from an Astro Private Cloud installation to Astro and are using the Astro CLI to perform actions on both accounts.

1. Run the following command to view a list of base domains for all APC installations that you can access and to confirm your default base domain:

   ```text wrap theme={null}
   astro context list
   ```

2. In the Astro CLI, run the following command to re-authenticate to the target base domain:

   ```text wrap theme={null}
   astro login
   ```

3. Run the following command to switch to a different base domain:

   ```text wrap theme={null}
   astro context switch <basedomain>
   ```

   For example, if the base domain you wanted to switch to was `astronomer.io`, you would run:

   ```text wrap theme={null}
   astro context switch astronomer.io
   ```


# Merge YAML configurations
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/merge-yaml

Merge Astro Private Cloud configurations with merge_yaml.py

When merging YAML configurations into `values.yaml`, you can merge manually or with a tool of your choosing.

You can use the following `merge_yaml.py` script to merge YAML excerpts into `values.yaml` automatically. This script requires both Python and the `ruamel.yaml` package, which you can install using `pip install ruamel.yaml`.
To run the program, ensure that `merge_yaml.py`, `values.yaml`, and the `yaml` file that contains the configuration you want to add are all in your project directory. Then, run:

```bash wrap theme={null}
python merge_yaml.py values-to-merge.yaml values.yaml
```

```python expandable wrap theme={null}
#!/usr/bin/env python
"""
Backup destination file and merge YAML contents of src into dest.

By default creates backups, overwrites destination, and clobbers lists.

Usage:
    merge_yaml.py src dest [--create-backup=True] [--dry-run] [--show-stacktrace=False] [--merge-lists=True] [--help]
"""

import argparse
import os
import shutil
from datetime import datetime
import sys
from pathlib import Path

# Check Python version
if sys.version_info < (3, 0):
    print("Error: This script requires Python 3.0 or greater.")
    sys.exit(2)

# Try importing ruamel.yaml
try:
    from ruamel.yaml import YAML
except ImportError:
    print(
        "Error: ruamel.yaml is not installed. Please install it using 'pip install ruamel.yaml'"
    )
    sys.exit(2)

yaml = YAML()

def deep_merge(d1, d2, **kwargs):
    """Deep merges dictionary d2 into dictionary d1."""
    merge_lists = kwargs.get("merge_lists")
    for key, value in d2.items():
        if key in d1:
            if isinstance(d1[key], dict) and isinstance(value, dict):
                deep_merge(d1[key], value, **kwargs)
            elif merge_lists and isinstance(d1[key], list) and isinstance(value, list):
                d1[key].extend(value)
            else:
                d1[key] = value
        else:
            d1[key] = value
    return d1

def load_yaml_file(filename):
    """Load YAML data from a file."""
    if not os.path.exists(filename):
        return {}
    with open(filename, "r") as file:
        return yaml.load(file)

def save_yaml_file(filename, data):
    """Save YAML data to a file."""
    with open(filename, "w") as file:
        yaml.dump(data, file)

def create_backup(filename):
    """Create a timestamped backup of the file."""
    # create a directory called backups relative to the filename
    backup_dir = filename.parent / "yaml_backups"
    try:
        backup_dir.mkdir(exist_ok=True)
    except Exception as e:
        print(
            f"Error: Could not create backup directory {backup_dir}. Check your file-permissions or use --no-create-backup to skip creating a backup."
        )
        exit(2)

    timestamp = datetime.now().strftime("%y%m%d%H%M%S")
    backup_filename = backup_dir / f"{filename.name}.{timestamp}.bak"
    shutil.copyfile(filename, backup_filename)
    print(f"Backup created: {backup_filename}")

def main():
    parser = argparse.ArgumentParser(
        description="Deep merge YAML contents of src into dest."
    )
    parser.add_argument("src", type=Path, help="Source filename")
    parser.add_argument("dest", type=Path, help="Destination filename")
    parser.add_argument(
        "--create-backup",
        type=bool,
        default=True,
        help="Create a backup of the destination file before merging",
    )
    parser.add_argument(
        "--dry-run",
        action="store_true",
        help="Print to stdout only, do not write to the destination file",
    )
    # add a argument for showing the stack trace on yaml parse errors
    parser.add_argument(
        "--show-stacktrace",
        action="store_true",
        help="Show stack trace on yaml parse errors",
    )
    # add an argument to clobber lists
    parser.add_argument(
        "--merge-lists",
        action="store_true",
        help="Merge list items instead of clobbering",
        default=False,
    )

    args = parser.parse_args()

    src_filename = args.src.resolve().expanduser()
    dest_filename = args.dest.resolve().expanduser()

    # make sure both files exist
    if not src_filename.exists():
        print(f"Error: {args.src} does not exist")
        exit(2)

    if not dest_filename.exists():
        print(f"Error: {args.dest} does not exist")
        exit(2)

    try:
        src_data = load_yaml_file(src_filename)
    except Exception as e:
        print(
            f"Error: {args.src} is not a valid YAML file. Run with --show-stacktrace to see the error."
        )
        if args.show_stacktrace:
            raise e
        exit(2)
    try:
        dest_data = load_yaml_file(dest_filename)
    except Exception as e:
        print(
            f"Error: {args.dest} is not a valid YAML file. Run with --show-stacktrace to see the error."
        )
        if args.show_stacktrace:
            raise e
        exit(2)

    if args.create_backup and not args.dry_run:
        create_backup(dest_filename)

    src_data = load_yaml_file(args.src)
    dest_data = load_yaml_file(args.dest)

    # if dest_data is empty, just copy src_data to dest_data
    if not dest_data:
        if not args.dry_run:
            save_yaml_file(args.dest, src_data)
    else:
        merged_data = deep_merge(dest_data, src_data, merge_lists=args.merge_lists)
        if not args.dry_run:
            save_yaml_file(args.dest, merged_data)
            print(f"Merged data from {args.src} into {args.dest}")
        else:
            yaml.dump(merged_data, sys.stdout)

if __name__ == "__main__":
    main()

```


# Migrate to Airflow 3 from Airflow 2 in Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/migrate-to-airflow-3

Migrate your Astro Private Cloud Deployments from Airflow 2 to Airflow 3.

This guide describes the steps required to migrate your Airflow deployments from Airflow 2 to Airflow 3 in Astro Private Cloud.

## 1. Enable Airflow 3 on the cluster

1. Navigate to the **Cluster Configuration Override** page in the Astro Private Cloud UI.
2. Enable the **Airflow 3 flag**:

```yaml wrap theme={null}
astronomer:
    houston:
        deployments:
            runtimeManagement:
                airflowV3:
                    enabled: true
```

3. Enable the mandatory Dag processor component for Airflow 3:

```yaml wrap theme={null}
astronomer:
    houston:
        deployments:
            airflowComponents:
                dagProcessor:
                    enabled: true
```

<Warning> Airflow 3 requires the Dag processor to be enabled. Deployments without this component will fail. </Warning>

4. Enable **Sidecar Logging** for Airflow tasks:

```yaml wrap theme={null}
global:
    logging:
        loggingSidecar:
            enabled: true
```

<Warning> Airflow 3 requires sidecar logging. Deployments without this component won't be able to view task logs. </Warning>

## 2. Configure registry access

1. Allowlist the Azure Container Registry (`azurecr`) repository and URL for pulling Airflow 3 images.
2. If using third-party registries, configure mirroring to ensure reliable image pulls.

<Note> Airflow 3 images are hosted at `astrocrpublic.azurecr.io`. </Note>

## 3. Check Dag compatibility before migration

Before upgrading, follow the [Airflow 2 → 3 upgrade guide](/docs/learn/airflow-upgrade-2-3) to verify that your Dags are compatible with Airflow 3.

## 4. Update Deployment Dockerfile

1. Modify your Dockerfile to use the Airflow 3 image and updated registry URL:

```dockerfile wrap theme={null}
FROM astrocrpublic.azurecr.io/runtime:3.1-2
```

2. Build and push your updated Docker image.
   <Note> Airflow 2 images are available on Quay (`quay.io/astronomer/airflow`), while Airflow 3 images are hosted on `astrocr` (`astrocrpublic.azurecr.io`). </Note>

## 5. Update Deployment

Perform the Deployment update using one of the following methods:

1. UI: Navigate to your Deployment and trigger an update with the new Airflow 3 image.
2. API: Use the UpsertDeployment API to update the Deployment image.
3. CLI: Use the Astro CLI to deploy the updated image.

Ensure the updated Deployment reflects the Airflow 3 runtime version.

## 6. Validate after migration

* Confirm that Dags are running as expected in Airflow 3.
* Monitor logs and metrics to ensure stability.
* Verify that the Dag processor component is active and functioning.

<Note>
  This guide provides a **safe and supported path** to migrate from Airflow 2 to Airflow 3 in Astro Private Cloud.
</Note>


# Use a MySQL or PostgreSQL database for metadata or storage
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/multi-db

Manually configure a pre-existing MySQL or PostgreSQL database to use with your Deployments.

You can create Astro Private Cloud Deployments with the [APC API](/docs/astro-private-cloud/v-2-x/houston-api) that use pre-created databases, external to the Airflow Deployment, as both a metadata storage and result storage backend.

## Prerequisites

* Workspace Admin [user privileges](/docs/astro-private-cloud/v-2-x/manage-permissions) and a Workspace ID
* (Optional) A MySQL or PostgreSQL database
* (Optional) An existing Deployment

<Note>If you create a new connection to an external database from a Deployment with existing Dag data, you must migrate that historic data to the new database. Information about your historic Deployment activity, such as task instances and Dag runs, won't be displayed as the database where you stored that information has changed.</Note>

## Step 1: Enable manual connection strings

In Astro Private Cloud 2.x, `databaseManagement.manualConnectionStrings.enabled` is a `deployments.*` setting. Cluster, Workspace, and Deployment overrides take precedence over `values.yaml` for `deployments.*` keys. See [Configure Astro Private Cloud](/docs/astro-private-cloud/v-2-x/configure-astro-private-cloud) for the precedence rules.

Choose one of the following options based on the scope you need:

* To enable manual connection strings for one data plane cluster, update the cluster's **Configuration Override**.
* To enable manual connection strings as the platform default for clusters that don't have their own saved value for this key, update `values.yaml` and run a Helm upgrade.

### Option A: Update the cluster configuration override (recommended)

Cluster overrides apply to `deployments.*` values for the selected data plane cluster, so don't include the `deployments.` prefix in the override.

1. In the Astro UI, open **Clusters**, select your data plane cluster, then click **Edit** on **Configuration Override**. See [Update data plane cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster).

2. Add the following to **Configuration Override**:

   ```yaml wrap theme={null}
   databaseManagement:
     manualConnectionStrings:
       enabled: true
   ```

3. Click **Update cluster** to apply the change. The override is deep-merged with the cluster's existing configuration.

### Option B: Update `values.yaml` (platform default)

Use this option only when no cluster has saved an override for `deployments.databaseManagement.manualConnectionStrings.enabled`. If a cluster already has a saved value for this key, you must update the cluster configuration as in Option A.

1. Open your `values.yaml` file.

2. Add the following under `astronomer.houston.config`:

   ```yaml wrap theme={null}
   astronomer:
     houston:
       config:
         deployments:
           databaseManagement:
             manualConnectionStrings:
               enabled: true
   ```

3. Push the configuration change. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config). If the `astronomer-houston` pods don't roll automatically after the Helm upgrade, restart them manually so they pick up the new configuration.

<Note>
  **Already-installed clusters**

  The `values.yaml` settings in this step only take effect during the initial cluster installation. For clusters that are already registered, Houston resolves Deployment configuration directly from the cluster's database record (`Cluster.config.deployments`) and ignores the Helm-derived ConfigMap. To enable manual connection strings on an existing cluster, apply the change through **System Admin → Clusters → Edit → Cluster Deployment Configuration** in Astronomer instead.
</Note>

## Step 2: (Optional) Create your database

Substitute `astro-db-name` with your own database name, if you need to create a new database.

```sql wrap theme={null}
CREATE DATABASE astro-db-name;
```

## Step 3: Add a user account to your database for the connection

Substitute `astro-user-name` and `astro-user-password` with your information. You can use an existing database for this step.

<Warning>PostgreSQL usernames must be lowercase. </Warning>

<Tabs>
  <Tab title="PostgreSQL">
    1. Create a user with a password for Astro Private Cloud to use to access the database.

    ```sql wrap theme={null}
    CREATE USER astro-user-name WITH PASSWORD 'astro-user-password';
    ```

    2. Grant all privileges on the database to the user.

    ```sql wrap theme={null}
    GRANT ALL PRIVILEGES ON DATABASE postgreSQL_linked_DB TO astro-user-name;
    ```

    3. Grant `USAGE` and `CREATE` privileges on the `public` schema to `astro-user-name`:

    ```sql wrap theme={null}
    GRANT USAGE, CREATE ON SCHEMA public TO astro-user-name;
    ```

    Now, go into the database you created, which is `astro-db-name` in this example, and run the following queries

    4. Grant all privileges on all tables, sequences, and functions to the user.

    ```sql wrap theme={null}
    GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO astro-user-name;
    GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO astro-user-name;
    GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public TO astro-user-name;
    ```

    5. Set default privileges for the user, so any new tables, sequences, or functions automatically have the user's access.

    ```sql wrap theme={null}
    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL PRIVILEGES ON TABLES TO astro-user-name;
    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL PRIVILEGES ON SEQUENCES TO astro-user-name;
    ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL PRIVILEGES ON FUNCTIONS TO astro-user-name;
    GRANT USAGE, CREATE ON SCHEMA public TO astro-user-name;
    ```
  </Tab>

  <Tab title="MySQL">
    1. Create a new user and password.

    ```sql wrap theme={null}
    CREATE USER 'astro-user-name'@'%' IDENTIFIED BY 'astro-user-password';
    ```

    2. Assign privileges.

    ```sql wrap theme={null}
    GRANT ALL PRIVILEGES ON astro-db-name.* TO 'astro-user-name'@'%';
    ```
  </Tab>
</Tabs>

## Step 4: Retrieve database host information

Retrieve the connection information for your external database. For example, with AWS, you can retrieve your endpoint information by [Finding the connection information for an RDS for MySQL DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToInstance.EndpointAndPort.html).

## Step 5: Compose a connection string for your database

You need connection strings that define how Astro Private Cloud configures the connection to your external databases from your Airflow Deployment. The values of these strings are used when you define your `metadataConnection` or `resultBackendConnection` when you create, update, or upsert your Deployment.

Use the values for your `astro-user-name`, `astro-user-password`, `astro-db-name`, and the host information you retrieved to compose the connection strings in the following format, depending on whether you want to define a result backend connection or a metadata database connection.

<Warning>
  **PgBouncer is enabled by default**

  For PostgreSQL Deployments, PgBouncer is enabled by default in Astro Private Cloud. When PgBouncer is enabled, URI-style connection strings (`postgresql://...`) are rejected during upsert and you must use the JSON format (`metadataConnectionJson` and `resultBackendConnectionJson`). Use the URI tabs below only if you have explicitly disabled PgBouncer for your Deployment.
</Warning>

<Tabs>
  <Tab title="PostgreSQL">
    ### With PgBouncer disabled

    * `metadataConnection`:

      ```text wrap theme={null}
      postgresql://astro-user-name:astro-user-password@host:5432/astro-db-name
      ```

    * `resultBackendConnection`:

      ```text wrap theme={null}
      db+postgresql://astro-user-name:astro-user-password@host:5432/astro-db-name
      ```

    <Warning>
      **Celery Executor**

      The connection string format validation regex don't cover the `resultbackend` connection string format, which includes `db+`. This is specifically required for the Celery executor worker. If the connection string doesn't include `db+`, then Celery worker pod fails. The regex validation isn't implemented because it adds the complications on format validation logic in different scenarios.
    </Warning>

    #### With PgBouncer enabled

    If you have PgBouncer enabled, and are using Postgres, you must configure `metadataConnectionJson` and `resultBackendConnectionJson` instead. PgBouncer is enabled by default in Astro Private Cloud, so this is the typical path.

    Use the values for your `astro-user-name`, `astro-user-password`, `astro-db-name`, and the host information you retrieved to compose the connection strings in the following format, depending on whether you want to define a result backend connection or a metadata database connection.

    * `metadataConnectionJson`:

      ```json wrap theme={null}

      "metadataConnectionJson": {
          "user": "astro-user-name",
          "pass": "astro-user-password",
          "protocol": "postgresql",
          "host": "host",
          "port": 5432,
          "db": "astro-db-name"
          },
      ```

    * `resultBackendConnectionJson`:

      ```json wrap theme={null}
      "resultBackendConnectionJson": {
          "user": "astro-user-name",
          "pass": "astro-user-password",
          "protocol": "postgresql",
          "host": "host",
          "port": 5432,
          "db": "astro-db-name"
          },

      ```
  </Tab>

  <Tab title="MySQL">
    * `metadataConnection`:

      ```text wrap theme={null}
      mysql+mysqldb://astro-user-name:astro-user-password@host:3306/astro-db-name
      ```
    * `resultBackendConnection`:

      ```text wrap theme={null}
      db+mysql+mysqldb://astro-user-name:astro-user-password@host:3306/astro-db-name
      ```

    <Warning>
      **Celery Executor**

      The connection string format validation regex don't cover the `resultbackend` connection string format, which includes `db+`. This is specifically required for the Celery executor worker. If the connection string doesn't include `db+`, then Celery worker pod fails. The regex validation isn't implemented because it adds the complications on format validation logic in different scenarios.
    </Warning>
  </Tab>
</Tabs>

## Step 6: Add to Deployment configuration

Use the [APC API](/docs/astro-private-cloud/v-2-x/houston-api) to create your Deployment configuration.

<Warning>
  **Required: skipAirflowDatabaseProvisioning**

  When you point a Deployment at an external database, you must set `skipAirflowDatabaseProvisioning: true` in the `upsertDeployment` mutation. Without this flag, Commander overwrites the `host` value in your `metadataConnection` or `metadataConnectionJson` with the data plane database URL before running `helm install`, regardless of the host you submitted. The Deployment then uses the platform database instead of your external database without reporting an error. Setting `skipAirflowDatabaseProvisioning: true` skips automatic database provisioning and preserves the host you provided.
</Warning>

The following example shows the mutation and queries for using `upsertDeployment`. See [APC API code examples](/docs/astro-private-cloud/v-2-x/houston-api-example-queries) for examples on how to use the `update` and `upsert` options for configuring your Deployment.

### Create a new Deployment

```graphQL expandable wrap theme={null}
mutation upsertDeployment(
  $workspaceUuid: Uuid!
  $releaseName: String
  $namespace: String!
  $label: String!
  $description: String
  $version: String
  $airflowVersion: String
  $runtimeVersion: String
  $executor: ExecutorType
  $workers: Workers
  $webserver: Webserver
  $scheduler: Scheduler
  $triggerer: Triggerer
  $properties: JSON
  $dagDeployment: DagDeployment
  $rollbackEnabled: Boolean
  $metadataConnection: String
  $resultBackendConnection: String
  $metadataConnectionJson: JSON
  $resultBackendConnectionJson: JSON
  $skipAirflowDatabaseProvisioning: Boolean
) {
  upsertDeployment(
    workspaceUuid: $workspaceUuid
    releaseName: $releaseName
    namespace: $namespace
    label: $label
    airflowVersion: $airflowVersion
    description: $description
    version: $version
    executor: $executor
    workers: $workers
    webserver: $webserver
    scheduler: $scheduler
    triggerer: $triggerer
    properties: $properties
    runtimeVersion: $runtimeVersion
    dagDeployment: $dagDeployment
    rollbackEnabled: $rollbackEnabled
    metadataConnection: $metadataConnection
    resultBackendConnection: $resultBackendConnection
    metadataConnectionJson: $metadataConnectionJson
    resultBackendConnectionJson: $resultBackendConnectionJson
    skipAirflowDatabaseProvisioning: $skipAirflowDatabaseProvisioning
  ) {
    id
    config
    urls {
      type
      url
      __typename
    }
    properties
    description
    label
    releaseName
    namespace
    status
    type
    version
    workspace {
      id
      label
      __typename
    }
    airflowVersion
    runtimeVersion
    dagDeployment {
      type
      nfsLocation
      repositoryUrl
      branchName
      syncInterval
      syncTimeout
      ephemeralStorage
      dagDirectoryLocation
      rev
      sshKey
      knownHosts
      __typename
    }
    createdAt
    updatedAt
    __typename
  }
}
```

### JSON query example

```json expandable wrap theme={null}
{
  "workspaceUuid": "cm3g0cjd2000008l74jigb54y",
  "skipAirflowDatabaseProvisioning": true,
 "metadataConnectionJson": {
    "user": "astro-user-name",
    "pass": "astro-password",
    "protocol": "postgresql",
    "host": "host",
    "port": 5432,
    "db": "astro-db-name"
    },
  "resultBackendConnectionJson": {
    "user": "astro-user-name",
    "pass": "astro-password",
    "protocol": "postgresql",
    "host": "postgres-db-lb.external-postgres.svc.cluster.local",
    "port": 5432,
    "db": "astro-db-name"
    },
  "namespace": "",
  "executor": "CeleryExecutor",
  "workers": {},
  "webserver": {},
  "scheduler": {
    "replicas": 1
  },
  "triggerer": {},
  "label": "Rt1160-Celery-Pgbouncer-Enabled-Json-5",
  "description": "",
  "runtimeVersion": "11.6.0",
  "properties": {
    "extra_capacity": {
      "cpu": 1000,
      "memory": 3840
    }
  },
  "rollbackEnabled": true,
  "dagDeployment": {
    "type": "dag_deploy",
    "nfsLocation": "",
    "repositoryUrl": "",
    "branchName": "",
    "syncInterval": 1,
    "syncTimeout": 120,
    "ephemeralStorage": 2,
    "dagDirectoryLocation": "",
    "rev": "",
    "sshKey": "",
    "knownHosts": ""
  }
}
```

### Example query string variables

```json expandable wrap theme={null}
{
  "workspaceUuid": "cm3g0cjd2000008l74jigb54y",
  "skipAirflowDatabaseProvisioning": true,
"metadataConnection": "postgresql://astro-user-name:astro-user-password@host:5432/astro-db-name"
"resultBackendConnection": "db+postgresql://astro-user-name:astro-user-password@host:5432/astro-db-name"
  "namespace": "",
  "executor": "CeleryExecutor",
  "workers": {},
  "webserver": {},
  "scheduler": {
    "replicas": 1
  },
  "triggerer": {},
  "label": "Rt1160-Celery-Pgbouncer-Enabled-Json-5",
  "description": "",
  "runtimeVersion": "11.6.0",
  "properties": {
    "extra_capacity": {
      "cpu": 1000,
      "memory": 3840
    }
  },
  "rollbackEnabled": true,
  "dagDeployment": {
    "type": "dag_deploy",
    "nfsLocation": "",
    "repositoryUrl": "",
    "branchName": "",
    "syncInterval": 1,
    "syncTimeout": 120,
    "ephemeralStorage": 2,
    "dagDirectoryLocation": "",
    "rev": "",
    "sshKey": "",
    "knownHosts": ""
  }
}
```


# Astro Private Cloud unified architecture
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/unified-architecture

Understand how Astro Private Cloud behaves when control plane and data plane components run in a single cluster.

Astro Private Cloud (APC) can run in *unified* mode, where the control plane and data plane components coexist inside the same Kubernetes cluster. Unified installations can bootstrap easily, but sacrifice the isolation and scaling benefits of dedicated planes found in split mode. This document explains which components run when you enable `global.plane.mode: unified`, how the components interact, and when unified mode makes sense.

## Responsibilities in Unified mode

Because control and data responsibilities share one cluster in unified mode, the following characteristics apply:

* **Management and execution share infrastructure**: The APC API platform management service, the Astro UI web interface, the deployment orchestrator, ingress controllers, logging pipelines, and metrics components all run together.
* **Reduced cross-cluster networking**: The deployment orchestrator communicates locally with the APC API, and Prometheus, the central metrics store, ingests and scrapes everything in-cluster without federation.
* **Shared security surface**: Ingress endpoints for the Astro UI and APC API (`app.<base-domain>`, `houston.<base-domain>`, `deployments.<base-domain>`, etc.) and registry/logging endpoints all originate from the same cluster.
* **Simplified operations for sandbox/test**: Backups, upgrades, and monitoring target a single cluster.

Unified mode is ideal for proof-of-concepts, lab environments, or small teams that need APC features without requiring multiple clusters.

## Components enabled in Unified mode

When you enable `global.plane.mode: unified` in the `values.yaml` file, APC enables both the control plane and data plane Helm charts, which includes the following services:

* **Control plane services**: The web interface (Astro UI), the platform management service (APC API), scheduled jobs, the event streaming broker (NATS JetStream), the alert routing service (Alertmanager), control-plane NGINX ingress, registry token jobs, and the central metrics store (Prometheus).
* **Data plane services**: The deployment orchestrator, the secret distribution job (Config Syncer), data plane NGINX ingress, the platform registry (Registry), the log forwarder daemonset (Vector), the log store (Elasticsearch - if enabled), the cluster state exporter (kube-state-metrics), metrics gateway endpoints (Prometheus federation/auth) used locally, and namespace-pool helpers.
* **Shared services**: Optional Postgres/PgBouncer (for the APC API), base Prometheus StatefulSet, Airflow CRDs.

Because everything runs in the same cluster, you must size and secure it appropriately to handle both management workloads and tenant Airflow namespaces.

## Network and DNS footprint

Unified clusters expose a superset of ingress hostnames, commonly:

* `app.<base-domain>`: Astro UI.
* `houston.<base-domain>`: APC API for UI, CLI, deployment orchestrator callbacks.
* `deployments.<base-domain>`: Ingress route for Airflow UIs.
* `registry.<base-domain>`: Platform container registry (Registry) if enabled.
* `alertmanager.<base-domain>` and `prometheus.<base-domain>`: Optional dashboards.
* `<base-domain>` (optional): Redirect to `app.<base-domain>`.

Because both planes run in the same cluster, there are no cross-cluster TLS certificates. This means that a single certificate that includes these names is sufficient.

## Operational implications

| Aspect           | Unified Mode                                                          | Split Mode                                                |
| ---------------- | --------------------------------------------------------------------- | --------------------------------------------------------- |
| Cluster Count    | 1                                                                     | 2+                                                        |
| Affected area    | Management + execution share failure domain                           | Control plane isolated from data plane failures           |
| Scaling          | Cluster must scale to meet both UI and Airflow workload demand        | Each plane scales independently                           |
| Upgrades         | Single maintenance window, but downtime affects both responsibilities | Control plane upgrades isolated from data plane workloads |
| Network Security | Fewer external egress rules                                           | Requires firewall between control plane and data planes   |
| Recommended For  | Test environments, small teams, early POCs                            | Production workloads needing higher isolation             |

## Transition to split mode

Many Organizations start with a unified mode and later migrate to split mode for isolation or scalability. Plan ahead for migrating to split mode by:

* Using external Postgres and registry storage so migrating the control plane doesn't require moving data twice.
* Keeping DNS hostnames consistent with the future split design, for example `app.<base-domain>` and `deployments.<base-domain>`, so you can later point records at different clusters.
* Managing configuration through values files or automation so you can reproduce control plane settings in a dedicated cluster.


# Upgrade to Astro Private Cloud 2.0 from 0.37
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/upgrade-037-to-2

Upgrade to Astro Private Cloud 2.0 from 0.37.4, including platform migration steps and values.yaml migration.

This guide provides instructions to upgrade directly from Astro Private Cloud (APC) 0.37.4 to 2.0. This is a major upgrade that includes platform-level infrastructure changes (NATS JetStream migration, component replacements) as well as a comprehensive `values.yaml` schema migration.

<Warning>
  **Upgrade requires platform downtime**

  This upgrade deletes and recreates several platform components (NATS, STAN, the APC API). The platform will be unavailable for creating or updating Deployments during the upgrade. Plan a maintenance window and notify your users before starting.
</Warning>

<Note>
  If you are already on APC 1.x, use the simpler [Upgrade to 2.0 from 1.x](/docs/astro-private-cloud/v-2-x/upgrade-1x-to-2) guide instead. That guide doesn't require the platform-level migration steps described here.
</Note>

## Prerequisites

* [Upgrade to APC 0.37.4](/docs/astro-private-cloud/v-0-37/upgrade-astronomer) if you're on an earlier version.

* Back up your platform database. At minimum, create a snapshot or backup of your PostgreSQL database (RDS snapshot, Azure backup, Cloud SQL backup, or `pg_dump`).

* Verify that all platform and Airflow Deployment Pods are healthy:

  ```bash wrap theme={null}
  kubectl get pods -n astronomer
  kubectl get pods -n astronomer-<deployment-release-name>
  ```

* If you use Astronomer Units (AU) for resource configuration, [convert to CPU/memory settings while still on 0.37.x](/docs/astro-private-cloud/v-1-x/breaking-changes-removals#removal-of-astronomer-units-au-and-standardization-of-cpu/memory-settings) — the steps live on the APC 1.0 breaking changes page because that migration was required for the 1.0 release. The AU-to-CPU/memory migration script was removed in 1.0 and isn't available in 2.0.

* Check for duplicate workspace labels in your database. The upgrade includes a migration that adds a unique constraint to workspace labels, and it will fail if duplicates exist:

  ```sql wrap theme={null}
  SET search_path TO "houston$default";
  SELECT label, COUNT(*) FROM "Workspace" GROUP BY label HAVING COUNT(*) > 1;
  ```

  If this query returns any results, rename or remove the duplicate workspaces before proceeding.

* Remove any deprecated Helm values from your `values.yaml` that are no longer recognized in 2.0. See [Breaking changes and removals](/docs/astro-private-cloud/v-2-x/breaking-changes-removals) for the full list.

* Python 3.10 or later installed on your local computer (required for the values migration script).

* Install the `ruamel.yaml` Python package:

  ```bash wrap theme={null}
  pip install ruamel.yaml
  ```

* (Optional) Capture logs from your NATS and STAN instances before the upgrade:

  ```bash wrap theme={null}
  kubectl logs -n astronomer -l component=nats --tail=1000 > nats-pre-upgrade.log
  kubectl logs -n astronomer -l component=stan --tail=1000 > stan-pre-upgrade.log
  ```

<Tip>
  For help with upgrade issues, see [Debug upgrade](/docs/astro-private-cloud/v-2-x/debug-upgrade).
</Tip>

<Warning>
  **Manual DNS and load balancer update required**

  When upgrading from APC 0.`x.x` to 2.0, APC creates a new control plane NGINX ingress Service (`astronomer-cp-nginx`) and a new LoadBalancer. The previous ingress and LoadBalancer are replaced.

  You must:

  * Update all DNS records to the new load balancer IP address.
  * Update firewall, allowlist, and security rules to point to the new public endpoint.
  * Re-issue or update any TLS/SSL certificates that reference the previous LoadBalancer hostname, if applicable.

  This change occurs because the control plane ingress Service name changes from `astronomer-nginx` (0.x) to `astronomer-cp-nginx` (2.0), which causes Kubernetes to provision a new external LoadBalancer with a new public IP/hostname. Prepare these updates before performing the upgrade.

  Example (your output will vary by cloud/provider):

  ```bash wrap theme={null}
  kubectl -n astronomer get svc astronomer-cp-nginx
  NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP                                        PORT(S)                      AGE
  astronomer-cp-nginx  LoadBalancer   10.100.223.78   7666ac61ef6-1683718677.us-east-2.elb.amazonaws.com 80:32255/TCP,443:32647/TCP  25d
  ```
</Warning>

<Note>
  If you are on OpenShift and manage your own Routes or use a third-party ingress controller instead of the platform's built-in NGINX ingress, this LoadBalancer change doesn't apply to you. You can skip the DNS update step later in this guide.
</Note>

<Note>
  **Static or internal loadBalancerIP**

  If you explicitly set a static or internal `loadBalancerIP` under `nginx` in your `values.yaml` (for example, when using an internal GKE load balancer with `privateLoadBalancer: true`), the Helm chart passes that IP through to the new `astronomer-cp-nginx` Service. Kubernetes binds the new LoadBalancer to the same IP you provided, so the external IP doesn't change and no DNS update is required.

  In this scenario, you can skip the DNS update step later in this guide. You should still update firewall, allowlist, and TLS configurations if they reference the previous Service name.

  The behavior depends on the cloud provider's `LoadBalancer` implementation and whether it supports reusing a specified IP. Astronomer recommends verifying on your target platform before the production upgrade. For more information, see the [Kubernetes Service documentation](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer).
</Note>

## Step 1: Get the migration script

The migration scripts are located in the `bin/` folder of the [Astronomer Helm chart repository](https://github.com/astronomer/astronomer). Clone the repository and check out the `release-2.0` branch:

```bash wrap theme={null}
git clone https://github.com/astronomer/astronomer.git
cd astronomer
git checkout release-2.0
```

The script you need for this upgrade path is `bin/migrate-helm-chart-values-037x-to-2x.py`.

## Step 2: Migrate your `values.yaml`

The 0.37.x-to-2.x migration is a comprehensive transformation that includes feature flag restructuring, removal of obsolete components, key renames, value updates, new platform keys, and APC API configuration flag restructuring under `astronomer.houston.config.deployments`. The migration script automates the entire transformation.

### Back up your override file

```bash wrap theme={null}
cp my-values.yaml my-values.yaml.backup
```

### Preview changes (dry run)

```bash wrap theme={null}
./bin/migrate-helm-chart-values-037x-to-2x.py --dry-run my-values.yaml
```

Example output:

```text wrap theme={null}
Found 28 migration(s) to apply:
  global.rbacEnabled -> global.rbac.enabled
  global.sccEnabled -> global.scc.enabled
  ...
  global.singleNamespace -> (deleted)
  global.veleroEnabled -> (deleted)
  ...
  fluentd -> vector
  global.pgbouncer.krb5ConfSecretName -> global.pgbouncer.secretName
  global.pgbouncer.servicePort: "5432" -> "6543"
  ...
  (new) -> global.plane
  ...
```

If you see `No migrations needed`, your file is already compatible with 2.0.

### Run the migration

<Tabs>
  <Tab title="In-place with backup (recommended)">
    ```bash wrap theme={null}
    ./bin/migrate-helm-chart-values-037x-to-2x.py --in-place --backup my-values.yaml
    ```

    This modifies your file directly and creates `my-values.yaml.bak` as a backup.
  </Tab>

  <Tab title="Write to a new file">
    ```bash wrap theme={null}
    ./bin/migrate-helm-chart-values-037x-to-2x.py my-values.yaml migrated-values.yaml
    ```
  </Tab>

  <Tab title="Output to stdout">
    ```bash wrap theme={null}
    ./bin/migrate-helm-chart-values-037x-to-2x.py my-values.yaml > migrated-values.yaml
    ```
  </Tab>
</Tabs>

### Review the migrated file

Open the migrated file and verify the changes look correct. Pay special attention to:

* **New default values** — review the [New keys added with defaults](#new-keys-added-with-defaults) table and override any defaults that don't match your environment.
* **Fluentd-to-Vector rename** — if you had custom Fluentd configuration, only the resource values carry over. See [Breaking changes](/docs/astro-private-cloud/v-2-x/breaking-changes-removals) for migration guidance.
* **PgBouncer port** — confirm the new port (6543) works for your setup, or override it back to 5432.
* **Unified mode** — Ensure `global.plane.mode` is set to `"unified"`. This is equivalent to how 0.37.x operates and is required for the initial upgrade.

## Step 3: Validate the Helm upgrade (dry run)

Run a Helm upgrade dry run to verify that the upgrade will succeed:

```bash wrap theme={null}
helm upgrade -f migrated-values.yaml -n astronomer astronomer astronomer/astronomer --version 2.0.x --dry-run
```

Replace `2.0.x` with the specific patch version you are upgrading to.

<Warning>
  Don't proceed with the remaining steps until the dry run completes successfully. If it fails, resolve the reported errors first.
</Warning>

## Step 4: Delete existing STAN and NATS StatefulSets

Before upgrading, you must migrate from STAN to JetStream. Run the following commands to remove legacy STAN components:

```bash wrap theme={null}
kubectl delete sts <release-name>-stan
kubectl delete sts <release-name>-nats --cascade=orphan
```

## Step 5: Patch `astronomer-bootstrap` secret with database name

Ensure the `astronomer-bootstrap` secret includes a database name suffix in `connection` (for example, `/postgres`). This prevents Postgres connection errors during or after the upgrade.

### Check your current connection string

```bash wrap theme={null}
kubectl get secret -n astronomer astronomer-bootstrap \
  --template='{{.data.connection | base64decode }}'
```

A connection string with a database name looks like:

```text wrap theme={null}
postgresql://user:pass@host:5432/postgres
                                 ^^^^^^^^
                                 database name present
```

A connection string without a database name looks like:

```text wrap theme={null}
postgresql://user:pass@host:5432
                                ^
                                no database name
```

If your connection string already includes a database name, skip to [Step 6](#step-6-delete-the-apc-deployment).

### Determine the correct database name

The default database name depends on your cloud provider:

* **AWS RDS**: `postgres` (standard default; `main` may appear only in specific legacy or custom configurations)
* **Azure Database for PostgreSQL**: `postgres`
* **GCP Cloud SQL**: `postgres`

If you're unsure, connect to your PostgreSQL instance and run `\l` to list available databases.

### Patch the secret

```bash wrap theme={null}
NAMESPACE=astronomer
DB_NAME=postgres

CURRENT=$(kubectl -n "$NAMESPACE" get secret astronomer-bootstrap -o jsonpath='{.data.connection}' | base64 -d)
NEW="${CURRENT%/}/$DB_NAME"
kubectl -n "$NAMESPACE" patch secret astronomer-bootstrap --type=merge -p "{\"data\":{\"connection\":\"$(printf '%s' "$NEW" | base64 -w0)\"}}"
```

## Step 6: Delete the APC Deployment

Delete the APC Deployment to avoid a known Helm patch conflict related to environment variable ordering changes between versions.

```bash wrap theme={null}
kubectl delete deployment/<release-name>-houston -n astronomer --cascade=orphan
```

The `--cascade=orphan` flag keeps the APC Pods running during the delete operation. The Helm upgrade in the next step recreates the Deployment with the correct configuration.

<Tip>
  If you skip this step and encounter a Helm patch error during the upgrade, see [Debug upgrade](/docs/astro-private-cloud/v-2-x/debug-upgrade#helm-upgrade-fails-with-patch-conflict) for the workaround.
</Tip>

## Step 7: Upgrade to APC 2.0

Ensure that your migrated `values.yaml` includes the unified mode configuration:

```yaml wrap theme={null}
global:
  plane:
    mode: "unified"
```

<Note>
  After upgrading to 2.0 in unified mode, you can optionally add separate data planes to run Airflow Deployments in other clusters.
</Note>

Complete all pre-upgrade steps (database backup, STAN/NATS deletion, bootstrap secret patch, APC Deployment deletion, values migration) before running the upgrade command.

```bash wrap theme={null}
helm upgrade -f migrated-values.yaml -n astronomer astronomer astronomer/astronomer --version 2.0.x
```

Replace `2.0.x` with the specific patch version you are upgrading to.

<Note>
  **Airgapped environments**

  For airgapped environments that can't access the internet, download the Helm chart `.tgz` file directly from the Astronomer Helm repository:

  ```text wrap theme={null}
  https://helm.astronomer.io/astronomer-<version>.tgz
  ```

  Replace `<version>` with the specific version you are upgrading to. Upload this file to your internal artifact repository, then reference it in your `helm upgrade` command.
</Note>

## Step 8: Restart NATS and APC API components

After the platform upgrade completes and all Pods are running, restart NATS and APC API components to ensure the new JetStream components and APC API services are connected and synchronized:

```bash wrap theme={null}
kubectl rollout restart sts/<release-name>-nats
kubectl rollout restart deploy/<release-name>-houston
kubectl rollout restart deploy/<release-name>-houston-worker
```

After the rollout completes, verify the Pods have been recreated by checking their age:

```bash wrap theme={null}
kubectl get pods -n astronomer -l "component in (nats,houston,houston-worker)" -o wide
```

All Pods should show a recent `AGE` (a few minutes). If any Pods show an older age, delete them manually to force recreation:

```bash wrap theme={null}
kubectl delete pod -n astronomer -l component=nats
kubectl delete pod -n astronomer -l component=houston
kubectl delete pod -n astronomer -l component=houston-worker
```

## Step 9: Update DNS records

APC 2.0 creates a new control plane NGINX ingress Service (`astronomer-cp-nginx`) with a new LoadBalancer. *You must complete this step before you can access the Astro UI or API*.

Skip this step if either of the following applies:

* You are on OpenShift and manage your own Routes or use a third-party ingress controller.
* You explicitly set a static `loadBalancerIP` under `nginx` in your `values.yaml`. In this case, the new Service binds to the same IP and no DNS update is required. Confirm by comparing the `EXTERNAL-IP` returned by the following command to the IP in your `values.yaml`.

Get the new load balancer address:

```bash wrap theme={null}
kubectl -n astronomer get svc astronomer-cp-nginx
```

Update your DNS records to point to the new load balancer IP or hostname. This includes all subdomains for your base domain (for example, `app.<baseDomain>`, `houston.<baseDomain>`, `registry.<baseDomain>`).

## Step 10: Upgrade all Airflow Deployments

After you have validated that all platform Pods are healthy, upgrade your Airflow Deployments to ensure compatibility with 2.0.

<Note>
  Existing Airflow Deployments typically continue to function after the platform upgrade without immediate action. However, Astronomer recommends upgrading Deployments to ensure full compatibility with the new platform version.
</Note>

To upgrade Deployments, use one of the following approaches:

* **Astro UI**: Upgrade each Deployment manually from the Astro UI.
* **APC API**: Use the APC API `upsertDeployment` mutation for programmatic or bulk upgrades.
* **Astro CLI**: Use the Astro CLI's [astro deploy](/docs/cli/v1.43/astro-deploy) command.

## Step 11: Validate the upgrade

<Steps>
  <Step title="Confirm that NATS Pods are running with JetStream enabled">
    Check if the JetStream job is created:

    ```bash wrap theme={null}
    kubectl -n astronomer get jobs | grep jetstream
    ```

    Check if NATS Pods are running:

    ```bash wrap theme={null}
    kubectl -n astronomer get pods -l component=nats
    ```
  </Step>

  <Step title="Verify APC Worker Pods are healthy and processing events">
    Check if APC Worker Pods are running:

    ```bash wrap theme={null}
    kubectl -n astronomer get pods -l component=houston-worker
    ```
  </Step>

  <Step title="Verify there are no remaining references to STAN">
    This command should return no results:

    ```bash wrap theme={null}
    kubectl -n astronomer get statefulsets | grep stan
    ```
  </Step>

  <Step title="Verify Vector is running (replaces Fluentd)">
    ```bash wrap theme={null}
    kubectl -n astronomer get pods -l component=vector
    ```
  </Step>

  <Step title="Verify you can access the Astro UI">
    Navigate to `app.<baseDomain>` in your browser and confirm the UI loads.
  </Step>
</Steps>

## Values migration reference

`bin/migrate-helm-chart-values-037x-to-2x.py` on the [Astronomer Helm chart `release-2.0` branch](https://github.com/astronomer/astronomer) includes **every** shared rule from [Values migration reference](/docs/astro-private-cloud/v-2-x/upgrade-1x-to-2#values-migration-reference) in `bin/helm_chart_values_migration_shared.py`, **and** the 0.37-only steps in the tables below. The script runs, in order: (1) `apply_global_feature_flag_rules_to_all`; (2) the 0.37 `MIGRATIONS` list in `migrate-helm-chart-values-037x-to-2x.py` (every `DeleteKey`, `RenameKey`, `SetValue`, and `AddKeyIfMissing` in that file); (3) the same `MIGRATIONS` against nested `global` maps; (4) `apply_houston_config_flag_migrations`; (5) `apply_houston_deployment_migrations`; (6) `apply_nginx_csp_policy_migrations`. The 1.x script uses the same APC API and nginx steps but a different order and adds `strictSchemaCheck` at the end; the 0.37 script adds `strictSchemaCheck` during step (2). Use `--dry-run` to see the exact list for your file.

The following tables are the **complete** 0.37-only portion (the full `MIGRATIONS` list from that script).

### Deleted keys (0.37 script)

| Key                                         | Notes                                                                        |
| ------------------------------------------- | ---------------------------------------------------------------------------- |
| `global.singleNamespace`                    | Obsolete feature                                                             |
| `global.veleroEnabled`                      | Obsolete feature                                                             |
| `global.enableHoustonInternalAuthorization` | Replaced by install-mode behavior                                            |
| `global.nodeExporterSccEnabled`             | No longer used                                                               |
| `global.stan`                               | STAN removed (JetStream)                                                     |
| `tags.stan`                                 | STAN tag removed                                                             |
| `stan` (top-level)                          | STAN resources removed                                                       |
| `kibana` (top-level)                        | See [Breaking changes](/docs/astro-private-cloud/v-2-x/breaking-changes-removals) |
| `prometheus-blackbox-exporter` (top-level)  | See [Breaking changes](/docs/astro-private-cloud/v-2-x/breaking-changes-removals) |

### Renamed keys (0.37 script)

| Old path                              | New name / path               | Notes                                                                |
| ------------------------------------- | ----------------------------- | -------------------------------------------------------------------- |
| `fluentd` (top-level key)             | `vector` (top-level)          | Subtree kept; custom Fluentd config isn't auto-translated to Vector. |
| `global.pgbouncer.krb5ConfSecretName` | `global.pgbouncer.secretName` | Same Secret object                                                   |

### Value updates (only when the current value matches the "old" column)

| Path                            | Old             | New             | Notes                                             |
| ------------------------------- | --------------- | --------------- | ------------------------------------------------- |
| `global.pgbouncer.servicePort`  | `5432` (string) | `6543` (string) | Set back to `5432` in overrides if required       |
| `global.nats.jetStream.enabled` | `false`         | `true`          | Set to `false` in overrides to keep JetStream off |

### New keys added with defaults

The migration script adds these keys with default values if they aren't already present. Review each key and override the default if it doesn't match your environment.

| Key Path                              | Default Value | Description                                               | Action Needed?                                                               |
| ------------------------------------- | ------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------- |
| `global.authHeaderSecretName`         | `~` (null)    | Name of Kubernetes secret for cross-plane authentication. | Set this if running in multi-plane mode. Not needed for unified mode.        |
| `global.plane.mode`                   | `"unified"`   | Platform operating mode: `control`, `data`, or `unified`. | Change to `control` or `data` if running a multi-plane deployment.           |
| `global.plane.domainPrefix`           | `""`          | Cluster identifier prefix for multi-plane DNS.            | Set to your cluster ID if running in multi-plane mode.                       |
| `global.podLabels`                    | `{}`          | Labels applied to every pod.                              | Add labels if needed for monitoring, cost allocation, or policy enforcement. |
| `global.logging.provider`             | `~` (null)    | Logging provider identifier.                              | Set if using a specific logging backend.                                     |
| `nats.init.resources.requests.cpu`    | `"75m"`       | NATS init container CPU request.                          | Override if your cluster needs different resource settings.                  |
| `nats.init.resources.requests.memory` | `"30Mi"`      | NATS init container memory request.                       | Override if your cluster needs different resource settings.                  |
| `nats.init.resources.limits.cpu`      | `"250m"`      | NATS init container CPU limit.                            | Override if your cluster needs different resource settings.                  |
| `nats.init.resources.limits.memory`   | `"100Mi"`     | NATS init container memory limit.                         | Override if your cluster needs different resource settings.                  |

## Roll back the upgrade

If you need to revert the values migration:

1. Restore your backup:

   ```bash wrap theme={null}
   cp my-values.yaml.backup my-values.yaml
   ```

2. To downgrade the chart after a Helm upgrade:

   ```bash wrap theme={null}
   helm rollback astronomer <previous-revision> --namespace astronomer
   ```


# Upgrade to Astro Private Cloud 2.0 from 1.x
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/upgrade-1x-to-2

Upgrade to Astro Private Cloud 2.0 from 1.x, including values.yaml migration and Helm chart upgrade.

Use this document to upgrade from Astro Private Cloud (APC) 1.x to 2.0. The upgrade primarily involves migrating your `values.yaml` override file to the new schema and running a Helm upgrade.

<Note>
  If you are upgrading from APC 0.37.x directly to 2.0, use the [Upgrade to 2.0 from 0.37](/docs/astro-private-cloud/v-2-x/upgrade-037-to-2) guide instead. That guide covers the additional platform-level steps required when skipping 1.x.
</Note>

## Prerequisites

* Verify that you are running APC 1.x. If you are on 0.37.x, see the [Upgrade to 2.0 from 0.37](/docs/astro-private-cloud/v-2-x/upgrade-037-to-2) guide.

* Back up your platform database. At minimum, create a snapshot or backup of your PostgreSQL database (RDS snapshot, Azure backup, Cloud SQL backup, or `pg_dump`).

* Verify that all platform and Airflow Deployment Pods are healthy:

  ```bash wrap theme={null}
  kubectl get pods -n astronomer
  kubectl get pods -n astronomer-<deployment-release-name>
  ```

* Python 3.10 or later installed on your local computer (required for the values migration script).

* Install the `ruamel.yaml` Python package:

  ```bash wrap theme={null}
  pip install ruamel.yaml
  ```

## Step 1: Get the migration script

The migration scripts are located in the `bin/` folder of the [Astronomer Helm chart repository](https://github.com/astronomer/astronomer). Clone the repository and check out the `release-2.0` branch:

```bash wrap theme={null}
git clone https://github.com/astronomer/astronomer.git
cd astronomer
git checkout release-2.0
```

The script you need for this upgrade path is `bin/migrate-helm-chart-values-1x-to-2x.py`.

## Step 2: Migrate your `values.yaml`

APC 2.0 introduces a unified feature flag schema that reorganizes scattered `global.*` boolean flags into domain-grouped structures with a consistent `.enabled` pattern. The migration script also restructures APC API configuration flags under `astronomer.houston.config.deployments` into nested paths. The migration script automates both transformations.

### Back up your override file

```bash wrap theme={null}
cp my-values.yaml my-values.yaml.backup
```

### Preview changes (dry run)

Run the migration script in dry-run mode to see what will change without modifying any files:

```bash wrap theme={null}
./bin/migrate-helm-chart-values-1x-to-2x.py --dry-run my-values.yaml
```

Example output:

```text wrap theme={null}
Found 12 migration(s) to apply:
  global.rbacEnabled -> global.rbac.enabled
  global.sccEnabled -> global.scc.enabled
  global.openshiftEnabled -> global.openshift.enabled
  global.networkNSLabels -> global.networkNSLabels.enabled
  global.namespaceFreeFormEntry -> global.namespaceManagement.namespaceFreeFormEntry.enabled
  global.taskUsageMetricsEnabled -> global.metricsReporting.taskUsageMetrics.enabled
  global.deployRollbackEnabled -> global.deploymentLifecycle.deployRollback.enabled
  global.features.namespacePools -> global.namespaceManagement.namespacePools
  global.dagOnlyDeployment -> global.deployMechanisms.dagOnlyDeployment
  global.loggingSidecar -> global.logging.loggingSidecar
  astronomer.houston.config.deployments.dagProcessorEnabled -> astronomer.houston.config.deployments.airflowComponents.dagProcessor.enabled
  astronomer.houston.config.deployments.triggererEnabled -> astronomer.houston.config.deployments.airflowComponents.triggerer.enabled
```

If you see `No migrations needed`, your file is already compatible with 2.0.

### Run the migration

<Tabs>
  <Tab title="In-place with backup (recommended)">
    ```bash wrap theme={null}
    ./bin/migrate-helm-chart-values-1x-to-2x.py --in-place --backup my-values.yaml
    ```

    This modifies your file directly and creates `my-values.yaml.bak` as a backup.
  </Tab>

  <Tab title="Write to a new file">
    ```bash wrap theme={null}
    ./bin/migrate-helm-chart-values-1x-to-2x.py my-values.yaml migrated-values.yaml
    ```
  </Tab>

  <Tab title="Output to stdout">
    ```bash wrap theme={null}
    ./bin/migrate-helm-chart-values-1x-to-2x.py my-values.yaml > migrated-values.yaml
    ```
  </Tab>
</Tabs>

### Review the migrated file

Open the migrated file and verify the changes look correct. The script preserves YAML comments and formatting.

## Step 3: Validate the Helm upgrade (dry run)

Before making any changes to your cluster, run a Helm upgrade dry run to verify that the upgrade will succeed:

```bash wrap theme={null}
helm upgrade -f migrated-values.yaml -n astronomer astronomer astronomer/astronomer --version 2.0.x --dry-run
```

Replace `2.0.x` with the specific patch version you are upgrading to.

<Warning>
  Don't proceed until the dry run completes successfully. If it fails, resolve the reported errors first.
</Warning>

## Step 4: Upgrade to APC 2.0

Perform a standard Helm upgrade using your migrated values file:

```bash wrap theme={null}
helm upgrade -f migrated-values.yaml -n astronomer astronomer astronomer/astronomer --version 2.0.x
```

Replace `2.0.x` with the specific patch version you are upgrading to.

<Note>
  **Airgapped environments**

  For airgapped environments that can't access the internet, download the Helm chart `.tgz` file directly from the Astronomer Helm repository:

  ```text wrap theme={null}
  https://helm.astronomer.io/astronomer-<version>.tgz
  ```

  Replace `<version>` with the specific version you are upgrading to. Upload this file to your internal artifact repository, then reference it in your `helm upgrade` command.
</Note>

## Step 5: Migrate cluster configuration in the APC database

The Helm values migration Python script updates `values.yaml` on disk. Separately, the APC database can still store legacy flat `deployments` keys in each registered cluster’s `config` and `configOverride` until the cluster config path migration has run.

That migration is part of the APC API’s `yarn migrate` script (it invokes `migrate-cluster-config-paths` along with other jobs). If your control-plane upgrade already runs the full `yarn migrate` pipeline, this step is usually not a separate manual task.

If the migration hasn’t run or you need to run it again (for example after a failed migrate, or to confirm with a dry run first), you can run the same job independently with `yarn migrate-cluster-config-paths` from the `houston-api` pod, with the platform database reachable. Without this database update—whether it ran inside `yarn migrate` or a standalone run—cluster-level rows might not line up with the 2.0 [config governance](/docs/astro-private-cloud/v-2-x/config-governance) model.

* **What it does**: For every registered cluster, the APC API rewrites stored `config` / `configOverride` from old paths to the domain-grouped nested structure (the same renames the Helm migrator applies under `astronomer.houston.config.deployments`). It also invalidates the cluster details cache in the APC API.

* **When you need a standalone run**: For **1.x → 2.0** when you had registered data plane clusters in 1.x, **and** the cluster rows were never migrated (for example `yarn migrate` didn't run, or the step failed). **Don't** treat this as a standard [0.37 → 2.0](/docs/astro-private-cloud/v-2-x/upgrade-037-to-2) requirement—you typically have no pre-existing 2.0 cluster records to fix on that path.

* **Command** (on a host with the APC API release and DB access):

  ```bash wrap theme={null}
  yarn migrate-cluster-config-paths --dry-run
  yarn migrate-cluster-config-paths
  ```

  A dry run reports how many clusters **would** be updated; a full run applies changes and reports migrated vs skipped counts. Exit is non-zero if any cluster failed.

* **Strict config validation in the APC API**: The chart sets `astronomer.houston.strictSchemaCheck`. For how the 1.x migrator injects it, see the [values migration reference](#values-migration-reference) section. When `strictSchemaCheck.enabled` is `true`, the APC API rejects `deployments` config overrides with unknown keys or bad types. If you need to allow keys not yet in the generated schema, you can set `astronomer.houston.strictSchemaCheck.enabled: false` in your platform `values.yaml` and run another `helm upgrade` (see [Git-Sync relay metrics](/docs/astro-private-cloud/v-2-x/git-sync-relay-metrics) for a related case).

## Step 6: Verify the upgrade

After the upgrade completes, verify that all platform components are running:

```bash wrap theme={null}
kubectl get pods -n astronomer
```

All Pods should be in a `Running` or `Completed` state. If any Pods are in a `CrashLoopBackOff` or `Error` state, see [Debug upgrade](/docs/astro-private-cloud/v-2-x/debug-upgrade).

## Step 7: Upgrade all Airflow Deployments

After you have validated that all platform Pods are healthy, upgrade your Airflow Deployments to ensure full compatibility with 2.0.

<Note>
  Existing Airflow Deployments typically continue to function after the platform upgrade without immediate action. However, Astronomer recommends upgrading Deployments to ensure full compatibility with the new platform version.
</Note>

To upgrade Deployments, use one of the following approaches:

* **Astro UI**: Upgrade each Deployment manually from the Astro UI by navigating to the Deployment and triggering an upgrade.
* **APC API**: Use the APC API `upsertDeployment` mutation for programmatic or bulk upgrades.
* **Astro CLI**: Use the Astro CLI's [astro deploy](/docs/cli/v1.43/astro-deploy) command.

## Values migration reference

`bin/migrate-helm-chart-values-1x-to-2x.py` applies rules in this order: (1) every `global` map in the file (values under each `global` key, including nested charts); (2) `astronomer.houston.config` flags; (3) `astronomer.houston.config.deployments` rewrites, plus chart-only key deletions; (4) `nginx.cspPolicy`; (5) inject `astronomer.houston.strictSchemaCheck` if missing. The following tables list every rule in `bin/helm_chart_values_migration_shared.py` that the 1.x script uses. Keep this list aligned with the `release-2.0` branch of the [Astronomer Helm chart](https://github.com/astronomer/astronomer). The script only migrates keys that are present in your file; it doesn't add keys you didn't set, except for `strictSchemaCheck` as noted. Keys with no rule here are left unchanged.

### `global` feature flags

For each `global` mapping in the document, the old key is removed after migration (or dropped if the new path already exists).

| Old path                                     | New path                                                    | Type             |
| -------------------------------------------- | ----------------------------------------------------------- | ---------------- |
| `global.rbacEnabled`                         | `global.rbac.enabled`                                       | boolean → nested |
| `global.sccEnabled`                          | `global.scc.enabled`                                        | boolean → nested |
| `global.openshiftEnabled`                    | `global.openshift.enabled`                                  | boolean → nested |
| `global.networkNSLabels`                     | `global.networkNSLabels.enabled`                            | boolean → nested |
| `global.namespaceFreeFormEntry`              | `global.namespaceManagement.namespaceFreeFormEntry.enabled` | boolean → nested |
| `global.taskUsageMetricsEnabled`             | `global.metricsReporting.taskUsageMetrics.enabled`          | boolean → nested |
| `global.deployRollbackEnabled`               | `global.deploymentLifecycle.deployRollback.enabled`         | boolean → nested |
| `global.features.namespacePools` (subtree)   | `global.namespaceManagement.namespacePools`                 | subtree move     |
| `global.dagOnlyDeployment` (subtree)         | `global.deployMechanisms.dagOnlyDeployment`                 | subtree move     |
| `global.loggingSidecar` (subtree)            | `global.logging.loggingSidecar`                             | subtree move     |
| `global.podDisruptionBudgetsEnabled`         | `global.podDisruptionBudgets.enabled`                       | boolean → nested |
| `global.postgresqlEnabled`                   | `global.postgresql.enabled`                                 | boolean → nested |
| `global.prometheusPostgresExporterEnabled`   | `global.prometheusPostgresExporter.enabled`                 | boolean → nested |
| `global.nodeExporterEnabled`                 | `global.nodeExporter.enabled`                               | boolean → nested |
| `global.manualNamespaceNamesEnabled`         | `global.namespaceManagement.manualNamespaceNames.enabled`   | boolean → nested |
| `global.enablePerHostIngress`                | `global.perHostIngress.enabled`                             | boolean → nested |
| `global.enableArgoCDAnnotation`              | `global.argoCD.annotation.enabled`                          | boolean → nested |
| `global.disableManageClusterScopedResources` | `global.manageClusterScopedResources.enabled`               | inverted boolean |
| `global.astronomerEnabled`                   | `global.astronomer.enabled`                                 | boolean → nested |
| `global.nginxEnabled`                        | `global.nginx.enabled`                                      | boolean → nested |
| `global.alertmanagerEnabled`                 | `global.alertmanager.enabled`                               | boolean → nested |
| `global.grafanaEnabled`                      | `global.grafana.enabled`                                    | boolean → nested |
| `global.kubeStateEnabled`                    | `global.kubeState.enabled`                                  | boolean → nested |
| `global.prometheusEnabled`                   | `global.prometheus.enabled`                                 | boolean → nested |
| `global.elasticsearchEnabled`                | `global.elasticsearch.enabled`                              | boolean → nested |
| `global.vectorEnabled`                       | `global.daemonsetLogging.enabled`                           | boolean → nested |
| `global.fluentdEnabled`                      | `global.daemonsetLogging.enabled`                           | boolean → nested |

### `astronomer.houston.config` (non-`deployments`) flags and move

| Old path                                                               | New path                                                                   | Type             |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------- | ---------------- |
| `astronomer.houston.config.emailConfirmation`                          | `astronomer.houston.config.emailConfirmation.enabled`                      | boolean → nested |
| `astronomer.houston.config.publicSignups`                              | `astronomer.houston.config.publicSignups.enabled`                          | boolean → nested |
| `astronomer.houston.config.updateRuntimeCheckEnabled`                  | `astronomer.houston.config.updateRuntimeCheck.enabled`                     | boolean → nested |
| `astronomer.houston.config.updateAirflowCheckEnabled`                  | `astronomer.houston.config.updateAirflowCheck.enabled`                     | boolean → nested |
| `astronomer.houston.config.subdomainHttpsEnabled`                      | `astronomer.houston.config.subdomainHttps.enabled`                         | boolean → nested |
| `astronomer.houston.config.useAutoCompleteForSensitiveFields`          | `astronomer.houston.config.autoCompleteForSensitiveFields.enabled`         | boolean → nested |
| `astronomer.houston.config.shouldLogUsername`                          | `astronomer.houston.config.logUsername.enabled`                            | boolean → nested |
| `astronomer.houston.config.disableSSLVerify`                           | `astronomer.houston.config.sslVerification.enabled`                        | inverted boolean |
| `astronomer.houston.config.auth.openidConnect.idpGroupsImportEnabled`  | `astronomer.houston.config.auth.openidConnect.idpGroupsImport.enabled`     | boolean → nested |
| `astronomer.houston.config.auth.openidConnect.idpGroupsRefreshEnabled` | `astronomer.houston.config.auth.openidConnect.idpGroupsRefresh.enabled`    | boolean → nested |
| `astronomer.houston.config.auth.openidConnect.insecureIDPTokenLog`     | `astronomer.houston.config.auth.openidConnect.insecureIDPTokenLog.enabled` | boolean → nested |
| `astronomer.houston.config.webserver.graphqlPlaygroundEnabled`         | `astronomer.houston.config.webserver.graphqlPlayground.enabled`            | boolean → nested |
| `astronomer.houston.config.nats.tlsEnabled`                            | `astronomer.houston.config.nats.tls.enabled`                               | boolean → nested |
| `astronomer.houston.config.workers.dplink.debugEnabled`                | `astronomer.houston.config.workers.dplink.debug.enabled`                   | boolean → nested |
| `astronomer.houston.config.apollo.auditMiddlewareEnabled`              | `astronomer.houston.config.apollo.auditMiddleware.enabled`                 | boolean → nested |
| `astronomer.houston.config.deployments.mockWebhook.krbEnabled`         | `astronomer.houston.config.deployments.mockWebhook.krb.enabled`            | boolean → nested |
| `astronomer.houston.config.helm.rbacEnabled`                           | `astronomer.houston.config.helm.rbac.enabled`                              | boolean → nested |
| `astronomer.houston.config.deployments.mockWebhook.krbRealm`           | `astronomer.houston.config.deployments.mockWebhook.krb.realm`              | move value       |

### `nginx.cspPolicy`

| Old path                     | New path                  | Type             |
| ---------------------------- | ------------------------- | ---------------- |
| `nginx.cspPolicy.cdnEnabled` | `nginx.cspPolicy.enabled` | boolean → nested |

### `astronomer.houston.config.deployments` path migrations

Let `d` = `astronomer.houston.config.deployments`. Migrations run in the order below (same as `HOUSTON_DEPLOYMENT_PATH_MIGRATIONS` in code). The **Transform** column names match the code: `boolean-to-airflowV3` adds a default `minimumAstroRuntimeVersion` when converting from a plain boolean, and `taskUsageReport-to-taskUsageMetrics` maps the legacy task usage object into `metricsReporting.taskUsageMetrics`. `deprecated-unset` means the key is **removed** with no replacement.

| Old key (`d.<key>`)                                   | New path (under `d`)                                                             | Transform                           |
| ----------------------------------------------------- | -------------------------------------------------------------------------------- | ----------------------------------- |
| `performanceOptimizationModeEnabled`                  | `performanceOptimization.enabled`                                                | boolean-to-enabled                  |
| `upsertExtraIniAllowed`                               | `upsertDeployment.extraIniAllowed`                                               | move                                |
| `logHelmValues`                                       | `logHelmValues.enabled`                                                          | boolean-to-enabled                  |
| `airflowV3`                                           | `runtimeManagement.airflowV3`                                                    | boolean-to-airflowV3                |
| `customImageShaEnabled`                               | `runtimeManagement.customImageSha.enabled`                                       | boolean-to-enabled                  |
| `enableListAllRuntimeVersions`                        | `runtimeManagement.listAllRuntimeVersions.enabled`                               | boolean-to-enabled                  |
| `runtimeEnvOverideSemverCheck`                        | `runtimeManagement.runtimeEnvOverrideSemverCheck`                                | move (note: old key name spelling)  |
| `astroRuntimeReleasesFile`                            | `runtimeManagement.astroRuntimeReleasesFile`                                     | move                                |
| `airflowMinimumAstroRuntimeVersion`                   | `runtimeManagement.airflowMinimumAstroRuntimeVersion`                            | move                                |
| `loggingSidecar`                                      | `logging.loggingSidecar`                                                         | move                                |
| `elasticsearch`                                       | `logging.elasticsearch`                                                          | move                                |
| `configureDagDeployment`                              | `deployMechanisms.configureDagDeployment.enabled`                                | boolean-to-enabled                  |
| `dagOnlyDeployment`                                   | `deployMechanisms.dagOnlyDeployment.enabled`                                     | boolean-to-enabled                  |
| `nfsMountDagDeployment`                               | `deployMechanisms.nfsMountDagDeployment.enabled`                                 | boolean-to-enabled                  |
| `gitSyncDagDeployment`                                | `deployMechanisms.gitSyncDagDeployment.enabled`                                  | boolean-to-enabled                  |
| `gitSyncRelay`                                        | `deployMechanisms.gitSyncRelay`                                                  | move                                |
| `triggererEnabled`                                    | `airflowComponents.triggerer.enabled`                                            | boolean-to-enabled                  |
| `dagProcessorEnabled`                                 | `airflowComponents.dagProcessor.enabled`                                         | boolean-to-enabled                  |
| `disableManageResourceQuotasAndLimitRanges`           | `resourceManagement.resourceQuotas.enabled`                                      | invert-to-enabled                   |
| `components`                                          | `resourceManagement.components`                                                  | move                                |
| `executors`                                           | `resourceManagement.executors`                                                   | move                                |
| `astroUnit`                                           | `resourceManagement.astroUnit`                                                   | move                                |
| `maxExtraCapacity`                                    | `resourceManagement.maxExtraCapacity`                                            | move                                |
| `maxPodCapacity`                                      | `resourceManagement.maxPodCapacity`                                              | move                                |
| `sidecars`                                            | `resourceManagement.sidecars`                                                    | move                                |
| `overProvisioningFactorMem`                           | `resourceManagement.overProvisioningFactorMem`                                   | move                                |
| `overProvisioningFactorCPU`                           | `resourceManagement.overProvisioningFactorCPU`                                   | move                                |
| `overProvisioningComponents`                          | `resourceManagement.overProvisioningComponents`                                  | move                                |
| `manualReleaseNames`                                  | `namespaceManagement.manualReleaseNames.enabled`                                 | boolean-to-enabled                  |
| `manualNamespaceNames`                                | `namespaceManagement.manualNamespaceNames.enabled`                               | boolean-to-enabled                  |
| `namespaceFreeFormEntry`                              | `namespaceManagement.namespaceFreeFormEntry`                                     | object-or-boolean-to-nested         |
| `preDeploymentValidationHook`                         | `namespaceManagement.namespaceFreeFormEntry.preDeploymentValidationHook`         | move                                |
| `preDeploymentValidationHookTimeout`                  | `namespaceManagement.namespaceFreeFormEntry.preDeploymentValidationHookTimeout`  | move                                |
| `namespaceLabels`                                     | `namespaceManagement.namespaceLabels`                                            | move                                |
| `preCreatedNamespaces`                                | `namespaceManagement.preCreatedNamespaces`                                       | move                                |
| `deployRollback`                                      | `deploymentLifecycle.deployRollback`                                             | object-or-boolean-to-nested         |
| `hardDeleteDeployment`                                | `deploymentLifecycle.hardDeleteDeployment.enabled`                               | boolean-to-enabled                  |
| `cleanupAirflowDb`                                    | `deploymentLifecycle.cleanupAirflowDb`                                           | object-or-boolean-to-nested         |
| `database`                                            | `databaseManagement.database`                                                    | move                                |
| `manualConnectionStrings`                             | `databaseManagement.manualConnectionStrings.enabled`                             | boolean-to-enabled                  |
| `pgBouncerResourceCalculationStrategy`                | `databaseManagement.pgBouncerResourceCalculationStrategy`                        | move                                |
| `exposeDockerWebhookEndpoint`                         | `deploymentImagesRegistry.exposeDockerWebhookEndpoint.enabled`                   | boolean-to-enabled                  |
| `enableUpdateDeploymentImageEndpoint`                 | `deploymentImagesRegistry.updateDeploymentImageEndpoint.enabled`                 | boolean-to-enabled                  |
| `enableUpdateDeploymentImageEndpointDockerValidation` | `deploymentImagesRegistry.updateDeploymentImageEndpointDockerValidation.enabled` | boolean-to-enabled                  |
| `serviceAccountAnnotationKey`                         | `deploymentImagesRegistry.serviceAccountAnnotationKey`                           | move                                |
| `grafanaUIEnabled`                                    | `metricsReporting.grafana.enabled`                                               | boolean-to-enabled                  |
| `taskUsageReport`                                     | `metricsReporting.taskUsageMetrics`                                              | taskUsageReport-to-taskUsageMetrics |
| `pagination`                                          | `metricsReporting.pagination`                                                    | move                                |
| `canUpsertDeploymentFromUI`                           | `upsertDeployment.allowFromUi.enabled`                                           | boolean-to-enabled                  |

**Removed keys (no replacement, `deprecated-unset` or chart-only delete)**

| Key under `d`                                  | Reason                                |
| ---------------------------------------------- | ------------------------------------- |
| `upsertDeploymentEnabled`                      | `deprecated-unset`                    |
| `enableSystemAdminCanCreateDeprecatedAirflows` | `deprecated-unset`                    |
| `defaultDistribution`                          | `deprecated-unset`                    |
| `astroUnitsEnabled`                            | Chart-only delete (no 2.x equivalent) |
| `resourceProvisioningStrategy`                 | Chart-only delete (no 2.x equivalent) |
| `maxPodAu`                                     | Chart-only delete (no 2.x equivalent) |

### Injected default

| Path                                   | Injected if missing | Notes                                                        |
| -------------------------------------- | ------------------- | ------------------------------------------------------------ |
| `astronomer.houston.strictSchemaCheck` | `enabled: true`     | `AddKeyIfMissing` in `migrate-helm-chart-values-1x-to-2x.py` |

## FAQ

<AccordionGroup>
  <Accordion title="Is the migration idempotent?">
    Yes. Running the script multiple times on the same file produces the same output. Running it on an already-migrated file reports "No migrations needed" and makes no changes.
  </Accordion>

  <Accordion title="What about YAML comments?">
    The script uses `ruamel.yaml` in round-trip mode, which preserves comments and formatting. Inline comments transfer to the new key names. Comments on untouched keys are unaffected.
  </Accordion>

  <Accordion title="What if I have both old and new keys?">
    If a key already exists at the new-schema path, the new-schema value takes precedence and the stale old key is removed. For example, if your file contains both `global.rbacEnabled: true` and `global.rbac.enabled: false`, the script keeps `global.rbac.enabled: false` and deletes `global.rbacEnabled`.
  </Accordion>

  <Accordion title="What if I have a full copy of values.yaml instead of just overrides?">
    The script works on full files, but Astronomer recommends extracting only your customizations into a separate override file. Running the migration on a full copy of the old defaults may carry forward old default values (like image tags) that should be updated to the new chart defaults.
  </Accordion>
</AccordionGroup>

## Roll back

If you need to revert the values migration:

1. Restore your backup:

   ```bash wrap theme={null}
   cp my-values.yaml.backup my-values.yaml
   ```

2. To downgrade the chart after a Helm upgrade:

   ```bash wrap theme={null}
   helm rollback astronomer <previous-revision> --namespace astronomer
   ```


# Remote Execution Agent Helm chart release notes
Source: https://astronomer.io/docs/astro/agent-helm-chart-release-notes

A changelog of changes to the Astro Remote Execution Agent Helm chart, by chart version.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

This page lists changes to the Astro Remote Execution Agent Helm chart. The Helm chart has its own version, separate from the Remote Execution Agent version that it deploys by default. See [Helm chart versioning](/docs/astro/agent-maintenance-policy#helm-chart-versioning) for how the two relate, and the [Helm chart to Astro agent compatibility matrix](/docs/astro/agent-maintenance-policy#helm-chart-to-astro-agent-compatibility-matrix) for which combinations Astronomer supports.

For changes to the Remote Execution Agent itself, see [Remote Execution Agent release notes](/docs/astro/agent-release-notes). For the full list of configurable values, see [Helm chart configuration reference](/docs/astro/remote-agents-helm-reference).

<Update label="Helm chart 2.3.5" description="August 20, 2026">
  * **Changed:** Default `appVersion` updated to Remote Execution Agent 1.8.4.
</Update>

<Update label="Helm chart 2.3.4" description="July 29, 2026">
  * **Changed:** Default `appVersion` updated to Remote Execution Agent 1.8.3.
</Update>

<Update label="Helm chart 2.3.3" description="July 22, 2026">
  * **Fixed:** The chart no longer overrides image entrypoints. For the default agent images this ensures that orphaned processes in agent containers are correctly reaped by the init process.
</Update>

<Update label="Helm chart 2.3.2" description="July 17, 2026">
  * **Changed:** Default `appVersion` updated to Remote Execution Agent 1.8.2.
</Update>

<Update label="Helm chart 2.3.1" description="July 15, 2026">
  * **Changed:** Default `appVersion` updated to Remote Execution Agent 1.8.1, which added Airflow 3.3 support.
</Update>

<Update label="Helm chart 2.3.0" description="July 7, 2026">
  * **Added:** A `stateStoreBackend` value, alongside the existing `xcomBackend` and `secretBackend` values, to configure the Airflow 3.3+ worker-side state store backend. See [Configure state store backend](/docs/astro/remote-execution-configure-state-store-backend).
</Update>

<Update label="Helm chart 2.2.2" description="July 1, 2026">
  * **Changed:** Default `appVersion` updated to Remote Execution Agent 1.7.2.
</Update>

<Update label="Helm chart 2.2.1" description="June 17, 2026">
  * **Fixed:** 2.2.0 broke `helm install`/`helm template` when the newly-added OpenLineage configuration wasn't provided explicitly in `values.yaml`. Fixed in this release.
</Update>

<Update label="Helm chart 2.2.0" description="June 17, 2026">
  * **Added:** Extended OpenLineage configuration options: `openLineage.endpoint` and `openLineage.facetsEnvironmentVariables`. See [Configure OpenLineage](/docs/astro/remote-execution-configure-openlineage).
</Update>

<Update label="Helm chart 2.1.0" description="June 4, 2026">
  * **Added:** The chart now attempts to validate that configured agent image tags meet the minimum supported Astro Agent version. See the [compatibility matrix](/docs/astro/agent-maintenance-policy#helm-chart-to-astro-agent-compatibility-matrix).
  * **Added:** Sentinel RBAC now grants access to Kubernetes events, so Sentinel can surface event-based failure reasons, such as readiness probe failures, for unhealthy components.
  * **Added:** Agent containers now receive `ASTRO_AGENT_CLIENT_COMPONENT_NAME` and `ASTRO_AGENT_CLIENT_REVISION` environment variables, so each agent component can report its resolved configuration keyed to a specific component and rollout.
  * **Fixed:** Changed the chart's `kubeVersion` constraint from `>= 1.30.0` to `>= 1.30.0-0` so that Kubernetes server version strings with a pre-release suffix, such as those reported by some EKS clusters (for example, `1.35.4-eks-40737a8`), satisfy the constraint.
</Update>

<Update label="Helm chart 2.0.0" description="April 17, 2026">
  <Warning>
    **Breaking change: worker label selectors**

    Label selectors on worker Deployments and Services now include a per-worker label. Kubernetes label selectors are immutable, so an in-place `helm upgrade` from a 1.x chart to 2.0.0 fails. See [Breaking change in Helm chart 2.0.0](/docs/astro/agent-maintenance-policy#helm-chart-to-astro-agent-compatibility-matrix) for the required upgrade steps.
  </Warning>

  * **Changed:** `sentinel.enabled` is now `true` by default. Previously, Sentinel monitoring was disabled unless you explicitly opted in. Task-level metrics collection is also enabled by default. See [Enable Sentinel monitoring](/docs/astro/remote-agents-sentinel).
  * **Changed:** Label selectors on worker Deployments and Services now include a per-worker label. See the breaking change note for required upgrade steps.
  * **Added:** JSON schema validation for Helm values, covering required fields and field types.
  * **Added:** Support for a custom service account and annotations for each worker.
  * **Added:** RBAC permissions allowing workers to launch pods using the `KubernetesPodOperator`.
</Update>

<Update label="Helm chart 1.0.0 to 1.5.0" description="April 22, 2025 to March 24, 2026">
  These chart versions were released in lockstep with the Remote Execution Agent client of the same version number, before chart versioning was decoupled in Helm chart 2.0.0. See [Remote Execution Agent release notes](/docs/astro/agent-release-notes) for the agent changes released during this period.
</Update>


# Remote Execution Agent image reference
Source: https://astronomer.io/docs/astro/agent-images

Reference of images included in each Astro Remote Execution Agent release.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Astro Deployments.
</Note>

This page is a reference of the images included in each release of the [Astro Remote Execution Agent](/docs/astro/remote-execution-configure-agents).

## Understanding image versions

Remote Execution Agent images use the following naming format:

```text wrap theme={null}
images.astronomer.cloud/baseimages/astro-remote-execution-agent:<RUNTIME_VERSION>-python-<PYTHON_VERSION>-astro-agent-<AGENT_VERSION>
```

The image tag contains three version components:

* `RUNTIME_VERSION`: The Astro Runtime version (for example, `3.1-11` or `3.0-13`). This corresponds to the Astro Runtime version used in your Astro Deployment.
* `PYTHON_VERSION`: The Python version (for example, `3.11` or `3.12`). Match this to your Airflow project's Python version.
* `AGENT_VERSION`: The Remote Execution Agent version (for example, `1.3.3`).

Images with the `-base` suffix are minimal base images without additional dependencies pre-installed.

Images with a `-ubi` or `-ubi9` segment after the runtime version use a Red Hat Universal Base Image (UBI) instead of the default Debian-based image: `-ubi` (UBI 10) for Astro Runtime 3.2 and 3.3, or `-ubi9` (UBI 9) for Astro Runtime 3.0 and 3.1.

<Warning>
  **Runtime version compatibility**

  The Astro Runtime version of your Astro Deployment in the Orchestration Plane must be greater than or equal to the Runtime version of the Remote Execution Agent image. Using a Remote Execution Agent image with a higher Runtime version than your Astro Deployment can cause compatibility issues.
</Warning>

## Agent image selection recommendations

Match the Python version in the image tag to your Airflow project’s Python version for compatibility.

<Info>
  **Selecting and using Sentinel images**

  For Sentinel images, always use the version tag that matches your deployed Remote Execution Agent version. Sentinel images do not have Python version tags. Do not use older or mismatched Sentinel versions with newer Remote Execution Agent releases.

  Deploy the Sentinel image as published by Astronomer without customization or modification. Only customize the Remote Execution Agent image if your pipelines require additional dependencies.
</Info>

## Astro Remote Execution Agent and Sentinel 1.8.4

* Release date: August 20, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-4-python-3.14-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-4-python-3.14-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-4-python-3.13-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-4-python-3.13-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-4-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-4-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-4-ubi-python-3.14-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-4-ubi-python-3.14-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-4-ubi-python-3.13-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-4-ubi-python-3.13-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-4-ubi-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-4-ubi-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.14-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.14-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.13-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.13-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-ubi-python-3.14-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-ubi-python-3.14-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-ubi-python-3.13-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-ubi-python-3.13-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-ubi-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-ubi-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-7-python-3.14-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-7-python-3.14-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-7-python-3.13-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-7-python-3.13-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-7-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-7-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-7-ubi-python-3.14-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-7-ubi-python-3.14-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-7-ubi-python-3.13-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-7-ubi-python-3.13-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-7-ubi-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-7-ubi-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-ubi-python-3.14-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-ubi-python-3.14-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-ubi-python-3.13-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-ubi-python-3.13-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-ubi-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-ubi-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-ubi-python-3.14-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-ubi-python-3.14-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-ubi-python-3.13-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-ubi-python-3.13-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-ubi-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-ubi-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-19-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-19-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-19-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-19-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-19-ubi9-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-19-ubi9-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-19-ubi9-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-19-ubi9-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-ubi9-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-ubi9-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-ubi9-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-ubi9-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-ubi9-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-ubi9-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-ubi9-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-ubi9-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-ubi9-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-ubi9-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-ubi9-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-ubi9-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-ubi9-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-ubi9-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-ubi9-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-ubi9-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-ubi9-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-ubi9-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-ubi9-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-ubi9-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-ubi9-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-ubi9-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-ubi9-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-ubi9-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-ubi9-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-ubi9-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-ubi9-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-ubi9-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-ubi9-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-ubi9-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-ubi9-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-ubi9-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-ubi9-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-ubi9-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-ubi9-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-ubi9-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-ubi9-python-3.12-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-ubi9-python-3.12-astro-agent-1.8.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-ubi9-python-3.11-astro-agent-1.8.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-ubi9-python-3.11-astro-agent-1.8.4-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.8.4`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.8.3

* Release date: July 23, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.14-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.14-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.13-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.13-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.14-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.14-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.13-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.13-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-18-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-18-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-18-python-3.11-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-18-python-3.11-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.8.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.8.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.8.3-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.8.3`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.8.2

* Release date: July 17, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.14-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.14-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.13-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.13-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.14-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.14-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.13-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.13-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.8.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.8.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.8.2-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.8.2`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.8.1

* Release date: July 15, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.14-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.14-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.13-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.13-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.14-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.14-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.13-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.13-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.8.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.8.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.8.1-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.8.1`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.8.0

* Release date: July 15, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.14-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.14-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.13-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.13-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.3-2-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.14-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.14-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.13-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.13-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.8.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.8.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.8.0-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.8.0`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.7.4

* Release date: July 23, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.14-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.14-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.13-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.13-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.7.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.7.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.7.4-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.7.4`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.7.3

* Release date: July 14, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.14-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.14-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.13-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.13-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.7.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.7.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.7.3-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.7.3`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.7.2

* Release date: July 1, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.14-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.14-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.13-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.13-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.7.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.7.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.7.2-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.7.2`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.7.1

* Release date: June 4, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.14-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.14-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.13-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.13-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-5-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.11-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.11-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.11-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.11-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.7.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.7.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.7.1-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.7.1`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.6.2

* Release date: July 23, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.14-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.14-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.13-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.13-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-6-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.6.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.6.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.6.2-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.6.2`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.6.1

* Release date: June 1, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.14-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.13-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-4-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.11-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.11-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.11-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.11-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.6.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.6.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.6.1-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.6.1`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Older versions

* [Versions 1.6.0 through 1.3.2](/docs/astro/agent-images-archive-1)
* [Versions 1.3.1 through 1.0.0](/docs/astro/agent-images-archive-2)


# Remote Execution Agent image reference archive: 1.6.0 through 1.3.2
Source: https://astronomer.io/docs/astro/agent-images-archive-1

Archived reference of images included in Astro Remote Execution Agent releases 1.6.0 through 1.3.2.

Archived image reference for Astro Remote Execution Agent versions 1.6.0 through 1.3.2.

* [Current agent image reference](/docs/astro/agent-images)
* [Older agent images](/docs/astro/agent-images-archive-2)

## Astro Remote Execution Agent and Sentinel 1.6.0

* Release date: April 16, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.14-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.13-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-2-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-1-python-3.14-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-1-python-3.14-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-1-python-3.13-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-1-python-3.13-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-1-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.2-1-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.11-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.11-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.6.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.6.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.6.0-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.6.0`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.5.3

* Release date: July 23, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.5.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.5.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.5.3-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.5.3`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.5.2

* Release date: June 1, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.12-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.12-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.11-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.11-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.12-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.12-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.11-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.11-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.5.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.5.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.5.2-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.5.2`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.5.1

* Release date: April 15, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.12-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.12-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.11-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.11-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.5.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.5.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.5.1-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.5.1`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.5.0

* Release date: March 24, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.12-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.12-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.11-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.11-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.5.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.5.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.5.0-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.5.0`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.3.12

* Release date: July 23, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.12-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-17-python-3.11-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.12-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-16-python-3.11-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.3.12-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.3.12`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.3.12-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.12`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.3.11

* Release date: June 4, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.12-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.12-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.11-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.11-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.12-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.12-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.11-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-15-python-3.11-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.3.11-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.3.11`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.3.11-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.11`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.3.10

* Release date: May 14, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.12-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.12-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.11-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-15-python-3.11-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.3.10-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.3.10`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.3.10-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.10`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.3.9

* Release date: April 16, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.12-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.12-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.11-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.11-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.3.9-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.3.9`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.3.9-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.9`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.3.8

* Release date: March 25, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.12-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.12-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.11-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-14-python-3.11-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.3.8-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.3.8`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.3.8-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.8`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.3.7

* Release date: March 4, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.12-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-13-python-3.11-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.7-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.7`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.7-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.7`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.3.6

* Release date: February 18, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-12-python-3.12-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-12-python-3.12-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-12-python-3.11-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-12-python-3.11-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.6-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.6`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.6-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.6`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.3.5

* Release date: February 13, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-12-python-3.12-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-12-python-3.12-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-12-python-3.11-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-12-python-3.11-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.12-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-14-python-3.11-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.5-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.5`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.5-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.5`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.3.4

* Release date: February 5, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.4-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.4`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.3.3

* Release date: January 30, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.3-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.3`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.3.2

* Release date: January 30, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.12-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-11-python-3.11-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.2-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.2`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>


# Remote Execution Agent image reference archive: 1.3.1 through 1.0.0
Source: https://astronomer.io/docs/astro/agent-images-archive-2

Archived reference of images included in Astro Remote Execution Agent releases 1.3.1 through 1.0.0.

Archived image reference for Astro Remote Execution Agent versions 1.3.1 through 1.0.0.

* [Current agent image reference](/docs/astro/agent-images)

## Astro Remote Execution Agent and Sentinel 1.3.1

* Release date: January 28, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.1-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.1`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.3.0

* Release date: January 28, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.3.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.3.0-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.3.0`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.2.3

* Release date: January 28, 2026

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.2.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.2.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.2.3-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.2.3`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.2.2

* Release date: December 9, 2025

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-6-python-3.12-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-6-python-3.12-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-6-python-3.11-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-6-python-3.11-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-5-python-3.12-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-5-python-3.12-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-5-python-3.11-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-5-python-3.11-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.2.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.2.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.2.2-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.2.2`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.2.1

* Release date: November 17, 2025

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-5-python-3.12-astro-agent-1.2.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-5-python-3.12-astro-agent-1.2.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-5-python-3.11-astro-agent-1.2.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-5-python-3.11-astro-agent-1.2.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.2.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.2.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.2.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.2.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.2.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.2.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.2.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.2.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.2.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.2.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.2.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.2.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.2.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.2.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.2.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.2.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.2.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.2.1-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.2.1`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent and Sentinel 1.2.0

* Release date: November 10, 2025

### Agent images included

The following Astro Agent images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.2.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.2.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.2.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.2.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.2.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.2.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.2.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.2.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.2.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.2.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.2.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.2.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.2.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.2.0-base`

### Sentinel image included

The following Sentinel image is included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-sentinel:1.2.0`

<Note>
  **Sentinel versioning**

  Sentinel images are released alongside Remote Execution Agent images. Always use matching version tags for both components to ensure compatibility.
</Note>

## Astro Remote Execution Agent 1.1.2

* Release date: January 28, 2026

### Images included

The following images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.12-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-9-python-3.11-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.12-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-7-python-3.11-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.1.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.1.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.1.2-base`

## Astro Remote Execution Agent 1.1.1

* Release date: November 17, 2025

### Images included

The following images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-5-python-3.12-astro-agent-1.1.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-5-python-3.12-astro-agent-1.1.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-5-python-3.11-astro-agent-1.1.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-5-python-3.11-astro-agent-1.1.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.1.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.1.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.1.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.11-astro-agent-1.1.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.1.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.1.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.1.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.1.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.1.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.12-astro-agent-1.1.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.1.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-2-python-3.11-astro-agent-1.1.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.1.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.1.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.1.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.1.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.1.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.1.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.1.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.1.1-base`

## Astro Remote Execution Agent 1.1.0

* Release date: September 29, 2025

### Images included

The following images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.1.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.1.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.1.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.11-astro-agent-1.1.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-1-python-3.12-astro-agent-1.1.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-1-python-3.12-astro-agent-1.1.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-1-python-3.11-astro-agent-1.1.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-1-python-3.11-astro-agent-1.1.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.1.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.12-astro-agent-1.1.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.1.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-13-python-3.11-astro-agent-1.1.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-12-python-3.12-astro-agent-1.1.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-12-python-3.12-astro-agent-1.1.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-12-python-3.11-astro-agent-1.1.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-12-python-3.11-astro-agent-1.1.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.1.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.1.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.1.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.1.0-base`

## Astro Remote Execution Agent 1.0.4

* Release date: August 21, 2025

### Images included

The following images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-11-python-3.12-astro-agent-1.0.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-11-python-3.12-astro-agent-1.0.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-11-python-3.11-astro-agent-1.0.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-11-python-3.11-astro-agent-1.0.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-10-python-3.12-astro-agent-1.0.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-10-python-3.12-astro-agent-1.0.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-10-python-3.11-astro-agent-1.0.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-10-python-3.11-astro-agent-1.0.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.0.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.12-astro-agent-1.0.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.0.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-8-python-3.11-astro-agent-1.0.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.0.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.12-astro-agent-1.0.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.0.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-6-python-3.11-astro-agent-1.0.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.12-astro-agent-1.0.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.12-astro-agent-1.0.4-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.11-astro-agent-1.0.4`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.11-astro-agent-1.0.4-base`

## Astro Remote Execution Agent 1.0.3

* Release date: July 15, 2025

### Images included

The following images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-5-python-3.12-astro-agent-1.0.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-5-python-3.12-astro-agent-1.0.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-5-python-3.11-astro-agent-1.0.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-5-python-3.11-astro-agent-1.0.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.12-astro-agent-1.0.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.12-astro-agent-1.0.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.11-astro-agent-1.0.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.11-astro-agent-1.0.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-2-python-3.12-astro-agent-1.0.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-2-python-3.12-astro-agent-1.0.3-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-2-python-3.11-astro-agent-1.0.3`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-2-python-3.11-astro-agent-1.0.3-base`

## Astro Remote Execution Agent 1.0.2

* Release date: July 3, 2025

### Images included

<Warning>
  Due to a known backwards compatibility issue in Airflow, use 3.0-1 images for Remote Deployments on Runtime 3.0-1, 3.0-2 images for Runtime 3.0-2, and 3.0-4 images for Runtime 3.0-3 and 3.0-4
</Warning>

The following images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.12-astro-agent-1.0.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.12-astro-agent-1.0.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.11-astro-agent-1.0.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.11-astro-agent-1.0.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-2-python-3.12-astro-agent-1.0.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-2-python-3.12-astro-agent-1.0.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-2-python-3.11-astro-agent-1.0.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-2-python-3.11-astro-agent-1.0.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-1-python-3.12-astro-agent-1.0.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-1-python-3.12-astro-agent-1.0.2-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-1-python-3.11-astro-agent-1.0.2`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-1-python-3.11-astro-agent-1.0.2-base`

## Astro Remote Execution Agent 1.0.1

* Release date: June 2, 2025

### Images included

<Info>
  Use 3.0-1 images for Remote Deployments on Runtime 3.0-1, and 3.0-2 images for Runtime 3.0-2.
</Info>

The following images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-2-python-3.12-astro-agent-1.0.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-2-python-3.12-astro-agent-1.0.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-2-python-3.11-astro-agent-1.0.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-2-python-3.11-astro-agent-1.0.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-1-python-3.12-astro-agent-1.0.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-1-python-3.12-astro-agent-1.0.1-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-1-python-3.11-astro-agent-1.0.1`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-1-python-3.11-astro-agent-1.0.1-base`

## Astro Remote Execution Agent 1.0.0

* Release date: April 22, 2025

### Images included

The following images are included in this Remote Execution Agent release:

* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-1-python-3.12-astro-agent-1.0.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-1-python-3.12-astro-agent-1.0.0-base`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-1-python-3.11-astro-agent-1.0.0`
* `images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-1-python-3.11-astro-agent-1.0.0-base`


# Remote Execution Agent maintenance policy
Source: https://astronomer.io/docs/astro/agent-maintenance-policy

Review the versioning and 6-month maintenance policy for the Astro Remote Execution Agent.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

## Remote Execution Agent versioning

Astro Remote Execution Agent versions are released regularly and use [semantic versioning](https://semver.org/). Astronomer ships major, minor, and patch releases of the Remote Execution Agent in the format of `major.minor.patch`. Each previous minor version is maintained for **6 months** from its initial `.0` release, during which it receives only critical security (CVE) fixes — not bug fixes. CVE backports use the latest available patch release of Runtime for each Airflow `x.y.z` version.

* **Major** versions are released for significant feature additions. Major versions aren't guaranteed to be backward compatible.
* **Minor** versions are released for functional changes. Minor releases are backward compatible.
* **Patch** versions are released for bug and security fixes that resolve unwanted behavior in the latest minor version only. Patch releases are backward compatible. Previous minor versions in maintenance receive only critical security (CVE) fixes as patch releases.

Bug fixes are delivered through new `minor.patch` versions on the latest minor release. If you report an issue with a maintained Astro Runtime image that isn't on the latest `minor.patch` version, Astronomer support may ask you to upgrade to see if it resolves the issue. Because minor versions are backward compatible, upgrading to the latest minor is the supported path for resolving bugs.

New Astro Runtime release images are built only for the latest minor agent release. When a new Runtime version is released, it is paired with the current latest minor agent version. If the new Runtime requires major functional agent changes, Astronomer releases a new minor agent version to ship with it.

Critical security (CVE) fixes are backported to previous minor versions still within their 6-month maintenance window. CVE backports use the latest available Runtime patch release for each supported Airflow `x.y.z` version.

You can find full information about releases in the [Remote Execution Agent release notes](/docs/astro/agent-release-notes).

### Release scenarios

The following examples illustrate how the policy applies. Assume agent `v1.6.0` on Runtime `3.2-2` with Python 3.14 is the current latest publicly available `minor.patch` version.

| Scenario                                                                       | Release                                                                                                              |
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------- |
| Runtime `3.2-3` is released and doesn't require agent changes                  | Released on agent `v1.6.0`                                                                                           |
| Runtime `3.2-3` is released and doesn't require major functional agent changes | Released on agent `v1.6.1`                                                                                           |
| Runtime `3.3-1` (new minor) is released                                        | Released on agent `v1.7.0`                                                                                           |
| A new Remote Execution feature requires functional agent changes               | Released on agent `v1.7.0`, Runtime `3.2-2`, and Python 3.14 (plus other supported Python versions for that Runtime) |
| Bug fix or non-critical security fix required                                  | Released on agent `v1.6.1`                                                                                           |
| Critical security (CVE) fix required                                           | Released on agent `v1.6.1`, `v1.5.2`, and `v1.3.11`                                                                  |

### Image tag naming conventions

The following table describes the naming conventions for the image tags, allowing you to specify particular versions or allow your environment to use the latest options. Astronomer recommends using a fixed tag, with the versions for the Runtime, Python, and Remote Execution Agent explicitly defined.

| Tag Format                                                         | Description                                                                                           |
| ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| `<runtime_version>-python-<python_version>-astro-agent-x.y.z`      | Recommended configuration format. Fixed tag with specific Remote Execution Agent and Python versions  |
| `<runtime_version>`                                                | Floating tag that pointed to the latest Remote Execution Agent and Python versions. No longer updated |
| `<runtime_version>-python-<python_version>`                        | Floating tag that pointed to the latest Remote Execution Agent versions. No longer updated            |
| `<runtime_version>-base`                                           | Floating tag for the base image without ONBUILD support. No longer updated                            |
| `<runtime_version>-python-<python_version>-base`                   | Floating tag for the base image without ONBUILD support. No longer updated                            |
| `<runtime_version>-python-<python_version>-astro-agent-x.y.z-base` | Fixed tag with specific Remote Execution Agent and Python versions for base image                     |

<Note>
  **Floating tags are frozen at v1.7.2**

  Astronomer no longer updates the floating tags (`<runtime_version>`, `<runtime_version>-python-<python_version>`, and the `-base` variant of each). They stay pinned to the images from Remote Execution Agent `v1.7.2`, the most recent release that published them:

  * `<runtime_version>` and `<runtime_version>-base` point to the Python 3.12 images, `<runtime_version>-python-3.12-astro-agent-1.7.2` and `<runtime_version>-python-3.12-astro-agent-1.7.2-base`.
  * `<runtime_version>-python-<python_version>` and `<runtime_version>-python-<python_version>-base` point to the `v1.7.2` images for that Python version.

  To upgrade, reference a fixed tag with the Runtime, Python, and Remote Execution Agent versions set explicitly, and change it when a new version is available.
</Note>

<Note>Remote Execution Agent images aren't currently compatible with the RHEL platform.</Note>

### Helm chart versioning

As of April 2026, the Remote Execution Agent Helm chart is versioned independently from the Remote Execution Agent. This allows Astronomer to make changes to the Helm chart independently from Remote Execution releases, while still maintaining compatibility with supported agent versions. Decoupling the Helm chart from the Remote Execution Agent release lets Astronomer ship Helm chart bug fixes and enhancements faster.

Each new Helm Chart version will also have an `appVersion` field, populated with the Remote Execution agent version that the chart is shipping with by default.

**What this means for you**:

With this separation between Astro Agent and Helm Chart, you will need to check the compatibility between the Chart's version and the Agent version you are running. See the compatibility matrix below.

### Helm chart to Astro agent compatibility matrix

| Helm Chart Version | Astro Agent Version |
| ------------------ | ------------------- |
| 1.5.x              | ≥ 1.3.0, \< 1.6.0   |
| 2.x                | ≥ 1.5.0             |

For a version-by-version history of what changed in each Helm chart release, see [Helm chart release notes](/docs/astro/agent-helm-chart-release-notes).

<Warning>
  **Breaking change in Helm chart 2.0.0**

  Upgrading from Helm chart 1.5.x to 2.0.0 is a breaking change that requires action before you run `helm upgrade`. See [Helm chart upgrade guide: To 2.0.0](#to-2-0-0) for the required steps.
</Warning>

## Helm chart upgrade guide

This section lists the `values.yaml` file and Kubernetes changes required when upgrading the Remote Execution Agent Helm chart across a major version. If a chart version isn't listed, no special upgrade steps are required beyond the standard `helm upgrade` process. See [Helm chart release notes](/docs/astro/agent-helm-chart-release-notes) for the full history of chart changes.

### To 2.0.0

Upgrading from Helm chart 1.5.x to 2.0.0 is a breaking change that requires action before you run `helm upgrade`.

#### Worker label selectors changed

The label selectors on worker Deployments and Services changed in 2.0.0 to include a per-worker label. Kubernetes label selectors are immutable, so an in-place upgrade fails with the following error:

```text wrap theme={null}
Error: UPGRADE FAILED: Deployment.apps "...-worker-..." is invalid: spec.selector: Invalid value: ... field is immutable
```

Before you upgrade, delete the existing worker Deployments and Services. Replace `<namespace>` with your agent namespace:

```bash wrap theme={null}
kubectl delete deployment -l component=worker -n <namespace>
kubectl delete svc -l component=worker -n <namespace>
```

This causes an interruption of a few seconds to worker pods. Dag processing and triggerer components are unaffected. After deletion, run `helm upgrade` to re-create the worker Deployments and Services with the correct label selectors.

#### Sentinel now enabled by default

`sentinel.enabled` changed from `false` to `true`. If you don't want to run Sentinel, set `sentinel.enabled: false` explicitly in your `values.yaml` file before you upgrade. See [Enable Sentinel monitoring](/docs/astro/remote-agents-sentinel).

#### Null values are no longer accepted for some optional fields

Chart 2.0.0 introduced stricter values validation. Optional fields that default to `~` (YAML null) in chart 1.5.x no longer accept `null` and must be a string, removed, or commented out. This affects the following fields, among others:

* `agentToken`, `agentTokenSecretName`, `agentTokenFile`
* `imagePullSecretName`, `imagePullSecretData`
* `sentinelAuthSecret`, `sentinelAuthSecretName`, `sentinelAuthSecretFile`
* `openLineage.apiKey`, `openLineage.apiKeySecret`, `openLineage.namespace`, `openLineage.url`

If your `values.yaml` file explicitly sets any of these fields to `~`, `helm install` and `helm template` fail against the chart's values schema. Remove the line, comment it out, or set it to a valid value before you upgrade. Fields you never set explicitly aren't affected.

Astronomer recommends downloading the latest `values.yaml` file from the Astro UI and comparing it with your current file before you upgrade.

## Upgrade considerations

The Remote Execution Agent is distributed as a Docker image through the Astro control plane registry, `images.astronomer.cloud`, and includes a Runtime image with multi-Python version support for the Agent and your Dag processor. This allows you to run an image in the Execution Plane with all requirements for running Dag code and the program used by the Agent. It also means that there are three foundational components in your execution plane that can be upgraded to ensure your Remote Execution Agent works with the most up-to-date versions of Airflow and Astro resources:

* Remote Execution Agent version
* Runtime version, in your Orchestration Plane/Astro UI
* Runtime version, in your Execution Plane/Agent image
* Python version

The Remote Execution Agents in the Execution Plane must always use an image with a Runtime version that is less than or equal to the Astro Runtime version in the Orchestration Plane. In general, Astro Runtime versions are backward compatible with Remote Execution Agent versions. If an incompatibility exists, you can find it listed in [Version upgrade considerations](#version-upgrade-considerations).

<Warning>
  Astronomer recommends upgrading your Orchestration plane, your Astro Runtime, and Execution plane, your Remote Execution Agent, separately.

  For upgrading the Execution Plane, Astronomer also recommends upgrading the agent version, the Astro Runtime version, and the Python version individually.
</Warning>

## Remote Execution Agent image upgrade process

1. Update your Astro project Dockerfile with the new version of Astro Remote Execution Agent image.

2. Build your image and publish to your image registry.

3. Update your Remote Execution Agent's Helm `values.yaml` file with the location of your new image in your image registry for the following parameters. To use the same image for all components, specify the image at the top level so all components inherit it. To use different images for different components, for example for each worker queue, specify the image for each component individually to override the top-level image.

   <Tabs>
     <Tab title="Single image">
       ```yaml wrap theme={null}
       image: <image-url>
       ```
     </Tab>

     <Tab title="Per-component image overrides">
       ```yaml wrap theme={null}
       image: <image-url> # Default image
       dagProcessor:
         image: <image-url-override>
       triggerer:
         image: <image-url-override>
       workers:
         - name: ...
           image: <image-url-override>
         - name: ...
           image: <image-url-override>
       ```
     </Tab>
   </Tabs>

4. Run the following `helm` commands to upgrade your installation:

   ```bash wrap theme={null}
   helm repo update
   helm upgrade astro-agent astronomer/astro-remote-execution-agent \
       --namespace astro-agent \
       --values values.yaml
   ```

   <Note>
     To use a specific version of the Remote Execution Agent Helm chart, specify it with the `--version` flag in the `helm upgrade` command. If you don't specify a version, the upgrade uses the latest available chart version after running `helm repo update`.
   </Note>

### Version upgrade considerations

The following sections include upgrade considerations for specific Astro Remote Execution Agent versions. This includes breaking changes, database migrations, incompatibilities, and other considerations.

<Tip>
  If a version isn't included in this section, then there are no specific upgrade considerations for that version.
</Tip>

#### Remote Execution Agent 1.0.0

The [Remote Execution Agent](/docs/astro/remote-execution-configure-agents) based on Airflow 3.0-1 has a known incompatibility with the [Astro Runtime 3.0-2](/docs/runtime/version-upgrade-considerations#runtime-3-0-2). Don't upgrade to or create Remote Deployments that use the combination of an Orchestration Plane version 3.0-2 and Remote Execution Agent version using Runtime 3.0-1.

* [Roll back to previous deploys](/docs/astro/deploy-history#what-happens-during-a-deploy-rollback)
* [Roll back Deployments after a broken upgrade](/docs/astro/best-practices/upgrading-astro-runtime#roll-back-deployments-after-a-broken-upgrade)


# Remote Execution Agent release notes
Source: https://astronomer.io/docs/astro/agent-release-notes

A summary of the latest Remote Execution Agent features and functionality.

<Tip>[Subscribe to Remote Execution Agent release notes](/docs/astro/release-notes-subscribe) to receive updates via RSS, email, or Slack.</Tip>

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

<Info>
  **Remote Execution Agent image release cadence**

  New Remote Execution Agent images are typically released when an Astro Runtime based on a new Airflow version becomes available, or when critical bug or security fixes are required for an existing Runtime or Airflow version.

  You can expect the updated Agent image to be published within **seven days** of the corresponding Runtime image release.
</Info>

<Warning>
  **Helm chart now versioned independently from the Remote Execution Agent**

  As of April 2026, the Remote Execution Agent Helm chart is versioned separately from the Agent, so chart releases no longer appear on this page. Helm chart 2.0.0 introduces a breaking change for upgrades from 1.x that requires action before you run `helm upgrade`. See [Helm chart versioning](/docs/astro/agent-maintenance-policy#helm-chart-versioning) for the compatibility matrix and upgrade steps, and [Helm chart release notes](/docs/astro/agent-helm-chart-release-notes) for the full history of chart changes.
</Warning>

The Astro Remote Execution Agent allows you to run your Airflow tasks remotely in your Kubernetes cluster, connecting to an existing Astro Deployment. This Agent provides the following components:

* Workers: Execute tasks from specified queues.
* Dag processor: Process and sync Dags from configured sources.
* Triggerer: Handle deferrable operators and triggers.

To upgrade Remote Execution Agent, see [Upgrade Remote Execution Agent](/docs/astro/agent-maintenance-policy). For general product release notes, see [Astro Release Notes](/docs/astro/release-notes). If you have any questions or a bug to report, contact [Astronomer support](https://cloud.astronomer.io/open-support-request).

## Known issues

The following known issues apply to all current Remote Execution Agent releases:

* The Dag processor supports reading Airflow variables, but you cannot currently create, update, or delete variables from the Dag processor.

<Update label="Astro Remote Execution Agent and Sentinel 1.8.4" description="August 20, 2026">
  #### Additional improvements

  * Remote Execution Agent images are now also published on Red Hat Universal Base Image (UBI), tagged with a `-ubi` (Astro Runtime 3.2 and 3.3) or `-ubi9` (Astro Runtime 3.0 and 3.1) segment. See [Agent image reference](/docs/astro/agent-images#understanding-image-versions).
  * The sentinel component now exposes a `/metrics` endpoint which includes task instance state metrics in OpenMetrics format.
  * The sentinel component now exposes an internal API which can be used for scale-to-zero worker autoscaling. Integrated support for this will be shipped separately in the Astro Agent helm chart.
  * Remediated multiple high-severity CVEs in the standalone client image.

  #### Bug fixes

  * Fixed a bug where the Dag processor could keep serving a stale parse for a Dag that started failing to import. The agent now evicts the cached parse when a file begins failing to import.

  #### Images included

  See [Astro Remote Execution Agent 1.8.4 images included](/docs/astro/agent-images#astro-remote-execution-agent-and-sentinel-1-8-4).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.8.3" description="July 23, 2026">
  #### Additional improvements

  * Updated bundled dependencies to address security vulnerabilities in `click`.

  #### Images included

  See [Astro Remote Execution Agent 1.8.3 images included](/docs/astro/agent-images#astro-remote-execution-agent-and-sentinel-1-8-3).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.8.2" description="July 17, 2026">
  #### Additional improvements

  * Improved how the worker, Dag processor, and triggerer components communicate with the Astro control plane to better support future Airflow versions.

  #### Bug fixes

  * Fixed a bug that could cause an agent component to send an unnecessary internal header on requests to the Astro control plane.

  #### Images included

  See [Astro Remote Execution Agent 1.8.2 images included](/docs/astro/agent-images#astro-remote-execution-agent-and-sentinel-1-8-2).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.8.1" description="July 15, 2026">
  #### Additional improvements

  * [Task-level utilization metrics](/docs/astro/task-level-metrics) support extends to all supported Remote Execution Agent versions 1.5.0 and later using the Helm 2.0.0+ chart.

  #### Bug fixes

  * Fixed a bug introduced in 1.8.0 where Remote Execution Agents on Astro Runtime versions earlier than 3.3, such as Runtime 3.2, failed to start because the agent incorrectly required a worker-side state store backend that those runtimes don't include. The agent now requires the backend only on Astro Runtime 3.3 and later images.
  * Fixed a bug where the local HTTP proxy in the agent could stop forwarding task heartbeats when an unresponsive upstream connection held a connection-pool slot indefinitely. The proxy client now sets explicit timeouts.

  #### Images included

  See [Astro Remote Execution Agent 1.8.1 images included](/docs/astro/agent-images#astro-remote-execution-agent-and-sentinel-1-8-1).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.8.0" description="July 15, 2026">
  #### Airflow 3.3 support

  Agent Client now supports Airflow 3.3 ([Astro Runtime 3.3-2](/docs/runtime/runtime-release-notes#astro-runtime-3-3-2)), with images for Python 3.12, 3.13, and 3.14. This includes support for the Airflow 3.3 worker-side state store: agents offload task and asset state to a backend that you configure, such as object storage, instead of storing it in the Astro control plane. Asset-watcher triggers running in the agent triggerer can also access the asset state store.

  When you run agents on a Runtime 3.3 image, you must [configure a state store backend](/docs/astro/remote-execution-configure-state-store-backend) through the `AIRFLOW__WORKERS__STATE_STORE_BACKEND` environment variable. Agents on Runtime 3.3 images do not start until you set this variable.

  #### Additional improvements

  * The Dag processor now caches parsed Dags by default and sends only new and changed Dags on each heartbeat, instead of every serialized Dag. Set `ASTRO_AGENT_CLIENT_DAG_PROCESSOR__ENABLE_DAG_CACHING=false` to opt out.
  * Agents now detect DNS changes for the Astro control plane and re-create their connections, so they reconnect automatically after a disaster recovery failover instead of requiring a restart.
  * New Agent Client releases no longer publish floating image tags, such as `3.2-5` or `3.2-5-python-3.13`. Existing floating tags remain in the registry but no longer receive updates. Use the pinned tags listed in the [Remote Execution Agent image reference](/docs/astro/agent-images), as recommended by the [Agent maintenance policy](/docs/astro/agent-maintenance-policy#image-tag-naming-conventions).
  * Where Sentinel is integrated, agent components now report an initializing status while their Pods start up, instead of briefly reporting as unhealthy.
  * Reduced Sentinel memory usage by excluding Kubernetes managed fields from the Pod cache.
  * The control plane now manages the Sentinel heartbeat interval, as it already does for the heartbeat intervals of other agent components.
  * Updated bundled dependencies to address security vulnerabilities in `cryptography`, `pyjwt`, `python-multipart`, and `starlette`.

  #### Bug fixes

  * Fixed a bug where the Dag processor on Airflow versions earlier than 3.3 did not respect configured Dag bundle refresh intervals, so bundles could refresh too often or on another bundle's schedule.
  * Fixed a bug where unpausing a Dag that uses asset watchers did not create the watcher triggers with Dag caching enabled. Dags that use asset watchers are now excluded from Dag caching.
  * Fixed a bug where Dag-level and task-level callbacks that pull XComs failed with `ForbiddenSessionUseError`, which could repeatedly crash the Dag processor on Airflow 3.1. The Dag processor now serves callback XCom reads through the Astro control plane.
  * Fixed a bug where a task that completed within a single metrics-collection interval produced a metrics record that the control plane rejected, blocking the whole metrics batch and causing repeated `Failed to send task metrics` errors.
  * Fixed docstring redaction for task groups: the Dag processor now redacts `doc_md` in addition to `tooltip`, including on task groups nested more than one level deep, before Dags leave the agent.
  * Fixed a bug where the Dag processor could send stale parse results to the control plane after a Dag bundle version update.

  #### Images included

  See [Astro Remote Execution Agent 1.8.0 images included](/docs/astro/agent-images#astro-remote-execution-agent-and-sentinel-1-8-0).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.7.4" description="July 23, 2026">
  #### Additional improvements

  * Updated Agent Client images to include the [3.2-6 Runtime version](/docs/runtime/runtime-release-notes#astro-runtime-3-2-6).
  * Updated bundled dependencies to address security vulnerabilities in `click`.

  #### Images included

  See [Astro Remote Execution Agent 1.7.4 images included](/docs/astro/agent-images#astro-remote-execution-agent-and-sentinel-1-7-4).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.7.3" description="July 14, 2026">
  #### Additional improvements

  * Updated bundled dependencies to address security vulnerabilities in `cryptography`, `pyjwt`, `python-multipart`, and `starlette`.

  #### Bug fixes

  * Fixed a bug where the local HTTP proxy in the agent could stop forwarding task heartbeats when an unresponsive upstream connection held a connection-pool slot indefinitely. The proxy client now sets explicit timeouts.

  #### Images included

  See [Astro Remote Execution Agent 1.7.3 images included](/docs/astro/agent-images#astro-remote-execution-agent-and-sentinel-1-7-3).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.7.2" description="July 1, 2026">
  #### Additional improvements

  * Updated Agent Client images to include the
    [3.0-16](/docs/runtime/runtime-release-notes#astro-runtime-3-0-16)
    and [3.1-17 Runtime
    versions](/docs/runtime/runtime-release-notes#astro-runtime-3-1-17).

  #### Images included

  See [Astro Remote Execution Agent 1.7.2 images included](/docs/astro/agent-images#astro-remote-execution-agent-and-sentinel-1-7-2).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.7.1" description="June 4, 2026">
  #### Additional improvements

  * Added Astro Runtime 3.2-5 to the Agent Client images.
  * Each agent component reports a snapshot of its resolved configuration to the control plane at startup, so you can review the effective settings and how they change over time. Secrets are redacted before the snapshot is sent. Reporting is on by default; set `ASTRO_AGENT_CLIENT_CONFIG_REPORTER_ENABLED=false` to opt out for a component.
  * Removed a blocking call from the worker heartbeat path, reducing task-start latency.
  * The Dag processor heartbeat path now exposes Prometheus metrics, including heartbeat duration, payload size, number of Dags sent, request outcomes, and parse-cache hit rate.
  * The agent now exposes a per-component health gauge on the `/metrics` endpoint, so you can alert on the health of individual components instead of relying only on the Kubernetes probe.
  * Where Sentinel is integrated, the Sentinel dashboard now surfaces Kubernetes Pod events for an unhealthy component, so you can see why a container failed its readiness check.

  #### Bug fixes

  * Fixed a bug where triggers that read connections or variables, for example through `BaseHook.get_connection()` or `Variable.get()`, failed with `ForbiddenSessionUseError`. The Triggerer now proxies connection and variable access through the execution API on all supported runtimes.
  * Fixed Human-in-the-Loop operators on Airflow 3.1 and later, where a deferred trigger could fail authentication when multiple trigger tokens were cached, or fail to parse `datetime` fields in the trigger payload.
  * Fixed a bug where Dag processor callbacks were dropped on Airflow versions before 3.1 because required fields were missing from the heartbeat payload. The agent now backfills these fields.
  * Fixed a bug where a task's reported maximum CPU usage could show an impossibly large value because of a spurious first sample.
  * Where Sentinel is integrated, fixed a bug where a healthy backend could report a misleading message that the agent status was unavailable.

  #### Images included

  See [Astro Remote Execution Agent 1.7.1 images included](/docs/astro/agent-images#astro-remote-execution-agent-and-sentinel-1-7-1).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.6.2" description="July 23, 2026">
  #### Additional improvements

  * Updated Agent Client images to include the [3.2-6 Runtime version](/docs/runtime/runtime-release-notes#astro-runtime-3-2-6).
  * Updated bundled dependencies to address security vulnerabilities in `click`, `cryptography`, `pyjwt`, and `python-multipart`.

  #### Images included

  See [Astro Remote Execution Agent 1.6.2 images included](/docs/astro/agent-images#astro-remote-execution-agent-and-sentinel-1-6-2).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.6.1" description="June 1, 2026">
  #### Additional improvements

  * JWT redaction now applies inside the isolated task subprocess, so trigger tokens and other JWTs are redacted from agent logs.
  * Updated bundled dependencies to address security vulnerabilities in `cryptography`, `gitpython`, `urllib3`, `python-multipart`, and `mako`.

  #### Bug fixes

  * Fixed a bug where an exception in the heartbeater's run loop, such as a subprocess IPC timeout, could prevent the agent from sending its final terminating heartbeat, leaving the control plane unaware of the shutdown.

  #### Images included

  See [Astro Remote Execution Agent 1.6.1 images included](/docs/astro/agent-images#astro-remote-execution-agent-and-sentinel-1-6-1).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.6.0" description="April 16, 2026">
  #### Airflow 3.2 support

  Agent Client now supports Airflow 3.2 (Astro Runtime 3.2-1), including Python 3.13 and 3.14 compatibility.

  #### Additional improvements

  * OpenTelemetry trace contexts are now propagated to Agent Clients.

  #### Bug fixes

  * Fixed a `ValidationError` when processing callbacks with `dag_run.state` on Airflow versions before 3.0.5.
  * Fixed a bug where the worker heartbeater did not report that the agent was terminating during graceful shutdown, so the control plane was not informed that the agent was draining tasks.
  * Fixed shutdown hangs in IPC socket communication where connected clients or pending write flushes could prevent the agent from shutting down.

  #### Images included

  See [Astro Remote Execution Agent 1.6.0 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-6-0).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.5.3" description="July 23, 2026">
  #### Additional improvements

  * Updated bundled dependencies to address security vulnerabilities in `click`, `cryptography`, `pyjwt`, and `python-multipart`.

  #### Images included

  See [Astro Remote Execution Agent 1.5.3 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-5-3).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.5.2" description="June 1, 2026">
  #### Additional improvements

  * JWT redaction now applies inside the isolated task subprocess, so trigger tokens and other JWTs are redacted from agent logs.
  * Bumped the minimum required version of `apache-airflow-providers-git` to 0.3.1, which is needed for compatibility with Airflow 3.1.7 and later.
  * Updated bundled dependencies to address security vulnerabilities in `cryptography`, `gitpython`, `urllib3`, `python-multipart`, and `pyjwt`.

  #### Bug fixes

  * Fixed a bug where an exception in the heartbeater's run loop, such as a subprocess IPC timeout, could prevent the agent from sending its final terminating heartbeat, leaving the control plane unaware of the shutdown.

  #### Images included

  See [Astro Remote Execution Agent 1.5.2 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-5-2).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.5.1" description="April 15, 2026">
  #### Bug fixes

  * Fixed a bug where the worker heartbeater did not report that the agent was terminating during graceful shutdown, so the control plane was not informed that the agent was draining tasks.
  * Fixed shutdown hangs in IPC socket communication where connected clients or pending write flushes could prevent the agent from shutting down.

  #### Images included

  See [Astro Remote Execution Agent 1.5.1 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-5-1).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.5.0" description="March 24, 2026">
  #### XCom support for Triggerer

  Triggers running in the Triggerer component can now read and write XCom values using the execution API. Each trigger receives a per-trigger JWT token for authenticated access, which is automatically refreshed to prevent expiration during long-running triggers.

  #### Additional improvements

  * Added `gc.freeze()` to the Dag processor to reduce garbage collection overhead and improve memory performance.
  * Updated Agent Client images to include the [3.1-14 Runtime version](/docs/runtime/runtime-release-notes#astro-runtime-3-1-14).
  * Unknown settings from configuration sources now produce a warning instead of causing a validation error.
  * The Dag processor and the worker each send a final heartbeat before shutting down.
  * HTTP(S) proxy environment variables are honored for clients that talk to the Astro API server.
  * SIGTERM handling is consolidated across Agent Client components for more predictable shutdowns.
  * Dag file queue handling was reimplemented, with more efficient behavior when Dag bundles refresh.
  * The Triggerer uses a dedicated heartbeater that is decoupled from the Triggerer subprocess lifecycle, improving heartbeat reliability across process restarts.
  * The Dag processor emits additional debug logging and applies a default timeout when sending messages to the coordinator over IPC.
  * `relative_fileloc` is carried in `DagFileParsingResult` and preserved through the Dag processor coordinator.
  * Callback payloads use the `EmailRequest` shape consistently with Airflow’s models.
  * Secrets masking uses imports that match the Airflow 3.0 package layout.
  * Where Sentinel is integrated, the client supports agent ID tracking and container restart detection.

  #### Bug fixes

  * Fixed a bug where failed triggers could remain in the running set and cause repeated cancel loops.
  * Fixed trigger handling when the task instance was null or missing (including skipping triggers with no task instance on Airflow 3.0 clients).
  * Fixed Triggerer coordinator process termination so subprocesses shut down as intended.
  * Fixed graceful shutdown behavior in edge cases.
  * Fixed the log level used when a process has finished.
  * Fixed email-related callback typing when values are built from API responses.
  * Fixed `DagCallback` behavior on older Airflow versions used with the agent.
  * Fixed a bug where `DagModel.bundle_version` did not update for cached DAGs after bundle changes.
  * Fixed a compatibility issue in the Dag processor where the wrong method was called for adding new files on Airflow 3.0 and 3.1 vs 3.2.
  * Fixed a bug where the Triggerer failed with a `LimitOverrunError` when processing large IPC messages.
  * Fixed a bug where non-serializable trigger event payloads caused Triggerer heartbeat failures.
  * Fixed `Stats` import compatibility across different Airflow 3 versions.
  * Fixed a bug where the Dag processor did not exit on fatal exceptions.
  * Fixed a case where the managed Astro subprocess had already exited but the agent could linger; the agent now exits promptly and cleanly.
  * Fixed uncaught errors in socket listener callbacks so a bad callback does not tear down the listener.
  * Fixed client-side stripping of `triggering_user_name` from callbacks for Airflow below 3.1.2.

  #### Images included

  See [Astro Remote Execution Agent 1.5.0 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-5-0).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.4.0">
  Internal-only release.
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.12" description="July 23, 2026">
  #### Additional improvements

  * Updated bundled dependencies to address security vulnerabilities in `click`, `cryptography`, `pyjwt`, and `python-multipart`.

  #### Images included

  See [Astro Remote Execution Agent 1.3.12 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-3-12).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.11" description="June 4, 2026">
  #### Additional improvements

  * Updated bundled dependencies to address security vulnerabilities in `cryptography`, `gitpython`, `urllib3`, `python-multipart`, `mako`, and `pyjwt`.

  #### Images included

  See [Astro Remote Execution Agent 1.3.11 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-3-11).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.10" description="May 14, 2026">
  #### Additional improvements

  * Bumped the minimum required version of `apache-airflow-providers-git` to 0.3.1, which is needed for compatibility with Airflow 3.1.7 and later.

  #### Bug fixes

  * Fixed a bug where an exception in the heartbeater run loop caused shutdown to skip the final terminating heartbeat, so the control plane wasn't informed that the agent had stopped.

  #### Images included

  See [Astro Remote Execution Agent 1.3.10 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-3-10).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.9" description="April 16, 2026">
  #### Bug fixes

  * Fixed a bug where the worker heartbeater did not report that the agent was terminating during graceful shutdown, so the control plane was not informed that the agent was draining tasks.
  * Fixed shutdown hangs in IPC socket communication where connected clients or pending write flushes could prevent the agent from shutting down.

  #### Images included

  See [Astro Remote Execution Agent 1.3.9 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-3-9).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.8" description="March 25, 2026">
  #### Bug fixes

  * Fixed a bug where the Triggerer failed with a `LimitOverrunError` when processing large IPC messages.

  #### Images included

  See [Astro Remote Execution Agent 1.3.8 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-3-8).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.7" description="March 4, 2026">
  #### Bug fixes

  * Fixed a bug where the Triggerer agent did not exit when receiving a `409 Conflict from the API Server`, which prevented it heartbeating successfully.

  #### Images included

  See [Astro Remote Execution Agent 1.3.7 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-3-7).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.6" description="February 18, 2026">
  #### Improvements

  * Small performance and telemetry enhancements to the remote Dag Processor agent.

  #### Images included

  See [Astro Remote Execution Agent 1.3.6 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-3-6).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.5" description="February 13, 2026">
  #### Bug fixes

  * Fixed a bug where agents shutting down didn't always correctly notify the Astro APIs, which could result in new containers being unable to register.
  * Fixed a bug where the remote Dag Processor agent wouldn't crash after a fatal exception. (In this case crashing was expected and required to allow the agent to re-register successfully.)

  #### Images included

  See [Astro Remote Execution Agent 1.3.5 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-3-5).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.4" description="February 5, 2026">
  #### Bug fixes

  * Fixed a bug where agents don't shutdown gracefully on SIGTERM. Worker agents now wait for running tasks to finish before shutting down or until the Pod's termination grace period expires. This fix addresses issues with running tasks being killed when an agent is terminated.

  #### Images included

  See [Astro Remote Execution Agent 1.3.4 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-3-4).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.3" description="January 30, 2026">
  <Warning>
    Agent Client versions 1.3.2 and 1.3.3 don't handle SIGTERM gracefully. Running tasks may fail when terminating agents on these versions. This issue is fixed in Agent Client version 1.3.4, which is included in the Astro Remote Execution Agent and Sentinel 1.3.4 release.
  </Warning>

  #### Bug fixes

  * Fixed a bug that could prevent Dag import errors from being cleared when fixed, affecting Airflow ≥ 3.1.6.

  #### Images included

  See [Astro Remote Execution Agent 1.3.3 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-3-3).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.2" description="January 30, 2026">
  <Warning>
    Agent Client versions 1.3.2 and 1.3.3 don't handle SIGTERM gracefully. Running tasks may fail when terminating agents on these versions. This issue is fixed in Agent Client version 1.3.4, which is included in the Astro Remote Execution Agent and Sentinel 1.3.4 release.
  </Warning>

  #### Proxy server support

  Agent Client now supports running behind an HTTP(S) proxy server. You can configure proxy settings using the usual `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` environment variables.

  #### Additional improvements

  * Updated Agent Client images to use the [3.1-11 Runtime version](/docs/runtime/runtime-release-notes#astro-runtime-3-1-11).

  #### Bug fixes

  * Fixed a bug where agents wouldn't terminate correctly if interrupted during initialization.

  #### Images included

  See [Astro Remote Execution Agent 1.3.2 images included](/docs/astro/agent-images-archive-1#astro-remote-execution-agent-and-sentinel-1-3-2).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.1" description="January 28, 2026">
  #### Additional improvements

  * Added support for Asset Watcher and Deadline Alert triggers in Airflow 3.1.

  #### Bug fixes

  * Fixed a bug where failed triggers weren't correctly removed from the list of triggers running in a triggerer agent.
  * Fixed a bug where the same trigger could be started multiple times in the same triggerer agent.
  * Fixed a bug affecting execution of callbacks on the Dag Processor in Airflow ≥ 3.1.1.
  * Fixed some nil pointer reference bugs in the Helm chart templates.
  * Fixed an issue where environment variables provided in `commonEnv` weren't able to override the defaults in the Helm chart.

  #### Images included

  See [Astro Remote Execution Agent 1.3.1 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-and-sentinel-1-3-1).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.3.0" description="January 28, 2026">
  #### Bug fixes

  * Fixed a bug where an agent could get stuck in a "cordoned" state indefinitely when temporarily partitioned from the API server. Agents in this state will now detect the situation and recover automatically.

  #### Images included

  See [Astro Remote Execution Agent 1.3.0 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-and-sentinel-1-3-0).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.2.3" description="January 28, 2026">
  #### Additional improvements

  * Added support for Asset Watcher and Deadline Alert triggers in Airflow 3.1.
  * Added support for setting default values for `image` and `imagePullPolicy` that apply to all Agent components in the Helm chart.

  #### Bug fixes

  * Fixed a bug where failed triggers weren't correctly removed from the list of triggers running in a triggerer agent.
  * Fixed a bug where the same trigger could be started multiple times in the same triggerer agent.
  * Fixed a bug affecting execution of callbacks on the Dag Processor in Airflow ≥ 3.1.1.
  * Fixed some nil pointer reference bugs in the Helm chart templates.
  * Fixed an issue where environment variables provided in `commonEnv` weren't able to override the defaults in the Helm chart.

  #### Images included

  See [Astro Remote Execution Agent 1.2.3 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-and-sentinel-1-2-3).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.2.2" description="December 9, 2025">
  #### Additional improvements

  * Moved vector logging sidecar to be part of the init container, so that it keeps running throughout the lifecycle of the Pod and doesn't terminate before the main Agent component container.
  * Added support for configuring `imagePullPolicy` in the Helm chart.
  * Updated Agent Client images to use the [3.1-7 Runtime version](/docs/runtime/runtime-release-notes#astro-runtime-3-1-7).

  #### Bug fixes

  * Fixed a bug in the Helm chart that prevented Worker component resources from being created correctly when multiple workers were configured.
  * Don't specify `worker.replicas` when `worker.hpa.enabled` is set to true in the Helm chart.
  * Fixed typo with using lowercase key `value` as part of `worker.hpa.metrics` in the Helm chart.
  * Report both running and creating triggers in the `running_triggers` part of the triggerer response to prevent starting a trigger with a given ID if it is already running or creating.

  #### Images included

  See [Astro Remote Execution Agent 1.2.2 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-and-sentinel-1-2-2).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.2.1" description="November 17, 2025">
  #### Additional improvements

  * Updated Agent Client images to use the [3.1-4 Runtime version](/docs/runtime/runtime-release-notes#astro-runtime-3-1-4).

  #### Bug fixes

  * Fixed incorrect timestamp values for Dag parsing stats in the Airflow UI.
  * Fixed structlog for Agent logging when using 3.0-x Runtime versions.

  #### Images included

  See [Astro Remote Execution Agent 1.2.1 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-and-sentinel-1-2-1).
</Update>

<Update label="Astro Remote Execution Agent and Sentinel 1.2.0" description="November 10, 2025">
  #### Added Sentinel to Remote Execution Agent Helm chart

  Added a new Sentinel service to Remote Execution Agent Helm chart. Sentinel helps collect monitoring datapoints such as Pod, XCom backend and secrets backend statuses and reports them back to API Server. This is an experimental feature to help with Remote Execution Agent monitoring and is turned off by default in the Agent Helm chart. See [Enable Sentinel for monitoring](/docs/astro/remote-agents-sentinel).

  #### Additional improvements

  * Added a check in the Agent Client to avoid breaking the Remote Deployment by running into forward compatibility issues. For example, Agent Client will get back 422 HTTP response code instead of regular heartbeat response if the Airflow version of the Agent Client is higher than that of the Astro Remote Deployment.
  * Added support to configure `imagePullPolicy` in the Remote Agent Helm chart.

  #### Images included

  See [Astro Remote Execution Agent 1.2.0 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-and-sentinel-1-2-0).
</Update>

<Update label="Astro Remote Execution Agent 1.1.2" description="January 28, 2026">
  #### Additional improvements

  * Added support for Asset Watcher and Deadline Alert triggers in Airflow 3.1.
  * Moved vector logging sidecar to be part of the init container, so that it keeps running throughout the lifecycle of the Pod and doesn't terminate before the main Agent component container.
  * Added support for configuring `imagePullPolicy` in the Helm chart.
  * Added support for setting default values for `image` and `imagePullPolicy` that apply to all Agent components in the Helm chart.

  #### Bug fixes

  * Fixed a bug where failed triggers weren't correctly removed from the list of triggers running in a triggerer agent.
  * Fixed a bug where the same trigger could be started multiple times in the same triggerer agent.
  * Fixed a bug affecting execution of callbacks on the Dag Processor in Airflow ≥ 3.1.1.
  * Fixed a bug in the Helm chart that prevented Worker component resources from being created correctly when multiple workers were configured.
  * Don't specify `worker.replicas` when `worker.hpa.enabled` is set to true in the Helm chart.
  * Fixed typo with using lowercase key `value` as part of `worker.hpa.metrics` in the Helm chart.
  * Fixed some nil pointer reference bugs in the Helm chart templates.
  * Fixed an issue where environment variables provided in `commonEnv` weren't able to override the defaults in the Helm chart.

  #### Images included

  See [Astro Remote Execution Agent 1.1.2 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-1-1-2).
</Update>

<Update label="Astro Remote Execution Agent 1.1.1" description="November 17, 2025">
  #### Additional improvements

  * Updated Agent Client images to use the [3.1-4 Runtime version](/docs/runtime/runtime-release-notes#astro-runtime-3-1-4).

  #### Images included

  See [Astro Remote Execution Agent 1.1.1 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-1-1-1).
</Update>

<Update label="Astro Remote Execution Agent 1.1.0" description="September 29, 2025">
  #### Additional improvements

  * Updated Agent Client images to use the [3.1-1 Runtime version](/docs/runtime/runtime-release-notes#astro-runtime-3-1-1).
  * Added support to directly reference the Agent token file in the Helm chart.
  * Added the ability to configure a custom service account and corresponding annotation per Agent component in the Helm chart.

  #### Remote Agent Worker Kubernetes service account change

  The naming convention for the Remote Agent Worker's Kubernetes service account has been updated. Users who have configured IRSA on AWS, Workload Identity on Azure, or GCP will need to annotate the new service account appropriately. In previous releases, the service account name for the Remote Execution Agent worker was `{{ resourceNamePrefix }}-worker`. The default service account name has been updated to `{{ resourceNamePrefix }}-worker-{{ worker.name }}`. `resourceNamePrefix` and `worker.name` are both attributes set in the `values.yaml` file. You can also provide a custom service account, which overrides the default service account name. The custom service account name applies to all workers.

  #### Bug fixes

  * Fixed a bug where the Dag processor process didn't close cleanly on completion.
  * Fixed a bug where the Agent process crashed in the event of heartbeat failures.

  #### Images included

  See [Astro Remote Execution Agent 1.1.0 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-1-1-0).
</Update>

<Update label="Astro Remote Execution Agent 1.0.4" description="August 21, 2025">
  #### Additional improvements

  * Updated Agent Client images to use the [3.0-8 Runtime version](/docs/runtime/runtime-release-notes#astro-runtime-3-0-8).

  #### Bug fixes

  * Fixed live task logs for Hosted Astro executor Deployments.

  #### Images included

  See [Astro Remote Execution Agent 1.0.4 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-1-0-4).
</Update>

<Update label="Astro Remote Execution Agent 1.0.3" description="July 15, 2025">
  #### Prevent `airflow_local_settings.py` Execution in Agent Processes

  The Remote Execution Agent process no longer imports `airflow_local_settings.py`. This prevents errors from user-defined settings that rely on dependencies not available in the Remote Execution Agent's isolated environment.

  #### Images included

  See [Astro Remote Execution Agent 1.0.3 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-1-0-3).
</Update>

<Update label="Astro Remote Execution Agent 1.0.2" description="July 3, 2025">
  #### Support for Custom Service Accounts and Annotations

  The Remote Execution Agent Helm chart now supports setting custom service account names and annotations for the Worker, Triggerer, and Dag processor components. This allows you to:

  * Bring your own service accounts (BYO-SA) and disable the Helm-managed service account creation
  * Add annotations, such as for AWS IRSA

  This update supports restricted or self-managed environments where service accounts and roles are centrally controlled.

  #### Bug fixes

  * Changed working directory for Remote Execution Agent images to `/usr/local/airflow`
    The workdir for Remote Execution Agent images has moved from `/opt/astro` to `/usr/local/airflow` to align with Runtime images. If you have added a manual `WORKDIR /usr/local/airflow` directive in your image builds you can now remove this if building from the 1.0.2 base images.

  <Warning>
    **Breaking Change**

    If you reference `/opt/astro` in your Dockerfile for a Remote Execution mode Deployment, you must update those references to `/usr/local/airflow`.
  </Warning>

  #### Images included

  See [Astro Remote Execution Agent 1.0.2 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-1-0-2).

  <Warning>
    Due to a known backwards compatibility issue in Airflow, use 3.0-1 images for Remote Deployments on Runtime 3.0-1, 3.0-2 images for Runtime 3.0-2, and 3.0-4 images for Runtime 3.0-3 and 3.0-4
  </Warning>
</Update>

<Update label="Astro Remote Execution Agent 1.0.1" description="June 2, 2025">
  #### Bug fixes

  * Fixed a bug where the proxy rejected HEAD requests to the XCom API with a `405 Method Not Allowed`. It now correctly supports HEAD requests.

  #### Images included

  See [Astro Remote Execution Agent 1.0.1 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-1-0-1).

  <Warning>
    Use 3.0-1 Agent images for Remote Deployments using the Astro Runtime/Orchestration Plane 3.0-1 due to an OSS Airflow Bug. You can use 3.0-2 Remote Agent images for any Astro Runtime ≥ 3.0-2.

    See [Upgrade considerations](/docs/astro/agent-maintenance-policy#upgrade-considerations) for more information about the Remote Execution Agent and Astro Runtime version considerations.
  </Warning>
</Update>

<Update label="Astro Remote Execution Agent 1.0.0" description="April 22, 2025">
  #### Introducing the Remote Execution Agent

  With the release of Airflow 3.0, Astro now supports Remote Execution of Dags with the **Astro Remote Execution Agent**. The Remote Execution Agent allows you to keep your sensitive data and execute your Dags in your own environment, while still leveraging Astro features like day zero Airflow Runtime support, observability, and user management. The Astro Remote Execution Agent manages running the Workers, Triggerer, and Dag processors in your environment, while the Airflow Metadata Database, Scheduler, Web/API Server, and Remote Execution API remain in the Astro Orchestration plane.

  See [Remote Execution Agents](/docs/astro/remote-execution-configure-agents) for installation, configuration, and advanced options.

  #### Images included

  See [Astro Remote Execution Agent 1.0.0 images included](/docs/astro/agent-images-archive-2#astro-remote-execution-agent-1-0-0).

  <Warning>
    Due to a known backwards compatibility issue in Airflow, use 3.0-1 images for Remote Deployments on Runtime 3.0-1, 3.0-2 images for Runtime 3.0-2, and 3.0-4 images for Runtime 3.0-3 and 3.0-4
  </Warning>
</Update>


# Apache Airflow feature support
Source: https://astronomer.io/docs/astro/airflow-feature-support

Astronomer's support policy for experimental and unsupported Apache Airflow features on Astro.

Apache Airflow releases features at varying levels of maturity. Some features ship as experimental in Airflow itself, and a small set of upstream features aren't supported on Astro. This document explains how Astronomer supports experimental Airflow features and lists the upstream Airflow features that Astro doesn't support.

For the lifecycle policy that applies to Astronomer-released products and features, see [Astronomer feature lifecycle](/docs/astro/feature-previews).

## Experimental Airflow features

Apache Airflow marks some features as experimental. These features ship in Airflow releases, but their interfaces, behaviors, and stability characteristics can change between versions without notice.

Astronomer provides best-effort support for experimental Airflow features on Astro. Best-effort support means that Astronomer support investigates issues and helps you find workarounds where possible. SLAs don't apply to experimental features, and some issues require an upstream fix in Apache Airflow before Astronomer can resolve them.

## Unsupported Airflow features

The following upstream Apache Airflow features aren't supported on Astro:

<Warning>
  This list isn't exhaustive and changes as Apache Airflow evolves. Before you adopt a new Airflow feature in a production Deployment, contact Astronomer support to confirm its support status on Astro.
</Warning>

* **Edge executor**: The Apache Airflow Edge executor isn't supported on Astro. For distributed task execution outside the Astro data plane, use [Remote Execution Agents](/docs/astro/remote-execution-overview).
* **Airflow multi-team**: The Airflow 3 multi-team model isn't supported on Astro. Astro provides tenant isolation through Organizations, Workspaces, and Deployments. See [About Astro](/docs/astro/astro-architecture).
* **Auth managers**: Apache Airflow auth managers aren't supported on Astro. This includes the simple auth manager, the FAB auth manager, and any custom or third-party auth managers. Astro uses its own authentication and authorization system. See [Organization users](/docs/astro/manage-organization-users) and [Set up SSO](/docs/astro/configure-idp).

If you have questions about a specific Airflow feature, [submit a support request](/docs/astro/astro-support).


# Airflow 3 features
Source: https://astronomer.io/docs/astro/airflow3/features-af3

Features of Airflow 3

## Remote Execution on Astro

Remote Execution in Airflow 3 allows tasks to run securely in customer-managed environments without opening inbound connections. Only essential scheduling and health data leave the execution plane.

[Remote Execution Agents](/docs/astro/remote-execution-configure-agents) on Astro enable tasks to run in user-managed hardware or private clouds with only outbound connections to Astro’s Orchestration Plane. Sensitive data stays local, ideal for regulated or multi-regional deployments.

Learn more about [Execution modes](/docs/astro/execution-mode) on Astro.

## Dag Versioning

Airflow 3’s [Dag Versioning](/docs/astro/dag-versioning) ensures that each pipeline run references its exact code snapshot, enabling complete historical traceability. Teams can rapidly audit, compare with and debug by viewing the historic Dag code/structure of each run, eliminating confusion and accelerating compliance checks.

## Backfills

Backfills solve one of the most common and time-consuming challenges in data orchestration: reliably reprocessing historical or newly available data. Previously, backfills in Airflow had to be triggered from a command-line process that could easily terminate if the session was lost, leaving longer reruns vulnerable to interruption and without robust monitoring. In Airflow 3, backfills become first-class citizens managed by the scheduler itself, enabling asynchronous API triggers, real-time monitoring through the UI, and the ability to pause or cancel jobs mid-run. This unified approach not only saves teams from manual scripting and fragile workarounds, but it also gives them confidence that large-scale historical recalculations—often critical for machine learning retraining and data integrity checks—will run consistently, even if they take hours or days to complete. To learn more about using backfills, see [Rerun Dags and tasks guide](/docs/learn/rerunning-dags#backfill).

## UI Modernization

Airflow 3 introduces a modern, React-based UI that unifies logs, task details, and dynamic Dag updates in a clean, intuitive interface. To learn more, see [Airflow UI guide](/docs/learn/airflow-ui).

## Event-driven Scheduling

Event-driven scheduling in Airflow 3 lets pipelines react to near real-time data changes or external triggers, rather than relying solely on fixed time-based schedules. This means a Dag can automatically start running as soon as a message arrives in the message queue of a supported service. By removing the need for constant polling or hard-coded cron schedules, event-driven pipelines can process data the instant it arrives. This not only saves resources and shortens end-to-end processing time, but also enables more dynamic, near–real-time workflows that are crucial for modern data science, streaming analytics, and AI/ML applications. To learn how to implement it, see [the event-driven scheduling guide](/docs/learn/airflow-event-driven-scheduling).

## Inference Execution

Airflow 3.0 introduces several enhancements to support AI Inference Execution:

* Ad-hoc scheduling: Airflow 3.0 allows Dags to be run independently of any data interval, which is crucial for supporting inference execution. This feature enables on-demand execution of inference tasks without being constrained by predefined schedules.
* Synchronous Dag execution: The new version supports simultaneous execution of the same dag, allowing for synchronous inference runs. This is particularly useful for scenarios where multiple inference requests need to be processed concurrently.
* API-triggered execution: Airflow 3.0 introduces the ability to trigger Dags via API calls, enabling multiple instances to be initiated simultaneously for inference tasks. This feature facilitates experimentation and allows for dynamic, near  real-time inference processing.
* Event-driven scheduling: The new version supports automatic triggering of Dags based on external events or data availability. This can be particularly useful for inference pipelines that need to react to new data or model updates in near real-time.
* Language-agnostic Task Execution Interface: Airflow 3.x lays the groundwork to run tasks in any language. This enables users to implement inference tasks in the most suitable language for their models, without expensive code refactoring such as using C++, Golang, Java, etc. for more efficient execution.

Collectively, these enhancements make Airflow 3 more capable of handling diverse inference scenarios, from batch processing to on-demand execution, while offering improved flexibility and performance for AI and ML workflows.


# Upgrade an Astro project to Airflow 3
Source: https://astronomer.io/docs/astro/airflow3/upgrade-af3

Learn how to upgrade your Astro project from Apache Airflow 2 to Apache Airflow 3.

This guide provides steps to upgrade your Astro project from Apache Airflow 2 to Apache Airflow 3.

<Tip>For more information on breaking changes between Airflow 2 and Airflow 3, as well as configuration linting, see the [Upgrade from Apache Airflow® 2 to 3](/docs/learn/airflow-upgrade-2-3) Learn guide and the [Airflow 3 Release Notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html).</Tip>

<Warning>
  If you are upgrading an existing Deployment in place (see [Strategy B](#strategy-b)), the following requirements apply:

  * Your Deployment must first run Astro Runtime 13.7.0 or above.
  * The upgrade target must be the latest available patch of a supported Airflow 3 minor version (for example, `3.1-{latest}` or `3.2-{latest}`). An older 3.x patch is allowed only if another Deployment in your Organization is already running it. For example, if one Deployment in your Organization is on `3.1-3`, others can also upgrade to `3.1-3` even after `3.1-4` has shipped. Otherwise, the upgrade is rejected with: *"… is not available as an upgrade target. Astro Runtime 3 upgrades are limited to the latest patch version for each minor version, or a version already used within your organization."*
  * No metadata database table can exceed 50 GB. If any table is over the limit, run the [metadata database cleanup Dag](/docs/learn/2.x/cleanup-dag-tutorial) to reduce table sizes, or contact [Astro support](/docs/astro/astro-support) for assistance.
  * Any Dag runs and task instances in a non-terminal state when the upgrade begins will be failed by the migration. An audit log entry is recorded for each failed run.
</Warning>

## Prerequisites

* An Astro account with [user permissions](/docs/astro/user-permissions) to create a new Deployment, such as **Workspace Operator** or **Deployment Admin**
* An existing Astro Deployment running Airflow 2.x
* [Astro CLI >=1.34.0](/docs/cli/v1.43/release-notes) installed on your local computer
* Your Astro project folder

## Step 1: Prepare your local project for Airflow 3

Before upgrading to Airflow 3, you must test your Dags and dependencies locally to identify and resolve any compatibility issues.

<Warning>
  **Breaking change**

  Astro Runtime for 3.0 and higher includes only the [provider packages](/docs/runtime/runtime-release-notes#astro-runtime-3-0-1) required to run on Astro. Previous versions contained additional providers, and if you are using those providers you must explicitly add them to your `requirements.txt`.
</Warning>

Follow these steps to prepare your local project:

### Upgrade test your local project

Test your project locally to identify any compatibility issues:

```sh wrap theme={null}
astro dev upgrade-test
```

Use the results of this test to identify compatibility issues. Address any compatibility issues by updating your Dags, custom operators, and dependencies. Common changes include:

* Updating import paths that have changed in Airflow 3
* Replacing deprecated features with their recommended alternatives
* Updating Airflow providers to versions compatible with Airflow 3 — see [minimum provider versions required for Airflow 3](https://github.com/apache/airflow/blob/main/pyproject.toml#L71)

### Run unit tests

In your local Astro project, update the `Dockerfile` to use an Airflow 3 image:

```dockerfile title="Dockerfile" wrap theme={null}
FROM astrocrpublic.azurecr.io/runtime:<astro-runtime-version>
```

Next, run the following to execute any unit tests:

```sh wrap theme={null}
astro dev pytest
```

For more information, see [testing your Astro project locally](/docs/cli/v1.43/test-your-astro-project-locally).

<Info>Replace `<astro-runtime-version>` with the specific Astro Runtime version that includes Airflow 3.x that you want to use.</Info>

### Run your project locally

For a final check and to test Dags manually, start Airflow locally:

```sh wrap theme={null}
astro dev start
```

Refer to the docs on [local Airflow development](/docs/cli/v1.43/run-airflow-locally) to make and test any changes.

## Step 2: Choose your upgrade strategy

There are two supported upgrade strategies for Airflow 3. Choose the approach that fits your operational needs and workflow requirements.

### Strategy A

1. Create a new Airflow 3 Deployment in Astro.
2. Update your local project to use an Airflow 3 image.
3. Deploy your updated project to the new Deployment.

**Pros:**

* Lower affected area; you can test your Dags without affecting production.
* Gradual, controlled migration of workflows.
* Easy rollback; you can switch back to your Airflow 2 Deployment until decommissioned.

**Cons:**

* No migration of historical task data.
* Slightly higher resource usage and cost while running both Deployments during migration.

To continue with this strategy, proceed to Step 3A.

### Strategy B

1. Prepare and test your Dags on Airflow 3 locally.
2. Make an Airflow 3 deploy against your Airflow 2 Deployment, initiating an upgrade.

**Pros:**

* Preserves historical task data and metadata.
* No need for extra or temporary Deployments.

**Cons:**

* Higher affected area; all Dags must be compatible at once.
* Immediate production impact if issues arise.
* Any in-flight dag runs and task instances will be failed when the upgrade begins.
* Rollbacks to Airflow 2 require specific Runtime versions. See [Airflow 3 to Airflow 2 rollback support and requirements](#airflow-3-to-airflow-2-rollback-support-and-requirements)

<Warning>
  Astronomer recommends running the [metadata database cleanup Dag](/docs/learn/2.x/cleanup-dag-tutorial) before you upgrade. In-place upgrades require a database migration that can take several hours for Deployments with large amounts of task history. Cleaning up old metadata before the upgrade significantly reduces migration time and the risk of issues during the upgrade process.
</Warning>

To continue with this strategy, move to Step 3B.

## Step 3A: Upgrade to Airflow 3 with a new Deployment

### 1. Create a new Airflow 3 Deployment in the Astro UI

See [Create a Deployment](/docs/astro/create-deployment) for more details.

### 2. Deploy to your new Airflow 3 Deployment

Deploy your updated project:

```sh wrap theme={null}
astro deploy <your-new-deployment-id>
```

For additional customization, refer to docs on [deploying Astro project images](/docs/astro/deploy-project-image).

Verify that your Dags are running correctly in the new environment.

### 3. Migrate your workflows

1. Update any CI/CD pipelines to deploy to your new Airflow 3 Deployment.
2. Redirect any integrations or external systems to use your new Deployment.
3. After you've confirmed everything is working correctly, you can decommission your Airflow 2 Deployment.

## Step 3B: Upgrade an existing Deployment to Airflow 3

### Upgrade your Airflow 2 Deployment to Airflow 3

<Warning>
  Rolling back from Airflow 3 to Airflow 2 is supported but has specific Runtime version requirements:

  * Your Deployment was on Astro Runtime 13.7.0 or above before upgrading.
  * Your Deployment is on Astro Runtime 3.0-11 or later after upgrading to Airflow 3.

  Review the [Airflow 3 to Airflow 2 rollback support and requirements](#airflow-3-to-airflow-2-rollback-support-and-requirements) before proceeding with this upgrade.
</Warning>

<Warning>
  In-place upgrades require that no metadata database table exceed 50 GB. If any table is over the limit, run the [metadata database cleanup Dag](/docs/learn/2.x/cleanup-dag-tutorial) to reduce table sizes, or contact [Astro support](/docs/astro/astro-support) for assistance.

  Even when tables are under the limit, upgrades can take several hours for Deployments with a large amount of task history. Astronomer recommends cleaning up old metadata before the upgrade to further reduce migration time and risk. Contact [Astro support](/docs/astro/astro-support) if you have concerns or questions about the progress of the upgrade.
</Warning>

<Warning>
  Any Dag runs and task instances in a non-terminal state when the upgrade begins will be failed by the migration. An audit log entry is recorded for each failed run so you can identify impacted Dags after the upgrade. Plan your upgrade window to minimize impact, and pause any Dags that should not be interrupted.
</Warning>

Deploy your updated project:

```sh wrap theme={null}
astro deploy
```

## Step 4: Update your workload identity policy bindings

<Warning>
  This step is applicable for Deployments configured with Customer Managed Workload identity.
</Warning>

* Update the Workload Identity configuration on the Deployment Details page. Re-enter the same IAM role (ARN/service account email/service principal) you were previously using. This step ensures your IAM role bindings are updated from the Webserver service account to the newly added API Server service account in Airflow 3, which is referenced in the upcoming setup script.
* When prompted, provide your IAM role ARN/service account email/service principal details. Copy and run the CLI command that is generated. After running it, click **Save Configuration** to store the IAM role so it can be reused as a selectable configuration.

## Troubleshoot common issues

### Pip package resolution

With Runtime 3.0, Python package resolution has switched by default from pip to [uv](https://docs.astral.sh/uv/), which is intended as a fast [drop in replacement](https://docs.astral.sh/uv/pip/compatibility/) for pip. However, uv has a different behavior for packages that exist on [multiple indexes](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes), because pip's behavior is unsafe against dependency confusion attacks. If you rely on the pip behavior, you can add `# ASTRO_RUNTIME_USE_PIP` to your `requirements.txt` file to use pip instead of uv. You can also opt to install your dependencies in a separate `RUN` line on your Dockerfile instead of using Runtime's built-in pip installation.

### Docker Compose override changes

When using `astro dev start`, you can specify a [`docker-compose.override.yml`](/docs/cli/v1.43/run-airflow-locally#override-the-astro-cli-docker-compose-file) file. If you specified any overrides for the `webserver` container in your override file for Airflow 2, these break your ability to use `astro dev start` because there is no longer a `webserver` container in Airflow 3. Replace all references to `webserver` with `api-server`.

### Dags folder location change

On Airflow 2, the `dags/` folder is located at `/usr/local/airflow/dags`, but with the addition of [Dag versioning](/docs/astro/dag-versioning) in Airflow 3, you can no longer be sure of the exact path of the relevant Dags folder. Instead of using this hardcoded path or the `AIRFLOW_HOME` environment variable, use paths relative to the current file using the Python keyword [**file**](https://docs.python.org/3/reference/datamodel.html#module.__file__). Note also that because Dag bundling is not enabled when using `astro dev start`, you don't find errors from this folder location change until your code is deployed to Astro.

For example, if you use the `AIRFLOW_HOME` variable to open a file in your Dags folder like the following code example, it breaks in Airflow 3:

```python wrap theme={null}
open(f"{AIRFLOW_HOME}/dags/path/to/file.txt", "r")
```

### PYTHONPATH doesn't include the Dags folder

On Astro with Airflow 3, the `dags/` folder isn't on the `PYTHONPATH`, and you can't customize the `PYTHONPATH` to include the `dags/` folder or any other folders. This is a consequence of [Dag versioning](/docs/astro/dag-versioning) and Dag bundles in Airflow 3, where the exact path of the Dags folder isn't fixed at runtime. Setting `PYTHONPATH` in your `Dockerfile` or as an environment variable has no effect on Astro.

If your Airflow 2 Dags imported modules directly from the `dags/` folder using an import like `from my_utils import helper`, where `my_utils.py` lives in `dags/`, these imports break on Airflow 3. Imports that include the `dags.` prefix, such as `from dags.my_utils import helper`, continue to work.

Astronomer recommends that you move shared modules out of the `dags/` folder and into the `include/` folder, which remains on the `PYTHONPATH`, then update your imports. For example:

```python wrap theme={null}
from include.my_utils import helper
```

If you want to keep helper files in the `dags/` folder, for example, to avoid an image deploy when you change helper code, update your imports to use the `dags.` prefix:

```python wrap theme={null}
from dags.my_utils import helper
```

See [Manage Airflow code](/docs/learn/managing-airflow-code) for the recommended project structure.

## Airflow 3 to Airflow 2 rollback support and requirements

Astro supports rolling back from Airflow 3 to Airflow 2 but not all Airflow 3.x Deployments can be reverted to all Airflow 2.x versions.

* You can only roll back to Airflow 2.11, which runs on Astro Runtime 13.7.0 or above.
* Rollback to Airflow 2.11 is only possible if your Deployment is running an Airflow 3 version that includes rollback support: Runtime versions 3.0-11, or later.

<Note>Airflow 3 to Airflow 2 rollbacks take longer than same-version rollbacks.</Note>


# Add Astro domains to your network
Source: https://astronomer.io/docs/astro/allowlist-domains

A list of Astro domains to add to your organization's network allowlist.

To use Astro in a network that requires you to allowlist new domains, make a request to allowlist the following domains on your network:

* `https://cloud.astronomer.io/` - Astro Cloud Hosted service, so your users can access the Astro UI to manage your Astro and Airflow infrastructure.
* `https://api.astronomer.io/` - Astro API, so you can programmatically manage your Astro and Airflow infrastructure.
* `https://images.astronomer.cloud/` - Only required for [Remote Execution](/docs/astro/execution-mode). The Remote Execution Agent pulls images from this domain.
* `https://auth.astronomer.io/` - Domain needed for authenticating requests to login to the Astro Cloud Hosted service.
* `https://updates.astronomer.io/` - Domain used to check for new versions of Astro Runtime images. Astro alerts you to new Runtime versions in Airflow Deployments’ UIs.
* `https://install.astronomer.io/` - Domain needed to install the Astro CLI, which is used for deploying project images to Astro.
* `https://*.astronomer.run/` (Recommended) - URLs for your individual Airflow Deployments in Astro. Allowlist needed so your users can connect to them to manage workflows.
* `https://<organization-id>.astronomer.run/` - Only required if you don't use the recommended wildcard hostname, `https://*.astronomer.run/`. Individual Deployment URL for your Organization, used instead of wildcard.
* `https://<clusterId>.registry.astronomer.run/` - Only required if you don't use the recommended wildcard hostname, `https://*.astronomer.run/`. Domain required to push images for your Airflow Deployments if you don't use the wildcard hostname.
* `https://<clusterId>.external.astronomer.run/` - Only required for [Remote Execution](/docs/astro/execution-mode) Deployments. These domains represent your Airflow Deployments' orchestration plane. They must be allowlisted so your Remote Execution Agents can connect to them.
* `https://o11y.astronomer.io` - Astro OpenLineage URL endpoint required for integration with the [Astro Observe](/docs/astro/observe-get-started) and [Alerts](/docs/astro/alerts) features.

To locate the value for `<organization-id>` in the Astro UI, click **Settings** (**Organization Settings** in the legacy UI). Copy the value listed under **Organization ID**.
To locate the value for `<clusterId>` in the Astro UI, go to **Settings** > **Clusters** (in the legacy UI, go to **Organization Settings** > **Clusters**), and copy the **ID** value under each cluster's details page.

To build and deploy Astro project images, additionally allowlist the following domains:

* `https://air.astronomer.io` - Domain needed for accessing Astro Runtime images.
* `https://astrocrpublic.azurecr.io` - Domain needed for accessing Astro Runtime images, in case a redirect from `https://air.astronomer.io` doesn't work with your networking.
* `https://pip.astronomer.io` - Domain needed for installing packages in images.
* `https://install.astronomer.io` - Domain needed in order to install the Astro CLI, which is used for deploying project images to Astro.
* `https://raw.githubusercontent.com` - Domain needed as an alternative way to access GitHub-hosted content.
* `https://pypi.org` - Domain needed for installing Python packages in images.
* `https://astroproddagdeployment.blob.core.windows.net` - Domain needed to upload Dag bundles to the Astronomer managed storage account.


# Astro glossary
Source: https://astronomer.io/docs/astro/astro-glossary

A quick reference for terms you'll encounter on Astro.

The following table contains definitions for all of the key terms and concepts you'll come across on Astro. For a glossary of Apache Airflow terms, see [Airflow glossary](/docs/learn/glossary).

| Term                          | Definition                                                                                                                                                                                                                                                                                                                                                                                                                                                 |   |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - |
| API token                     | An API token is an alphanumeric token that grants programmatic access to Astro for automated workflows. An API token can be scoped to an [Organization](/docs/astro/organization-api-tokens) or a [Workspace](/docs/astro/workspace-api-tokens)."                                                                                                                                                                                                                    |   |
| Astro                         | [Astro](https://www.astronomer.io/product/) is a SaaS application that provides fully managed Apache Airflow environments for teams of all sizes. To get started, [start a trial](https://www.astronomer.io/lp/signup/).                                                                                                                                                                                                                                   |   |
| Astro alerts                  | [Astro alerts](/docs/astro/alerts) are customizable notifications that can alert teams of disruptions on Astro Deployments. Astro alerts are configured in the Astro UI and integrate with tools like Slack and Pagerduty. Unlike Airflow alerts, Astro alerts require no changes to Dag code.                                                                                                                                                                  |   |
| Astro CLI                     | The [Astro CLI](/docs/cli/v1.43/overview) is an open source command line interface built by Astronomer. You can use the Astro CLI to run Apache Airflow locally or interact programmatically with Astronomer products.                                                                                                                                                                                                                                          |   |
| Astro Cloud MCP Server        | The [Astro Cloud MCP Server](/docs/astro/astro-mcp-server) is an experimental remote MCP server that provides tools for discovering and managing your Astro resources, including Deployments, Workspaces, and environment variables. The current Astro Cloud MCP server is deprecated and shouldn't be used.                                                                                                                                                    |   |
| Astro                         | [Astro](/docs/astro/astro-architecture) is a distribution of Astro where the infrastructure required to run Airflow is fully managed by Astronomer in Astronomer's cloud. It's recommended for most teams running Airflow.                                                                                                                                                                                                                                      |   |
| Astro Hypervisor              | The [Astro Hypervisor](https://www.astronomer.io/press-releases/astronomer-introduces-new-capabilities-to-enable-the-future-of-effortless-cost-effective-managed-airflow/) an Astronomer-managed component of the Astro platform that facilitates operating and optimizing your Deployments.                                                                                                                                                               |   |
| Astro Private Cloud           | [Astro Private Cloud](/docs/astro-private-cloud/) is Astronomer's commercial offering for running Apache Airflow on Kubernetes in a private cloud or air-gapped environment. The infrastructure required to run the service is hosted and managed entirely by your organization instead of by Astronomer. It's only recommended for extremely security conscious organizations running at a unique level of scale.                                              |   |
| Astro project                 | An [Astro project](/docs/cli/v1.43/develop-project) contains the set of files necessary to run Airflow either locally or on Astro, including dedicated folders for the Dag files, plugins, and dependencies. Create a new Astro project by running `astro dev init` with the [Astro CLI](/docs/cli/v1.43/overview).                                                                                                                                                  |   |
| Astro Registry MCP Server     | The [Astro Registry MCP Server](/docs/astro/astro-mcp-server) is an experimental public remote MCP server that provides resources for the [Astro Registry](https://registry.astronomer.io), including Airflow modules and connection configurations.                                                                                                                                                                                                            |   |
| Astro Runtime                 | [Astro Runtime](/docs/runtime/runtime-image-architecture) is a Docker image for running Airflow that's built and maintained by Astronomer. Every Astro project and Deployment is configured with a version of Astro Runtime. Compared to the Apache Airflow Docker image, Astro Runtime additionally includes smart Airflow configurations, pre-installed packages, a security manager for role-based access control (RBAC), and expedited vulnerability fixes. |   |
| Astro UI                      | The Astro UI is the user interface for Astro. From the Astro UI, users can manage Organizations, Workspaces, and Deployments. The Astro UI is available at `https://cloud.astronomer.io`.                                                                                                                                                                                                                                                                  |   |
| Cluster                       | An Astro cluster is a Kubernetes cluster that hosts the infrastructure required to run Deployments. Clusters can be [standard](/docs/astro/resource-reference-hosted#standard-cluster-regions) or [dedicated](/docs/astro/resource-reference-hosted#dedicated-cluster-regions).                                                                                                                                                                                      |   |
| Control Plane                 | The Astro control plane is Astro's interface for managing Airflow environments running in the cloud. Use the Astro UI or the Astro CLI to interact with the control plane. It provides end-to-end visibility, control, and management of users, teams, Workspaces, Deployments, metrics, and logs.                                                                                                                                                         |   |
| Dag Bundle Version            | A Dag Bundle Version is a unique timestamp generated by the Astro CLI after a user completes a Dag-only deploy and identifies the version of code that Astro is running.                                                                                                                                                                                                                                                                                   |   |
| Data assets                   | Assets can include tasks, datasets, warehouse tables, and local files. They are used to create Data Products.                                                                                                                                                                                                                                                                                                                                              |   |
| Freshness                     | An observability alert for data products that measures how recently an asset was updated.                                                                                                                                                                                                                                                                                                                                                                  |   |
| Data lineage                  | [Data lineage](/docs/astro/observe-openlineage) is the concept of tracking and observing data flowing through a data pipeline. Data lineage can be used to understand data sources, troubleshoot run failures, manage personally identifiable information (PII), and ensure compliance with data regulations. Astro includes data lineage features in the Astro UI.                                                                                             |   |
| Data product                  | [Data products](/docs/astro/create-data-products) are abstractions that gives you observability into the health and performance of data pipelines.                                                                                                                                                                                                                                                                                                              |   |
| Timeliness                    | An observability alert for data products that measures whether or not it meets a Timeliness SLA.                                                                                                                                                                                                                                                                                                                                                           |   |
| Deploy                        | A [deploy](/docs/astro/deploy-code) is the process of pushing code to a Deployment on Astro. A code push can include either a complete Astro project or Dag code.                                                                                                                                                                                                                                                                                               |   |
| Deployment                    | An [Astro Deployment](/docs/astro/create-deployment) is an Airflow environment that is powered by all core Airflow components, including schedulers and workers. Astro users can deploy Dags to a Deployment, and can have one or more Deployments within a Workspace.                                                                                                                                                                                          |   |
| Deployment file               | A Deployment file is a YAML-formatted snapshot of a Deployment's current configuration. In addition to configurations, a Deployment file includes a Deployment's metadata and a timestamp to associate the configuration with a specific Deployment. Deployment files are used to [manage Deployments as code](/docs/astro/manage-deployments-as-code).                                                                                                         |   |
| Deployment template file      | A Deployment template file is a YAML configuration that can be used to create or update a Deployment. Unlike a Deployment file, Deployment template files don't contain metadata and can be applied to any new or existing Deployment. Deployment template files are used to [manage Deployments as code](/docs/astro/manage-deployments-as-code).                                                                                                              |   |
| Environment variable          | An [environment variable](/docs/astro/environment-variables) is a key-value pair that defines a configuration or value for a Deployment.                                                                                                                                                                                                                                                                                                                        |   |
| High availability (HA)        | [High availability (HA)](/docs/astro/deployment-resources#enable-high-availability) is a feature on Astro for ensuring that the components of a Deployment continue to run even in the event of an outage. On Astro, HA can be enabled or disabled per Deployment.                                                                                                                                                                                              |   |
| Namespace                     | A [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) is a Kubernetes component which isolates Airflow environments within a Kubernetes cluster. Each Deployment uses a separate namespace to isolate resources.                                                                                                                                                                                                    |   |
| Organization                  | An Organization is the highest management level on Astro. An Organization contains Workspaces, which are collections of Deployments, or Airflow environments, that are typically owned by a single team.                                                                                                                                                                                                                                                   |   |
| Service Level Agreement (SLA) | An SLA defines the time by which a data product needs to be delivered or other process completed.                                                                                                                                                                                                                                                                                                                                                          |   |
| Standard cluster              | A [standard cluster](/docs/astro/resource-reference-hosted#standard-cluster-regions) is a cluster type available on Astro Hosted. It's multi-tenant and runs Deployments from multiple Organizations.                                                                                                                                                                                                                                                           |   |
| Worker Node                   | A [worker node](/docs/astro/configure-worker-queues#worker-queue-settings) is a node used to run Airflow worker Pods, which are responsible for executing Airflow tasks in the Deployments.                                                                                                                                                                                                                                                                     |   |
| Worker Node Pool              | A worker node pool is a Kubernetes node pool that's used to run worker nodes of the same type in the Astro Orchestration Plane. Each worker node pool has a worker type and a maximum node count.                                                                                                                                                                                                                                                          |   |
| Worker Queue                  | A [worker queue](/docs/astro/configure-worker-queues) is a set of configurations that apply to a group of workers in a Deployment running the Celery executor. Within a worker queue, users can configure worker type, worker size, and autoscaling behavior.                                                                                                                                                                                                   |   |
| Worker Type                   | The worker type defines the quantity of resources a Celery worker can consume. On Astro, worker types are defined in terms of Astronomer units (A5, A10, A20). Each Ax type has a different configuration for memory, CPU and default number of concurrent tasks per Celery worker.                                                                                                                                                                        |   |
| Workspace                     | [Workspaces](/docs/astro/manage-workspaces) are collections of Deployments that can be accessed by a specific group of users. Workspaces can be used to group Deployments that share a business use case or environment trait.                                                                                                                                                                                                                                  |   |


# Book Astro office hours
Source: https://astronomer.io/docs/astro/astro-office-hours

Attend office hours with Astronomer data engineers to ask non-support questions about Airflow.

<Warning>
  Astro Office Hours will be decommissioned effective August 31, 2026. For any questions or assistance, please reach out to your Account Manager.
</Warning>

<Note> This is feature is only available if you are on the **Team** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

Office hours are a way for you to meet with Astronomer data engineers and ask about Astro and Airflow features and best practices. Office hours are available weekly on a first-come, first-served basis. An office hour session is typically 30 minutes long.

<Warning>
  Office hours are not a replacement for support. [Submit a support request](/docs/astro/astro-support) if you have any urgent issue, bug, or something that needs fixing. Astronomer data engineers will not be able to answer support requests during office hour sessions.

  If you need more extensive assistance, guidance, or regular long-term engagement, reach out to the [Professional Services](https://www.astronomer.io/professional-services/) team.
</Warning>

During an office hours meeting, you can:

* Ask questions about Airflow features and best practices.
* Discuss new Astro features or provide feedback.
* Get insight on integrating Airflow with new systems.
* Do an architecture review or discuss scaling Airflow and Astro Deployments .
* Conduct a code review for a dag.

## Book an office hours appointment

Before you book an office hours appointment, please browse the documentation, [Astronomer Academy](https://academy.astronomer.io/), or [Airflow Guides](https://www.astronomer.io/docs/learn/).

To book office hours, click **Help** > **Book Office Hours** in the [Astro UI](https://cloud.astronomer.io). Then, schedule a 30-minute virtual meeting on the scheduling form. In the form, provide details about any issues and questions that you want to discuss during the meeting.

<Frame>
  <img alt="A screenshot showing the menu in cloud.astronomer.io which contains a Book Office Hours entry" />
</Frame>

Alternatively, you can use the [**Book office hours** link](https://scheduler.zoom.us/d/ljvcm2p9/astro-office-hours) in the documentation navigation.


# Submit a support request
Source: https://astronomer.io/docs/astro/astro-support

Get Astro support when you need it.

In addition to product documentation, the following resources are available to help you resolve issues:

* [Astronomer knowledge base](https://support.astronomer.io/hc/en-us)
* [Airflow guides](https://www.astronomer.io/docs/learn/)

If you're experiencing an issue or have a question that requires Astronomer expertise, use one of the following methods to contact Astronomer support:

* Submit a support request in the [Astro UI](https://cloud.astronomer.io/open-support-request).
* Submit a support request on the [Astronomer support portal](https://support.astronomer.io/hc/en-us).

## Best practices for support request submissions

The following are the best practices for submitting support requests in the Astro UI or the Astronomer support portal:

### Check the Astro status page

Before you open a ticket for unexpected or disruptive behavior on Astro, check the [Astro status page](https://status.astronomer.io/) to see if the problem you're experiencing has already been reported.

### Be as descriptive as possible

The more information you can provide about the issue you're experiencing, the quicker Astronomer support can start the troubleshooting and resolution process. When submitting a support request, include the following information:

* Have you made any recent changes to your Deployment or running dags?
* What solutions have you already tried?
* Is this a problem in more than one Deployment?

<a />

### Include logs or code snippets as text and not screenshots

If you've already copied task logs or Airflow component logs, send them a part of your request as text. The more context you can provide in your request, the better.

### Check recommended support articles

If you draft your support ticket on the [Astronomer support portal](https://support.astronomer.io), the portal automatically recommends support articles to you based on the content in your ticket. Astronomer recommends looking through these recommendations to see if your issue has a documented solution before submitting your ticket.

You can also proactively search support articles without submitting a support ticket on the [Astronomer knowledge base](https://support.astronomer.io/hc/en-us).

## Submit a support request in the Astro UI

1. In the Astro UI, click **Help** > **Submit Support Request**.

   <Frame>
     <img alt="Submit Support Request menu location" />
   </Frame>

   Alternatively, you can directly access the support form by going to `https://cloud.astronomer.io/open-support-request`.

2. Select a **Request Type**. Your request type determines which other fields appear in the support request.

3. Complete the rest of the support request.

4. Click **Submit**.

   You'll receive an email when your ticket is created and follow-up emails as Astronomer support replies to your request. To check the status of a support request, you can also sign in to the [Astronomer support portal](https://support.astronomer.io).

## Submit a support request on the Astronomer support portal

Astronomer recommends that you submit support requests in the Astro UI. If you can't access the Astro UI, sign in to the [Astronomer support portal](https://support.astronomer.io) and create a new support request.

If you're new to Astronomer, you'll need to create an account on the Astronomer support portal to submit a support request. Astronomer recommends that you create an account with the same email address that you use to access Astro. This allows you to view support tickets from other team members that have email addresses with the same domain.

If your team uses more than one email domain, add all domains to your Organization so that team members with different email domains can view each others' support requests. See [Create and manage domains for your Organization](/docs/astro/manage-domains).

## Monitor existing support requests

If you've submitted your support request on the Astronomer support portal, sign in to the [Astronomer support portal](https://support.astronomer.io) to:

* Review and comment on requests from your team.
* Monitor the status of all requests in your organization.

<Tip>To add a teammate to an existing support request, cc them when replying on the support ticket email thread.</Tip>

## Ticket Priorities

To help Astronomer Support respond effectively to your support request, priorities are determined automatically by Astronomer. Refer to the [Astronomer Technical Support and Success Packages](https://www.astronomer.io/legal/technical-support-success-packages/) to read more about ticket priorities and their SLAs.

The following sections show the four ticket priorities with examples and descriptions for each:

### P1: Critical impact

A production Deployment is completely unavailable, or a dag that was previously working in production stops working, even though it was not changed.

Astronomer handles P1 tickets with the highest levels of urgency. If Astronomer Support responds to a P1 ticket, and subsequently does not hear back from you for 2 hours, the ticket priority is automatically changed to P2.

Additionally, if the immediate problem is solved, but follow-up investigations continue, those investigations are conducted in a separate ticket at a lower priority.

### P2: High impact

Your ability to use Astro is severely impaired, but does not affect any critical, previously working pipelines in production.

Examples:

* A newly deployed production dag is not working, even though it worked successfully in a development or test environment.
* The Airflow UI is unavailable.
* You can't deploy code to a production Deployment, but existing dags and tasks run as expected.
* You need to modify a Hybrid cluster setting that is required for running tasks, such as adding a new worker instance type.
* Task logs are missing in the Airflow UI.

### P3: Medium impact

Service is partially impaired.

Examples:

* A newly deployed dag is not working in a development Deployment, even though it worked successfully in a local environment using the Astro CLI.
* You need to modify a Hybrid cluster setting that affects your cluster's performance, but isn't required to run tasks, such as changing the size of your cluster's database or adding a new VPC peering connection.
* Astro CLI usage is impaired. For example, there are incompatibility errors between installed packages.
* There is an Airflow issue that has a code-based solution.
* You received a log alert on Astronomer.
* You lost the ability to use a [Preview](/docs/astro/feature-previews) feature that does not affect general services.
* You can't deploy code to a non-production Deployment, but existing dags and tasks run as expected.

### P4: Low impact

Astro is fully usable, but you have a question for the Support team.

Examples:

* There are package incompatibilities caused by a specific, complex use case.
* You have an inquiry or a small bug report for a Preview feature.

## Business Hours and Holidays

Astronomer Support Business Hours are 9:00 PM Sunday New York local time through 9:00 PM Friday New York local time. In addition, Astronomer observes the following holidays that are excluded from Business Hours. If [your plan](https://www.astronomer.io/pricing/) or contract includes tickets eligible for 24x7 support, holidays and business hours do not affect response time for those specific cases.

| Holiday                    | 2026 Observed Date |
| -------------------------- | ------------------ |
| New Year’s Day             | January 1          |
| Martin Luther King Jr. Day | January 19         |
| Memorial Day               | May 25             |
| Juneteenth                 | June 19            |
| Independence Day           | July 3             |
| Labor Day                  | September 7        |
| Thanksgiving               | November 26        |
| Day After Thanksgiving     | November 27        |
| Christmas Eve              | December 24        |
| Christmas Day              | December 25        |

## Book office hours

<Warning>
  Astro Office Hours will be decommissioned effective August 31, 2026. For any questions or assistance, please reach out to your Account Manager.
</Warning>

If you don't require break-fix support, Astronomer recommends scheduling office hours. In an office hours meeting, you can ask questions, make feature requests, or get expert advice for your data pipelines.

For more information about booking an office hour meeting, see [Book an office hours appointment](/docs/astro/astro-office-hours#book-an-office-hours-appointment).

## Request an escalation for an existing support ticket

<Note> This is feature is only available if you are on the **Business** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

Business and Enterprise customers can request escalated support for an existing ticket. To request a support escalation, email escalations\[ɑt]astronomer.io with the ticket number in the subject line and the following information in the body:

* Ticket number
* Issue summary
* Reason for escalation

Before requesting an escalation, give Astronomer support adequate time to respond to your ticket. Please do not escalate a ticket within the first hour of submitting it.

Reasons for escalation include:

* The ticket is taking much longer than expected to resolve.
* Multiple attempts to solve the issue have failed.
* The urgency of the ticket has increased substantially.

When you make an escalation request, it alerts the on-call shift manager to the ticket, who evaluates the escalation request. The on-call shift manager either assumes ownership of the ticket or ensures meaningful and timely progress is made on the ticket.


# Azure Native ISV retirement
Source: https://astronomer.io/docs/astro/azure-liftr-migration

Learn about the Azure Native ISV retirement and how to migrate to standard Astro billing and SSO.

Astronomer is retiring the [Azure Native ISV offering](https://learn.microsoft.com/en-us/azure/partner-solutions/astronomer/overview) for Astro. The program is fully deprecated on July 1, 2026.

Your Astro platform keeps running throughout the migration. Your Organization, Workspaces, Astro Deployments, Dags, Dag history, users, and data are all preserved. This change affects how you're billed and how you sign in, not the Astro product you use every day.

Two affected groups need to take action:

* **Azure Native ISV PayGo plans**: If you purchased Astro through the Team PayGo or Developer PayGo plans on the Azure Native ISV offering, you complete a few short steps yourself.
* **Azure Native ISV annual contracts**: If you're on an annual Azure contract that runs on the Azure Native ISV integration, contact your Astronomer account representative for a guided migration.

If you purchased Astro directly through Astronomer (at `cloud.astronomer.io` or through a direct order form), this change doesn't affect you, regardless of which cloud you run on.

## Timeline

The following dates apply to both affected groups:

| Date                   | What happens                                                                         |
| ---------------------- | ------------------------------------------------------------------------------------ |
| Early April 2026       | The Azure Native ISV offering was delisted. Service is still active.                 |
| May 1 to June 30, 2026 | Migration window. Billing and SSO move to the standard Astronomer surface.           |
| July 1, 2026           | Full deprecation. The Azure Native ISV billing path and SSO connections are retired. |

<Note>
  The Azure Native ISV plan options remain visible in the Azure portal UI even though they're no longer purchasable.
</Note>

## Migrate from an Azure Native ISV PayGo plan

You can complete this migration yourself in a few minutes.

<Steps>
  <Step title="Add a credit card">
    1. Go to the [Astronomer billing page](https://cloud.astronomer.io/billing).
    2. Add a credit card as your payment method.

    After your card is on file, your Astro usage bills against it on a monthly cycle. Your current pricing and plan terms are honored, so you don't need to re-select a plan or re-sign anything.

    Astronomer terminates your Azure Native ISV subscription, so don't cancel it yourself in the Azure portal.

    Your current Azure billing remains active until July 1, 2026. After that, your credit card on file becomes the default payment method.
  </Step>

  <Step title="Verify that standard Azure SSO works">
    1. Go to [cloud.astronomer.io](https://cloud.astronomer.io/).
    2. Sign in with your corporate email address (for example, `you@yourcompany.com`). This routes you to the standard Azure SSO connection.
    3. Confirm that you have full access to your account.
  </Step>

  <Step title="Invite a backup Organization Owner">
    Before you remove the old SSO connection, invite a user that can sign in without SSO. This account gives you a way to access Astro during the period between removing the old SSO connection and configuring a new one.

    1. Invite a new user with an email address from a domain that you don't manage through SSO, such as a personal email address rather than `you@yourcompany.com`. Astro routes addresses outside your SSO domain to email and password authentication. See [Add a user to an Organization](/docs/astro/manage-organization-users#add-a-user-to-an-organization).
    2. Assign the user the Organization Owner role.
    3. Accept the invitation and confirm that the backup user can sign in with an email and password.

    If your Organization enforces SSO logins on an email address you want to use for the backup user, temporarily allow all login methods first. See [SSO enforcement](/docs/astro/configure-idp#sso-enforcement).
  </Step>

  <Step title="Remove the current SSO connection">
    If you're an Organization Owner, go to the [authentication settings page](https://cloud.astronomer.io/settings/authentication) and remove the old SSO connection. For step-by-step instructions, see [Reconfigure SSO to a new identity provider](/docs/astro/configure-idp#reconfigure-sso-to-a-new-identity-provider).

    The Developer PayGo and Team PayGo tiers default to [Allow all login methods](/docs/astro/configure-idp#sso-enforcement), so you can sign in with an email and password combination or Google authentication. You can stop the SSO reconfiguration here unless you want to add a new SSO connection.
  </Step>

  <Step title="(Optional) Reset the SSO connection to a new Azure AD connection">
    If you want to add a new SSO connection, follow the steps in [Register Astro as an application on Azure](/docs/astro/configure-idp#microsoft-entra-id).
  </Step>
</Steps>

## Migrate from an Azure Native ISV annual contract

<Note>
  Your Astronomer account representative can guide you through this migration. Contact them directly if you have any questions.
</Note>

### What to expect

Your Astronomer account representative contacts you within a few weeks to confirm your setup, walk through the billing and SSO migration on a short call, and schedule any coordination needed on your side.

Your billing method stays valid through the program deprecation. There is no billing interruption, no double-billing, and no action required from your procurement or finance team until your account representative proposes the next step. Your current pricing and plan terms are honored through the migration.

### Prepare for the migration call

Complete these steps before your call:

<Steps>
  <Step title="Verify that standard Azure SSO works">
    1. Go to [cloud.astronomer.io](https://cloud.astronomer.io/).
    2. Sign in with your corporate email address (for example, `you@yourcompany.com`). This routes you to the standard Azure SSO connection.
    3. Confirm that you have full access to your account.
  </Step>

  <Step title="Invite a backup Organization Owner">
    Before you remove the old SSO connection, invite a user that can sign in without SSO. This account gives you a way to access Astro during the period between removing the old SSO connection and configuring a new one.

    1. Invite a new user with an email address from a domain that you don't manage through SSO, such as a personal email address rather than `you@yourcompany.com`. Astro routes addresses outside your SSO domain to email and password authentication. See [Add a user to an Organization](/docs/astro/manage-organization-users#add-a-user-to-an-organization).
    2. Assign the user the Organization Owner role.
    3. Accept the invitation and confirm that the backup user can sign in with an email and password.

    If your Organization enforces SSO logins on an email address you want to use for the backup user, temporarily allow all login methods first. See [SSO enforcement](/docs/astro/configure-idp#sso-enforcement).
  </Step>

  <Step title="Remove the current SSO connection">
    If you're an Organization Owner, go to the [authentication settings page](https://cloud.astronomer.io/settings/authentication) and remove the old SSO connection. For step-by-step instructions, see [Reconfigure SSO to a new identity provider](/docs/astro/configure-idp#reconfigure-sso-to-a-new-identity-provider).
  </Step>

  <Step title="(Optional) Enable all login methods for your Organization">
    If your security or IT teams allow it, you can enable all login methods for your Organization. This shortens the migration because you don't have to set up a new SSO connection.

    1. Go to your [Organization's authentication settings page](https://cloud.astronomer.io/settings/authentication).
    2. In the **Advanced** section, select the option to enable all login methods.
  </Step>

  <Step title="(Optional) Reset the SSO connection to a new Azure AD connection">
    If you need to add a new SSO connection, follow the steps in [Register Astro as an application on Azure](/docs/astro/configure-idp#microsoft-entra-id).
  </Step>
</Steps>

If you have questions before your account representative contacts you, email `platform@astronomer.io`. Astronomer routes the question internally or loops in your representative if that's faster.

## What stays the same

Regardless of which group you're in:

* Your Astro Deployments continue running with no interruption.
* Your Dags, schedules, run history, connections, and variables are preserved.
* API tokens, service accounts, and Deployment keys continue to work.
* Workspace permissions and User Roles persist.
* You work with the same Astronomer support team under the same SLAs.

## Common questions

<AccordionGroup>
  <Accordion title="Why are we making this change?">
    Microsoft and Astronomer have jointly decided to retire the Azure Native ISV integration to enable a more unified Astro experience across all customers.
  </Accordion>

  <Accordion title="Will my Dags keep running during the migration?">
    Yes. Your Deployments, schedulers, workers, Dag history, connections, and variables are all unaffected.
  </Accordion>

  <Accordion title="Will my API keys and service accounts keep working?">
    Yes. API authentication is separate from SSO and doesn't change.
  </Accordion>

  <Accordion title="What happens if I do nothing by July 1, 2026?">
    * If you're on a PayGo plan, your Astro platform keeps running, but your Azure Native ISV subscription terminates. Without a credit card on file, billing moves to a manual state. Add a credit card at the [Astronomer billing page](https://cloud.astronomer.io/billing) to avoid any billing interruption. Resources are automatically spun down on July 15, 2026.
    * If you're on an annual contract, your account representative makes sure you land safely. Respond when they contact you.
  </Accordion>

  <Accordion title="Can I stay on the same Astro plan?">
    Yes, or you can move to a different plan. Your current pricing and plan terms are honored through the migration.
  </Accordion>

  <Accordion title="I bought directly through Astronomer. Does this affect me?">
    No.
  </Accordion>

  <Accordion title="What happens to the Astronomer resource in my Azure portal?">
    Astronomer terminates the Azure Native ISV resource connection after the migration. The resource served as a billing and authentication link between Azure and Astro, but didn't provision or manage any Azure infrastructure on your behalf.

    After migration:

    * The Astronomer resource in your Azure portal shows as terminated or delisted.
    * Your Astro Deployments continue running unchanged — they were never dependent on this Azure portal resource.
    * You can safely remove the terminated resource from your Azure portal view.

    After July 15, 2026, Astronomer spins down and deletes the Azure resource from the Azure portal. No action is required on your part — the resource removal is automatic.

    The Astro platform itself runs in Astronomer-managed infrastructure and is unaffected by the removal of this Azure portal integration resource.
  </Accordion>
</AccordionGroup>


# Airflow Edge Cases on Astro
Source: https://astronomer.io/docs/astro/best-practices/airflow-edge-cases

Edge cases when running Airflow that give unintuitive behavior and how to avoid them

Apache Airflow gives huge flexibility in how it can be used. While this is a benefit of Airflow, it also means that there are a few edge case behaviors that are somewhat unintuitive.

## Workers Scaling Down Frequently

Astro guarantees that each task will have at least 24 hours to complete before being disrupted by a scaling event. This means that when a worker is scaled down, it can't actually be removed until the last task on it completes or 24 hours have passed since it was marked for scale down. And if the last task to complete runs for a considerably longer time than the other tasks on the worker, a single task could be on the worker for a long time, leading to rather inefficient use of resources. This problem is most pronounced if there is frequent scaling up and down. These workers that are waiting on a task to complete before shutting down are called "terminating" and are *not* counted against the max number of workers for the queue. This leads to the possibility that more workers can be running on a worker queue than its max number of workers, leading to increased cost.

Walking through a worst case scenario, imagine that every hour, 100 tasks are started on a worker queue with a concurrency of 10. Out of the 100 tasks, 5 of them take 3 hours to complete, while the rest finish within 15 minutes. At the top of the hour, 10 workers are spun up to perform the work. After 15 minutes, all but 5 of the tasks are completed, so Astro determines that with only 5 running tasks, only 1 worker needs to remain. However, 4 other workers will also be kept in a terminating state while they complete their long running tasks. Once 1 hour passes, 10 new workers will be up to complete the new tasks, in addition to the 4 terminating workers. At 1 hour and 15 minutes, the pattern will repeat and now there are 9 terminating workers. At 2 hours elapsed, the total number of workers in the worker queue will be 19, nearly double what might normally be expected from this worker queue.

The solution is to avoid putting tasks with wildly different execution times on the same worker queue. Long running tasks should run on a separate worker queue so that they are scaled up and down less frequently, and when they are scaled down, there are multiple tasks with similar durations leading to efficient use of the terminating worker.

## More than 1000 Deferred Tasks

With default settings, an Astro deployment will only evaluate the condition of 1000 deferred tasks, with other deferred tasks waiting unchecked until one of the 1000 has its condition met, opening up space. There is no indication in Airflow as to which deferred tasks are actually being checked. If you have CPU and memory headroom on the triggerer per your Analytics page, you can increase the [`default_capacity`](https://airflow.apache.org/docs/apache-airflow/2.11.0/configurations-ref.html#default-capacity) in Airflow 2 or [capacity](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#capacity) in Airflow 3. If you need to run more than 1000 deferred tasks at a time and do not have the headroom in CPU or memory to increase capacity, contact Astronomer Support.

## Impact of Long Dag Parse Times

You may be familiar with the best practice of [avoiding top level code](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#top-level-python-code) in dags, as it leads to a long Dag parse time. What may be less clear is what the impact of an increased Dag parse time is. First, the more Dags you have that take longer to parse, the slower changes in those Dags will show up in Airflow's UI and scheduling behavior. All the Dags are parsed in a loop continuously, and the longer it takes to get through that loop, the longer it takes for changes to take effect.

Additionally, the worker needs to parse the Dag file before it can actually run the task. While the Dag file is being parsed, the task will be in the queued state. Thus, a Dag that takes a long time to parse will see long queued times for all of its tasks.

## Logging and Out of Memory on Tasks

There are two ways that task logs are retrieved. While a task is running, its logs are read directly from the worker via a small webserver that the worker runs for this purpose. This enables "live logging". But after the task is completed, it takes the logs from the worker and uploads them to cloud storage. However, if the worker is killed with an out of memory (OOM) error, the worker is killed before the logs from its currently running tasks can be uploaded, and of course the worker is no longer around to serve live logs. Thus, worker out of memory errors lead to there being no logs for a task. After the worker OOM, the tasks will appear to be Running for another 5 minutes[^1] at which point the scheduler will notice that there has been no communication from the task in the form of periodic heartbeats for too long. It will mark the task as failed and state that it was killed because of no task heartbeat in the task's Event Log. Thus, if you see a task with no logs, but a heartbeat missing event in its Event Log, it's almost certainly an Out of Memory error.

[^1]: Unless [configured](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#task-instance-heartbeat-timeout) for a different amount of time


# When to use Airflow or Astro alerts for your pipelines on Astro
Source: https://astronomer.io/docs/astro/best-practices/airflow-vs-astro-alerts

Decide when to use Airflow callbacks or Astro alerts to monitor your pipelines on Astro.

When orchestrating data pipelines, it's key that you know when something goes wrong. It could be that a business critical Dag failed or that a Dag that provides data for another team took longer than normal to complete. A common consideration when running Airflow at scale is how to alert your team in different scenarios.

Airflow has built-in notification mechanisms for common use cases, but they have some limitations. For the cases where Airflow notifications aren't sufficient, [Astro alerts](/docs/astro/alerts) provide an additional level of observability. For many use cases, Astronomer recommends using a combination of Airflow and Astro alerts to best cover all alerting scenarios.

When you combine Airflow notifications with Astro alerts, you can:

* Use Astro alerts to configure Dag failure or success alerts for common communication channels like Slack, PagerDuty, or email.
* Use Astro alerts to implement pipeline SLAs based on task duration or Timeliness.
* Use Airflow email notifications and callbacks for custom task-level alerting logic.

This guide provides guidance on when to use Astro or Airflow alerts, as well as an example implementation covering a couple of common alerting scenarios.

## Feature overview

This guide highlights when to use the following Astro and Airflow features to create different types of alerts for your pipelines:

* [Astro alerts](/docs/astro/alerts) for configuring Dag SLAs and failure notifications.
* [Airflow callbacks](/docs/learn/error-notifications-in-airflow#airflow-callbacks) for custom task-level alerts.

## Best practice guidance

Using a combination of Astro alerts and Airflow notifications allows your team to implement alerting logic using the best tool for each scenario. You might need different types of alerts that are better suited towards one option or the other, for example:

* **Task-level alerts with custom logic** are recommended if you want be notified of success or failure only for a specific task, or you want to run custom code if a task succeeds or fails. Task-level success or failure alerts or alerts with custom logic are not currently supported with Astro alerts, but they're straightforward to implement in Airflow.
* **Dag-level success or failure alerts** can be created using either Airflow notifications or Astro alerts, but on Astro they are easy to configure and don't require any code changes. To implement these alerts with Airflow, you need to update your Dag code and complete additional configuration for your communication channel (for example, setting up an SMTP server for email alerts).
* **Timeliness alerts** are recommended when you need your Dag to complete by a certain time of day. These types of alerts are available only on Astro.
* **Task duration alerts** are recommended when you need to know if your task has taken longer than a certain amount of time to complete. These types of alerts are easy to implement with Astro alerts, but are unintuitive in Airflow.

In some cases, whether you set up an alert on Astro or in Airflow will depend on how your Dags were configured before you moved to Astro. For example, if all of your Dags already have email-on-failure notifications configured, it will likely be easier for you to configure an SMTP server on Astro and rely on the existing notification than implement a new Astro alert for each dag. On the other hand, if you are developing new Dags on Astro, it will likely be easier to implement Astro alerts which require less configuration for the communication channels.

<Info>Programmatically creating Astro alerts is coming soon. This will make it easy to configure alerts for a large number of Dags without  having to create each alert manually.</Info>

In other cases, Astro alerts are always the better option. If you need to implement SLAs for your pipelines so you immediately know when a Dag task is taking too long to run, you should use the Timeliness or task duration Astro alerts. There are a couple of reasons these are preferable to Airflow SLAs:

* The Astro alert for task duration is relevant to the task start time, making it easy to understand and implement for your use case. Airflow SLAs often cause confusion because they are relevant to the Dag execution date, not the task start time.
* The Astro alert for Timeliness allows you to implement alerts based on a specific time of day your Dag should complete. This type of SLA is not available in Airflow.
* Astro alerts allow you to easily choose between multiple common communication channels. With Airflow SLAs, if you want your notification sent somewhere other than your email, you have to write your own custom callback logic. Additionally, if you have an SMTP service configured in your Airflow environment you will get email notifications for SLA misses; there is no way to turn this behavior off.

### Recommended alerts

Astronomer recommends choosing the following alerts in these common scenarios:

| Scenario                                      | Astro alert | Airflow email notification | Airflow callback | Airflow SLA | Airflow timeout |
| --------------------------------------------- | :---------: | :------------------------: | :--------------: | :---------: | :-------------: |
| Email on Dag success or retry                 |      X      |              X             |                  |             |                 |
| Email on specific task success or retry       |             |              X             |                  |             |                 |
| Custom dag-level alerting logic               |             |                            |         X        |             |                 |
| Custom task-level alerting logic              |             |                            |         X        |             |                 |
| Timeliness Dag SLA                            |      X      |                            |                  |             |                 |
| Task duration SLA                             |      X      |                            |                  |             |                 |
| Stop a running task after some amount of time |             |                            |                  |             |        X        |
| Trigger another Dag based on an alert         |      X      |                            |                  |             |                 |

For more information on different types of alerts, check out the documentation linked in [See also](#see-also).

## Example

This example shows how to implement a combination of Airflow and Astro alerts to cover notifications for common scenarios including Dag failures, specific task failures, and pipeline SLAs.

### Prerequisites

To implement the alerts shown in this example, you need:

* At least one [Astro Deployment](/docs/astro/create-deployment). Your Deployment must run Astro Runtime 7.1.0 or later and it must have [OpenLineage enabled](/docs/astro/observe-openlineage).
* An [Astro project](/docs/cli/v1.43/develop-project) with at least one dag. Your Dag should have at least two tasks.

However, you can extend this example to encompass any number of Astro Deployments and dags.

### Implementation

To implement this use case:

1. Open the Dag in your Astro project and configure a pre-built SlackNotifier for one of its tasks as a task-level argument. See [Example pre-built notifier: Slack](/docs/learn/error-notifications-in-airflow#example-pre-built-notifier-slack) for sample code.
2. Deploy your project to your Astro Deployment. See [Deploy code to Astro](/docs/astro/deploy-code).
3. Add a connection to Slack in the Astro UI. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections). This connection will be used by your SlackNotifier, so make sure the connection ID matches what you used in your Dag code in Step 1.
4. In your Astro Deployment, configure an Astro **Dag failure** alert for your Dag using the communication channel of your choice. See [Set up Astro alerts](/docs/astro/alerts).
5. Configure an Astro **Timeliness** alert for your Dag based on the amount of time you expect your Dag to complete in. Use the communication channel of your choice. See [Set up Astro alerts](/docs/astro/alerts).

## See also

* [Set up Astro alerts](/docs/astro/alerts)
* [Manage Airflow Dag notifications](/docs/learn/error-notifications-in-airflow)
* [Airflow timeouts](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#timeouts)


# Astro API best practices
Source: https://astronomer.io/docs/astro/best-practices/astro-api-best-practices

Best practices for developing with the Astro API

The [Astro API](/docs/astro/api/v-1/overview) is Astronomer's REST API for managing resources on Astro, for example to create a Deployment. This guide provides the following best practices for general development, and specifically development with the Astro API, to ensure a safe and optimal experience:

* Considerations for using the [Astro API](/docs/astro/api/v-1/overview), [Astro CLI](/docs/cli/v1.43/overview), and [Astro Terraform provider](/docs/astro/terraform-provider)
* Error handling
* Authenticating scripts using API tokens
* Using a graphical REST API client for development
* Handling rate limiting with exponential backoff retries
* Reusing HTTP connections

## Feature overview

This guide highlights the following Astro features:

* [The Astro API](/docs/astro/api/v-1/overview)
* [The Astro CLI](/docs/cli/v1.43/overview)
* [The Astro Terraform provider](/docs/astro/terraform-provider)
* Astro API Tokens
  * [Deployment API tokens](/docs/astro/deployment-api-tokens)
  * [Workspace API tokens](/docs/astro/workspace-api-tokens)
  * [Organization API tokens](/docs/astro/organization-api-tokens)

## Considerations for using the Astro API, Astro CLI, and Astro Terraform provider

Astronomer provides several tools to manage Astro resources. Each tool offers benefits for specific use cases:

### Astro API

* Has structured input and output, which allows for convenient automation
* Allows you to use any programming language or framework that can make HTTP requests
* Requires you to implement status checking mechanisms yourself, such as polling the API to check for Deployment creation completion

### Astro CLI

* Provides a local development environment
* Produces human-readable output such as text or table format, therefore less suitable for automation
* Makes deploying Airflow code to Astro convenient

### Astro Terraform provider

* Allows you to use Terraform, the industry standard for managing infrastructure as code
* Requires Terraform development familiarity
* Uses declarative language, which means you define your desired end state and, don't have to think about details, such as polling for infrastructure creation status

The tool that suits you best depends on factors such as your technical experience, existing knowledge in your organization, and use of Astronomer. It's common to use all available tools.

## Error handling with the Astro API

It's a best practice to ensure your code handles unexpected situations properly, such as errors. Because Astro's tooling generally distinguishes between client-side and server-side errors, you can automate a process for determining the cause of an error. All HTTP 4XX code indicate a client-side error and all HTTP 5XX status codes indicate a server-side error.

For example, Deployment names are unique within an Astronomer Workspace. If you create a second Deployment with the same name as an already existing Deployment, the Astro API returns an HTTP 400 error.

To ensure your code handles errors correctly, wrap requests in a `try`/`except` code block:

```python expandable wrap theme={null}
import requests

organization_id = "<your-organization-id>"
workspace_id = "<your-workspace-id>"
astro_api_token = "<your-astro-bearer-token>"

try:
    # Create a Deployment
    # https://www.astronomer.io/docs/astro-api/platform-api-reference/deployment/create-deployment
    response = requests.post(
        f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments",
        headers={"Authorization": f"Bearer {astro_api_token}"},
        json={
            "astroRuntimeVersion": "3.1-5",
            "defaultTaskPodCpu": "0.25",
            "defaultTaskPodMemory": "0.5Gi",
            "executor": "CELERY",
            "isCicdEnforced": False,
            "isDagDeployEnabled": False,
            "isHighAvailability": False,
            "name": "my_deployment",  # <== this will fail if "my_deployment" already exists
            "resourceQuotaCpu": "10",
            "resourceQuotaMemory": "20Gi",
            "schedulerSize": "SMALL",
            "type": "STANDARD",
            "workspaceId": workspace_id,
        },
    )
    response.raise_for_status()
except requests.exceptions.HTTPError as e:
    print("Failed creating deployment. Reason: " + e.response.json()["message"])
    raise e
```

The statement, `response.raise_for_status()`, raises an exception on any HTTP response code that's not `2XX`, which are all non-successful HTTP response codes. In the `except` clause, you can handle this exception however you want.

In case of an error, the Astro API returns a reason in the response body, with the key `message`. For this specific example, it returns HTTP code 400. This code example prints the error reason. Without the reason, you can't know why a request failed. In the example of a duplicated Deployment name, the logs show the following:

```text wrap theme={null}
Failed creating Deployment. Reason: Invalid request: Deployment name 'my_deployment' already exists in this workspace
```

The complete error response structure is:

```json wrap theme={null}
{
    "message": "Invalid request: Deployment name 'my_deployment' already exists in this workspace",
    "requestId": "f004d12a-29c8-40d8-b239-2b1b615ea45b",
    "statusCode": 400
}
```

For traceability purposes, you can also include the `requestId` in your logs, which is an internal Astronomer identifier that Astronomer support can use to track down your request.

## Authenticating scripts using API tokens

In automated scripts, such as CI/CD pipelines, you can query the Astro API with an API token. API tokens grant access to certain Astronomer resources, so it's important to keep the token safe. **Do not hardcode the token in code.** Instead, store the token in a secret and expose it as an environment variable, `ASTRO_API_TOKEN`. Storing API tokens in a system dedicated for storing secret values, like [GitHub Actions Secrets](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#creating-secrets-for-a-repository), ensures secret values are not visible to humans and only referenced by code when needed.

Additionally, a best security practice is the [*Principle of Least Privilege*](https://en.wikipedia.org/wiki/Principle_of_least_privilege), where you grant only the permissions necessary to perform an action. This reduces the attack surface, or the number of ways a bad actor could cause damage, in case of a leaked API token.

Astronomer provides three levels of API tokens, from least to most privilege:

* [Deployment API tokens](/docs/astro/deployment-api-tokens)
* [Workspace API tokens](/docs/astro/workspace-api-tokens)
* [Organization API tokens](/docs/astro/organization-api-tokens)

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>
Consider [custom Deployment roles](/docs/astro/customize-deployment-roles) for configuring Deployment-level roles with only the necessary permissions.

## Use a graphical REST API client for development

The [Astro API documentation](https://www.astronomer.io/docs/astro/api) provides a convenient web interface to try out the API:

<Frame>
  <img alt="Screenshot of the Astro API documentation" />
</Frame>

A graphical REST API client can be a helpful addition when developing with any REST API. Popular tools include [Postman](https://www.postman.com) and [Insomnia](https://insomnia.rest/products/insomnia). The Astro API documentation provides [downloads to the API specifications](/docs/astro/api/v-1/overview#download-openapi-specification), which are YAML files that define the API structure, that you can load into your tool of choice. Graphical REST API clients often provide convenience features over web-based documentation including query history, value sharing with variables, the ability to define environments with different values, and chained requests, which lets you use results from one query in a follow-up query.

## Handle rate limiting with exponential backoff retries

The [Astro API limits requests](/docs/astro/api/v-1/overview#rate-limiting) in case the request rate passes certain thresholds, depending on the type of the request. When a request is rate limited, the API returns an `HTTP 429` status code.

It's a best practice to apply an exponential backoff strategy to not overload the server side for rate-limited requests. An exponential backoff strategy gradually increases the time between requests to allow for the server to recover and respond correctly, for example, by waiting `1`, `2`, `4`, or `8` seconds between consecutive requests.

Consider the scenario of waiting for a Deployment to receive status `HEALTHY` after creation. Creating a Deployment can take a moment, so repeatedly requesting the status from the Astro API without any pause between requests can cause rate limiting.

The following example checks the HTTP response code and handle HTTP 429 separately from other response codes. While this script won't hit any rate limit threshold on the Astro API since the code waits 5 seconds between requests, running multiple scripts simultaneously can quickly increase requests. When receiving an HTTP 429 response, it calculates a waiting period of `2 ** (timeout_attempts - 1)` seconds and then increases the period depending on the attempt number.

```python {14-19,32-36} expandable wrap theme={null}
from datetime import datetime
import requests
import time

organization_id = "<your-organization-id>"
deployment_id = "<your-deployment-id>"
astro_api_token = "<your-astro-bearer-token>"

timeout_secs = 600
timeout_attempts = 0
start = datetime.now()
while True:
    try:
        response = requests.get(
            f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}",
            headers={"Authorization": f"Bearer {astro_api_token}"},
        )
        response.raise_for_status()
        timeout_attempts = 0

        deployment_status = response.json()["status"]
        if deployment_status == "HEALTHY":
            print("Deployment is healthy")
            break
        if (datetime.now() - start).total_seconds() > timeout_secs:
            raise Exception("Timeout")
        else:
            print(f"Deployment status is currently {deployment_status}. Waiting...")
            time.sleep(5)

    except requests.exceptions.HTTPError as e:
        if e.response.status_code == 429:
            timeout_attempts += 1
            sleep_duration = 2 ** (timeout_attempts - 1)
            print(f"Request was rate limited. Sleeping {sleep_duration} seconds and trying again.")
            time.sleep(sleep_duration)
        else:
            print("Failed fetching deployment status. Reason: " + e.response.json()["message"])
            raise e
```

In the previous example, the code defines custom logic for handling rate limits and exponential backoffs. While this works, [Python's requests library](https://requests.readthedocs.io) comes with several built-in utilities you can use to simplify the code. For example, you can avoid defining your own exponential backoff logic by using the Python `requests` library.

`requests.session()` creates a persistent session between requests and replaces `requests.get(...)` with `session.get(...)` to use the settings configured in the session. This way you don't have to duplicate the same `if e.response.status_code == 429` business logic for every request. The code example below configures the `urllib3.Retry` exponential backoff logic for HTTP status code 429 and up to 10 attempts. This means it waits `1`, `2`, `4`, ..., `128`, `256`, and `512` seconds in between the ten attempts.

```python wrap theme={null}
import requests
from requests.adapters import HTTPAdapter
from urllib3 import Retry

session = requests.session()
ratelimit_retry = Retry(status_forcelist=[429], backoff_factor=1, total=10)
session.mount(prefix="https://api.astronomer.io", adapter=HTTPAdapter(max_retries=ratelimit_retry))
response = session.get(...)
```

You can set a default value for `headers` using Python's request library as a way to simplify your code. While you can write `headers={"Authorization": f"Bearer {astro_api_token}"}` with every request, it's cleaner to define this value once and then automatically apply it to every request using the `session` object:

```python wrap theme={null}
import requests

session = requests.session()
session.headers = {"Authorization": f"Bearer {astro_api_token}"}

# Before
requests.get(
    f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}",
    headers={"Authorization": f"Bearer {astro_api_token}"},
)

# After
session.get(f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}")
```

## Reuse HTTP connections

For repeated requests to the Astro API, or any REST API, it's a best practice to reuse connections. That means you (client) keep a connection open to the Astro API (server) for multiple requests, instead of opening and closing a connection for every request. This reduces latency and CPU usage. Reusing HTTP connections is also referred to as [HTTP persistent connections or HTTP keep-alive](https://en.wikipedia.org/wiki/HTTP_persistent_connection), where keep-alive refers to the [Keep-Alive](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Keep-Alive) header that [used to be](https://datatracker.ietf.org/doc/html/rfc9113#section-8.2.2) transmitted with the message.

Using Python's requests library again, `requests.session` object reuses connections:

```python wrap theme={null}
import requests

session = requests.session()
session.headers = {"Authorization": f"Bearer {astro_api_token}"}
session.get(url=f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}")
session.get(url=f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}")
```

A single connection then handles the two `GET` requests, instead of recreating a connection. This becomes visible when configuring `DEBUG` logging:

```python {10,12,21} wrap theme={null}
import logging
import requests

logging.basicConfig(level=logging.DEBUG)

# Before
requests.get(url=f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}", headers={"Authorization": f"Bearer {astro_api_token}"})
requests.get(url=f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}", headers={"Authorization": f"Bearer {astro_api_token}"})

# DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.astronomer.io:443
# DEBUG:urllib3.connectionpool:https://api.astronomer.io:443 "GET /platform/v1beta1/organizations/clkvh3b46003m01kbalgwwdcy/deployments/clw9bhr2n0d9801gp7zuigsqn HTTP/11" 200 None
# DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.astronomer.io:443
# DEBUG:urllib3.connectionpool:https://api.astronomer.io:443 "GET /platform/v1beta1/organizations/clkvh3b46003m01kbalgwwdcy/deployments/clw9bhr2n0d9801gp7zuigsqn HTTP/11" 200 None

# After
session = requests.session()
session.headers = {"Authorization": f"Bearer {astro_api_token}"}
session.get(url=f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}")
session.get(url=f"https://api.astronomer.io/platform/v1beta1/organizations/{organization_id}/deployments/{deployment_id}")

# DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.astronomer.io:443
# DEBUG:urllib3.connectionpool:https://api.astronomer.io:443 "GET /platform/v1beta1/organizations/clkvh3b46003m01kbalgwwdcy/deployments/clw9bhr2n0d9801gp7zuigsqn HTTP/11" 200 None
# DEBUG:urllib3.connectionpool:https://api.astronomer.io:443 "GET /platform/v1beta1/organizations/clkvh3b46003m01kbalgwwdcy/deployments/clw9bhr2n0d9801gp7zuigsqn HTTP/11" 200 None
```

In the logs, you can see the first example creates two connections. The second example uses `requests.session`, which reuses connections, so it only creates one connection.

## See also

* [Astro API documentation](https://www.astronomer.io/docs/astro/api)
* [Make requests to the Airflow REST API](/docs/astro/airflow-api)
* [Python requests documentation](https://requests.readthedocs.io)


# Manage Astro connections in branch-based deploy workflows
Source: https://astronomer.io/docs/astro/best-practices/connections-branch-deploys

Manage Airflow connections across branch-based Astro Deployments using the Environment Manager.

Airflow Dags often interact with a multitude of external systems, such as data warehouses and APIs. Dags access these systems using [Airflow connections](/docs/astro/manage-connections-variables). A common logistical consideration when running Airflow at scale is deciding how to manage connections between development and production environments. Different environment types require different levels of access to external resources.

Astro's [branch-based development](/docs/astro/automation-overview) and [connection management](/docs/astro/manage-connections-variables) features allow you to automatically share specific Airflow connections with Astro Deployments based on their development context.

When you combine branch-based deploys and connection management, you can:

* Automatically create Deployments for development branches with a set of development Airflow connections.
* Override preset Airflow connections on a per-Deployment basis to troubleshoot or customize your development environment.
* Run a development and production Deployment for the same project without changing Airflow connection names between the two contexts.

## Feature overview

This use case depends on the following Astro features to create a fully integrated CI/CD pipeline:

* Configuring Airflow connections in the [Astro Environment Manager](/docs/astro/manage-connections-variables).
* Branch-based, or multi-branch Deployments using [CI/CD](/docs/astro/set-up-ci-cd#multiple-environments).

## Prerequisites

This use case assumes you have:

* At least two Astro Deployments, one for development and one for production.
* A Git-based repository where you manage an Astro project.
* A configured multi-branch CI/CD pipeline. See [CI/CD templates](/docs/astro/ci-cd-templates/template-overview).

However, you can extend this use case to encompass multiple development or production environments.

## Implementation

To implement this use case:

1. As a Workspace Operator or Admin, create connections in the Astro Environment manager to your development-level resources. See [Create a connection](/docs/astro/create-and-link-connections#create-a-connection).
2. Set these connections to be available to all Deployments in the Workspace by default. See [Configure connection sharing for a Workspace](/docs/astro/create-and-link-connections#configure-connection-sharing-for-a-workspace). Your Dag authors can now access external development and testing resources across all Deployments.
3. Choose a Deployment in your Workspace that you want to run your production workflows. In the Deployment, override the connections you configured to instead connect to your production resources. See [Override connection fields](/docs/astro/create-and-link-connections#override-connection-fields).
4. In the Git repository for your Astro project, define a multi-branch CI/CD pipeline for deploying to Astro. For an example of how to do this in Astro, see the [Astro GitHub integration](/docs/astro/deploy-github-integration).

Now, when a Dag author deploys to either a production or development Deployment through CI/CD, they can run their Dags in Astro without needing to configure connections. Additionally, because your production connection shares the same connection ID as your default development connection, Dag authors don't have to update their code to point towards different connections when promoting their code to production.

## Explanation

Using branch-based Deployments with the Astro Environment Manager allows your team to focus on the parts of Astro that matter most for their roles. For example, using the Astro Environment Manager means that you only need one administrative user to manage connections across multiple Deployments:

* A Workspace Owner [creates an Airflow connection](/docs/astro/create-and-link-connections#create-a-connection) in the Astro Environment Manager that connects to external development resources. They share this connection to all Deployments by default by turning on the [**Linked to all Deployments** setting](/docs/astro/create-and-link-connections#configure-connection-sharing-for-a-workspace).
* In the production Deployment, the Workspace Owner [overrides the linked connection](/docs/astro/create-and-link-connections#override-connection-fields) to instead connect to production resources on the same external system. Because the connection ID and code are the same, this override requires no updates at the Dag level.

After a Workspace Owner creates connections, Dag authors can develop Dags without needing to reconfigure connections between development and production:

* A Dag author creates a new development branch of an Astro project. The CI/CD pipeline for the repository deploys their branch to the development Deployment on Astro. This is known as a [multi-branch CI/CD pipeline](/docs/astro/set-up-ci-cd#multiple-environments).
* The Dag author has access to development resources because the Workspace Author configured a default Airflow connection for the Deployment. The author can [pull this connection onto their local machine](/docs/cli/v1.43/local-connections) for testing purposes, or test by deploying to the development Deployment on Astro.
* When the Dag author finishes development, they merge their development branch into production. The CI/CD pipeline deploys this change to the production Deployment.
* When the Dag author's code runs in the production Deployment, it now accesses production resources based on the overrides configured by the Workspace Owner.

This use case provides several benefits for both Workspace managers and Dag authors:

* When you use branch-based deploys, your CI/CD pipeline automatically deploys your branches to development Deployments. This saves resources and reduces the complexity of development for Dag authors.
* Workspace Operators and Owners can manage connections without needing access to Dag code.
* Dag authors only need a connection ID to connect their dags, meaning they can focus on data engineering instead of connection configuration.
* Connection IDs don't need to be updated when you promote code to production, reducing development timelines and reducing the number of resources to manage.

## See also

* [Manage Airflow connections and variables](/docs/astro/manage-connections-variables)
* [Automate actions on Astro](/docs/astro/automation-overview)


# Cross-deployment dependencies
Source: https://astronomer.io/docs/astro/best-practices/cross-deployment-dependencies

How to implement dependencies between your Airflow deployments.

[Cross-Dag dependencies](/docs/learn/cross-dag-dependencies) serve a common use case: configuring a Dag to run when a separate Dag or a task in another Dag completes or updates an asset. But what about situations in which an asset or Dag you monitor exists in a separate deployment? For example, you might want to make a task dependent on an asset update in a Dag that is owned by a different team and located in a separate deployment. Astro also supports the orchestration of tasks using this kind of relationship, which is referred to as a *cross-deployment dependency*.

This guide uses the following terms to describe cross-deployment dependencies:

* **Upstream deployment**: A deployment where a Dag must reach a specified state before a Dag in another deployment can run.
* **Downstream deployment**: A deployment in which a Dag cannot run until a Dag in an upstream deployment reaches a specified state.

Cross-deployment dependencies require special implementation because some methods, like the `TriggerDagRunOperator`, `ExternalTaskSensor`, and direct Airflow Asset dependencies, are only designed for Dags in the same deployment. On Astro, there are two recommended methods available for implementing cross-deployment dependencies: Astro Alerts and triggering updates to Airflow Assets using the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/create_asset_event).

## Feature overview

In this guide, you'll learn when to use the following Astro and Airflow features to create dependencies across Astro deployments. Astro supports cross-deployment dependencies in any Workspace or cluster.

* [Astro Alerts](/docs/astro/alerts). Recommended for most Astro use cases as no code modification is required.
* [Airflow Assets](/docs/learn/airflow-datasets). Can be used to trigger a Dag after a task in another Dag updates an asset.

## Best practice guidance

Astro Alerts and Airflow Assets are the best methods for implementing cross-deployment dependencies. The `dagRuns` endpoint of the Airflow API can also be used for this purpose and might be appropriate in cases where you want tasks that don't update assets to trigger Dags. This method isn't covered here, but you can implement it by following the guidance in [Airflow REST API](/docs/astro/airflow-api#trigger-a-dag-run).

To determine whether an Astro Alert or the Assets feature is the right solution for your use case, consider the following guidance.

**Astro Alerts:**

You can use Astro alerts to implement cross-deployment Dag dependencies using the Dag trigger communication channel. They are simple to implement and are the preferred method in the following situations:

* If you need to implement a dependency trigger based on any Dag state other than success, such as a Dag failure, a task taking longer than expected, or a Dag not completing by a certain time.
* If you need to implement a simple one-to-one cross-deployment dependency (one upstream Dag triggers one downstream dag) and do not want to update your Dag code.
* When your Dags don't already use the Assets feature and when it is easy to identify the relevant dependent dags, which isn't always the case in larger organizations.

**Airflow Assets:**

Assets represent a significant evolution in the way Airflow can be used to define dependencies and, for some, offer a more natural way of expressing pipelines than traditional dags. Assets, which offer more flexibility for cross-deployment dependencies than Astro alerts, are the preferred method in the following scenarios:

* You need to implement dependencies in a many-to-one pattern, so you can make a Dag dependent on the completion of multiple other Dags or tasks. This is not possible using Astro Alerts.
* You need to implement dependencies in a one-to-many pattern, so one Dag or task triggers multiple Dags or tasks. While this is also possible using Astro Alerts, it requires a separate alert for each dependency.

<Tip>In Airflow 3, Datasets were renamed to Assets. You can update an asset with a `POST` request to the [assets endpoint of the Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/create_asset_event), which supports implementation across deployments. This guide uses the Airflow REST API v2 (`/api/v2/assets/events`) available in Airflow 3.</Tip>

## Astro alerts example

### Assumed knowledge

To use Astro Alerts to create cross-deployment dependencies, you should have an understanding of:

* Airflow dags. See [Introduction to Airflow dags](/docs/learn/dags).
* Creating and managing Astro deployments. See [Create a deployment](/docs/astro/create-deployment).

### Implementation

This example shows you how to create dependencies between Dags in different Astro deployments using Astro Alerts.

#### Prerequisites

* Two [Astro Deployments](/docs/astro/create-deployment), each containing at least one dag.

#### Process

Create a dependency between Dags in separate deployments with an alert trigger on Astro.

1. First, [create a Deployment API token](/docs/astro/deployment-api-tokens) for the upstream Deployment.
2. Click [**Alerts**](/docs/astro/alerts) in the Workspace menu, and create a new alert.
3. Enter an **Alert name** that you'll remember, select the **Alert type** (like **Dag Success**), and then select **Dag Trigger** as the **Communication Channel**.
4. In the **Deployment** menu for the **Dag Trigger** communication channel, select the downstream deployment from the list.
5. In the **Dag NAME** menu, select the Dag you want to trigger.
6. Paste your API token in the **DEPLOYMENT API TOKEN** field.
7. Run the upstream dag, verify that the alert triggers, and confirm the downstream Dag runs as expected.

## Assets example

### Assumed knowledge

To use Airflow Assets to create cross-deployment dependencies, you should have an understanding of:

* Airflow dags. See [Introduction to Airflow dags](/docs/learn/dags).
* [Airflow Assets](/docs/learn/airflow-datasets).
* The [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html).

### Implementation

This section explains how to use the [Airflow REST API v2 `create_asset_event` endpoint](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/create_asset_event) to trigger the downstream Dag in another deployment when an asset is updated. Typical asset implementation only works for Dags in the same Airflow deployment, but by using the Airflow REST API, you can implement this pattern across deployments.

The `create_asset_event` endpoint identifies the target asset by its integer `asset_id`, which Airflow assigns automatically when the asset is first referenced in the downstream deployment. This example looks up the `asset_id` at runtime by querying the assets endpoint with the asset's URI, so you don't need to hard-code the identifier.

#### Prerequisites

* Two [Astro Deployments](/docs/astro/create-deployment).
* A [Deployment API token](/docs/astro/deployment-api-tokens), [Workspace API token](/docs/astro/workspace-api-tokens), or [Organization API token](/docs/astro/organization-api-tokens) for one of your deployments. This deployment will host your downstream dag.
* Two [Astro projects](/docs/cli/v1.43/develop-project#create-an-astro-project).

#### Process

1. In your upstream Deployment, which is the Deployment for which you did **not** create an API Token, in the Deployment's **Environment Variables** tab in your Deployment's **Environment** settings, create an environment variable for your API token and use `API_TOKEN` for the key.
2. For your downstream Deployment, follow the guidance in [Make requests to the Airflow REST API - Step 2](/docs/astro/airflow-api#step-2-retrieve-the-deployment-url) to obtain the Deployment URL for your downstream Deployment. The Deployment URL should be in the format of `clq52ag32000108i8e3v3acml.astronomer.run/dz3uu847`.
3. In your upstream Deployment, use Variables in the Astro UI to create an environment variable where you can store your downstream Deployment URL, using `DEPLOYMENT_URL` for the key.
4. In the upstream Deployment, add the following Dag to your Astro project running in the upstream Deployment. The `get_bear` task declares `MY_ASSET` as an outlet, which produces an asset event for `MY_ASSET` in the *same* Airflow deployment on successful completion. The dependent `update_asset_via_api` task produces an asset event for the same-named asset in a *different* Airflow deployment by calling the Airflow REST API v2 `create_asset_event` endpoint.

```python expandable wrap theme={null}
from airflow.sdk import Asset, dag, task
from pendulum import datetime
import os
import requests

URI = "file://include/bears"
MY_ASSET = Asset(URI)
TOKEN = os.environ.get("API_TOKEN")
DEPLOYMENT_URL = os.environ.get("DEPLOYMENT_URL")

@dag(
    start_date=datetime(2023, 12, 1),
    schedule="0 0 * * 0",
    catchup=False,
    doc_md=__doc__,
)
def producer_dag():
    @task(outlets=[MY_ASSET])
    def get_bear():
        print("Update the bears asset")

    @task
    def update_asset_via_api():
        headers = {
            "Authorization": f"Bearer {TOKEN}",
            "Content-Type": "application/json",
            "Accept": "application/json",
        }

        # Look up the asset_id in the downstream deployment by URI.
        # Airflow assigns the integer asset_id when the asset is first
        # referenced in that deployment.
        lookup = requests.get(
            url=f"https://{DEPLOYMENT_URL}/api/v2/assets",
            headers=headers,
            params={"uri_pattern": URI},
        )
        lookup.raise_for_status()
        assets = lookup.json().get("assets", [])
        if not assets:
            raise RuntimeError(
                f"No asset with URI {URI} found in the downstream deployment."
            )
        asset_id = assets[0]["id"]

        # Create an asset event in the downstream deployment.
        payload = {
            "asset_id": asset_id,
            "extra": {},
        }
        response = requests.post(
            url=f"https://{DEPLOYMENT_URL}/api/v2/assets/events",
            headers=headers,
            json=payload,
        )
        response.raise_for_status()
        print(response.json())

    get_bear() >> update_asset_via_api()

producer_dag()

```

5. Deploy the project to Astro.
6. Deploy a Dag of any kind to your downstream Deployment. In this dag, use a `dag_id` of `consumer_dag` and schedule it on the same asset as the `producer_dag`. Deploying this Dag registers the asset in the downstream deployment and assigns it an `asset_id` that the upstream producer can look up at runtime.

For example:

```python wrap theme={null}
from airflow.sdk import Asset, dag, task
from datetime import datetime

URI = "file://include/bears"

@dag(
    dag_id="consumer_dag",
    start_date=datetime(2023, 12, 1),
    schedule=[Asset(URI)],
    catchup=False,
    doc_md=__doc__,
)
def consumer_dag():
    @task
    def wait_for_bears():
        print("The bears are here!")

    wait_for_bears()

consumer_dag()

```

After deploying both projects to their respective Deployments on Astro, runs of your `producer_dag` trigger your `consumer_dag` automatically. If the downstream Dag isn't firing, verify that the upstream Deployment's environment variables, the API token and deployment URL, correspond to the downstream Deployment's API Token and Deployment URL. A successful request to the Airflow REST API emits a payload containing the newly created asset event. Check the task logs for output that looks similar to the following:

```text wrap theme={null}
[2025-06-02, 11:21:06 UTC] {logging_mixin.py:188} INFO - {'id': 4, 'asset_id': 1, 'uri': 'file://include/bears', 'extra': {'from_rest_api': True}, 'source_task_id': None, 'source_dag_id': None, 'source_run_id': None, 'source_map_index': -1, 'created_dagruns': [], 'timestamp': '2025-06-02T11:21:06.252976+00:00'}
```

## See also

* Creating [cross-Dag dependencies](/docs/learn/cross-dag-dependencies).
* Setting up [alerts](/docs/astro/alerts).


# Dag writing on Astro
Source: https://astronomer.io/docs/astro/best-practices/dag-writing-on-astro

Follow best practices for writing Airflow dags that take full advantage of Astro features.

You can run any valid Airflow Dag on Astro.

That said, following best practices and taking full advantage of Astro features will help you:

* Write Dags more efficiently.
* Improve your development workflow.
* Better organize your deployment and Dags.
* Optimize resource usage and task execution efficiency.

This guide describes best practices for taking advantage of Astro features when writing Dags. While this focuses on writing Airflow Dags for Astro, all [general Airflow best practices](/docs/learn/dag-best-practices) are also recommended.

<Info>
  **New to Airflow?**

  If you are new to Airflow, Astronomer suggests starting with the following resources:

  * Hands-on tutorial: [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
  * Astronomer Academy: [Airflow 101 Learning Path](https://academy.astronomer.io/path/airflow-101).
  * Webinar: [Airflow 101: How to get started writing data pipelines with Apache Airflow](https://www.astronomer.io/events/webinars/airflow-101-how-to-get-started-writing-data-pipelines-with-apache-airflow-video/).
</Info>

## Feature overview

In this guide, you'll learn about a number of Astro and Airflow features for optimizing your Dags for Astro:

* [Astro CLI](/docs/cli/v1.43/overview). An OSS tool for developing Dags and deploying projects tailor-made for Astro.
* [Worker queues](/docs/astro/configure-worker-queues). A feature of Astro that enables resource optimization beyond what is possible using OSS Airflow.
* [Astro Cloud UI Environment Manager](/docs/astro/manage-connections-variables). A tool for managing connections across multiple deployments locally and on Astro.
* [Astro alerts](/docs/astro/alerts). No-code alerting on Astro configurable for various trigger types and communication channels.

## Best practice guidance

The guidance in this section is split into two categories: writing Dag code for Astro and managing Dag code on Astro.

### Write and develop Dags for Astro

Astro users have several tools available when it comes to developing and writing Dags: the Astro CLI, the TaskFlow API, Dag Factory, and Cosmos:

* The Astro CLI is recommended for those who want to develop code locally in a containerized setup using their preferred IDE. It is especially useful for those migrating existing Dags to Astro.
* The [TaskFlow API](/docs/learn/airflow-decorators) is a functional API for using decorators to define Dags and tasks, which simplifies the process for passing data between tasks and defining dependencies. The TaskFlow API simplifies the Dag authoring experience by eliminating the boilerplate code required by traditional operators.
* [Dag Factory](https://github.com/astronomer/dag-factory) is an open source tool for dynamic Dag generation from YAML files that makes it easy for people who don't know Python to use Airflow. See the [Dag Factory Documentation](https://astronomer.github.io/dag-factory/latest/) for more information.
* [Cosmos](https://github.com/astronomer/astronomer-cosmos) is an open source tool that turns dbt workflows into Airflow Dags. See the [Cosmos documentation](https://astronomer.github.io/astronomer-cosmos/) for more information.

See [Dags](/docs/learn/dags) for detailed information on structuring your Airflow data pipelines.

#### Use the Astro CLI for local development

When writing Dags intended for Astro Deployments, use the [Astro CLI](/docs/cli/v1.43/install-cli) for containerized local development. The Astro CLI is an open-source interface you can use to:

* Test Airflow Dags locally.
* Deploy code to Astro.
* Automate key actions as part of a CI/CD process.

To get the most out of the Astro CLI:

* Use the `dags` directory to store your Dags. If you have multiple Dags, you can use subdirectories to keep your environment organized.
* Modularize your code and store all supporting classes, files and functions in the `include` directory.
* Store all your [tests](#best-practice-guidance) in the `tests` directory.

<Tip>If you are unable to install the Astro CLI on your local machine, due to company policy or for other reasons, you can use it in GitHub Codespaces by forking the [Astro CLI Codespaces](https://github.com/astronomer/astro-cli-codespaces) repository.</Tip>

Learn more and get started by following the guidance in [Astro CLI](/docs/cli/v1.43/overview).

### Manage Dag code on Astro

Compared to open-source Airflow, Astro offers a distinct feature set for optimizing your Airflow pipelines. Some of these features impact individual Dags, and you should consider them not only when writing new Dag code but also when migrating Dags to Astro. You don't need to make changes to existing Dags when you migrate them, but by implementing some small changes you can get access to an upgraded experience when it comes to managing connections, defining and storing environment variables, allocating resources, and implementing alerts.

Making use of the following features can require Dag code changes or adding Airflow configuration environment variables in Astro instead of Airflow. Astronomer recommends considering these features during Dag development to streamline the development process and make use of everything Astro has to offer.

* **[Astro Cloud UI Environment Manager](/docs/astro/manage-connections-variables)** for managing connections. This allows you to define connections once and use them in multiple deployments (with or without overrides for individual fields), as well as in your local development environment. Connections in the Environment Manager are stored in the Astronomer managed secrets backend. Using the Environment Manager requires adding the relevant Airflow provider packages to your Astro project and adding configuring the connection in the Astro UI.
* **[Deployment environment variables in the Astro UI](/docs/astro/environment-variables)** for storing environment variables. Common uses of environment variables include adding tokens or URLs required by your Dags, integrating with third-party tooling to export metrics, customizing core settings of Airflow or Airflow Providers, and storing Airflow connections and variables. A number of approaches are supported. For example, you can mark variables as secret for storage in the Astronomer managed secrets backend. You can store [Airflow variables](/docs/learn/airflow-variables), which are Airflow native key-value pairs, as environment variables as well by using the format `AIRFLOW_VAR_MYVARIABLENAME`. You can also manage environment variables using your Astro project Dockerfile and, locally, your project `.env` file. Astro project modification and additional configuration in Astro may be required depending on the strategy chosen.
* **[Worker queues](/docs/astro/configure-worker-queues)** for optimizing task execution efficiency. Worker queues allow you to define sets of workers with specific resources that execute tasks in parallel. You can assign tasks to specific worker queues to optimize resource usage. This feature is an improvement over OSS Airflow pools, which only manage concurrency and not resources. You can use both worker queues and pools in combination. Config in Astro and Dag code modification are required to use this feature.
* **[Astro Alerts](/docs/astro/alerts)** for setting up alerts for your Dags. Astro alerts add another level of observability to Airflow's notification systems. For example, you can configure an alert to notify you in Slack when a Dag run completes or fails. Unlike Airflow callbacks and SLAs, Astro alerts require no changes to Dag code. For more information, see [When to use Airflow or Astro alerts for your pipelines on Astro](/docs/astro/best-practices/airflow-vs-astro-alerts). Config in the Astro UI required.

## See also

* Webinar: [Best practices for managing Airflow across teams](https://www.astronomer.io/events/webinars/best-practices-for-managing-airflow-across-teams-video/).
* Webinar: [Dag writing for data engineers and data scientists](https://www.astronomer.io/events/webinars/dag-writing-for-data-engineers-and-data-scientists-video/).
* OSS Learn guide: [Dag writing best practices in Apache Airflow](/docs/learn/dag-best-practices).


# Use Git submodules with an Astro project
Source: https://astronomer.io/docs/astro/best-practices/git-submodules

Combine code from multiple repositories into a single Astro project using Git submodules.

When your Astro project depends on code maintained in a separate repository, Git submodules let you include that external repository as a nested folder inside your Astro project. This is useful when you want to keep shared code, such as a dbt project or a Python utility library, in its own repository while still deploying it as part of your Astro project.

This document covers how to add a Git submodule to an Astro project, configure CI/CD to deploy with submodules, and test your setup locally.

## When to use Git submodules

Use Git submodules when:

* A separate team maintains code that your Dags depend on, and that code lives in its own repository.
* You want to pin your Astro project to a specific version of an external repository.
* You need to combine code from multiple repositories into a single Astro project before deployment.

A common example is including an external dbt project as a submodule so that [Cosmos](https://astronomer.github.io/astronomer-cosmos/) can orchestrate dbt models as Airflow tasks.

<Note>
  Git submodules add complexity to your development and deployment workflows. If the external code changes infrequently or is small, consider copying the code directly into your Astro project instead.
</Note>

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* An [Astro project](/docs/cli/v1.43/develop-project#create-an-astro-project).
* [Git](https://git-scm.com/downloads) installed on your local computer.
* A remote Git repository containing the code you want to include as a submodule.

## Add a submodule to your Astro project

<Steps>
  <Step title="Add the submodule">
    Run the following command from your Astro project root folder to add the external repository as a submodule:

    ```sh wrap theme={null}
    git submodule add <repository-url> <folder-name>
    ```

    Replace `<repository-url>` with the URL of the external repository and `<folder-name>` with the name of the folder where the submodule code appears in your project. For example, to add a dbt project called `jaffle-shop`:

    ```sh focus={1} wrap theme={null}
    git submodule add https://github.com/jessicaschueler/jaffle-shop-classic jaffle-shop
    ```

    This command creates a `.gitmodules` file in your project root and clones the external repository into the specified folder. The `.gitmodules` file tracks the submodule configuration:

    ```text title=".gitmodules" wrap theme={null}
    [submodule "jaffle-shop"]
    	path = jaffle-shop
    	url = https://github.com/jessicaschueler/jaffle-shop-classic
    ```
  </Step>

  <Step title="Commit the submodule">
    After you add the submodule, commit the changes to your repository:

    ```sh wrap theme={null}
    git add .gitmodules <folder-name>
    git commit -m "Add <folder-name> as a submodule"
    ```
  </Step>

  <Step title="Verify the submodule">
    Confirm the submodule is correctly linked by running the following command:

    ```sh wrap theme={null}
    git submodule status
    ```

    The output displays the commit hash of the submodule and its folder path.
  </Step>
</Steps>

## Example: dbt project with Cosmos

The [cosmos-demo-submod](https://github.com/jessicaschueler/cosmos-demo-submod) repository demonstrates using a Git submodule to include an external dbt project in an Astro project. In this example, the [jaffle-shop](https://github.com/jessicaschueler/jaffle-shop-classic) dbt project is included as a submodule and orchestrated with [Cosmos](https://astronomer.github.io/astronomer-cosmos/).

The project structure looks like the following:

```text wrap theme={null}
astro-project/
├── .gitmodules
├── Dockerfile
├── dags/
│   └── basic_cosmos_dag.py
├── jaffle-shop/         # Git submodule (external dbt project)
│   ├── models/
│   ├── seeds/
│   └── dbt_project.yml
├── requirements.txt
└── packages.txt
```

The `Dockerfile` installs dbt into a virtual environment so that Cosmos can run dbt models:

```dockerfile title="Dockerfile" wrap theme={null}
FROM quay.io/astronomer/astro-runtime:8.8.0

RUN python -m venv dbt_venv && source dbt_venv/bin/activate && \
    pip install --no-cache-dir dbt-postgres==1.5.4 && deactivate
```

The `requirements.txt` file includes the Cosmos package:

```text title="requirements.txt" wrap theme={null}
astronomer-cosmos>=1.0.2
```

## Clone an Astro project that contains submodules

When you clone a repository that contains submodules, the submodule folders are empty by default. To initialize and fetch the submodule contents, use one of the following methods:

* Clone the repository with the `--recurse-submodules` flag:

  ```sh wrap theme={null}
  git clone --recurse-submodules <repository-url>
  ```

* If you already cloned the repository without the flag, initialize the submodules manually:

  ```sh wrap theme={null}
  git submodule init
  git submodule update
  ```

## Update a submodule to the latest commit

When the external repository has new changes that you want to pull into your Astro project, update the submodule:

<Steps>
  <Step title="Pull the latest changes">
    Run the following command from your Astro project root folder:

    ```sh wrap theme={null}
    git submodule update --remote <folder-name>
    ```

    This updates the submodule to the latest commit on its default branch.
  </Step>

  <Step title="Commit the updated reference">
    After you update the submodule, your Astro project repository tracks a new commit hash for it. Commit this change:

    ```sh wrap theme={null}
    git add <folder-name>
    git commit -m "Update <folder-name> submodule to latest"
    ```
  </Step>
</Steps>

## Configure CI/CD for submodules

When you deploy an Astro project that contains submodules, your CI/CD pipeline must clone the submodule contents before running `astro deploy`. Without this step, the submodule folders are empty and the deploy fails.

### GitHub Actions

Add the `submodules` option to your checkout step:

```yaml title=".github/workflows/deploy.yml" focus={5} wrap theme={null}
steps:
  - name: Checkout repository
    uses: actions/checkout@v4
    with:
      submodules: recursive
```

If your submodule is in a private repository, configure authentication by adding a personal access token (PAT) or deploy key:

```yaml title=".github/workflows/deploy.yml" focus={5-6} wrap theme={null}
steps:
  - name: Checkout repository
    uses: actions/checkout@v4
    with:
      submodules: recursive
      token: ${{ secrets.GIT_PAT }}
```

### GitLab CI/CD

Set the `GIT_SUBMODULE_STRATEGY` variable in your `.gitlab-ci.yml` file:

```yaml title=".gitlab-ci.yml" focus={2} wrap theme={null}
variables:
  GIT_SUBMODULE_STRATEGY: recursive
```

### Other CI/CD tools

For other CI/CD tools, run the following commands after cloning the repository and before deploying:

```sh wrap theme={null}
git submodule init
git submodule update --recursive
```

## Pin a submodule to a specific branch

By default, a submodule tracks a specific commit. To configure it to follow a specific branch, run the following command:

```sh wrap theme={null}
git config -f .gitmodules submodule.<folder-name>.branch <branch-name>
```

After setting the branch, run `git submodule update --remote` to pull the latest commit from that branch.

## Troubleshoot submodules

### Empty submodule folder after cloning

If the submodule folder is empty after you clone the Astro project, run:

```sh wrap theme={null}
git submodule init && git submodule update
```

### Submodule points to the wrong commit

If the submodule shows an unexpected version of the code, check the commit hash with `git submodule status`. To update the submodule to the latest commit:

```sh wrap theme={null}
git submodule update --remote <folder-name>
git add <folder-name>
git commit -m "Update <folder-name> to latest commit"
```

### CI/CD deploy fails with missing files

Ensure your CI/CD pipeline includes a recursive submodule checkout. See [Configure CI/CD for submodules](#configure-ci/cd-for-submodules).

## See also

* [Choose a code repository strategy](/docs/astro/best-practices/repo-structure)
* [Deploy code to Astro](/docs/astro/deploy-code)
* [Develop a CI/CD workflow](/docs/astro/set-up-ci-cd)
* [Cosmos documentation](https://astronomer.github.io/astronomer-cosmos/)
* [cosmos-demo-submod example repository](https://github.com/jessicaschueler/cosmos-demo-submod)


# Attribute Astro spend across teams
Source: https://astronomer.io/docs/astro/best-practices/internal-chargeback

Use Organization dashboard data to attribute Astro spend across teams, Workspaces, and Deployments.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

Does your platform team own the Astro contract while other teams use Astro through their own Workspaces or Deployments? If so, you need a repeatable way to split that shared cost across teams. [Organization dashboards](/docs/astro/organization-dashboard) show you Astro credit usage and who's using it, but they don't build a chargeback report for you.

This guide walks you through turning that dashboard data into a chargeback report. It covers the following steps:

* Choose how you attribute credit usage to teams: by Workspace, by Deployment, or both.
* Adjust reported credit usage for your contract's discount, since dashboards report list-price figures by default.
* Attribute Workspace and Deployment credit usage using the Cost Breakdown dashboard.
* Attribute shared infrastructure credit usage, like a dedicated cluster that multiple Workspaces share.
* Account for teams with different usage patterns. Some teams run Dags often, and others run them rarely.
* Automate recurring chargeback reporting instead of pulling data manually each cycle.
* Pull the same credit usage data programmatically using the Billing API.

Repeat this process each billing cycle to keep your chargeback reports consistent as teams, Workspaces, and Deployments change.

## Relevant dashboards

* Use the [Cost Breakdown dashboard](/docs/astro/organization-dashboard#cost-breakdown) to see Astro credit usage broken down by Workspace, Deployment, compute type, worker queue, and dedicated cluster.
* Use the [Organization Overview and Operator Use dashboards](/docs/astro/organization-dashboard) to compare how much each team actually uses Astro.
* Use [Organization dashboard exports](/docs/astro/org-dash-exports) to schedule recurring chargeback reports instead of downloading data manually.

<Note>All dashboard and export figures are in USD.</Note>

## Prerequisites

This guide assumes that you have:

* Organization Billing Admin [user permissions](/docs/astro/user-permissions#organization-roles).
* At least one [Astro Workspace](/docs/astro/manage-workspaces) per team you want to attribute credit usage to, or an equivalent naming convention that maps Workspaces or Deployments to teams.

## Build your chargeback report

<Steps>
  <Step title="Choose your attribution model">
    Decide how you map credit usage data to teams before you pull any reports:

    * Attribute credit usage by Workspace: base each team's credit usage on the Workspace or Workspaces that team owns. This works well when each team has dedicated Workspaces.
    * Attribute credit usage by Deployment: attribute credit usage at the Deployment level when a single Workspace contains Deployments that belong to different teams.
    * Attribute credit usage by both: use Workspace as the primary attribution unit and Deployment as a secondary breakdown when a team needs credit usage detail for individual pipelines.

    The model you choose determines which tab you use in the next step.
  </Step>

  <Step title="Adjust for your contract discount">
    The Cost Breakdown, Clusters, Worker Queues, and Underlying Data dashboards all report credit usage at Astronomer's list price for your Organization's tier. If your contract includes a discount, that discount doesn't appear in these figures.

    1. On the **Contract Details** tab, use the **Viewing As** control to note the **Current Contract Balance** under both **Cost Basis** and **Credits**.
    2. Divide the **Cost Basis** balance by the **Credits** balance to calculate your discount factor. For example, a Cost Basis balance of \$49,500 against a Credits balance of \$50,000 gives a discount factor of 0.99, meaning your contract has a 1% discount.
    3. Multiply the credit usage figures from these dashboards by your discount factor before you charge a team.

    <Note>Recalculate your discount factor each time you build a chargeback report, since it can change at contract renewal. See [Track credit usage against your contract](#track-credit-usage-against-your-contract) for more on monitoring your overall contract balance.</Note>
  </Step>

  <Step title="Attribute Workspace and Deployment credit usage">
    1. In the Astro UI, click **Dashboards**, then click the **Cost Breakdown** tab.
    2. Use the **Time Period**, **Workspace Name**, and **Deployment Name** filters to scope the dashboard to the billing period you're reporting on.
    3. Review the **Workspace Cost** chart to compare total credit usage across teams at a glance.
    4. Under **Detailed Costs**, click **Deployments** to see credit usage broken down by Workspace Name, Deployment Name, and billable item. Click **Compute Types** for the same breakdown by compute type, if a team needs to understand what's driving their compute usage.

    <Tip>The **Detailed Costs** tables include Workspace Name and Deployment Name filters, so you can isolate a single team's credit usage before exporting it to share with that team.</Tip>
  </Step>

  <Step title="Attribute shared infrastructure credit usage">
    Dedicated clusters are often shared by multiple Workspaces, so their credit usage doesn't attribute directly to a single team through the Deployments or Compute Types breakdowns.

    1. On the **Cost Breakdown** tab, under **Detailed Costs**, click **Clusters**.
    2. Review the credit usage breakdown by Cluster Name and Workspace Name, which splits cluster credit usage into dedicated cluster cost, disaster recovery cluster cost, disaster recovery replication cost, and network cost.
    3. Sum the dedicated cluster cost and network cost for each cluster. Network chargeback cost comes from the dedicated cluster, so allocate the two together rather than as separate line items.
    4. Divide each cluster's combined credit usage across the Workspaces that use it, based on the attribution model you chose in the first step. For example, split credit usage evenly across consuming teams, or weight it by each team's usage from the next step.

    <Note>The Clusters breakdown excludes Workspace credit usage that can't be tied to a specific cluster, such as AI token usage. Attribute that credit usage using the Deployments or Compute Types breakdown instead.</Note>

    <Note>Some credit usage figures take time to fully populate. Network credit usage in particular can take a couple of days to arrive, so a chargeback report you build right after a billing period ends might report a lower network credit usage than it should.</Note>

    <Info>Worker queue credit usage only applies to Deployments running the Celery Executor or Astro Executor. Use the **Worker Queues** tab under **Detailed Costs** if a team needs credit usage broken down by individual worker queue.</Info>
  </Step>

  <Step title="Account for different usage patterns">
    Splitting shared credit usage evenly across teams can be inaccurate if some teams run Dags far more frequently than others. Compare usage before you finalize an allocation:

    1. Click the **Organization Overview** tab and review **Top 10 Workspaces by Active Dag Count** and **Top 10 Workspaces by Code Deploy Count** to see which teams drive the most activity.
    2. Click the **Operator Use** tab and filter by Workspace Name to compare task counts and Dag counts for each team.
    3. Use these usage numbers to weight the shared credit usage allocation from the previous step. For example, a team responsible for 70% of task runs on a shared cluster takes on 70% of that cluster's credit usage, rather than an even split.
  </Step>

  <Step title="Automate recurring chargeback reporting">
    Instead of pulling each dashboard manually every billing cycle, export chargeback data on a schedule.

    1. Click the **Underlying Data** tab to see the **Daily Cost Breakdown** table, along with the same data in the FinOps Open Cost and Usage Specification (FOCUS) format, a standardized billing schema that's easier to load into an existing chargeback or finance tool.
    2. See [Export Astro reporting data](/docs/astro/org-dash-exports) to set up a recurring export or a conditional dashboard alert for this table.

    <Tip>Astronomer recommends scheduling exports from the **Underlying Data** tab instead of from individual charts, since it gives you a consistent, standardized format each cycle.</Tip>
  </Step>

  <Step title="Pull chargeback data with the Billing API">
    <Info>
      **Labs**

      This feature is in [Labs](/docs/astro/feature-previews).
    </Info>

    Instead of exporting from the UI, pull the same credit usage data programmatically with the Astro API. This is useful if you want to load chargeback data directly into your own finance or BI tooling. See the [Astro Labs API overview](/docs/astro/api/v-1-labs/labs-overview) for authentication, base URL, and versioning details. The Billing API is available on the Enterprise tier and above, and requires the same Organization Billing Admin-level permission as the rest of this guide.

    Use the following endpoint:

    * `GET /organizations/{organizationId}/billing/daily-usage`: provides fine-grained Astro credit usage data on a daily time grain. Corresponds to the **Daily Cost Breakdown** table on the **Underlying Data** tab by default, or the **FOCUS Format** table when you set `format=focus` to get the same data in the FinOps FOCUS format.

    This endpoint accepts `startDate` and `endDate` query parameters, `workspaceId` and `deploymentId` filters for scoping to specific teams, and `limit`/`offset` pagination:

    ```bash wrap theme={null}
    curl --location 'https://api.astronomer.io/labs/v1/organizations/<organization-id>/billing/daily-usage?startDate=2026-08-01&endDate=2026-08-07&format=focus&limit=1000' \
    --header 'Authorization: Bearer <your-api-token>'
    ```

    <Note>Usage data can be delayed, so exporting only the previous day's data might not capture every cost row accurately. Export daily to keep your data current, but query a 7-day lookback window each time (for example, set `startDate` to 7 days before `endDate`) so any rows that arrived late get included in your chargeback calculations.</Note>
  </Step>
</Steps>

## Track credit usage against your contract

After you attribute credit usage to teams, use the **Contract Details** tab to monitor your Organization's overall credit or dollar balance and projected burn rate. This dashboard reports credit usage at the Organization level and isn't team-attributable on its own, but it's useful context for the central team responsible for procurement when reviewing chargeback totals against the overall contract.

Set the **Viewing As** control to **Cost Basis** when you reconcile chargeback totals against the overall contract, since it reflects what your Organization actually paid rather than list-price **Credits**.


# Manage development Deployments on Astro
Source: https://astronomer.io/docs/astro/best-practices/manage-dev-deployments

Choose between permanent or preview development Deployments to support your CI/CD workflow on Astro.

For most teams working on Astro, Astronomer recommends using multiple Airflow Deployments for running and testing development and production versions of your pipelines, and promoting code between them using CI/CD. This allows you to develop your pipelines faster, more securely, and more reliably.

There are many ways to organize your code, CI/CD pipelines, and Deployments to support a sustainable development lifecycle on Astro, and no single setup will work for all teams. However, there are a two main options for managing your development Deployments and promoting code from development to production:

* Maintain a permanent development Deployment that contains the code from a permanent `dev` branch of a version-controlled code repository. You can hibernate this Deployment so that it doesn't consume resources when you're not using it.
* Configure CI/CD workflows to create preview Deployments that map to feature branches which are deleted when the feature branch is merged into production.

This guide covers how to choose which of these methods is best for your team and how to implement both using Astro features.

<Info>This guide does not provide detailed guidance on how to set up code repositories such as GitHub or how to configure CI/CD tools. Consult the documentation for your organization's tooling for specific and up-to-date guidance.</Info>

## Feature overview

This guide highlights how to use the following Astro features to manage your Deployments:

* Astro's [GitHub integration](/docs/astro/deploy-github-integration).
* [Deployment hibernation](/docs/astro/deployment-resources#hibernate-a-development-deployment).
* Multi-Deployment [CI/CD](/docs/astro/set-up-ci-cd#multiple-environments).

## Best practice guidance

It's a best practice to maintain [multiple environments](/docs/astro/set-up-ci-cd#multiple-environments) for separate development and production versions of a data pipeline. While there are many ways to do this, the two options covered in this guide — permanent development Deployments and ephemeral preview Deployments mapped to feature branches — will work for most teams.

In general, preview (ephemeral) Deployments offer a better development experience. Since each feature branch maps to its own Astro Deployment, you don't have to worry about conflicts from other developers working on a development branch at the same time. If you have larger teams working on Astro or deploy changes frequently, this is a good option. However, managing Deployments in this way requires more setup to ensure the Deployments have access to external systems and resources, which in turn requires a more complex CI/CD implementation. Your team should have experience with CI/CD to make this pattern successful. Preview Deployments can also come with less predictable costs, especially at larger scale with many feature Deployments being regularly spun up and down.

Permanent development Deployments are easier to set up and manage. You only need to set up the environment once, and Astro's GitHub integration offers far simpler CI/CD implementation for GitHub users. In many cases, this pattern is also more cost-effective, as you can maintain one development Deployment and use Astro's hibernation feature to reduce costs for the Deployment when you aren't using it. This option is often best for smaller teams, teams that deploy infrequently, and teams who are very cost conscious.

<Tip>[Hibernating development Deployments](/docs/astro/deployment-resources#hibernate-a-development-deployment) reduces the cost of maintaining multiple permanent Deployments.</Tip>

## Hibernating development Deployment example

This example shows you how to map a permanent branch on GitHub to a permanent development Deployment with a hibernation schedule. If your team doesn't use GitHub, you can also implement a branch-based hibernating development Deployment using [CI/CD templates](/docs/astro/ci-cd-templates/template-overview).

### Prerequisites

* One [Astro Deployment](/docs/astro/create-deployment) for production.
* One [Astro project](/docs/cli/v1.43/develop-project) in a GitHub repository.

<Info>You can extend this example to encompass any number of Astro Deployments and development branches.</Info>

### Implementation

1. Create a new development Deployment. Make sure **Development Mode** is enabled when you create the Deployment. See [Enable development mode](/docs/astro/deployment-resources#enable-development-mode).
2. Create a hibernation schedule for your development Deployment. Choose a schedule that will not interfere with your typical development times. Note that you cannot deploy to a Deployment that is hibernating. See [Hibernate a development Deployment](/docs/astro/deployment-resources#hibernate-a-development-deployment).
3. Configure branch-based deployment using the GitHub integration, mapping one branch of a GitHub repository to the development Deployment you created. See [Deploy code with the Astro GitHub integration](/docs/astro/deploy-github-integration).

## Ephemeral preview Deployment example

This example shows how to implement ephemeral Deployments for feature development that are spun up and down by CI/CD.

### Prerequisites

* One [Astro Deployment](/docs/astro/create-deployment) for production.
* A CI/CD tool such as GitHub Actions, Jenkins, or CircleCI.
* A version control tool such as GitHub or GitLab.
* One [Astro project](/docs/cli/v1.43/develop-project) in a Git repository.
* [Deployment API tokens](/docs/astro/deployment-api-tokens) to automate your code deploys.

<Info>You can extend this example to encompass any number of Astro Deployments.</Info>

### Implementation

1. Obtain an API token in the Astro UI. See [Create an API token](/docs/astro/automation-authentication#step-1-create-an-api-token).
2. Install the Astro CLI in your CI/CD tool. See [Authenticate an automation tool to Astro](/docs/astro/automation-authentication#step-2-install-the-astro-cli-in-your-automation-tool).
3. Create a CI/CD pipeline using a [GitHub Actions template](/docs/astro/ci-cd-templates/github-actions-deployment-preview) or, if using a different CI/CD tool, [shell scripts](/docs/astro/ci-cd-templates/preview-deployments) that contain logic for managing preview Deployments based on branches. These workflows require at least an Astro Deployment name or ID and a branch name. See [Create a CI/CD pipeline](/docs/astro/set-up-ci-cd#create-a-ci/cd-pipeline) and [Preview Deployments](/docs/astro/ci-cd-templates/preview-deployments).

<Tip>Business-tier users can add an additional layer of security by enforcing CI/CD for deploys rather than allowing manual deploys with the Astro CLI. See [Enforce CI/CD](/docs/astro/set-up-ci-cd#enforce-ci/cd).</Tip>

## See also

* [Manage Astro connections in branch-based deploy workflows](/docs/astro/best-practices/connections-branch-deploys)


# Manage resources on Astro
Source: https://astronomer.io/docs/astro/best-practices/manage-resources

Compare approaches for managing Astro Deployment resources with the UI, API, CLI, or Terraform.

Astro supports several approaches to managing [Deployment resources](/docs/astro/deployment-settings#deployment-resources), so you can provision the resources you need whether you are just starting out or you are deploying projects programmatically at scale.

## Recommended approaches

Astronomer recommends the following approaches in these common scenarios:

| Scenario                                                       | Astro UI | Terraform provider | Astro API | Deployment files | Astro CLI |
| -------------------------------------------------------------- | :------: | :----------------: | :-------: | :--------------: | :-------: |
| Manually managing resources                                    |     x    |                    |           |                  |     x     |
| Programmatically managing resources                            |          |          x         |     x     |         x        |           |
| Managing resources at scale                                    |          |          x         |     x     |         x        |           |
| Managing resources for a small team or an individual developer |     x    |                    |     x     |                  |     x     |
| Automating management with CI/CD                               |          |          x         |     x     |         x        |           |
| Managing resource config with version control                  |          |          x         |     x     |         x        |           |
| An approach using Python is needed                             |          |                    |     x     |                  |           |
| An approach using Bash is needed                               |          |                    |     x     |                  |     x     |

For more detailed information about when to choose each option, see the sections below.

## Managing resources manually

Astronomer recommends the Astro UI for managing resources if you do not need to deploy or modify project config programmatically. If you are a small team getting started or you are an individual dev creating projects on an ad hoc basis, the Astro UI will likely meet your needs when managing resources. In addition to all the options you need in order to create and customize resources, you will get guidance directly in the UI that will help ensure that your instance is right-sized for your use case.

On the Astro UI, you can:

* Optimize Deployment processing.
* Optimize compute resources and cost.
* Enable use cases with intensive workloads.

For more details about configuring resources with the UI, see [Deployment resources](/docs/astro/deployment-resources).

<Info>
  If you prefer to use a command-line tool, you can use the Astro CLI to manage all the resources configurable with the UI.

  For the commands and settings available in the Astro CLI, see [Command reference](/docs/cli/v1.43/reference).
</Info>

## Managing resources programmatically

Astronomer recommends the Terraform Provider, Astro API, or Deployment files when you need to manage resources programmatically. Typically, teams manage resources as code as they start managing instances at scale. Benefits of managing resources as code include the ability to have your infrastructure configuration in your version control solution, which allows for tracking and rolling back changes as well as recreating resources easily if something goes wrong. Also, you can create and modify large numbers of Deployments quickly, making it easy to onboard new teams, reallocate resources, reassign Deployments, and more.

* [Terraform provider](/docs/astro/terraform-provider). Terraform is an industry-standard tool for managing infrastructure as code (IaC). With the provider, you can use Terraform to automate, templatize, or programmatically manage Astro environments. For example, you can automate creating Workspaces and Teams based on existing resources. Astronomer recommends this approach in general but especially for teams in organizations where Terraform is already in use.
* [Astro API](/docs/astro/api/v-1/overview). The API enables you to create or update resources such as Organizations, Deployments, Clusters, Deploys, and Workspaces. An Organization API token is required. You can [download the OpenAPI spec](/docs/astro/api/v-1/overview#download-openapi-specification) for easy configuration of tools such as Postman and Swagger. Astronomer recommends the API for Python-centric use cases.
* [Deployment files](/docs/astro/manage-deployments-as-code). You can configure Deployments programmatically using Deployment files, which you can generate automatically from existing Deployments. You can standardize Deployment configuration for specific use cases using Deployment template files, which you can also generate automatically from existing Deployments. Astronomer recommends this approach when automating Deployment management at scale.

<Info>
  If you prefer to use Bash scripts to manage your infrastructure, you can use the Astro CLI as a wrapper on the API. You can use CLI commands to automate tasks such as creating and deleting Deployments, hibernating and waking Deployments, creating and updating Deployment pools, and creating and updating worker queues. You can also automate command execution in CI/CD pipelines using API tokens. Note: the Astro CLI requires Docker.

  For an overview of the CLI, see [Astro CLI](/docs/cli/v1.43/overview).
</Info>


# Choose a code repository strategy
Source: https://astronomer.io/docs/astro/best-practices/repo-structure

Choose between monorepo and multirepo strategies for organizing Astro project code.

Astro supports a range of options when it comes to organizing your project code.

This guide covers the options along with their pros and cons, so you can choose the best option for your team.

## Feature overview

Astro supports two basic repository strategies. You can:

1. Use a **monorepo**: host an Astro project in one repository for deploys to one or multiple Astro Deployments, based, for example, on permanent dev and prod branches with their own Deployments or a single permanent branch. Astronomer recommends this approach for most teams.
2. Use a **multirepo**: separate your Dags from other files in your project and maintain multiple repositories linked to one or multiple Astro Deployments. Astronomer recommends this approach for teams needing to meet strict security requirements and teams planning to automate the creation of Deployments.

## Best practice guidance

Depending on your organization's structure and needs, Astronomer recommends either a monorepo or multirepo approach to organizing the code you deploy to Astro. In addition to a repository strategy, Astronomer recommends implementing version control and CI/CD.

### Option 1: Monorepo

<Info> This is the most common strategy. </Info>

When your Dags are critical to your business, the ability to test your Dags is critical, as well. Using a monorepo, you can set up automated deploys from multiple branches of a single repository to support multiple Deployments of the same codebase, enabling implementation of robust testing of the code you intend to deploy to production, support for multiple teams, and more.

Within a monorepo, you can also maintain multiple Astro projects, all with Astro Deployments and testing.

This strategy requires:

* One Astro Workspace for your project(s).
* One or more permanent branches in your repository, each representing an environment. Many teams use two permanent branches named `main` and `dev`.
* One or more Astro Deployments, each representing an environment.

Pros of this approach:

* You can test code changes to your Dags in an isolated environment on Astro before deploying the changes to production.
* It is flexible and scalable. Mapping branches to additional Deployments allows for permanent testing environments or ephemeral sandbox Deployments for feature testing. See [Manage dev Deployments on Astro](/docs/astro/best-practices/manage-dev-deployments). Also, if you work at a larger organization, you can deploy to different Workspaces and Astro projects to support multiple teams or organize Deployments according to specific business use cases.
* It supports separate environment configurations. For example, your development Deployment can query a development database, and your production Deployment can query a production database without needing complex logic to switch between the two.

Cons of this approach:

* Dependencies and environments might be more difficult to manage, to the extent that they differ from project to project.

<Tip>For guidance on setting up CI/CD for deploys to multiple Deployments from a single repository, see [Multiple environments](/docs/astro/set-up-ci-cd#multiple-environments).</Tip>

<Info>Individuals and small teams getting started on Astro can keep their instances simple with one permanent branch mapped to an Astro Deployment.</Info>

### Option 2: Multirepo

A common use case for multiple repositories is keeping Astro Project configuration (such as the `Dockerfile`, `requirements.txt`, or Deployment Configuration-as-Code) separate from Dag code. Astronomer recommends a multirepo approach when:

* You have strict security requirements for who can manage Deployments.
* You want to minimize complexity for project contributors, automate the creation of Deployments, or manage Deployments more efficiently.
* You can set up and maintain a more complex CI/CD pipeline.

This strategy requires:

* A Workspace for your project(s).
* Multiple repositories.
* One or more Astro Deployments.
* [**Dag-only deploys**](/docs/astro/deploy-dags#enable-or-disable-dag-only-deploys-on-a-deployment) on the target Deployment and CI/CD pipeline setup for each repository.

Pros of this approach:

* Keeping project configuration separate from Dag code enables the use of a single repository for configuring multiple Deployments.
* The possibility of accidental changes to Astro Deployment settings is minimized.
* You can automate the creation of Deployments and streamline your Deployment management process.

Cons of this approach:

* You must keep any local copies of an Astro project synchronized with multiple repositories in order to test Deployment code locally.
* Team members might have inconsistencies in their local environments if they lack access to the configuration repository. For this reason, Astronomer recommends [setting up a Dev Deployment](/docs/astro/best-practices/manage-dev-deployments) where Dag authors can see and modify project configuration for testing purposes.

### Version control and CI/CD

In addition to choosing a repository strategy suited to your use case, Astronomer recommends implementing version control and CI/CD.

Implementing version control is a longstanding software development best practice that enables:

* Simultaneous collaboration.
* Easy review of code changes.
* Safe and secure experimentation on new features and fixes.
* Improved traceability and auditing.

Using CI/CD is a general software development best practice that enables automated building, testing, merging, and delivery of code. If you haven't yet implemented CI/CD pipelines for your Deployments, benefits of doing so include:

* Faster feedback loops.
* Improved code quality.
* Streamlined development workflows.
* Consistent deployments across environments.
* Less time spent getting code to production.
* Enhanced collaboration.

With a version control system in place, you can set up CI/CD by following the guidance in [Develop a CI/CD workflow for deploying code to Astro](/docs/astro/set-up-ci-cd). The option you choose for organizing the code in your shared repository and deploying to Astro should reflect the size and needs of your team. [Deployment API tokens](/docs/astro/deployment-api-tokens) are required for implementing CI/CD for automated deploys to Astro.

Once you have set up CI/CD, Astronomer recommends [enabling CI/CD enforcement](/docs/astro/set-up-ci-cd#enforce-ci/cd) so that code pushes can be completed only when using a Deployment or Workspace token.


# Best practices for rightsizing Airflow resources on Astro
Source: https://astronomer.io/docs/astro/best-practices/rightsize-airflow-on-astro

Use Astro Deployment metrics to rightsize your Airflow resources on Astro.

Astro gives you options to customize your Deployment settings, enabling you to choose the right amount of resources for your processing needs. This means you can customize the size of your Airflow components so that your processes can sufficiently scale for heavier workloads, without you needing to reserve resources that you don't need.

This guide shares an example process for how to use Deployment Metrics in Astro to analyze your Deployment's performance and determine whether or not to adjust your allocated resources. The process covers the following steps:

* Define thresholds for the minimum and maximum resource use that you feel are an optimal performance range.
* Examine your historical Dag and Deployment performance data to see if your Deployment operates within your optimal performance range.
* Determine any required changes to your resource allocations for your workers and scheduler.
* Make those adjustments in your Deployment settings.

You can periodically repeat this process to make sure that your Airflow resources are optimized for your workloads as your needs change.

<Tip>If you want to analyze your Deployment metrics in greater detail, you can use the [Universal Metrics Exporter](/docs/astro/export-metrics) to configure a metrics export at the Deployment or Workspace level.</Tip>

## Feature overview

* Use [Deployment metrics](/docs/astro/deployment-metrics) to estimate the right size of your Airflow resource.
* Adjust your [Deployment resource sizes](/docs/astro/deployment-resources) to meet your needs.

## Prerequisites

This guide assumes that you have:

* At least one [Astro Deployment](/docs/astro/create-deployment).
* At least one active [Astro project](/docs/cli/v1.43/develop-project), because you need your Dag performance data.
* Workspace Owner, Workspace Operator, or Deployment Admin [user permissions](/docs/astro/user-permissions).
* Familiarity with the different Airflow Executors available on Astro, the [Astro Executor, Celery Executor and Kubernetes Executor](/docs/astro/executors-overview).

## Step 1: Determine performance thresholds

Define two performance thresholds:

* **Maximum threshold** Choose a capacity percentage, between 0% and 100%, that you want to consider the upper limit for ideal performance, but still leaves some additional capacity for pipeline growth. This example uses 75% for the maximum threshold.

* **Minimum threshold** Choose a capacity percentage, between 0% and 100%, that you want to consider the lower limit for ideal performance, where you don't think that unconsumed resources are wasted. This example uses 50% for the minimum threshold.

## Step 2: Retrieve your Deployment metrics

In the Astro UI, open the Deployments page and choose the Deployment you want to rightsize.  Navigate to the **Analytics** tab and choose **Last 7 days** for the timeframe.

## Step 3: Adjust your scheduler resource settings

For Astro Hosted Deployments that are **Medium**, **Large**, or **Extra Large**, your scheduler has a separate Dag processor component, which appears as a separate line in each line graph of your Deployment metrics. In these Deployments, scheduler and Dag processor resources should be considered separately. For Small deployments, which combine the scheduler and Dag processor, use the following **Scheduler Resources** steps to right-size your Deployment size.

### Scheduler Resources

The Deployment metrics show the CPU and Memory use per Pod as percentages, which you can compare to the **Minimum Threshold** and **Maximum Threshold** that you defined in Step 1. For scheduler resources, follow the same process:

* If CPU or Memory is between the **Minimum Threshold** and the **Maximum Threshold**, no changes are needed.
* If CPU or Memory is greater than the **Maximum Threshold**, then increase those resources.
* If CPU or Memory is less than the **Minimum Threshold**, then decrease those resources.

In the following example, you can see the resource metrics for a small scheduler and Dag processor, where both run in one single process and the metrics show a single line graph. If you had a minimum threshold set for `50%` and maximum for `75%`, the following example that shows around 5% CPU use and 14% for Memory use indicates that you can allocate more processes to this Deployment, or decrease the memory and CPU for its scheduler.

<Frame>
  <img alt="Screenshot of Schedulers CPU and Memory use metrics, showing a line graph of a consistent 3-6% for the CPU and around 9-19% use for memory over the previous 7 days." />
</Frame>

### Dag processor Resources and Performance

<Info>For **Medium**, **Large**, and **Extra Large** Deployments only</Info>

For **Medium**, **Large**, and **Extra Large** Deployments, the Dag processor can run close to, or at, maximum vCPU utilization without competing for resources with the scheduler, since the two components run separately. A Dag processor running at 100% vCPU utilization results in slower Dag file processing times without causing scheduler unavailability. For these Deployment sizes, high Dag processor vCPU utilization is a good sign because this means your Dag processor is running near full capacity and is processing Dags efficiently.

You can rightsize Dag processor memory following the same process as other resources:

* If CPU or Memory is between the **Minimum Threshold** and the **Maximum Threshold**, no changes are needed.
* If CPU or Memory is greater than the **Maximum Threshold**, then increase those resources.
* If CPU or Memory is less than the **Minimum Threshold**, then decrease those resources.

To rightsize Dag processor vCPU, consider your current Dag file processing speed. In large deployments, **Large** and **Extra Large** size, the limitation for Dag file processing time tends to be the number of parsing processes, which can be adjusted through an [Airflow environment variable](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#parsing-processes), `parsing_processes`. You can create and store an Airflow environment variable directly in the [Astro UI](/docs/astro/create-and-link-variables).

If Dag file processing is slow and you want to increase performance, use the following steps:

* If Dag processor vCPU utilization is low (\< 50%), increase Dag processor vCPU utilization by increasing the number of parsing processes. You should see Dag processor vCPU utilization increase.
* If Dag processor vCPU utilization is high (> 80%), or the number of parsing processes is higher than 12 parsing processes, increase Deployment size.

In the following example, you can see the resource metrics for a Medium size scheduler that includes a Dag processor. If you had a minimum threshold set for `50%` and maximum for `75%`, the following example that shows consistent 4-8% for the Dag processor vCPU, consistent 2-6% use for scheduler CPU use, and 7-15% memory use by both the scheduler and Dag processor indicates that you can allocate more processes to this Deployment, or scale down the size of the size of the resources.

<Frame>
  <img alt="Screenshot of Schedulers CPU and Memory use metrics, showing a line graph of a consistent 4-8% for the Dag processor vCPU, consistent 2-6% use for scheduler CPU use, and 7-15% memory use by both the scheduler and Dag processor over the previous 7 days." />
</Frame>

Learn more about adjusting `parsing_processes` and how to see the number of default parsing processes across each Deployment size in [Scaling Airflow](/docs/learn/airflow-scaling-workers#scheduler-settings).

See [Scheduler resources](/docs/astro/deployment-resources#size-options) for more details about configuring your scheduler.

## Step 4: Adjust your executor resource settings

<Tabs>
  <Tab title="Celery executor">
    Go to the **Workers** section of your Deployment analytics to view the **CPU** and **Memory** use for your workers. This shows the CPU and Memory use per worker Pod as percentages, which you can compare to the **Minimum threshold** and **Maximum threshold** you defined in Step 1.

    <Tip>Enable **Dynamic Y-Axis scaling** to autoscale the graphs to best fit the data reported for each metric.</Tip>

    * If CPU or Memory is between the **Minimum Threshold** and the **Maximum Threshold**, no changes are needed.
    * If CPU or Memory is greater than the **Maximum Threshold**, then increase those resources.
    * If CPU or Memory is less than the **Minimum Threshold**, then decrease those resources.

    Using the following example, with the thresholds defined as `50%` for the minimum and `75%` for the maximum, you would reduce the available CPU for your workers because CPU use over time remained around 25% to 30%. Because the metrics show resource use to remain approximately 50%, which is within the optimal performance range but consistently on the lower end of the scale, you could choose to reduce the resources you use. Depending on whether you use the CeleryExecutor or Kubernetes executor, you adjust your resources in different ways.

    <Frame>
      <img alt="Screenshot of Workers CPU and Memory use metrics, showing a line graph of a consistent 30% use for the CPU and around 50% use for memory over the previous 7 days." />
    </Frame>

    #### Changing Celery Executor resources

    If you use the Celery Executor, you cannot directly configure the CPU or memory available to your workers. Instead, you can configure the worker types, such as `A5` or `A10`, which have a fixed amount of CPU and memory. See more information about [Configuring worker queues](/docs/astro/configure-worker-queues) to select a different resource size.
  </Tab>

  <Tab title="Kubernetes executor/KubernetesPodOperator">
    #### Configuring Kubernetes executor resources

    If you use the Kubernetes Executor or `KubernetesPodOperator`, you can edit the CPU and memory available to each pod. For more information, see:

    * [Configuring a KubernetesExecutor-deployment default pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources)
    * [Configuring per-task resources worker pod resources](/docs/astro/kubernetes-executor#example-set-cpu-or-memory-limits-and-requests)
    * [Configuring `KubernetesPodOperator` pod resources](/docs/astro/kpo-task-level-resources)

    To determine optimal resources for each KubernetesExecutor/`KubernetesPodOperator` pod, start with low resources, and increase resources gradually if you find the task is running slow or is terminated by Kubernetes due to insufficient resources.

    For example, start with 1 GiB memory and gradually increase the memory allotment until the task runs successfully. You can avoid wasting excess resources by following the best practice of gradually increasing your resource settings, instead of assigning the maximum option available.
  </Tab>
</Tabs>

## Step 5: (Optional - Celery only) Examine Pod count

If you have minimum and maximum Celery pod counts configured for your Deployment, you can compare the **Pod count per status** to your **CPU use**. This diagram shows you how many pods your Deployment uses.

The following example shows the Pods used by a Deployment that can autoscale to a maximum of ten worker Pods, and a minimum of zero Pods. Because the maximum number of Pods used is only five Pods, you can see that there are an additional five Pods available, if needed.

<Frame>
  <img alt="Screenshot of Pod count metrics, showing a line graph of the number of Pods used by the Deployment over the previous seven days. The maximum number of Pods used was 5, rarely, but most consistently the Deployment used 2." />
</Frame>


# Best practices for upgrading Astro Runtime on Astro
Source: https://astronomer.io/docs/astro/best-practices/upgrading-astro-runtime

A list of best practices to ensure that your Astro Runtime upgrades are safe and easy.

Astro includes features that make upgrading to the latest version of Astro Runtime easy and, more importantly, safe. When leveraging these features, follow these best practices to ensure that your Deployments remain stable after you upgrade:

* Monitor for upgrade opportunities using [Organization dashboards](/docs/astro/organization-dashboard), as well as by following the [Astro Runtime maintenance and lifecycle policy](/docs/runtime/runtime-version-lifecycle-policy).
* Check [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime) for advisories on the specific version you're upgrading to.
* Prepare to upgrade safely using [upgrade tests](/docs/cli/v1.43/test-your-astro-project-locally#test-before-an-astro-runtime-upgrade).
* In the case of a malfunction after upgrading, revert to your original Astro Runtime version using [deploy rollbacks](/docs/astro/deploy-history).

Use this document to learn more about each of these best practices for upgrading Astro Runtime.

## Best practice guidance

Astronomer allows you to control the version of Astro Runtime that you run your Astro deployments on [^1], but we do also have an opinionated best practice that we believe most Astro users would benefit from following.

Astronomer recommends:

* Using the [slim version](/docs/runtime/runtime-image-architecture#slim-images) of Astro Runtime
* Staying on the very latest release of Astro Runtime, i.e., always upgrading as soon as possible
* Controlling your provider dependencies explicitly, by specifying their versions in your `requirements.txt` file

In your Dockerfile:

```dockerfile title="Dockerfile" wrap theme={null}
# Upgrade this as often as you can - it's safe
# Use astro-runtime-slim instead of astro-runtime
FROM quay.io/astronomer/astro-runtime-slim:12.8.0
```

In your `requirements.txt` file:

```text title="requirements.txt" wrap theme={null}
# Use ~= to allow upgrading minor and patch version, but not breaking changes
# Only upgrade major versions explicitly, and only after reviewing release notes

apache-airflow-providers-google ~= 14.0  # release notes: https://airflow.apache.org/docs/apache-airflow-providers-google/stable/changelog.html
apache-airflow-providers-snowflake ~= 6.1  # release notes: https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/changelog.html
```

### Rationale

Astronomer thinks about the Airflow code in two major categories:

* **The core** is the apache-airflow package plus the providers that are required to run successfully on Astro, for example the astronomer-logging-provider. This provider is part of the core because without it, task logs will not work on Astro.

* **The optional providers** are packages like [apache-airflow-providers-google](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/index.html) that extend Airflow by providing operators that help you interface with services or perform common tasks without having to write a lot of new code. You only need to install a optional provider packages for the specific services and tasks that you actually use in your Astro deployment.

The main concept underpinning Astronomer's upgrade recommendations is that you should always strictly prefer the latest version of core, but you may not necessarily want the latest provider versions for the operators that you use. Great pains are taken in core to prevent backwards incompatibilities and bugs that could be disruptive to your Airflow pipelines. Because the slim version of Runtime contains only the core parts of Airflow and no optional providers, you never need to change your Dag code to upgrade to the latest Runtime slim version[^2], unless you are upgrading from Airflow 2 to Airflow 3, which rarely necessitates code level changes.

Fortunately, the optional providers can be both upgraded *and downgraded* freely, without requiring rollbacks. Which means that the risk introduced by upgrading pre-maturely is low. In many cases it can be a rational choice to simply upgrade your optional providers to the latest version, run your Dags and see if anything breaks. If you get an unexpected Dag import error or task failure, evaluate if that failure could have been caused by the provider upgrade.

However, for greater safety when working with production deployments, it is prudent to check the [provider release notes](https://airflow.apache.org/docs/#providers-packages-docs-apache-airflow-providers-index-html) for the major version you are upgrading to and assess them for any breaking changes that might impact your dags.

### Performing an upgrade

The Astro CLI has [built-in functionality](/docs/cli/v1.43/test-your-astro-project-locally#test-before-an-astro-runtime-upgrade) to identify major version upgrades in packages which could introduce breaking changes to your dags. Astronomer recommends making use of this feature prior to every upgrade.

Use the process described in [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime#step-3-optional-run-upgrade-tests-with-the-astro-cli) as the basis for all upgrades. It gives step-by-step guidance on completing an upgrade, as well as version-specific advice. In particular, the following steps should be followed for a smooth upgrade:

* Run upgrade tests to anticipate and address problems. This involves a single command that:

  * Compares dependency versions between the current and upgraded environments.

  * Checks Dags in the project against the new Airflow version for errors.

  * Runs a Dag parse test with the new Airflow version.

  * Produces in new project directory `upgrade-test-<current_version>--<upgraded_version>`:
    * An upgraded `Dockerfile`.
    * `pip freeze` output for both versions.
    * A report with environment metadata and a Results table including any errors logged.

  * A Dependency Compare report listing the upgrades, additions and removals coming in the new Astro runtime version and underlying Airflow package. For example, a "Major Updates" section lists the packages receiving new major versions, and a "Minor Updates" lists the packages receiving new minor versions:

  ```text wrap theme={null}
  Major Updates:
  azure-mgmt-datafactory 6.1.0 >> 7.0.0

  Minor Updates:
  Flask-Caching 2.1.0 >> 2.2.0
  ```

For more details about the `upgrade-test` command, see [Testing your Astro project locally](/docs/cli/v1.43/test-your-astro-project-locally#test-before-an-astro-runtime-upgrade).

### Monitoring for upgrades

Astro users on the Enterprise plan can use Organization dashboards to check whether their presently running Astro Runtime versions are currently maintained, as well as how much time is left before they become unmaintained. This is especially helpful in cases where your Organization has many Deployments to track. See [Organization dashboards](/docs/astro/organization-dashboard#deployment-detail) for steps to access these dashboards.

If you are not on the Astro Enterprise plan, you can still reference the [Astro Runtime maintenance and lifecycle policy](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-maintenance-policy) to see when you'll need to upgrade your Deployments to a new version of Astro Runtime. A [Deployment Health Alert](/docs/astro/alerts#deployment-health-alerts) will also be displayed if a Deployment is on an unmaintained Astro Runtime version.

### Roll back Deployments after a broken upgrade

<Info>Astro supports rolling back from Airflow 3 to Airflow 2 but not all Airflow 3.x Deployments can be reverted to all Airflow 2.x versions. Review the [Airflow 3 to Airflow 2 rollback requirements](/docs/astro/airflow3/upgrade-af3#airflow-3-to-airflow-2-rollback-support-and-requirements).</Info>

In the case of an emergency, you can roll back to your previous deploy to downgrade the Astro Runtime version. Astro's rollback functionality is similar to the Git `revert` command. Rollbacks revert project code only, keeping environment variables and other configuration unchanged.

Some rollbacks require reverting the underlying database schema used by Airflow. Sometimes, data cannot be represented in the older schema and so will be dropped by the rollback process. For example, if you upgrade to Runtime 12, new tasks will have the try number associated with Event Logs, but if you roll back to a previous version, these try numbers will be lost. This data is always data that would not have been present to begin with if you had not upgraded your Runtime, so there is not risk of data loss from the overall process of upgrading and subsequently rolling back.

<Tip>The `upgrade-test` command can also be used to test the current Runtime version against a *downgraded* version.</Tip>

## See also

* [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime#step-3-optional-run-upgrade-tests-with-the-astro-cli).

[^1]: Subject to certain restrictions

[^2]: Unless you are directly accessing the Airflow metadata database from your task code. We strongly recommend that you never do this; it's a great way to break things, and it won't work with Airflow 3.


# Create a network connection between Astro and GCP
Source: https://astronomer.io/docs/astro/connect-gcp

Create a network connection to Google Cloud Platform.

Use this document to learn how you can grant an Astro cluster and its Deployments access to your external Google Cloud Platform (GCP) resources.

Publicly accessible endpoints allow you to quickly connect your Astro clusters or Deployments to GCP through an Airflow connection. If your cloud restricts IP addresses, you can add the external IPs of your Deployment or cluster to an GCP resource's allowlist.

If you have stricter security requirements, you can [create a private connection](#create-a-private-connection-between-astro-and-gcp) to GCP in a few different ways.

After you create a connection from your cluster to GCP, you might also need to individually authorize Deployments to access specific resources. See [Authorize your Deployment using workload identity](/docs/astro/authorize-deployments-to-your-cloud).

## Standard and dedicated cluster support for GCP networking

Standard clusters have different connection options than dedicated clusters.

Standard clusters can connect to GCP in the following ways:

* Using [static external IP addresses](#allowlist-a-deployment’s-external-ip-addresses-on-gcp).
* Using Private Service Connect to all managed [Google APIs](https://cloud.google.com/vpc/docs/private-service-connect-compatibility#google-apis-global).

Dedicated clusters can use all of the same connection options as standard clusters. Additionally, they support a number of private connectivity options including:

* VPC peering

If you require a private connection between Astro and GCP, Astronomer recommends configuring a dedicated cluster. See [Create a dedicated cluster](/docs/astro/create-dedicated-cluster).

## Access a public GCP endpoint

All Astro clusters include a set of external IP addresses that persist for the lifetime of the cluster. When you create a Deployment in your workspace, Astro assigns it one of these external IP addresses. To facilitate communication between Astro and your cloud, you can allowlist these external IPs in your cloud. If you have no other security restrictions, this means that any cluster with an allowlisted external IP address can access your GCP resources through a valid Airflow connection.

### Allowlist a Deployment's external IP addresses on GCP

1. In the Astro UI, select a Workspace, click **Deployments**, and then select a Deployment.
2. Select the **Details** tab.
3. In the **Other** section, you can find the **External IPs** associated with the Deployment.

Add the IP addresses to the allowlist of any external services that you want your Deployment to access.

When you use publicly accessible endpoints to connect to GCP, traffic moves directly between your Astro cluster and the GCP API endpoint. Data in this traffic never reaches the Astronomer managed control plane. Note that you still might also need to authorize your Deployment to some resources before it can access them. For example, you can [Authorize deployments to your cloud with workload identity](/docs/astro/authorize-deployments-to-your-cloud) so that you can avoid adding passwords or other access credentials to your Airflow connections.

<Accordion title="Dedicated cluster external IP addresses">
  If you use Dedicated clusters and want to allowlist external IP addresses at the cluster level instead of at the Deployment level, you can find the list cluster-level external IP addresses in your Organization's **Clusters** page.

  1. In the **Organization** section of the Astro UI, click **Organization Settings**, then click **Clusters**, then select a cluster.
  2. In the Details page, copy the IP addresses listed under **External IPs**.

  Add the IP addresses to the allowlist of any external services that you want your cluster to access. You can also access these IP addresses from the **Details** page of any Deployment in the cluster.

  After you allowlist a cluster's IP addresses, all Deployments in that cluster have network connectivity to GCP.
</Accordion>

## Create a private connection between Astro and GCP

Choose one of the following setups based on the security requirements of your company and your existing infrastructure.

<Tabs>
  <Tab title="VPC peering">
    <Info>This connection option is available only for dedicated Astro clusters.</Info>

    VPC peering ensures private and secure connectivity, reduces network transit costs, and simplifies network layouts. Because Astro uses source network address translation (SNAT) that performs many-to-one IP address translations for connections to your data sources, to minimize the risk and concern with IP overlap and exhaustion with dedicated GCP clusters, you might need to confirm that the default Astro subnet and peering ranges don't overlap with the ranges used by your target resource. See [create a dedicated GCP cluster](/docs/astro/create-dedicated-cluster?tab=gcp#setup) for more information about default ranges and alternative configurations.

    To create a VPC peering connection between an Astro VPC and a GCP VPC:

    1. Contact [Astronomer support](https://cloud.astronomer.io/open-support-request) and provide the following information:

       * Astro cluster ID and name.
       * Google Cloud project ID of the target VPC.
       * VPC NAME of the target VPC.
       * Classless Inter-Domain Routing (CIDR) block of the target VPC.

         After receiving your request, Astronomer support will create a VPC peering connection from your Astro VPC to your target VPC. The support team will then provide you with your Astro cluster GCP project ID and VPC name.

    2. Using the information provided by Astronomer support, [create a peering connection](https://cloud.google.com/vpc/docs/using-vpc-peering#creating_a_peering_configuration) from your target VPC to your Astro cluster VPC. For example, you can use the following gcloud CLI command to create the connection:

       ```sh wrap theme={null}
       gcloud compute networks peerings create <choose-any-name> --network=<your-target-vpc-network-name>  --peer-project=<your-cluster-project-id> --peer-network=<your-cluster-vpc-name>
       ```

    After both VPC peering connections have been created, the connection becomes active.
  </Tab>

  <Tab title="Private Service Connect">
    <Info>
      For GCP dedicated clusters, [Private Service Connect](https://cloud.google.com/vpc/docs/about-accessing-vpc-hosted-services-endpoints#global-access) endpoints must be configured to allow global access to access service endpoints that reside in a different GCP region than the Astro cluster.

      [Google API services](https://cloud.google.com/vpc/docs/about-accessing-google-apis-endpoints) can be accessed through Private Service Connect endpoints from any region out of the box.
    </Info>

    Use Private Service Connect (PSC) to create private connections from Astro to GCP services without connecting over the public internet. See [Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect) to learn more.

    Astro clusters are by default configured with a PSC endpoint with a target of [All Google APIs](https://cloud.google.com/vpc/docs/private-service-connect-compatibility#google-apis-global). To provide a secure-by-default configuration, a DNS zone is created with a resource record that will route all requests made to `*.googleapis.com` through this PSC endpoint. This ensures that requests made to these services are made over PSC without any additional user configuration. As an example, requests to `storage.googleapis.com` will be routed through this PSC endpoint.

    You can check if the service that you want to connect Airflow to is available through the **All Google APIs** target by running the following command:

    ```sh wrap theme={null}
    gcloud services list --available --filter="name:googleapis.com"
    ```

    If you don't see your service listed, open a support case with [Astronomer support](/docs/astro/astro-support) to set up the necessary PSC connectivity and provide a Service attachment URI in the following format: `projects/SERVICE_PROJECT/regions/REGION/serviceAttachments/SERVICE_NAME`.
  </Tab>

  <Tab title="VPN">
    <Info>This connection option is only available for dedicated Astro clusters.</Info>

    Use this connectivity type to access on-premises resources or resources in other cloud providers.

    ### Prerequisites for GCP HA VPN

    * Grant temporary permissions for `astronomer@astro-remote-mgmt.iam.gserviceaccount.com` service account. You can delete the policy binding and role after creating the VPN.

    ```sh wrap theme={null}
    gcloud iam roles create AstroVPNGatewayRole \
        --project $HA_VPN_PROJECT \
        --title "Temporary access to VPN Gateway from Astro" \
        --description "This role allows creation a HA VPN connection with Astro" \
        --permissions compute.vpnGateways.use

    gcloud projects add-iam-policy-binding $HA_VPN_PROJECT \
        --member="astronomer@astro-remote-mgmt.iam.gserviceaccount.com" \
        --role="projects/$HA_VPN_PROJECT/roles/AstroVPNGatewayRole"
    ```

    * Retrieve the full name of VPN Gateway in format `projects/<project name>/regions/<region>/vpnGateways/<gateway name>`.

    #### Prerequisites for GCP Classic VPN

    Retrieve the following information about your VPN device or application:

    * Public IP address
    * Subnet CIDR range, multiple if needed, for your side of the connection
    * Preferences regarding shared key and BGP usage
    * IKE settings for the tunnel

    #### Contact Astronomer support for VPN configuration on Astro side

    Submit all collected details to [Astronomer support](https://cloud.astronomer.io/open-support-request). The Astronomer CRE team will proceed with the required steps. The CRE team will contact you using your support ticket to ask follow-up questions, request clarification, or let you know about connectivity tests.
  </Tab>

  <Tab title="Network Connectivity Center">
    <Info>This connection option is only available for dedicated Astro clusters.</Info>

    The [Network Connectivity Center](https://cloud.google.com/network-connectivity/docs/network-connectivity-center/concepts/overview) is an orchestration framework that simplifies network connectivity among spoke resources that are connected to a central management resource called a hub.
    Astro VPC can connect as a Spoke to the existing Hub for unlocking a communication between Airflow and data sources within your infrastructure.

    #### Prerequisites

    You must grant the following permissions for the Astronomer Service account, `astronomer@astro-remote-mgmt.iam.gserviceaccount.com`, in a GCP Project with the Network Connectivity Center Hub:

    * `roles/networkconnectivity.groupUser`
    * `roles/networkconnectivity.hubViewer`

    Retrieve the following information about your GCP project and Network Connectivity Center Hub:

    * GCP Project name
    * Hub name
    * Group name

    #### Contact Astronomer support for Network Connectivity Center Spoke configuration on Astro side

    Submit all collected details to [Astronomer support](https://cloud.astronomer.io/open-support-request). The Astronomer CRE team will contact you using your support ticket to ask follow-up questions, request clarification, or let you know about connectivity tests as they complete the required setup steps.
  </Tab>
</Tabs>

## Hostname resolution options

Securely connect Astro to resources running in other VPCs or on-premises through a resolving service.

As most flexible and reliable solution Astronomer recommends using Domain Name System (DNS) forwarding. In case of small mount of records and immutable IP addresses, support team can create a Private zone with DNS records, pointed to customer's resources.

<Tabs>
  <Tab title="Domain Name System forwarding">
    Use Domain Name System (DNS) forwarding to allow Astro to resolve DNS queries for resources running in other VPCs or on-premises. You have access to internal resources through private names. All changes in zone will be available for Astro environment immediately.

    To use this solution, make sure Astro can connect to the DNS server using a VPC peering or VPN connection and then submit a request to [Astronomer support](https://cloud.astronomer.io/open-support-request). With your request, include the following information:

    * The domain name for forwarding requests
    * The IP address of the DNS server where requests are forwarded

    ### (Optional) Create an Airflow connection to confirm connectivity

    After Astronomer support confirms that DNS forwarding was successfully set up, you can confirm that it works by creating an Airflow connection to a resource running in a VPC or on-premises. See [Managing Connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html).
  </Tab>

  <Tab title="Private hosted zone">
    Astronomer support can create Private hosted zones for reflecting particular DNS records in your environment without any changes or additional configurations. Private zones work well when the number of zones and records is small and stable. Otherwise, name resolution accuracy and connectivity in general can be affected.

    To use this solution, submit a request to [Astronomer support](https://cloud.astronomer.io/open-support-request). With your request, include the following information:

    * List of DNS records for the Private zone
    * IP addresses that have to be assigned respectively

    #### (Optional) Create an Airflow connection to confirm connectivity

    After Astronomer support confirms that DNS forwarding was successfully set up, you can confirm that it works by creating an Airflow connection to a resource running in a VPC or on-premises. See [Managing Connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html).
  </Tab>

  <Tab title="DNS peering">
    Astronomer can create a [DNS peering zone](https://cloud.google.com/dns/docs/zones/peering-zones) so an Astro Project can have read access to your DNS zone hosted in GCP.

    To use this solution, you must grant the Astronomer service account a role in your GCP project by adding an IAM policy binding. Replace `ZONE_OWNER_PROJECT_ID` in the following code with your GCP project name. Then, execute the following command:

    ```sh wrap theme={null}
    gcloud projects add-iam-policy-binding <ZONE_OWNER_PROJECT_ID> \
      --member='serviceAccount:astronomer@astro-remote-mgmt.iam.gserviceaccount.com' \
      --role=roles/dns.peer
    ```

    Submit a request to [Astronomer support](https://cloud.astronomer.io/open-support-request). With your request, include the following information about your infrastructure:

    * Private zone name
    * GCP Project name
    * Network name

    After Astronomer support confirms that DNS peering zone was successfully created, you can delete any role bindings that were previously created:

    ```sh wrap theme={null}
    gcloud projects remove-iam-policy-binding <ZONE_OWNER_PROJECT_ID> \
      --member='serviceAccount:astronomer@astro-remote-mgmt.iam.gserviceaccount.com' \
      --role=roles/dns.peer
    ```
  </Tab>
</Tabs>

## See Also

* [Manage Airflow connections and variables](/docs/astro/manage-connections-variables)
* [Authorize your Deployment using workload identity](/docs/astro/authorize-deployments-to-your-cloud)


# Configure Customer Managed Egress
Source: https://astronomer.io/docs/astro/customer-managed-egress

Enable Customer Managed Egress on a dedicated AWS cluster to control outbound traffic from Dag workloads through a Transit Gateway attachment.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

<Info>Customer Managed Egress is currently only available on dedicated AWS clusters.</Info>

Astro supports Customer Managed Egress for Dag Workloads on dedicated AWS clusters.

Customer Managed Egress for Dag Workloads lets you control egress to ensure compliance with security standards and regulations, and provides a data loss protection architecture to secure against unauthorized data transfer.

Enabling Customer Managed Egress through a Transit Gateway attachment between Astro and your corporate network allows you to manage and have full visibility into private and public data flows from your Astro Deployments and from Metrics Export configurations.

<Info>An icon on Deployments and Deployment details pages indicate when a Deployment is on a cluster with Customer Managed Egress enabled.</Info>

## Prerequisites

* An existing dedicated AWS cluster. [Create a dedicated cluster](/docs/astro/create-dedicated-cluster)
* Organization Owner user permissions. See [User permissions reference](/docs/astro/user-permissions) for more information.

## Step 1: Create a resource share for Transit Gateway with Astro and submit Transit Gateway ID

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Clusters**. For an existing dedicated AWS cluster, click the cluster you want to edit. Then, navigate to the **Customer Managed Egress for DAG Workloads** section of the Cluster Details page.

    2. Click **Configure Customer Managed Egress...**

    3. [Share your Transit Gateway](https://repost.aws/knowledge-center/transit-gateway-sharing) with your Astro cluster account using AWS Resource Access Manager (AWS RAM). Your Astro cluster Account ID is provided to enter into AWS RAM.

    4. Retrieve your [AWS Transit Gateway ID](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-transit-gateways.html) from your AWS console.

    5. Enter your AWS Transit Gateway ID into **Transit Gateway ID**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**, then click **Clusters**. For an existing dedicated AWS cluster, click the cluster you want to edit. Then, navigate to the **Customer Managed Egress for DAG Workloads** section of the Cluster Details page.

    2. Click **Configure Customer Managed Egress...**

    3. [Share your Transit Gateway](https://repost.aws/knowledge-center/transit-gateway-sharing) with your Astro cluster account using AWS Resource Access Manager (AWS RAM). Your Astro cluster Account ID is provided to enter into AWS RAM.

    4. Retrieve your [AWS Transit Gateway ID](https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-transit-gateways.html) from your AWS console.

    5. Enter your AWS Transit Gateway ID into **Transit Gateway ID**.
  </Tab>
</Tabs>

## Step 2: Astro accepts resource share and creates Transit Gateway attachment

Monitor the automatically generated Astro support ticket to accept the resource share and confirm Transit Gateway attachment is created. This activity is completed by Astronomer Support, so you don't need to take action during this step.

## Step 3: Enable Customer Managed Egress for Dag Workloads

1. Enable **Customer Managed Egress for DAG Workloads**.

2. Astro routes all public and private traffic to your Transit Gateway from your Astro Deployments and [Private Network Egress mode](/docs/astro/private-network-egress) is enabled.

<Warning>Resetting the Transit Gateway ID disables the routing of workload traffic (public and private) to your Transit Gateway, which might cause task failures. You must re-configure a new Transit Gateway to enable Customer Managed Egress again.</Warning>


# Data protection
Source: https://astronomer.io/docs/astro/data-protection

Learn how Astronomer uses encryption to protect clusters and data.

Astro uses both encryption in transit and encryption at rest to protect data across and within the Data Plane and Control Plane. Additionally, Deployment namespaces are network isolated with restricted communications. This document contains details about each type of encryption and isolation currently in place on Astro.

## Encryption in transit

All communication between control and data planes is encrypted in transit using [TLS](https://www.acunetix.com/blog/articles/tls-security-what-is-tls-ssl-part-1/) 1.2, strong ciphers, and secure transfer (data layer).

All customer data flows within the control plane transit through a mTLS mesh, enforcing TLS 1.2 and secure strong ciphers. Encrypted secret environment variables transit from the Cloud API to a secrets manager in the control plane using TLS 1.2 and strong ciphers. Data planes pull base64 encoded secret environment variables, along with other Airflow configurations over an encrypted TLS connection. As part of the application of configuration manifests in the data plane, all secret and sensitive information is stored in an encrypted etcd cluster at rest.

All internal service communication within the data plane is transmitted using TLS 1.2 and secure ciphers.

Every cluster in your data plane has its own certificates which were generated when the cluster was created and signed by Astronomer's certificate management platform. The certificates are automatically renewed every 90 days, and no longer require public ingress as part of the certificate signing process.

## Encryption at rest

All data at rest across control and data planes is encrypted with AES-256, one of the strongest block ciphers available. This is done using native cloud provider technologies.

Specifically, control plane data is encrypted on disk with a platform-managed key, including backups and the temporary files created while DB queries are running. Likewise, data plane data is server-side encrypted and volume encrypted, using encryption keys managed by the cloud provider and anchored by hardware security appliances. All resources provisioned across both planes leverage cloud provider envelope encryption wherever possible in accordance with a [defense in depth security strategy](https://en.wikipedia.org/wiki/Defense_in_depth_\(computing\)).

## Deployment network isolation

All pods and services specific to a single Deployment on Astro are isolated to a corresponding Kubernetes namespace within the Astro cluster in which the Deployment is hosted. All Deployment namespaces on Astro, including those running on the same cluster, are network isolated from each other by default. In addition to the isolation between Deployment namespaces, Astronomer also restricts communication within a namespace to only what is required between components and associated ports.

This level of network isolation is achieved using [network policies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) enabled by the [Calico](https://kubernetes.io/docs/concepts/cluster-administration/networking/#calico) kubernetes network plugin.

The network isolation between and within Deployment namespaces ensures that communication is restricted to only allow necessary communications within a namespace, and that communication between Deployments is denied. This ensures that unintended communications and attempted data exchanges are blocked.


# Custom role permissions reference
Source: https://astronomer.io/docs/astro/deployment-role-reference

Learn about each possible permission that you can assign to custom Deployment and Dag roles.

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

This document contains all available permissions that you can assign to a [custom Deployment role](/docs/astro/customize-deployment-roles) or a [custom Dag role](/docs/astro/dag-level-access-control#create-a-custom-dag-role). Permissions are organized by scope and object.

## Deployment

* `deployment.get`: Get information about the Deployment, including its details and resource configurations.
* `deployment.update`: Update information about the Deployment, including its details and resource configurations.
* `deployment.delete`: Delete the Deployment.

## Deployment Agents

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Remote Execution Agents are Airflow Workers, Triggerers, and Dag processors running in your remote environment with an Airflow 3 Deployment.

* `deployment.agents.get`: Able to get the information about Remote Execution Agents running in Remote Deployments.
* `deployment.agents.update`: Allow users with permissions to cordon and uncordon Remote Execution Agents in Remote Deployments.
* `deployment.agents.delete`: Allow users with permissions to delete Remote Execution Agents in Remote Deployments.

## Deployment Agents Tokens

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Tokens used by external Airflow agents, including the Remote Execution Agent, to access Airflow APIs for remote task execution.

* `deployment.agentTokens.get`: View access tokens for Remote Agents.
* `deployment.agentTokens.create`: Create an access token to your Deployment for a Remote Execution Agent.
* `deployment.agentTokens.delete`: Delete access tokens for Remote Agents.

## Deployment Airflow AdminMenu

* `deployment.airflow.adminMenu.get`: View the **Admin** menu in the Airflow UI.

## Deployment Airflow Astronomer

* `deployment.airflow.astronomer.get`: View the **Astronomer** menu in the Airflow UI.

## Deployment Airflow AuditLog

* `deployment.airflow.auditLog.get`: View the **Audit Log** menu in the Airflow UI.

## Deployment Airflow Backfill

* `deployment.airflow.backfill.get`: View backfills in the Airflow UI.
* `deployment.airflow.backfill.create`: Create backfills in the Airflow UI.
* `deployment.airflow.backfill.update`: Update backfills in the Airflow UI.
* `deployment.airflow.backfill.delete`: Delete backfills in the Airflow UI.

## Deployment Airflow BrowseMenu

* `deployment.airflow.browseMenu.get`: View the **Browse** menu in the Airflow UI.

## Deployment Airflow ClusterActivity

* `deployment.airflow.clusterActivity.get`: View the **Cluster Activity** menu in the Airflow UI.

## Deployment Airflow Config

* `deployment.airflow.config.get`: View the **Config** menu in the Airflow UI.

## Deployment Airflow custom menu

* `deployment.airflow.customMenu.get`: Create and use custom plugins for the Airflow UI menu.

### Deployment Airflow Connection

* `deployment.airflow.connection.get`: View connections in the Airflow UI.
* `deployment.airflow.connection.create`: Create connections in the Airflow UI.
* `deployment.airflow.connection.update`: Update connections in the Airflow UI.
* `deployment.airflow.connection.delete`: Delete connections in the Airflow UI.

### Deployment Airflow dag

* `deployment.airflow.dag.get`: View dags in the Airflow UI.
* `deployment.airflow.dag.update`: Update dags in the Airflow UI.
* `deployment.airflow.dag.delete`: Delete dags in the Airflow UI.

## Deployment Airflow DagCode

* `deployment.airflow.dagCode.get`: View the source code for dags in the Airflow UI.

## Deployment Airflow DagDependencies

* `deployment.airflow.dagDependencies.get`: View the task dependencies for dags in the Airflow UI.

### Deployment Airflow DAGRun

* `deployment.airflow.dagRun.get`: View dag runs in the Airflow UI.
* `deployment.airflow.dagRun.create`: Create dag runs in the Airflow UI.
* `deployment.airflow.dagRun.update`: Update dag runs in the Airflow UI.
* `deployment.airflow.dagRun.delete`: Delete dag runs in the Airflow UI.

## Deployment Airflow Datasets

* `deployment.airflow.datasets.get`: View the datasets for dags in the Airflow UI.
* `deployment.airflow.datasets.create`: Create datasets in the Airflow UI.
* `deployment.airflow.datasets.delete`: Delete datasets in the Airflow UI.

## Deployment Airflow Docs

* `deployment.airflow.docs.get`: View the documentation for dags in the Airflow UI.

## Deployment Airflow HITL detail

* `deployment.airflow.hitlDetail.get`: View human-in-the-loop task details in the Airflow UI.
* `deployment.airflow.hitlDetail.update`: Approve human-in-the-loop task actions in the Airflow UI.

## Deployment Airflow ImportError

* `deployment.airflow.importError.get`: View any import errors for dags in the Airflow UI.

## Deployment Airflow Job

* `deployment.airflow.job.get`: View scheduler jobs in the Airflow UI.

## Deployment Airflow Plugin

* `deployment.airflow.plugin.get`: View plugins in the Airflow UI.

## Deployment Airflow Pool

* `deployment.airflow.pool.get`: View pools in the Airflow UI.
* `deployment.airflow.pool.create`: Create pools in the Airflow UI.
* `deployment.airflow.pool.update`: Update pools in the Airflow UI.
* `deployment.airflow.pool.delete`: Delete pools in the Airflow UI.

## Deployment Airflow Provider

* `deployment.airflow.provider.get`: View installed provider packages in the Airflow UI.

## Deployment Airflow RequiredActionsMenu

* `deployment.airflow.requiredActionsMenu.get`: View the **Required Actions** menu in the Airflow UI.

## Deployment Airflow SlaMiss

* `deployment.airflow.slaMiss.get`: View details about SLA misses in the Airflow UI.

## Deployment Airflow task

* `deployment.airflow.task.get`: View tasks in the Airflow UI.
* `deployment.airflow.task.create`: Create tasks in the Airflow UI.
* `deployment.airflow.task.update`: Update tasks in the Airflow UI.
* `deployment.airflow.task.delete`: Delete tasks in the Airflow UI.

## Deployment Airflow TaskInstance

* `deployment.airflow.taskInstance.get`: View task instances in the Airflow UI.
* `deployment.airflow.taskInstance.create`: Create task instances in the Airflow UI.
* `deployment.airflow.taskInstance.update`: Update task instances in the Airflow UI.
* `deployment.airflow.taskInstance.delete`: Delete task instances in the Airflow UI.

## Deployment Airflow TaskLog

* `deployment.airflow.taskLog.get`: View task logs in the Airflow UI.

## Deployment Airflow TaskReschedule

* `deployment.airflow.taskReschedule.get`: View task reschedules in the Airflow UI.

## Deployment Airflow Trigger

* `deployment.airflow.trigger.get`: View information about triggers in the Airflow UI.

## Deployment Airflow Variable

* `deployment.airflow.variable.get`: View variables in the Airflow UI.
* `deployment.airflow.variable.create`: Create variables in the Airflow UI.
* `deployment.airflow.variable.update`: Update variables in the Airflow UI.
* `deployment.airflow.variable.delete`: Delete variables in the Airflow UI.

## Deployment Airflow version

* `deployment.airflow.version.get`: View Dag version information in the Airflow UI.

## Deployment Airflow warning

* `deployment.airflow.warning.get`: View Dag warnings in the Airflow UI.
* `deployment.airflow.warning.delete`: Delete Dag warnings in the Airflow UI.

## Deployment Airflow Website

* `deployment.airflow.website.get`: Access the Airflow UI homepage.

## Deployment Airflow XCom

* `deployment.airflow.xcom.get`: View XComs data in the Airflow UI.
* `deployment.airflow.xcom.create`: Create XCom entries in the Airflow UI.
* `deployment.airflow.xcom.update`: Update XCom entries in the Airflow UI.
* `deployment.airflow.xcom.delete`: Delete XCom data in the Airflow UI.

## Deployment Alerts

* `deployment.alerts.get`: View configured Deployment alerts.
* `deployment.alerts.create`: Create Deployment-level Astro alerts in the Astro UI or through the Astro API.
* `deployment.alerts.update`: Update Deployment-level Astro alerts in the Astro UI or through the Astro API.
* `deployment.alerts.delete`: Delete Deployment-level Astro alerts in the Astro UI or through the Astro API.

## Deployment ApiTokens

* `deployment.apiTokens.get`: View Deployment API tokens in the Astro UI or through the Astro API.
* `deployment.apiTokens.create`: Create Deployment API tokens in the Astro UI or through the Astro API.
* `deployment.apiTokens.update`: Update Deployment API tokens in the Astro UI or through the Astro API.
* `deployment.apiTokens.delete`: Delete Deployment API tokens in the Astro UI or through the Astro API.
* `deployment.apiTokens.rotate`: Rotate Deployment API tokens in the Astro UI or through the Astro API.

## Deployment Deploys

* `deployment.deploys.get`: View deploy history for the Deployment in the Astro UI or through the Astro API.
* `deployment.deploys.create`: Make Deploys to the Deployment.
* `deployment.deploys.update`: Roll back deploys to the Deployment.

## Deployment EnvObjects

* `deployment.envObjects.get`: View Airflow objects created in the Astro **Environments** page.
* `deployment.envObjects.create`: Create Airflow objects in the Astro **Environments** page.
* `deployment.envObjects.update`: Update Airflow objects in the Astro **Environments** page.
* `deployment.envObjects.delete`: Delete Airflow objects in the Astro **Environments** page.

## Deployment Image

* `deployment.image.push`: Push an image to the Deployment.
* `deployment.image.pull`: Pull the Deployment's image.

<Note>
  `astro deploy` checks these permissions when it runs. Pushing an image often requires both `deployment.image.push` and `deployment.image.pull`, because the push can reference pieces of a previously pushed image.
</Note>

## Deployment incidents

* `deployment.incidents.get`: View incidents affecting the Deployment in the Astro UI.

## Deployment Logs

* `deployment.logs.get`: View Deployment logs in the Astro UI.

## Deployment Metrics

* `deployment.metrics.get`: View Deployment **Analytics** in the Astro UI.

## Deployment notification channels

* `deployment.notificationChannels.get`: View notification channels configured for the Deployment.
* `deployment.notificationChannels.create`: Create notification channels for the Deployment.
* `deployment.notificationChannels.update`: Update notification channels for the Deployment.
* `deployment.notificationChannels.delete`: Delete notification channels for the Deployment.

## Deployment Observability

* `deployment.observability.event.create`: Upload OpenLineage events.
* `deployment.observability.metrics.create`: Upload OpenLineage metrics.

## Deployment Teams

* `deployment.teams.get`: View Teams belonging to a Deployment.
* `deployment.teams.update`: Update Team membership to a Deployment.

## Deployment Users

* `deployment.users.get`: View users belonging to a Deployment.
* `deployment.users.update`: Update user membership to a Deployment.

## Deployment webhook

* `deployment.webhook.get`: View webhooks configured for the Deployment.
* `deployment.webhook.create`: Create webhooks for the Deployment.
* `deployment.webhook.update`: Update webhooks for the Deployment.
* `deployment.webhook.delete`: Delete webhooks for the Deployment.

## Dag scope permissions

The following permissions are available when creating a custom role with the **Scope** set to **Dag**. Unlike Deployment-scope permissions that apply to all Dags in a Deployment, Dag-scope permissions apply only to the Dags that the role is bound to by Dag tag or Dag ID. See [Dag-level access control](/docs/astro/dag-level-access-control) for more information about creating and assigning custom Dag roles.

### Dag Airflow AuditLog

* `dag.airflow.auditLog.get`: View the audit log for the assigned Dags.

### Dag Airflow Dag

* `dag.airflow.dag.get`: View the assigned Dags in the Airflow UI.
* `dag.airflow.dag.update`: Update the assigned Dags in the Airflow UI.
* `dag.airflow.dag.delete`: Delete the assigned Dags in the Airflow UI.

### Dag Airflow DagCode

* `dag.airflow.dagCode.get`: View the source code for the assigned Dags in the Airflow UI.

### Dag Airflow DagDependencies

* `dag.airflow.dagDependencies.get`: View the task dependencies for the assigned Dags in the Airflow UI.

### Dag Airflow DagRun

* `dag.airflow.dagRun.get`: View Dag runs for the assigned Dags in the Airflow UI.
* `dag.airflow.dagRun.create`: Create Dag runs for the assigned Dags in the Airflow UI.
* `dag.airflow.dagRun.update`: Update Dag runs for the assigned Dags in the Airflow UI.
* `dag.airflow.dagRun.delete`: Delete Dag runs for the assigned Dags in the Airflow UI.

### Dag Airflow HitlDetail

* `dag.airflow.hitlDetail.get`: View human-in-the-loop task details for the assigned Dags.
* `dag.airflow.hitlDetail.update`: Update human-in-the-loop task details for the assigned Dags.

### Dag Airflow Task

* `dag.airflow.task.get`: View tasks for the assigned Dags in the Airflow UI.

### Dag Airflow TaskInstance

* `dag.airflow.taskInstance.get`: View task instances for the assigned Dags in the Airflow UI.
* `dag.airflow.taskInstance.update`: Update task instances for the assigned Dags in the Airflow UI.
* `dag.airflow.taskInstance.delete`: Delete task instances for the assigned Dags in the Airflow UI.

### Dag Airflow TaskLog

* `dag.airflow.taskLog.get`: View task logs for the assigned Dags in the Airflow UI.

### Dag Airflow Version

* `dag.airflow.version.get`: View version details for the assigned Dags.

### Dag Airflow Warning

* `dag.airflow.warning.get`: View warnings for the assigned Dags.

### Dag Airflow Xcom

* `dag.airflow.xcom.get`: View XCom data for the assigned Dags in the Airflow UI.
* `dag.airflow.xcom.create`: Create XCom entries for the assigned Dags in the Airflow UI.
* `dag.airflow.xcom.update`: Update XCom entries for the assigned Dags in the Airflow UI.


# Astro deprecations
Source: https://astronomer.io/docs/astro/deprecations

Learn about Astro feature deprecations

Astronomer marks features as "deprecated" to evolve our products and services to meet modern best-practices. Deprecated features are not recommended for production use and will be removed in the future. Deprecations of GA features align with our support commitments and policies.

To get more information or file a support request for a deprecated feature, see [Submit a support request](/docs/astro/astro-support).


# Disaster recovery
Source: https://astronomer.io/docs/astro/disaster-recovery

Learn how Astronomer handles disaster recovery scenarios and how to set up self-service cross-region disaster recovery for dedicated clusters.

The Astro Data Plane is designed to withstand in-region Availability Zone (AZ) degradations and outages as described in [Resilience](/docs/astro/resilience). For full region outages on dedicated clusters, Astro supports self-service cross-region disaster recovery (DR).

For a detailed overview of Astro's AWS disaster recovery architecture, see the [AWS disaster recovery whitepaper](https://trust.astronomer.io/?itemUid=ace38601-eed9-412b-970e-49b8d729c77a\&source=click) in the Astronomer Trust Center.

## Cross-region disaster recovery

Cross-region disaster recovery requires the Enterprise Business Critical tier. It is generally available for AWS and GCP dedicated clusters. Azure support is planned.

Cross-region DR lets you configure a pair of dedicated clusters, a primary and a secondary, in two regions of the same cloud provider. The secondary cluster stays continuously synchronized with the primary so you can fail over with minimal downtime and data loss. After failover, Astro automatically enables synchronization in the reverse direction, keeping the original primary ready for failback. When the primary region recovers, you can fail back with a single click.

### How disaster recovery works

* The primary cluster runs all Deployments in Region A.
* A multi-region database replicates Deployment metadata to the secondary cluster in Region B.
* Multi-region object storage copies task logs to the secondary cluster.
* User-deployed images are replicated to the secondary cluster.
* On failover, the secondary cluster is promoted to active. All Deployments, configuration, environment variables, connections, and Airflow variables transfer automatically.
* Clusters and Deployments retain their IDs, names, namespaces, and system-managed configuration after failover. All hostnames — including the Airflow UI, Airflow API, and Remote Execution API URLs — are updated to point to the secondary cluster and remain the same.
* For Deployments that use Remote Execution, only the Astro-managed orchestration plane fails over. The execution plane runs in your own infrastructure and isn't part of Astro's region failover, but your Remote Execution Agents reconnect to the promoted secondary cluster automatically. See [Remote Execution in DR pairs](/docs/astro/disaster-recovery-failover#remote-execution-in-dr-pairs).

### RTO and RPO

The following table defines the recovery time objective (RTO) and recovery point objective (RPO) for DR clusters. Targets are benchmarked with 80+ Deployments and 1,250+ concurrent task runs.

| Metric                         | Target                                                    |
| ------------------------------ | --------------------------------------------------------- |
| Recovery time objective (RTO)  | Less than one hour                                        |
| Recovery point objective (RPO) | Less than 15 minutes (requires Task Logs Replication SLA) |

See [Task Logs Replication SLA](/docs/astro/disaster-recovery-prepare#task-logs-replication-sla) for details on the RPO guarantee.

### What gets failed over

The following items transfer to the secondary cluster automatically during failover:

* Deployments and data pipelines
* Dag run history, task instance metadata, and XComs
* Deployment configuration
* Environment variables, connections, Airflow variables, and metrics exports — whether configured via Environment Manager or directly on the Deployment
* Task logs. Enable Task Logs Replication SLA for a guaranteed 15-minute RPO.

The following items do not transfer automatically and require manual steps after configuring the secondary cluster:

* Networking and DNS configuration. Configure these using self-service networking features, or work with [Astronomer support](https://support.astronomer.io). See [Networking considerations](/docs/astro/disaster-recovery-prepare#networking-considerations).
* `imagePullSecrets` for Kubernetes Pod Operators (KPOs)
* Customer-managed workload identities. You must configure the workload identity and trust relationships for the secondary cluster separately. See [Workload identity](/docs/astro/disaster-recovery-prepare#workload-identity).
* Customer-managed network routing on the secondary cluster, such as AWS Transit Gateway
* Remote Execution Agents and their supporting infrastructure. Agents run in your own environment and aren't part of Astro's region failover. See [Remote Execution in DR pairs](/docs/astro/disaster-recovery-failover#remote-execution-in-dr-pairs).


# Trigger failover and failback
Source: https://astronomer.io/docs/astro/disaster-recovery-failover

Trigger a failover to your secondary cluster or fail back to the primary cluster after it recovers.

## Trigger failover

<Note>
  Failover is also supported using the [Astro API](https://www.astronomer.io/docs/astro/api).
</Note>

<Warning>
  Running, scheduled, and event-triggered tasks may be impacted during the failover window. Tasks may fail and require a retry.
</Warning>

<Steps>
  <Step title="Open the DR tab">
    In the Astro UI, go to **Settings** > **Clusters** (**Organization Settings** > **Clusters** in the legacy UI), select your primary cluster, and open the **Disaster Recovery** tab. Confirm the **Status** indicator shows the primary region is active.
  </Step>

  <Step title="Initiate failover">
    On the **Disaster Recovery** tab, click **Failover**. Alternatively, open the cluster's actions menu (**⋯**) at the top right of the page and select **Failover to Secondary…**. Follow the prompts to confirm.

    The secondary cluster is promoted to active, and all Deployments and data become available in the secondary cluster.
  </Step>

  <Step title="Validate Deployments">
    After failover completes, check the health and status of your Deployments, especially mission-critical ones. Validate that your Dags and tasks are running as expected and retry any failures if necessary.
  </Step>
</Steps>

## Trigger failback

After the primary region recovers, you can fail back to the original primary cluster.

<Note>
  Failback is also supported using the [Astro API](https://www.astronomer.io/docs/astro/api).
</Note>

<Warning>
  Running, scheduled, and event-triggered tasks may be impacted during the failback window. Tasks may fail and require a retry.
</Warning>

<Steps>
  <Step title="Open the DR tab">
    In the Astro UI, go to **Settings** > **Clusters** (**Organization Settings** > **Clusters** in the legacy UI), select your original primary cluster, and open the **Disaster Recovery** tab. Confirm the **Status** indicator shows the cluster has failed over to the secondary region.
  </Step>

  <Step title="Initiate failback">
    On the **Disaster Recovery** tab, click **Failback**. Alternatively, open the cluster's actions menu (**⋯**) at the top right of the page and select **Failback to Primary…**. Follow the prompts to confirm.
  </Step>
</Steps>

## Remote Execution in DR pairs

<Warning>
  As with other Deployments, running, scheduled, and event-triggered tasks may be impacted during the failover window and might fail and require a retry.
</Warning>

Deployments that use [Remote Execution](/docs/astro/remote-execution-overview) are supported in DR cluster pairs. During failover, Astro promotes the orchestration plane in the secondary cluster and the Remote Execution API URL remains the same, so your Remote Execution Agents reconnect to the secondary cluster without changing their configuration.

<Note>
  Only Remote Execution Agents running version `1.8.0 ` and later detect that a failover occurred and reconnect automatically to the secondary cluster. Agents on earlier versions might need to be restarted after a failover.
</Note>

The execution plane runs in your own infrastructure and isn't part of Astro's region failover. To keep tasks running after failover, plan for the following:

* Run your Remote Execution Agents in a location that remains available during an outage of the primary Astro region, or have standby Remote Execution Agents in the same secondary region as your Astro orchestration plane. Agent infrastructure and its availability are your responsibility. See the [Remote Execution shared responsibility model](/docs/astro/remote-execution-shared-responsibility).
* If you use [AWS PrivateLink](/docs/astro/remote-agents-aws-privatelink) or [Azure Private Link](/docs/astro/remote-agents-azure-privatelink) for agent connectivity, configure private connectivity to the secondary cluster separately. Private endpoints are region-specific and don't transfer automatically during failover.

## Universal Metrics Export in DR pairs

If you have Universal Metrics Export (UME) configured, the same UME configuration applies to both the primary and secondary clusters. Metrics exported from each cluster include a `cloud_region` attribute so you can distinguish data from each cluster in your metrics system.

After failover, update your UME settings if needed to reflect the new active cluster.


# Prepare for disaster recovery
Source: https://astronomer.io/docs/astro/disaster-recovery-prepare

Prepare your Astro environment for cross-region disaster recovery, including networking, workload identity, and Dag configuration.

## Networking considerations

Networking primitives and DR network requirements differ by cloud provider.

<Tabs>
  <Tab title="AWS">
    **VPC CIDR for the secondary cluster**

    By default, the secondary cluster uses the same VPC subnet range and Pod CIDR range as the primary cluster. Astronomer recommends configuring a different CIDR range for the secondary cluster to avoid IP conflicts when both clusters are connected to shared networks. You can set a different CIDR range for the secondary cluster when you create the primary cluster. See [Create a dedicated Astro cluster](/docs/astro/create-dedicated-cluster).

    **VPC peering**

    Self-service VPC peering is supported for both clusters. After Astronomer creates the secondary cluster, you must create all VPC peering connections and routes for the secondary cluster. Astronomer does not automatically initiate any peerings after creating the secondary cluster.

    **Private Network Egress**

    If Private Network Egress (PNE) is enabled on the primary cluster, Astronomer enables it on the secondary cluster as well.

    **Customer Managed Egress and Transit Gateway**

    Customer Managed Egress (CME) is self-service and must be configured separately for both the primary and secondary clusters. See [Customer Managed Egress](/docs/astro/customer-managed-egress).
  </Tab>

  <Tab title="GCP">
    **DR CIDR ranges for the secondary cluster**

    GCP requires four network ranges for the secondary cluster: a DR VPC subnet range, DR Pod CIDR range, DR service subnet range, and DR service peering range. Each range is required and must not overlap with the corresponding range on the primary cluster. Unlike AWS, GCP does not default these ranges from the primary cluster. Configure them when you create the cluster or in the DR enablement request. See [Create a dedicated Astro cluster](/docs/astro/create-dedicated-cluster).

    **Region pairing**

    The secondary region must be compatible with the primary region, based on supported GCP dual-region pairings. For example, the region `us-east5` can pair only with `us-central1` or `us-east1`. The Astro UI shows only compatible secondary regions when you configure DR.

    **Networking and DNS**

    Unlike AWS, GCP networking connectivity isn't self-service. To configure networking for either the primary or secondary cluster, including VPC peering and DNS, contact [Astronomer support](https://support.astronomer.io).
  </Tab>
</Tabs>

## Workload identity

**Astro-managed workload identity**

If you use Astro-managed workload identity, the same workload identity is used in both the primary and secondary regions.

**Customer-managed workload identity**

If you use customer-managed workload identity, the secondary cluster defaults to the Astro-managed workload identity. You must configure the workload identity and IAM policy binding for the secondary cluster separately. You can only do this after Astronomer creates the secondary cluster, because the identity provider information for the secondary cluster is not available until then. You can do this in the Deployment details **Advanced** section.

## Task Logs Replication SLA

The Task Logs Replication SLA is an optional feature that guarantees a 15-minute RPO for task logs. Enabling this feature incurs additional pass-through costs.

You can enable the Task Logs Replication SLA when you create a new DR cluster pair, or after creating. See [Create a dedicated Astro cluster](/docs/astro/create-dedicated-cluster).

## Prepare Dags for disaster recovery

Astro automatically sets the `ASTRONOMER_IS_DR_ENV` environment variable on all Deployments in a DR cluster pair:

* **Secondary cluster**: `ASTRONOMER_IS_DR_ENV=True`
* **Primary cluster**: The variable is not set.

Use this variable in your Dag code to branch logic based on whether a Deployment is running on the secondary cluster. For example, to switch connections, change resource configurations, or skip certain tasks during a DR event. Astronomer recommends updating your Dags to handle this variable before triggering a failover.

Alternatively, you can update relevant configuration such as connections and environment variables after failover using the [Astro API](https://docs.astronomer.io/astro/api) or [Terraform](https://registry.terraform.io/providers/astronomer/astro/latest/docs).


# Set up disaster recovery
Source: https://astronomer.io/docs/astro/disaster-recovery-setup

Enable cross-region disaster recovery on new or existing dedicated clusters.

## Set up DR on a new cluster

You can enable DR when creating a new dedicated cluster through the Astro UI. See [Create a dedicated Astro cluster](/docs/astro/create-dedicated-cluster) for configuration steps and field descriptions. After the cluster is created, complete the [required steps after enabling DR](#required-steps-after-enabling-dr) before triggering a failover.

<Info>
  Terraform support for DR cluster creation are planned for general availability (GA).
</Info>

<Note>
  DR cluster creation is also supported using the [Astro API](https://www.astronomer.io/docs/astro/api).
</Note>

## Set up DR on an existing cluster

To enable DR on an existing dedicated cluster, submit a support request through the Astro UI. Astronomer processes the request during your specified maintenance window.

#### Prerequisites

* Organization Owner role with `organization.clusters.update` permission
* A dedicated AWS or GCP cluster that is not already DR-enabled

#### Submit a DR enablement request

<Steps>
  <Step title="Open the support request form">
    You can open the form in two ways:

    * From the **Disaster Recovery** tab: In the Astro UI, go to **Settings** > **Clusters** (**Organization Settings** > **Clusters** in the legacy UI), select your cluster, open the **Disaster Recovery** tab, then click **Enable Disaster Recovery**. The form opens for your cluster's cloud provider.
    * From the support menu: In the Astro UI, open **New Support Request** and select the request type for your cloud provider, either **Enable AWS Data Plane Disaster Recovery** or **Enable GCP Data Plane Disaster Recovery**.
  </Step>

  <Step title="Configure the request">
    Complete the following fields. The required network ranges depend on your cluster's cloud provider:

    <Tabs>
      <Tab title="AWS">
        * **Cluster**: Select the AWS cluster to enable DR on. Only eligible clusters appear. The cluster must be an AWS dedicated cluster that isn't already DR-enabled.
        * **Failover Region**: Select the AWS region for the secondary cluster.
        * **DR VPC Subnet Range**: (Optional) Specify a VPC subnet range for the secondary cluster. Leave blank to use the same range as the primary cluster.
        * **DR Pod CIDR Range**: (Optional) Specify a Pod CIDR range for the secondary cluster. Leave blank to use the same range as the primary cluster.
        * **Task Logs Replication SLA**: Enable to guarantee a 15-minute RPO for task logs. Additional charges apply. See [Task Logs Replication SLA](/docs/astro/disaster-recovery-prepare#task-logs-replication-sla).
        * **Maintenance Window**: Select a date and time for the maintenance window. The date must be at least 5 days from today. Weekends are not available.
        * **Additional Details**: (Optional) Include any additional context or requirements.
        * **CC Emails**: (Optional) Add email addresses to copy on the support ticket.
      </Tab>

      <Tab title="GCP">
        * **Cluster**: Select the GCP cluster to enable DR on. Only eligible clusters appear. The cluster must be a GCP dedicated cluster that isn't already DR-enabled.
        * **Failover Region**: Select the GCP region for the secondary cluster. Only regions compatible with your primary region appear, based on supported GCP dual-region pairings.
        * **DR VPC Subnet Range**: Specify the VPC subnet range for the secondary cluster. This range is required and must not overlap with the primary **VPC Subnet Range**.
        * **DR Pod CIDR Range**: Specify the Pod range for the secondary cluster. This range is required and must not overlap with the primary **Pod Subnet Range**.
        * **DR Service Subnet Range**: Specify the Service range for the secondary cluster. This range is required and must not overlap with the primary **Service Subnet Range**.
        * **DR Service Peering Range**: Specify the Private Service connection range for the secondary cluster. This range is required and must not overlap with the primary **Service Peering Range**.
        * **Task Logs Replication SLA**: Enable to guarantee a 15-minute RPO for task logs. Additional charges apply. See [Task Logs Replication SLA](/docs/astro/disaster-recovery-prepare#task-logs-replication-sla).
        * **Maintenance Window**: Select a date and time for the maintenance window. The date must be at least 5 days from today. Weekends are not available.
        * **Additional Details**: (Optional) Include any additional context or requirements.
        * **CC Emails**: (Optional) Add email addresses to copy on the support ticket.
      </Tab>
    </Tabs>
  </Step>

  <Step title="Submit the request">
    Click **Submit Support Request**. Astronomer confirms the maintenance window and contacts you before beginning the conversion.
  </Step>
</Steps>

<Warning>
  Running, scheduled, and event-triggered tasks are affected during the maintenance window. Tasks may fail and require a retry. Plan for 1-2 hours of downtime during the maintenance window.
</Warning>

The conversion process depends on your cluster's cloud provider:

<Tabs>
  <Tab title="AWS">
    * **Database migration**: Migrates the metadata database. This phase requires no downtime.
    * **Infrastructure switch**: Enables cross-region replication. This phase requires approximately 1-2 hours of maintenance downtime.
  </Tab>

  <Tab title="GCP">
    * **Configuring the DR region**: Astro copies your cluster's storage buckets to a dual-region configuration. This phase requires no downtime.
    * **Enabling DR**: Astro finalizes the storage relocation and enables cross-region replication. Each bucket has a brief write outage as its location changes.
  </Tab>
</Tabs>

Enabling DR incurs additional Astro credits. Secondary cluster resources and data replication incur ongoing charges based on your cluster configuration.

## Disable DR

Disabling DR deprovisions the secondary cluster and deletes all compute and data stores in the secondary region. This stops data replication and can't be undone without re-enabling DR.

<Warning>
  You can disable DR only while the cluster is running from its primary region. If the cluster has failed over to the secondary region, Astro blocks the request until you [fail back to the primary region](/docs/astro/disaster-recovery-failover#trigger-failback).
</Warning>

<Note>
  Disabling DR is also supported using the [Astro API](https://www.astronomer.io/docs/astro/api).
</Note>

#### Prerequisites

* Organization Owner role with `organization.clusters.update` permission

#### Disable DR on your cluster

<Steps>
  <Step title="Open the DR tab">
    In the Astro UI, go to **Settings** > **Clusters** (**Organization Settings** > **Clusters** in the legacy UI), select your DR-enabled cluster, and open the **Disaster Recovery** tab.
  </Step>

  <Step title="Disable DR">
    On the **Disaster Recovery** tab, click **Disable**. Alternatively, open the cluster's actions menu (**⋯**) at the top right of the page and select **Disable Disaster Recovery…**. Enter the cluster name to confirm, then click **Disable Disaster Recovery**.
  </Step>
</Steps>

<Warning>
  This action is irreversible without re-enabling DR. Ensure you no longer need cross-region failover capability before disabling DR.
</Warning>

## Required steps after enabling DR

After Astronomer creates the secondary cluster, complete the following steps before triggering a failover:

1. **Networking and DNS**: Configure all required networking and DNS customizations for the secondary cluster. See [Networking considerations](/docs/astro/disaster-recovery-prepare#networking-considerations).
2. **imagePullSecrets**: If your Deployments use Kubernetes Pod Operators (KPOs), configure `imagePullSecrets` on the secondary cluster. See [Pull images from a private registry](/docs/astro/kpo-private-registry).
3. **Customer-managed workload identity**: If your Deployments use customer-managed workload identities, configure the appropriate workload identity and update the trust relationships for the secondary cluster. See [Workload identity](/docs/astro/disaster-recovery-prepare#workload-identity).
4. **Dag logic**: Update your Dag logic to handle the `ASTRONOMER_IS_DR_ENV` environment variable for secondary-specific connections or configurations. See [Prepare Dags for disaster recovery](/docs/astro/disaster-recovery-prepare#prepare-dags-for-disaster-recovery).


# Astronomer feature lifecycle
Source: https://astronomer.io/docs/astro/feature-previews

Learn about the stages Astronomer products and features go through, from experimentation to general availability.

Astronomer uses a structured release program to bring new products and features from early experimentation to general availability (GA). This document defines each stage and what it means for you, including stability, support, and SLA coverage at each phase.

<Note>
  This lifecycle program applies to any Astronomer-released product or feature carrying a Labs, Preview, or GA designation, including Astro platform features, Astronomer-owned open source projects, and standalone tools and applications. It doesn't apply to Apache Airflow open source software, which follows the Apache release process.
</Note>

To view the support commitment for GA releases of Astro Runtime, see [Astro Runtime maintenance and lifecycle policy](/docs/runtime/runtime-version-lifecycle-policy). To submit feedback or a support request for a Labs or Preview feature, see [Submit a support request](/docs/astro/astro-support).

## Labs

Labs features are experimental and might not be fully stable. Frequent and breaking changes can occur based on user feedback. Labs features might not be fully documented and have limited support, no SLA, and no guarantee of continued development. Astronomer provides Labs features at no extra cost, although they might become paid features when they reach Preview or general availability. Access to Labs features varies by product. See the relevant feature documentation for details.

## Preview

Preview features are stable and ready for production use. Some planned additions and modifications to feature behaviors can occur before they become generally available. Preview features are fully documented, have reduced support through standard channels with a 24-hour response time, and have a reduced 95% uptime SLA. For paid Preview features, pricing is subject to change. Access to Preview features depends on your Astro plan.

## GA

GA features are production-hardened, with proven stability and reliability across many Deployments. Full SLAs and support apply per your Astro plan.

## Deprecated

Deprecated features are no longer under active development. They remain functional and documented until Astronomer establishes a removal or end-of-life date, but Astronomer no longer provides technical support for them. Astronomer notifies you of feature deprecations directly and through release notes.


# GDPR compliance
Source: https://astronomer.io/docs/astro/gdpr-compliance

Learn how Astronomer and Astro are GDPR compliant.

## What is the GDPR?

The [General Data Protection Regulation](https://gdpr-info.eu/) (GDPR) is a legal framework that sets guidelines for the collection and processing of personal information from individuals who live in the [European Economic Area](https://www.gov.uk/eu-eea) (EEA). The European privacy law became enforceable on May 25, 2018, and replaces the EU’s [Data Protection Directive](http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CELEX:31995L0046:en:HTML), which went into effect in 1995. The GDPR is intended to harmonize national data privacy laws throughout the EEA and enhance the protection of all EEA residents with respect to their personal data.

The European Commission provides official definitions about the legislation [here](https://ec.europa.eu/info/law/law-topic/data-protection/data-protection-eu_en).

## Who is impacted by GDPR?

The GDPR applies to companies that are established within and outside of the EU that offer goods or services to EEA residents, or monitor their behavior. In essence, it impacts and applies to all companies processing and holding the personal data of data subjects located in the EEA. The GDPR defines "[personal data](https://gdpr-info.eu/issues/personal-data/)" to be "any information which are related to an identified or identifiable natural person".

## Is Astronomer subject to GDPR?

Astronomer is subject to GDPR because the personal data of EEA residents is processed and stored through consumption of Astronomer services.

## Is my company subject to GDPR?

If there is a possibility that your company collects or processes personal data of individuals located in the EEA, you are most likely subject to GDPR. Confirm with your privacy and legal counsel.

## How does using Astro help me comply with the GDPR?

Simply using Astro does not ensure compliance with GDPR, but the combination of a Data Processing Agreement (DPA), Astro architecture, and various controls, features, and modules available in Astro, Astro Runtime, and the Airflow Registry can help you with your GDPR compliance.

Astro is designed and architected with security and privacy by default. Astro boasts a hybrid deployment model founded on a control plane hosted by Astronomer and a data plane that is hosted in your cloud environment. Both are fully managed by Astronomer. This model offers the self-service convenience of a fully managed service while respecting the need to keep data private, secure, and within corporate boundaries.

All customer business data never leaves your environment (for example, a cloud database) or is required to be uploaded to Astronomer's own cloud service, thus reducing any concerns that Astronomer may not properly respond to a GDPR request in the allotted time as prescribed by GDPR requirements.

The customer (the [data controller](https://gdpr-info.eu/art-4-gdpr/)) maintains full control over how their data is accessed by their data plane through a combination of network, authentication and authorization controls. Running a current and supported version of [Astro Runtime](/docs/runtime/upgrade-astro-runtime) ensures the latest security and bug fixes are in effect, while the [Airflow Registry](https://airflow.apache.org/registry/) provides a suite of [provider-maintained modules](https://airflow.apache.org/registry/providers/) that you can use to interact with your data in a secure and standard way.

Some basic personal information about Astro users, such as email addresses, names, and IP addresses, as well as data pipeline metadata, such as deployments metrics, scheduler logs, and lineage, is collected and processed by Astronomer (the [data processor](https://gdpr-info.eu/art-4-gdpr/)) in the control plane to provide Astro services like user management, deployment management, and observability. Customers may [exercise their data protection rights](https://www.astronomer.io/privacy#exercising-of-your-gdpr-data-protection-rights) if they have concerns about the management of this personal data.

## Does Astronomer offer a Data Processing Agreement (DPA)?

Yes, Astronomer offers a Data Processing Agreement which complies with the requirements of the current GDPR legal framework in relation to data processing. If your company requires a DPA with Astronomer to satisfy the requirements the GDPR imposes on data controllers with respect to data processors, contact [privacy@astronomer.io](mailto:privacy@astronomer.io).

Please note that if you have previously executed a DPA with Astronomer, it is likely that the DPA already contains sufficient provisions to satisfy the requirements the GDPR imposes on data controllers with respect to data processors. If you believe a new DPA is required, contact [privacy@astronomer.io](mailto:privacy@astronomer.io) with any questions or concerns.

## How does Astronomer perform transfer of personal data outside of the EEA?

The [European Commission](https://ec.europa.eu/info/index_en) (EC) issued modernized [Standard Contractual Clauses](https://ec.europa.eu/info/law/law-topic/data-protection/international-dimension-data-protection/standard-contractual-clauses-scc_en) (SCCs) on June 4, 2021, under the GDPR (Article 46) for data transfers from controllers or processors in the EU/EEA (or otherwise subject to the GDPR) to controllers or processors established outside the EU/EEA (and not subject to the GDPR).

Astronomer is subject to the new SCCs to transfer personal data to countries outside of the EEA where necessary, and has incorporated them into a standard Data Processing Agreement for the purposes of providing our Services (inclusive of support).

<Info>This page is for informational purposes only. Customers should not consider the information or recommendations presented here to constitute legal advice. Customers should engage their own legal and privacy counsel to properly evaluate their use of Astronomer services, with respect to their legal and compliance requirements and objectives.</Info>


# HIPAA compliance
Source: https://astronomer.io/docs/astro/hipaa-compliance

Learn how to achieve HIPAA compliance on Astro.

The Health Insurance Portability and Accountability Act of 1996 (HIPAA) is United States legislation that provides data privacy and security provisions for safeguarding protected health information (PHI).

HIPAA applies to organizations that are classified as [covered entities](https://www.hhs.gov/hipaa/for-professionals/covered-entities/index.html), as well as other persons or businesses, known as [business associates](https://www.hhs.gov/hipaa/for-professionals/covered-entities/index.html), that provide services with the handling, transmission, storage, or processing of PHI data. By providing the managed service Astro for data orchestration of PHI data for a HIPAA covered entity or business associate, Astronomer becomes a business associate under HIPAA.

HIPAA requires covered entities or business associates that work with other business associates to produce a contract that imposes specific safeguards on the PHI that the business associate uses or discloses to provide services to a covered entity. The contract is known as a Business Associate Agreement (BAA).

## PHI data processing on the Astro dedicated clusters

Upon signing of a HIPAA [Business Associate Agreement (BAA)](https://www.hhs.gov/hipaa/for-professionals/covered-entities/sample-business-associate-agreement-provisions/index.html), Astronomer permits the processing of PHI data in the Astro dedicated cluster. A signed BAA between Astronomer and you, the customer, helps support your HIPAA compliance program, but it is your responsibility to have required internal processes and a security program in place that align with HIPAA requirements. Compliance with HIPAA on Astro is a shared responsibility as outlined in the BAA and the model documented below.

## Shared Responsibility Model for HIPAA compliance

Astro operates on a model of shared responsibility, which means that Astronomer employees and Astronomer customers are equally responsible for ensuring platform security and compliance. This document expands on the general [shared responsibility model](/docs/astro/shared-responsibility-model) to include specific responsibilities for HIPAA compliance. Maintaining HIPAA compliance is a joint effort that is shared by the public cloud providers, Astronomer, and the customer. Each party must fulfill their individual obligations to ensure HIPAA compliance.

This document references the Astro control plane and dedicated clusters, which are core parts of the Astro Hosted deployment model:

* The control plane provides end-to-end visibility, control, and management of users, workspaces, deployments, metrics, and logs.
* The dedicated cluster is the single tenant foundation in Astro and orchestrates your data pipelines on Astro Runtime deployments.

### Astronomer obligations

* Provide a single-tenant cluster (Dedicated Astro Cluster) to ensure that PHI data processed on Astro Runtime deployments is completely network, compute, and data resources isolated.
* Provide cluster infrastructure options and configuration that enforce encryption in-transit and at rest.
* Encrypt data in transit between control plane and dedicated clusters.
* Encrypt data at rest in control plane and dedicated clusters.
* Monitor control plane and dedicated clusters for but not limited to unauthorized access, malicious activity, intrusions and threats at runtime, and unauthorized configuration changes.
* Deprovision compute and data resources when they are no longer required for task execution, so that the cloud provider can permanently remove the compute and data resources.
* Execute dedicated cluster deletion when initiated by the customer, so that the cloud provider can permanently remove the network, compute, and data resources.

### Customer obligations

* Execute a Business Associate Agreement (BAA) with your public cloud provider to process PHI on cloud infrastructure.
* [Configure an identity provider](/docs/astro/configure-idp) (IdP) for single sign-on to your Astro Organization.
* Use a [supported](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-lifecycle-schedule) (preferably latest patch) version of [Astro Runtime](/docs/runtime/runtime-image-architecture), to take advantage of the most recent security features and fixes.
* Use [supported and compatible versions](https://github.com/apache/airflow/blob/main/README.mdx#release-process-for-providers) of [Airflow providers](https://airflow.apache.org/registry/providers/), to take advantage of the most recent security features and fixes.
* Create a [secrets backend](/docs/astro/secrets-backend) to access sensitive information and secrets from your data pipelines that will be used to access PHI. If you do not have a secrets backend, you must [store your environment variables as secrets](/docs/astro/environment-variables).
* Ensure all PHI data that is orchestrated or processed by the dedicated cluster is encrypted at rest and in transit at all times using modern cryptographic protocols and ciphers, and at no point is stored or can be read in clear text. For example, when reading data from an RDS instance, transforming it in on an Astro Runtime deployment running in your dedicated cluster, and writing it out to an S3 bucket.
* Do not output PHI to scheduler and/or task logs, especially in clear text. See [View Logs](/docs/astro/view-logs) for more information.
* Do not store PHI as part of your dag image or code.
* Do not store unencrypted PHI in [XComs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html). Ensure encrypted PHI stored in XComs for task execution is purged following task execution.
* Ensure your [lineage metadata](/docs/astro/observe-openlineage) does not contain any PHI.
* Do not add, delete or modify dedicated cluster infrastructure that is provisioned and managed by Astronomer as that may be a violation of HIPAA. For example, disabling encryption of the S3 bucket (AWS), Cloud Storage (GCP), or Storage Account (Azure).

### Cloud provider responsibilities:

* Comply with the business associate obligations outlined in the BAA between Astronomer and cloud provider, and between the customer and cloud provider.
* Provide cloud infrastructure, specifically virtual machines, that support HIPAA compliance:
  * AWS: hardware-enabled encryption at rest and in transit with [EC2 Nitro instances](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#ec2-nitro-instances)
  * GCP: [Shielded GKE nodes](https://cloud.google.com/kubernetes-engine/docs/how-to/shielded-gke-nodes) leveraging built in encryption [at rest](https://cloud.google.com/docs/security/encryption/default-encryption) and [in transit](https://cloud.google.com/docs/security/encryption-in-transit) cluster features
  * Azure: AKS managed [Virtual Machine Scale Sets](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/overview) (VMSS) leveraging built in encryption [at rest](https://docs.microsoft.com/en-us/azure/security/fundamentals/encryption-overview#encryption-of-data-at-rest) and [in transit](https://docs.microsoft.com/en-us/azure/security/fundamentals/encryption-overview#encryption-of-data-in-transit) cluster features
* Permanently delete and remove data disks, databases, object storage, and encryption keys when released or deleted by Astro.

<Info>This page is for informational purposes only. Customers should not consider the information or recommendations presented here to constitute legal advice. Customers should engage their own legal and privacy counsel to properly evaluate their use of Astronomer services, with respect to their legal and compliance requirements and objectives.</Info>


# Manage Astro billing
Source: https://astronomer.io/docs/astro/manage-billing

Change your billing details and view your current spend from the Astro UI.

Astro meters and bills based on consumption of cloud resources associated with clusters, Deployments, and workers. Pricing is charged at an hourly rate, but is measured by the second. See [Pricing](https://www.astronomer.io/pricing/) for complete pricing and billing details.

You can configure payment information and check your total Astro spend from the Astro UI so that you don't go over your budget for running Airflow.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## View billing details

<Tabs>
  <Tab title="New Astro UI">
    In the Astro UI, go to **Settings** > **Billing**. The **Billing** page is organized into sections, including:

    * **Current Billing Cycle** shows your total spend for the current billing cycle.
    * **Payment** shows your payment information.
    * **Resources** shows your invoices, usage breakdown, and credits.
  </Tab>

  <Tab title="Legacy UI">
    In the Astro UI, click **Organization Settings**, then click **Billing**. The **Billing** menu includes the following tabs for tracking your payments:

    * **Overview** shows your current payment information and your total spend for the current billing cycle.
    * **Invoices** shows a breakdown of charges for current and previous billing cycles.
    * **Usage** shows your spend over time for each billable Airflow component.
  </Tab>
</Tabs>

<Info>Astro users on the Enterprise plan can view a detailed billing breakdown in their [Organization Dashboard](/docs/astro/organization-dashboard#cost-breakdown).</Info>

## Update billing details

To add or update your payment details, email [billing@astronomer.io](mailto:billing@astronomer.io).

## View total spend

<Tabs>
  <Tab title="New Astro UI">
    In the Astro UI, go to **Settings** > **Billing**. The **Current Billing Cycle** section contains high level information about your total Astro spend.
  </Tab>

  <Tab title="Legacy UI">
    In the Astro UI, click **Organization Settings**, then click **Billing**. The **Overview** page contains high level information about your total Astro spend.
  </Tab>
</Tabs>

* **Accrued charges** is the total amount you've spent on Astro resources for the current billing cycle before trial credits and discounts are applied.
* **Balance due** is the total amount you owe for the current billing cycle after trial credits and discounts are applied. If you're on a trial, this is also where you can view how many credits you have left.

## View invoices

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Billing**.
    2. Go to the **Resources** section.
    3. Click **Invoices**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**, then click **Billing**.
    2. Click **Invoices**.
  </Tab>
</Tabs>

By default, the Astro UI shows a draft invoice for your current billing cycle.

* Each **Charge** is an Astro component that you used at some point in the billing cycle. Note that your worker resources are charged separately from the Deployments they run on and each other. For example, a Deployment with two worker queues will appear as three separate charges in your invoice.
* **Quantity** represents the number of hours you ran a given Astro component.

<Info>The **Billing** page updates every hour with your total spend in the previous complete hour. For example, at 10:00 AM, the page updates with your spend from 8:00 AM - 9:00 AM on that same day. If your spending metrics don't look accurate based on recent usage, wait until the next complete hour for the page to update.</Info>

## View component usage by date

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Billing**.
    2. Go to the **Resources** section.
    3. Click **Usage**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**, then click **Billing**.
    2. Click **Usage**.
  </Tab>
</Tabs>

Each chart on this page shows your different types of usage over the last 30, 60, or 90 days for worker, compute, Deployments, and dedicated clusters. For a particular usage type, the chart shows combined usage for all instances of a given component type across your Organization. Therefore, it's possible to have more than 24 hours of usage for a given day. For example, if you have two Deployment with Medium Schedulers running for 24 hours in a day, the usage chart will show that your total usage for the day was 48 hours.

## View your credits

You can check your available credit balance and total credit consumption in the Astro UI. This includes [Astro Trial](/docs/astro/trial) credits as well as any other credits you might have accrued during your time using Astro.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Billing**.
    2. Go to the **Resources** section.
    3. Click **Commits & Credits**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**, then click **Billing**.
    2. Click **Commits and Credits**.
  </Tab>
</Tabs>

## Cancel your plan

<Tabs>
  <Tab title="Pay-as-you-go">
    To cancel your plan, email [billing@astronomer.io](mailto:billing@astronomer.io).

    <Info>If you cancel your Astro plan, all Deployments are deleted immediately but your Organization can be reactivated at any time.</Info>
  </Tab>

  <Tab title="AWS marketplace subscription">
    You can cancel your Astro subscription from the [AWS management console](https://docs.aws.amazon.com/marketplace/latest/buyerguide/cancel-subscription.html#cancel-saas-subscription).
  </Tab>

  <Tab title="Azure marketplace subscription">
    To cancel your Astro subscription, delete the Astro resource [in the Azure portal](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resources-portal#delete-resources). Make sure to delete all Deployments before deleting the Astro resource.
  </Tab>

  <Tab title="Annual contract">
    Reach out to your point of contact at Astronomer or [billing@astronomer.io](mailto:billing@astronomer.io) for further assistance.
  </Tab>
</Tabs>


# Create and manage Organization API tokens
Source: https://astronomer.io/docs/astro/organization-api-tokens

Create and manage Organization API tokens to automate key actions across all Workspaces in your Organization, like adding users and creating Deployments.

Use Organization API tokens to automate across all Workspaces in your Organization, such as creating Deployments and managing users as part of your CI/CD pipelines. For an overview of how Astro authenticates API requests, including JWT validation, token lifetimes, and rotation behavior, see [API authentication and token security](/docs/astro/api-authentication).

Organization API tokens are particularly helpful for automating:

* Creating Workspaces.
* Inviting users to an Organization or Workspace. See [Add a group of users to Astro using the Astro CLI](/docs/astro/manage-workspace-users#add-a-group-of-users-to-a-workspace-using-the-astro-cli).
* Creating and updating Deployments using a [Deployment file](/docs/astro/manage-deployments-as-code).
* Exporting audit logs.
* Gathering metadata about Deployments using the Airflow REST API.
* Completing any of the actions you can complete with a Workspace API token or Deployment API token across all Deployments in your Organization.

<Info>
  **API Token details**

  You can view the Workspaces and Deployments roles associated with an API token by clicking on any token in the Organization Settings API Token table to open its details page.
</Info>

For Deployments running Astro Runtime 3.1-12 or later, you can assign Dag-level roles to Organization, Workspace, and Deployment API tokens. See [Assign Dag roles to API tokens](/docs/astro/dag-level-access-control#assign-dag-roles-to-api-tokens).

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Create an Organization API token

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Access Management** section, click **API Tokens**.

    2. Click **+ New Organization API Token**.

    3. Configure the new Organization API token:

       * **Name**: The name for the API token.
       * **Description**: Optional. The Description for the API token.
       * **Organization Role**: The role that the API token can assume. See [User permissions](/docs/astro/user-permissions#organization-roles).
       * **Expiration**: The number of days that the API token can be used before it expires.

    4. Click **Create API token**. A confirmation screen showing the token appears.

    5. Copy the token and store it in a safe place. You will not be able to retrieve this value from Astro again.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**.

    2. Go to **Access Management** > **API Tokens**.

    3. Click **+ API Token**.

    4. Configure the new Organization API token:

       * **Name**: The name for the API token.
       * **Description**: Optional. The Description for the API token.
       * **Organization Role**: The role that the API token can assume. See [User permissions](/docs/astro/user-permissions#organization-roles).
       * **Expiration**: The number of days that the API token can be used before it expires.

    5. Click **Create API token**. A confirmation screen showing the token appears.

    6. Copy the token and store it in a safe place. You will not be able to retrieve this value from Astro again.
  </Tab>
</Tabs>

## Update or delete an Organization API token

If you delete an Organization API token, make sure that no existing automation workflows are using it. After it's deleted, an API token can't be recovered. If you unintentionally delete an API token, create a new one and update any automation workflows that used the deleted API token.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Access Management** section, click **API Tokens**.
    2. Open the **More actions** menu (⋯) next to your API token, then click **Edit Token**.
    3. Update the name, description, or Organization role of your token, then click **Update API Token**.
    4. Optional. To delete an Organization API token, click **Delete API Token**, enter `Delete`, and then click **Yes, Continue**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**.
    2. Go to **Access Management** > **API Tokens**.
    3. Open the action menu (⋯) then click **Edit Token** next to your API token.
    4. Update the name, description, or Organization role of your token, then click **Update API Token**.
    5. Optional. To delete an Organization API token, click **Delete Token**, enter `Delete`, and then click **Yes, Continue**.
  </Tab>
</Tabs>

## Rotate an Organization API token

Rotating an Organization API token lets you renew a token without needing to reconfigure its name, description, and permissions. You can also rotate a token if you lose your current token value and need it for additional workflows.

When you rotate an Organization API token, you receive a new valid token from Astro that can be used in your existing workflows. The previous token value becomes invalid and any workflows using those previous values stop working.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then in the **Access Management** section, click **API Tokens**.
    2. Open the **More actions** menu (⋯) next to your API token, then click **Rotate Token**. Type in `ROTATE` to confirm, and click **Yes, Continue**. The Astro UI rotates the token and shows the new token value.
    3. Copy the new token value and store it in a safe place. You won't be able to retrieve this value from Astro again.
    4. In any workflows using the token, replace the old token value with the new value you copied.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**.

    2. Go to **Access Management** > **API Tokens**.

    3. Click **Edit** next to your API token.

    4. Click **Rotate token**. The Astro UI rotates the token and shows the new token value.

    5. Copy the new token value and store it in a safe place. You will not be able to retrieve this value from Astro again.

    6. In any workflows using the token, replace the old token value with the new value you copied.
  </Tab>
</Tabs>

## Use an Organization API token with the Astro CLI

To use an Organization API token with Astro CLI, specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI.

For example, to automate Astro CLI commands on a Mac, run the following command to set a temporary value for the environment variable:

```sh wrap theme={null}
export ASTRO_API_TOKEN=<your-token>
```

After you set the variable, you can run `astro deployment`, `astro workspace`, and `astro organization` commands for your Workspace without authenticating yourself to Astronomer. Astronomer recommends storing `ASTRO_API_TOKEN` as a secret before using it to automate the Astro CLI for production workflows.

### Use an Organization API token for CI/CD

You can use Organization API tokens and the Astro CLI to automate various Organization, Workspace, and Deployment management actions in CI/CD.

For all use cases, you must make the following environment variable available to your CI/CD environment:

```text wrap theme={null}
ASTRO_API_TOKEN=<your-token>
```

After you set this environment variable, you can run Astro CLI commands from CI/CD pipelines without needing to manually authenticate to Astro. For more information and examples, see [Automate code deploys with CI/CD](/docs/astro/set-up-ci-cd).


# Update settings for your Astro Organization
Source: https://astronomer.io/docs/astro/organization-settings

Update high-level settings for an Astro Organization that apply to all Workspaces, Deployments, and users.

An Organization is the highest management level on Astro. In addition to configuring infrastructure for your Organization, you can update Organization settings that apply to all clusters, Workspaces, Deployments, and users within the Workspace.

This document includes instructions to configure high-level Organization settings from the Astro UI. To configure more specific Organization-level infrastructure, see:

* [Manage users in an Astro Organization](/docs/astro/manage-organization-users)
* [Set up authentication and single sign-on for Astro](/docs/astro/configure-idp)
* [Create a dedicated Astro cluster](/docs/astro/create-dedicated-cluster)
* [Create and manage Organization API tokens](/docs/astro/organization-api-tokens)

## Prerequisites

* Organization Owner permissions

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

## Update your Organization name

Your Organization name is a human-readable name that appears in the Astro UI and in the Astro CLI. Updating your Organization name is a cosmetic change that has no affect on any unique Organization information, such as your Organization ID, which persists for the lifetime of the Organization.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**.
    2. In the **Organization Name** row, click the **Edit** pencil.
    3. Enter the new Organization name.
    4. Click **Save**.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**. This opens the **General** Organization page.
    2. In the **Organization Detail** section, click **Edit Details**.
    3. Give your Organization a new **Organization Name**, then click **Update Organization**.
  </Tab>
</Tabs>

## Configure environment secrets fetching for the Astro Environment Manager

When members of your Organization create a local Airflow environment using the Astro CLI, they can pull connections configured in the [Astro Environment Manager](/docs/astro/create-and-link-connections) based on their Workspace credentials. This enables users to share connection details between Astro and their local Airflow environments and avoid creating connections twice. See [Import and export Airflow connections and variables](/docs/cli/v1.43/local-connections) for more details.

You can enable or disable this feature based on whether you want Organization members to access Astro-configured connection details from their local machines.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**.
    2. Scroll to the **Organization Policies** section.
    3. Click the **Environment Secrets Showable** toggle to enable or disable the feature.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**. This opens the **General** Organization page.
    2. In the **Organization Detail** section, click **Edit Details**.
    3. Click the **Environment Secrets Fetching** toggle to **Enabled** or **Disabled**, then click **Update Organization**.
  </Tab>
</Tabs>

## Enforce dedicated clusters for new Deployments

Organization Owners can restrict new Deployments to dedicated clusters only. When you enable **Enforce Dedicated Clusters**, Astro rejects requests to create new Deployments on standard clusters, both in the Astro UI and through the Astro API. Existing Deployments on standard clusters continue to run without changes.

This setting is available to eligible Organizations.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**.
    2. Scroll to the **Organization Policies** section.
    3. Click the **Enforce Dedicated Clusters** toggle to enable or disable it.
  </Tab>

  <Tab title="Legacy UI">
    <Steps>
      <Step title="1. Open Organization settings">
        In the Astro UI, click **Organization Settings**. This opens the **General** Organization page.
      </Step>

      <Step title="2. Edit Organization details">
        In the **Organization Detail** section, click **Edit Details**.
      </Step>

      <Step title="3. Toggle enforce dedicated clusters">
        Find the **Enforce Dedicated Clusters** option. Set the toggle to **Enabled** to require new Deployments to use a dedicated cluster, or **Disabled** to allow new Deployments on standard clusters.
      </Step>

      <Step title="4. Save changes">
        Click **Update Organization** to apply your changes.
      </Step>
    </Steps>
  </Tab>
</Tabs>

## Configure security contacts

*Security contacts* are email addresses, other than Organization Owners, that receive security notifications for your Organization.

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings**, then scroll to the **Security** section.
    2. Next to **Security Contacts**, click the **Edit** pencil.
    3. To add an address, click **+ Add Email** and enter a valid email. To remove an address, click the trash can next to it.
    4. Click **Update Security Contacts**.
  </Tab>

  <Tab title="Legacy UI">
    <Steps>
      <Step title="1. Open Organization settings">
        In the Astro UI, click **Organization Settings**. This opens the **General** Organization page.
      </Step>

      <Step title="2. Edit security contacts">
        In the **Security Contacts** section, click **Edit**.
      </Step>

      <Step title="3. Add or remove email addresses">
        Click **Add Email** to add an entry, or click **Remove** next to an existing entry. Each address must be a valid email.
      </Step>

      <Step title="4. Save changes">
        Click **Update Security Contacts**.
      </Step>
    </Steps>
  </Tab>
</Tabs>


# Global environment variables
Source: https://astronomer.io/docs/astro/platform-variables

A list of environment variables that are set globally on Astro and should not be modified.

This document is a reference for all environment variables on Astronomer with different default values than open source Apache Airflow. You can override default Runtime environment variables, but you can't override [system environment variables](#system-environment-variables).

For information on setting your own environment variables, see [Environment variables](/docs/astro/manage-env-vars).

## System environment variables

On Astro, certain environment variables have special handling. There are three types of special handling:

* *Override variables* have their value set by Astro, regardless of what you set in your Deployment.
* *Default Change* variables have values that are different from the values in open source Apache Airflow. But, if you specify a value, your value takes precedence.
* *Unsafe to Change* variables are environment variables used by Astro to function. Do not override variables listed as **Unsafe to Change**, because it can break Astro functionality.

The following table provides information about each global environment variable set by Astronomer.

<Danger>
  The Astro UI does not currently prevent you from setting the environment variables listed as **Unsafe to Change**. Attempting to set them can result in unexpected behavior that can include access problems, missing task logs, and failed tasks.

  If you need to set one of these variables for a particular use case, contact [Astronomer support](https://cloud.astronomer.io/open-support-request).
</Danger>

### Default Changed

| Environment Variable                                                             | Applicability                                                                            | Value                                                                                                                                                                                                                           |
| -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AIRFLOW__TRIGGERER__DEFAULT_CAPACITY`                                           | Both Airflow 2 and 3                                                                     | `1000`                                                                                                                                                                                                                          |
| `AIRFLOW__CORE__PARALLELISM`                                                     | Both Airflow 2 and 3                                                                     | based on worker queue settings                                                                                                                                                                                                  |
| `AIRFLOW__DAG_PROCESSOR__FILE_PARSING_SORT_MODE`                                 | Only applies if you use HA or have an extra large size Deployment. Both Airflow 2 and 3. | `random_seeded_by_host`                                                                                                                                                                                                         |
| `AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL`                                       | Airflow 3                                                                                | `30`                                                                                                                                                                                                                            |
| `AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL`                                      | Airflow 2                                                                                | `30`                                                                                                                                                                                                                            |
| `AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE`                  | Both Airflow 2 and 3                                                                     | `16`                                                                                                                                                                                                                            |
| `AIRFLOW__LOGGING__LOG_FILENAME_TEMPLATE`                                        | Both Airflow 2 and 3                                                                     | `dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number\|default(ti.try_number) }}/{{ ti.id }}.log` (for AstroExecutor) |
| `AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT`                                        | Both Airflow 2 and 3                                                                     | `600` (for `<12.5.0,11.15.0`), `300` otherwise                                                                                                                                                                                  |
| `OPENLINEAGE_DISABLED=True`                                                      | Airflow 3 Remote Deployments                                                             |                                                                                                                                                                                                                                 |
| `AIRFLOW__SCHEDULER__SCHEDULER_ZOMBIE_TASK_THRESHOLD`                            | Both Airflow 2 and 3                                                                     | `60`, unless on Astro Executor, then `120`                                                                                                                                                                                      |
| `AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT`                            | Both Airflow 2 and 3                                                                     | `60`, unless on Astro Executor, then `120`                                                                                                                                                                                      |
| `AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION`                              | Airflow 2                                                                                | `FALSE`                                                                                                                                                                                                                         |
| `AIRFLOW__KUBERNETES_EXECUTOR__WORKER_POD_PENDING_FATAL_CONTAINER_STATE_REASONS` | Only applies if you are on Runtime Version 11.15.0 or higher. Both Airflow 2 and 3.      | `CreateContainerConfigError,CreateContainerError,ImageInspectError,InvalidImageName`                                                                                                                                            |

### Override

| Environment Variable                                 | Applicability                                                                                  | Value                                                                    |
| ---------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ |
| `AIRFLOW__ASTRONOMER__CASBIN_DEPLOYMENT`             | Both Airflow 2 and 3                                                                           | Your Deployment ID                                                       |
| `ASTRO_LOGGING_ROLE_ARN`                             | Both Airflow 2 and 3                                                                           |                                                                          |
| `AIRFLOW_CONN_ASTRO_GCS_LOGGING`                     | Both Airflow 2 and 3                                                                           | `URI`                                                                    |
| `AIRFLOW_CONN_ASTRO_AZURE_LOGS`                      | Both Airflow 2 and 3                                                                           |                                                                          |
| `AIRFLOW_CONN_ASTRO_S3_LOGGING`                      | Both Airflow 2 and 3                                                                           |                                                                          |
| `AIRFLOW__WEBSERVER__STATIC_CDN`                     | Both Airflow 2 and 3                                                                           | [https://cdn.astronomer.io/airflow/](https://cdn.astronomer.io/airflow/) |
| `AIRFLOW__API__BASE_URL`                             | Airflow 3 only                                                                                 | `https://{orgid}.astronomer.run/d{deployment_id_suffix}`                 |
| `AIRFLOW__OPENSEARCH__HOST`                          | Airflow 3 only                                                                                 | `“”`                                                                     |
| `ASTRO_AGENT_CLIENT_DISALLOW_DEFAULT_SECRET_BACKEND` | Airflow 3 Astro Hosted                                                                         | `FALSE`                                                                  |
| `ASTRO_AGENT_CLIENT_DISALLOW_DEFAULT_XCOM_BACKEND`   | Airflow 3 Astro Hosted                                                                         | `FALSE`                                                                  |
| `AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST`     | Airflow 3 hosted                                                                               |                                                                          |
| `AIRFLOW__ASTRO__TASK_LOG_URL_PATTERN`               | Airflow 3 Remote                                                                               | `URI`                                                                    |
| `AIRFLOW__METRICS__STATSD_ALLOW_LIST`                | Both Airflow 2 and 3                                                                           | Variable is removed                                                      |
| `AIRFLOW__METRICS__STATSD_STATSD_CUSTOM_CLIENT_PATH` | Both Airflow 2 and 3                                                                           | Variable is removed                                                      |
| `AIRFLOW__LOGGING__LOG_FILENAME_TEMPLATE`            | Both Airflow 2 and 3                                                                           | Variable is removed                                                      |
| `AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST`     | Both Airflow 2 and 3                                                                           | Variable is removed                                                      |
| `AIRFLOW__API__PORT`                                 | Both Airflow 2 and 3                                                                           | `9091`                                                                   |
| `AIRFLOW__API_AUTH__JWT_ALGORITHM`                   | Both Airflow 2 and 3                                                                           | `EdDSA`                                                                  |
| `AIRFLOW__API_AUTH__JWT_KID`                         | Both Airflow 2 and 3                                                                           |                                                                          |
| `AIRFLOW__API_AUTH__JWT_PRIVATE_KEY_PATH`            | Both Airflow 2 and 3                                                                           | `/var/run/secrets/airflow/api-token-secret/api-token-signing-key`        |
| `AIRFLOW__API_AUTH__TRUSTED_JWKS_URL`                | Both Airflow 2 and 3                                                                           | `/var/run/secrets/airflow/api-token-secret/api-token-jwks-key`           |
| `AIRFLOW__CELERY__BROKER_URL`                        | Both Airflow 2 and 3                                                                           |                                                                          |
| `AIRFLOW__CORE__FERNET_KEY`                          | The secret key for saving connection passwords in the metadata database. Both Airflow 2 and 3. | `fernetKeySecret`                                                        |
| `AIRFLOW__CORE__LAZY_LOAD_PLUGINS`                   | Both Airflow 2 and 3                                                                           | `FALSE`                                                                  |
| `AIRFLOW__DATABASE__CHECK_MIGRATIONS`                | Both Airflow 2 and 3                                                                           | `FALSE`                                                                  |
| `AIRFLOW__DATABASE__EXTERNAL_DB_MANAGERS`            | Both Airflow 2 and 3                                                                           | `fab` in Airflow 2, `fab+astro` in Airflow 3                             |
| `AIRFLOW__KUBERNETES_EXECUTOR__NAMESPACE`            | Both Airflow 2 and 3                                                                           |                                                                          |
| `AIRFLOW__KUBERNETES__POD_TEMPLATE_FILE`             | Both Airflow 2 and 3                                                                           | reference to mounted `/usr/local/airflow/pod_template_file.yaml`         |
| `AIRFLOW__LOGGING__COLORED_CONSOLE_LOG`              | Both Airflow 2 and 3                                                                           | `FALSE`                                                                  |
| `AIRFLOW__OPERATORS__DEFAULT_QUEUE`                  | Both Airflow 2 and 3                                                                           | Worker queue defined as default, usually `"default"`                     |
| `AIRFLOW__SCHEDULER__STANDALONE_DAG_PROCESSOR`       | Both Airflow 2 and 3                                                                           | `True`/`False` depending on Scheduler Size                               |
| `AIRFLOW__WEBSERVER__SECRET_KEY`                     | Both Airflow 2 and 3                                                                           |                                                                          |
| `AIRFLOW__CELERY__RESULT_BACKEND`                    | Both Airflow 2 and 3                                                                           |                                                                          |
| `AIRFLOW__DATABASE__SQL_ALCHEMY_CONN`                | Both Airflow 2 and 3                                                                           |                                                                          |
| `AIRFLOW__EXECUTION_API__JWT_AUDIENCE`               | Both Airflow 2 and 3                                                                           |                                                                          |
| `ASTRO_ORGANIZATION_ID`                              | Both Airflow 2 and 3                                                                           | Your Astro Organization ID                                               |
| `ASTRO_WORKSPACE_ID`                                 | Both Airflow 2 and 3                                                                           | Your Astro Workspace ID                                                  |
| `ASTRO_DEPLOYMENT_ID`                                | Both Airflow 2 and 3                                                                           | Your Astro Deployment ID                                                 |
| `ASTRO_DEPLOYMENT_NAMESPACE`                         | Both Airflow 2 and 3                                                                           | Your Astro Deployment Namespace                                          |

### Unsafe to Change

| Environment Variable                                               | Description                                                                                                                                               | Value                                                      |
| ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| `OPENLINEAGE_NAMESPACE`                                            | Used internally by Astro for lineage and alerting. Do not change or overwrite.                                                                            |                                                            |
| `AIRFLOW__OPENLINEAGE__NAMESPACE`                                  | Used internally by Astro for lineage and alerting. Do not change or overwrite.                                                                            |                                                            |
| `AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_TIMEOUT`         | The amount of time that the Celery executor waits for a response from the Celery backend before throwing an error. Both Airflow 2 and 3.                  | `30`                                                       |
| `AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_CONNECT_TIMEOUT` | The amount of time that the Celery executor will attempt to connect to the Celery backend before retrying. Both Airflow 2 and 3                           | `5`                                                        |
| `AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_KEEPALIVE`       | Whether the Celery executor will check whether the connection to the Celery backend is still alive. Both Airflow 2 and 3                                  | `TRUE`                                                     |
| `AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__RETRY_ON_TIMEOUT`       | Whether the Celery executor will retry a connection to the Celery backend when the connection fails. Both Airflow 2 and 3                                 | `TRUE`                                                     |
| `AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_TARGET`                       | Routes scheduler logs to `stdout`. Both Airflow 2 and 3                                                                                                   | `stdout`                                                   |
| `AIRFLOW__LOGGING__REMOTE_LOGGING`                                 | Enables remote logging. Airflow 2 and only Airflow 3 Azure remote Deployments.                                                                            | `TRUE`                                                     |
| `AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER`                         | Location of remote logging storage. Enables remote logging. Airflow 2 and only Airflow 3 Azure remote Deployments.                                        | `baseLogFolder`                                            |
| `AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID`                             | ID of the connection that provides access to your remote logging location. Enables remote logging. Airflow 2 and only Airflow 3 Azure remote Deployments. |                                                            |
| `AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS`                           | Class name that specifies the logging configuration. Enables remote logging. Airflow 2 and only Airflow 3 Azure remote Deployments.                       | `astronomer.runtime.logging.logging_config`                |
| `AIRFLOW_CONN_ASTRO_S3_LOGGING`                                    | Connection URI for writing task logs to Astro's managed S3 bucket. Enables remote logging. Airflow 2 and only Airflow 3 Azure remote Deployments.         |                                                            |
| `AIRFLOW__LOGGING__ENCRYPT_S3_LOGS`                                | Determines whether to use server-side encryption for S3 logs. Both Airflow 2 and 3.                                                                       | `FALSE`                                                    |
| `AIRFLOW__WEBSERVER__BASE_URL`                                     | The base URL of the Airflow UI. Both Airflow 2 and 3                                                                                                      | `https://${fullIngressHostname`                            |
| `AIRFLOW__CORE__SQL_ALCHEMY_CONN`                                  | The SqlAlchemy connection string for the metadata database. Both Airflow 2 and 3.                                                                         | `dbConnSecret`.                                            |
| `AIRFLOW__WEBSERVER__UPDATE_FAB_PERMS`                             | Determines whether to update FAB permissions on webserver startup. Airflow 2 Only.                                                                        | `TRUE`                                                     |
| `AIRFLOW__WEBSERVER__ENABLE_PROXY_FIX`                             | Determines whether to enable werkzeug ProxyFix middleware for reverse proxy. Airflow 2 only.                                                              | `TRUE`                                                     |
| `AIRFLOW__API__ENABLE_PROXY_FIX`                                   | Determines whether to enable werkzeug ProxyFix middleware for reverse proxy. Airflow 3 only.                                                              | `TRUE`                                                     |
| `AIRFLOW__CORE__EXECUTOR`                                          | The executor class that Airflow uses. Astro supports the Celery and Kubernetes executor. Both Airflow 2 and 3.                                            | executor                                                   |
| `AIRFLOW_HOME`                                                     | The home directory for an Astro project. Both Airflow 2 and 3.                                                                                            | `usr/local/airflow`                                        |
| `AIRFLOW__KUBERNETES__NAMESPACE`                                   | The Kubernetes namespace where Airflow workers are created. Both Airflow 2 and 3.                                                                         | `namespace`                                                |
| `AIRFLOW__CORE__HOSTNAME_CALLABLE`                                 | Path to a callable, which resolves to the hostname. Both Airflow 2 and 3.                                                                                 | `airflow.utils.net.get_host_ip_address`                    |
| `AIRFLOW__SCHEDULER__STATSD_ON`                                    | Determines whether StatsD is on. Both Airflow 2 and 3.                                                                                                    | `TRUE`                                                     |
| `AIRFLOW__SCHEDULER__STATSD_HOST`                                  | The hostname for StatsD. Both Airflow 2 and 3.                                                                                                            | `statsd.Hostname`                                          |
| `AIRFLOW__SCHEDULER__STATSD_PORT`                                  | The port for StatsD. Both Airflow 2 and 3.                                                                                                                | `<statsd-port>`                                            |
| `AIRFLOW__METRICS__STATSD_ON`                                      | Determines whether metrics are sent to StatsD. Both Airflow 2 and 3.                                                                                      | `TRUE`                                                     |
| `AIRFLOW__METRICS__STATSD_HOST`                                    | The hostname for sending metrics to StatsD. Both Airflow 2 and 3.                                                                                         | `statsd.Hostname`                                          |
| `AIRFLOW__METRICS__STATSD_PORT`                                    | The port for sending metrics to StatsD. Both Airflow 2 and 3.                                                                                             | `<statsd-metrics-port>`                                    |
| `AIRFLOW__METRICS__STATSD_PREFIX`                                  | The prefix for sending the metrics to StatsD. Both Airflow 2 and 3.                                                                                       | `airflow`                                                  |
| `AIRFLOW__WEBSERVER__COOKIE_SECURE`                                | Sets a secure flag on server cookies. Airflow 2 only.                                                                                                     | `TRUE`                                                     |
| `AIRFLOW__WEBSERVER__INSTANCE_NAME`                                | Shows the name of your Deployment in the Home view of the Airflow UI. Airflow 2.                                                                          | `<Deployment-Name>`                                        |
| `AIRFLOW__API__INSTANCE_NAME`                                      | Shows the name of your Deployment in the Home view of the Airflow UI. Airflow 3.                                                                          | `<Deployment-Name>`                                        |
| `AIRFLOW__CELERY__WORKER_CONCURRENCY`                              | Determines how many tasks each Celery worker can run at any given time and is the basis of worker auto-scaling logic                                      | `<Max-Tasks-Per-Worker>`                                   |
| `AIRFLOW__WEBSERVER__EXPOSE_CONFIG`                                | Exposes the Configuration tab of the Airflow UI and hides sensitive values. Airflow 2                                                                     | `NON-SENSITIVE-ONLY`                                       |
| `AIRFLOW__API__EXPOSE_CONFIG`                                      | Exposes the Configuration tab of the Airflow UI and hides sensitive values. Airflow 3.                                                                    | `NON-SENSITIVE-ONLY`                                       |
| `AIRFLOW__USAGE_DATA_COLLECTION__ENABLED`                          | Disables the Airflow usage data collection & reporting to Scarf. Both Airflow 2 and 3.                                                                    | `FALSE`                                                    |
| `AWS_DEFAULT_REGION`                                               | (AWS clusters only) The region where your cluster is located. Both Airflow 2 and 3.                                                                       | The region where you configured your cluster.              |
| `AWS_SECRET_ACCESS_KEY`                                            | The key secret for accessing Astro's managed S3 bucket¹. Both Airflow 2 and 3                                                                             |                                                            |
| `INSTANCE_TYPE`                                                    | Provides the instance size of the node the dag is scheduled on. Both Airflow 2 and 3                                                                      | `(v1:metadata.labels['beta.kubernetes.io/instance-type'])` |
| `OPENLINEAGE_URL`                                                  | The URL for your Astro lineage backend. The destination for lineage metadata sent from external systems to the OpenLineage API. Both Airflow 2 and 3      |                                                            |
| `OPENLINEAGE_API_KEY`                                              | Your OpenLineage API key. Both Airflow 2 and 3                                                                                                            |                                                            |


# Enable Private Network Egress
Source: https://astronomer.io/docs/astro/private-network-egress

Enable Private Network Egress on dedicated AWS clusters to require private connections from Astro.

<Note> This is feature is only available if you are on the **Team** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

<Note>
  **Preview**

  This feature is in [Preview](/docs/astro/feature-previews).
</Note>

<Info>Private Network Egress is currently only available on dedicated AWS clusters.</Info>

Astro supports Private Network Egress Mode for dedicated AWS clusters to ensure security and compliance, and to provide a data loss protection architecture to secure against unauthorized data transfer.

Enabling Private Network Egress Mode disables public internet connectivity from your Astro Deployments and makes [private networking connections](/docs/astro/networking-overview#network-connection-recommendations) required to connect to external data services and external endpoints for Metrics Export. With Private Network Egress enabled, exports to public observability platforms like Grafana Cloud will be impacted.

<Info>An icon on Deployments and Deployment details pages indicate when a Deployment is on a cluster with Private Network Egress enabled.</Info>

## Enable or disable Private Network Egress Mode for a cluster

Private Network Egress Mode can be enabled for new and existing dedicated AWS clusters.

<Tip>
  **A new Astro UI is here**

  Astronomer has redesigned the Astro UI. Try the new experience and switch your instructions using the **New Astro UI** and **Legacy UI** tabs on this page. Your selection is remembered across the docs.
</Tip>

<Tabs>
  <Tab title="New Astro UI">
    1. In the Astro UI, go to **Settings** > **Clusters**.
    2. Configure the egress mode:

    * For an existing dedicated cluster, click the cluster you want to edit. Then, enable **Private Network Egress Mode** in the **Cluster Details**.
    * For a [new dedicated cluster](/docs/astro/create-dedicated-cluster), click **+ New Cluster** and then enable **Private Network Egress Mode** in the **Network Egress Management** section during the initial configuration of the cluster.
  </Tab>

  <Tab title="Legacy UI">
    1. In the Astro UI, click **Organization Settings**, then click **Clusters**.
    2. Configure the egress mode:

    * For an existing dedicated cluster, click the cluster you want to edit. Then, enable **Private Network Egress Mode** in the **Cluster Details**.
    * For a [new dedicated cluster](/docs/astro/create-dedicated-cluster), click **Add cluster** and then enable **Private Network Egress Mode** in the **Network Egress Management** section during the initial configuration of the cluster.
  </Tab>
</Tabs>


# Astro release notes
Source: https://astronomer.io/docs/astro/release-notes

Astro release notes covering new features, bug fixes, and version updates for Astro.

<Tip>[Subscribe to Astro release notes](/docs/astro/release-notes-subscribe) to receive updates via RSS, email, or Slack.</Tip>

Astronomer is committed to continuous delivery of both features and bug fixes to Astro. To keep your team up to date on what's new, this document will provide a regular summary of all changes released to Astro.

**Latest Astro CLI Version**: 1.45.0 ([Release notes](/docs/cli/v1.45/release-notes))

**Latest Astro Runtime Version**: 3.3-2 ([Release notes](/docs/runtime/runtime-release-notes))

**Latest Remote Execution Agent Version**: 1.8.4 ([Release notes](/docs/astro/agent-release-notes))

<Update label="August 20, 2026">
  ### Bug fixes

  * Fixed an issue where every Airflow variable migrated from a Deployment's metadata database into the Environment Manager was masked as a secret. A migrated variable is now masked only when its name matches one of Airflow's sensitive-name patterns, such as a name that contains `password`, `secret`, `token`, or `api_key`, which matches Airflow's own masking behavior.
  * Fixed an issue where you couldn't create Redshift, Snowflake, or other connection types that have required **extra** fields, because the **Create Connection** button stayed disabled even after you filled in those fields. The button now enables when all required fields, including **extra** fields, have values.
</Update>

<Update label="August 13, 2026">
  ### Billing API added to the Astro v1 Labs API

  <Info>
    **Labs**

    This feature is in [Labs](https://www.astronomer.io/docs/astro/feature-previews).
  </Info>

  Enterprise+ Organizations can now query Astro cost and usage data programmatically through the new [**List daily usage**](https://www.astronomer.io/docs/astro/api/v-1-labs/billing/list-daily-usage) endpoint on the [Astro v1 Labs API](https://www.astronomer.io/docs/astro/api/v-1-labs/labs-overview), which returns per-Deployment resource cost data with a FOCUS-formatted response option.

  It syncs from the same cost breakdown export that powers the Organization dashboard, so you can pull the data on a schedule instead of exporting it by hand. See [Attribute Astro spend across teams](https://www.astronomer.io/docs/astro/best-practices/internal-chargeback) to build a chargeback report from this data.
</Update>

<Update label="July 29, 2026">
  ### Cross-region disaster recovery for GCP dedicated clusters is generally available

  Self-service cross-region disaster recovery (DR) for GCP dedicated clusters is now generally available. You can configure a primary and a secondary dedicated cluster in different GCP regions, and fail over with minimal downtime and data loss. After failover, Astro automatically enables synchronization in the reverse direction, so you can fail back once the primary region recovers.

  See [Disaster recovery](/docs/astro/disaster-recovery).

  ### Task-level utilization metrics

  Astro now shows CPU and memory utilization at the Dag, task, and task instance level in a new **Resource Metrics** tab in the Airflow UI, with task-to-worker mapping that ties resource use back to the Dag, task, task run, and time window. Use it to identify which tasks consume the most resources, which are likely causing out-of-memory errors, and which Dags are driving cost.

  This feature is available on Astro executor Hosted Runtime 3.1+ Deployments, and on Remote Execution 3.1+ Deployments using Remote Execution Agents 1.5.0+ and Helm chart 2.0.0+.

  See [View task-level utilization metrics](/docs/astro/task-level-metrics).
</Update>

<Update label="July 15, 2026">
  ### Try the new Astro experience

  <Info>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Info>

  Astro's navigation now gives you a single view across your entire Organization instead of a single Workspace. Deployments, Dags, and Astro IDE projects all appear across Workspaces in one view, with Workspace context on every row. Filter to a specific Workspace when you need to, or work from the full picture across your Organization. Workspace, Organization, and personal settings now share a single settings layout.

  The homepage also surfaces:

  * **Recently Failed Dags**: An Organization-wide table of failing Dags, sorted by failure rate, with time-range filters. Each row shows the Workspace and Deployment, run history, and two actions: **Investigate with Otto** and **Open in Airflow**.
  * **Jump Back In**: A strip of recently visited Deployments, Dags, data products, and Astro IDE projects.
  * **Favorite Deployments**: An Organization-wide view of Deployment health with **All**, **Healthy**, and **Hibernating** status filters. Each card shows status, cluster and repository context, and four metrics: Dag Runs, Tasks, Worker CPU, and Worker Memory.

  You can favorite Dags (Airflow 3.1+) and Deployments to pin them to the top of these views and filter to only your favorites.

  Enable the new experience from the banner in Astro. To switch back, go to **Your Profile**, click **Edit** next to **Astronomer Labs**, and turn off the **Astro Next** toggle. Switching between experiences doesn't change your pipelines or configuration.
</Update>

<Update label="July 9, 2026">
  ### Test against an existing Deployment in the Astro IDE

  <Info>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Info>

  You can now test code in the [Astro IDE](/docs/astro/ide-overview) against an existing Astro Deployment in your Workspace, instead of only an ephemeral test Deployment. Testing against an existing Deployment reuses its configuration, connections, and resources, and skips provisioning a new environment.

  When you attach a session to an existing Deployment, the Astro IDE deploys your session's image and Dags to that Deployment, overwriting the image and Dags currently running there. The Deployment keeps its own connections, Airflow variables, environment variables, and resource configuration. Testing against an existing Deployment requires the `workspace.deployments.create` permission.

  See [Test against an existing Deployment](/docs/astro/ide-test-run#test-against-an-existing-deployment).
</Update>

<Update label="July 2, 2026">
  ### New Deployment analytics experience

  <Info>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Info>

  Astro now offers a new version of the Deployment analytics experience with expanded Airflow visibility, including API server and webserver metrics, Dag processor coverage, Kubernetes executor and `KubernetesPodOperator` coverage, parse time, and richer per-status and per-component breakdowns. The new experience also includes improved charting controls, multi-pool visibility, and analytics charts on the Deployments list.

  During the preview period, you can switch between the new and previous analytics experience from the **Analytics** page.

  See [View metrics for Astro Deployments](/docs/astro/deployment-metrics).
</Update>

<Update label="July 1, 2026">
  ### Additional improvements

  * Updated Astro infrastructure to improve reliability and performance.
</Update>

<Update label="June 24, 2026">
  ### Enable Sub-Second Pipelines

  <Info>
    **Labs**

    This feature is in [Labs](/docs/astro/feature-previews).
  </Info>

  You can now enable [Sub-Second Pipelines](/docs/astro/sub-second-pipelines) on Airflow 3.2+ Deployments that use the [Astro executor](/docs/astro/astro-executor). Sub-Second Pipelines pair the Astro executor with a dedicated event-driven Event Scheduler that picks up and dispatches API-triggered Dag runs in under a second and sustains approximately 1,000 Dag runs per minute, where the Celery executor queues and falls behind. Use them for high-throughput triggered workloads, on-demand inference, programmatic workflow invocation, and API-driven reverse ETL.

  Enable the **Sub-Second Pipelines** toggle on the Deployment, create a worker queue with the **Sub-Second** toggle on, and route your Dag's tasks to that queue. Sub-Second Pipelines support API-triggered Dag runs only; runs triggered by cron schedules, timetables, asset and data-aware scheduling, or message queues continue to use the standard scheduler. You can confirm a run took the sub-second path from the Dag's `sub_second` tag, the **Sub-Second Metrics** tab in the Airflow UI, the Event Scheduler logs, or the SLA Metrics API.

  See [Enable Sub-Second Pipelines](/docs/astro/sub-second-pipelines).
</Update>

<Update label="June 23, 2026">
  ### Review code with Otto

  <Info>
    **Labs**

    This feature is in [Labs](/docs/astro/feature-previews).
  </Info>

  You can now use [Otto](/docs/astro/otto-overview) to review your Airflow code automatically when you open a pull request or merge request. The Otto review action posts a sticky summary comment, adds inline comments where it finds issues, and offers commit suggestions where a concrete fix is available. Each review returns a structured verdict of `approve`, `comment`, or `request_changes`, with line-anchored findings rated `high`, `medium`, or `low` severity.

  Otto reviews changes against Astronomer compatibility knowledge, Airflow community standards, and your team's conventions captured in Otto Memory, so it catches issues that generic code review misses. The action supports GitHub through a Marketplace action and GitLab through a CI/CD template and Docker image. You can also run a review on demand from the [Astro CLI](/docs/cli/v1.43/astro-otto) with `astro otto --persona reviewer`.

  See [Review code with Otto](/docs/astro/otto-code-review).
</Update>

<Update label="June 17, 2026">
  ### Additional improvements

  * When you add the first IP address range to your organization's IP access list, the Astro API now rejects the request with a `400 Bad Request` error if your current IP address falls outside the submitted ranges, which prevents you from accidentally locking yourself out. The same protection applies when you add multiple ranges at once, and users with IP access list bypass permission can still add the first range even when their IP falls outside it. See [Set up IP Access List](/docs/astro/ip-access-list).

  ### Bug fixes

  * Fixed the SMTP connection form in the Astro UI so its fields map to the correct Airflow SMTP provider parameter keys. The SSL and TLS fields are now labeled **Disable SSL** and **Disable TLS** and map to `disable_ssl` and `disable_tls`, where setting a field to `true` disables that protocol and `false` allows it. The sender email field now maps to `from_email` instead of `mail_from`.
</Update>

<Update label="June 11, 2026">
  ### Investigate Dag failures with Otto

  <Info>
    **Labs**

    This feature is in [Labs](/docs/astro/feature-previews).
  </Info>

  You can now use [Otto](/docs/astro/otto-overview) to investigate Dag failures on Astro from the Astro UI, Astro alerts, or the Astro API. Each investigation produces a structured diagnosis with a root cause type, severity, suggested fix, and a checklist of Dag- and task-level checks, drawing on Airflow, Astro, and Astro Observe context. You can also ask Otto to investigate a Dag failure interactively in the [Astro CLI](/docs/cli/v1.43/astro-otto).

  To investigate critical Dag failures automatically, pair a Dag failure alert's **Dag Trigger** notification channel with a Dag that calls the [investigation API](https://www.astronomer.io/docs/astro/api/v-1-labs/observability/start-a-dag-failure-diagnosis-run), then route the diagnosis to Slack, a pull request, or an incident management system. You can also tailor investigations with Otto investigation guidance set at the Workspace or Deployment level.

  See [Investigate with Otto](/docs/astro/otto-investigate).

  ### Dag-level access control is generally available

  Dag-level access control is now generally available. You can assign roles scoped to individual Dags within a Deployment to enforce least-privilege security and enable multiple teams to collaborate in a single Deployment without exposing Dags across team boundaries.

  Dag roles can be bound to Dags by **Dag tag** or **Dag ID** and assigned to users, Teams, and API tokens. Astro provides two default Dag roles, **Dag Viewer** and **Dag Author**, and you can create custom Dag roles with granular permissions.

  This feature requires Astro Runtime 3.1-12+ and an Enterprise plan. See [Dag-level access control](/docs/astro/dag-level-access-control).
</Update>

<Update label="June 2, 2026">
  ### Additional improvements

  * The **Basic** section of Deployment details in the Astro UI now includes a **Deployment Namespace** row with copyable text, so you can retrieve a Deployment's namespace without using the Astro CLI or API.
</Update>

<Update label="May 28, 2026">
  ### API server horizontal autoscaling

  <Info>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Info>

  You can now enable horizontal autoscaling for the Airflow API server on Airflow 3 Deployments. Autoscaling scales between a fixed minimum of two replicas and a configurable maximum from `2` to `10`. Autoscaling works on all Airflow 3 Deployments, and supports the Astro, Celery, and Kubernetes executors.

  See [Configure API server autoscaling](/docs/astro/api-server-autoscaling).

  ### Project-scoped connections and variables in the Astro IDE

  <Info>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Info>

  You can now link Workspace-level connections, Airflow variables, and environment variables to specific [Astro IDE](/docs/astro/ide-overview) projects. Project-linked objects only affect that project's ephemeral test Deployments, without affecting Deployments that aren't started from the Astro IDE. You can also set value overrides per project, or auto-link a Workspace object to every Astro IDE project in a Workspace.

  See [Make connections, Airflow variables, and environment variables available](/docs/astro/ide-test-run#make-connections-airflow-variables-and-environment-variables-available).

  ### Migrate Airflow connections and variables to the Environment Manager

  <Info>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Info>

  You can now bulk migrate connections and Airflow variables from a Deployment's Airflow metadata database into the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager). Astro creates a Workspace environment object linked to the source Deployment, then removes the source object from the metadata database, so you can manage and reuse the object centrally without losing access on the source Deployment.

  See [Migrate Airflow connections and variables from a Deployment](/docs/astro/migrate-metadata-db-to-environment-manager#migrate-airflow-connections-and-variables-from-a-deployment).

  ### Promote a Deployment environment object to the Workspace

  <Info>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Info>

  You can now promote a single Deployment-scoped connection, Airflow variable, environment variable, or metrics export to the Workspace level in the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager). Astro keeps the source Deployment linked, so it continues to use the same values, and you can reuse the object on other Deployments or Astro IDE projects.

  See [Promote a Deployment environment object to the Workspace](/docs/astro/migrate-metadata-db-to-environment-manager#promote-a-deployment-environment-object-to-the-workspace).

  ### Additional improvements

  * When you select one or more Deployments on the Astro UI **DAGs** page, the tag filter dropdown now shows only tags from those Deployments.
  * The workspace filter on the Astro Observe homepage now displays your current Workspace by default, matching the Workspace context the page is already using to query data.
  * The maximum threshold for the **DAG Duration** Astro Alert is now 7 days (604,800 seconds), up from 1 day.
  * You can now designate security contact email addresses for your organization, in addition to Organization Owners, to receive security notifications. See [Configure security contacts](/docs/astro/organization-settings#configure-security-contacts).

  ### Bug fixes

  * Fixed an issue where users with the **Organization Observe Admin** or **Organization Observe Member** role received a 403 error when accessing the Airflow `/dags` page on Astro Runtime 3.2-3 or later. These roles now include read access to Dag runs, Human-in-the-Loop details, and task instances.
  * Fixed an issue on slow or large Workspaces where the Astro Observe **Recently Failed DAGs** table could show stale paginated results with mismatched page contents and totals. The table now resets to the first page in that scenario. Failed run counts also now use Dag run end time when available, so they better match the selected time range.
</Update>

<Update label="May 14, 2026">
  ### Additional improvements

  * Organization, Workspace, and Deployment API token tables and details pages in the Astro UI now show a **Last Rotated** column and row, with the UTC timestamp of the most recent rotation. Tokens that have never been rotated display `Never`.
</Update>

<Update label="May 6, 2026">
  ### Additional improvements

  * Disaster recovery configuration and actions have moved from the cluster **Details** page to a dedicated **Disaster Recovery** tab on eligible cluster detail pages. Use this tab to view DR status, initiate failover or failback, and enable or disable DR. See [Disaster recovery](/docs/astro/disaster-recovery).
  * The lineage sidebar now includes an **Alerts** tab when you select an Airflow Dag or task asset. From this tab, you can view existing alerts for the asset, create Dag or task failure alerts pre-scoped to the selected context, and edit or delete alerts based on your existing permissions.
  * The Astro UI color mode toggle now persists your dark or light mode preference across page refreshes.

  ### Bug fixes

  * Fixed an issue where AWS disaster recovery cluster creation through the Astro API incorrectly used the primary cluster's `SecondaryVpcCidr` for the disaster recovery cluster secondary VPC CIDR. The API now correctly applies the `DRSecondaryVpcCidr` request value.
</Update>

<Update label="April 30, 2026">
  ### Introducing Otto, Astronomer's data engineering agent

  <Info>
    **Labs**

    This feature is in [Labs](/docs/astro/feature-previews).
  </Info>

  [Otto](/docs/astro/otto-overview) is Astronomer's data engineering agent, purpose built for Airflow and designed to get smarter every session. Use Otto to build pipelines, debug failures, investigate production incidents, and manage Airflow upgrades, grounded in Astronomer's compatibility knowledge base and your team's accumulated memory.

  Key capabilities include:

  * **Exploration**: Query your data warehouse, trace lineage, and profile tables without leaving the agent session.
  * **Dag authoring**: Describe what you need in natural language. Otto writes the Dag, configures connections, triggers a run, and iterates until it works. When something breaks during development, Otto pulls task logs, reads Dag source, inspects variables and connections, and traces through the failure to propose a fix.
  * **Investigation**: When a production Dag fails, ask Otto to pull the logs, analyze the failure, and propose a fix. Otto reads production task logs, run history, connections, and Dag source, then works through the diagnosis interactively with you.
  * **Airflow upgrades**: Otto analyzes your Dag fleet against Astronomer's compatibility knowledge base, identifies what breaks, proposes specific code changes, and produces a prioritized plan. See [Upgrade Airflow with Otto](/docs/astro/otto-upgrades).

  Access Otto from your terminal with [`astro otto`](/docs/cli/v1.43/astro-otto), or in Astro through the [Astro IDE](/docs/astro/ide-overview).

  See [Get started with Otto](/docs/astro/get-started-otto).

  ### Enforce dedicated clusters for new Deployments

  Organization Owners can now restrict new Deployments to dedicated clusters only. When you enable **Enforce Dedicated Clusters**, Astro rejects requests to create new Deployments on standard clusters, both in the Astro UI and through the Astro API. Existing Deployments on standard clusters continue to run without changes.

  See [Enforce dedicated clusters for new Deployments](/docs/astro/organization-settings#enforce-dedicated-clusters-for-new-deployments).
</Update>

<Update label="April 21, 2026">
  ### Cross-region disaster recovery is generally available

  Self-service cross-region disaster recovery (DR) for AWS dedicated clusters is now generally available. You can configure a primary and a secondary dedicated cluster in different AWS regions, and fail over with minimal downtime and data loss. After failover, Astro automatically enables synchronization in the reverse direction, so you can fail back once the primary region recovers.

  See [Disaster recovery](/docs/astro/disaster-recovery).

  ### AWS VPC peering is generally available

  Self-service VPC peering configuration for Astro dedicated clusters on AWS is now generally available. You can create VPC peering connections between an external AWS VPC and an Astro dedicated cluster directly from the Astro UI, without contacting Astronomer support.

  See [AWS Networking: VPC Peering](/docs/astro/connect-aws-vpc-peering).

  ### Additional improvements

  * You can now create Azure dedicated clusters in the `southafricanorth` region. See [Astro Hosted resource reference](/docs/astro/resource-reference-hosted).
  * The Dag access management page now shows an error when assigning or editing a Dag role for a user, team, or API token fails, replacing the previous silent-failure behavior. See [Dag-level access control](/docs/astro/dag-level-access-control).
  * The Astro UI has an updated visual style to align with the Astronomer brand.

  ### Bug fixes

  * Fixed an issue where configuring custom workload identity for Remote Execution Deployments on AWS and GCP could fail when workload identity options did not provide derived project or account data. The identity configuration flow now uses the cluster cloud provider account directly.
</Update>

<Update label="April 15, 2026">
  ### Blueprint in the Astro IDE

  <Info>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Info>

  Blueprint is now available in the [Astro IDE](/docs/astro/ide-overview). Blueprint is a template-based Dag authoring system that lets platform teams define reusable workflow templates in Python, while analysts, analytics engineers, and data scientists assemble those templates into production Dags through a visual builder and form-driven configuration — without writing Airflow code.

  Key capabilities include:

  * **No-code Dag authoring**: Drag blueprints onto a visual canvas, configure them through forms, and connect them to define dependencies.
  * **Platform-defined templates**: Platform teams write blueprints using the open-source [airflow-blueprint](https://github.com/astronomer/blueprint) library, encoding best practices, operational defaults, and guardrails.
  * **Dag YAML generation**: Blueprint produces plain YAML configuration files that live in your Git repository alongside your other project files.
  * **Built-in testing**: Start an ephemeral test Deployment directly from the IDE to validate individual tasks or full Dags before committing.
  * **Blueprint versioning**: Pin Dags to specific blueprint versions so existing workflows continue to work when a blueprint is updated.

  See [Blueprint](/docs/astro/ide-blueprint).

  ### Additional improvements

  * Clicking a failed Dag row on the Observe home page now navigates directly to the corresponding Asset Catalog details page. See [Observe](/docs/astro/astro-observe).
  * You can now copy an alert ID directly from the actions menu in the Alerts list. See [Alerts](/docs/astro/alerts).
  * The Dag API tokens table in Dag access management now supports search, making it faster to locate specific tokens. See [Dag-level access control](/docs/astro/dag-level-access-control).
  * The Ask Astro web application has been removed from Astro.
</Update>

<Update label="April 8, 2026">
  ### Additional improvements

  * The universal metrics exporter now scrapes metrics from triggerer, webserver, and apiserver components in addition to base, worker, scheduler, and Dag processor containers. These metrics are available automatically without additional configuration. See [Export metrics](/docs/astro/export-metrics).
  * Observe event timelines now include a **Run ID** column and display indented task and dataset events under their parent Dag events, making it easier to scan and correlate related timeline activity.
  * Clicking a Dag start, success, or failure event in the Observe timeline now opens the corresponding Airflow run page directly in a new tab.

  ### Bug fixes

  * Fixed an issue where updating environment variables, changing the default Pod size, or editing executor settings on non-development Deployments returned the error `Hibernation spec is not allowed for non-development deployments`. This was caused by an empty `hibernationSpec` object stored in the Deployment's scaling configuration data.
</Update>

<Update label="April 1, 2026">
  ### API token access management pages

  You can now click an API token's row in the Astro UI to open a dedicated access management page for Workspace and Deployment API tokens. From these pages, you can view token details and manage applicable roles, including Deployment roles and Dag roles, without navigating away from the token context.

  See [Workspace API tokens](/docs/astro/workspace-api-tokens) and [Deployment API tokens](/docs/astro/deployment-api-tokens).

  ### Self-service user profile updates

  You can now update your user profile directly from the Astro UI. Users managed by an Identity Provider (IdP) can update profile preferences but cannot edit name fields, which are managed by the IdP.

  ### Self-service disable Disaster Recovery

  You can now disable Disaster Recovery (DR) for eligible clusters directly from the cluster actions menu in the Astro UI.

  See [Set up disaster recovery](/docs/astro/disaster-recovery-setup#disable-dr).

  ### Custom GCP workload identity at Deployment creation

  You can now specify a custom workload identity for eligible GCP Deployments at creation time, instead of configuring it after Deployment creation.

  See [Authorize deployments to cloud resources](/docs/astro/authorize-deployments-to-your-cloud#dedicated-clusters-only-share-or-reuse-a-managed-identity-using-a-wildcard).

  ### Additional improvements

  * Cluster details in the Astro UI now display the **GCP Project Number** for GCP clusters. You can copy this value directly from the cluster details page.

  ### Bug fixes

  * Fixed a 500 error that occurred when Git deployments to Astro used a trailing slash in the Astro project path.
  * Fixed an issue where dbt bundle deletion failed when the object version ID was missing from cloud storage metadata.
  * Fixed an issue where Airflow variable values and value overrides in the Astro UI required a non-empty value. You can now create or update Airflow variables with empty values.
  * Fixed missing Airflow 3 metric exports for scheduler heartbeat, triggerer heartbeat, task heartbeat kill, and asset metrics. These metrics are now available in telemetry output.
  * Fixed an issue where clicking **Open Airflow** in the Astro UI also navigated the current page to Deployment details. The button now only opens Airflow in a new tab.
  * Fixed an issue where the alert edit page in the Astro UI showed stale values after saving changes. The page now reflects updates immediately without a manual refresh.
</Update>

<Update label="March 18, 2026">
  ### Async sessions and session history in the Astro IDE

  <Info>
    **Preview**

    The Astro IDE is in [Preview](/docs/astro/feature-previews).
  </Info>

  Sessions in the [Astro IDE](/docs/astro/ide-overview) are no longer tied to an open browser tab. You can submit a prompt, close the browser, and return later to see the completed response. The AI agent continues processing in the background.

  Key changes include:

  * **Async processing**: After submitting a prompt, you can navigate away and return later to see the completed response.
  * **Session history**: The **Sessions** tab shows a chronological list of your past sessions. Click any session to resume it with full context.
  * **Session persistence**: Conversation data is stored in cloud object storage and can be fully restored at any time.
  * **Private sessions**: Sessions are private to the user who created them, while projects remain shared across the Workspace.

  ### Underlying Data tab and updated cost categories in Organization Dashboards

  Organization Dashboards now include an **Underlying Data** tab with raw data tables in a standardized financial format for Deployment activity, Workspace activity, and daily cost breakdowns. You can filter, download, email, and schedule exports directly from these tables. The **Cost Breakdown** dashboard also now includes additional cost categories and a new **Detailed Costs** view with granular breakdowns by Deployment, Compute Types, Worker Queues, or Clusters.

  See [Export and schedule Organization Dashboard data](/docs/astro/org-dash-exports) and [Organization Dashboard](/docs/astro/organization-dashboard).

  ### Smaller VPC subnet support for AWS clusters

  You can now create dedicated clusters on AWS with a VPC subnet range as small as /22, down from the previous minimum of /21. This gives you more flexibility to configure cluster networking with smaller address spaces when a full /21 block isn't available.

  See [Create a dedicated Astro cluster](/docs/astro/create-dedicated-cluster#setup).

  ### GCP europe-west3 (Frankfurt) region support for standard clusters

  You can now create standard deployments in the GCP `europe-west3` (Frankfurt, Germany) region.
</Update>

<Update label="March 11, 2026">
  ### Cross-region disaster recovery for AWS dedicated clusters now in Preview

  <Info>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Info>

  You can now configure cross-region disaster recovery (DR) for AWS dedicated clusters. DR lets you pair a primary and secondary cluster in different AWS regions so you can fail over with minimal downtime and data loss if a full region outage occurs.

  Key capabilities include:

  * Self-service failover and failback from the Astro UI.
  * Automatic replication of Deployments, Dag run history, task instance metadata, environment variables, connections, Airflow variables, and task logs to the secondary cluster.
  * Recovery time objective (RTO) of less than one hour and recovery point objective (RPO) of less than 15 minutes with the optional Task Logs Replication SLA.
  * All hostnames, including Airflow UI, Airflow API, and Remote Execution API URLs, are automatically updated to point to the active cluster after failover.

  Cross-region DR requires the Enterprise Business Critical tier. You can enable DR when creating a new dedicated cluster or submit a support request to enable it on an existing cluster. See [Disaster recovery](/docs/astro/disaster-recovery).

  ### Improved authentication resilience

  Astro now runs authentication services directly in each data plane, removing the dependency on a single-region control plane component for authenticating requests. This reduces the affected area of control plane incidents, so your Deployments remain accessible even when parts of the control plane are unavailable.

  Deployment URLs now route directly through the data plane. Old URLs continue to work and automatically redirect for 6-12 months. If you have strict network allowlists or custom networking configurations, Astronomer will contact you with guidance on any required updates.

  ### Direct access Deployment API tokens

  You can now create direct access Deployment API tokens that authenticate requests directly at the Deployment level without depending on the control plane. During a control plane outage, Dags and automations that use direct access tokens continue to access the Airflow API without interruption.

  Direct access tokens are available to Organization Owners through the Astro UI when creating a Deployment API token. The role assigned to a direct access token cannot be changed after creation. See [Deployment API tokens](/docs/astro/deployment-api-tokens#direct-access-token-api-tokens).
</Update>

<Update label="February 25, 2026">
  ### Additional improvements

  * Updated Astro infrastructure to improve reliability and performance.
</Update>

<Update label="February 19, 2026">
  ### Dag-level access control now in Labs

  <Note>
    **Labs**

    This feature is in [Labs](/docs/astro/feature-previews). Reach out to your account team to enable this feature.
  </Note>

  You can now assign roles scoped to individual Dags within a Deployment. Dag-level access control lets you enforce least-privilege security and enable multiple teams to collaborate in a single Deployment without exposing Dags across team boundaries.

  Dag roles can be bound to Dags by **DAG Tag** or **DAG ID** and assigned to users, Teams, and Organization API tokens. Astro provides two default Dag roles, **Dag Viewer** and **Dag Author**, and you can create custom Dag roles with granular permissions.

  This feature requires Astro Runtime 3.1-12+ and an Enterprise plan. See [Dag-level access control](/docs/astro/dag-level-access-control).

  ### Additional improvements

  * Added support for AWS PrivateLink and Azure Private Link to enable private connectivity between Astro clusters and customer-managed Remote Execution Agents.
</Update>

<Update label="February 11, 2026">
  ### Additional improvements

  * Updated Astro infrastructure to improve reliability and performance.

  ### Bug fixes

  * Fixed an issue where the event timeline in the lineage graph did not correctly filter events based on the **before** time selection.
</Update>

<Update label="February 4, 2026">
  ### New GCP regions available for dedicated clusters

  You can now create dedicated clusters in the following GCP regions:

  * `northamerica-northeast2`
  * `southamerica-west1`
  * `australia-southeast2`

  ### Additional improvements

  * Added an optional **Description** field to all environment manager objects, including Connections, Environment Variables, Airflow Variables, and Metrics Exports.
  * In the [Astro IDE](/docs/astro/ide-overview), you can now include whole directories in chat context instead of selecting files individually, improving bulk operations and multi-file questions.

  ### Bug fixes

  * Fixed an issue where the **Extra** field in the Astro connection manager used a text input instead of a JSON editor, causing errors like `TypeError: 'str' object does not support item assignment` when setting JSON session parameters on connections.
  * Fixed incorrect dag duration metrics caused by duplicate events.
  * Fixed **Run History** date range filtering to include long-running runs that started within the selected range but spanned beyond the end date.
</Update>

<Update label="January 28, 2026">
  ### Astro API v1 now generally available

  The Astro API v1 is now generally available. The v1 API provides a production-ready API with coverage under standard Astronomer support SLAs. The previous v1beta1 API is deprecated and will reach end of support in January 2027. See the [v1beta1 deprecation notice](https://www.astronomer.io/docs/astro/api/v-1-beta-1/v1beta1-deprecation-notice) and [migration guide](/docs/astro/api/v-1-beta-1/migrate-v1-api) for details.

  ### Pre-emptive migrations to shorten Airflow 2 to Airflow 3 upgrades

  Shortened the duration of future in-place Airflow 2 to Airflow 3 upgrades by executing a database migration and preparation across all Airflow 2 Deployments. This applies only to Astro Runtime 12 and 13, as those are the minimum versions required to upgrade to Airflow 3 and Astro Runtime 3. All Runtime versions below 12 are deprecated. Once you upgrade from a deprecated version to Runtime 12 or 13, the pre-emptive migration will be executed. In some cases, this optimization has reduced in-place Airflow 2 to Airflow 3 upgrade times from hours to minutes.

  ### Bug fixes

  * Fixed an issue where Deployments remained stuck in the `Deploying` state if a worker queue name contained Airflow component keywords such as "scheduler."
</Update>

<Update label="January 21, 2026">
  ### Astro Observe data quality now in Preview

  <Info>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Info>

  Astro Observe data quality for Snowflake and Databricks is now available in Preview. Data quality helps you monitor tables and columns for data quality issues including row volume changes, schema changes, and null columns, alerting on issues and displaying upstream and downstream table-level lineage to help keep your data accurate and complete.

  See [data quality](/docs/astro/observe-data-quality) for setup and more information.

  ### Additional improvements

  * Astro Observe [SLAs](/docs/astro/observe-slas) now support flexible cron-based schedules to capture complex business logic and allow more control over SLA sensitivity.
</Update>

<Update label="January 14, 2026">
  ### Additional improvements

  * Organization Owners can now manage API token assignments and roles from the API Token Details page. You can now add an API token to a Workspace or Deployment, update an API token’s role within a Workspace or Deployment, and remove an API token from a Workspace or Deployment directly from **Organization Settings -> Access Management -> API Tokens**.
</Update>

<Update label="January 6, 2026">
  ### Additional improvements

  * You can now view the details and roles assigned to an API token by clicking on any token in the **Organization Settings -> Access Management -> API Tokens** table, which opens the new API Token Details page.
</Update>

## Previous years

* [2025 release notes](/docs/astro/release-notes-2025)
* [2024 release notes](/docs/astro/release-notes-2024)
* [2023 release notes](/docs/astro/release-notes-2023)
* [2022 release notes](/docs/astro/release-notes-2022)
* [2021 release notes](/docs/astro/release-notes-2021)


# 2021 Astro release notes
Source: https://astronomer.io/docs/astro/release-notes-2021

Astro release notes from 2021, covering features, bug fixes, and version updates for Astro, the Astro CLI, Astro Runtime, and the Remote Execution Agent.

<Tip>[Subscribe to Astro release notes](/docs/astro/release-notes-subscribe) to receive updates via RSS, email, or Slack.</Tip>

Astro release notes from 2021. See the [current release notes](/docs/astro/release-notes) for the latest updates.

<Update label="December 16, 2021">
  ### View scheduler error logs from the Astro UI

  The new **Logs** tab in the Astro UI shows scheduler error and warning logs for all Deployments in your Workspace. When you select a Deployment in this menu, all error logs generated over the last 30 minutes appear in the UI.

  To access logs directly for a given Deployment, click the new **Logs** button on the Deployment's page or in the **Deployments** table.

  For more information on how to view logs, read [View logs](/docs/astro/view-logs).

  ### Bug fixes

  Fixed various bugs in the Astro UI to better handle nulls and unknowns in Deployment metrics
</Update>

<Update label="December 9, 2021">
  ### Additional improvements

  * In the Astro UI, the **Open Airflow** button now shows more specific status messages when a Deployment's Airflow UI is inaccessible.

  ### Bug fixes

  * Fixed Deployment table scrolling and alignment issues in the UI
</Update>

<Update label="December 6, 2021">
  ### New "Usage" tab in the Astro UI

  Total task volume for your Organization is now available in a new **Usage** tab in the Astro UI. Astro is priced based on successful task runs, so this view can help you monitor both Astro cost as well as Airflow usage in aggregate and between Deployments.

  <Frame>
    <img alt="Usage tab in the Astro UI" />
  </Frame>

  ### New AWS regions available

  You can now create new clusters in:

  * `us-west-1`
  * `ap-northeast-1`
  * `ap-southeast-1`
  * `ap-northeast-2`
  * `ap-southeast-2`
  * `ap-south-1`
  * `us-west-1`
  * `us-west-2`

  For a full list of AWS regions supported on Astro, see [Resources required for Astro on AWS](https://www.astronomer.io/docs/resource-reference-aws#aws-region).

  ### Additional improvements

  * You can now see your Deployment's **Namespace** in the **Deployments** menu and on the Deployment information screen in the Astro UI. Namespace is a required argument to run tasks with the KubernetesPodOperator. It is also required to submit an issue to [Astronomer support](https://cloud.astronomer.io/open-support-request).

  * The Astro UI now shows a warning if you attempt to exit Environment Variable configuration without saving your changes.

  * A Deployment's health status is now based on the health of both the Airflow webserver and scheduler. Previously, a Deployment's health status was only based on the health of the webserver. Now, the Astro UI will show that your Deployment is "Healthy" only if both components are running as expected.

  ### Bug fixes

  * The Astro UI now has error handling for attempts to access a Deployment that does not exist.
  * If you attempt to modify an existing secret environment variable, the **Value** field is now blank instead of showing hidden characters.

  ### Data plane improvements

  * Amazon EBS volumes have been upgraded from gp2 to [gp3](https://aws.amazon.com/about-aws/whats-new/2020/12/introducing-new-amazon-ebs-general-purpose-volumes-gp3/) for improved scale and performance.
  * EBS volumes and S3 buckets are now encrypted by default.
  * The ability to enable public access to any Amazon S3 bucket on an Astro data plane is now blocked per a new AWS account policy. Previously, public access was disabled by default but could be overridden by a user creating a new S3 bucket with public access enabled. This AWS account policy could be overridden by AWS account owners, but Astronomer strongly recommends against doing so.
</Update>

<Update label="November 19, 2021">
  ### Secret environment variables

  You can now set secret environment variables via the Astro UI. The values of secret environment variables are hidden from all users in your Workspace, making them ideal for storing sensitive information related to your Astro projects.

  For more information, read [Set environment variables via the Astro UI](/docs/astro/manage-env-vars#use-the-astro-ui).

  ### Additional improvements

  * You can now create new clusters in AWS `sa-east-1`.
  * Extra whitespace at the end of any environment variable that is set via the Astro UI is now automatically removed to ensure the variable is passed correctly.
</Update>

<Update label="November 11, 2021">
  ### Deployment metrics dashboard

  In the Astro UI, your Deployment pages now show high-level metrics for Deployment health and performance over the past 24 hours.

  <Frame>
    <img alt="New metrics in the Astro UI" />
  </Frame>

  For more information on this feature, read [Deployment metrics](/docs/astro/deployment-metrics).

  ### Bug fixes

  * Resolved a security vulnerability by setting `AIRFLOW__WEBSERVER__COOKIE_SECURE=True` as a global environment variable
</Update>

<Update label="November 5, 2021">
  ### Bug fixes

  * Fixed an issue where a new user could not exit the Astro UI "Welcome" screen if they hadn't yet been invited to a Workspace
</Update>

<Update label="October 29, 2021">
  ### Cloud UI redesign

  The Astro UI has been redesigned so that you can more intuitively manage Organizations, Workspaces, and your user profile.

  To start, the homepage is now a global view. From here, you can now see all Workspaces that you have access to, as well as information and settings related to your **Organization**: a collection of specific users, teams, and Workspaces. Many features related to Organizations are coming soon, but the UI now better represents how Organizations are structured and what you can do with them in the future.

  You can now also select specific Workspaces to work in. When you click in to a Workspace, you'll notice the left menu bar is now entirely dedicated to Workspace actions:

  * The Rocket icon brings you to the **Deployments** menu.
  * The People icon brings you to the **Workspace Access** menu.
  * The Gear icon brings you to the **Workspace Settings** menu.

  To return to the global menu, you can either click the Astro "A" or click the Workspace name to produce a dropdown menu with your Organization.

  All user configurations can be found by clicking your user profile picture in the upper right corner of the UI. From the dropdown menu that appears, you can both configure user settings and access other Astronomer resources such as documentation and the Astronomer Registry.

  ### Additional improvements

  * You can now create new clusters in `us-east-2` and `ca-central-1`.
  * In the Deployment detail page, **Astro Runtime** now shows the version of Apache Airflow that the Deployment's Astro Runtime version is based on.
  * You can now create or modify an existing Astro cluster to run any size of the `t2`,`t3`, `m5`, or `m5d` [AWS EC2 instances](/docs/astro/resource-reference-aws-hybrid).

  ### Bug fixes

  * Fixed an issue where a new Deployment's health status did not update unless you refreshed the Astro UI
</Update>

<Update label="October 28, 2021">
  ### Bug fixes

  * Fixed an issue where you couldn't push code to Astro with a Deployment API key via a CI/CD process
  * Fixed an issue where you couldn't update or delete an API key after creating it
</Update>

<Update label="October 25, 2021">
  ### Additional improvements

  * When deleting a Deployment via the UI, you now have to type the name of the Deployment in order to confirm its deletion.

  ### Bug fixes

  * Fixed an issue where you could not access Airflow's REST API with a Deployment API key
  * Fixed an issue where calling the `imageDeploy` API mutation with a Deployment API key would result in an error
</Update>

<Update label="October 15, 2021">
  ### Additional improvements

  * When creating a new Deployment, you can now select only the latest patch version for each major version of Astro Runtime.

  * When creating a new Deployment in the Astro UI, the cluster is pre-selected if there is only one cluster available.

  * The name of your Astro Deployment now appears on the main dags view of the Airflow UI.

  * You can now see the health status for each Deployment in your Workspace on the table view of the **Deployments** page in the Astro UI.

  * In the Astro UI, you can now access the Airflow UI for Deployments via the **Deployments** page's card view.

  * The Astro UI now saves your color mode preference.
</Update>

<Update label="October 1, 2021">
  ### Additional improvements

  * In the Astro UI, the **Open Airflow** button is now disabled until the Airflow UI of the Deployment is available.
  * Workspace Admins can now edit user permissions and remove users within a given Workspace.
</Update>

<Update label="September 28, 2021">
  <Danger>This release introduces a breaking change to code deploys via the Astro CLI. Starting on September 28, you must upgrade to v1.0.0+ of the CLI to deploy code to Astro. [CI/CD processes](/docs/astro/set-up-ci-cd) enabled by Deployment API keys will continue to work and will not be affected. For more information, read the [CLI release notes](/docs/cli/v1.43/release-notes).</Danger>

  ### Additional improvements

  * In the Astro UI, a new element on the Deployment information screen shows the health status of a Deployment. Currently, a Deployment is considered unhealthy if the Airflow webserver is not running and the Airflow UI is not available:

    <Frame>
      <img alt="Deployment Health text in the UI" />
    </Frame>

  * The documentation home for Astro has been moved to `www.astronomer.io/docs`, and you no longer need a password to access the page.

  ### Bug fixes

  * The Astro UI now correctly renders a Deployment's running version of Astro Runtime.
</Update>

<Update label="September 17, 2021">
  ### Support for Deployment API keys

  Astro now officially supports Deployment API keys, which you can use to automate code pushes to Astro and integrate your environment with a CI/CD tool such as GitHub Actions. For more information on creating and managing Deployment API keys, see Deployment API keys. For more information on using Deployment API keys to programmatically deploy code, see [CI/CD](/docs/astro/set-up-ci-cd). Support for making requests to Airflow's REST API using API keys is coming soon.
</Update>

<Update label="September 3, 2021">
  ### Bug fixes

  * Added new protections to prevent S3 remote logging connections from breaking
  * Fixed an issue where environment variables with extra spaces could break a Deployment
  * Fixed an issue where Deployments would occasionally persist after being deleted via the UI
  * In the UI, the **Organization** tab in **Settings** is now hidden from non-admin users
  * In the UI, the table view of Deployments no longer shows patch information in a Deployment's **Version** value
</Update>

<Update label="August 27, 2021">
  ### Additional improvements

  * You can now remain authenticated to Astro across multiple active browser tabs. For example, if your session expires and you re-authenticate to Astro on one tab, all other tabs running Astro will be automatically updated without refreshing.
  * If you try to access a given page on Astro while unauthenticated and reach the login screen, logging in now brings you to the original page you requested.

  ### Bug fixes

  * Fixed an issue where an incorrect total number of team members would appear in the **People** tab
</Update>

<Update label="August 20, 2021">
  ### Support for the Airflow REST API

  You can now programmatically trigger dags and update your Deployments on Astro by making requests to Airflow's [REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html). Currently this feature works only with temporary tokens, which are available at `cloud.astronomer.io/token`. Support for Deployment API keys is coming soon. For more information on using this feature, read [Airflow API](/docs/astro/airflow-api).

  ### Additional improvements

  * Set `AIRFLOW_HOME = 'usr/local/airflow'` as a permanent global environment variable
  * In the Astro UI, long environment variable keys and values now wrap to fit the screen
  * Added links for the Astronomer Registry and certification courses to the left-hand navbar
  * Moved the **Teams** and **People** tabs into the **Settings** page of the UI
  * Added **Cluster** information to the metadata section of a Deployment's information page in the UI
  * Renamed various UI elements to better represent their functionality
  * Increased the maximum **Worker Termination Grace Period** from 600 minutes (10 hours) to 1440 minutes (24 hours)

  ### Bug fixes

  * The left-hand navbar in the UI is no longer cut off when minimized on smaller screens
  * Fixed an issue where you could not delete a Workspace via the UI
  * Fixed an issue where expired tokens would occasionally appear on `cloud.astronomer.io/token`
  * Fixed an issue where the UI would initially load an inaccurate number of team members on the **Access** page
  * Fixed alphabetical sorting by name in the **People** tab in the UI
  * Removed placeholder columns from various tables in the UI
</Update>

<Update label="August 6, 2021">
  ### Additional improvements

  * Informational tooltips are now available on the **New Deployment** page.

  ### Bug fixes

  * Fixed an issue where adding a user to a Workspace and then deleting the user from Astro made it impossible to create new Deployments in that Workspace
  * Improved error handling in the Airflow UI in cases where a user does not exist or does not have permission to view a Deployment
</Update>

<Update label="July 30, 2021">
  ### Improvements

  * Increased the limit of **Worker Resources** from 30 AU to 175 AU (17.5 CPU, 65.625 GiB RAM). If your tasks require this many resources, reach out to us to make sure that your cluster is sized appropriately
  * Collapsed the **People** and **Teams** tabs on the left-hand navigation bar into a single **Access** tab
  * Added a **Cluster** field to the Deployments tab in the Astro UI. Now, you can reference which cluster each of your Deployments is in
  * Replaced our white "A" favicon to one that supports color mode
  * Informational tooltips are now available in **Deployment Configuration**

  ### Bug fixes

  * Fixed an issue where a deleted user could not sign up to Astro again
  * Removed Deployment-level user roles from the Astro UI. Support for them coming soon
  * Fixed an issue where a newly created Deployment wouldn't show up on the list of Deployments in the Workspace
</Update>


# 2022 Astro release notes
Source: https://astronomer.io/docs/astro/release-notes-2022

Astro release notes from 2022, covering features, bug fixes, and version updates for Astro, the Astro CLI, Astro Runtime, and the Remote Execution Agent.

<Tip>[Subscribe to Astro release notes](/docs/astro/release-notes-subscribe) to receive updates via RSS, email, or Slack.</Tip>

Astro release notes from 2022. See the [current release notes](/docs/astro/release-notes) for the latest updates.

<Update label="December 20, 2022">
  ### Additional improvements

  * You can now configure OneLogin and Ping Identity as identity providers on Astro.
  * Workspace Members can now view **Workspace settings** in the Astro UI.
  * Node groups that are collapsed in the lineage graph in the Astro UI now show only the total number of connected **Jobs** and **Datasets**, instead of listing each job and dataset. This makes the lineage graph easier to navigate.

    <Frame>
      <img alt="Collapsed node in lineage graph of Astro UI" />
    </Frame>

  ### Bug fixes

  * Fixed an issue where the lineage UI did not show dataset metrics when a dataset had no column-level metrics.
  * Fixed an issue where some instances of a dataset's name were inconsistent in the lineage UI.
</Update>

<Update label="December 13, 2022">
  ### Improvements to the Cloud IDE

  The Cloud IDE includes several new features which improve dag authoring and testing:

  * There is a new **Commit** button in the Astro UI that is separate from the **Configuring GitHub** menu.
  * The default CI/CD pipeline included in the Cloud IDE project supports dag-only deploys. Deploying dag changes to Astro using the CI/CD pipeline is now significantly faster.
  * The **Configure GitHub** menu in the Astro UI now includes a **Clone Repo** settings menu. Enabling this option makes other files in your GitHub repository, such as helper functions in the `include` folder of your project, accessible when you run dags in the Cloud IDE.
  * You can now explicitly mark upstream dependencies for a task cell from the cell's configuration menu.

  ### Support for n2 worker types on GCP

  You can now configure worker queues with the following `n2` worker types on Google Cloud Platform (GCP) clusters:

  * `n2-standard-4`
  * `n2-standard-8`
  * `n2-standard-16`
  * `n2-highmem-4`
  * `n2-highmem-8`
  * `n2-highmem-16`
  * `n2-highcpu-4`
  * `n2-highcpu-8`
  * `n2-highcpu-16`

  For more information about these worker types, see [N2 machine series](https://cloud.google.com/compute/docs/general-purpose-machines#n2_machines). For a list of all worker types available on GCP, see [Worker node size resource reference](/docs/astro/resource-reference-gcp-hybrid#supported-worker-node-pool-instance-types).

  ### Additional improvements

  * In the **Clusters** tab of the Astro UI, you can now click a cluster entry to see details about the cluster configuration, including which **Worker Types** are enabled for the cluster.
  * The Deployment details page in the Astro UI now includes an **ID** pane. A Deployment ID is required when you deploy code using a CI/CD process.
  * The **OpenLineage URL** for your Organization is now available on the **Settings** page in the Astro UI. An OpenLineage URL is required to [integrate data lineage from some external systems](/docs/astro/observe-openlineage).
  * Workspaces are now sorted alphabetically in the Astro UI.
  * In Astro CLI version 1.8.0 or later, running `astro deploy` with an empty or missing `dags` folder does not erase or override existing dags. Instead, the directory is excluded from the build and push process to Astro. This lets you manage your dags and project files in separate repositories when using [dag-only deploys](/docs/astro/deploy-dags).

  ### Bug fixes

  * Fixed an issue where Astro temporarily stored dags for dag-only deploys in a new directory named `/usr/local/airflow/dags/current`, which could cause import errors in user code.
  * Fixed an issue where task runs triggered in the Cloud IDE did not have access to project environment variables.
  * Fixed an issue where Deployment metrics for memory usage were not always accurate.
</Update>

<Update label="November 15, 2022">
  ### Additional improvements

  * In the Astro UI, the **People** page now shows the IDs of users belonging to your Organization.
  * In the Astro UI, the **Deployments** page now shows the user or API key that most recently updated each Deployment and when they updated it.

  ### Bug fixes

  * Availability zone (AZ) rebalancing has been disabled for worker node pools on AWS clusters. This change should result in fewer [zombie tasks](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#zombie-undead-tasks) and less volatility across workers. AZ rebalancing is enabled for other system components on Astro.
  * The **Updated at** field for a transferred Deployment now displays the correct time.
  * `astro deploy --dags` now handles deferrable tasks correctly.
</Update>

<Update label="November 8, 2022">
  ### Deploy only dags with `astro deploy -—dags`

  Using Astro CLI 1.7, you can run `astro deploy -—dags` to push only the `dags` directory of your Astro project to a Deployment on Astro. This is an additional option to `astro deploy` that makes for a faster development experience and gives you more flexibility in how you configure CI/CD processes.

  For more information, see [Astro CLI 1.7](/docs/cli/v1.43/release-notes#deploy-only-dags-with-astro-deploy-—dags) or [Deploy dags only](/docs/astro/deploy-dags). For example CI/CD workflows with this feature enabled, see [CI/CD](/docs/astro/ci-cd-templates/template-overview#dag-deploy-templates).

  ### Improved data lineage interface

  The **Lineage** tab has new features and is better integrated into the Astro UI.

  <Frame>
    <img alt="Updated lineage page" />
  </Frame>

  Specifically, the tab includes the following improvements:

  * The process for comparing runs uses a simpler interface and provides more information about the runs you're comparing.
  * Names for UI elements have been updated to more clearly represent Airflow resources. For example, **jobs** is now **runs**, and the **Explore** tab is now **Runs**.
  * Lineage graphs include new colors and animations to show the flow of data as it moves between runs and datasets.

  ### Transfer a Deployment

  You can now transfer a Deployment from one Workspace to another in your Organization. This feature is helpful if you need to change the group of users that have access to a Deployment, or if you create a Deployment in the wrong Workspace.

  See [Transfer a Deployment to another Workspace](/docs/astro/transfer-a-deployment).

  ### Additional improvements

  * The Kubernetes API is no longer exposed to the public internet on AWS data planes. The allowlist is limited to control plane IPs. New clusters will be created with this configuration, while all existing clusters will be updated by end of next week.
</Update>

<Update label="November 1, 2022">
  ### Introducing the Astro Cloud IDE, a new Airflow development experience

  Astronomer is excited to introduce the Astro Cloud IDE, which is a notebook-inspired development environment for writing, running, and deploying data pipelines. Now you can develop an entire Airflow project, including dags, dependencies, and connections entirely within the Astro UI.

  <Frame>
    <img alt="Example page in the Astro Cloud IDE" />
  </Frame>

  The Astro Cloud IDE was created with the following objectives:

  * Configuring Airflow shouldn't be a barrier to running Airflow.
  * Passing data between tasks should be seamless regardless of what language is used to write the task.
  * Data pipelines should be quick to deploy and easy to test with CI/CD.

  Most importantly, the Astro Cloud IDE was developed to make it easier for new Airflow users to get started and to provide experienced users with a robust development environment.

  ### Additional improvements

  * In the Astro UI, cluster selection menus are now alphabetized.

  ### Bug fixes

  * Fixed an issue where the KubernetesPodOperator was not aware of available ephemeral storage in `m5d` and `m6id` worker nodes. This issue resulted in Pods being evicted to free up storage even when there was enough available storage for tasks.
  * Fixed an issue in the Astro UI where you could select a worker type before selecting a cluster when creating a Deployment.
  * Fixed an issue where Deployments on Runtime 5.0.10 and earlier showed a nonfunctional **Configuration** tab in the Airflow UI.
  * Fixed [CVE-2022-32149](https://nvd.nist.gov/vuln/detail/CVE-2022-32149).
</Update>

<Update label="October 25, 2022">
  ### Additional improvements

  * In the Astro UI, you can now view a cluster's external IP addresses in the **Clusters** tab.

  ### Bug fixes

  * Fixed an issue where some Deployments were running tasks after being deleted.
</Update>

<Update label="October 18, 2022">
  ### Additional improvements

  * In the Astro UI, **Access** has been moved from the left menu to a tab on the **Workspace Settings** page.
  * In the Astro UI, **Workspace Settings** in the left menu is now available to all Workspace members.

  ### New Azure regions

  You can now [create an Astro cluster on Azure](/docs/astro/manage-hybrid-clusters#create-a-cluster) in the following regions:

  * `japaneast`
  * `southafricanorth`
  * `southcentralus`
</Update>

<Update label="October 11, 2022">
  ### Additional improvements

  * New worker node pools on Azure and Google Cloud Platform (GCP) clusters can now scale to zero. When you set your minimum worker count to 0, you don't incur costs for enabling a new worker type for your cluster until it's used in a Deployment.

  ### Bug fixes

  * Fixed an issue where worker queues with a minimum worker count of zero would appear with a minimum worker count of one in the Astro UI.
</Update>

<Update label="October 4, 2022">
  ### New permissions boundary for managed AWS Accounts

  The operational roles that Astronomer assumes on dedicated customer AWS accounts now have new [permissions boundaries](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_boundaries.html) that limit the roles to a subset of their permissions. The remote management role is now limited to the following actions across all contexts:

  * `autoscaling:*`
  * `cloudformation:*`
  * `cloudwatch:*`
  * `ec2:*`
  * `ecr:*`
  * `eks:*`
  * `elasticloadbalancing:*`
  * `iam:*OpenID*`
  * `kms:DescribeKey`
  * `lambda:*`
  * `logs:*`
  * `route53:AssociateVPCWithHostedZone`
  * `s3:*`
  * `secretsmanager:*`
  * `servicequotas:*`
  * `ssm:*`
  * `tag:*`

  These permissions might change in the future to enable new Astro features or to refine permissions for specific contexts.

  ### Additional improvements

  * Users with the required permissions can now access a **Configuration** tab in the **Admin** menu of the Airflow UI. This page no longer shows sensitive values in plain-text and can be used to verify all configurations running on your Deployment.
  * In the Astro UI, the maximum time for Deployment metrics has been extended from 24 hours to 7 days.
  * The [Deployment metrics overview](/docs/astro/deployment-metrics#deployment-overview) now shows metrics for the `default` worker queue instead of an aggregate of all worker queues. Improved worker queue metrics coming soon.

  ### Bug fixes

  * Added the global environment variable `AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_TARGET=stdout` so that a scheduler's logs don't overcrowd its local storage
  * Removed misleading maximum CPU and memory lines from Deployment metric graphs
</Update>

<Update label="September 28, 2022">
  ### Additional improvements

  * All worker queue configurations in the **Worker Queues** tab of the Astro UI now have tooltips.
  * The **Worker CPU** and **Worker Memory** metrics in the **Analytics** tab of the Astro UI now show metrics only for the default worker queue instead of an average across queues. Improved worker queue metrics coming soon.

  ### Bug fixes

  * Values in the Organization **Settings** page no longer overlap with other UI elements.
  * Organization Owners can now [push code](/docs/astro/deploy-code) to a Deployment even if they aren't explicit members of the Deployment's Workspace.
</Update>

<Update label="September 21, 2022">
  ### A simpler Deployment page

  All of a Deployment's configurations, including analytics, API keys, environment variables, and resource configurations, are now organized as tabs within the Deployment's page in the Astro UI.

  This new UI moves the **Analytics** and **Logs** from the left sidebar to the main Deployment page so that you no longer have to filter those views separately by Deployment. The left sidebar now exclusively contains Workspace-level menus.

  ### New Account Dashboard

  You can now access your Account Dashboard to manage your user account settings and find links to helpful resources. Access this page by going to `account.astronomer.io` in your browser or by clicking **Profile** > **Manage your Astro Account** in the Astro UI. You must be authenticated to Astro.

  ### Additional improvements

  * You can now use the `m6id` worker node type series for Deployments on AWS clusters. This worker type is general purpose and includes significant storage as well as up to 15% better performance compared to `m5d` nodes. For more information, see [Worker instance types](/docs/astro/resource-reference-aws-hybrid#supported-worker-node-pool-instance-types).
  * New worker node pools on Amazon Web Services (AWS) clusters can now scale to zero. This means that enabling a new worker type for your cluster does not cost you until it's used in a Deployment.

  ### Bug fixes

  * Fixed an issue where the Astro UI Deployment metrics showed a maximum worker CPU and memory that was inconsistent with your configured worker queues.
</Update>

<Update label="September 14, 2022">
  ### Additional improvements

  * When you create a new worker queue, the default worker type in your cluster is now pre-selected in the **Worker Type** list.
  * You can now configure multiple instances of the same identity provider (IdP). See [Configure an identity provider](/docs/astro/configure-idp).
  * You can now expand and collapse the **Workspace** menu in the Astro UI.

  ### Bug fixes

  * Fixed an issue where you could not open the Airflow UI from a Deployment.
</Update>

<Update label="August 31, 2022">
  ### Export Deployment metrics to Datadog

  You can now export over 40 Airflow metrics related to the state of your Astro Deployment to [Datadog](https://www.datadoghq.com/) by adding a Datadog API key to the Deployment. Metrics include task successes, dag processing time, frequency of import errors, and more.

  For organizations already using the observability service, this integration allows your team to standardize on tooling and gain a more granular view of Deployment metrics in a single place. Once the integration is configured, Astro automatically exports all available metrics to Datadog. For a complete list of supported metrics, see [Data Collected](https://docs.datadoghq.com/integrations/airflow/?tab=host#data-collected).

  To learn more, see [Export Airflow metrics to Datadog](/docs/astro/export-datadog).

  ### Additional improvements

  * The Astro UI now automatically ensures that worker queue names are valid as you type in real time.
  * The number of times that a user can enter the wrong credentials for Astro before being locked out has been reduced from 10 to 6.
  * You can now configure [worker queues](/docs/astro/configure-worker-queues#worker-queue-settings) to have a minimum **Worker count** of 0 workers. Note that depending on your cloud provider and Deployment configurations, some Deployments still might not be able to scale to 0 workers.

  ### Bug fixes

  * The timestamp shown in the **Updated** field of the Deployment view in the Astro UI is now properly updated when you create or modify environment variables.
  * Fixed an issue where logging in to the Airflow UI with unrecognized credentials could freeze you on an error page.
</Update>

<Update label="August 24, 2022">
  ### Additional improvements

  * When you configure worker queues in the Astro UI, the total CPU and memory capacity of each worker instance type is now shown instead of the nominal available resources.
  * Improved error handling for creating new worker queues when soft-deleted worker queues might still exist on the data plane.

  ### Bug fixes

  * Fixed an issue where running `astro deploy` with a Deployment API key could revert changes to a worker queue's size that were previously set in the Astro UI.
  * Fixed an issue where the **Lineage** tab in the Astro UI showed all job durations as having a length of 0.
</Update>

<Update label="August 18, 2022">
  ### Create multiple worker queues

  Worker queues are a new way to configure your Deployment to best fit the needs of your tasks. A worker queue is a set of configurations that apply to a group of workers in your Deployment. Within a worker queue, you can configure worker type and size as well as autoscaling behavior. By configuring multiple worker queues for different types of tasks, you can better optimize for the performance, reliability, and throughput of your Deployment.

  In the Astro UI, you can now create multiple worker queues. Once you create a worker queue, you can assign a task to that worker queue by adding a simple `queue='<worker-queue-name>'` argument in your dag code.

  This feature enables the ability to:

  * Use more than one worker type within a single Deployment and cluster. Previously, a single cluster on Astro supported only one worker type.
  * Isolate long-running tasks from short-running tasks to avoid errors related to competing resource requests.
  * Fine-tune autoscaling behavior for different groups of tasks within a single Deployment.

  For example, if you have a task that requires significantly more CPU than memory, you can assign it to a queue that's configured with workers that are optimized for compute usage.

  To learn more about configuring worker queues, see [Configure Deployment resources](/docs/astro/configure-worker-queues).

  ### New worker sizing

  This Astro release introduces a new, simple way to allocate resources to the workers in your Deployment. Instead of choosing a varying combination of CPU and memory, you can now select a worker type in the Astro UI as long as it's enabled in your cluster. For example, `m5.2xlarge` or `c6i.8xlarge` on AWS. Once you select a worker type, Astronomer will create the biggest worker that the worker type can support to ensure that your tasks have enough resources to execute successfully.

  Astro's worker sizing enables a few benefits:

  * You can no longer configure a worker that is too large or otherwise not supported by your underlying cluster. Previously, misconfiguring worker size often resulted in task failures.
  * A more efficient use of infrastructure. Astronomer has found that a lower number of larger workers is more efficient than a higher number of smaller workers.
  * A higher level of reliability. This worker sizing model results in less volatility and a lower frequency of cluster autoscaling events, which lowers the frequency of errors such as zombie tasks and missing task logs.
  * The legacy **AU** unit is no longer applicable in the context of the worker. You only have to think about CPU, memory, and worker type.

  Worker sizing on Astro is now defined in the context of worker queues. For more information about worker sizing, see [Configure Deployment resources](/docs/astro/configure-worker-queues). For a list of supported worker types, see the [AWS](/docs/astro/resource-reference-aws-hybrid#supported-worker-node-pool-instance-types), [GCP](/docs/astro/resource-reference-gcp-hybrid#supported-worker-node-pool-instance-types), and [Azure](/docs/astro/resource-reference-azure-hybrid#supported-worker-node-pool-instance-types) resource references.

  ### New Maximum Tasks per Worker setting

  A new **Maximum Tasks per Worker** configuration is now available in the Deployment view of the Astro UI. Maximum tasks per worker determines the maximum number of tasks that a single worker can process at a time and is the basis of worker autoscaling behavior. It is equivalent to [worker concurrency](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#worker-concurrency) in Apache Airflow.

  Previously, maximum tasks per worker was permanently set to 16 and was not configurable on Astro. Now, you can set maximum tasks per worker anywhere between 1 and 64 based on the needs of your tasks. It can be set per worker queue on a Deployment.

  To learn more, see [Worker autoscaling logic](/docs/astro/celery-executor#celery-worker-autoscaling-logic).

  ### New Worker Count (Min-Max) setting

  A new **Worker Count (Min-Max)** configuration is now available in the Deployment view of the Astro UI. This value defines the minimum and maximum number of workers that can run at a time.

  Use this setting to fine-tune worker autoscaling behavior in your Deployment. By default, the minimum number of workers is 1 and the maximum is 10.

  ### Support for multiple Organizations

  A single user account can now belong to multiple Organizations. A user with multiple Organizations can switch to another Organization by clicking on their current Organization's name in the Astro UI and then clicking **Switch Organization**.

  Note that switching Organizations with the Astro CLI is not yet supported. For more information, see [Switch Organizations](/docs/astro/log-in-to-astro#switch-organizations).

  ### New Azure region (Australia East)

  You can now [create an Astro cluster on Azure](/docs/astro/manage-hybrid-clusters#create-a-cluster) in Australia East (New South Wales).

  ### New Google Cloud Platform regions

  You can now [create an Astro cluster on GCP](/docs/astro/manage-hybrid-clusters#create-a-cluster) in the following regions:

  * `australia-southeast2` (Melbourne)
  * `asia-east1` (Taiwan)
  * `asia-south2` (Delhi)
  * `asia-southeast2` - (Jakarta)
  * `europe-north1` (Finland)
  * `europe-southwest1` (Madrid)
  * `europe-west8` (Milan)
  * `europe-west9` (Paris)
  * `northamerica-northeast2` (Toronto)
  * `southamerica-west1` (Santiago)
  * `us-east5` (Columbus)
  * `us-south1` (Dallas)

  ### Bug fixes

  * Fixed an issue where the Astro UI's **Resource Settings** page wasn't showing units for CPU and Memory values.
</Update>

<Update label="August 10, 2022">
  ### Updated user permissions for Organization and Workspace roles

  The following user roles have new and modified permissions:

  * Organization Owners now have Workspace Admin permissions for all Workspaces in their Organization. This role can now access Organization Workspaces, Deployments, and usage data.
  * Organization Billing Admins can now view usage for all Workspaces in their Organization regardless of their Workspace permissions.
  * Workspace Editors can now delete any Deployment in their Workspace.

  ### Automatic access for new users authenticating with an identity provider

  If your organization has [implemented an identity provider (IdP)](/docs/astro/configure-idp), any new user who authenticates to Astro through your IdP is now automatically assigned the Organization Member role. This means that users authenticating through your IdP don't need to be invited by email before joining your Organization.

  ### Additional improvements

  * Added a security measure that ensures Workspace roles can only be assigned to users who have an Organization role in the Organization in which the Workspace is hosted. This ensures that a user who does not belong to your Organization cannot be assigned a Workspace role within it.
</Update>

<Update label="August 2, 2022">
  ### Support for Astro on Azure Kubernetes Service (AKS)

  Astro now officially supports Astro clusters on AKS. This includes support for an initial set of AKS regions.

  For more information about the installation process and supported configurations, see [Install Astro on Azure](/docs/astro/install-azure-hybrid) and [Resource Reference Azure](/docs/astro/resource-reference-azure-hybrid).

  ### Bug fixes

  * Pending invites no longer appear for active users in the Astro UI.
</Update>

<Update label="July 27, 2022">
  ### New Deployment optimizations for high availability (HA)

  This release introduces two changes that ensure a higher level of reliability for Deployments on Astro:

  * [PgBouncer](https://www.pgbouncer.org/), a microservice that increases resilience by pooling database connections, is now considered highly available on Astro. Every Deployment must now have 2 PgBouncer Pods instead of 1, each assigned to a different node within the cluster. This change protects against pod-level connection issues resulting in [zombie tasks](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#zombie-undead-tasks), which was previously seen during cluster downscaling events. PgBouncer is fully managed by Astronomer and is not configurable.

  * The Airflow scheduler is now configured with an [anti-affinity policy](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) to limit the possibility of all schedulers for a single Deployment being impacted by an incident within a single node on an Astro cluster. For users who set **Scheduler Count** in the Astro UI to 2, this means that those 2 scheduler Pods cannot be assigned to the same node and instead require a minimum of 2 nodes total. To avoid significant increases in cost, 3 or 4 schedulers can share the same 2 nodes and will not necessarily result in a higher node count minimum.

  For more information on Deployment configurations, see [Deployment settings](/docs/astro/deployment-settings).

  ### Additional improvements

  * Added tooltips for [Deployment overview metrics](/docs/astro/deployment-metrics#deployment-overview) in the Astro UI.
</Update>

<Update label="July 21, 2022">
  ### Additional improvements

  * You can now access an Organization's AWS external ID from the **Settings** tab of the Astro UI.
  * Organizations now need only a single AWS external ID for all clusters. Previously, each cluster required a unique external ID, which added complexity to the installation and cluster creation process.
  * You can now remove a user from an Organization from the Astro UI.
  * Organization Billing Admins can now view task usage for all Workspaces regardless of their Workspace permissions.
</Update>

<Update label="July 14, 2022">
  ### Additional improvements

  * The Astro UI **Clusters** page now includes the cluster ID value.
  * Organization Owners and Organization Billing Admins can now update the Organization name in the Astro UI **Settings** page.
  * The Astro UI **Analytics** page can now show data for the last 30 minutes.

  ### Bug fixes

  * When you change **Worker Resources** for a Deployment in the Astro UI, any errors related to your worker size request are now based on the correct node instance type that your cluster is running.
  * When you select a Workspace and click **Go back** in a browser, the page now reloads as expected.
  * A **Page not found** error message no longer appears when you select a Deployment in the **Usage** page of the Astro UI.
  * The **Deployment Analytics** page now displays the correct date.
  * Deprecated versions of Astro Runtime now appear correctly in the Deployment page of the Astro UI. Previously, versions were appended with `-deprecated`.
</Update>

<Update label="June 30, 2022">
  ### New Google Cloud Platform regions

  You can now [create an Astro cluster on GCP](/docs/astro/manage-hybrid-clusters#create-a-cluster) in the following regions:

  * `asia-northeast1` (Tokyo)
  * `asia-northeast2` (Osaka)
  * `asia-northeast3` (Seoul)
  * `asia-south1` (Mumbai)
  * `europe-central2` (Warsaw)
  * `europe-west6` (Zurich)
  * `northamerica-northeast1` (Montreal)
  * `us-west3` (Salt Lake City)

  ### Additional improvements

  * You can now search for Organization members by name, email address, and role in the **People** tab of the Organization view in the Astro UI. You can also search for members in the **Access** tab of the Workspace view.

  ### Bug fixes

  * Fixed an issue where you could not use the KubernetesPodOperator to execute tasks in a Kubernetes cluster outside of your Astro cluster. See [KubernetesPodOperator](/docs/astro/kubernetespodoperator).
</Update>

<Update label="June 23, 2022">
  ### New Google Cloud Platform regions

  You can now [create an Astro cluster on GCP](/docs/astro/manage-hybrid-clusters#create-a-cluster) in the following regions:

  * `asia-southeast1` (Singapore)
  * `australia-southeast1` (Sydney)
  * `europe-west1` (Belgium)
  * `europe-west2` (England)
  * `europe-west3` (Frankfurt)
  * `southamerica-east1` (São Paulo)
  * `us-west2` (Los Angeles)
  * `us-west4` (Nevada)
</Update>

<Update label="June 16, 2022">
  ### Submit Support Requests in the Astro UI

  Support requests can now be created and submitted in the Astro UI. You no longer need to open an account on the Astronomer support portal to reach the Astronomer team. To streamline the request process, the **Submit Support Request** form auto-populates your currently selected Workspace and Deployment in the Astro UI.

  ### Parallelism Now Autoscales with a Deployment's Worker Count

  To better scale concurrent task runs, Astro now dynamically calculates [`parallelism`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#parallelism), which is an Airflow configuration that determines the maximum number of tasks that can run concurrently within a single Deployment.

  A Deployment's `parallelism` is now equal to the current number of workers multiplied by the [`worker_concurrency`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#worker-concurrency) value. This change ensures that your task runs won't be limited by a static parallelism limit as workers autoscale in your Deployment. See [Worker Autoscaling Logic](/docs/astro/celery-executor#celery-worker-autoscaling-logic) for more information.

  Note that you can still use a static `parallelism` value by setting `AIRFLOW__CORE__PARALLELISM` as an [environment variable](/docs/astro/environment-variables).

  ### Bug Fixes

  * Fixed a rare issue where some user emails would be associated with the wrong username.
  * Fixed an issue where you could not properly sort entries in the **People** tab by name.
</Update>

<Update label="June 9, 2022">
  ### Update Deployment configurations with the Astro CLI

  You can now programmatically update the configurations for your Astro Deployments using Deployment API keys and the Astro CLI. Updating a Deployment with an API key doesn't require manual user authentication, meaning that you can now add Deployment configuration steps to automated processes such as CI/CD pipelines.

  Specifically, you can now run the following commands with Deployment API keys:

  * [`astro deployment list`](/docs/cli/v1.43/astro-deployment-list)
  * [`astro deployment update`](/docs/cli/v1.43/astro-deployment-update)
  * [`astro deployment variable create`](/docs/cli/v1.43/astro-deployment-variable-create)
  * [`astro deployment variable list`](/docs/cli/v1.43/astro-deployment-variable-list)
  * [`astro deployment variable update`](/docs/cli/v1.43/astro-deployment-variable-update)

  ### Bug fixes

  * Fixed an issue where a Deployment's logs wouldn't load in the Astro UI if it was the only Deployment in the Workspace
</Update>

<Update label="June 2, 2022">
  ### Support for the `us-east4` GCP region

  You can now [create an Astro cluster on GCP](/docs/astro/manage-hybrid-clusters#create-a-cluster) in the `us-east4` region, which is located in northern Virginia, USA.
</Update>

<Update label="May 26, 2022">
  ### New Datasets page in the Astro UI

  You can now use the new **Datasets** page in the **Lineage** tab to view a table of datasets that your dags have read or written to. This information can help you quickly identify dataset dependencies and data pipeline access requirements.

  Click the name of a dataset to show its lineage graph.

  ### Bug fixes

  * Fixed an issue where the **Astro Runtime** field of the Astro UI listed the running version as **Unknown**  for Deployments using an unsupported version of Astro Runtime
</Update>

<Update label="May 5, 2022">
  ### Data lineage Is now available on Astro

  We are excited to introduce data lineage to Astro. You now have access to a new **Lineage** view in the Astro UI that visualizes data movement across datasets in your Organization based on integrations with Airflow, Apache Spark, dbt, Great Expectations, and more.

  Built around the [OpenLineage](https://openlineage.io/) open source standard, the data lineage graphs and metadata in the Astro UI can help you better understand your ecosystem and diagnose issues that may otherwise be difficult to identify.

  For example, if an Airflow task failed because the schema of a database changed, you might go to the Lineage page on Astro to determine which job caused that change and which downstream tasks failed because of it.

  To learn more about data lineage and how you can configure it on Astro, see:

  * [Integrate Airflow and OpenLineage](/docs/learn/airflow-openlineage)
  * [Enable data lineage for External Services](/docs/astro/observe-openlineage)
  * [OpenLineage Compatibility Matrix](https://openlineage.io/docs/integrations/about#capability-matrix)

  <Info>This functionality is still early access and under active development. If you have any questions or feedback about this feature, reach out to [Astronomer support](https://cloud.astronomer.io/open-support-request).</Info>

  ### Support for Astro on Google Cloud Platform (GCP)

  Astro now officially supports Astro clusters on Google Cloud Platform (GCP). This includes support for an initial set of GCP regions as well as [Workload Identity](https://cloud.google.com/iam/docs/manage-workload-identity-pools-providers) for secure connection to other GCP data services in your ecosystem.

  For more information about the installation process and supported configurations, see [Install Astro on GCP](/docs/astro/install-gcp-hybrid) and [Resource Reference GCP](/docs/astro/resource-reference-gcp-hybrid).

  ### Support for Organization-Level user invites

  You can now [invite users to an Astro Organization](/docs/astro/manage-organization-users#add-a-user-to-an-organization) without having to first invite them to a specific Workspace. Users invited to an Organization will receive an activation email which brings them directly to the Organization view of the Astro UI.

  ### Additional improvements

  * Improved the templated emails sent out for user invites with clear instructions for how to get started on Astro
  * Improved error messaging behavior on the **DAGs** and **Usage** pages of the Astro UI
  * New user accounts must now be verified via email before they can access Astro
</Update>

<Update label="April 28, 2022">
  ### New AWS node instance types available

  To widen our support for various use cases and levels of scale, we've expanded the types of AWS node instances that are supported on Astro. You can now create clusters with:

  * [General Purpose M6i instances](https://aws.amazon.com/ec2/instance-types/m6i/)
  * [Compute Optimized C6i instances](https://aws.amazon.com/ec2/instance-types/c6i/)
  * [Memory Optimized R6i instances](https://aws.amazon.com/ec2/instance-types/r6i/)

  To modify an existing Astro cluster to use any of these instance types, see [Modify a Cluster](/docs/astro/manage-hybrid-clusters).

  ### Additional improvements

  * Improve the error message that renders in the Astro UI if you try to create a worker that is too large for the Deployment's node instance type to support. This error message now specifies a clear call to action
</Update>

<Update label="April 21, 2022">
  ### Feedback in Astro UI on worker size limits

  The Astro UI now renders an error if you try to modify the **Worker Resources** to a combination of CPU and memory that is not supported by the node instance type of the cluster that the Deployment is hosted on. This validation ensures that the worker size you request is supported by the infrastructure available in your Astro cluster, and minimizes silent task failures that might have occurred due to invalid resource requests.

  If your Astro cluster is configured with the `m5d.8xlarge` node type, for example, the Astro UI will show an error if you try to set **Worker Resources** to 350 AU. This is because the maximum worker size an `m5d.8xlarge` node can support is 307 AU.
</Update>

<Update label="April 14, 2022">
  ### Additional improvements

  * The data plane now connects to various AWS services via [AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/endpoint-services-overview.html). This ensures that traffic to AWS services is kept private and does not traverse the NAT and Internet gateways, reducing the risk of exposing your resources to the internet.

  ### Bug fixes

  * Fixed an issue where you could not add a new user to a Workspace if the user had an email address that contained uppercase characters
</Update>

<Update label="March 31, 2022">
  ### New analytics page in Astro UI to monitor Deployments

  The Astro UI now includes a dedicated **Analytics** page that contains various Deployment-level metrics. These metrics are collected in real time and can provide insight into how your data pipelines are performing over time.

  For more information about accessing the **Analytics** page and the available metrics, see [Deployment Analytics](/docs/astro/deployment-metrics#deployment-analytics).

  ### Lineage backend upgrade scheduled for all Organizations

  As part of [Astronomer's acquisition of Datakin](https://www.astronomer.io/blog/astronomer-acquires-datakin-the-data-lineage-tool/), data lineage features are coming soon to Astro. The first step in enabling these features is to implement lineage backends for existing Astro customers.

  Starting on March 31st and continuing over the next couple of weeks, all Astro Deployments on Runtime 4.2.0+ will be upgraded to emit lineage events. As a result of this change, you might start seeing lineage-related scheduler logs such as the following:

  ```text wrap theme={null}
  [2022-03-30, 12:17:39 UTC] {great_expectations_extractor.py:17} INFO - Did not find great_expectations_provider library or failed to import it
  [2022-03-24, 23:40:01 UTC] {client.py:74} INFO - Constructing openlineage client to send events to https://api.astro-astronomer.datakin.com
  ```

  A few additional notes about this upgrade:

  * You can ignore any lineage logs that indicate an error or failed process, such as the first line in the example logs above. These logs will more accurately reflect the state of your lineage functionality once lineage features are launched on Astro.
  * Deployments on Runtime 4.2.0+ will be updated to emit data lineage events only after you [push code](/docs/astro/deploy-code). Until you do so, this change will not be applied.
  * Because Astronomer is upgrading each customer individually over time, the exact date that you will start seeing these logs will vary.
  * When you push code to a Deployment on Runtime 4.2.0+ and trigger this update, all other Deployments on Runtime 4.2.0+ in the same Workspace will also restart in order to receive the lineage backend update. If you plan to push code to any Deployment affected by this change, then we recommend doing so at a time where you can tolerate some Airflow components restarting. For more information about expected behavior, see [What Happens During a Code Deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy).

  For more information about what to expect when lineage tools go live, read Astronomer's [OpenLineage and Airflow guide](/docs/learn/airflow-openlineage).

  ### New AWS regions available

  You can now [create new Clusters](/docs/astro/manage-hybrid-clusters#create-a-cluster) in:

  * `af-south-1` (Cape Town)
  * `ap-east-1` (Hong Kong)
  * `ap-northeast-3` (Osaka)
  * `me-south-1` (Bahrain)

  ### Additional improvements

  * The Astro UI now includes a button that links to Astronomer [support](https://support.astronomer.io/) and [status](https://status.astronomer.io/) pages.
</Update>

<Update label="March 25, 2022">
  ### Maximum node count is now configurable per Cluster

  As of this release, **Maximum Node Count** is now a configurable setting for new and existing clusters. On Astro, maximum node count represents the total number of EC2 nodes that your cluster can support at any given time. For an Astro cluster on AWS, EC2 nodes are the primary unit of infrastructure required to run a Deployment and its components, including workers and the Airflow scheduler. New clusters have a maximum node count of 20 by default, but the setting can be modified to any value from 2 to 100 at any time.

  Previously, maximum node count was a fixed, global setting that applied to all customers on Astro and could not be configured per cluster. Now, your organization can modify this setting as your workloads evolve and more Deployments are created. Once the limit is reached, your cluster will not be able to auto-scale and worker pods may fail to schedule.

  To update this setting for an existing cluster, reach out to [Astronomer support](https://cloud.astronomer.io/open-support-request) and provide the name of your cluster and the desired maximum node count.

  ### Additional improvements

  * In **Resource Settings**, the maximum allowed value for **Worker Resources** has been increased to 400 AU.
</Update>

<Update label="March 17, 2022">
  ### Export task usage as a CSV file

  In the Astro UI, you can now export your task usage data from the **Usage** tab as a CSV file to perform more complex data analysis related to your Airflow usage and costs. For example, you can use the file as the basis for a pivot table that shows total task usage by Workspace.

  To export your task usage data as a CSV file, click the **Export** button in the **Usage** tab.

  ### Bug fixes

  * Fixed an issue where saving new environment variables in the Astro UI would occasionally fail
</Update>

<Update label="March 10, 2022">
  ### Running Docker image tag in Airflow UI

  The Docker image that is running on the Airflow webserver of your Deployment is now shown as a tag in the footer of the Airflow UI. Depending on how your team deploys to Astro, this tag is either a unique identifier generated by a CI tool or a timestamp generated by the Astro CLI on `astro deploy`. Both represent a unique version of your Astro project.

  <Frame>
    <img alt="Runtime Tag banner" />
  </Frame>

  When you push code to a Deployment on Astro via the Astro CLI or CI/CD, reference this tag in the Airflow UI to verify that your changes were successfully applied.

  To upgrade a Deployment to the latest Runtime version, see [Upgrade Runtime](/docs/runtime/upgrade-astro-runtime).

  <Info>
    While it is a good proxy, the tag shown in the Airflow UI does not forcibly represent the Docker image that is running on your Deployment's scheduler, triggerer, or workers.

    This value is also distinct from the **Docker Image** that is shown in the Deployment view of the Astro UI, which displays the image tag as specified in the Cloud API request that is triggered on `astro deploy`. The image tag in the Airflow UI can be interpreted to be a more accurate proxy to what is running on all components of your Deployment.

    If you ever have trouble verifying a code push to a Deployment on Astro, reach out to [Astronomer support](https://cloud.astronomer.io/open-support-request).
  </Info>
</Update>

<Update label="March 3, 2022">
  ### Additional improvements

  * The threshold in order for bars in the **Worker CPU** and **Worker Memory** charts to appear red has been reduced from 95% to 90%. This is to make sure that you get an earlier warning if your workers are close to hitting their resource limits.

  ### Bug fixes

  * Fixed an issue where malformed URLs prevented users from accessing the Airflow UI of some Deployments on Astro
  * Fixed an issue where Astro Runtime 4.0.11 wasn't a selectable version in the **Astro Runtime** menu of the Deployment creation view in the Astro UI
</Update>

<Update label="February 24, 2022">
  ### Bug fixes

  * Removed the **Teams** tab from the Astro UI. This view was not yet functional but coming back soon
  * Fixed an issue where the number of users per Workspace displayed in the Organization view of the Astro UI was incorrect
  * Fixed an issue where if a secret environment value was updated in the Astro UI and no other values were modified, the change was not applied to the Deployment
</Update>

<Update label="February 17, 2022">
  ### Introducing Astro and a new look

  This week's release introduces a reimagined Astronomer brand that embraces **Astro** as a rename of Astronomer Cloud. The rebrand includes a new Astronomer logo, color palette, and font.

  The new Astronomer brand is now reflected both in the [Astro UI](https://cloud.astronomer.io) as well as in the main [Astronomer website](https://astronomer.io) and [documentation](https://www.astronomer.io/docs).

  In addition to visual changes, we've renamed the following high-level Astro components:

  * **Astronomer Cloud CLI** is now **Astro CLI**
  * **Astronomer UI** is now **Cloud UI**
  * **Astro Runtime** is now **Astro Runtime**

  We hope you find this exciting. We're thrilled.

  ### New Organization roles for users

  The following Organization-level roles are now supported on Astro:

  * **Organization Member**: This role can view Organization details and membership. This includes everything in the **People**, **Clusters**, and **Settings** page of the Astro UI. Organization members can create new Workspaces and invite new users to an Organization.
  * **Organization Billing Admin:** This role has all of the Organization Member's permissions, plus the ability to manage Organization-level settings and billing. Organization Billing Admins can access the **Usage** tab of the Astro UI and view all Workspaces across the Organization.
  * **Organization Owner:** This role has all of the Organization Billing Admin's permissions, plus the ability to manage and modify anything within the entire Organization. This includes Deployments, Workspaces, Clusters, and users. Organization Owners have Workspace Admin permissions to all Workspaces within the Organization.

  Organization roles can be updated by an Organization Owner in the **People** tab of the Astro UI. For more information about these roles, see [User permissions](/docs/astro/user-permissions).

  ### Create new Workspaces from the Astro UI

  All users can now create a new Workspace directly from the **Overview** tab of the Astro UI:

  When you create a new Workspace, you will automatically become a Workspace Admin within it and can create Deployments. For more information about managing Workspaces, see [Manage Workspaces](/docs/astro/manage-workspaces).

  ### Bug fixes

  * Fixed an issue where authentication tokens to Astro weren't properly applied when accessing the Airflow UI for a Deployment. This would result in an authenticated user seeing `Error: Cannot find this astro cloud user` in the Airflow UI.
  * Fixed an issue where long environment variable values would spill out of the **Value** column and onto the **Updated** column in the **Environment Variables** view of a Deployment in the Astro UI.
</Update>

<Update label="February 11, 2022">
  ### Monitor dag runs across all Deployments in a Workspace

  You can view key metrics about recent dag runs through the new **DAGs** page in the Astro UI. Use this page to view dag runs at a glance, including successes and failures, across all Deployments in a given Workspace. You can also drill down to a specific dag and see metrics about its recent runs.

  For more information about the **DAGs** page, see [Deployment metrics](/docs/astro/deployment-metrics#dag-and-task-runs).

  ### Additional improvements

  * All resource settings in the Deployment view of the Astronomer UI now show exact CPU and Memory usage to the right of every slider, previously shown only in Astronomer Units (AUs). This makes it easy to know exactly how many resources you allocate to each component.
  * A banner now appears in the Astronomer UI if a Deployment is running a version of Astro Runtime that is no longer maintained. To make the most of features and bug fixes, we encourage users to upgrade to recent versions as much as possible.
  * Added more ways to sort pages that utilize card views, such as the **Deployments** page
  * Added user account avatars next to usernames in several places across the Astro UI

  ### Bug fixes

  * Removed the **Environment** field from the Deployment view of the Astronomer UI. This field is not currently functional and will be re-added as soon as it is.
</Update>

<Update label="February 3, 2022">
  ### Support for third-party identity providers

  You can now integrate both Azure AD and Okta as identity providers (IdPs) for federated authentication on Astro. By setting up a third-party identity provider, a user in your organization will be automatically logged in to Astro if they're already logged in via your IdP. By adding new Astro users through your IdP's own user management system, Workspace Admins can automatically add new users to their Workspace without those users needing to individually sign up for Astro.

  For more information about this feature read [Set up an identity provider](/docs/astro/configure-idp).

  ### Support for the Astro CLI

  The Astro CLI (`astro`) is now generally available as the official command-line tool for Astro. It is a direct replacement of the previously released `astro` executable and comes with various significant improvements. We encourage all customers to upgrade.

  For more information on the Astro CLI, see [CLI Release Notes](/docs/cli/v1.43/release-notes). For install instructions, read [Install the CLI](/docs/cli/v1.43/install-cli).

  ### Multiple authentication methods for a single user account

  Astro now supports multiple authentication methods for a single user account. This means that as long as you're using a consistent email address, you now have the flexibility to authenticate with GitHub, Google, username/password, and/or [an external identity provider (idP)](/docs/astro/configure-idp) at any time. Previously, a single user account could only be associated with one authentication method, which could not be changed after the account was created.

  This also means that all Organizations now have GitHub, Google, and username/password authentication methods enabled by default for all users.

  ### Additional improvements

  * Changed the default RDS instance type for new clusters from `db.r5.xlarge` to `db.r5.large`, which represents a monthly cost reduction of \~50% for newly provisioned clusters. Customers with existing clusters will need to request a downscale via [Astronomer support](https://cloud.astronomer.io/open-support-request)
</Update>

<Update label="January 13, 2022">
  ### Identity-based login flow

  Astro now utilizes an identity-based login flow for all users. When you first log in via the Astro UI, you now only need to enter the email address for your account. Astro assumes your Organization and brings you directly to your Astro Organization's login screen.

  This change serves as a foundation for future SSO and authentication features. In upcoming releases, users will be able to authenticate via custom identity providers like Okta and Azure Active Directory.

  ### Additional improvements

  * Significant improvements to the load times of various Astro UI pages and elements.
  * In the Astro UI, the tooltips in the **Resource Settings** section of a Deployment's page now show the definition of 1 AU. This should make it easier to translate AU to CPU and Memory.
  * Scheduler logs in the Astro UI no longer show `DEBUG`-level logs.
  * To ensure that all workers have enough resources to run basic workloads, you can no longer allocate less than 10 AU to **Worker Resources**.
</Update>

<Update label="January 6, 2022">
  ### Improvements to "Scheduler Logs" in the Astro UI

  The **Scheduler Logs** tab in the Astro UI has been updated to make logs easier to read, separate, and parse. Specifically:

  * You can now filter logs by type (`DEBUG`, `INFO`, `WARN`, and `ERROR`).
  * The page now shows logs for the past 24 hours instead of the past 30 minutes.
  * The page now shows a maximum of 500 logs instead of a lower maximum.
  * When looking at a Deployment's logs, you can return to the Deployment's information using the **Deployment Details** button.

  ### Removal of worker termination grace period

  The **Worker Termination Grace Period** setting is no longer available in the Astro UI or API. Previously, users could set this to anywhere between 1 minute and 24 hours per Deployment. This was to prevent running tasks from being interrupted by a code push. Today, however, existing Celery workers don't have to terminate in order for new workers to spin up and start executing tasks. Instead, existing workers will continue to execute running tasks while a new set of workers gets spun up concurrently to start executing the most recent code.

  To simplify Deployment configuration and reflect current functionality:

  * The worker Termination Grace Period was removed from the Astro UI
  * This value was permanently set to 24 hours for all Deployments on Astro

  This does not change or affect execution behavior for new or existing Deployments. For more information, read [What Happens During a Code Deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy).

  ### Additional improvements

  * Removed *Kubernetes Version* column from the **Clusters** table. This value was previously inaccurate and is not needed. The Kubernetes version of any particular Astro cluster is set and modified exclusively by Astro as part of our managed service.
</Update>


# 2023 Astro release notes
Source: https://astronomer.io/docs/astro/release-notes-2023

Astro release notes from 2023, covering features, bug fixes, and version updates for Astro, the Astro CLI, Astro Runtime, and the Remote Execution Agent.

<Tip>[Subscribe to Astro release notes](/docs/astro/release-notes-subscribe) to receive updates via RSS, email, or Slack.</Tip>

Astro release notes from 2023. See the [current release notes](/docs/astro/release-notes) for the latest updates.

<Update label="December 20, 2023">
  ### Bug fixes

  * Fixed an issue where creating an alert for a dag through the Astro API would apply the alert to the incorrect Deployment.
  * Fixed an issue where Deployment worker Pods could crash when running the Kubernetes executor.
  * Fixed an issue where Deployment API key expiration dates were not applied correctly if you configured multiple API keys at once.
  * Fixed an issue where logging features could be disrupted if you set `AZURE_CLIENT_ID `as an environment variable. Note that this fix applies only to Astro Runtime 10 and later.
</Update>

<Update label="December 12, 2023">
  ### Bug fixes

  * Fixed an issue where the Astro UI would produce an error if you updated an environment variable on an Astro Hybrid Deployment running the Kubernetes Executor.
</Update>

<Update label="December 6, 2023">
  ### Additional improvements

  * The [Astro Environment Manager](/docs/astro/create-and-link-connections) is now generally available. This feature allows you to create and manage Airflow connections in the Astro UI.

  ### Bug fixes

  * Fixed an issue where dag code that appeared in the Airflow UI did not roll back when you rolled back a Deployment, even though the running code was successfully rolled back.
  * Fixed an issue where you could not view billing information from the Astro UI when you installed Astro through the Azure Marketplace.
  * Fixed an issue where the Astro UI would produce a console error when a user accessed their Workspace list.
</Update>

<Update label="November 30, 2023">
  ### Support for Microsoft Entra Workload ID

  You can now use [Microsoft Entra Workload ID](https://www.microsoft.com/en-us/security/business/identity-access/microsoft-entra-workload-id) to authorize Deployments to resources in Azure. Workload identity is a simple and secure way to authorize access external resources, as it doesn't require creating or storing long-term credentials. To set up Microsoft Entra Workload ID, see [Authorize Deployments to cloud resources](/docs/astro/authorize-deployments-to-your-cloud?tab=azure#setup).

  ### Bug fixes

  * Removed the ability to create Hybrid Azure clusters in `eastasia` because some workload identity features aren't supported in this region.
</Update>

<Update label="November 16, 2023">
  ### Install Astro from the Azure Marketplace

  Astro is now available as an Azure Native ISV Service. If your team is considering Astro and you use Azure, Astronomer recommends installing Astro from the Azure Marketplace because:

  * You can manage billing from the Azure Portal.
  * Microsoft Entra ID is pre-configured for all Organizations.
  * It's easier to create Astro resources and get started directly from Azure.

  ### Create Airflow connections in the Astro UI and link them to Deployments

  You can now create Airflow connections in the Astro UI through the new Environment Manager menu. The Environment Manager lets you create Airflow connections directly in the Astro UI and stores all connections in an Astro-managed secrets backend. You can then share connections between Deployments and set default connections so that your team members always have access to external resources when they create new Deployments. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).

  Note that this feature is currently available only for Deployments running the Celery executor.

  ### Trigger a dag from an Astro alert

  You can now configure [Astro alerts](/docs/astro/alerts) to trigger any dag in your Workspace through the Airflow REST API. You can configure the triggered dag to complete any action, such as sending an alert through a custom communication channel or writing data about the incident to a table.

  ### New Azure regions available on Astro Hosted

  You can now create Hosted dedicated clusters in the following Azure regions:

  * `eastus`
  * `canadacentral`
  * `uksouth`
  * `brazilsouth`
  * `centralindia`
  * `francecentral`
  * `japaneast`

  See [Astro Hosted resource reference](/docs/astro/resource-reference-hosted) for more information.
</Update>

<Update label="November 7, 2023">
  ### Roll back Deployments to previous versions of your code

  <Warning>This feature is in [Preview](/docs/astro/feature-previews).</Warning>

  Astro now maintains snapshots of your past deploys, including your Deployment image and dag code, for the previous three months. If you need to quickly revert a Deployment back to a working version of your code, you can roll back to a past deploy from the **Deploy History** page in the Astro UI.

  Deploy rollbacks are a powerful safety mechanism to ensure that your production pipelines continue to run when something unexpected happens after a deploy. See [Roll back to a past deploy](/docs/astro/deploy-history#roll-back-to-a-past-deploy) for more information and configuration steps.

  ### Bug fixes

  * Fixed an issue where, you could inadvertently open the support request window if you opened a dag that included "Support" in its name from the **DAGs** view. As a result of this change, the support request window URL has been updated from `https://cloud.astronomer.io/support` to `https://cloud.astronomer.io/open-support-request`.
  * Fixed an issue where the Deployment configuration menu in the Astro UI didn't always show your Deployment's current configuration.
  * Fixed an issue where you could not create a Deployment with some stable Runtime versions using the Astro Platform API.
</Update>

<Update label="October 31, 2023">
  ### Deployment API keys are now deprecated

  Deployment API keys have been officially deprecated in favor of [Deployment API tokens](/docs/astro/deployment-api-tokens). This means:

  * You can't create new Deployment API keys.
  * If a Deployment has no configured Deployment API keys, the **API keys** tab will not appear.
  * You can continue using existing Deployment API keys until a future end-of-support date.

  ### Additional improvements

  * Workspace Operators can now create, update, and delete Deployment API tokens.

  ### Bug fixes

  * Fixed an issue where a Deployment's **Updated By** field was not updated if you transferred the Deployment.
</Update>

<Update label="October 24, 2023">
  ### New Azure regions available on Astro Hosted

  You can now create Deployments in standard clusters in the following Azure regions:

  * `eastus2`
  * `westus2`
  * `westeurope`

  See [Astro Hosted resource reference](/docs/astro/resource-reference-hosted) for more information.

  ### Bug fixes

  * Fixed an issue where Deployment API tokens weren't deleted after their associated Deployment was deleted.
  * Fixed an issue where you could not create a Deployment with some available Runtime versions using the Astro Platform API.
</Update>

<Update label="October 17, 2023">
  ### Additional improvements

  * You can now view deploy history for both Hosted and Hybrid Astro Deployments in the Astro UI. For more information, see [View deploy history](/docs/astro/deploy-history).

  ### Bug fixes

  * Modified behavior for Astro Hosted so that KubernetesExecutor and KubernetesPodOperator pods running in-cluster have equivalent resource requests and limits. If you don't configure them to have equivalent resource requests and limits, Astro modifies them to the become the limits. Previously, the dag deploy would fail if `resources` did not equal `limits`.
</Update>

<Update label="October 10, 2023">
  ### Deployment API tokens

  Deployment API tokens are now generally available and replace Deployment API keys as the most secure and customizable way to programmatically update Deployments. This includes using them to [deploy code](/docs/astro/deploy-code) and update [environment variables](/docs/astro/environment-variables).

  See [Deployment API tokens](/docs/astro/deployment-api-tokens) to learn how to create and manage Deployment API tokens.

  <Warning>
    Deployment API tokens are a direct replacement for Deployment API keys, which are now supported only on a limited basis on Astro.

    After October 31, 2023, you will not be able to create new API keys. While you can still continue to use and manage existing Deployment API keys, Astronomer will soon require using Deployment API tokens.
  </Warning>

  ### New Edit Deployments in the Astro UI

  Editing Deployments in the Astro UI has a new, consolidated flow. All of the configuration options are now editable in a single form, similar to Deployment creation, instead of spread across multiple forms. See [Deployment Settings](/docs/astro/deployment-settings) for a detailed description of how to create, update, and configure your Deployment options.
</Update>

<Update label="October 3, 2023">
  ### Additional Improvements

  * Added a **DAG Success** alert so you can now set up an alert for successful completion events. See how to set up [Astro alerts](/docs/astro/alerts).

  ### Bug Fixes

  * Fixed a problem in the Astro UI where a warning about Deployment Health was displayed when a Workspace had zero Deployments.
</Update>

<Update label="September 26, 2023">
  ### Introducing the Astro API

  <Info>The Astro API is currently in beta. See [Astro API versioning and support](/docs/astro/api/v-1/versioning-and-support).</Info>

  You can now use the [Astro API](/docs/astro/api/v-1/overview) to create applications and scripts to programmatically interact with Astro. The Astro API is a standard REST API that includes endpoints for interacting with all key resources and components on Astro.

  Using the Astro API, you can create robust and secure applications for managing Deployment resources, updating user permissions, and performing many other key Astro operations. To make your first API call, see [Get started with the Astro API](/docs/astro/api/v-1/get-started).
</Update>

<Update label="September 19, 2023">
  ### Manage Deployments programmatically using Deployment API tokens

  Deployment API tokens replace Deployment API keys as the most secure and customizable way to manage Deployments programmatically. You can use Deployment API tokens to perform all of the same actions as a Deployment API key, including:

  * [Pushing code](/docs/astro/deploy-code) to a Deployment.
  * Updating a Deployment's [environment variables](/docs/astro/environment-variables).
  * Making requests to update your Deployment's Airflow environment using the [Airflow REST API](/docs/astro/airflow-api).

  Unlike Deployment API keys, you can set an expiration date for Deployment API tokens and rotate them to better manage access to your Deployment. See [Deployment API tokens](/docs/astro/deployment-api-tokens) to learn how to create and manage Deployment API tokens.

  <Warning>
    Deployment API tokens are a direct replacement for Deployment API keys. Therefore, Astronomer recommends always using Deployment API tokens over API keys. While you can still continue to use and manage existing Deployment API keys, Astronomer will soon require using Deployment API tokens.

    After API tokens are generally available, Deployments with zero API keys will not show the **API Keys** tab and you will no longer be able to create Deployment API keys. If you want to continue using API keys, ensure that you always have at least one API key configured for the Deployment.
  </Warning>

  ### Additional improvements

  * When you create a new Deployment, the Astro UI now presents new options and suggestions for running your first dag.
  * You can now retrieve a Workspace's ID from the Astro UI. To find a Workspace's ID, open the Workspace in the Astro UI and go to **Workspace Settings** > **General**.
</Update>

<Update label="September 12, 2023">
  ### Per-Deployment IAM workload identities on AWS

  Astro Hybrid clusters on AWS now support per-Deployment IAM workload identities, meaning that you can now limit your trust policies to authorize only specific Deployments to your cloud resources.

  <Info>
    This change required an automatic update to the cross-account role that Astro uses to manage clusters in your cloud. In addition to enabling per-Deployment IAM workload identities, this update also adds the following permissions to reduce the risk of partial deletions in your cloud:

    ```text wrap theme={null}
    { "elasticloadbalancing:DescribeLoadBalancers", "elasticloadbalancing:DeleteLoadBalancer" }
    ```

    For more information about this change, see [Automatic updates coming to cross-account roles for Astro Hybrid on AWS](https://support.astronomer.io/hc/en-us/articles/19833616584723).
  </Info>

  To migrate from using cluster workload identities to Deployment workload identities:

  1. In the AWS Management Console, go to the **Identity and Access Management (IAM) dashboard**. Identify all of your trust policies that specify your cluster workload identity. They should look similar to the following trust policy:

     ```json wrap theme={null}
     {
         "Version": "2012-10-17",
         "Statement": [
             {
                 "Effect": "Allow",
                 "Principal": {
                     "AWS": [
                         "arn:aws:iam::<dataplane-AWS-account-ID>:role/AirflowS3Logs-<cluster-ID>"
                     ]
                 },
                 "Action": "sts:AssumeRole"
             }
         ]
     }
     ```

  2. For each trust policy, add the workload identities for any Deployments that you want to access the related resource. To locate your Deployment workload identity, open the Deployment in the Astro UI and copy the **Workload Identity** from the **Details** page. Your trust policy should now look like the following:

     ```json wrap theme={null}
     {
         "Version": "2012-10-17",
         "Statement": [
             {
                 "Effect": "Allow",
                 "Principal": {
                     "AWS": [
                         "arn:aws:iam::123456789876:role/AirflowS3Logs-cl6zcnlc641hr0voibivf21jh",
                         "arn:aws:iam::<dataplane-AWS-account-ID>:role/astro-<namespace>"
                     ]
                 },
                 "Action": "sts:AssumeRole"
             }
         ]
     }
     ```

  3. For each Deployment that you specified in your trust policies, open the Deployment in the Astro UI and click **Details**, then click **Edit Details**. In the **Workload Identity** section, select the new Deployment identity from the dropdown list, then click **Update**. To avoid disruption to tasks, don't complete this step until you have added the Deployment workload identity to all of the trust policies it needs for access.

  4. [Upgrade](/docs/cli/v1.43/upgrade-cli) to the latest Astro CLI release, which includes support for Per Deployment IAM Workload Identity.

  5. After you've tested the policies with your Deployment workload identities, remove the cluster workload identity from your trust policies

  ### Bug fixes

  * Fixed an issue where Billing Admins could view task usage on the **Usage** page only for Workspaces that they belonged to. Now, Billing Admins can view usage for all Workspaces regardless of their Workspace role.

  ### Additional improvements

  * The Astro UI **Usage** page now shows task usage for deleted Deployments. If you're an Astro Hybrid Billing Admin, this means that task usage metrics now better reflect your billable usage.
  * When you create a Deployment through the Astro UI and choose an Astro Runtime version, you can now select only the most recent supported patch for each major version of Astro Runtime.
  * You can now filter task logs by log level or source from the [**DAGs** page](/docs/astro/manage-dags) in the Astro UI.
</Update>

<Update label="September 6, 2023">
  ### View deploy history in the Astro UI

  When you view a Deployment in the Astro UI, you can now open the **Deploy History** tab to view a table of all code deploys. The table shows who made deploys, when they made the deploys, and what Astro Runtime image they used for the deploy.

  You can also now use the Astro CLI to specify an optional description for your deploys using the `--description` flag. Deploy descriptions appear in the **Deploy History** table and are useful for telling other Workspace members why you made a deploy or what changes it contains. For more information, see [View deploy history](/docs/astro/deploy-history).

  ### View rendered data figures in the Astro Cloud IDE

  Python cells that generate figures using [Matplotlib](https://matplotlib.org/), [Plotly](https://plotly.com/graphing-libraries/), or any libraries that extend these tools now show the figures they render in the Astro Cloud IDE. When you run a Python cell, any figures it generates appear in a new **Figures** tab.

  <Frame>
    <img alt="A Python cell that renders a chart with plotly, and the rendered chart in the Figures tab" />
  </Frame>
</Update>

<Update label="August 29, 2023">
  ### Changes to Workspace user roles

  <Info>
    **Upgrade the Astro CLI**

    To take advantage of these new user roles programmatically, you must [upgrade the Astro CLI](/docs/cli/v1.43/upgrade-cli) to version 1.19 or later.
  </Info>

  To increase granularity and better serve each user persona on Astro, Workspace roles have been updated with new names and permissions:

  * The **Workspace Author** role is a new role for users who primarily write and deploy dags. Users with this role can push code changes, but they can't update Deployment or Airflow settings such as Airflow variables, Astro environment variables, or connections.
  * The Workspace Admin role has been renamed to **Workspace Owner**. Users with this role are responsible for administrating membership to the Workspace.
  * The Workspace Editor role has been renamed to **Workspace Operator**. In addition to pushing code changes, Workspace Editors can now also edit Airflow objects such as variables, connections, and XComs. Users with this role are responsible for managing the environments that dags run in.
  * The Workspace Viewer role has been renamed to **Workspace Member**. Users with this role only need viewing permissions for a Deployment and don't have permissions to make any code or configuration changes.

  For more information about these role changes, see [User permissions reference](/docs/astro/user-permissions#workspace-roles) and [Enhanced Astro Workspace Roles for more granular permissions](https://www.astronomer.io/blog/introducing-updated-astro-workspace-roles-for-more-granular-permissions/)

  ### Additional improvements

  * The lifespan of the personal user access token you can retrieve from `cloud.astronomer.io/token` has been reduced from 24 hours to 1 hour.
  * The **DAGs** view of the Astro UI now shows your configured dependency edge labels in the graph view.
  * The Astro UI now shows more detailed instructions for deploying code when you create a new Deployment.
  * The Deployment **Analytics** page in the Astro UI has been renamed to **Overview**.

  ### Bug fixes

  * Fixed an issue where Deployments using the Kubernetes executor could not run dags with lower resource requests than the **Default Pod Size**. Minimum requests are now hard-coded and decoupled from default requests.
</Update>

<Update label="August 21, 2023">
  ### Additional improvements

  * You can now configure [task log forwarding to Datadog](/docs/astro/export-datadog) at the Deployment level.
  * In the **DAGs** view of the Astro UI, you can now double click a task run node in the graph view to view the task run's logs and mapped tasks.
  * The A50 worker type has been renamed to A60 to make it consistent in scale with other worker types.
  * The max possible **CPU quota** and **Memory quota** for a Deployment running in a Hosted dedicated cluster has increased to 1600 vCPU/ 3200 GiB respectively.
</Update>

<Update label="August 15, 2023">
  ### Additional improvements

  * You can now see how many Astro alerts you've configured for a dag in the **DAGs** page of the Astro UI.

  ### Bug fixes

  * Fixed an issue where you couldn't run Astro Cloud IDE pipelines that included a Markdown cell.
  * Fixed an issue where an Organization's SSO bypass link was formatted incorrectly in the Astro UI.
</Update>

<Update label="August 8, 2023">
  ### New Hosted worker type

  You can now configure Deployments with the `A50` machine type, which has 12 vCPU and 24 GiB. See [Astro hosted resource reference](/docs/astro/resource-reference-hosted).

  ### Additional improvements

  * When you create a new Astro Cloud IDE project, you can now specify whether you want the project to include an example pipeline.
  * You can now access Organization-level settings in the Astro UI only through the **Organization Settings** link. Additionally, some Organization settings have been moved to the top level of navigation so that there is no longer a **Settings** menu.
  * You can now use commas, apostrophes, and ampersands in Workspace and Organization names.
  * The Workspace list view in the Astro UI has been redesigned so that Organization Owners can now edit and delete Workspaces directly from the list.

  ### Bug fixes

  * Fixed an issue where the Astro UI showed incorrect CPU and memory limits in bar charts on the Deployments list and **Details** page.
</Update>

<Update label="August 1, 2023">
  ### Hosted Deployments have dag-only deploys enabled by default

  New Astro Hosted Deployments now have [dag-only deploys](/docs/astro/deploy-dags) enabled by default. When dag-only deploys are enabled, some workflows for your Deployment, including image-based deploys, are different compared to when dag-only deploys are disabled. For more information about how code and image deploys work when dag-only deploys are enabled, see [What happens during a project deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy). To disable dag-only deploys, see [Enable/ disable dag-only deploys on a Deployment](/docs/astro/deploy-dags#enable-or-disable-dag-only-deploys-on-a-deployment).

  ### Teams now have Organization-level roles

  All Teams on Astro now have an Organization role. Existing Teams have been given the [Organization Member](/docs/astro/user-permissions#organization-roles) role, which doesn't result in any additional automatic permissions.

  Coupled with [SCIM user groups](/docs/astro/set-up-scim-provisioning), you can now manage your Organization Owners and Billing Admins from your identity provider. See [Manage teams](/docs/astro/manage-teams) for more information.

  ### Additional improvements

  * The Astro UI now shows how many Workspaces, dags, clusters, and Astro Cloud IDE projects you have in the left sidebar.
  * You can now create Deployments in standard clusters hosted in `europe-west2` on GCP and `eu-central-1` on AWS.
  * The default metadata database instance type for new Deployments on GCP clusters in Astro Hybrid has been reduced to `Small General Purpose` with 2 vCPUs and 8GiB. See [GCP Hybrid cluster settings](/docs/astro/resource-reference-gcp-hybrid).

  ### Bug fixes

  * Because some regions don't support specific machine types that Astro Hosted uses, you can no longer create Hosted dedicated clusters in the following AWS regions:

    * `af-south-1`
    * `ap-east-1`
    * `ap-northeast-2`
    * `ap-northeast-3`
    * `ca-central-1`
    * `eu-south-1`
    * `eu-west-3`
    * `me-south-1`
</Update>

<Update label="July 25, 2023">
  ### Additional improvements

  * The templates for [Astro alert](/docs/astro/alerts) messages have been updated to include more information about the Deployment that the alert was triggered in, including a link to the dag that triggered the alert.

  ### Bug fixes

  * Fixed an issue where Azure AD single sign-on (SSO) connections were incorrectly labeled as SAML connections in the Astro UI.
</Update>

<Update label="July 18, 2023">
  ### Configure default Pod sizes

  You can now configure the default minimum CPU and memory for tasks that you run with the Kubernetes executor or KubernetesPodOperator. If you don't specify CPU or memory in a task definition, Astro runs the task in a Pod that uses your default resource configurations. Configure default minimum resources to ensure that tasks always have enough CPU and memory to run successfully. See [Deployment settings](/docs/astro/deployment-settings#deployment-resources).

  ### New regions available on Astro Hosted

  You can now create Hosted dedicated clusters in the following regions:

  * AWS
    * `af-south-1` - Africa (Cape Town)
    * `ap-east-1` - Asia Pacific (Hong Kong)
    * `ap-northeast-1` - Asia Pacific (Tokyo)
    * `ap-northeast-2` - Asia Pacific (Seoul)
    * `ap-northeast-3` - Asia Pacific (Osaka)
    * `ap-southeast-1` - Asia Pacific (Singapore)
    * `ap-southeast-2` - Asia Pacific (Sydney)
    * `ap-south-1` - Asia Pacific (Mumbai)
    * `ca-central-1` - Canada (Central)
    * `eu-central-1` - Europe (Frankfurt)
    * `eu-south-1` - Europe (Milan)
    * `eu-west-1` - Europe (Ireland)
    * `eu-west-2` - Europe (London)
    * `eu-west-3` - Europe (Paris)
    * `me-south-1` - Middle East (Bahrain)
    * `sa-east-1` - South America (São Paulo)
    * `us-east-1` - US East (N. Virginia)
    * `us-east-2` - US East (Ohio)
    * `us-west-1` - US West (N. California)
    * `us-west-2` - US West (Oregon)

  * GCP
    * `asia-east1` - Taiwan, Asia
    * `asia-northeast1` - Tokyo, Asia
    * `asia-northeast2` - Osaka, Asia
    * `asia-northeast3` - Seoul, Asia
    * `asia-south1` - Mumbai, Asia
    * `asia-south2` - Delhi, Asia
    * `asia-southeast1` - Singapore, Asia
    * `asia-southeast2` - Jakarta, Asia
    * `australia-southeast1` - Sydney, Australia
    * `australia-southeast2` - Melbourne, Australia
    * `europe-central2` - Warsaw, Europe
    * `europe-north1` - Finland, Europe
    * `europe-southwest1` - Madrid, Europe
    * `europe-west1` - Belgium, Europe
    * `europe-west2` - England, Europe
    * `europe-west3` - Frankfurt, Europe
    * `europe-west4` - Netherlands, Europe
    * `europe-west6` - Zurich, Europe
    * `europe-west8` - Milan, Europe
    * `europe-west9` - Paris, Europe
    * `northamerica-northeast1` - Montreal, North America
    * `northamerica-northeast2` - Toronto, North America
    * `southamerica-east1` - São Paulo, South America
    * `southamerica-west1` - Santiago, South America
    * `us-central1` - Iowa, North America
    * `us-east1` - South Carolina, North America
    * `us-east4` - Virginia, North America
    * `us-east5` - Columbus, North America
    * `us-south1` - Dallas, North America
    * `us-west1` - Oregon, North America
    * `us-west2` - Los Angeles, North America
    * `us-west3` - Salt Lake City, North America
    * `us-west4` - Nevada, North America

  See [Astro Hosted resource reference](/docs/astro/resource-reference-hosted#dedicated-cluster-regions) for all available configurations.

  ### Bug fixes

  * You can no longer create Hybrid GCP clusters in `eu-north-1`.
</Update>

<Update label="July 11, 2023">
  ### Send Astro alerts to email

  You can now send Astro alerts to multiple email addresses. Sending Astro alerts to email requires no configuration outside of Astro, which makes it a quick option to improve your alerting infrastructure. See [Configure Astro alerts](/docs/astro/alerts?tab=Email#step-1-configure-your-notification-channel) for setup steps.

  ### Configure SCIM provisioning for Azure AD

  If your Organization uses Azure for single sign-on (SSO), you can now set up SCIM provisioning for Astro. SCIM provisioning simplifies user management by allowing you to add and remove Astro users from Okta based on your existing user groups. See [Set up SCIM provisioning](/docs/astro/set-up-scim-provisioning?tab=Azure#setup) for more information.

  ### Additional improvements

  * On Astro Hosted deployments, the `astronomer_monitoring_dag` has been paused for all image-based Deployments and removed entirely from all Deployments with dag deploys enabled. It has been replaced with an implementation that allows workers on Deployments to fully scale to 0.
</Update>

<Update label="July 5, 2023">
  ### Configure SCIM provisioning for Okta

  If your Organization uses Okta for single sign-on (SSO), you can now set up SCIM provisioning for Astro. SCIM provisioning simplifies user management by allowing you to add and remove Astro users from Okta based on your existing user groups. See [Set up SCIM provisioning](/docs/astro/set-up-scim-provisioning?tab=Okta#setup) for more information.

  ### See pricing estimate when creating a Deployment

  The Deployment creation page in the Astro UI has been reorganized to make it easier to focus on specific configurations for your Deployment. Each configuration is now collapsible and includes guidance for different environment sizes. Additionally, the page now shows cost estimates for a Deployment before you create it.

  <Frame>
    <img alt="Deployment creation screen with new pricing information" />
  </Frame>

  ### Additional improvements

  * You can now configure Deployments with the `A40` machine type, which has 8 vCPU and 16 GiB. See [Astro hosted resource reference](/docs/astro/resource-reference-hosted).
  * You can now access your Organization settings from a Workspace by clicking the name of your Organization/ Workspace.
  * On Astro Hybrid, the default Azure DB instance is now `Standard D2ds_v4`.
  * The Deployment creation screen for Astro Hybrid has received several updates that were previously only available on Astro Hosted.

  ### Bug fixes

  * Fixed an issue where dags that used the KubernetesPodOperator and had tasks with `in_cluster=False` could not be parsed.
</Update>

<Update label="June 27, 2023">
  ### Support for dedicated clusters on Azure

  You can now create a dedicated cluster in the following Azure regions:

  * `australiaeast`
  * `eastus2`
  * `northeurope`
  * `westeurope`
  * `uswest2`

  See [Astro Hosted resource reference](/docs/astro/resource-reference-hosted) for more information.

  ### Additional improvements

  * The Astro UI now shows how many Workspaces each Team belongs to in **Settings** > **Access Management** > **Teams**.
  * You can now create dedicated clusters in `us-west1` on GCP.
</Update>

<Update label="June 20, 2023">
  ### Additional improvements

  * You can now add a new Astro user to Workspaces before the user has accepted their invite.
  * If you have Organization Owner permissions, you can now add a user to a Workspace even if the user hasn't been added to your Organization. Users added to Workspaces this way are automatically added to your Organization as an Organization Member.
  * The Astro UI now shows your Team IDs in **Settings** > **Access Management** > **Teams**. Use Team IDs to add Teams to Workspaces using the Astro CLI.
  * A Team's **Updated At** and **Updated By** values are now updated when you change the Team's permissions in a Workspace or Organization.

  ### Bug fixes

  * Fixed an issue where a Workspace descriptions were incorrectly required when creating a new Workspace through the Astro CLI.
</Update>

<Update label="June 13, 2023">
  ### Manage billing and track usage for Astro Hosted

  Use the new **Billing** page in the Astro UI to see both high-level and detailed metrics about your spend in Astro Hosted. You can also use this page to configure your billing details and view invoices. See [Manage billing](/docs/astro/manage-billing) for more details.

  ### New cell type for using Airflow operators in the Astro Cloud IDE

  You can now use any Airflow operator available on the Astronomer Registry in your Astro Cloud IDE pipeline. Operator cells apply formatting and checks for parameter inputs, making it easy to configure operators as part of your pipeline.

  Additionally, you can configure custom cells to use your team's custom operators in a pipeline.

  ### IMDSv2 is now enforced on AWS clusters

  <Warning>
    **Breaking change**

    If your dags assume IAM roles to directly access metadata on your cluster using IMDSv1, this change can result in dag run failures. Upgrade your dags to use IMDSv2 for all cluster metadata requests.

    See [Use IMDSv2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html) for more information.
  </Warning>

  Astronomer now enforces IMDSv2 on all AWS clusters. Any requests for resources on your clusters are now session-based and muse include a token in the request body.

  ### Additional improvements

  * Trial Deployments now have [dag-only deploys](/docs/astro/deploy-code) enabled by default.
  * The Astro UI now shows your **Organization Short Name** and **Astro SAML Connection Name** in the Astro UI.
  * You can now view mapped tasks from the **DAGs** page in the Astro UI.

  ### Bug fixes

  * Fixed an issue where worker node pools in Hosted dedicated clusters on Azure were not being updated correctly.
  * Fixed an issue the Astro UI would reset a Deployment's **Min Worker Count** from 0 to 1 after you edited the Deployment in any way.
</Update>

<Update label="June 6, 2023">
  ### Track user actions and ensure compliance with audit logs

  You can now export audit logs from the Astro UI to view all actions taken in your Organization over a given time period. See [Export audit logs](/docs/astro/audit-logs) for setup steps.

  ### Additional improvements

  * You can now configure Hybrid GCP clusters with additional Memory Optimized and Compute Optimized Cloud SQL instance types. See [Supported Cloud SQL instance types](/docs/astro/resource-reference-gcp-hybrid#supported-cloud-sql-instance-types).
</Update>

<Update label="May 30, 2023">
  ### Manage permissions for groups of users with Teams

  Configure Teams from the Astro UI to manage the permissions for many users across Workspaces from a single page. *Teams* are a group of users in an Organization that you grant the same Workspace permissions, without needing to define them individually.

  See [Make a Team](/docs/astro/manage-teams) for setup steps.

  ### Bug fixes

  * In Astro Hosted, an irrelevant **AWS external ID** info page has been removed from the Astro UI.
  * Fixed an issue where dag-only deploys could be unreliable due to the deploy process not requesting enough resources in the cluster.
</Update>

<Update label="May 23, 2023">
  ### Introducing Astro Hosted and Hybrid

  *Astro Hosted* is a new way to run Airflow on Astronomer's cloud. On Astro Hosted, Airflow environments are managed and hosted entirely by Astronomer, enabling you to shift your focus from infrastructure to data.

  For more information about how Astro Hosted works, see the [Architecture overview](/docs/astro/astro-architecture).

  If you're already an Astro user and your Deployments run in your company's own cloud, you're using *Astro Hybrid*. This version of Astro was formerly known as Astro - Bring Your Own Cloud.

  To see whether you're an Astro Hybrid user, open your Organization in the Astro UI and go to **Settings** > **General**. Your version of Astro is listed under **Product Type**.

  See [Documentation refactor for Astro Hybrid](#documentation-refactor-for-astro-hybrid) to learn how the documentation has changed for current Astro Hybrid users.

  ### Configure default Kubernetes Pods on Astro Hosted

  One of the biggest risks of running the Kubernetes executor or KubernetesPodOperator is that your tasks can accidentally request more resources than expected, which can drive up costs. To limit this risk, you can now configure default and maximum Pod resources from the Astro UI. If a task tries to request Pod resources that are more than your configured limits, the task fails.

  See [Configure Kubernetes Pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources) for setup steps.

  ### Documentation refactor for Astro Hybrid

  The following updates have been made to documentation to accommodate new Astro Hosted information:

  * Astro Hybrid documentation now has a dedicated menu under "Administration" that contains all docs related to Hybrid installation and cluster management. See [Astro Hybrid overview](/docs/astro/hybrid-overview).
  * All other docs now assume Astro Hosted by default. If a feature is functionally different between Hosted and Hybrid, the documentation for that feature will include a note about how that setup differs for Hybrid. Look for the blue **Alternative Astro Hybrid setup** notes throughout documentation.
</Update>

<Update label="May 16, 2023">
  ### Automate Organization management with Organization API tokens

  You can now create Organization API tokens to automate key actions across your Organization and all of the Workspaces in it. You can customize the role and expiration date of the token to give it the minimum required permissions for the task it completes. Some common actions that you can automate with Organization API token are:

  * Creating Workspaces.
  * Inviting users to an Organization or Workspace.
  * Creating and updating Deployments using a [Deployment file](/docs/astro/manage-deployments-as-code).
  * Exporting audit logs.
  * Gathering metadata about Deployments using the Airflow REST API.
  * Completing any of the actions you can complete with a Workspace API token or Deployment API key across all Deployments in your Organization.

  See [Manage Organization API tokens](/docs/astro/organization-api-tokens) for more information.
</Update>

<Update label="May 2, 2023">
  ### Receive Astro alerts on Slack or PagerDuty

  Astro alerts are a new way to be notified when your dags aren't running as expected. Unlike Airflow callbacks and SLAs, Astro alerts require no changes to dag code and integrate with Slack and PagerDuty.

  You can set an alert on any dag to be notified when the dag fails or when a task takes longer to run than expected. See [Astro alerts](/docs/astro/alerts) for configuration steps.

  ### Bug fixes

  * Fixed an issue where SSO configurations made through Astronomer support could be overridden by updating the SSO configuration through the Astro UI.
</Update>

<Update label="April 26, 2023">
  ### Improved log viewing in the Astro UI

  The Deployment **Logs** page in the Astro UI now shows logs for your Deployment's workers, schedulers, triggerers, and webserver. Additionally, you can now view up to the last 10,000 logs emitted by your Deployment from the Astro UI.

  To make it easier to parse this larger log volume, the **Logs** page now lets you filter by log type, date, and keyword. See [View logs](/docs/astro/view-logs) for more information.
</Update>

<Update label="April 18, 2023">
  ### Self-service configuration for single sign-on (SSO) connections

  You can now configure SSO connections directly from the Astro UI without assistance from Astronomer support. Use the **Authentication** page to configure different authentication environments for your Organization by creating and managing multiple SSO connections and domains.

  To review the new process for creating SSO connections, see [Set up authentication and SSO](/docs/astro/configure-idp). To create new managed domains to map to your SSO connections, see [Manage domains](/docs/astro/manage-domains).
</Update>

<Update label="April 11, 2023">
  ### Additional improvements

  * The node type for running Airflow system components on GCP clusters has been reduced from `n2-standard-4`  to `e2-standard-4`.
  * To optimize infrastructure costs for running the Kubernetes executor, Kubernetes executor worker Pods from different Deployments can now run on the same worker node. This occurs only when the Deployments are hosted in the same cluster and use the same worker node instance type.
</Update>

<Update label="April 4, 2023">
  ### Preview Deployments

  You can now create preview Deployments from feature branches in your Git repository. Use a [preview Deployment template](/docs/astro/ci-cd-templates/template-overview#preview-deployment-templates) or [GitHub Actions template](/docs/astro/ci-cd-templates/github-actions-deployment-preview) to configure your Astro pipelines to:

  * Create the preview Deployment when you create a new branch.
  * Deploy code changes to Astro when you make updates in the branch.
  * Delete the preview Deployment when you delete the branch.
  * Deploy your changes to your base Deployment after you merge your changes into your main branch.

  ### Additional improvements

  * Added the ability to enforce CI/CD deploys. You can now configure your Deployment to only accept code deploys if they are triggered by a Deployment API key or Workspace token.
  * When you create a new cell in the Astro Cloud IDE, the editor auto-scrolls to your new cell and selects it.

  ### Bug fixes

  * Fixed a bug where the UI passed the wrong cluster type.
  * Fixed an issue where the Deployment status shows as 'deploying' when KPOs are running.
</Update>

<Update label="March 28, 2023">
  ### New GCP node instance types available

  You can now use the following node instance types for worker nodes in GCP clusters:

  * `e2-standard-32`
  * `e2-highcpu-32`
  * `n2-standard-32`
  * `n2-standard-48`
  * `n2-standard-64`
  * `n2-highmem-32`
  * `n2-highmem-48`
  * `n2-highmem-64`
  * `n2-highcpu-32`
  * `n2-highcpu-48`
  * `n2-highcpu-64`

  For a list of all instance types available for GCP, see [Supported worker node pool instance types](/docs/astro/resource-reference-gcp-hybrid#supported-worker-node-pool-instance-types).

  ### Additional improvements

  * You can now use `db.m6g` and `db.r6g` RDS instance types on AWS clusters.
  * The default RDS instance type for new AWS clusters has been reduced from `db.r5.large` to `db.m6g.large`
  * The default CIDR range for new AWS clusters has been reduced from /19 to /20.
  * You can now submit a **Request type** in the [Astro UI support form](https://cloud.astronomer.io/open-support-request). When you choose a request type, the form updates to help you submit the most relevant information for your support request.
  * You can no longer delete a Workspace if there are any Astro Cloud IDE projects still in the Workspace.
  * Organization role permissions have changed so that only Organization Owners can create Workspaces.

  ### Bug fixes

  * Fixed an issue where you could set a Deployment's scheduler resources to less than 5 AU.
</Update>

<Update label="March 21, 2023">
  ### Automate Workspace and Deployment actions using Workspace API tokens

  Use Workspace API tokens to automate Workspace actions, such as adding users to a Workspace and creating new Deployments, or for processes that a Deployment API key can automate. You can customize the role and expiration date of the token to give it the minimum required permissions for the task it completes.

  To create and use Workspace API tokens, see [Workspace API tokens](/docs/astro/workspace-api-tokens).

  ### Additional improvements

  * In the Astro Cloud IDE, you can now specify the output table for a Warehouse SQL cell using both literal and Python expressions.
  * Port 80 is no longer used for certificate management on the data plane.
  * To switch Organizations in the Astro UI, you now use the **Switch Organization** button next to your Organization's name.

    <Frame>
      <img alt="Switch Organizations button" />
    </Frame>
</Update>

<Update label="March 15, 2023">
  ### Run the Kubernetes executor in Astro

  You can now configure your Deployments to use the Kubernetes executor for executing tasks. Using the Kubernetes executor, you can:

  * Run tasks with different version dependencies in the same Astro project.
  * Request specific amounts of CPU and memory for individual tasks.
  * Automatically down your resources when no tasks are running.

  The Kubernetes executor runs each task in its own Kubernetes Pod instead of in shared Celery workers. Astronomer fully manages the infrastructure required to run the executor and automatically spins Pods up and down for each of your task runs. This executor is a good fit for teams that want fine-grained control over the execution environment for each of their tasks.

  To learn whether the Kubernetes executor works for your use case, see [Choose an executor](/docs/astro/executors-overview#choose-an-executor). To configure the Kubernetes executor for a task or Deployment, see [Configure the Kubernetes executor](/docs/astro/kubernetes-executor).

  ### Simplified Organization management in the Astro UI

  The Astro UI has been redesigned so that Organization settings tabs are now available in the left menu. Use this new menu to switch between pages as you can for Workspace settings.

  While most tabs were migrated directly to the left menu with the same name, some pages have been renamed and moved:

  * Formerly located in **Overview**, your Workspace list is now available in **Workspaces**.
  * Formerly located in the **People** tab, Organization user management settings are now in **Settings** > **Access Management**.
  * Formerly located in the **Settings** tab, general Organization settings are now in **Settings** > **General**.

  ### New Astro Cloud IDE integration with GitLab

  You can now configure a GitLab repository in your Astro Cloud IDE project. Configuring a GitLab repository allows you to commit your pipelines and deploy them to Astro directly from the Astro Cloud IDE.

  ### Additional improvements

  * Clusters on an Astro - Hosted installation no longer retain Airflow logs which are older than 90 days.
  * The Data Plane System node pool instance type on GCP clusters has been reduced from `n2-standard-4` to `n2-standard-2`.
</Update>

<Update label="March 7, 2023">
  ### Get expert advice on Astro and Airflow in office hours

  Office hours are a new way for Astro customers to meet with the Astronomer Data Engineering team. In an office hour meeting, you can ask questions, make feature requests, or get expert advice for your data pipelines.

  You can now schedule a 30-minute office hour meeting in the **Help** menu next to your user profile in the Astro UI.

  <Frame>
    <img alt="Button to book office hours in the Astro UI" />
  </Frame>

  For more information, see [Book office hours in the Astro UI](/docs/astro/astro-support#book-office-hours).

  ### Additional improvements

  * The node pool instance type used for Astro system components on GCP clusters has been reduced from `n2-standard-4` to `n2-standard-2`.
  * Dags generated by the Astro Cloud IDE now use UTC instead of your current timezone as the default timezone for scheduling dag runs.

  ### Bug fixes

  * Fixed an issue in the Astro Cloud IDE where you could not update a pipeline that was configured with an invalid cyclic dependency chain.
  * Fixed an issue where deploying an Astro project with a custom Docker image tag resulted in the Deployment always having the **Deploying** status in the Astro UI.
  * Fixed an issue where worker Pods on Azure clusters were sometimes unable to scale because there was no prioritization for starting up essential scheduling Pods.
</Update>

<Update label="March 1, 2023">
  ### Astro no longer requires administrator access on AWS

  Astro no longer requires administrator permissions for its dedicated AWS account. Instead, Astro now assumes a cross-account IAM role with the minimum necessary permissions for running and managing clusters. See [Install Astro on AWS](/docs/astro/install-aws-hybrid) for more information.

  ### IdP-initiated logins through the Okta dashboard

  If your Organization uses Okta as your Astro identity provider, you can now log in to Astro directly from your [Okta Apps dashboard](https://help.okta.com/eu/en-us/Content/Topics/end-user/dashboard-overview.htm). If you've been authenticated by Okta, you no longer need to be authenticated by Astro when you access it through your dashboard.

  ### Additional improvements

  * Ingress to the Kubernetes API on Google Cloud Platform (GCP) and Azure clusters is now limited to Astro control plane IPs. This change will be implemented on all clusters in the coming weeks.

  ### Bug fixes

  * To protect the functionality of Astro monitoring services, you can no longer override the values of the following environment variables:

    * `AIRFLOW__METRICS__STATSD_ON`
    * `AIRFLOW__METRICS__STATSD_HOST`
    * `AIRFLOW__METRICS__STATSD_PORT`
    * `AIRFLOW__METRICS__STATSD_ALLOW_LIST`
    * `AIRFLOW__METRICS__STATSD_STATSD_CUSTOM_CLIENT_PATH`
    * `AIRFLOW__METRICS__STATSD_PREFIX`

      You can still set new values for these variables, but the values will be automatically overwritten in the Astro data plane. See [Platform variables](/docs/astro/platform-variables).

  * Fixed an issue where a user could provision multiple accounts when their login email address included differently cased characters.
</Update>

<Update label="February 21, 2023">
  ### New identity-first authentication model

  Astro has migrated to an identity-first authentication model. Users now authenticate to the Astro platform instead of individual Organizations, and Organizations can set permissions for how users can modify and access resources. This model prioritizes identity verification and enforces authentication policies for user email domains.

  For all users logging in to Astro, this migration has the following effects:

  * Instead of being redirected to separate login pages for each Organization, all Astro users log in through a universal login page.
  * Users belonging to multiple Organizations no longer have to log in again when switching Organizations.
  * Users no longer need to enter their email on a separate page before they log in to the Astro UI.
  * If your Organization enforces single sign-on (SSO), users can now authenticate to Astro with a username and password when your email domain doesn't enforce SSO.

  For Organization Owners, this migration has the following additional effects:

  * You can now use an SSO bypass link to log in to Astro if your SSO connection is disrupted.
  * Your Organization now has a list of owned email domains, and any users logging into Astro with one of those domains will be redirected to your configured identity provider.

  To configure authentication behavior, see [Configure SSO](/docs/astro/configure-idp#advanced-setup).

  ### New Hosted regions available

  You can now create clusters in the following regions on an Astro - Hosted installation.

  * AWS

    * `ap-northeast-1`
    * `ap-southeast-2`
    * `eu-central-1`
    * `eu-west-1`
    * `us-east-1`
    * `us-west-2`

  * Google Cloud

    * `asia-northeast1`
    * `australia-southeast1`
    * `europe-west1`
    * `europe-west2`
    * `us-central1`
    * `us-east4`

  * Microsoft Azure

    * `australiaeast`
    * `japaneast`
    * `northeurope`
    * `westeurope`
    * `eastus2`
    * `westus2`

  ### Additional improvements

  The default CIDR ranges for new GCP clusters have been reduced. The following are the new CIDR ranges:

  * **Subnet CIDR**: `172.20.0.0/22`
  * **Pod CIDR**: `172.21.0.0/19`
  * **Service Address CIDR**: `172.22.0.0/22`
  * **Service VPC Peering**: `172.23.0.0/20`

  ### Bug fixes

  In the Astro UI, when using **Compare** on the **Lineage Graph** page, you can now compare shorter run lengths.
</Update>

<Update label="February 14, 2023">
  ### Authorize Workspaces to clusters

  You can now keep teams and projects isolated by authorizing Workspaces to specific clusters. Use this feature to better manage cloud resources by ensuring that only authorized Deployments are running on specific clusters.

  ### New Deployment health statuses and information in the Astro UI

  The Astro UI now includes three additional [Deployment health statuses](/docs/astro/deployment-health-incidents) that you might see when creating or pushing code to a Deployment.

  * The **Creating** status indicates that Astro is still provisioning the resources for the Deployment.
  * The **Deploying** status indicates that a code deploy is in progress. Hover over the status indicator to view specific information about the deploy, including whether it was an image deploy or a dag-only deploy.
  * The **Unknown** status indicates that Deployment status can't be determined.

  Additionally, the Deployment information page in the Astro UI now includes fields for **Docker Image** and **DAG Bundle Version** that show unique timestamps and tags based on your latest code deploy. Use this information as the source of truth for which version of your code is currently running on the Deployment.

  ### View OpenLineage facets for lineage job runs

  [OpenLineage facets](https://openlineage.io/docs/spec/facets/) are JSON objects that provide additional context about a given job run. By default, a job run includes facets that show what kind of job was completed, whether the job run was successful, and who owns the job.

  You can now view all available facets for a job run, including [custom facets](https://openlineage.io/docs/spec/facets/custom-facets), by opening the job run's **Lineage Graph** and then selecting the **Info** tab. You can check the status of your facets, including whether they are correctly formatted, so that you can resolve potential issues in your data pipelines.

  <Frame>
    <img alt="Example OpenLineage facet page in the Astro UI" />
  </Frame>

  ### Additional improvements

  * You can now create AWS clusters in `ap-northeast-1` and `ap-southeast-2` on an Astro - Hosted installation.
  * You can now create GCP clusters in `australia-southeast1` on an Astro - Hosted installation.

  ### Security fixes

  * Fixed [CVE-2023-0286](https://nvd.nist.gov/vuln/detail/CVE-2023-0286).
</Update>

<Update label="February 7, 2023">
  ### Additional improvements

  * The instructions on the welcome page for new Astro users who are not yet part of an Organization have been improved to make getting started easier.

  ### Bug fixes

  * Removed nonfunctioning date filtering functionality from the Lineage UI.
  * Fixed an issue where triggering an image deploy from an older version of the Astro CLI could unintentionally turn off dag deploys on a Deployment.
</Update>

<Update label="January 31, 2023">
  ### Bug fixes

  * When you select **Mark Success** or **Clear** for Deployment task actions in the Airflow UI, you are now correctly redirected to the dag **Tree** view instead of the **DAGs** homepage.
  * Fixed [CVE-2022-41721](https://avd.aquasec.com/nvd/2022/cve-2022-41721/).
</Update>

<Update label="January 24, 2023">
  ### New Workspace Home page

  When you select a Workspace in the Astro UI, the **Home** page now appears first. On this page, you can:

  * Check the status of your Deployments.
  * Quickly access your most recently viewed Deployments and Cloud IDE projects.
  * View release notes for all Astro products.

  <Frame>
    <img alt="Workspace home page in the Astro UI" />
  </Frame>

  See [Introducing Astro’s New Workspace Homepage](https://www.astronomer.io/blog/introducing-astros-new-workspace-homepage/) for more information.

  ### Additional improvements

  * Ingress to the Airflow UI and API on Astro clusters is now limited to control plane IPs. This change will be implemented on all clusters in the coming weeks.
  * You can now request custom tags for your AWS clusters by submitting a support request to [Astronomer support](https://cloud.astronomer.io/open-support-request). You can view your cluster tags in the Astro UI by selecting **Clusters**, selecting a cluster, and then clicking the **Details** tab.
  * You can now create new clusters in France Central for Bring Your Own Cloud installations of Astro on Azure.
  * Improved the speed of dags appearing in the Airflow after completing a dag-only deploy.

  ### Bug fixes

  * Fixed [CVE-2022-48195](https://avd.aquasec.com/nvd/2022/cve-2022-48195/).
</Update>

<Update label="January 18, 2023">
  ### Bug fixes

  * Fixed an issue with Google Cloud Platform (GCP) clusters where the metadata database for a Deployment could persist after the Deployment was deleted.
</Update>

<Update label="January 10, 2023">
  ### New Astro Cloud IDE cell types

  To simplify the creation of new tasks, the following new cell types are now available in the Astro Cloud IDE:

  * **SQL**: Run a SQL query against an existing database connection and save the query results in an XCom file for use by other cells. Use this cell type to run smaller queries and store the results in Airflow for quick access by other cells.
  * **Warehouse SQL**: Run a SQL query against an existing database connection and store the query results in your data warehouse. Use this cell type for data operations that require more storage and reliability.
  * **Markdown**: Add inline Markdown comments to your generated dag code. Use this cell type to document code decisions and to make it easier for team members to collaborate on shared pipelines.

  ### Additional improvements

  * To reduce the time it takes for Airflow to parse new dag files, the default value for `AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL` has been reduced from 5 minutes to 30 seconds for all Deployments regardless of Runtime version. For most users, this means that you will see new dags appear in the Airflow UI faster.
  * In the Astro UI, a banner now appears if there is an incident reported on the [Astro status page](https://status.astronomer.io/).

  ### Bug fixes

  * Sorting the **Organization Role** column in the **People** tab of the Astro UI now works as expected.
  * Fixed an issue where lineage groups would occasionally not collapse as expected in the **Lineage Graph** view.
</Update>


# 2024 Astro release notes
Source: https://astronomer.io/docs/astro/release-notes-2024

Astro release notes from 2024, covering features, bug fixes, and version updates for Astro, the Astro CLI, Astro Runtime, and the Remote Execution Agent.

<Tip>[Subscribe to Astro release notes](/docs/astro/release-notes-subscribe) to receive updates via RSS, email, or Slack.</Tip>

Astro release notes from 2024. See the [current release notes](/docs/astro/release-notes) for the latest updates.

<Update label="December 17, 2024">
  ### Additional improvements

  * Two new [Astro alert types](/docs/astro/alerts#notification-channels-scope), **Task Failure** and **DAG Duration** are generally available.
  * Add icon to certain Deployment fields to indicate that they cannot be edited after the Deployment is created.

  ### Bug fixes

  * Fixed a bug that prevented users with custom roles at their Team level from making API tokens with the same level of permissions as themselves.
</Update>

<Update label="December 10, 2024">
  ### Additional improvements

  * Added `bundle` as a deploy type for [Deploy APIs](https://www.astronomer.io/docs/astro/api/v-1-beta-1/platform-api-reference/deploy/) to support [dbt deploys](/docs/astro/deploy-dbt-project#option-2) via the Astro API.
</Update>

<Update label="December 4, 2024">
  ### Pay-as-you-go Astro now available on AWS Marketplace

  You can now sign up for [Developer-tier Astro](https://www.astronomer.io/pricing/) as a month-to-month subscription through the AWS Marketplace. Choosing to subscribe to Astro through the AWS Marketplace allows you to start an Astro Trial, and then proceed with a monthly subscription that uses your AWS Billing information.

  See [Subscribe to Astro from the AWS Marketplace](/docs/astro/subscribe-aws) to get started.

  ### Additional improvements

  * Two new Airflow application metrics are available for the Universal Metrics Exporter, `dagrun.first_task_scheduling_delay` and `task_instance_created_<operator_name>`. See [Export metrics](/docs/astro/export-metrics#airflow-application-metrics) for more information.
  * Improved Astro's logic for choosing Deployment [fallback contact emails](/docs/astro/deployment-details#fallback-emails).
  * Added message in Deployment details to clarify if the contact emails listed are a fallback value or not.

  ### Bug fixes

  * Fixed an issue where Data Product Alerts were unexpectedly showing up when using the Workspace filter in the Organization Alert Management List.
  * Fixed a bug where asset filters weren't incorporating the entered search.
</Update>

<Update label="November 19, 2024">
  ### Additional improvements

  * [dbt deploy](/docs/astro/deploy-dbt-project) is now generally available.
  * Improved error messaging when an Organization Owner email is not supplied, preventing you from creating default Deployment alerts.
  * Updated the onboarding flow so that newly invited users see a signup page, instead of the login page.
  * Added the ability to retrieve audit logs via the Astro API. See [Audit logs reference](/docs/astro/audit-logs#audit-logs-reference) and [Astro API changelog](https://astronomer.io/docs/astro/api/platform/changelog/2024/11/19) for more information.

  ### Bug fixes

  * Fixed a bug in Observe where selecting a dataset with only downstream dependencies did not fully update the asset lineage graph. Now, all dependencies are correctly displayed.
</Update>

<Update label="November 12, 2024">
  ### Network egress management

  <Note>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Note>

  You can now enable [Private Network Egress](/docs/astro/private-network-egress) or configure [Customer Managed Egress for Dag Workloads](/docs/astro/customer-managed-egress) for dedicated AWS clusters to ensure security and compliance, and to provide a data loss protection architecture to secure against unauthorized data transfer.

  ### Additional improvements

  * Added the ability to work with the Astro Environment Manager through the Astro API, by working with **Environment Objects**. Environment Objects are the API representation of the different functionalities the Environment Manager supports such as Universal Metrics Export, Airflow Variables, and Connections.
  * Added the ability to filter assets list by asset type, namespace, and dag.
  * Updated Deployment configuration page.
  * Removed **Airflow Variable Key** from UI when editing Airflow Variable.
  * Moved the **IP Access List** from being a sub-section of the Authentication settings page to a dedicated page within Organization **Access Management**.

  ### Bug fixes

  * Fixed a bug where only Organization owners were considered as fallback emails for Deployment Health Alerts alerts. With this fix, Workspace owners will be considered first and Organization owners will only be considered as a final fallback if no human Workspace owners exist.
</Update>

<Update label="November 5, 2024">
  ### Proactive Alerts for Deployment health

  <Note>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Note>

  Astro now proactively alerts you on Deployment-level health issues when infrastructure components aren’t running as expected. New Deployments now receive four Deployment health alerts by default. These alerts provide insight into infrastructure-level issues and suggest specific actions to remediate the problem. Alerts notify the **Contact Emails** associated with a Deployment by default and are fully customizable. See [Astro Alerts](/docs/astro/alerts#deployment-health-alerts).

  ### Create data products with Astro Observe

  <Note>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Note>

  You can now use Astro Observe to create Data Products, which allow you to configure alerts and receive insights into the performance of the pipelines where you integrate them. See [Create a Data Product](/docs/astro/create-data-products).

  ### Additional improvements

  * Added validation of environment variable keys in `deployment create` and `update` requests. Environment variables must now match the following regex: `^[a-zA-Z_]+[a-zA-Z0-9_]*$`.
  * Upgraded Azure clusters to Kubernetes 1.29
  * Improved descriptions for scheduler sizes and **High Availability** in advanced section of cluster creation.
  * Improved description of **Development Mode** in advanced section of Deployment creation.
  * Removed a warning that displayed in the Astro UI when a user's Deployment was using **CI/CD Enforcement**.

  ### Bug fixes

  * Fixed an issue where **Open Airflow** button was disabled, even when the Deployment was not getting created or hibernating.
  * Fixed a bug where the Airflow webserver would restart for dag-only deploys.
</Update>

<Update label="October 29, 2024">
  ### Airflow mini-scheduler turned off by default

  The [mini-scheduler config](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#schedule-after-task-execution) (`AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION`) is now disabled by default. Previously enabled to optimize performance at scale, the mini-scheduler sometimes led to unexpected task state changes and failures, especially when the main scheduler got out of sync. This change reduces debugging complexity, but users with very large dags, many downstream dependencies, and the need to run over 10,000 tasks per hour can re-enable it for potential performance benefits by setting `AIRFLOW__SCHEDULER__SCHEDULE_AFTER_TASK_EXECUTION` to `True`.

  ### Bug fixes

  * Fixed a bug where only users with Organization-scoped roles could create or update Deployment alerts, even if they had Deployment alert creation and update permissions enabled.
</Update>

<Update label="October 22, 2024">
  ### Bug fixes

  * Fixed a bug affecting dbt deploys for Deployments using the Kubernetes executor.
</Update>

<Update label="October 15, 2024">
  ### Additional improvements

  * The **Scope** and **Scope Entity ID** columns of the **Alerts** and **Notification Channels** lists are now consolidated to the **Scope** column, which displays the entity name and links to the Deployment or Workspace the alert is enabled for. If the scope of the Alert or Notification Channel is the entire Organization, the **Scope** column value is `This Organization`. This enhancement allows you to navigate directly to the Deployment or Workspace from the list.

  ### Bug fixes

  * Fixed a bug where the **Default Pod Size** field was showing incorrect sizes.
</Update>

<Update label="October 9, 2024">
  ### Updates and changes to Astro Alerting UI

  * You can now define whether a notification channel is available to specific Deployments, an entire Workspace, or to an entire Organization.
  * You can create and edit alerts and notification channels of any type through the **Organization Alerting** page, whether they are Deployment or Organization alerts. If you don't have user permissions with privileges to make a particular type of alert, you can still access the **Alerting** editor in the **Organization** menu to create, edit, and manage both alerts and notification channels.
  * Previously, you could access and edit Workspace alerts from the **Alerting** page in the main navigation of the Astro UI. Now, you can create and manage your Organization alerts through the **Alerting** page in the **Organization** menu, or your Deployment-specific alerts in your Deployment's **Alerts** page.

  ### Bug fixes

  * Fixed a bug where editing Airflow variable links in the Environment Manager would delete any links to Deployments. Now, editing Airflow variables in the Environment Manager does not affect linked Deployments.
</Update>

<Update label="October 1, 2024">
  ### Additional improvements

  * Added **Updated By** and **Created By** columns to the Workspaces list.
  * Enhanced the onboarding experience so you can use one of the [template Deployments](/docs/astro/first-dag-onboarding#step-3-select-a-template) without connecting to your GitHub account.

  ### Bug fixes

  * Fixed an issue that showed incorrect pagination numbers in the table footer when searching the Alerts list.
  * Resolved an issue where the `globalFilter` search paramameter would not clear if you deleted the input value entirely.
  * Fixed an issue with the dbt deploy rollback functionality where dbt bundles weren't updated to the rollback deploy version's bundle state.
</Update>

<Update label="September 24, 2024">
  ### Additional improvements

  * The Universal Metrics Exporter is now generally available to Team-tier level customers. See [Export metrics](/docs/astro/export-metrics) for information about how to export infrastructure metrics about your Airflow use on Astro to your preferred third-party observability tools.
  * Updated the **Alerts** tab in the Astro UI to improve bulk-selecting alerts, increase the number of characters you can use when naming an alert, and allow auto-generated alert names to include the alert type in the alert name.
  * Improved the display of long strings to have a truncated display, which expands to show the full string when you hover over or copy the value.

  ### Bug fixes

  * Fixed an issue where you couldn't submit Log Summary feedback for different task runs that had the same task ID.
</Update>

<Update label="September 17, 2024">
  ### Additional improvements

  * Updated the Organization Dashboards' **Cost Breakdown** tab, which now displays top-level **Network** and **Cluster** costs.

  ### Bug fixes

  * Fixed an issue which prevented KPO task Pods from terminating correctly.
</Update>

<Update label="September 9, 2024">
  ### Scale Deployments Reliably with XL Deployment size and standalone Dag processor

  As your team and Airflow use cases grow, scaling up your Airflow environment reliably can be a challenge. Complex, dynamically-generated dags, sub-optimal dag parsing practices, or a growing business that requires a larger data pipeline can strain dag processing and threaten your Airflow scheduler’s availability.

  Astro Hosted Deployments now support high-scale environments more reliably by separating the Dag processor from the scheduler. You can take advantage of this Airflow best practice in Astro Hosted Deployments with **Medium** and larger sizes. By separating the Dag processor from the scheduler, Astro protects your core scheduling function from competing with the Dag processor for resources. This separation ensures scheduler availability for your most complex use cases and improves overall dag parsing performance.

  In addition to this change, you can now create Extra Large size Deployments  to confidently run your largest workloads on Astro. The Extra Large Deployment provides 8 vCPU and 16 GiB of memory and comes with two Dag processors to support your largest and most complex workloads, ranging from [dynamically-generated dags](/docs/learn/dynamically-generating-dags) to high-scale pipelines that can grow alongside your business.

  <Frame>
    <img alt="An example of the available Scheduler sizes" />
  </Frame>

  This update is available on Runtime version 9.7.0 and greater; if you are running a lower Runtime version, [upgrade your Runtime version](https://www.astronomer.io/docs/astro/../runtime/upgrade-astro-runtime). To learn more about this feature, read more about [Deployment scheduler resources](/docs/astro/deployment-resources#scheduler).

  ### Astro UI improvements

  Improved the navigation and design for managing clusters, editing settings, and accessing analytics for Organizations and Deployments.

  These changes include:

  * A new section in the main menu to include an **Organization** section with links to important resources for administering your Astro Organization. These include links to your [Organization Settings](/docs/astro/organization-settings), [Organization Dashboards](/docs/astro/organization-dashboard), and quick access to your cluster settings.
  * A dedicated **Deployment Analytics** page in your detailed **Deployment** information, which now includes improvements to the tooltips and metrics visualization design. See [Deployment metrics](/docs/astro/deployment-metrics) for more information about the available metrics.
  * Your **Deploy History** now appears on the **Overview** page of your Deployment. See [Deploy history](/docs/astro/deploy-history) for more information about how to view your code deploy history.

  ### Create and manage Airflow variables with the Environment Manager

  <Note>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Note>

  You can now create and manage Airflow variables for Deployments across your Workspace through the Astro Environment Manager. This allows you to quickly and securely create Airflow variables once and share them to multiple Deployments without having to set up your own secrets backend.

  See [Create Airflow variables in the Astro UI](/docs/astro/create-and-link-variables) for more information.
</Update>

<Update label="September 3, 2024">
  ### Additional improvements

  * Made updates to the Astro UI for editing **Astro Alerts**, to clarify whether you are discarding your edits to an alert, or discarding the alert.

  ### Bug fixes

  * Fixed an issue that blocked some users from pushing project images to the Docker registry.
</Update>

<Update label="August 27, 2024">
  ### Additional improvements

  * You can now give Airflow Dataset **Create** and **Delete** permissions to custom roles. This capability is provided to **Deployment Admin**, **Workspace Author**, **Workspace Operator**, and **Workspace Owner** default roles. Previously only **Workspace Owner** and above had access to this permission set. See more information about [User Permissions](/docs/astro/user-permissions) and [Custom Deployment Roles](/docs/astro/customize-deployment-roles).
  * If your Astro Deployments run on AWS or GCP clusters, you can now see the **Account** ID of the clusters in the **Cluster Details** page in your **Organization Settings**.
  * For Deployment alert emails, the sender now shows as **Astro**. It previously showed the sender as **Postmaster**.
  * You can now see Deployment details while Astro creates your Deployment, where only the specific actions that are not available until after creation is complete are disabled.
</Update>

<Update label="August 20, 2024">
  ### Additional improvements

  * Added the ability to specify Custom Workload Identity at Deployment creation time when creating an AWS Deployment.
  * Improved the descriptions of default hibernation schedules in the UI.
  * Added more detail in descriptions for code deploy failures in the **Deploy History** page.

  ### Bug fixes

  * Fixed an issue where you could not use global text search for dags in your Alerts.
  * Resolved a problem where search query parameters might be deleted when using pagination.
</Update>

<Update label="August 13, 2024">
  ### Additional improvements

  * Improved the descriptions of [Deployment Health Alerts](/docs/astro/alerts#deployment-health-alerts) to provide more specific information about performance thresholds.

  ### Bug fixes

  * Fixed an issue where dag lists didn't update when you switched Deployments in the Astro UI.
  * Fixed a bug where the cursor would disappear while typing in certain fields in the Astro UI.
</Update>

<Update label="August 7, 2024">
  ### Customize labels on your metrics export

  You can now add Key:Value pair labels to metrics that you export using the Universal Metrics Exporter. This allows you to tag and record metrics coming from specific Deployments or Workspaces. See [Export metrics from Astro](/docs/astro/export-metrics) for more information.

  ### Additional improvements

  * Added a new template demonstrating how to run dbt projects on Astro using [Cosmos](https://www.astronomer.io/integrations/dbt/) when you're onboarding. See [Get started on Astro](https://cloud.astronomer.io/start-astro) to set up a trial account and try out a template.

  ### Bug fixes

  * Fixed an issue where a user could not bulk select alerts if they had different names but the same configurations.
</Update>

<Update label="July 30, 2024">
  ### New alerts for Deployment health incidents

  <Note>
    **Labs**

    This feature is in [Labs](/docs/astro/feature-previews). Please reach out to your account team to enable this feature.
  </Note>

  This release introduces four new alert types that correspond to four of the existing [Deployment health incident types](/docs/astro/deployment-health-incidents#deployment-incidents). These new **Deployment Health Alerts** allow Astro to proactively notify you when Deployment health issues arise.

  Using Deployment health alerts, you can:

  * Improve alerting coverage beyond dag and task failures to address infrastructure-level incidents that are otherwise difficult to monitor.
  * Proactively monitor Deployment health and take immediate remediation actions through email, Slack, and PagerDuty to reduce mean time to resolution.
  * Share Deployment health visibility across teams.

  See [Astro Alerts: Trigger Types](/docs/astro/alerts#alert-types) and [Deployment Health Incidents](/docs/astro/deployment-health-incidents) for more information.

  ### Additional improvements

  * Now, when you link directly to your Astro or Airflow UI, any link previews successfully have improved visuals and include metadata information.
  * Improved the Astro Alerts UI to streamline creating notification channels and alerts across dags and Deployments. See [Astro Alerts](/docs/astro/alerts) for setup information.
  * Added an example Universal Metrics Export dashboard configuration file for Grafana Cloud. See [Export metrics from Astro](/docs/astro/export-metrics) for setup instructions and [Grafana example](/docs/astro/export-metrics#grafana-example) for a configuration example.
  * Organization Dashboards are now generally available to Enterprise-tier customers. See [View Organization Dashboards](/docs/astro/organization-dashboard) for more information.

  ### Bug fixes

  * Fixed a bug where the Astro UI would show an error in your Deploy History instead of your Runtime Version, if the Runtime version is `yanked`.
  * Fixed an issue when creating an alert where you couldn't select tasks with the same `task_id` across multiple selected dags.
</Update>

<Update label="July 24, 2024">
  ### Run your first dag on Astro with GitHub Integration

  Now, when you first set up your Astro account, you can choose to connect your GitHub account and run a sample Astro project template and dag. Previously, to run your first dag on Astro, you needed to either download and use the Astro CLI or work with GitHub Actions. Now, the onboarding process allows you to customize your first experience with Astro to focus on your use case, whether that's Business Operations, Generative AI, or learning Airflow, and set up a GitHub Integration to quickly clone and deploy the example dags. Try it now with [Start Astro](https://cloud.astronomer.io/start-astro).

  ### Work with dbt projects on Astro

  <Note>
    **Labs**

    This feature is in [Labs](/docs/astro/feature-previews). Please reach out to your account team to enable this feature.
  </Note>

  You can now deploy dbt code to Astro quickly and independently of your dag or image deploys. Data build tool (dbt) core is an open source tool for data transformation that uses SQL to transform your data instead of Python in your dags. You can also use both Cosmos, an open source tool that integrates dbt models into Airflow and treats them like dags, and dbt deploys on Astro to have unparalleled visibility into your dbt tasks with the Airflow UI and a streamlined code deploy process.

  See [Deploy dbt projects](/docs/astro/deploy-dbt-project) for more information about how to get started.

  ### Additional improvements

  * Improved how the GitHub Integration in the Astro UI links directly to the Astro Project in a GitHub repository.
  * (*Astro Hosted only*) The customer managed workload identity setting for Deployments is now generally available. This allows you to grant Astro Deployments all of the permissions of an AWS IAM role. See [Attach an IAM role to your Deployment](/docs/astro/authorize-deployments-to-your-cloud#attach-an-iam-role-to-your-deployment) for detailed information.
  * The ability to deploy automatically from GitHub using the official Astro GitHub integration is now generally available. See [Deploy code with GitHub](/docs/astro/deploy-github-integration) for setup steps.
  * Astro Runtime version 11 is now classified as Long Term Support (LTS). This means that instead of being supported until October 2024, Astro will support Runtime version 11 until October 2025. See the [Astro Runtime maintenance and lifecycle schedule](/docs/runtime/runtime-version-lifecycle-policy) for more information, including information about restricted versions.

  ### Bug fixes

  * Fixed some bugs in the Astro UI for defining ephemeral storage to show storage limits, fix maximum pod size for storage, and add storage for default pod size.
</Update>

<Update label="July 16, 2024">
  ### Additional improvements

  * Added the following metrics to [Metrics Export](/docs/astro/export-metrics):
    * The new `airflow_executor_open_slots`, `airflow_dagrun_dependency-check`, and `airflow_dagrun_dependency-check.<dag_id>` metrics allow you to collect metrics about your dags and executor status.
    * Monitor task execution with `kube_pod_container_resource_limits`. This new metric enables you to track resource use against the configured limits so you can understand if task execution meets your configured CPU, memory, or storage limits for your Celery Workers or Kubernetes Executor and KubernetesPodOperator pods.
  * Added new supporting documentation for the Astro Terraform Provider, including a Getting Started guide and code examples for common uses. Read [Astro Terraform Provider](/docs/astro/terraform-provider) for more information.

  ### Bug fixes

  * Fix an error where start times in the UI were different from actual dag run trigger times.
</Update>

<Update label="July 9, 2024">
  ### Additional improvements

  * Fixed a bug that prevented adding a date input for a dag filter in the Astro UI.
</Update>

<Update label="July 2, 2024">
  ### Export metrics about your Astro Deployments to observability tools

  <Note> This is feature is only available if you are on the **Team** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

  <Note>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Note>

  You can now export comprehensive, operational metrics about the performance of your Astro Deployments to third-party observability tools, such as New Relic, using the new Universal Metrics Exporter. This new feature allows you to configure a Prometheus endpoint to export metrics using the Prometheus data model, which means you can integrate your Astro observability metrics directly into your existing monitoring tools. See [Export metrics from Astro](/docs/astro/export-metrics) for setup instructions.

  ### Added ephemeral storage metrics to Astro Deployment Analytics

  To assist you in determining the amount of custom ephemeral storage to configure for your workers and schedulers, you can now use Deployment Analytics to see relevant usage metrics. These include:

  * **Ephemeral Storage Usage** metric that shows a % of usage against the configured limit for your Celery Workers, KubernetesPodOperator, Kubernetes Executor, and Scheduler.
  * **Dynamic Y-axis Scaling** to the Celery Worker, KPO/KE, and Scheduler Deployment analytics panel to include dynamic zooming.

  See [Deployment Analytics](/docs/astro/deployment-metrics#deployment-analytics) for more information.

  ### Additional improvements

  * The `Scheduler heartbeat not found` Deployment health incident is downgraded to `Warning` from `Critical`.
</Update>

<Update label="June 25, 2024">
  ### Customer managed workload identity for AWS

  <Note>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Note>

  The **Customer Managed Identity** Deployment setting is now available on AWS. This means that you can now assign an existing workload identity and AWS IAM role to your Airflow Deployments on Astro. When you use this setting, your Deployment uses the identity to assume the permissions of your IAM role and gain secure access to your data services. With this feature, you can:

  * Re-use or share a customer managed identity across many Deployments, either ephemeral or static.
  * Leverage existing identities when migrating from MWAA or open source Airflow environments

  This can reduce friction when migrating to Astro. See [Attach an IAM role to your Deployment](/docs/astro/authorize-deployments-to-your-cloud#attach-an-iam-role-to-your-deployment) for detailed information.

  ### Self-healing workers automatically address stuck queued tasks

  A new improvement to the Astro Data Plane now automatically identifies when Celery workers are online and healthy, but not actually processing new tasks. When this happens, it looks like many tasks are stuck in a `queued` state. With this new feature, you will experience a lower frequency of tasks stuck in a `queued` state, which causes performance and reliability issues for your Airflow implementation.

  Self-healing workers use the following process:

  * It first identifies workers that both have tasks stuck in a `queued` state for an extensive time period and are in Deployments where the concurrency available means that tasks should not be queued.
  * The healer then kills catatonic workers that are not running tasks or shifts workers that are still running tasks into a warm shutdown period.
  * After the catatonic worker is shut down, a new healthy, worker comes online and resumes tasks.

  This feature is automatically enabled for the Astro Hosted infrastructure and does not require any action.

  ### Additional Improvements

  * Improved the formatting for how IP addresses are listed in the Astro UI to make it easier to copy and paste them.

  ### Bug fixes

  * `europe-west6` is no longer available as a region for dedicated clusters on GCP.
</Update>

<Update label="June 18, 2024">
  ### Additional Improvements

  * Moved the **Environment variable** tab in **Deployment Settings** to the **Environment** tab, and renamed it to **Environment Variables**. This helps disambiguate between Environment Variables and Airflow Variables in the Astro UI. Refer to [Manage environment variables](/docs/astro/manage-env-vars#use-the-astro-ui) for more information about the different methods to set up and manage Environment variables.

  ### Bug fixes

  * Fixed a bug where a duplicate **Customer managed identity** option displayed when configuring Deployment settings.
</Update>

<Update label="June 11, 2024">
  ### Restrict Astro access to specific IP Address ranges with IP Access list

  <Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

  Organization owners can now limit the IP addresses of users to control who can access the Astro UI or programmatically work with the Astro API for their organization. This allows you to restrict access to your Astro Organization to users on a VPN or to specific network ranges. See [Set up IP Access list](/docs/astro/ip-access-list).

  ### Additional improvements

  * When using the GitHub integration, you can now retry a failed deploy or trigger a Git deploy directly from your Deployment settings page in the Astro UI. See [GitHub Integration](/docs/astro/deploy-github-integration) for more information.
  * Deployments at the end of an Astro trial now automatically hibernate and are not immediately deleted. If your trial has ended, add a credit card number to your Organization to access your Workspaces and wake your hibernating Deployment. While in hibernation, Deployment settings and configurations are preserved. If you don't enter a payment method within 30 days, your hibernating Deployments and all corresponding data are deleted. See [Start a trial](/docs/astro/trial) for more information.
  * The ability to create, update, and delete Deployment API tokens is restricted to users with the Workspace Owner role. Read [Workspace user permissions](/docs/astro/user-permissions#workspace-roles) for more information.

  ### Bug fixes

  * Fixed an issue where the Organization Settings Dashboard failed to load correctly.
</Update>

<Update label="June 6, 2024">
  ### API keys are no longer supported

  As of June 1, 2024, Deployment API keys are no longer supported. Replace your API keys with [Deployment API tokens](/docs/astro/deployment-api-tokens), confirm successful operation with your API tokens, and then delete your API keys.

  ### Configure ephemeral storage on worker Pods

  <Note>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Note>

  You can now customize the amount of ephemeral storage for data intensive workloads on Celery, Kubernetes, and KubernetesPodOperator workers. Previously, to accommodate larger workloads, you needed to integrate an external object storage or database tool to process large datasets within a single task. Now, you can customize the ephemeral storage when creating or updating your Deployment so that all data processing happens directly in your task Pod.

  You are only charged for requested resources which are greater than the minimum defaults for each worker type:

  * **Celery worker**: 10 GiB minimum by default. 100 GiB maximum.
  * **Kubernetes executor/ Kubernetes pod operator**: 0.25 GiB minimum by default. 100 GiB maximum.

  ### Automate Airflow, resource, and infrastucture management with the Astro Terraform Provider

  <Note>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Note>

  You can now use Terraform to automate managing resources and changes to large organizations and Airflow infrastructure with the Astronomer Terraform provider package. The provider is available through both the Terraform Registry and a public Github repository, where you can review the provider's code, make issues, and create pull requests.

  Refer to the Astronomer Terraform Provider  docs  in the [Terraform registry](https://registry.terraform.io/providers/astronomer/astro/latest/docs) or the [GitHub repository](https://github.com/astronomer/terraform-provider-astro) for more information.

  ### Additional improvements

  * The Astro UI now validates cron expressions for development Deployment hibernation schedules and shows commonly used hibernation schedules that you can enable or disable. Refer to [Create a hibernation schedule](/docs/astro/deployment-resources#hibernate-a-development-deployment) for more information.
  * If you don't already have a GitHub repository, you can now create one when you authorize the [GitHub Integration](/docs/astro/deploy-github-integration) from the Astro UI. Previously, you could only connect Astro to existing repositories.

  ### Bug fixes

  * Fixed an issue where you could not use Airflow connection testing with Azure Managed Identities on Astro.
</Update>

<Update label="May 30, 2024">
  ### Additional improvements

  * Dedicated clusters are now available only to [**Team**](https://www.astronomer.io/pricing/) tier customers and above.
  * Added a centralized docs reference page that lists all open source Apache Airflow provider packages and their versions for each Astro Runtime version. See [Provider package reference](/docs/runtime/runtime-provider-reference).

  ### Bug fixes

  * Fixed an issue where you could make changes to clusters and Deployments on an inactive Organization.
  * The default storage resources for the Kubernetes executor and KubernetesPodOperator Pods are now enforced with `0.25Gi` instead of `10Gi` in some cases. You can still customize the resource allocation for Kubernetes pods depending on your needs. See [Configure Kubernetes Pod Resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources)
</Update>

<Update label="May 22, 2024">
  ### The Astro GitHub integration is now in Preview

  <Note>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Note>

  The ability to deploy automatically from GitHub using the official Astro GitHub integration is now in Preview.

  The Astro GitHub integration is a new way to automatically deploy code from a GitHub repository to Astro by merging pull requests or making commits directly to specific branches, without needing to configure a GitHub Action. Additionally, the GitHub integration displays Git metadata directly in the Astro UI, including Git commit descriptions and gives you greater visibility into the status and logs of individual code deploys.

  See [Deploy code with the Astro GitHub integration](/docs/astro/deploy-github-integration) for more information.

  ### Bug fixes

  * Fixed an issue where users with custom roles could see roles that could not be assigned in the Astro UI.
</Update>

<Update label="May 15, 2024">
  ### Additional improvements

  * Airflow connections that you configure through the Astro UI environment manager are now mounted to Deployment schedulers, meaning that scheduler processes can now make use of these Airflow connections.

  ### Bug fixes

  * Fixed an issue where you couldn't update users who were added through SCIM but didn't belong to an Organization.
  * Fixed an issue where a user with a custom role could create API Tokens, users, or teams with greater permissions than their own.
</Update>

<Update label="May 8, 2024">
  ### Updates to address ranges for dedicated clusters on Google Cloud Provider

  Astro on GCP Dedicated Clusters uses source network address translation (SNAT) that performs many-to-one IP address translations for connections to your data sources and defaults secondary ranges to RFC 6598 address space (non-standard Private IP addresses), to minimize the risk and concern with IP overlap and exhaustion. Your target data sources will see connections from Astro using the VPC Subnet Range when using private networking, like VPC Peering or VPN. If you want to configure private connectivity, ensure the default subnet and peering ranges don't overlap with your target data source network when you're creating your dedicated cluster. See [Create a dedicated Astro cluster](/docs/astro/create-dedicated-cluster?tab=gcp) for more details.

  ### Improvements to Astro performance

  As part of continued investment in the reliability, performance, and scalability of Astro, Astronomer is embarking on a migration of public and private image registries. Astro Runtime clusters will benefit from more performant, globally distributed and geo-replicated image registries, with built-in registry resilience if a regional outage occurs. This is in addition to the previous release of the registry cache local to every Astro cluster.

  No end user or task runtime change or impact is expected as part of the backend cutover during the week of May 6th, 2024.

  ### Additional improvements

  * You can no longer create Deployments using Astro Runtime versions marked as `yanked` in `https://updates.astronomer.io/astronomer-runtime`, even if your Organization has enabled creating Deployments with deprecated Runtime versions. These versions of the Astro Runtime have known issues and should not be used. For more information, see [Restricted Runtime Versions](/docs/runtime/runtime-version-lifecycle-policy#restricted-runtime-versions).

  ### Bug fixes

  * Fixed a bug where multiple users could not access [Organization Dashboards](/docs/astro/organization-dashboard) simultaneously.
</Update>

<Update label="April 30, 2024">
  ### Deploy automatically from GitHub using the official Astro GitHub integration

  <Note>
    **Labs**

    This feature is in [Labs](/docs/astro/feature-previews). Please reach out to your account team to enable this feature.
  </Note>

  The Astro GitHub integration is a new way to automatically deploy code from a GitHub repository to Astro without needing to configure a GitHub Action. Compared to using GitHub Actions, the Astro GitHub integration:

  * Allows you to enforce software development best practices without maintaining custom CI/CD scripts.
  * Enables developers to iterate on dag code quickly.
  * Shows Git metadata directly in the Astro UI, including Git commit descriptions.
  * Gives you greater visibility into the status and detailed logs of an individual deploy.

  See [Deploy code with the Astro GitHub integration](/docs/astro/deploy-github-integration) for more information.
</Update>

<Update label="April 23, 2024">
  ### Restrict a custom Deployment role to specific Workspaces

  You can now restrict the use of a custom Deployment role to specific Workspaces. Use Workspace role restriction when some Workspaces in your Organization have different requirements for how users interact with Deployments. See [Restrict a custom Deployment role to specific Workspaces](/docs/astro/customize-deployment-roles#restrict-a-custom-deployment-role-to-specific-workspaces) for setup steps.

  ### Additional improvements

  * The [custom Deployment roles](/docs/astro/customize-deployment-roles) feature is now generally available.

  * You can now promote a [development Deployment](/docs/astro/deployment-resources#hibernate-a-development-deployment) to a production Deployment by switching off the **Development Mode** toggle in the Deployment's configuration.

  * Workspace Members can now see and use [custom Airflow menu items](/docs/learn/2.x/using-airflow-plugins#appbuilder-menu-items). To give a custom role this permission, you can add `deployment.airflow.customMenu.get` to the role's permissions list. This permission works only on Deployments running Astro Runtime 9 or later.

    Note that you might have to modify the code for your menu item plugins to make them work on Astro. See [Appbuilder menu items](/docs/learn/2.x/using-airflow-plugins#appbuilder-menu-items) for more information.

  * You can now filter the Workspaces and clusters lists in the Astro UI by name.

  ### Bug fixes

  * To improve the reliability of data lineage for customers who leverage it, data lineage is now a private preview feature that can be enabled upon request. To reenable data lineage for an Astro Organization, reach out to your account team.
</Update>

<Update label="April 16, 2024">
  ### Bug fixes

  * Fixed an issue where you couldn't grant a custom Deployment role to a Deployment API token using the Astro API.
</Update>

<Update label="April 9, 2024">
  ### Additional improvements

  * You can now manage [custom Deployment roles](/docs/astro/customize-deployment-roles) using the Astro API.
  * Improved the time it takes to load Deployment analytics in the Astro UI.
  * You can now view info-level incidents from the Deployment health status indicator in the Astro UI.
  * It is now possible for workers to use up to 6400 CPUs and 12800 GiB of memory on a single Deployment.

  ### Bug fixes

  * Fixed an issue where you couldn't configure boolean values for Airflow connections in the Astro UI.
  * Fixed an issue where Airflow connections configured through the Astro UI did not work with deferrable tasks.
</Update>

<Update label="April 3, 2024">
  ### Additional improvements

  * [Deployment health incidents](/docs/astro/deployment-health-incidents) are now generally available.
  * [Deployment hibernation](/docs/astro/deployment-resources#hibernate-a-development-deployment) is now in public preview.

  ### Bug fixes

  * Fixed an issue where connections configured in the Astro UI were not mounted to triggerer Pods, resulting in failed runs for tasks that use deferrable operators.
  * Fixed an issue where a field in the **Snowflake - Private Key (Content)** connection type was not parsed correctly.
  * Fixed an issue where Organization Members could delete and update Astro alerts.
  * Fixed an issue where the Astro UI didn't show the correct amount of available ephemeral storage for the default worker queue.
  * Removed additional dependencies to make Astro more resilient to Quay outages.
</Update>

<Update label="March 26, 2024">
  ### Refactored Astro API documentation

  Astro API documentation is now hosted at [https://www.astronomer.io/docs/api](/docs/astro/api/v-1/overview). In the new API documentation center, you can:

  * Format and test API requests directly in your browser.
  * Export requests to Python, Javascript, and curl.
  * View weekly changelogs for the API.

  ### Create Deployments with deprecated versions of Astro Runtime

  You can now use the Astro API to create Deployments with deprecated versions of Astro Runtime. Using deprecated Astro Runtime versions is sometimes necessary if you're migrating existing Airflow environments to Astro, or if you need to maintain deprecated environments for testing purposes.

  Note that this feature is disabled by default. To use this feature, reach out to your account team and request for the feature to be enabled. See [Run a deprecated Astro Runtime version](https://www.astronomer.io/docs/astro/../runtime/upgrade-astro-runtime#run-a-deprecated-astro-runtime-version) for more information.

  ### New GCP database instance types available

  You can now use the following node instance types for database instances in GCP clusters:

  * XLarge Compute Optimized (24 CPU, 48 GiB MEM)
  * XXLarge Compute Optimized (32 CPU, 64 GiB MEM)

  See [GCP Hybrid cluster settings](/docs/astro/resource-reference-gcp-hybrid#supported-cloud-sql-instance-types) for a list of all available database instance types.

  ### Additional improvements

  * The Astro UI now includes [Learning Bytes](https://academy.astronomer.io/learning-bytes-reporting) for features that are not yet configured within your Organization.
  * The Astro UI now loads the status for dag and task runs more quickly.

  ### Bug fixes

  * When you change a worker type for an existing worker queue, the Astro UI no longer resets the worker queue's concurrency configurations.
  * Fixed an issue where the Astro API did not return the correct value for `IsHibernating` when you queried Deployment information.
</Update>

<Update label="March 19, 2024">
  ### New Azure regions available on Astro Hosted

  You can now create Hosted dedicated clusters in the following Azure regions:

  * `centralus`
  * `westus3`
  * `southcentralus`

  See [Astro Hosted resource reference](/docs/astro/resource-reference-hosted) for more information.

  ### Custom Deployment roles are now in Preview

  <Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

  <Note>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Note>

  The ability to customize Deployment-level permissions with Deployment roles is now in [Preview](/docs/astro/feature-previews).

  You can additionally use the new **Workplace Accessor** and **Deployment Admin** roles to define which users have access to specific Deployments in your Workspace. See more in [User permissions reference](/docs/astro/user-permissions#workspace-roles) and and [Create and assign custom Deployment roles](/docs/astro/customize-deployment-roles).

  ### Export data from reporting dashboards using webhooks

  <Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

  <Note>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Note>

  In addition to exporting report data with downloads or email, you can now [export reporting data](/docs/astro/org-dash-exports) using webhooks. Use webhooks to send your reporting data to services such as Segment, Airtable, or Marketo.

  Note that webhook exports are a [Sigma feature in beta](https://help.sigmacomputing.com/docs/webhook-exports) and might experience behavior changes.

  ### Additional improvements

  * You can now use the new **Credits** tab in the [Organization Billing page](/docs/astro/manage-billing) to see your credit balance.
  * On Astro Hybrid, the maximum worker concurrency on a worker queue has increased from `64` to `256`.
</Update>

<Update label="March 7, 2024">
  ### Reporting dashboards are now in public preview

  [Organization dashboards](/docs/astro/organization-dashboard) are now in [Preview](/docs/astro/feature-previews) to use for examining key metrics across your Organization.

  You can also export data from dashboards in the format of your choice. Exports can be triggered on a regular schedule or as an alert when specific criteria are met in your data. Export reporting data to share with other team members or to keep a record of key performance indicators. See [Export reporting data](/docs/astro/org-dash-exports) for more information.

  ### Customize Deployment-level permissions using Deployment roles

  <Note>
    **Labs**

    This feature is in [Labs](/docs/astro/feature-previews). Please reach out to your account team to enable this feature.
  </Note>

  Custom Deployment roles are a new way to define granular permissions for Astro users. For the first time, you can set a user's permissions at the Deployment level and define which specific parts of a Deployment they can access or modify. Use custom Deployment roles to have users collaborate in the same Workspace with only the minimum permissions they require. See [Customize Deployment roles](/docs/astro/customize-deployment-roles) for more information.

  ### New Deployment registry cache to improve resiliency

  Deployments now include a cache of Astronomer's image registry that stores the current Astro Runtime image for your Deployment. Because Deployments now always have access to their running image, image registry outages should no longer result in failed dag runs.

  ### Additional improvements

  * Removed nonfunctional network usage per Pod metrics from the Deployment **Analytics** page.
  * The end-of-life date for Deployment API keys is June 1, 2024.
  * The Cloud UI has been renamed to the Astro UI across all help text and documentation.
  * Due to a minor change to Astronomer cluster architecture, you can no longer add custom tags to Hybrid clusters on AWS.

  ### Bug fixes

  * Fixed an issue where CPU usage per Pod metrics did not render correctly in the Deployment **Analytics** page.
  * Fixed an issue where the Astro UI didn't show all available Teams when selecting Teams to add to a Workspace.
</Update>

<Update label="February 27, 2024">
  ### Use a custom service account to authorize Deployments to GCP

  You can now attach a custom GCP [service account](https://cloud.google.com/iam/docs/service-account-overview) to your Deployment to grant the Deployment all of the service account's permissions to your cloud. Using a custom service account provides the greatest amount of flexibility for authorizing Deployments to your cloud. For example, you can use existing service accounts on new Deployments, or your can attach a single service account to multiple Deployments that should all have the same level of access to your cloud. For setup steps, see [Authorize Deployments to your cloud](/docs/astro/authorize-deployments-to-your-cloud).

  ### Bug fixes

  * Fixed an issue where the Astro API failed to list Deployments after you deleted a hibernation override setting.
</Update>

<Update label="February 21, 2024">
  ### New worker types

  Astro Hosted Deployments now support A120 and A160 workers, which include enough CPU and memory to handle the most resource-intensive tasks in your dags. See [Astro Hosted resource reference](/docs/astro/resource-reference-hosted#astro-worker-types) for more information about each worker type.

  ### New platform variables to improve Celery executor reliability

  Astro Deployments now have the following environment variables set by default. This is to ensure that the Celery executor doesn't freeze when it attempts to connect with a Pod in the Celery backend that has unexpectedly terminated.

  ```text wrap theme={null}
  AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_TIMEOUT=30
  AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_CONNECT_TIMEOUT=5
  AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__SOCKET_KEEPALIVE=True
  AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__RETRY_ON_TIMEOUT=True
  ```

  For more information about each of these variables, see [Platform variables](/docs/astro/platform-variables)

  ### Ephemeral storage limit on schedulers

  Astro now limits that amount of ephemeral storage in a scheduler to 5Gi. If a scheduler attempts to use more than 5Gi of ephemeral storage, it will be terminated.

  It is rare for schedulers to require more than 5Gi of ephemeral storage. If your schedulers start to terminate after this update, ensure the following:

  * Your Deployment is not producing large amounts of temporary files without cleaning them up.
  * If you're using the BingAds Python SDK, your Deployment uses version 13.0.14 or later. Earlier releases include a [bug](https://github.com/BingAds/BingAds-Python-SDK/issues/117) that generates large amounts of temporary files.

  ### Bug fixes

  * You can no longer create Teams using the Astro API in an Organization that has SCIM provisioning enabled.
  * Astro API responses have been standardized to always return cloud provider names in uppercase (for example, `AWS`).
</Update>

<Update label="February 13, 2024">
  ### New Astro reporting dashboards show metrics for Deployments across your Organization

  <Warning>This feature is in [Labs](/docs/astro/feature-previews). Please reach out to your customer success manager to enable this feature.</Warning>

  The new **Dashboards** page includes a suite of dashboards that you can use to asses the performance of Deployments and dags across your entire Organization. Each dashboard focuses on a different aspect of your data pipelines to show you opportunities for cost and performance improvements. You can additionally configure Astro to send you alerts when a given metric reaches a specific threshold. See [Organization Dashboards](/docs/astro/organization-dashboard) for summaries of each available dashboard.

  ### Additional improvements

  * When you submit a support request from the Astro UI, you must now define an **Active Engagement Period** when you or a member of your team can engage with a member of Astronomer support.
  * Workspace Members can now access the **Clusters** view in the Airflow UI for a Deployment.

  ### Bug fixes

  * Fixed an issue where network connections between clusters could be disrupted occasionally.
  * When you retrieve information about a Deployment through the Astro API, the API now returns an empty value for `EnvironmentVariables` if the Deployment has no environment variables.
  * Deleting a Workspace through the Astro API now deletes all Astro Cloud IDE projects associated with the Workspace.
  * Fixed an issue where you could not clear optional fields in a Deployment's configuration using the Astro API.
</Update>

<Update label="February 6, 2024">
  ### Bug fixes

  * Fixed an issue where all Deployment task logs included the error `Not exporting configs to configmap...`
</Update>

<Update label="January 30, 2024">
  ### New messages for Deployment health status

  <Warning>This feature is in [Preview](/docs/astro/feature-previews).</Warning>

  Astro now automatically monitors Deployments and notifies you when a Deployment isn't running as expected, such as when it can't detect a heartbeat in a scheduler. These notifications, known as Deployment incidents, appear in your Deployment's health status in the Astro UI.

  See [Deployment health incidents](/docs/astro/deployment-health-incidents) to learn more about each available incident type and how to address them.

  <Frame>
    <img alt="An example of an incident message in a Deployment health status" />
  </Frame>

  ### Additional improvements

  * You can now access [Ask Astro](https://ask.astronomer.io/) from the Astro UI **Help** menu:

    <Frame>
      <img alt="The support menu, accessed using the Help button in the top menu of the Astro UI" />
    </Frame>

  * User role titles are now consistently formatted across the Astro UI.
</Update>

<Update label="January 25, 2024">
  ### Self-service VPC peering and route management for AWS

  <Warning>This feature is in [Preview](/docs/astro/feature-previews).</Warning>

  You can now configure a network connection between Astro and an AWS VPC without contacting Astronomer support. Astro automatically handles creating a connection request and provides instructions for completing the setup yourself. After you create a VPC connection, you can configure routes whenever you need to connect to an additional service in your external VPC. See [Create a private connection between Astro and AWS](/docs/astro/connect-aws-vpc-peering).

  ### Additional improvements

  * There is no longer a minimum on the amount of CPU and memory that you can request for Kubernetes Pods.

  ### Bug fixes

  * Fixed an issue in Astro Hybrid where refreshing the browser could occasionally reset a worker queue's worker type setting.
</Update>

<Update label="January 16, 2024">
  ### Additional improvements

  * The Astro UI Deployment analytics page now shows **CPU Usage Per Pod (%)** and **Memory Usage Per Pod (MB)** as a percentage of your total available resources rather than the resources of a single worker Pod, such that these metrics will never show Deployment resources usage as exceeding 100%.
  * The maximum value for worker queue **Max # of workers** has increased from 30 to 100.
</Update>

<Update label="January 9, 2024">
  ### Additional improvements

  * When you create a Deployment, the Astro UI now shows the Airflow version of your Deployment instead of the equivalent Astro Runtime version.
  * Enabled point-in-time restore (PITR) on Hybrid GCP clusters to improve resiliency to outages. Note that this might result in increased costs for cloud storage.

  ### Bug fixes

  * Fixed an issue where you occasionally couldn't access the Airflow UI for a Deployment with the error "No healthy upstream".
  * Fixed an issue where deleting a user with no Workspace membership from an Organization would affect how their Workspace membership appeared in other, unrelated Organizations.
  * Fixed an issue where the **Open in Airflow** button on the dag details page in the Astro UI did not open the Airflow UI as expected.
</Update>


# 2025 Astro release notes
Source: https://astronomer.io/docs/astro/release-notes-2025

Astro release notes from 2025, covering features, bug fixes, and version updates for Astro, the Astro CLI, Astro Runtime, and the Remote Execution Agent.

<Tip>[Subscribe to Astro release notes](/docs/astro/release-notes-subscribe) to receive updates via RSS, email, or Slack.</Tip>

Astro release notes from 2025. See the [current release notes](/docs/astro/release-notes) for the latest updates.

<Update label="December 16, 2025">
  ### New Azure region available on Astro

  You can now create dedicated clusters in the following Azure region:

  * `germanywestcentral`

  ### Centralized access management for Organization Owners

  [Organization Owners](/docs/astro/user-permissions#organization-roles) can now manage user and Team access to Deployments and Workspaces directly from Organization Settings, making it easier to update roles across your Astro Organization.
</Update>

<Update label="December 9, 2025">
  ### Improved VPC networking for AWS dedicated clusters

  Astro now uses a new VPC network architecture for AWS dedicated clusters that increases scalability and reduces IP exhaustion risk by assigning Kubernetes Pod IPs to a secondary CIDR range. Source Network Address Translation (SNAT) ensures all external traffic appears from the primary VPC subnet, allowing you to use smaller VPC ranges and simplify private networking. This architecture is now the default for new clusters created through the Astro UI and can be enabled for existing clusters in coordination with Astronomer support.

  ### New User Details page for centralized user management

  You can now view and edit a member’s Organization, Workspace, and Deployment roles from a single place with the new **User Details** page. Team member names now link to this page, enabling centralized permission management. Team detail pages also allow you to manage Workspace and Deployment roles directly.
</Update>

<Update label="November 18, 2025">
  ### Connect to any Git project in the Astro IDE

  <Info>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Info>

  The [Astro IDE](/docs/astro/ide-overview) now supports importing projects from any Git provider, including GitHub, GitLab, Bitbucket, and Azure Repos. All Astro IDE version control and deployment features are available for connected repositories. For workflow details, see [Import a Git project to Astro IDE](/docs/astro/ide-import-git-project).

  ### Workspace-level environment variables

  You can now create and manage environment variables at the Workspace level using the Environment Manager in the Astro UI. This enables you to define environment variables once and share them across multiple Deployments within a Workspace, with the ability to override values per Deployment.

  When you create an environment variable in the Environment Manager, Astro stores it in an
  Astronomer-hosted secrets manager and applies it to Deployments as Kubernetes Secrets. You can configure environment variables to automatically link to all current and future Deployments in your Workspace, or link them to specific Deployments individually.

  Key capabilities include:

  * Share environment variables across multiple Deployments within a Workspace
  * Override environment variable values for individual Deployments
  * Auto-link environment variables to all Deployments by default
  * Use environment variables in branch-based deploys and PR previews
  * Access workspace environment variables in ephemeral test Deployments started in Astro IDE
  * Connect to secrets backends for secure, centralized management of sensitive values

  Environment variables set at the Deployment level take precedence over Workspace-level environment variables, allowing you to maintain both standard configurations and Deployment-specific customizations.

  See [Create environment variables in the Astro UI](/docs/astro/create-and-link-environment-variables) and [Environment
  variables overview](/docs/astro/environment-variables).

  ### Additional improvements

  * In [Astro Observe](/docs/astro/astro-observe), you can now see the last run status for dags and tasks, including those that started but haven’t completed their run.
  * [Astro Observe](/docs/astro/astro-observe) now summarizes task failures by default for SLA breach events caused by a dag failure.
</Update>

<Update label="November 12, 2025">
  ### Additional improvements

  * Updated Astro infrastructure to improve reliability and performance.
</Update>

<Update label="November 4, 2025">
  ### Triggerer chargeback and billing invoice UI

  Beginning November 1, Astronomer enables chargebacks for customers who request [additional compute](/docs/astro/deployment-resources#triggerer) for the Triggerer component. This update applies to all new organizations and most existing organizations. Additionally, a new **Billing -> Invoices** UI is available for all customers, providing improved visibility and transparency into billing details.

  <Frame>
    <img alt="Example of new invoice page" />
  </Frame>

  ### Proactive failure monitor for data products

  <Info>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Info>

  A new **Proactive Failure Monitor** for data products is now available in Preview in Astro Observe.
  This monitor automatically alerts your team whenever any upstream or final DAG in a data product fails, notifying you proactively before failures impact downstream data delivery or jeopardize SLAs.

  For more details and setup instructions, see [Create a data product monitor](/docs/astro/observe-monitors#data-product-monitors).

  <Frame>
    <img alt="Create Monitor button in Monitoring tab" />
  </Frame>

  ### Additional improvements

  * Astro Observe now supports searching for [data products](/docs/astro/create-data-products#create-a-data-product) by name, making it easier to find relevant data products in a large team or Organization.
</Update>

<Update label="October 29, 2025">
  ### Enable or disable Astro AI Organization-wide

  Astro now allows Organizations to self-service the enabling and disabling of AI features directly from Organization Settings. Organization Owners can access this new toggle in **Organization Settings → Organization Details** to manage whether Astro AI is enabled for the entire Organization. When **Astro AI** is disabled, all AI-driven features are turned off throughout the platform, including prompting features in the [Astro IDE](/docs/astro/ide-overview) and AI Log Summaries in [Astro Observe](/docs/astro/root-cause-analysis).
</Update>

<Update label="October 21, 2025">
  ### Airflow 3 to Airflow 2 rollback support

  Astro now supports rolling back Deployments from Airflow 3 to Airflow 2, expanding beyond the previous limitation of same-version rollbacks only. This enhancement provides a recovery path if you encounter critical issues after upgrading to Airflow 3. See the [Airflow 3 to Airflow 2 rollback requirements](/docs/astro/airflow3/upgrade-af3#airflow-3-to-airflow-2-rollback-support-and-requirements) for full Airflow 3 to Airflow 2 rollback requirements.

  ### Troubleshoot dag issues in Observe

  You can now troubleshoot dag failures in Astro Observe directly from the Astro dag list. From any dag, click **Open in Observe** to view recent runs, or see upstream and downstream dependencies in lineage. Use Observe to investigate specific runs by viewing the event timeline of task execution and [summarizing task failure logs](/docs/astro/root-cause-analysis), or assess the impact of a recent issue by seeing what [data products](/docs/astro/create-data-products#create-a-data-product) contain the dag.

  To learn more about Observe and get access, click [Observe](https://cloud.astronomer.io/observe) in the Astro UI main navigation.

  <Frame>
    <img alt="An example of the Observe dag run list." />
  </Frame>

  ### Additional improvements

  * In Observe, you can now filter the dag and task timelines by Airflow run to investigate a specific run.
  * In Observe, you can now see all data products an asset belongs to in the **Data Products** tab to assess downstream impacts.
</Update>

<Update label="October 16, 2025">
  ### Additional improvements

  * Added the **Never Wake** wake schedule type, allowing Deployments to hibernate immediately and remain hibernated indefinitely.
  * Improved configuration defaults for Airflow 3 API Servers to improve performance, reliability, and resource usage.
</Update>

<Update label="October 1, 2025">
  ### Quick start Deployment creation flow

  A new **Quick Start** mode is available for Deployment creation in the Astro UI. This enhancement introduces template-based Deployment setup, enabling Deployment launches with a single click by selecting from predefined templates for Development, Pre-Production, and Production environments. Switch to advanced Custom Settings for full configuration control. This improvement streamlines the Deployment experience and ensures best-practice configurations are accessible by default. See [Create a Deployment](/docs/astro/create-deployment).

  ### Enhanced Wake Schedule section

  The **Wake Schedules** section has been enhanced and now replaces the previous **Hibernation Schedules**. **Wake Schedules** are available when **Development mode** is enabled. Multiple schedules can be defined for each Deployment, supporting precise timezone, day, and hour targeting for efficient resource utilization and improved cost management. See [Hibernate a Deployment](/docs/astro/deployment-resources#hibernate-a-development-deployment).

  ### Enhanced Authentication for Metrics Export with SigV4Authorization

  Universal Metrics Export now supports SigV4Authorization for AWS Deployments on dedicated clusters. You can use IAM role-based authentication to securely connect to Amazon Managed Prometheus without managing static credentials.

  In the Metrics Export configuration, select **SigV4Authorization** as your authentication type to enable secure, role-based access to your AWS observability services. Configure your AWS Role and Region, then copy the auto-generated trust policy from the new **Trust Policies** tab to complete the setup in your AWS account.

  This authentication method streamlines credential management while following AWS security best practices, making it easier to export Astro metrics to Amazon Managed Prometheus and other AWS-based observability tools.

  ### Additional improvements

  * Updated the Observe Lineage Graph to center on expanded nodes when clicking to expand. Expanding a node now selects and centers the expanded node, improving usability for navigation with selected assets.
  * Synced the run status highlighter’s date range control in Observe Lineage with the asset event timeline panel. The event timeline now reflects the highlighted date range selection.
  * Added a **Final Assets Only** toggle on the Data Product Assets page, allowing you to easily filter and focus on key final assets.
  * Removed the **All DAGs** option from DAG Timeliness alerts to prevent configuration errors and ensure alerts target specific dags.
  * The IP Access List form now supports IPv6 addresses.

  ### Bug fixes

  * Fixed an issue in Observe Lineage Impact Analysis where the downstream asset list did not respect pagination page size.
  * Fixed an issue in the Observe lineage view where entering an invalid date would crash the page.
  * Fixed an issue in the Data Product Lineage Graph where the view would auto-recenter on a selected asset every 5 seconds.
</Update>

<Update label="September 17, 2025">
  ### Build pipelines with the Astro IDE

  <Note>
    **Preview**

    This feature is in [Preview](/docs/astro/feature-previews).
  </Note>

  Astro IDE delivers a browser-based workspace built for Apache Airflow development, testing, and deployment—without requiring local setup. The IDE streamlines dag authoring with project-aware, context-driven AI assistance and allows real-time testing in isolated, ephemeral environments. You can deploy Airflow code directly to Astro or GitHub, import projects from GitHub or the Astro CLI, and enforce coding standards with custom project rules. For details on core workflows and features, see [Astro IDE overview](/docs/astro/ide-overview), [authoring dags](/docs/astro/ide-author-dags), [testing code](/docs/astro/ide-test-run), [deploying code](/docs/astro/ide-deploy), and [importing projects](/docs/astro/ide-import-github-project).

  ### Improved task eviction procedures

  The Astro platform has improved how Kubernetes manages and responds to container out of memory (OOM) situations, specifically with Celery workers. Previously, a single task running on a Celery worker could cause an OOM issue and termination of the entire pod resulting in:

  * Killing all other tasks running on that container (zombie tasks)
  * Losing task logs for both problematic tasks and all other tasks

  These platform changes reduce the number of zombie tasks and improve the availability of task logs for both running and killed OOM task processes.

  By improving how Astro handles zombie tasks and task logs, this change helps you identify high memory consuming tasks and/or tasks with incorrectly configured concurrency. You can use this information to self-serve solutions like increasing worker size, splitting tasks into larger workers, or reducing worker concurrency.

  ### Additional improvements

  * Added support for creating the Remote Execution Agent Token with the Astro API, instead of exclusively through the Astro UI. See the Astro [Platform API changelog](https://www.astronomer.io/docs/astro/api/v-1-beta-1/platform/changelog/2025/9/17), [IAM API changelog](https://www.astronomer.io/docs/astro/api/v-1-beta-1/iam/changelog#2025-09-17-summary), and the [Remote Execution Agent](/docs/astro/remote-execution-configure-agents) docs.
</Update>

<Update label="August 19, 2025">
  ### Bug fixes

  * Fixed a bug where Dag Processors would inherit the default of two parsing processes unless you manually set the environment variable `AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES`.
</Update>

<Update label="August 14, 2025">
  ### Added Triggerer metrics to Deployment analytics

  Added support for [Airflow triggerer metrics and dashboards](/docs/astro/deployment-metrics#airflow-triggerer). You can now view pod count, CPU and memory usage per triggerer Pod, including maximum and average values, triggers per status over time, and running triggerers as timeseries in the **Deployment Analytics** dashboard. These metrics are also available for export with UME.

  ### Additional improvements

  * Added asset search and enhanced navigation controls to the **Observe Lineage Graph**. You can now search for assets, center the graph on a selected asset, and benefit from improved zoom and pan controls.
  * You can now see the Remote Execution Agent **Version** in the Remote Agents Table.

  ### Bug fixes

  * Fixed an issue in the Data Product form where editing an existing Data Product did not reliably load and display the current selections.
</Update>

<Update label="July 30, 2025">
  ### Additional improvements

  * The [Universal Metrics Exporter](/docs/astro/export-metrics) can now export the `kube_resourcequota` infrastructure metric. Additionally, `scheduler` and `dag-processor` are now included as containers in the `kube_pod_container_resource_limits` metric.
  * In Observe, all references to *Graph* are renamed to *Lineage*, including URLs and UI labels, for consistency.
</Update>

<Update label="July 16, 2025">
  ### Additional improvements

  * **Clusters** and all sub-pages have been moved from the top-level navigation into **Organization Settings**.
</Update>

<Update label="July 8, 2025">
  ### Additional improvements

  * You can now search for Deployments within the **Connections**, **Metrics Exports**, and **Airflow Variables** pages. This improves navigation and filtering for linked Deployments.

  ### Bug fixes

  * Fixed a bug where switching between executors caused Deployment updates to fail with a `default worker queue is required` error. The required queue is now automatically recreated during executor changes.
  * Removed non-functional search bars from the **Connections**, **Metrics Exports**, and **Airflow Variables** pages. These will be reintroduced once backend support is implemented.
</Update>

<Update label="July 2, 2025">
  ### Enhanced Support Access

  Enhanced Support Access is now enabled by default for all Organizations to ensure faster, more effective assistance from the Astronomer Support team. This feature grants **Read-only** Admin access to your Organization’s details, allowing the Astronomer Support team to troubleshoot issues in real time and provide premium-level support. Support does not have access to make any changes to your environment, and you can always view access activity through [Audit logs](/docs/astro/audit-logs). See [Enhanced security access](/docs/astro/user-permissions#enhanced-support-access) for more information.

  ### Additional improvements

  * Added Astro API support for Airflow 3 Deployments. See [Platform API changelog](https://www.astronomer.io/docs/astro/api/v-1-beta-1/platform/changelog#2025-07-30-summary).
  * Added a **Applied Commit Or Credit ID** column to the billing invoices table within Organization settings.
  * You can now click rows in the **Commits and Credits** table to view detailed transaction history directly in the Astro UI.
  * Improved the Dag list by adding status icons and enabling sorting by next run time, with the default showing the next-to-run dag at the top.

  ### Bug fixes

  * Fixed a bug in where you could not update the **Values** field for Astro Alerts.
  * Fixed a bug where the Observe Lineage Graph failed to render nodes.
</Update>

<Update label="June 24, 2025">
  ### Additional improvements

  * In Runtime 11.15.0 and later, Kubernetes executor workers no longer fail immediately on `ErrImagePull` events. Instead, they retry pulling the image until the `task_queued_timeout` is reached. If the image pull still fails, the task is requeued.
  * Astro now shows an error page with a redirect option when navigating to deleted or non-existent resources.

  ### Bug fixes

  * Fixed a bug where Remote Deployments could be created and updated with a non-small scheduler size.
</Update>

<Update label="June 17, 2025">
  ### Bug fixes

  * Fixed a bug where characters were lost when typing quickly in the **Filter by Dag Name** input field.
  * Fixed an issue in Astro Executor where logs for previous task attempts on Hosted execution mode Deployments were not retrievable.
</Update>

<Update label="June 11, 2025">
  ### Improved resilience of Airflow UI and API for non-HA Deployments

  Improved reliability for non-HA Deployments by ensuring there's minimal traffic disruption during Pod restarts. With better coordination between shutdown and startup, traffic continues to flow even during infrastructure events like evictions, significantly improving reliability for non-HA customers. These enhancements apply to interactions between both your dags and the Airflow UI and your dags and the API, leading to a 95% availability improvement.

  ### Additional improvements

  * Updated the Azure Workload Identity input validation patterns to match the officially allowed characters.

  ### Bug fixes

  * Fixed a bug in Observe where the namespace for OpenLineageDatasets in the assets view was incorrectly showing the namespace of the Deployment instead of the namespace of the OpenLineageDataset.
  * Fixed a bug where the **Bucket URL** option for Azure Remote Deployments was not working as expected.
  * Fixed a bug that prevented scrolling in the **Create/Update Connections** modal.
</Update>

<Update label="June 3, 2025">
  ### Metrics Exports logs now available in Astro

  You can now view error logs for **Metrics Exports** connections at both the Deployment and Workspace level in Astro to quickly identify and troubleshoot connection issues. For common errors, a **Resolution** column provides guidance on how to resolve them. See [Metrics Exports error logs](/docs/astro/export-metrics) for more information.

  ### Additional improvements

  * You can now create Dedicated clusters in `europe-west6` on GCP.

  ### Bug fixes

  * Fixed scheduler size default for Remote Execution Deployments in the Astro UI. Previously, when switching to the Remote Execution mode after selecting the Celery or Kubernetes executor, the scheduler size incorrectly remained set to `medium`. The scheduler size now correctly defaults to `small` when Remote Execution mode is selected.
</Update>

<Update label="May 27, 2025">
  ### Additional improvements

  * Updated Astro infrastructure to improve reliability and performance.
</Update>

<Update label="May 22, 2025">
  ### New AWS regions available on Astro

  You can now create dedicated clusters in the following AWS regions:

  * `ca-central-1`

  ### Additional improvements

  * Improved Deployment creation experience so that when you change the **Executor** it does not reset **Worker Queues** values.

  ### Bug fixes

  * Fixed a bug where Astro Executor Deployments could assign multiple task allocations if multiple identical dag run IDs exist for multiple dags.
  * Fixed a bug where the incorrect auth manager config was set, breaking the webserver for some Runtime versions.
  * Fixed a bug in the Astro UI where back buttons didn’t behave as expected. Now, if you make page filtering changes, the back button no longer reverts those change. Instead, it goes to the previous entire page.
</Update>

<Update label="May 13, 2025">
  ### Bug fixes

  * Fixed a bug in the **Astro Observe Asset Catalog** where namespaces would show up as `n/a` if Observe could not find the respective Hosted Deployment.

  * Fixed a bug where the **Open Airflow** button on the Deployment Details page was clickable even if Astro was still creating the Deployment.
</Update>

<Update label="May 6, 2025">
  ### Additional improvements

  * Added the ability to switch between a table view for CPU Usage Per Pod and Memory Usage Per Pod for Deployment analytics.

  ### Bug fixes

  * Fixed a bug where the Kubernetes Executor Deployments ignored the Airflow logging level for the Airflow Scheduler.
  * Fixed a bug where the Astro UI truncated displayed values for the Airflow Variable and the Airflow Override Variable.
</Update>

<Update label="April 29, 2025">
  ### New AWS regions available for Standard clusters

  You can now create Deployments on a Standard cluster in the following AWS regions:

  * `eu-west-1`

  ### Bug fixes

  * Fixed a bug where Airflow Asset linking was broken in Observe.
  * Fixed a bug in the **Deployment Analytics** Pod metrics charts where utilisation appeared to be greater than 100% of the resource limit.
  * Fixed a bug where the **Code** tab in the Airflow UI on Hosted Execution mode Deployments showed a Remote Execution warning instead of the source code.
</Update>

<Update label="April 22, 2025">
  ### Introducing Airflow 3 on Astro

  Apache Airflow 3 introduces a suite of [new features](/docs/astro/airflow3/features-af3) such as dag versioning, event-driven scheduling, advanced inference execution, a redesigned modern UI, and high-performance backfills. A new distributed architecture decouples task execution from direct database connections, enhancing security and operational agility for your mission-critical data pipelines.

  Astro delivers Apache Airflow 3 capabilities on Day Zero in an enterprise-ready, fully managed, secure and auto-scaling platform, supported by Airflow experts and code committers.

  To support your Airflow 3 work on Astro, see the [Astro CLI 1.34](/docs/cli/archive) release notes, [Runtime 3.0-1](/docs/runtime/runtime-release-notes#astro-runtime-3-0-1) release notes, and [Airflow 3 blog post](https://airflow.apache.org/blog/airflow-three-point-oh-is-here/).

  See [Upgrade to Airflow 3](/docs/astro/airflow3/upgrade-af3) for steps to upgrade your Astro project from Apache Airflow 2 to Apache Airflow 3.

  ### Remote Execution Mode on Astro

  <Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

  For enterprises needing the highest level of security, Astronomer offers Remote Execution Agents on Astro as a premium managed orchestration solution. This approach keeps data, code, and secrets confined to your environment, letting only scheduling and health data travel to Astro’s Orchestration Plane. By leveraging Astro’s Remote Execution Agents, companies meet strict regulatory requirements and maintain data locality while benefiting from Airflow 3’s full feature set.

  See [Execution modes](/docs/astro/execution-mode) and [Remote Execution Agents](/docs/astro/remote-execution-configure-agents) for more information.

  ### Astro Executor

  The [Astro Executor](/docs/astro/astro-executor) is an exclusive feature on Astro that’s optimized for throughput and latency, ensuring your data workflows perform efficiently. Combined with straightforward in-place upgrades, you get to experience Airflow 3 benefits with Astro's proven reliability.

  ### Dag Versioning on Astro

  Airflow 3’s dag versioning ensures that each pipeline run references its exact code snapshot, enabling complete historical traceability.

  On Astro Hosted Deployments, dag versioning is handled seamlessly and integrated with your existing deploy mechanisms and requires no additional configuration.

  See [Dag versioning](/docs/astro/dag-versioning).

  ### Additional improvements

  * Code deployment architecture has been simplified for improved reliability.
  * Added the new custom Deployment role template, [Observe Ingest](/docs/astro/customize-deployment-roles#deployment-role-templates). This allows you to create Deployment API Tokens scoped to only ingest OpenLineage observability data.
</Update>

<Update label="April 15, 2025">
  ### Deprecated dag details view

  The dag details view replicated information readily available in the Airflow UI. Deprecating the dag details view enables Astronomer to invest in enhancing the core Airflow experience for all users.

  ### Additional improvements

  * You can now see a breakdown of task metrics such as **Failed Task Runs**, **Task Retry Rate**, and **Long-Running Tasks**  on the Observe Health Dashboard.
</Update>

<Update label="April 8, 2025">
  ### New Azure region available for Dedicated clusters

  You can now create Deployments on a Dedicated cluster in the following Azure region:

  * `uaenorth`

  ### Additional improvements

  * Updated the Observe Timeline filters to only show entities that are relevant to what is currently displayed on the timeline.
  * In the Observe Health dashboard, added links to entity names in the **Alerts Triggered** and **SLAs Missed** sections.
  * Removed the **Historical Landing** and **Current Lookback** charts from the **Data Product Overview** tab. A revision of this page is in the works.
</Update>

<Update label="April 2, 2025">
  ### Additional improvements

  * Combined the **Sent** and **Failed** counts of the Triggered Alerts section in the Observe Health dashboard into a summed **Triggered** count.
  * The **High Availability** toggle is now in the **Basic** section of Deployment details. Astronomer recommends enabling High Availability for production Deployments.

  ### Bug fixes

  * Fixed a bug where the `deployment.get` permission wasn't being enforced when accessing **Environment Variables** for Deployments.
  * Fixed a bug where `AIRFLOW__SCHEDULER__TASK_QUEUED_TIMEOUT` was not respected in user Deployment configurations.
  * Fixed an issue where upon selecting a value from the Event Type Status selector, you would not be able to close the menu by clicking outside of the menu.
</Update>

<Update label="March 19, 2025">
  ### Break down cost per task for expensive dags in Observe

  In Astro Observe, you can now see the cost per task (in Snowflake credits) of your most expensive dags. Hover over any dag in the **Most Expensive DAGs** graph to see a dynamic view of the most expensive tasks in that dag. Click into any task to drill down into cost metrics over time and begin optimizing to save costs.

  To learn more about Observe and get access, click [Observe](https://cloud.astronomer.io/observe) in the Astro UI main navigation.

  ### Additional improvements

  * In Astro Observe, added an **SLA Consumption Metric** in the **Alerts & SLAs** section.
  * In Astro Observe, the **Event Timeline** list now uses Entity Name instead of ID if the Name is available.
</Update>

<Update label="March 11, 2025">
  ### Alert rules enhancements

  For dag and task-level Alerts, you can now specify which dags and tasks the alert applies to by matching against a string. Use matching operators such as `is one of` for exact matches and `contains` for partial matches within a dag or task ID. You can also select `All Dags` for all dags in a Deployment and `All Tasks` for all tasks in the defined dag(s). See [Astro alerts](/docs/astro/alerts) for more information.

  ### Data Pipelines Health Dashboard improvements in Astro Observe

  In Astro Observe, the Data Pipelines Health Overview dashboard now supports drilling down into Cost Management and Alert Graphs. In Cost Management section, click on the most expensive dags or Tasks to see their underlying costs over time to pinpoint costly issues and understand where to optimize.

  The Alerts graph now shows a list of alerts that fired in the reporting window and links to the alert’s notification history. For example, to investigate an SLA breach alert that fired far more often than expected, you can go to the alert’s notification history to pinpoint the specific time range when the alert failed.

  To learn more about Observe and get access, click [Observe](https://cloud.astronomer.io/observe) in the Astro UI main navigation.

  ### Additional improvements

  * In Astro Observe, the Lineage graph now displays a detail view of individual assets. Click on any asset (like an Airflow task or dataset) to open the detail view, which includes metadata like a task’s operator class, a link to open the asset in Airflow, and the reason for inclusion in the lineage graph.
</Update>

<Update label="March 6, 2025">
  ### Cloud IDE is now deprecated

  The Cloud IDE has been officially deprecated. Stay tuned for a next-generation dag authoring experience!

  ### [Ask Astro](https://ask.astronomer.io/) enhancements

  * You can now log in to Ask Astro using your existing Astro account or create a new one to submit private questions.
  * Answers are now streamed back in real time, making the experience quicker and more responsive.
  * A specialized dag generation experience has been introduced.

  ### Additional improvements

  * Added the ability to work with Astro Alerts and their Notification Channels through the Astro API. See [Astro API changelog](https://astronomer.io/docs/api/platform/changelog) for more information.
  * Added an [Astro Observe](/docs/astro/astro-observe) asset sidebar so you can preview an asset's metadata without leaving the graph.
  * You can now choose the **Team** plan in addition to the **Developer** plan for monthly "pay-as-you-go" billing. This update gives you the ability to self-service to the plan that best suits your needs.
  * Updated the default scheduler size for Deployments to `medium` so that new Deployments include a separate Dag processor by default.

  ### Bug fixes

  * Fixed a bug where metric charts failed to populate for queues with long names.
  * Fixed a bug where the Deployment connections table only showed a maximum of 20 items.
  * Fixed a bug where changing tabs in Data Product and Asset Catalog Metrics would reset the currently selected time range.
  * Fixed a bug where environment variable secrets would toggle when the label was clicked.
</Update>

<Update label="February 25, 2025">
  ### Additional improvements

  * Updated Astro infrastructure to improve reliability and performance.
</Update>

<Update label="February 19, 2025">
  ### Astro Observe now generally available

  Observe is Astronomer's comprehensive observability solution for [Apache Airflow®](https://airflow.apache.org/) that provides visibility into the health and performance of your data pipelines. Using Observe, you can create data products to monitor performance of key data assets and set SLAs that alert on data product timeliness and freshness. With this GA release, you can now:

  * Visualize real-time health and cost metrics across your data pipelines in a single place with the [pipeline health dashboard](/docs/astro/astro-observe#health)
  * Connect Snowflake cost trends to reliability metrics to pinpoint inefficiencies across your dags and data products
  * Reduce mean time to resolution with [targeted root cause analysis](/docs/astro/root-cause-analysis) that identifies upstream issues that caused SLA breaches or failures
  * Understand failures at a glance with AI Log Summaries, which provide a human-readable explanation of Airflow task logs, including what went wrong, where to look in order to fix it, and how to prevent it from happening again
  * Proactively mitigate risks to data product health and increase alerting coverage with [Observe Insights](/docs/astro/create-data-products#overview)

  To learn more about Observe and get access, click [Observe](https://cloud.astronomer.io/observe) in the Astro UI main navigation.

  To get started using Observe with a step by step tutorial, check out the [Observe Quickstart](/docs/learn/astro-observe-quickstart).

  ### Notification History now available for Astro Alerts

  You can now see a historical log of all previously triggered Alerts for your Organization, and filter them based on status, time range, notification channel, and the Alert that triggered the notification. This lets you see a historical view of issues across deployments and data products, like SLA breaches, dag and task failures, and Deployment health incidents.

  To see the history of all alerts across your Organization, click **Notification History** in the **Alerting** section of the navigation. To see the history for a particular alert, go to the **Alerts** page, where you can see the number of times an alert was sent or failed to send. Click on the number in the **Sent** column to see alert notifications sent for that alert.

  ### Bug fixes

  * Fixed a bug that disabled editing of all existing environment variables instead of just existing secrets.
</Update>

<Update label="February 11, 2025">
  ### Bug fixes

  * Fixed a bug affecting the Deployment environment variables create and update form in the Astro UI.
  * Fixed a bug that caused the self-healing worker to prematurely kill Deployments when the dag downloader took a longer time than expected to start working. Now, Astro has updated logic that accounts for slow starting workers for Self-Healing Workers.
</Update>

<Update label="February 6, 2025">
  ### Additional improvements

  * Updated Astro infrastructure to improve reliability and performance.
</Update>

<Update label="January 28, 2025">
  ### Bug fixes

  * Fixed a bug where changes to the payment method were not reflected in the Astro UI.
</Update>

<Update label="January 22, 2025">
  ### High Availability Deployments now run two webservers by default

  High Availability Deployments in Astro Hosted now run two webservers by default, improving Airflow API availability during events like node consolidation and image deploys.

  ### Bug fixes

  * Fixed a SCIM integration issue that prevented users from pushing all their teams to Astro when an Organization had more than 100 teams.
</Update>

<Update label="January 13, 2025">
  ### Region based pricing now visible in Billing Dashboard

  Now, when you view your **Invoices**, you can see the region your cluster is based in and the cloud provider. See [Manage Astro billing](/docs/astro/manage-billing).
</Update>

<Update label="January 9, 2025">
  ### New Azure regions available for Standard clusters

  You can now create deployments on a Standard cluster in the following Azure regions:

  * `australiaeast`

  ### Additional improvements

  * Organized Alert types into categories instead of displayed in one list, so you can more easily find a particular alert within its type.
</Update>


# Subscribe to release notes
Source: https://astronomer.io/docs/astro/release-notes-subscribe

Subscribe to Astronomer Release Notes RSS feeds, or have updates sent to your email or Slack account.

You can subscribe to RSS feeds for Astro release notes, Astro CLI release notes, and Astro Runtime release notes. You can receive RSS feeds directly, or updates can be sent to your email or Slack account.

## Subscribe to RSS email updates

To have RSS feed updates sent to your email, use a browser RSS feed reader extension such as the [Chrome RSS Feed Reader](https://chrome.google.com/webstore/detail/rss-feed-reader/pnjaodmkngahhkoihejjehlcdlnohgmp?hl=en).

<Note>Safari does not natively support viewing or configuring RSS feeds on Desktop or Mobile.</Note>

## Subscribe to Slack updates

To add a feed to a public channel, the Slack Primary Owner role might be required. To learn more about adding RSS feeds to Slack, see [Add RSS feeds to Slack](https://slack.com/help/articles/218688467-Add-RSS-feeds-to-Slack).

Run the following Slack slash commands to subscribe to different Astro release notes:

* Astro product release notes: `/feed subscribe https://www.astronomer.io/astro-release-notes.xml`
* Astro Runtime release notes: `/feed subscribe https://www.astronomer.io/runtime-release-notes.xml`
* Astro CLI release notes: `/feed subscribe https://www.astronomer.io/cli-release-notes.xml`
* Astro Remote Execution Agent release notes: `/feed subscribe https://www.astronomer.io/astro-agent-release-notes.xml`


# Resilience
Source: https://astronomer.io/docs/astro/resilience

Learn how Astronomer leverages Availability Zones to make the control plane and data plane resilient.

The Astro control and data planes are architected and deployed on major public clouds to take advantage of their resilient and highly available regions. Regions provide multiple physically separated and isolated Availability Zones (AZs) which are connected through low-latency, high-throughput, and highly redundant networking within a geographic region. Additionally, each AZ has independent power, cooling, and physical security.

Astro leverages AZs for both the control and data planes. The control plane leverages 3 AZs per region, while the data plane leverages 2 AZs per region on AWS, and 3 AZs per region on GCP and Azure. Control planes and data planes are also segregated by cloud providers. As a result, both planes are expected to survive and recover from an AZ outage, though they may experience some degradation until the impacted resources are re-provisioned/promoted to a non-impacted AZ.

Astronomer utilizes automated backup features on public clouds. If a region experiences a partial outage or some other incident that affects your clusters, Astronomer can use these backups to restore all details of your cluster, including the metadata database.

In the case of a full region failure of the control plane, the services and data would be recovered in an alternate region by Astronomer. For dedicated clusters on AWS and GCP, Astro supports self-service cross-region disaster recovery (DR), which keeps a secondary cluster continuously synchronized with your primary cluster and allows you to fail over with minimal downtime. See [Disaster recovery](/docs/astro/disaster-recovery) for details and setup instructions.

Astro also provides DDoS protection with always-on traffic monitoring, detection, and automatic attack mitigation for all inbound network traffic, along with suspicious IP throttling and brute-force protection as part of our secure authentication service.


# Secrets management
Source: https://astronomer.io/docs/astro/secrets-management

Learn how Astronomer secures your sensitive information and supports secrets management integration

As the modern data orchestration service, Astro has been built and deployed with security as a guiding architectural principle. This same principle extends into how your sensitive information and credentials are stored and secured. Astro includes a [managed secrets backend](/docs/astro/manage-connections-variables#astro-environment-manager) for secure value encryption and storage, plus [integration with popular secrets management tools](/docs/astro/secrets-backend).

All secrets management configuration performed in the Astro UI is [securely transmitted and stored](/docs/astro/data-protection), is [resilient](/docs/astro/resilience) to in-region cloud failures, and can be [recovered](/docs/astro/disaster-recovery) in the case of a full control or data plane disaster.


# Security in Astro
Source: https://astronomer.io/docs/astro/security

Learn how Astro responds to and implements a variety of security concepts

Astro is a fully managed data orchestration service that allows you to run your data pipelines in your public cloud account on Amazon Web Services (AWS) or Google Cloud Platform (GCP), respecting the need to keep your data private, secure, and within corporate boundaries.

The Astro architecture is secure by default, using encryption in transit, encryption at rest, strong cryptographic protocols, authentication, and role-based access control for authorization to your data pipelines, with a host of flexible and secure connectivity options to your critical data sources. This page serves as a summary of all Astro features that ensure the security and reliability of your systems.

## Shared responsibility model

Astro operates on a model of shared responsibility, which means that both the Astronomer team and Astronomer customers are responsible for the security of the platform. For more information, see [Shared responsibility model](/docs/astro/shared-responsibility-model).

## Architecture

Astro utilizes a deployment model where the core components for managing Airflow are hosted in Astronomer's cloud, allowing you to run Airflow with as little friction as possible and securely connect to your data infrastructure. For more information, see [Architecture](/docs/astro/astro-architecture).

## Resilience

The Astro control and data planes are architected and deployed on major public clouds to take advantage of their resilient, secure and highly available regions. Additionally, both planes are designed and architected to take advantage of best-in-class security products offered by the public clouds. For more information, see [Resilience](/docs/astro/resilience).

## Disaster recovery

While Astro data plane is designed to withstand and survive in-region Availability Zone (AZ) degradations and outages, rest assured Astronomer can also help you recover from major region failures by restoring configuration and secrets from a secure and highly available data store. For more information, see [Disaster Recovery](/docs/astro/disaster-recovery).

## Physical and environment security

Astro leverages all three major public cloud providers (Azure, Google Cloud Platform, Amazon Web Services), thus physical and environmental security is handled entirely by those providers. Each cloud service provider provides an extensive list of compliance and regulatory assurances that they are rigorously tested against, including SOC 1/2-3, PCI-DSS, and ISO27001. For more information, see [Cloud provider security responsibilities](/docs/astro/shared-responsibility-model#cloud-provider-security-responsibilities).

Astronomer is a global remote company first. The Astronomer offices in the United States are treated as trustless. Employees need to authenticate to all applications and systems using Okta with multi-factor authentication (MFA) when using the office WiFi.

## Data privacy and compliance

Astro is compliant with AICPA SOC 2 controls with respect to the security, availability, and confidentiality Trust Service Categories. To obtain reports related to Astronomer's compliance, such as a SOC 2 Type 2 Report or a Penetration Test Report, visit the [Astronomer Trust Center](https://trust.astronomer.io/).

Astronomer is also [GDPR-compliant](/docs/astro/gdpr-compliance) as an organization, and the Astro platform is GDPR-ready. Astronomer offers a Data Processing Agreement (DPA), which satisfies the requirements the GDPR imposes on data controllers with respect to data processors.

For organizations operating with protected health information, Astronomer is [HIPAA-compliant](/docs/astro/hipaa-compliance) as an organization, and the Astro platform is HIPAA-ready.

Additionally, for organizations processing payment card information, Astro is certified as compliant with PCI DSS security standards.

## Data protection

Astro uses both encryption in transit and encryption at rest to protect data across and within the Data and control planes, using strong and secure protocols and ciphers, and built-in public cloud features. For more information, see [Data protection](/docs/astro/data-protection).

## Secrets management

Astro is designed to secure sensitive information about your external systems. Sensitive information shared with Astronomer is securely stored and transmitted for consumption by your data pipelines. For more information, see [Secrets management](/docs/astro/secrets-management).

## Patch management in Astro Runtime

Astronomer continuously checks for available security fixes for all software used in Astro Runtime and is committed to delivering these fixes in a timely manner. For more information, see [Security fixes](/docs/runtime/runtime-version-lifecycle-policy#security).


# Shared responsibility model
Source: https://astronomer.io/docs/astro/shared-responsibility-model

Astronomer's policy on shared responsibilities between our team and our customers.

Astronomer's highest priority is the security and reliability of your tasks. As an Astro customer, you benefit from a fully-managed data orchestration platform that meets the requirements of the most security-sensitive organizations.

Astro operates on a model of shared responsibility, which means that both the Astronomer team and Astronomer customers are responsible for the security of the platform. This document specifies areas of security ownership for both Astronomer customers and the Astronomer team.

## Astronomer's responsibilities

Astronomer is responsible for providing a secure and reliable managed service offering, including:

* Managing the control plane and core services (Astro UI, Cloud API, Deployment Access, and Cloud image Repository).
* Securing authentication and authorization to all interfaces (UI, API, and CLI).
* Automating provisioning, scaling, and configuration management of Astro resources in the data plane.
* Completing ongoing maintenance (currency, hardening, patching) and uptime monitoring of Astro resources in the data plane. For example, Kubernetes cluster upgrades.
* Maintaining data encryption (at rest/in flight) of Astro managed components (control and data planes).
* Consistently releasing production-ready and supported distributions of [Astro Runtime](/docs/runtime/upgrade-astro-runtime) for net-new and to-be-upgraded Deployments.
* Upon customer request, execute [Disaster Recovery](/docs/astro/disaster-recovery) procedure for dedicated Hybrid or Hosted clusters.

## Customer's responsibilities

The customer is responsible for managing certain security aspects of their Astro Organization and Deployments, including:

* Managing roles and permissions of users and API tokens within their organization and Workspace(s).
* Storing and retrieving [authentication tokens](/docs/astro/automation-authentication), connections, and [environment variables](/docs/astro/environment-variables) for data pipelines.
* Integrating with their federated identity management platform for secure single sign-on (SSO) authentication with multi-factor authentication (MFA) and customer managed credentials.
* Developing and maintaining data pipelines with security and quality coding best practices, inclusive of vulnerability management of plugins and dependencies.
* Regularly [upgrading their Deployment(s)](/docs/runtime/upgrade-astro-runtime) to the latest Astro Runtime version to take advantage of new functionality, as well as bug and security fixes.
* [Configuring and managing Deployment resource settings](/docs/astro/deployment-resources) for data pipeline workloads.
* Securing the network communications between their data plane and sensitive data resources.

## Cloud provider security responsibilities

Physical and environmental security is handled entirely by our cloud service providers. Each of our cloud service providers provides an extensive list of compliance and regulatory assurances that they are rigorously tested against, including SOC 1/2-3, PCI-DSS, and ISO27001.

### Azure

See the Azure [compliance](https://azure.microsoft.com/en-ca/overview/trusted-cloud/compliance/), [security](https://azure.microsoft.com/en-ca/overview/security/), and [data center security](https://azure.microsoft.com/en-ca/global-infrastructure/) documentation for more detailed information.

### Amazon

See the AWS [compliance](https://aws.amazon.com/compliance/), [security](https://aws.amazon.com/security/), and [data center security](https://aws.amazon.com/compliance/data-center/controls/) documentation for more detailed information.

### Google

See the GCP [compliance](https://cloud.google.com/security/compliance), [security](https://cloud.google.com/security), and [data center security](https://cloud.google.com/security/infrastructure) documentation for more detailed information.


# Add labels to Pods created by the Astro Private Cloud Helm chart
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/add-podlabels

Configure Astro Private Cloud to apply labels to all Pods created by Astro Private Cloud.

If your organization enforces Pod labeling for security, workload identity, multi-tenancy, or policy enforcement, you can configure Astro Private Cloud to automatically add labels to all Pods it creates during installation or upgrades.

## Prerequisites

* You must have [System Admin permissions](/docs/astro-private-cloud/v-2-x/role-permission-reference#system-admin) to perform an upgrade or installation.

## Configure Pod labels

In your Helm chart, add the `podLabels` configuration to the `values.yaml` file during your update or installation process.

```yaml wrap theme={null}
global:
    podLabels:
        key: "value"

```

For example, you can use the label `security.level: "high"` as a way to identify Pods created by Astro Private Cloud using the following code:

```yaml wrap theme={null}
global:
    podLabels:
        security.level: "high"
```

## Find labeled Pods

You can search for Pods using the kubectl command `get pods` with the label flag. For example, to retrieve a list of all Pods labeled with the key-value pair, `security.level=high`, you can use the following command:

```bash wrap theme={null}
kubectl -n astronomer get pods -l security.level=high
```


# Adopt Astro Runtime Operator managed Deployments (Preview)
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/adopt-operator-deployments

Bring Airflow Deployments you already run with the Astro Runtime Operator under Astro Private Cloud management, without recreating them.

If you already run Airflow with the Astro Runtime Operator, you can bring those Deployments under Astro Private Cloud (APC) management by **adopting** them. Your Airflow keeps running. Nothing is recreated, and the operator continues to reconcile the underlying resources. After adoption, you manage the Deployment from the APC control plane like any other: deploy code, change environment variables, resize components, and view metrics and logs.

<Note>
  **Astro Private Cloud 2.1**

  This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
</Note>

<Info>
  **Preview**

  Adoption is in Preview because APC does not yet manage everything the Astro Runtime Operator can express. An adopted Deployment keeps some of its settings under the operator rather than under APC, and operator mode itself is still missing some Helm-mode features. See [What adoption changes](#what-adoption-changes) for the ownership split, and [Feature support](/docs/astro-private-cloud/v-2-x/airflow-operator-mode#feature-support) for what operator mode does not cover yet.
</Info>

Adoption is the second stage of a two-stage path onto the platform:

* **Stage 1**: you run Airflow with the Astro Runtime Operator on your own Kubernetes cluster. Each Airflow Deployment is defined by one Airflow custom resource, and the operator turns that resource into the running Kubernetes workloads.
* **Stage 2**: you register that cluster as an APC data plane and adopt its Airflow Deployments. APC takes ownership of a defined set of settings, and the operator continues to own the rest.

Adoption leaves the operator itself alone: you keep installing and upgrading it. If you would rather APC took that over too, see [Move the operator under APC](/docs/astro-private-cloud/v-2-x/transition-operator-to-apc).

How the pieces fit together, since the terms are easy to mix up:

* The operator's **Custom Resource Definition (CRD)** is installed once on the cluster. It only defines the shape of an Airflow resource; it doesn't hold any Deployment's configuration.
* Each of your Airflow Deployments is one **Airflow custom resource**, created against that definition. It holds that Deployment's configuration.
* The **operator** watches those custom resources and builds the real Kubernetes objects from them: schedulers, workers, services, and the rest.

Adoption doesn't rearrange any of that. APC writes directly to an individual custom resource, and the operator reconciles the change exactly as it would if you had edited the resource yourself. Nothing is written to the CRD, and APC never replaces the operator.

```mermaid theme={null}
flowchart TD
  crd["Airflow CRD: installed once per cluster, defines the shape only"]
  apc["APC control plane"]
  you["Your pipeline, Helm chart, or kubectl"]
  cr["Airflow custom resource: one per Deployment, holds its configuration"]
  operator["Astro Runtime Operator"]
  workloads["Scheduler, workers, API server or webserver, triggerer"]

  crd -.->|defines the shape of| cr
  apc -->|writes only the fields it owns| cr
  you -->|writes everything else| cr
  cr -->|watched by| operator
  operator -->|creates and reconciles| workloads
```

Both writers act on the same custom resource, so which fields each one owns is the thing to understand before you adopt: see [What adoption changes](#what-adoption-changes). If you stop touching the resource yourself after adoption, the second arrow simply goes away; if you keep managing it, read [Keep your own pipeline and APC from fighting](#keep-your-own-pipeline-and-apc-from-fighting).

When you adopt a Deployment, APC applies the configuration it owns to that Deployment's Airflow custom resource, so expect the Deployment's pods to restart once shortly after you adopt. Read [What adoption changes](#what-adoption-changes) before you begin so you know what APC takes over.

## What adoption changes

When APC adopts an Airflow Deployment, it takes ownership of a specific set of fields and leaves everything else to you and the operator. In this release, nothing outside the "APC takes over" column is modified, either at adoption or on any later update.

Ownership is split rather than transferred wholesale because your custom resource can express things APC has no equivalent for: more than one worker queue, KEDA autoscaling, per-component pod templates, sidecars. APC claims only the fields it needs in order to manage the Deployment, which is what image it runs, which executor, the web component it puts authentication in front of, and the labels its monitoring and log shipping key on. If it claimed the rest, every update would have to overwrite your configuration with APC's narrower model. Leaving those fields alone is what makes adoption non-destructive. The trade-off is that they stay managed where they are today, through the operator, rather than through APC.

This is where the line falls today, not a permanent boundary. APC does not yet cover everything the Astro Runtime Operator can express, and the set of settings it manages is expected to widen in future releases. Operator mode has its own gaps against Helm mode, listed in [Feature support](/docs/astro-private-cloud/v-2-x/airflow-operator-mode#feature-support). Check this page against the version of APC you are running rather than assuming the split is fixed.

| APC takes over                                                                                                                                                                                                       | Stays yours                                                                                                                                                                                                                            |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Airflow image and Astro Runtime version. APC owns these fields from adoption, but seeds them from your existing custom resource, so the Deployment keeps running the image it already had until you deploy new code. | Sizing and replica counts for every component except the webserver or API server, including the scheduler, workers, triggerer, and Dag processor                                                                                       |
| Executor selection (see the warning under [Worker queues and autoscaling](#worker-queues-and-autoscaling))                                                                                                           | Your `airflow.cfg` and any config you set through it                                                                                                                                                                                   |
| The webserver (Airflow 2) or API server (Airflow 3) component **in full**, including its ingress, authentication, resources, and replicas                                                                            | Pod template overrides on every other component, including custom volumes, sidecars, tolerations, and node selectors. APC adds its own labels to those pod templates for attribution and log routing, but changes nothing else in them |
| Environment variables you set through APC                                                                                                                                                                            | Environment variables referencing your own Secrets or ConfigMaps                                                                                                                                                                       |
| Metrics exporter labels and the network policy rules needed to scrape them                                                                                                                                           | Your metadata database, its connection Secrets, and its credentials                                                                                                                                                                    |
| Nothing in the worker section                                                                                                                                                                                        | Every worker queue and its KEDA autoscaling, including queues beyond the first                                                                                                                                                         |
| Task logging destination and image pull Secret (**only if you opt in**)                                                                                                                                              | Task logging and image registry configuration if you do not opt in                                                                                                                                                                     |

<Warning>
  **APC takes over Airflow web access.** APC applies its own authentication and ingress to the webserver or API server component on every adopted Deployment. If your Airflow currently authenticates users through LDAP, a `REMOTE_USER` proxy, or a custom `webserver_config.py`, that configuration is replaced with APC sign-in the first time APC applies its configuration.

  This is why [giving the Deployment's users APC accounts](#step-4-give-the-deployments-users-apc-accounts) is a required step and not an optional one. Everyone who signs in to this Airflow today needs a matching APC account, and anyone who exists only in Airflow loses access until they have one.
</Warning>

Two more things before you start:

* **Component resources are brought into your platform's supported range.** If a component in your custom resource requests less than your platform's minimum or more than its maximum, APC adjusts it to the nearest supported value at adoption. See [Configure component size limits](/docs/astro-private-cloud/v-2-x/configure-component-size-limits).
* **Adoption is not a migration of history.** Existing task logs stay wherever they are today. If you switch task logging to APC, only logs written after the switch are readable from the Airflow UI; older ones remain in your own store but the Airflow UI no longer resolves them.

<Warning>
  **Don't adopt a Deployment whose image is pinned by digest.** If your custom resource references its image by digest (`myrepo/airflow@sha256:...`) rather than by tag, adoption rewrites it to a tag reference built from the Deployment's Astro Runtime version (`myrepo/airflow:<runtime-version>`). The digest pin is lost, and if that tag doesn't exist in your repository the Deployment stops being able to pull its image.

  This happens on the first apply, before you deploy anything. Re-tag the image and update the custom resource to reference it by tag before adopting, or hold off on adopting that Deployment. Astronomer is addressing this.
</Warning>

## Prerequisites

* Airflow Deployments running under the Astro Runtime Operator, on a cluster where operator support is enabled. See [Airflow Operator mode](/docs/astro-private-cloud/v-2-x/airflow-operator-mode).
* The operator's cluster registered as an APC data plane. See [Install a data plane cluster](/docs/astro-private-cloud/v-2-x/install-data-plane) and [Register a data plane cluster](/docs/astro-private-cloud/v-2-x/register-data-plane).
* Operator support and adoption enabled on your platform. See [Enable adoption on your platform](#enable-adoption-on-your-platform).
* Permission to adopt. Two permissions are involved: `workspace.deployments.adopt` to adopt a Deployment into a Workspace, and `system.deployments.adopt` to browse adoption candidates, which is separate because listing candidates scans a whole cluster. Among the built-in roles, **Workspace Admin** carries the first and **System Admin** the second. **Cluster Admin does not carry either**, because it governs cluster configuration rather than Deployments. If your platform uses custom roles, both permissions can be granted to one. See [Manage permissions](/docs/astro-private-cloud/v-2-x/manage-permissions) and the [role and permission reference](/docs/astro-private-cloud/v-2-x/role-permission-reference).
* No existing APC Deployment using the custom resource's name, or the namespace it runs in. Adoption is rejected if either is already taken.
* The custom resource references its image **by tag, not by digest**. See the warning under [What adoption changes](#what-adoption-changes).
* To use the Astro CLI instead of the UI, APC **2.1.0 or later** and a matching Astro CLI. See [Install the Astro CLI](https://www.astronomer.io/docs/astro/cli/install-cli).

## Enable adoption on your platform

Adoption builds on operator support, so both have to be on.

**Operator support** is a prerequisite and is configured separately, including its webhook TLS certificate. Follow [Enable operator support](/docs/astro-private-cloud/v-2-x/airflow-operator-mode#enable-operator-support) first if it isn't on yet.

**Adoption** is then controlled by one additional value, which is already on by default:

| Value                                     | Default | What it does                                                                                                                     |
| ----------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `global.airflowOperator.enabled`          | `false` | Turns on operator support. Required for adoption. See [Airflow Operator mode](/docs/astro-private-cloud/v-2-x/airflow-operator-mode). |
| `global.airflowOperator.adoption.enabled` | `true`  | Allows APC to adopt Deployments that already exist on the cluster.                                                               |

<Note>
  `global.airflowOperator.adoption.enabled` is already `true`, but it does nothing on its own. Adoption is enabled only when **both** values are `true`, so on a default install you enable adoption by turning on operator support. Leave the adoption value alone unless you specifically want operator support without adoption, in which case set it to `false`.
</Note>

Both values are read on the control plane and on the data plane, for different things: the control plane uses them to accept the adoption operations, and the data plane uses them to grant APC access to the Airflow custom resources and to scrape operator-managed Deployments. Set them wherever you set operator support.

### Don't install a second operator

Your cluster already runs the Astro Runtime Operator, so tell the platform chart not to install its own alongside it:

```yaml theme={null}
global:
  airflowOperator:
    enabled: true
    adoption:
      enabled: true

airflow-operator:
  # Skip installing APC's own operator; this cluster already runs one.
  enabled: false
```

<Warning>
  Set `airflow-operator.enabled: false` but keep `global.airflowOperator.enabled: true`. Setting the global value to `false` to avoid installing a second operator also switches off the permissions and the API operations adoption depends on, so adoption stops working entirely.
</Warning>

Add the values to the platform configuration file you install with, then upgrade the release. See [Apply platform configuration](/docs/astro-private-cloud/v-2-x/apply-platform-config). Enabling this does not change Deployments that already exist; it adds the adoption capability.

### Confirm it's enabled

Sign in to the control plane as a System Admin and look for **System** > **Adoption Candidates (Preview)**. If the section is there, adoption is enabled. If it's missing, or an adopt call reports that operator adoption is disabled, one of the two values is still `false` on the control plane.

## Step 1: Decide how to handle logging, images, and metrics

Adoption asks you to make two choices, logging and images. Both are set when you adopt and are not intended to be changed afterwards, so decide before you start. Metrics need no decision: they are always on.

### Task logging

Choose whether APC becomes the destination for your task logs.

* **Route logs to APC.** APC configures Airflow to write task logs to APC's configured log store, and the Airflow UI reads them back from there. This **overrides your Deployment's existing remote logging**. If your tasks currently log to Amazon S3, Google Cloud Storage, or your own Elasticsearch, they log to APC instead from then on. Logs written before the switch stay where they are, and the Airflow UI no longer resolves them.
* **Keep your own logging.** APC changes nothing about logging. Your tasks keep logging where they do prior to adoption, not in APC's configured log store.

See [Configure logging](/docs/astro-private-cloud/v-2-x/logs-configuration), [Export task logs](/docs/astro-private-cloud/v-2-x/export-task-logs), and [Send logs to S3](/docs/astro-private-cloud/v-2-x/logs-to-s3).

### Image registry

Two separate settings decide where your Deployment's image comes from. Don't confuse them.

**Your platform's registry** is configured once, for every Deployment on the platform, adopted or not. By default that's APC's built-in registry. To use your own instead, configure a custom image registry before you adopt: see [Use a custom image registry](/docs/astro-private-cloud/v-2-x/custom-image-registry) and [Registry backend](/docs/astro-private-cloud/v-2-x/registry-backend). APC synchronizes that registry's credential into every Deployment namespace, including adopted ones.

**The adoption choice** is narrower. It decides whether APC manages *this Deployment's* image reference and pull credential:

* **Use APC's registry** (`--use-apc-registry`). APC takes over the Deployment's image and provisions the pull credential its pods need. You don't have to move the image yourself first: adoption leaves the Deployment on the image it already runs, and the switch to your platform's registry happens on your first `astro deploy`, which moves the image and the pull credential together. Deploy code with `astro deploy` or a CI/CD pipeline as normal. See [Deploy code overview](/docs/astro-private-cloud/v-2-x/deploy-code-overview) and [CI/CD](/docs/astro-private-cloud/v-2-x/ci-cd).
* **Keep your own** (the default). APC leaves the Deployment's image and pull Secret exactly as they are and never manages them. Use this when something outside APC builds and pushes the image.

#### Deploy code when you keep your own image

You can still ship new code through APC. Build and push the image to your own registry, then point the Deployment at it:

```bash theme={null}
astro deploy --remote --image-name=<your-registry>/<repository>:<tag> --runtime-version=<runtime-version> <deployment-id>
```

APC updates the Deployment to run that image without touching your registry or your pull credential. `--runtime-version` is required with `--remote`. Your platform administrator must have set `deployments.enableUpdateDeploymentImageEndpoint: true`, which the [custom image registry](/docs/astro-private-cloud/v-2-x/custom-image-registry) setup already covers.

<Warning>
  **Don't run plain `astro deploy` on a Deployment that kept its own image.** Without `--remote`, `astro deploy` builds your project and pushes it to APC's built-in registry, then repoints the Deployment at it. Because you opted out, APC never provisioned a credential for that registry, so the Deployment's pods fail to pull the new image and stop starting. The command reports success, and the previous working image reference is gone.

  Use `--remote --image-name` as shown above, or adopt with `--use-apc-registry` if you want APC to own the Deployment's image.
</Warning>

<Note>
  The adoption choice is fixed at adoption. To change it later, release the Deployment and adopt it again with the setting you want.
</Note>

<Note>
  If APC detects that your custom resource already points at this cluster's own log store or image registry, for example because the Deployment was previously managed by a different control plane, it takes ownership of that wiring regardless of what you choose here. Leaving it half-owned would break the Deployment.
</Note>

### Metrics

Metrics are always enabled and have no option. APC labels the Deployment's metrics exporters so its monitoring stack collects them, which is additive and changes nothing about how your Airflow runs. If you collect metrics with your own Prometheus, keep doing so. APC's collection does not interfere. See [Deployment metrics](/docs/astro-private-cloud/v-2-x/deployment-metrics) and [Configure metrics](/docs/astro-private-cloud/v-2-x/configure-metrics).

### Settings APC can't fully represent

Some custom resource settings have no exact equivalent in APC. Examples include an environment variable set to different values on different components, and more than one worker group. By default, adoption proceeds and records these as partially represented, leaving the underlying setting in place and working. You can instead require a clean match, in which case adoption fails and reports what didn't fit rather than adopting. Use that mode when you want to review the differences first.

## Step 2: Review adoption candidates

An adoption candidate is an operator-managed Airflow custom resource on a registered data plane that no APC Deployment claims yet.

<Tabs>
  <Tab title="Astro UI">
    1. In the control plane, go to **System** > **Adoption Candidates (Preview)**.
    2. Select the data plane cluster from the dropdown. Candidates appear only after you pick a cluster.
    3. Review the candidates. Each shows the custom resource's name, its Kubernetes namespace, and the Astro Runtime and Airflow versions read from the resource.

    <Frame>
      <img alt="The Adoption Candidates page with a cluster selected, listing one candidate with an Adopt button." />
    </Frame>
  </Tab>

  <Tab title="APC API">
    ```graphql theme={null}
    query {
      adoptionCandidates(clusterId: "<data-plane-cluster-id>") {
        crName
        crNamespace
        runtimeVersion
        airflowVersion
      }
    }
    ```
  </Tab>
</Tabs>

There is no Astro CLI command for listing adoption candidates. Use the Astro UI or the APC API.

If a Deployment you expected doesn't appear, it is usually because an APC Deployment already uses that custom resource's name, or the namespace it runs in, or because the cluster you selected isn't the one it runs on.

## Step 3: Adopt the Deployment

Adopting is a single operation, available from all three surfaces. Pick one.

<Tabs>
  <Tab title="Astro UI">
    1. Go to **System** > **Adoption Candidates (Preview)**.
    2. Select the data plane cluster from the dropdown. The candidates on that cluster then appear. Until you pick one, the page prompts you to select a cluster and shows no candidates.
    3. Select **Adopt** on the candidate you want. The adoption drawer opens, showing the custom resource and the namespace and cluster it runs on.
    4. Choose the **Workspace** to adopt the Deployment into. This is the only required field.
    5. Optionally set a **Label** and **Description**. The label defaults to the custom resource name when left blank.
    6. Set the options you decided on in [Step 1](#step-1-decide-how-to-handle-logging-images-and-metrics):
       * **Route logs to APC (Elasticsearch / Vector)**
       * **Use APC's image registry**
       * **Adopt even if some settings can't be mapped to APC**, which is selected by default. Clear it to require a clean match.
    7. Select **Adopt Deployment**.

    <Frame>
      <img alt="The adoption drawer for a candidate, showing the custom resource, Workspace picker, label, description, and the three adoption options." />
    </Frame>
  </Tab>

  <Tab title="APC API">
    ```graphql theme={null}
    mutation {
      adoptDeployment(
        workspaceUuid: "<workspace-id>"
        clusterId: "<data-plane-cluster-id>"
        crNamespace: "<airflow-cr-namespace>"
        crName: "<airflow-cr-name>"
        label: "My adopted deployment"
        useApcLogging: false
        useApcRegistry: false
        acceptIncompatibilities: true
      ) {
        id
        releaseName
        namespace
        isAdopted
        adoptedAt
      }
    }
    ```

    `workspaceUuid`, `clusterId`, `crNamespace`, and `crName` are required. `label` defaults to the custom resource name, `useApcLogging` and `useApcRegistry` default to `false`, and `acceptIncompatibilities` defaults to `true`.
  </Tab>

  <Tab title="Astro CLI">
    The Deployment is adopted into your **currently selected Workspace**. Switch Workspaces first if you need a different one:

    ```bash theme={null}
    astro workspace switch
    ```

    Then adopt:

    ```bash theme={null}
    astro deployment adopt --cluster-id=<cluster-id> --name=<cr-name> --namespace=<cr-namespace>
    ```

    `--cluster-id`, `--name`, and `--namespace` are required. Add any of the following:

    | Flag                         | Description                                                                                                                              |
    | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
    | `--label`, `-l`              | Display label. Defaults to the custom resource name.                                                                                     |
    | `--description`              | Longer description for the Deployment.                                                                                                   |
    | `--use-apc-logging`          | Route task logs to APC. Off by default.                                                                                                  |
    | `--use-apc-registry`         | Pull images from APC's registry. Off by default.                                                                                         |
    | `--accept-incompatibilities` | Adopt even when some fields have no APC representation. On by default; pass `--accept-incompatibilities=false` to require a clean match. |
  </Tab>
</Tabs>

### What happens when you adopt

Each Airflow Deployment has one custom resource, and you adopt them one at a time. Whichever surface you use, APC then:

1. Reads the live Airflow custom resource from the cluster and maps it onto an APC Deployment.
2. Creates the Deployment in your Workspace and grants you Deployment Admin on it.
3. Applies its managed configuration to the running custom resource. **The Deployment's pods restart once during this step.**

Your Airflow is not recreated, its namespace is not changed, and its metadata database is left exactly as it is.

## Step 4: Give the Deployment's users APC accounts

**Every person who uses this Airflow needs their own APC account.** After adoption, APC's authentication guards the Airflow UI, and an APC account is the only way in. People reach Airflow through the token APC mints for their APC role, not through the Airflow account they signed in with before.

This step exists because the Deployment is adopted rather than created. A Deployment you create in APC starts with no users and gains them as you invite people. An adopted Deployment arrives with a user population that has been signing in all along, so there is an existing set of Airflow users to reproduce one-for-one in APC. Anyone you don't reproduce loses access at cutover.

Where that roster comes from depends on how the Airflow you adopted authenticated people. APC asks the running Airflow which auth manager it uses:

* **Flask-AppBuilder (FAB), which is the Airflow 2 default.** Airflow keeps its own user table, so APC reads it and offers you the roster. Follow the rest of this step.
* **Airflow 3.** APC sets Airflow 3's auth manager to its own at adoption, and there is no FAB user table behind it, so there is no roster to read. APC tells you this explicitly rather than showing an empty list. [Invite the users to APC](/docs/astro-private-cloud/v-2-x/manage-permissions#invite-users) instead, and skip the rest of this step.

### Import users from Airflow 2

<Tabs>
  <Tab title="Astro UI">
    1. Open the adopted Deployment and go to the **Import Users** tab. It appears only on adopted Deployments.
    2. Review the discovered users. For each, APC suggests a Workspace role and a Deployment role based on the user's Airflow role.
    3. Adjust the roles and clear the checkbox next to anyone you don't want to import.
    4. If your platform uses an identity provider, select **Skip invite email**. Those users sign in through your IdP, so they never need to set a password.
    5. Select **Import selected users**.

    <Frame>
      <img alt="The Import Users tab on an adopted Deployment, listing discovered Airflow users with their Airflow roles and a suggested Workspace and Deployment role for each." />
    </Frame>
  </Tab>

  <Tab title="APC API">
    Discover the users:

    ```graphql theme={null}
    query {
      adoptedAirflowUsers(deploymentId: "<deployment-id>") {
        username
        email
        fullName
        active
        fabRoles
        suggestedWorkspaceRole
        suggestedDeploymentRole
        alreadyImported
      }
    }
    ```

    Import the ones you want:

    ```graphql theme={null}
    mutation {
      deploymentUserBulkImport(
        deploymentId: "<deployment-id>"
        bypassInvite: false
        users: [
          { email: "user@example.com", fullName: "Example User", workspaceRole: WORKSPACE_VIEWER, deploymentRole: DEPLOYMENT_EDITOR }
        ]
      ) {
        email
        status
        created
        rolesAssigned
        inviteToken
        error
      }
    }
    ```

    Each row succeeds or fails independently, so a single bad address doesn't fail the whole import.

    Set `bypassInvite: true` for an IdP-backed platform to skip the invitation email, the same as **Skip invite email** in the Astro UI.
  </Tab>
</Tabs>

There is no Astro CLI command for importing users. Use the Astro UI or the APC API.

### How roles are mapped

How Airflow roles are suggested, and what the imported roles mean in Airflow:

| Airflow role found                     | Suggested Deployment role                                                | Resulting Airflow access    |
| -------------------------------------- | ------------------------------------------------------------------------ | --------------------------- |
| `Admin`                                | Deployment Admin                                                         | Airflow Admin               |
| `User`                                 | Deployment Editor                                                        | Airflow User                |
| `Viewer`                               | Deployment Viewer                                                        | Airflow Viewer              |
| Any other role, including custom roles | None. The dropdown shows **View only (no elevation)** until you pick one | Follows the role you choose |

Every imported user gets Workspace membership as well, because a Deployment role requires it. Users who already have access in this Workspace are marked as already imported and are skipped.

The Airflow role APC found is only used to suggest a role. Importing someone carries none of their old Airflow permissions across, so what they can do in Airflow afterwards comes entirely from the APC Deployment role you give them. Check the suggestions rather than accepting them wholesale, particularly for anyone who held a custom Airflow role.

<Note>
  **Discovery reads Airflow's own FAB user table.** If the Deployment doesn't have one, which is the case on an adopted Airflow 3, discovery returns an error saying no FAB user table was found rather than an empty list. That is expected, not a failed adoption. [Invite those users to APC](/docs/astro-private-cloud/v-2-x/manage-permissions#invite-users) instead.
</Note>

<Note>
  Imported users are created as pending invitations with no password. They set their own password by completing the invitation, or sign in directly if your platform uses an identity provider. If your platform can't send email, the import result returns each user's invitation token so you can deliver it yourself. See [Integrate an auth system](/docs/astro-private-cloud/v-2-x/integrate-auth-system) and [Import IdP groups](/docs/astro-private-cloud/v-2-x/import-idp-groups).
</Note>

## Step 5: Verify the adoption

1. The Deployment appears in your Workspace and reports healthy.
2. Your Airflow is still serving, and its Dags and history are intact.
3. Sign in to the Airflow UI as an imported user and confirm the expected role.
4. Change something in APC and confirm it reaches the running Airflow. Adding an environment variable is the simplest check. See [Environment variables](/docs/astro-private-cloud/v-2-x/environment-variables).
5. If you routed logs to APC, run a task and confirm its logs appear in the Airflow UI.
6. Confirm the Deployment's metrics are populating. See [Deployment metrics](/docs/astro-private-cloud/v-2-x/deployment-metrics).

## Manage an adopted Deployment

An adopted Deployment behaves like any other APC Deployment for everything APC owns:

* **Deploy code**: [Deploy code overview](/docs/astro-private-cloud/v-2-x/deploy-code-overview), [Deploy Dags](/docs/astro-private-cloud/v-2-x/deploy-dags), [CI/CD](/docs/astro-private-cloud/v-2-x/ci-cd). If the Deployment kept its own image, deploy with `--remote --image-name` instead: see [Deploy code when you keep your own image](#deploy-code-when-you-keep-your-own-image).
* **Environment variables**: [Environment variables](/docs/astro-private-cloud/v-2-x/environment-variables). Variables that were set per-component in your custom resource are shown read-only, because APC applies variables to all components uniformly. Variables that reference your own Secrets or ConfigMaps are not shown and keep working untouched.
* **Executor**: [Kubernetes executor](/docs/astro-private-cloud/v-2-x/kubernetes-executor). APC applies the change and the operator adjusts the supporting components.
* **Resources**: [Scale Deployment resources](/docs/astro-private-cloud/v-2-x/scale-deployment-resources), [Configure component size limits](/docs/astro-private-cloud/v-2-x/configure-component-size-limits). Webserver or API server sizing is set through APC; scheduler, worker, and triggerer sizing stays with your custom resource. Worker settings in particular are not applicable, see [Worker queues and autoscaling](#worker-queues-and-autoscaling).
* **Runtime upgrades**: [Migrate to Airflow 3](/docs/astro-private-cloud/v-2-x/migrate-to-airflow-3).
* **Secrets backends**: [Secrets backend](/docs/astro-private-cloud/v-2-x/secrets-backend). Unchanged by adoption; these are ordinary environment variables to APC.

## Worker queues and autoscaling

APC models a single worker queue per Deployment and does not use KEDA autoscaling for operator-based Deployments. Many operator-managed Deployments use more than one worker queue, KEDA autoscaling, or both.

Adoption handles this by leaving the worker section alone entirely. APC never writes it on an adopted Deployment, so:

* **Your worker queues keep running unchanged**, including every queue beyond the first. APC does not collapse them into one, rename them, or add a queue of its own.
* **Your KEDA autoscaling keeps running unchanged.** APC does not disable it, even though APC's own operator-based Deployments don't use it.
* **Extra worker queues are recorded as partially represented** at adoption. APC's own view of the Deployment shows the first queue; the others are stored but not surfaced as editable.

The trade-off is that worker settings in APC don't reach an adopted Deployment:

<Warning>
  **Changing worker count, worker resources, or autoscaling in APC has no effect on an adopted Deployment.** APC accepts the change and stores it, and the Deployment's own view shows the new value, but the running Airflow keeps the worker configuration it already had. There is no error and no warning. Manage worker sizing and autoscaling through your operator configuration instead, and treat APC's worker settings as not applicable for these Deployments.
</Warning>

<Warning>
  **Changing the executor on an adopted Deployment discards your worker queues and autoscaling.** The executor *is* APC-managed, and the operator rebuilds worker topology from whatever the executor implies: switching to KubernetesExecutor removes the worker queues entirely, and switching to CeleryExecutor replaces them with a single default queue. Extra queues and KEDA configuration do not survive either change. If your Deployment relies on multiple worker queues, do **not** change its executor from APC.
</Warning>

If you need APC to manage worker sizing and autoscaling for these Deployments, keep them on the operator for now rather than adopting them.

## Data plane failover

**Adopted Deployments are not covered by data plane failover.** APC's failover recreates the Deployments it created on a target cluster; it has no path for a Deployment whose definition lives in a custom resource on the original cluster, so adopted Deployments are not brought up on the failover target.

<Warning>
  Nothing currently stops you from initiating failover for a data plane that has adopted Deployments. Failover starts, and the adopted Deployments simply do not arrive on the target cluster. Do **not** count failover as disaster recovery for an adopted Deployment.

  If you run adopted Deployments on a data plane that has failover enabled, plan their recovery separately: keep the Airflow custom resource and its supporting configuration in source control, and be ready to apply it to the target cluster and adopt it again there.
</Warning>

See [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover) and [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover) for how failover works for Deployments APC created.

## Keep your own pipeline and APC from fighting

After adoption, two things write to the same Airflow custom resource: APC, and whatever you use to manage the resource yourself, such as a GitOps controller, a Helm chart, or `kubectl` in a CI job. If you still patch or upgrade the Deployment through your own pipeline, read this section. If you manage the Deployment only through APC after adoption, you can skip it.

APC writes only the fields listed in [What adoption changes](#what-adoption-changes). It writes them with Kubernetes server-side apply, under the field manager `houston`, and it force-claims them, so a Deployment update always wins over whatever wrote those fields last.

What happens when you upgrade the resource yourself depends entirely on how your tooling writes it:

```mermaid theme={null}
flowchart TD
  start["You upgrade the custom resource from your own pipeline"]
  scoped["Server-side apply, with APC-owned fields removed from your manifest"]
  full["Client-side apply, kubectl replace, or delete and recreate"]
  forced["Server-side apply with force-conflicts, APC-owned fields still in your manifest"]
  ok["APC fields untouched. Nothing to do"]
  wiped["APC fields wiped from the resource. Airflow keeps running without them"]
  flap["Your tooling and APC revert each other on every reconcile"]
  resync["Trigger a Deployment update to restore the APC fields"]
  strip["Remove the APC-owned fields from your manifest, then trigger a Deployment update"]

  start --> scoped
  start --> full
  start --> forced
  scoped --> ok
  full --> wiped
  wiped --> resync
  forced --> flap
  flap --> strip
```

The middle path is the one to watch: your upgrade succeeds, Airflow keeps running, and nothing reports a problem, but the Deployment is now missing the configuration APC applied, including the authentication on its web component. Only a Deployment update puts it back.

### Use server-side apply, and remove APC-owned fields from your manifest

Both halves are necessary. Doing only the first makes things worse, not better.

1. **Apply with server-side apply** so your write only touches the fields your manifest actually declares:

   ```bash theme={null}
   kubectl apply --server-side -f airflow-cr.yaml
   ```

   In Argo CD, set `ServerSideApply=true`. Flux's kustomize-controller already uses server-side apply.

2. **Delete the APC-owned fields from the manifest you apply.** Your manifest is usually the custom resource as it looked before adoption, so it still declares fields APC now owns, such as `spec.image`, `spec.runtimeVersion`, `spec.executor`, and the whole `spec.webserver` or `spec.apiserver` block.

<Warning>
  **Server-side apply on its own turns a one-time problem into a permanent one.** Argo CD's `ServerSideApply=true` runs with `--force-conflicts`, and Flux corrects drift the same way. If your manifest still declares APC-owned fields, your tool force-claims them back, the next Deployment update force-claims them again, and the two keep reverting each other indefinitely. Removing those fields from the manifest is what stops the loop. If your tool supports drift-ignore rules, exclude the APC-owned paths instead.
</Warning>

### Trigger a Deployment update after any full-object write

Client-side `kubectl apply`, `kubectl replace`, and deleting and recreating the custom resource all write the whole object, so they wipe APC's fields in one pass. Your Airflow keeps running, but it now runs without the configuration APC applied, including the authentication on its web component.

Every field APC owns is restored by the next Deployment update, because APC re-applies its full set of managed fields each time and force-claims them. After any write of that kind, and after any pipeline upgrade or patch that you're not certain was field-scoped, trigger a Deployment update. This is the Deployment-level re-sync, not an APC platform upgrade. Nothing needs to change for it to do its job, so a no-op update is enough:

<Tabs>
  <Tab title="Astro UI">
    Open the Deployment's settings and save without changing anything.
  </Tab>

  <Tab title="APC API">
    ```graphql theme={null}
    mutation {
      upsertDeployment(deploymentUuid: "<deployment-id>") {
        id
      }
    }
    ```

    Every other argument is optional, and omitted settings keep their current values.
  </Tab>
</Tabs>

If the Deployment is cordoned, uncordon it first. A cordoned Deployment ignores updates, so the resync won't happen. If APC's fields had actually drifted, restoring them changes the custom resource and the affected pods restart; if nothing had drifted, the update makes no change and nothing restarts.

### Never let your pipeline prune APC's Secrets

<Warning>
  **A Deployment update cannot restore deleted Secrets.** APC creates `<cr-name>-registry`, `<cr-name>-elasticsearch`, and `<cr-name>-env` once and cannot recreate them later, because it no longer holds the credentials they contain. If your pipeline prunes resources it doesn't manage, or you recreate the Deployment's namespace, and those Secrets are removed, the Deployment breaks in ways a Deployment update makes worse rather than better: the custom resource still references the missing Secrets, so pods fail to pull images or fail to start at all.

  Exclude the Deployment's namespace from pruning, or restrict pruning to the resources your pipeline created. If these Secrets are already gone, contact [Astronomer support](/docs/astro-private-cloud/v-2-x/support) to have them reissued.
</Warning>

### Check which fields APC owns

To see exactly what APC claims on a Deployment, read the custom resource's field ownership and look for the `houston` manager:

```bash theme={null}
kubectl get airflow <cr-name> -n <cr-namespace> --show-managed-fields -o yaml
```

`--show-managed-fields` is required. Without it, `kubectl` hides the ownership information from `-o yaml` and `-o json` output.

## Pause management with cordon

Cordoning a Deployment stops APC applying changes to it while it keeps running. It is not specific to adopted Deployments, so it has its own page: see [Cordon a Deployment](/docs/astro-private-cloud/v-2-x/cordon-deployment).

<Tip>
  Cordon adopted Deployments before a platform upgrade. See [Known limitations](#known-limitations).
</Tip>

## Release a Deployment

Releasing, also called unadopting, returns a Deployment to operator-only management.

<Tabs>
  <Tab title="Astro UI">
    In the Deployments list, open the Deployment's actions menu, select **Unadopt Deployment**, and confirm. Adopted Deployments carry an **Adopted** badge in the **Adopted** column.

    <Frame>
      <img alt="The Deployments list with an adopted Deployment's actions menu open on Unadopt Deployment." />
    </Frame>
  </Tab>

  <Tab title="APC API">
    ```graphql theme={null}
    mutation {
      unadoptDeployment(deploymentUuid: "<deployment-id>") {
        id
        releaseName
      }
    }
    ```
  </Tab>

  <Tab title="Astro CLI">
    ```bash theme={null}
    astro deployment unadopt --deployment-id=<deployment-id>
    ```

    The CLI asks for confirmation before releasing.
  </Tab>
</Tabs>

### What happens when you release

APC deletes its own record of the Deployment, along with its deploy history and the Deployment-scoped roles it granted. Workspace membership is left in place.

Nothing is removed from your cluster. The Airflow custom resource, its namespace, its metadata database, and its data are all left as they are, the operator continues to reconcile it, and Airflow keeps running. You can adopt the same Deployment again later.

<Warning>
  **Releasing does not undo the configuration APC applied.** In particular, the Airflow web authentication and ingress that APC applied at adoption stay on the custom resource, but APC no longer recognizes the Deployment, so users can't sign in to the Airflow UI until you restore your own web authentication configuration. Restore it as part of releasing, not afterwards.
</Warning>

Releasing is not the same as deleting. Deleting a Deployment removes the underlying Airflow and its database; releasing removes only APC's record of it.

## Known limitations

* **The cluster must already be an APC data plane.** You cannot adopt Deployments from a cluster the control plane doesn't know about. Register it first. See [Register a data plane cluster](/docs/astro-private-cloud/v-2-x/register-data-plane).
* **Airflow web authentication takeover is one-way.** Once APC applies its authentication to an adopted Deployment, there is no supported path back to your original configuration while the Deployment remains adopted.
* **Releasing a Deployment interrupts Airflow web access** until you restore its original web authentication configuration. See [Release a Deployment](#release-a-deployment).
* **A platform upgrade can restart adopted Deployments.** Cordon any adopted Deployment you don't want APC to act on during an upgrade, and uncordon it afterwards.
* **Worker settings in APC don't reach an adopted Deployment.** Worker count, worker resources, and autoscaling are accepted and stored but never applied, and changing the executor discards the Deployment's worker queues and KEDA configuration. See [Worker queues and autoscaling](#worker-queues-and-autoscaling).
* **Data plane failover does not cover adopted Deployments,** and nothing blocks you from initiating failover on a data plane that has them. Plan their recovery separately. See [Data plane failover](#data-plane-failover).
* **Images pinned by digest are converted to tag references at adoption.** The digest pin is not preserved, and the substituted tag may not exist. Re-tag before adopting. See the warning under [What adoption changes](#what-adoption-changes).
* **Plain `astro deploy` breaks a Deployment that kept its own image.** It pushes to APC's built-in registry and repoints the Deployment there, but no pull credential was provisioned for it. Use `--remote --image-name`. See [Deploy code when you keep your own image](#deploy-code-when-you-keep-your-own-image).
* **The registry and logging choices are fixed at adoption.** Changing either means releasing the Deployment and adopting it again.
* **Users can't be imported from an adopted Airflow 3 Deployment.** APC takes over its auth manager at adoption, so there are no Airflow-local users to read. [Invite them to APC](/docs/astro-private-cloud/v-2-x/manage-permissions#invite-users) instead.
* **Existing task logs are not migrated** when you route logging to APC. Only logs written after the switch are readable from the Airflow UI; older logs stay in your own store and the Airflow UI no longer resolves them.
* **Deleted APC Secrets can't be restored by a Deployment update.** If `<cr-name>-registry`, `<cr-name>-elasticsearch`, or `<cr-name>-env` is deleted or pruned, reissuing it requires Astronomer support. See [Never let your pipeline prune APC's Secrets](#never-let-your-pipeline-prune-apcs-secrets).
* **Clusters using the authentication sidecar don't get an ingress for adopted Deployments**, so the Airflow UI link is not reachable from APC on those clusters.

## Reference

### Adoption fields on a Deployment

| Field                          | Meaning                                                                                      |
| ------------------------------ | -------------------------------------------------------------------------------------------- |
| `isAdopted`                    | Whether the Deployment came from an operator-managed custom resource.                        |
| `adoptedAt`                    | When it was adopted.                                                                         |
| `adoptionLoggingManagedByApc`  | Whether APC is the task-log destination. Always `true` for Deployments APC created itself.   |
| `adoptionRegistryManagedByApc` | Whether APC's registry supplies the image. Always `true` for Deployments APC created itself. |

### API operations

| Operation                                                                         | Purpose                                                                                    |
| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `adoptionCandidates(clusterId)`                                                   | List operator-managed custom resources on a cluster that aren't adopted yet. System Admin. |
| `adoptDeployment(...)`                                                            | Adopt one custom resource into a Workspace. Workspace Admin.                               |
| `adoptedAirflowUsers(deploymentId)`                                               | List the Deployment's Airflow users with suggested roles. Read-only.                       |
| `deploymentUserBulkImport(deploymentId, users)`                                   | Import the reviewed users and grant their roles.                                           |
| `cordonDeployment(deploymentUuid, reason)` / `uncordonDeployment(deploymentUuid)` | Pause and resume APC management.                                                           |
| `unadoptDeployment(deploymentUuid)`                                               | Release the Deployment back to operator-only management.                                   |

See [Use the APC API](/docs/astro-private-cloud/v-2-x/houston-api) and [Example APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-example-queries) for authenticating and running these operations.

## Related documentation

* [Install a data plane cluster](/docs/astro-private-cloud/v-2-x/install-data-plane)
* [Register a data plane cluster](/docs/astro-private-cloud/v-2-x/register-data-plane)
* [Airflow Operator mode](/docs/astro-private-cloud/v-2-x/airflow-operator-mode)
* [Move the operator under APC](/docs/astro-private-cloud/v-2-x/transition-operator-to-apc)
* [Data plane architecture](/docs/astro-private-cloud/v-2-x/data-plane-architecture)
* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)
* [Configure a Deployment](/docs/astro-private-cloud/v-2-x/configure-deployment)
* [Cordon and uncordon a Deployment](/docs/astro-private-cloud/v-2-x/cordon-deployment)
* [Environment variables](/docs/astro-private-cloud/v-2-x/environment-variables)
* [Deploy code overview](/docs/astro-private-cloud/v-2-x/deploy-code-overview)
* [Configure logging](/docs/astro-private-cloud/v-2-x/logs-configuration)
* [Deployment metrics](/docs/astro-private-cloud/v-2-x/deployment-metrics)
* [Manage permissions](/docs/astro-private-cloud/v-2-x/manage-permissions)
* [Manage platform users](/docs/astro-private-cloud/v-2-x/manage-platform-users)
* [Invite users to a Workspace or Deployment](/docs/astro-private-cloud/v-2-x/manage-permissions#invite-users)


# Deploy Airflow with the Airflow Operator
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/airflow-operator-mode

Manage an Astro Private Cloud Deployment through the Airflow Kubernetes Operator instead of a Helm release.

Astro Private Cloud (APC) manages Airflow Deployments with Helm by default. Operator mode is an alternative that manages a Deployment through the Airflow Kubernetes Operator — a kubebuilder-based controller that reconciles Airflow Deployments from a custom resource definition (CRD) — instead of a Helm release. You choose the mode for each Deployment. Helm remains the default and is fully supported.

<Note>
  **Astro Private Cloud 2.1**

  This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
</Note>

<Warning>
  Airflow operator support is under active development for Astro Private Cloud. Several Helm-mode features aren't available in operator mode yet. See [Feature support](#feature-support).
</Warning>

If your cluster already runs the Astro Runtime Operator, you can bring its existing Airflow Deployments under APC management with [Adopt Astro Runtime Operator managed Deployments](/docs/astro-private-cloud/v-2-x/adopt-operator-deployments), and hand the operator itself over with [Move the operator under APC](/docs/astro-private-cloud/v-2-x/transition-operator-to-apc).

## Concepts

* Deployment mode: each APC Deployment is either Helm (a Helm release, the default) or operator (an Airflow CRD reconciled by the operator).
* Reconciliation: the control plane builds the Airflow custom resource (CR) spec from the Deployment configuration, and the data plane applies the CR and monitors the operator's reconciliation, rather than managing a Helm release.
* Mixed mode: a single APC installation can run Helm and operator Deployments side by side, and you can enable operator support for each data plane.
* Cluster-scoped resources: the operator requires cluster-level CRDs, mutating and validating webhooks, and cert-manager. Review these with your security team before you enable operator support. See [Security and governance](#security-and-governance).

## Prerequisites

* An APC 2.1 or later installation running in [split mode](/docs/astro-private-cloud/v-2-x/data-plane-architecture) or [unified mode](/docs/astro-private-cloud/v-2-x/unified-architecture).
* Permission to update the Helm values for the data plane, or for the unified installation.
* Cluster-level permission to install CRDs and to configure mutating and validating webhooks.
* Either [cert-manager](https://cert-manager.io/) in the cluster, or a serving certificate you generate yourself. See [Provide the webhook TLS certificate](#provide-the-webhook-tls-certificate).

## Enable operator support

Operator support is off by default, and it installs with the data plane, which hosts operator Deployments. In [split mode](/docs/astro-private-cloud/v-2-x/data-plane-architecture), enable it in the Helm configuration for the data plane. In [unified mode](/docs/astro-private-cloud/v-2-x/unified-architecture), where the control plane and data plane run in the same cluster, enable it in that installation's Helm configuration:

```yaml theme={null}
global:
  airflowOperator:
    enabled: true
```

Enabling operator support installs the operator CRDs, configures the mutating and validating webhooks, adds the Prometheus label filters for operator Deployments, and grants the data plane the role-based access control (RBAC) it needs for `airflow.apache.org` CRD resources.

### Provide the webhook TLS certificate

The Kubernetes API server calls the operator's mutating and validating webhooks over TLS, so the webhooks need a serving certificate and the CA bundle that signed it. Choose one of the following methods based on whether cert-manager is available in your cluster.

<Warning>
  Configure exactly one of these methods. The operator sub-chart uses cert-manager by default; if you disable cert-manager without providing your own certificate, the chart fails to render with an error instead of installing a broken webhook.
</Warning>

#### Use cert-manager

cert-manager is enabled for the operator sub-chart by default. If [cert-manager](https://cert-manager.io/) is available in your cluster, keep it enabled: it generates the serving certificate with the correct DNS names and injects the CA bundle into the webhook configurations automatically:

```yaml theme={null}
airflow-operator:
  certManager:
    enabled: true
```

This is the recommended method. It requires no manual certificate management, and cert-manager renews the certificate before it expires.

#### Provide your own certificate

If cert-manager isn't available, generate a serving certificate yourself and pass it to the operator sub-chart. The webhook Service is named `<release-name>-airflow-operator-webhook-service`.

<Steps>
  <Step title="Set the Service DNS names">
    Set the DNS names as shell variables. The certificate must be valid for both, or the API server rejects the webhook connection:

    ```bash theme={null}
    export NAMESPACE=<apc-namespace>
    export SERVICE=<release-name>-airflow-operator-webhook-service
    export DNS1="${SERVICE}.${NAMESPACE}.svc"
    export DNS2="${SERVICE}.${NAMESPACE}.svc.cluster.local"
    ```
  </Step>

  <Step title="Generate the CA and serving certificate">
    Generate a CA and a serving certificate whose Subject Alternative Names (SANs) cover both DNS names:

    ```bash theme={null}
    # Generate a CA key and self-signed CA certificate. This becomes the caBundle.
    openssl genrsa -out ca.key 2048
    openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 \
      -subj "/CN=airflow-operator-webhook-ca" -out ca.crt

    # Generate the serving key and certificate signing request.
    openssl genrsa -out tls.key 2048
    openssl req -new -key tls.key -subj "/CN=${DNS1}" -out tls.csr

    # Sign the request with the CA, embedding the required SANs.
    cat > san.ext <<EOF
    subjectAltName = DNS:${DNS1}, DNS:${DNS2}
    extendedKeyUsage = serverAuth
    EOF

    openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
      -out tls.crt -days 3650 -sha256 -extfile san.ext
    ```

    Don't reuse an ingress or general-purpose certificate. A certificate that lacks the `.svc` SANs fails the webhook TLS handshake even when the CA bundle is correct.
  </Step>

  <Step title="Create the TLS secret">
    Create a `kubernetes.io/tls` secret in the release namespace from the serving certificate and key:

    ```bash theme={null}
    kubectl -n "${NAMESPACE}" create secret tls airflow-operator-webhook-tls \
      --cert=tls.crt --key=tls.key
    ```
  </Step>

  <Step title="Configure the operator sub-chart">
    Point the operator sub-chart at the secret and provide the CA bundle. Set `certManager.enabled` to `false`, `webhooks.useCustomTlsCerts` to `true`, `webhooks.customCertsSecretName` to the secret name, and `webhooks.caBundle` to the raw contents of `ca.crt`:

    ```yaml theme={null}
    airflow-operator:
      certManager:
        enabled: false
      webhooks:
        useCustomTlsCerts: true
        customCertsSecretName: airflow-operator-webhook-tls
        caBundle: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
    ```

    Provide `caBundle` as raw PEM text, not base64. When `webhooks.useCustomTlsCerts` is `true`, both `webhooks.customCertsSecretName` and `webhooks.caBundle` are required — the chart fails to render if either is missing.
  </Step>
</Steps>

<Note>
  Certificates you provide yourself don't renew automatically. Before `tls.crt` expires, regenerate it, update the secret, and redeploy. If the CA changes, update `caBundle` as well.
</Note>

## Create an operator Deployment

After you enable operator support, a **Deployment Mode** selector appears when you create a Deployment. To create an operator-managed Deployment, set **Deployment Mode** to **Operator**, then complete the rest of the Deployment configuration as usual.

**Helm** is the default, and the selector doesn't appear when operator support is off.

<Frame>
  <img alt="The New Deployment page with the Deployment Mode selector showing Helm and Operator options, with Operator selected." />
</Frame>

## Feature support

Operator mode reaches most Helm capabilities, but not all. Use Helm mode for a Deployment that needs a feature in the second of the following lists.

Available in operator mode in APC 2.1:

* Celery executor and Kubernetes executor
* Airflow 2 and Airflow 3
* PostgreSQL-backed Deployments
* Private registry
* Auth sidecar and bring-your-own ingress
* Network policies for Airflow components and platform-level network policy
* Custom resource configuration (CPU and memory)
* Manual release name
* DaemonSet logging
* In-cluster and external Elasticsearch logging
* Airflow rollback
* OpenShift support

Not yet available in operator mode — use Helm mode:

* Dag-only deploy, Network File System (NFS) volume and git-sync Dag Deployment
* Namespace pools
* Sidecar logging
* MySQL-backed Deployments
* Resource-quota enforcement
* Enabling or disabling the triggerer independently
* Celery Flower UI
* Disaster-recovery (DR) failover and control-plane high availability (HA)

## Security and governance

Operator mode installs cluster-scoped resources:

* The Airflow CRDs
* A mutating and validating webhook
* cert-manager integration, when you use cert-manager to issue the webhook certificate

Because these are cluster-level, review them with your security team before you enable operator support.

If your change-control process requires it, you can manage the CRDs out-of-band — for example, install them separately as a cluster admin or through GitOps — so the platform chart doesn't create them. Set `crd.create` to `false` on the operator sub-chart:

```yaml theme={null}
airflow-operator:
  crd:
    create: false
```

The default is `true`, which lets the chart create the CRDs. When you set it to `false`, install the operator CRDs yourself before you create any operator Deployment.

<Note>
  The operator CRDs carry a `helm.sh/resource-policy: keep` annotation, so Helm never deletes them, even when you set `crd.create` to `false` on an existing installation. This prevents an upgrade from removing the CRDs, which would delete every operator Deployment. Remove the CRDs manually only after you delete all operator Deployments.
</Note>

## Known limitations

* Several Helm-mode features aren't available in operator mode. See [Feature support](#feature-support). Helm remains the default and the fuller-featured mode.
* Operator mode isn't a migration path between modes. You set the mode when you create a Deployment, and you can't convert an existing Deployment from Helm to operator or from operator to Helm. To change modes, create a new Deployment in the target mode.


# Airflow system components
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/airflow-system-components

Understand the components that make up a Deployment in Astro Private Cloud.

A Deployment in Astro Private Cloud (APC) consists of multiple components that work together to orchestrate and execute your data pipelines. Each component has a specific role and configuration options.

## Core components

### Scheduler

The scheduler is the heart of Airflow. It monitors all Dags and tasks, triggers task instances when dependencies are complete, and submits tasks to the executor for running.

Default configuration:

```yaml wrap theme={null}
scheduler:
  enabled: true
  replicas: 1
  terminationGracePeriodSeconds: 10
  livenessProbe:
    initialDelaySeconds: 10
    timeoutSeconds: 20
    failureThreshold: 5
    periodSeconds: 60
```

Key responsibilities:

* Schedule tasks based on dependencies and triggers.
* Monitor task states and handle retries.
* Manage pools and task queues.
* Parse Dag files and create Dag runs. When the Dag processor is enabled, Dag parsing moves to the Dag processor and the scheduler handles only scheduling.

### Webserver

The webserver provides the Airflow UI for monitoring Dags, viewing logs, triggering runs, and managing configurations.

Default configuration:

```yaml wrap theme={null}
webserver:
  enabled: true
  replicas: 1
  terminationGracePeriodSeconds: 30
  allowPodLogReading: true
  livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
    failureThreshold: 5
    periodSeconds: 10
```

Key features:

* Dag visualization and monitoring.
* Task log viewing.
* Variable and connection management.
* User authentication and authorization.

### Workers

Workers execute tasks. Workers run as a persistent deployment only when using Celery Executor. When using Kubernetes executor, Airflow launches ephemeral task pods instead.

Default configuration (Celery Executor):

```yaml wrap theme={null}
workers:
  enabled: true
  replicas: 1
  terminationGracePeriodSeconds: 600
```

### Triggerer

The triggerer handles deferrable operators, allowing tasks to release worker slots while waiting for external events.

Default configuration:

```yaml wrap theme={null}
triggerer:
  enabled: true
  replicas: 1
  terminationGracePeriodSeconds: 60
  livenessProbe:
    initialDelaySeconds: 10
    timeoutSeconds: 20
    failureThreshold: 5
    periodSeconds: 60
```

### Dag processor

The Dag processor parses Dag files and updates the metadata database with Dag definitions. Available in Airflow 2.3+ and mandatory in Airflow 3+.

Default configuration:

```yaml wrap theme={null}
dagProcessor:
  enabled: ~  # Auto-enabled for Airflow 3+
  replicas: 1
  terminationGracePeriodSeconds: 60
  waitForMigrations:
    enabled: true
```

<Note>
  In Airflow 3, the Dag processor is automatically enabled and required for Dag discovery.
</Note>

### API server (Airflow 3+)

The API server is a new component in Airflow 3 that provides the REST API, separated from the webserver for better scalability.

Default configuration:

```yaml wrap theme={null}
apiServer:
  enabled: true
  allowPodLogReading: true
```

<Note>
  The platform manages the number of API server replicas.
</Note>

## Supporting components

### Redis

Message broker for Celery Executor. Handles task queue communication between scheduler and workers.

```yaml wrap theme={null}
redis:
  enabled: true
  persistence:
    enabled: true
    size: 1Gi
```

### StatsD exporter

Collects and exports Airflow metrics for monitoring systems like Prometheus.

```yaml wrap theme={null}
statsd:
  enabled: true
  terminationGracePeriodSeconds: 30
```

### PgBouncer (optional)

Connection pooler that sits between Airflow components and the metadata database. Reduces the number of direct database connections opened by the scheduler, webserver, and workers.

PgBouncer is only enabled when the cluster uses PostgreSQL and `pgbouncer.enabled` is set to `true` in your platform configuration. It is disabled when the cluster uses MySQL.

```yaml wrap theme={null}
pgbouncer:
  enabled: true  # depends on cluster database type and platform config
```

### Flower

Web UI for monitoring Celery workers. Only active when using Celery Executor.

```yaml wrap theme={null}
flower:
  enabled: true
```

## Airflow 2 vs Airflow 3 components

| Component       | Airflow 2                  | Airflow 3                  |
| --------------- | -------------------------- | -------------------------- |
| Scheduler       | Required                   | Required                   |
| Webserver       | Required (includes API)    | Required (UI only)         |
| API server      | N/A                        | Required                   |
| Dag processor   | Optional (2.3+)            | Required                   |
| Workers         | Celery Executor only       | Celery Executor only       |
| Triggerer       | Optional (2.2+)            | Optional                   |
| Redis           | Celery Executor only       | Celery Executor only       |
| Flower          | Celery Executor only       | Celery Executor only       |
| StatsD exporter | Required                   | Required                   |
| PgBouncer       | Optional (PostgreSQL only) | Optional (PostgreSQL only) |

## Resource recommendations

### Small workloads (\< 50 Dags)

```yaml wrap theme={null}
scheduler:
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "500m"
      memory: "1Gi"

webserver:
  resources:
    requests:
      cpu: "500m"
      memory: "1920Mi"
    limits:
      cpu: "500m"
      memory: "1920Mi"
```

### Medium workloads (50-200 Dags)

```yaml wrap theme={null}
scheduler:
  resources:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "1000m"
      memory: "2Gi"

dagProcessor:
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "500m"
      memory: "1Gi"
```

### Large workloads (200+ Dags)

```yaml wrap theme={null}
scheduler:
  replicas: 2
  resources:
    requests:
      cpu: "2000m"
      memory: "4Gi"
    limits:
      cpu: "2000m"
      memory: "4Gi"

dagProcessor:
  replicas: 2
  resources:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "1000m"
      memory: "2Gi"
```

## Scaling components

### Horizontal scaling

Components that support multiple replicas. Default limits apply unless your platform administrator overrides them in the platform configuration.

* **Scheduler**: Up to 4 replicas by default.
* **API server**: Up to 4 replicas by default.
* **Dag processor**: Up to 3 replicas by default.
* **Workers**: Up to 10 replicas by default.
* **Triggerer**: Up to 2 replicas by default.

### Vertical scaling

Increase resources for:

* **Scheduler**: Complex dependencies or high task volume. When the Dag processor is enabled, the scheduler focuses on scheduling only.
* **Dag processor**: Large number of Dag files or complex parsing requirements.
* **Workers**: Memory-intensive tasks.


# Apply an Astro Private Cloud platform Helm configuration change
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/apply-platform-config

Apply platform-level Helm configuration changes to an Astro Private Cloud installation.

Astro Private Cloud uses [Helm](https://helm.sh/) to install and manage some platform-level settings that apply to all users and Deployments. The [Astronomer Helm chart](https://github.com/astronomer/astronomer/blob/master/values.yaml) includes configurations for areas such as:

* Identity provider integrations
* Registry backends
* Resource allocation limits

You can apply any platform customizations to your cluster using a YAML configuration. `values.yaml` contains settings for both the Astronomer Helm chart, as well as Helm charts for system components like [Elasticsearch](https://github.com/astronomer/astronomer/blob/master/charts/elasticsearch/values.yaml) and [nginx](https://github.com/astronomer/astronomer/blob/master/charts/nginx/values.yaml). Using Helm allows you to keep all of your configurations in a single file that you can version and store securely.

Use this document to learn how to retrieve your existing Helm configuration, modify configurations, and apply your changes to your cluster.

<Warning>
  Many settings under `astronomer.houston.config.deployments.*` aren't applied through Helm at all — they're cluster, workspace, or Deployment config, set through the **Configuration Override** section of the Astro UI or the `updateCluster`/`updateWorkspaceDeploymentsConfig`/`updateDeploymentConfig` APIs instead. Editing `values.yaml` only seeds the platform default the first time a cluster is created; it doesn't update an already-provisioned cluster. Before you edit `values.yaml` for anything under `deployments.*`, read [Configure Astro Private Cloud](/docs/astro-private-cloud/v-2-x/configure-astro-private-cloud) and [Config governance](/docs/astro-private-cloud/v-2-x/config-governance) to confirm Helm is the right layer, or see [Override data plane cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster) if it isn't.
</Warning>

<Steps>
  <Step title="Retrieve your current cluster configuration">
    The best way to add or modify configurations is to start with your existing `values.yaml` file. To retrieve the current `values.yaml` file of an existing APC cluster, run the following command:

    ```bash wrap theme={null}
    helm get values -o yaml <your-installation-release-name> -n <your-installation-namespace> > values.yaml
    ```

    <Note>
      This command overwrites any file called `values.yaml` in your current directory.
    </Note>

    Alternatively, your team might use version management to store your existing Helm configuration. In this case, retrieve, update, and store your configuration file according to your team's workflows. If your configuration file includes secrets, ensure you encrypt it before storing it in version control.
  </Step>

  <Step title="Update your configurations">
    1. Create a copy of your `values.yaml` file so that you can compare your existing configuration to your new configuration.
    2. In your copied file, update the values for the configurations you want to change. To update a configuration you haven't already specified, copy the corresponding default values from the relevant [default Helm chart](https://github.com/astronomer/astronomer/tree/master/charts) into your `values.yaml` file and then modify the value.

    When you have finished updating the configuration, ensure that the configurations have the same relative order and indentation as they do in the [default configuration file](https://github.com/astronomer/astronomer/blob/master/values.yaml). If they don't, your changes might not be properly applied. The name of the Helm charts you're modifying should be the first items in the file, such as in the following example:

    ```yaml wrap theme={null}
    global:
      <your-global-configuration>

    astronomer:
      <your-astronomer-configuration>

    alertmanager:
      <your-alertmanager-configuration>

    nginx:
      <your-nginx-configuration>
    ```

    <Info>Identity provider settings and other platform-wide defaults live in `astronomer.houston.config`. Settings under `astronomer.houston.config.deployments.*` are layered — see the callout at the top of this page before assuming a `deployments.*` change belongs here.</Info>
  </Step>

  <Step title="Push changes to your cluster">
    1. Copy your platform namespace and release name. These are both likely to be `astronomer`. Find your platform release name in your list of active namespaces by running the following command:

       ```bash wrap theme={null}
       kubectl get ns
       ```

       To locate your platform release name, run:

       ```bash wrap theme={null}
       helm ls -n <your-installation-namespace>
       ```

    2. Save your updated `values.yaml` file and run the following command to apply it as a Helm upgrade:

       ```bash wrap theme={null}
       helm upgrade <your-installation-release-name> astronomer/astronomer -f <your-updated-config-yaml-file> -n <your-installation-namespace> --set astronomer.houston.upgradeDeployments.enabled=false
       ```

       Setting `astronomer.houston.upgradeDeployments.enabled=false` ensures that Apache Airflow components in Deployments don't restart during your upgrade.

    3. Run the following command to confirm that your configuration was applied:

       ```bash wrap theme={null}
       helm get values <your-installation-release-name> -n <your-installation-namespace>
       ```
  </Step>
</Steps>


# APC API audit log schema and operations
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/audit-log-schema

Reference for the APC API audit event schema, field definitions, redaction rules, and the full inventory of audited APC API operations on Astro Private Cloud.

This document describes the shape of each Astro Private Cloud (APC) API audit event and lists every operation that produces one. Use it when you build queries, alerts, or dashboards on top of APC API audit logs. For background on the feature, see [APC API audit logging overview](/docs/astro-private-cloud/v-2-x/audit-logging-overview).

## Event shape

The APC API emits each audit event as a single JSON object on standard output. The object has stable field names and a consistent top-level structure. The Vector sidecar adds Kubernetes metadata and renames `timestamp` to `@timestamp` before delivering the event to the configured sink.

The following example shows a raw event emitted by the APC API:

```json expandable wrap theme={null}
{
  "timestamp": "<timestamp>",
  "audit": true,
  "audit_version": "1.0",
  "correlationId": "<correlation-id>",
  "sessionId": "<session-id>",
  "userId": "<user-id>",
  "username": "<username>",
  "userEmail": "<user-email>",
  "role": "SYSTEM_ADMIN",
  "clientType": "ui",
  "origin": {
    "requestIp": "<request-ip>",
    "podName": "<pod-name>"
  },
  "action": "workspace.create",
  "operation": "create",
  "entity": {
    "type": "workspace",
    "id": "<workspace-id>",
    "name": "<workspace-name>"
  },
  "outcome": "success",
  "statusCode": 200,
  "request": {
    "graphql": {
      "operation": "createWorkspace",
      "variables": {
        "label": "<workspace-name>"
      }
    }
  },
  "response": {
    "duration": 85.23 // milliseconds
  },
  "component": "houston-api",
  "level": "INFO",
  "message": "Workspace <workspace-name> created successfully"
}
```

The Vector sidecar doesn't drop APC API audit fields. Before delivery to a sink, Vector:

* Renames `timestamp` to `@timestamp`.
* Adds a `kubernetes` object with `pod_name`, `pod_namespace`, and `container_name`.
* Adds `platform`, `component`, and `service` fields that identify the emitting control plane component.
* Converts the `level` field to uppercase.

After the transform, the same event in the sink contains every original APC API audit field plus the added metadata:

```json expandable wrap theme={null}
{
  "@timestamp": "<timestamp>",
  "audit": true,
  "audit_version": "1.0",
  "correlationId": "<correlation-id>",
  "sessionId": "<session-id>",
  "userId": "<user-id>",
  "username": "<username>",
  "userEmail": "<user-email>",
  "role": "SYSTEM_ADMIN",
  "clientType": "ui",
  "origin": {
    "requestIp": "<request-ip>",
    "podName": "<pod-name>"
  },
  "action": "workspace.create",
  "operation": "create",
  "entity": {
    "type": "workspace",
    "id": "<workspace-id>",
    "name": "<workspace-name>"
  },
  "outcome": "success",
  "statusCode": 200,
  "request": {
    "graphql": {
      "operation": "createWorkspace",
      "variables": {
        "label": "<workspace-name>"
      }
    }
  },
  "response": {
    "duration": 85.23 // milliseconds
  },
  "kubernetes": {
    "pod_name": "<pod-name>",
    "pod_namespace": "<pod-namespace>",
    "container_name": "houston"
  },
  "platform": "<release-name>-control-plane",
  "component": "houston-api",
  "service": "houston-api",
  "level": "INFO",
  "message": "Workspace <workspace-name> created successfully"
}
```

Each sink wraps the delivered event in its own envelope. In CloudWatch Logs, the fields appear as a structured log event. In GCP Cloud Logging, they appear inside `jsonPayload`. In Elasticsearch, they appear inside the `_source` object of each document. Field names and values inside the event are identical across sinks.

When you build queries against a sink, use `@timestamp`. When you inspect Pod standard output with `kubectl logs`, use `timestamp`.

## Field reference

| Field                       | Type      | Description                                                                                                           |
| --------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------- |
| `timestamp`                 | string    | ISO 8601 timestamp when the APC API emitted the event. Renamed to `@timestamp` by Vector before delivery to a sink.   |
| `audit`                     | `boolean` | Always `true` for audit events. Vector filters delivered events on this field.                                        |
| `audit_version`             | string    | Schema version of the audit event contract. Currently `1.0`.                                                          |
| `correlationId`             | string    | UUID that links related events in the same request or operation chain.                                                |
| `sessionId`                 | string    | Identifier for the session that initiated the action.                                                                 |
| `userId`                    | string    | APC API user ID of the actor. Set to `system` for system-initiated worker events.                                     |
| `username`                  | string    | Username of the actor. Set to `system` for system-initiated worker events.                                            |
| `userEmail`                 | string    | Email address of the actor when available.                                                                            |
| `role`                      | string    | Role of the actor at the time of the action. Set to `SYSTEM` for system-initiated worker events.                      |
| `clientType`                | string    | Client that initiated the action, for example `ui` or `api`. Set to `system` for system-initiated worker events.      |
| `origin`                    | object    | Origin metadata when available. See [Origin fields](#origin-fields).                                                  |
| `action`                    | string    | Action identifier in the form `<entity>.<operation>`, for example `workspace.create`.                                 |
| `operation`                 | string    | Operation portion of `action`, for example `create`.                                                                  |
| `entity`                    | object    | Target of the action. Always contains `type`. May contain `id`, `name`, and other entity-specific fields.             |
| `outcome`                   | string    | One of `success`, `failure`, or `partial`.                                                                            |
| `statusCode`                | number    | HTTP-style status code. For GraphQL failures, the APC API derives this from the GraphQL error code.                   |
| `request.graphql.operation` | string    | GraphQL operation name for API events, for example `createWorkspace`.                                                 |
| `request.graphql.variables` | object    | GraphQL variables with sensitive values redacted. See [Sensitive data handling](#sensitive-data-handling).            |
| `response.duration`         | number    | Duration of the operation, in milliseconds.                                                                           |
| `component`                 | string    | `houston-api` for GraphQL events, `houston-worker` for background worker events.                                      |
| `level`                     | string    | `ERROR` when `outcome` is `failure`, otherwise `INFO`.                                                                |
| `message`                   | string    | Human-readable message generated from the action, entity, and outcome. See [Message generation](#message-generation). |

### Conditional fields

The following fields are present only in some events:

* `errorMessage` appears only when the APC API records an explicit error message.
* `changes` appears only for change-aware operations, such as `deployment.update_flag`. It holds before and after values for each changed field.
* `origin` appears only when origin metadata is available.
* `username` and `userEmail` can be absent when the APC API can't derive them from the request.

### Origin fields

When present, `origin` contains any of the following fields:

| Field          | Description                                            |
| -------------- | ------------------------------------------------------ |
| `requestIp`    | Remote address observed by the APC API server.         |
| `forwardedFor` | Value of the `X-Forwarded-For` header, when present.   |
| `podName`      | Name of the APC API Pod that handled the request.      |
| `podIp`        | IP address of the APC API Pod.                         |
| `hostIp`       | IP address of the Kubernetes node that hosted the Pod. |

## Sensitive data handling

The APC API sanitizes GraphQL variables before it writes them to `request.graphql.variables`. Matching is case-insensitive.

| Variable name   | Behavior                                                                                                                                                                                                               |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `password`      | Replaced with the string `<redacted>`.                                                                                                                                                                                 |
| `token`         | Replaced with the string `<redacted>`.                                                                                                                                                                                 |
| `secret`        | Replaced with the string `<redacted>`.                                                                                                                                                                                 |
| `authorization` | Replaced with the string `<redacted>`.                                                                                                                                                                                 |
| `apiKey`        | Replaced with `sha256:<12 hex chars>...<last 4 chars>`. The hash prefix and the last four characters of the original value are kept so that the same key hashes identically across events, without exposing the value. |

## Message generation

Each audit event includes a `message` field that APC API generates from the action, entity, and outcome. The patterns follow these rules:

* For most entity operations, success messages use the pattern `<Entity type> <entity> <past-tense action> successfully`, and failure messages use `Failed to <verb> <entity type> <entity>: <error>`.
  * Example: `Workspace abc-123 created successfully`
  * Example: `Failed to delete deployment analytics-prod: Not authorized`
* Authentication events use a dedicated pattern:
  * `User <identity> logged in successfully`
  * `Failed to login as <identity>: <error>`
  * `User <identity> logged out successfully`
* Workspace and Deployment membership operations, such as adding or removing a user or updating a role, use subject-aware messages:
  * `User alice@example.com added to workspace finance successfully`
  * `Team team-123 removed from deployment analytics successfully`
  * `User alice@example.com role changed to WORKSPACE_EDITOR in workspace finance successfully`
* Role binding operations use the pattern `<Entity type> <action> successfully (<role> to <subject>)`.
* Invite token operations include the invited email, for example `Invite for alice@example.com deleted successfully`.
* Flag update operations read from the `changes` payload, for example `Deployment analytics flags updated (paused: changed from disabled to enabled)`.

Messages use plain identifiers without surrounding quotes. When a failure carries an error message, it is appended after a colon.

## Operation inventory

The following tables list every operation that the APC API currently audits, grouped by category. Each row maps the GraphQL mutation or worker action to its audit `action` value. Expand a category to see its operations.

<AccordionGroup>
  <Accordion title="Workspace operations">
    | Mutation                           | Audit action                          |
    | ---------------------------------- | ------------------------------------- |
    | `createWorkspace`                  | `workspace.create`                    |
    | `updateWorkspace`                  | `workspace.update`                    |
    | `deleteWorkspace`                  | `workspace.delete`                    |
    | `workspaceAddUser`                 | `workspace.add_user`                  |
    | `workspaceRemoveUser`              | `workspace.remove_user`               |
    | `workspaceUpdateUserRole`          | `workspace.update_user_role`          |
    | `workspaceUpsertUserRole`          | `workspace.upsert_user_role`          |
    | `workspaceAddTeam`                 | `workspace.add_team`                  |
    | `workspaceRemoveTeam`              | `workspace.remove_team`               |
    | `workspaceUpdateTeamRole`          | `workspace.update_team_role`          |
    | `updateWorkspaceDeploymentsConfig` | `workspace.update_deployments_config` |
    | `deleteWorkspaceDeploymentsConfig` | `workspace.delete_deployments_config` |
  </Accordion>

  <Accordion title="Deployment operations">
    | Mutation                     | Audit action                           |
    | ---------------------------- | -------------------------------------- |
    | `deleteDeployment`           | `deployment.delete`                    |
    | `upsertDeployment`           | `deployment.upsert`                    |
    | `upgradeDeployment`          | `deployment.upgrade`                   |
    | `deployRollback`             | `deployment.rollback`                  |
    | `updateDeploymentImage`      | `deployment.update_image`              |
    | `updateDeploymentVariables`  | `deployment.update_variables`          |
    | `updateDeploymentKedaConfig` | `deployment.update_keda_config`        |
    | `updateDeploymentsResources` | `deployment.update_resources`          |
    | `deploymentAlertsUpdate`     | `deployment.alerts_update`             |
    | `deploymentAddUserRole`      | `deployment.add_user_role`             |
    | `deploymentRemoveUserRole`   | `deployment.remove_user_role`          |
    | `deploymentUpdateUserRole`   | `deployment.update_user_role`          |
    | `deploymentAddTeamRole`      | `deployment.add_team_role`             |
    | `deploymentRemoveTeamRole`   | `deployment.remove_team_role`          |
    | `deploymentUpdateTeamRole`   | `deployment.update_team_role`          |
    | `updateDeploymentConfig`     | `deployment.update_deployments_config` |
    | `deleteDeploymentConfig`     | `deployment.delete_deployments_config` |
  </Accordion>

  <Accordion title="Deploy revision operations">
    | Mutation                 | Audit action              |
    | ------------------------ | ------------------------- |
    | `createDeployRevision`   | `deploy_revision.create`  |
    | `cleanupDeployRevisions` | `deploy_revision.cleanup` |
  </Accordion>

  <Accordion title="User operations">
    | Mutation                 | Audit action                |
    | ------------------------ | --------------------------- |
    | `createUser`             | `user.create`               |
    | `removeUser`             | `user.remove`               |
    | `inviteUser`             | `user.invite`               |
    | `resendConfirmation`     | `user.resend_confirmation`  |
    | `updateSelf`             | `user.update_self`          |
    | `verifyEmail`            | `user.verify_email`         |
    | `deleteTestUsersInBulk`  | `user.delete_bulk`          |
    | `updateUserTeamBindings` | `user.update_team_bindings` |
  </Accordion>

  <Accordion title="Service account operations">
    | Mutation                         | Audit action                        |
    | -------------------------------- | ----------------------------------- |
    | `createServiceAccount`           | `service_account.create`            |
    | `updateServiceAccount`           | `service_account.update`            |
    | `deleteServiceAccount`           | `service_account.delete`            |
    | `createWorkspaceServiceAccount`  | `workspace_service_account.create`  |
    | `updateWorkspaceServiceAccount`  | `workspace_service_account.update`  |
    | `deleteWorkspaceServiceAccount`  | `workspace_service_account.delete`  |
    | `createDeploymentServiceAccount` | `deployment_service_account.create` |
    | `updateDeploymentServiceAccount` | `deployment_service_account.update` |
    | `deleteDeploymentServiceAccount` | `deployment_service_account.delete` |
    | `createSystemServiceAccount`     | `system_service_account.create`     |
    | `updateSystemServiceAccount`     | `system_service_account.update`     |
    | `deleteSystemServiceAccount`     | `system_service_account.delete`     |
  </Accordion>

  <Accordion title="Role binding operations">
    | Mutation                      | Audit action                      |
    | ----------------------------- | --------------------------------- |
    | `createSystemRoleBinding`     | `system_role_binding.create`      |
    | `deleteSystemRoleBinding`     | `system_role_binding.delete`      |
    | `createTeamSystemRoleBinding` | `team_system_role_binding.create` |
    | `deleteTeamSystemRoleBinding` | `team_system_role_binding.delete` |
  </Accordion>

  <Accordion title="Team operations">
    | Mutation     | Audit action  |
    | ------------ | ------------- |
    | `createTeam` | `team.create` |
    | `updateTeam` | `team.update` |
    | `removeTeam` | `team.remove` |
  </Accordion>

  <Accordion title="Authentication operations">
    | Mutation         | Audit action           |
    | ---------------- | ---------------------- |
    | `createToken`    | `auth.login`           |
    | `logout`         | `auth.logout`          |
    | `forgotPassword` | `auth.forgot_password` |
    | `resetPassword`  | `auth.reset_password`  |
    | `confirmEmail`   | `auth.confirm_email`   |
  </Accordion>

  <Accordion title="Invite token operations">
    | Mutation            | Audit action          |
    | ------------------- | --------------------- |
    | `deleteInviteToken` | `invite_token.delete` |
  </Accordion>

  <Accordion title="Cluster operations">
    | Mutation              | Audit action            |
    | --------------------- | ----------------------- |
    | `registerCluster`     | `cluster.register`      |
    | `updateCluster`       | `cluster.update`        |
    | `deregisterCluster`   | `cluster.deregister`    |
    | `cleanupClusterAudit` | `cluster.cleanup_audit` |
  </Accordion>

  <Accordion title="Worker operations">
    The APC API emits worker events from background jobs with `component` set to `houston-worker`. Worker actions aren't triggered by a GraphQL mutation, so they have no mutation mapping. When a worker action has no authenticated actor, the APC API records `userId: "system"`, `username: "system"`, `role: "SYSTEM"`, and `clientType: "system"`.

    | Audit action                           | Description                                           |
    | -------------------------------------- | ----------------------------------------------------- |
    | `deployment.worker_create`             | Applies a newly created Deployment in the data plane. |
    | `deployment.worker_update`             | Applies a Deployment update in the data plane.        |
    | `deployment.worker_delete`             | Applies a Deployment deletion in the data plane.      |
    | `deployment.worker_update_image`       | Applies a Deployment Airflow image update.            |
    | `deployment.worker_update_variables`   | Applies a Deployment environment variable update.     |
    | `deployment.worker_cleanup_db`         | Cleans up the Deployment's Airflow metadata database. |
    | `deployment.worker_refresh_task_usage` | Refreshes task usage metrics for a Deployment.        |
    | `cluster.worker_update_config`         | Applies a cluster configuration update.               |
  </Accordion>
</AccordionGroup>

## Next steps

* To enable audit log shipping, see [Set up audit log shipping](/docs/astro-private-cloud/v-2-x/audit-logging-setup).
* For every Helm value that controls the sidecar and its sinks, see [Audit logging configuration reference](/docs/astro-private-cloud/v-2-x/audit-logging-reference).


# APC audit logging overview
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/audit-logging-overview

Learn how APC audit logging captures administrative actions on Astro Private Cloud and ships them to a supported external sink.

Astro Private Cloud (APC) audit logging produces a structured, per-action record of administrative activity on the Astro Private Cloud control plane. Each event captures who performed the action, what was changed, and the outcome. Events are emitted as JSON by the APC API and APC Worker components and are shipped to one configured external sink by a Vector sidecar.

Use audit logging to support internal compliance programs, answer after-the-fact questions about control plane changes, and investigate security events.

<Note>
  APC audit logging is introduced in Astro Private Cloud 2.x.
</Note>

## What gets audited

APC audit logging emits an audit event for each successful or failed administrative action, including:

* Workspace and Deployment lifecycle changes.
* User, team, service account, and role binding changes.
* Authentication events such as sign-in, sign-out, and password reset.
* Cluster registration and cluster configuration changes.
* Background worker operations that modify Deployments or cluster state.

Each event is a JSON object with stable field names. For the full event schema, the list of audited operations, and message formatting rules, see [Audit log schema and operations](/docs/astro-private-cloud/v-2-x/audit-log-schema).

### How it works

APC audit logging uses a Vector sidecar that runs alongside the APC API and APC Worker Pods.

1. The APC API writes structured JSON events to standard output as it normally does.
2. A built-in log wrapper script tees the APC API's output to a shared volume that the Vector sidecar reads.
3. Vector parses each line, keeps only entries where `audit == true`, adds Kubernetes and platform metadata, normalizes the timestamp to `@timestamp`, and converts the `level` field to uppercase.
4. Vector delivers the transformed event to the configured sink.

APC audit logging doesn't depend on the sidecar. If the sidecar is disabled, the APC API continues to emit audit events to standard output.

#### Supported sinks

Each sink is supported on a specific Kubernetes platform for the Astro Private Cloud control plane. Cross-cloud configurations, such as shipping to AWS CloudWatch Logs from a GKE cluster, aren't supported.

| Sink                | Supported Kubernetes platform  | Authentication options                                               |
| ------------------- | ------------------------------ | -------------------------------------------------------------------- |
| AWS CloudWatch Logs | Amazon EKS                     | IAM Roles for Service Accounts (IRSA); static AWS credentials secret |
| GCP Cloud Logging   | Google Kubernetes Engine (GKE) | Workload Identity; GCP service account JSON key secret               |
| Elasticsearch       | Amazon EKS, GKE, or AKS        | Basic auth or anonymous, each optionally with a custom CA            |

<Note>
  For this release, exactly one sink can be enabled per installation. Enabling more than one sink results in a Helm validation error. Enabling the sidecar without a sink also fails validation.
</Note>

#### Default state

APC audit logging is disabled by default. The sidecar isn't injected and no audit events are shipped to any external system until you enable it through Helm values.

Because APC audit logging always writes audit events to standard output, you can inspect recent events with `kubectl logs` on the APC API or APC Worker Pods even when the sidecar is disabled. Use the sidecar when you need durable, long-term retention in a sink that you can query.

## Next steps

* To enable audit log shipping, see [Set up audit log shipping](/docs/astro-private-cloud/v-2-x/audit-logging-setup).
* To review every Helm value that controls the sidecar and its sinks, see [Audit logging configuration reference](/docs/astro-private-cloud/v-2-x/audit-logging-reference).
* To understand the shape of each audit event and the full inventory of audited operations, see [Audit log schema and operations](/docs/astro-private-cloud/v-2-x/audit-log-schema).


# APC audit logging configuration reference
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/audit-logging-reference

Reference for every Helm value that controls the APC audit logging sidecar and its supported sinks on Astro Private Cloud.

This document lists every Helm value that controls the Astro Private Cloud (APC) audit logging sidecar and its sinks. For task-oriented instructions, see [Set up audit log shipping](/docs/astro-private-cloud/v-2-x/audit-logging-setup).

All values in this document live under `houston.logging.loggingSidecar` in the `astronomer` chart's values file. When you use the umbrella chart, prefix the path with `astronomer.`, so the full path becomes `astronomer.houston.logging.loggingSidecar.*`.

## Sidecar-level values

These values control the Vector sidecar itself, independent of which sink you enable.

| Key                                        | Type      | Default | Description                                                                                                                                     |
| ------------------------------------------ | --------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled`                                  | `boolean` | `false` | Enables the Vector sidecar on the APC API and APC Worker Pods. When `false`, no audit events are shipped to any sink.                           |
| `resources.requests.cpu`                   | string    | `50m`   | CPU request for the Vector sidecar container.                                                                                                   |
| `resources.requests.memory`                | string    | `128Mi` | Memory request for the Vector sidecar container.                                                                                                |
| `resources.limits.cpu`                     | string    | `200m`  | CPU limit for the Vector sidecar container.                                                                                                     |
| `resources.limits.memory`                  | string    | `256Mi` | Memory limit for the Vector sidecar container.                                                                                                  |
| `securityContext.runAsNonRoot`             | `boolean` | `true`  | Vector sidecar's `runAsNonRoot` security context. Override only when your cluster's Pod security policy requires a different value.             |
| `securityContext.allowPrivilegeEscalation` | `boolean` | `false` | Vector sidecar's `allowPrivilegeEscalation` security context. Override only when your cluster's Pod security policy requires a different value. |

The Vector sidecar ships with a secure-by-default security context. The chart already applies `securityContext.runAsNonRoot: true` and `securityContext.allowPrivilegeEscalation: false`, so most installations don't need to set these values.

The Vector image is controlled at the chart level, not on the sidecar, through `images.vector.repository` and `images.vector.tag`. These values are managed by the chart and change between chart releases, so they aren't pinned in this reference. For the current defaults, see the `astronomer/astronomer` chart values file.

### CloudWatch sink values

Set under `houston.logging.loggingSidecar.cloudwatch`. Use this sink on Amazon EKS.

| Key            | Type      | Default                     | Description                                                                                                                                                                                               |
| -------------- | --------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled`      | `boolean` | `false`                     | Enables the CloudWatch sink. Only one sink can be enabled at a time.                                                                                                                                      |
| `region`       | string    | `""`                        | AWS region of the target log group. Required when `enabled` is `true`.                                                                                                                                    |
| `logGroupName` | string    | `/astronomer/houston/audit` | Target CloudWatch log group. The log group must exist and the IAM principal used by Vector must be allowed to write to it.                                                                                |
| `useIRSA`      | `boolean` | `true`                      | When `true`, Vector authenticates with AWS using IAM Roles for Service Accounts (IRSA). The `eks.amazonaws.com/role-arn` annotation on `houston.serviceAccount.annotations` must reference the IRSA role. |
| `secretName`   | string    | `houston-cloudwatch-creds`  | Name of a Kubernetes secret containing `aws_access_key_id` and `aws_secret_access_key`. Used when `useIRSA` is `false`.                                                                                   |

### GCP Cloud Logging sink values

Set under `houston.logging.loggingSidecar.gcpCloudLogging`. Use this sink on Google Kubernetes Engine (GKE).

| Key                     | Type      | Default                     | Description                                                                                                                                                                                              |
| ----------------------- | --------- | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled`               | `boolean` | `false`                     | Enables the GCP Cloud Logging sink. Only one sink can be enabled at a time.                                                                                                                              |
| `projectId`             | string    | `""`                        | Google Cloud project that receives the logs. Required when `enabled` is `true`.                                                                                                                          |
| `logId`                 | string    | `houston-audit`             | Custom log ID that identifies this APC audit stream in Cloud Logging.                                                                                                                                    |
| `resource.type`         | string    | `k8s_container`             | Monitored resource type. `k8s_container` is the expected value for GKE Pods.                                                                                                                             |
| `resource.location`     | string    | `""`                        | GKE cluster location, for example `us-east4` or `us-east4-b`. Required when `enabled` is `true`.                                                                                                         |
| `resource.clusterName`  | string    | `""`                        | GKE cluster name as it appears in Cloud Logging. Required when `enabled` is `true`.                                                                                                                      |
| `severityKey`           | string    | `level`                     | Field in each audit event that Vector maps to GCP severity.                                                                                                                                              |
| `useWorkloadIdentity`   | `boolean` | `true`                      | When `true`, Vector authenticates with GCP through Workload Identity. The `iam.gke.io/gcp-service-account` annotation on `houston.serviceAccount.annotations` must reference the Google service account. |
| `credentialsSecretName` | string    | `houston-gcp-logging-creds` | Name of a Kubernetes secret that contains a GCP service account JSON key. Used when `useWorkloadIdentity` is `false`.                                                                                    |
| `credentialsSecretKey`  | string    | `key.json`                  | Key within `credentialsSecretName` that holds the JSON key file. The file is mounted at `/etc/gcp-credentials/<credentialsSecretKey>`.                                                                   |

<Note>
  `projectId`, `resource.location`, and `resource.clusterName` are required when `gcpCloudLogging.enabled` is `true`. Empty or whitespace-only values are rejected by the chart.
</Note>

### Elasticsearch sink values

Set under `houston.logging.loggingSidecar.elasticsearch`. Use this sink to ship events to an external Elasticsearch cluster. The Elasticsearch sink is supported when the Astro Private Cloud control plane runs on Amazon EKS, GKE, or AKS.

| Key                | Type      | Default                       | Description                                                                                                                                             |
| ------------------ | --------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled`          | `boolean` | `false`                       | Enables the Elasticsearch sink. Only one sink can be enabled at a time.                                                                                 |
| `endpoint`         | string    | `~` (null)                    | External Elasticsearch URL, for example `https://es.example.com:9200`. Required when `enabled` is `true`.                                               |
| `index`            | string    | `houston-audit-%Y.%m.%d`      | Index name pattern. Accepts `strftime` tokens for date-based indices.                                                                                   |
| `apiVersion`       | string    | `v8`                          | Elasticsearch API version.                                                                                                                              |
| `auth.strategy`    | string    | `basic`                       | Authentication strategy. Supported values are `basic` and `none`.                                                                                       |
| `auth.secretName`  | string    | `houston-elasticsearch-creds` | Name of a Kubernetes secret that contains `username` and `password` keys. Required when `auth.strategy` is `basic`.                                     |
| `tls.enabled`      | `boolean` | `false`                       | When `true`, Vector uses the CA certificate in `caSecretName` to validate the Elasticsearch server certificate.                                         |
| `tls.caSecretName` | string    | `""`                          | Name of a Kubernetes secret that contains a `ca.pem` entry. The chart mounts the secret at `/etc/es-tls/ca.pem`. Required when `tls.enabled` is `true`. |

`auth.strategy` and `tls.enabled` are independent. The chart supports all four combinations: `none`, `basic`, `none` with a custom CA, and `basic` with a custom CA.

## APC service account annotations

The IRSA and Workload Identity integration points are annotations on the `<release>-houston-bootstrapper` service account. The chart passes any key-value pairs you set on `houston.serviceAccount.annotations` through to the rendered `ServiceAccount` manifest.

| Annotation                       | Use with                                                |
| -------------------------------- | ------------------------------------------------------- |
| `eks.amazonaws.com/role-arn`     | CloudWatch sink with `useIRSA: true`                    |
| `iam.gke.io/gcp-service-account` | GCP Cloud Logging sink with `useWorkloadIdentity: true` |

Both the APC API and APC Worker Pods use the same `<release>-houston-bootstrapper` service account, so a single annotation applies to both deployments.

## Validation rules

The chart validates `houston.logging.loggingSidecar` at render time. The following rules fail `helm upgrade` with a descriptive error when violated:

* When `loggingSidecar.enabled` is `true`, exactly one of `cloudwatch.enabled`, `gcpCloudLogging.enabled`, or `elasticsearch.enabled` must also be `true`.
* When `gcpCloudLogging.enabled` is `true`, `projectId`, `resource.location`, and `resource.clusterName` must each be set to a non-whitespace value.
* When `elasticsearch.enabled` is `true`, `endpoint` must be set.

## Unsupported values

`extraSinks` on `houston.logging.loggingSidecar` isn't accepted by the chart. APC audit logging ships only to the three sinks documented on this page.

## Next steps

* To apply these values to an installation, see [Set up audit log shipping](/docs/astro-private-cloud/v-2-x/audit-logging-setup).
* For the shape of each audit event and the list of audited operations, see [Audit log schema and operations](/docs/astro-private-cloud/v-2-x/audit-log-schema).


# Set up APC audit log shipping
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/audit-logging-setup

Enable the Vector sidecar and configure a supported sink to ship APC audit events from Astro Private Cloud to AWS CloudWatch, GCP Cloud Logging, or Elasticsearch.

This document explains how to enable the Astro Private Cloud (APC) audit log sidecar and ship events to one supported sink. For background on what the feature does and which configurations are supported, see [APC audit logging overview](/docs/astro-private-cloud/v-2-x/audit-logging-overview).

<Note>
  Exactly one sink can be enabled per installation for this release. Enabling the sidecar with zero or more than one sink causes a Helm validation error.
</Note>

## Prerequisites

* An Astro Private Cloud 2.x installation that you can upgrade with Helm.
* Access to the Helm values file used by the installation.
* `kubectl` configured against the target cluster.
* Permissions to create or modify cloud resources for the sink you choose:
  * CloudWatch on EKS: permissions to create IAM policies and roles, and a CloudWatch log group.
  * GCP Cloud Logging on GKE: permissions to create Google service accounts and grant IAM bindings in the target project.
  * Elasticsearch: a reachable endpoint and, if required, credentials and a CA certificate.

### Choose a sink

Use the AWS CloudWatch Logs sink when the Astro Private Cloud control plane runs on Amazon EKS. Use the GCP Cloud Logging sink when it runs on Google Kubernetes Engine (GKE). Use the Elasticsearch sink on Amazon EKS, GKE, or Azure Kubernetes Service (AKS).

<Tabs>
  <Tab title="AWS CloudWatch Logs">
    Use this sink when the Astro Private Cloud control plane runs on Amazon EKS. The recommended authentication method is IAM Roles for Service Accounts (IRSA). Static AWS credentials held in a Kubernetes secret are supported as a fallback when IRSA isn't in use on the EKS cluster.

    #### Prerequisites

    * The Astro Private Cloud control plane runs on Amazon EKS.
    * The AWS CLI is installed and authenticated against the target account.
    * For the IRSA path, the EKS cluster has, or can be associated with, an OIDC identity provider.
    * For the static-credentials path, an IAM principal with permission to write to the target CloudWatch log group.

    ##### Environment variables

    The following variables are referenced throughout this section. Set them to match your installation before running the commands.

    ```bash wrap theme={null}
    export EKS_CLUSTER_NAME="<cluster-name>"
    export AWS_REGION="<region>"
    export AWS_ACCOUNT_ID="<account-id>"
    export K8S_NAMESPACE="astronomer"
    export HELM_RELEASE="astronomer"
    export K8S_SA="${HELM_RELEASE}-houston-bootstrapper"
    export IRSA_ROLE_NAME="HoustonCloudWatchRole"
    export CW_LOG_GROUP="/astronomer/houston/audit"
    ```

    ##### Configure IRSA (recommended)

    <Steps>
      <Step title="Create the CloudWatch log group">
        ```bash wrap theme={null}
        aws logs create-log-group \
          --log-group-name "$CW_LOG_GROUP" \
          --region "$AWS_REGION"
        ```

        Optionally set a retention policy on the log group:

        ```bash wrap theme={null}
        aws logs put-retention-policy \
          --log-group-name "$CW_LOG_GROUP" \
          --region "$AWS_REGION" \
          --retention-in-days 30
        ```
      </Step>

      <Step title="Associate the EKS cluster OIDC provider">
        ```bash wrap theme={null}
        OIDC_URL=$(aws eks describe-cluster \
          --name "$EKS_CLUSTER_NAME" \
          --region "$AWS_REGION" \
          --query "cluster.identity.oidc.issuer" \
          --output text)
        OIDC_ID=${OIDC_URL#https://}

        aws iam list-open-id-connect-providers \
          | grep "$(echo "$OIDC_ID" | awk -F/ '{print $NF}')" \
          || eksctl utils associate-iam-oidc-provider \
               --cluster "$EKS_CLUSTER_NAME" \
               --region "$AWS_REGION" \
               --approve
        ```
      </Step>

      <Step title="Create the IAM policy">
        ```bash expandable wrap theme={null}
        cat > /tmp/houston-cloudwatch-policy.json <<EOF
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "CloudWatchDescribe",
              "Effect": "Allow",
              "Action": [
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
              ],
              "Resource": "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:*"
            },
            {
              "Sid": "CloudWatchWriteAuditGroup",
              "Effect": "Allow",
              "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
              ],
              "Resource": [
                "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:${CW_LOG_GROUP}",
                "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:log-group:${CW_LOG_GROUP}:*"
              ]
            }
          ]
        }
        EOF

        aws iam create-policy \
          --policy-name HoustonCloudWatchLogsPolicy \
          --policy-document file:///tmp/houston-cloudwatch-policy.json
        ```
      </Step>

      <Step title="Create the IRSA role and attach the policy">
        ```bash wrap theme={null}
        cat > /tmp/trust-policy.json <<EOF
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ID}"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                "StringEquals": {
                  "${OIDC_ID}:sub": "system:serviceaccount:${K8S_NAMESPACE}:${K8S_SA}",
                  "${OIDC_ID}:aud": "sts.amazonaws.com"
                }
              }
            }
          ]
        }
        EOF

        aws iam create-role \
          --role-name "$IRSA_ROLE_NAME" \
          --assume-role-policy-document file:///tmp/trust-policy.json

        aws iam attach-role-policy \
          --role-name "$IRSA_ROLE_NAME" \
          --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/HoustonCloudWatchLogsPolicy"
        ```

        If the IRSA role already exists and you only need to bind it to a new EKS cluster, use `aws iam update-assume-role-policy` against the existing role instead of `aws iam create-role`.
      </Step>

      <Step title="Apply the Helm values override">
        Add the following to the values file used by the installation and run `helm upgrade`:

        ```yaml wrap theme={null}
        astronomer:
          houston:
            serviceAccount:
              annotations:
                eks.amazonaws.com/role-arn: "arn:aws:iam::<AWS_ACCOUNT_ID>:role/HoustonCloudWatchRole"
            logging:
              loggingSidecar:
                enabled: true
                cloudwatch:
                  enabled: true
                  region: "<AWS_REGION>"
                  logGroupName: "/astronomer/houston/audit"
                  useIRSA: true
        ```
      </Step>
    </Steps>

    #### Configure static AWS credentials (fallback)

    Use this configuration on EKS when IRSA isn't in use.

    <Steps>
      <Step title="Create the CloudWatch log group">
        ```bash wrap theme={null}
        aws logs create-log-group \
          --log-group-name "$CW_LOG_GROUP" \
          --region "$AWS_REGION"
        ```
      </Step>

      <Step title="Create a Kubernetes secret with AWS credentials">
        ```bash wrap theme={null}
        kubectl create secret generic houston-cloudwatch-creds \
          --from-literal=aws_access_key_id="<AWS_ACCESS_KEY_ID>" \
          --from-literal=aws_secret_access_key="<AWS_SECRET_ACCESS_KEY>" \
          -n "$K8S_NAMESPACE"
        ```

        The IAM principal whose credentials you use must be allowed to write to the target log group. The policy shown in the IRSA section is a suitable template.
      </Step>

      <Step title="Apply the Helm values override">
        ```yaml wrap theme={null}
        astronomer:
          houston:
            logging:
              loggingSidecar:
                enabled: true
                cloudwatch:
                  enabled: true
                  region: "<AWS_REGION>"
                  logGroupName: "/astronomer/houston/audit"
                  useIRSA: false
                  secretName: "houston-cloudwatch-creds"
        ```
      </Step>
    </Steps>

    #### Verify

    After the upgrade completes, confirm that the Vector sidecar is running and that audit events are reaching CloudWatch:

    ```bash wrap theme={null}
    HOUSTON_POD=$(kubectl get pods -n "$K8S_NAMESPACE" \
      -l component=houston \
      -o jsonpath='{.items[0].metadata.name}')

    kubectl logs -n "$K8S_NAMESPACE" "$HOUSTON_POD" -c vector --tail=20

    aws logs tail "$CW_LOG_GROUP" --region "$AWS_REGION" --since 5m
    ```

    Perform any action that the APC API audits, such as creating a Workspace, and confirm a matching event appears in the log group within a few seconds.
  </Tab>

  <Tab title="GCP Cloud Logging">
    Use this sink when the Astro Private Cloud control plane runs on Google Kubernetes Engine (GKE). The recommended authentication method is Workload Identity. A GCP service account JSON key held in a Kubernetes secret is supported as a fallback when Workload Identity isn't in use on the GKE cluster.

    #### Prerequisites

    * The Astro Private Cloud control plane runs on GKE.
    * `gcloud` is installed and authenticated against the target Google Cloud project.
    * For the Workload Identity path, the GKE cluster has Workload Identity enabled.
    * For the service-account-key path, a Google service account with the `roles/logging.logWriter` role in the target Google Cloud project.

    ##### Environment variables

    ```bash wrap theme={null}
    export GCP_PROJECT_ID="<project-id>"
    export GKE_CLUSTER_NAME="<cluster-name>"
    export GKE_LOCATION="<zone-or-region>"
    export K8S_NAMESPACE="astronomer"
    export HELM_RELEASE="astronomer"
    export K8S_SA="${HELM_RELEASE}-houston-bootstrapper"
    export GCP_SA_NAME="houston-logging"
    export GCP_SA_EMAIL="${GCP_SA_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com"
    ```

    ##### Configure Workload Identity (recommended)

    <Steps>
      <Step title="Verify Workload Identity is enabled">
        ```bash wrap theme={null}
        gcloud container clusters describe "$GKE_CLUSTER_NAME" \
          --location "$GKE_LOCATION" \
          --format="value(workloadIdentityConfig.workloadPool)"
        ```

        Expected output is `${GCP_PROJECT_ID}.svc.id.goog`. If the output is empty, enable Workload Identity on the cluster before continuing.
      </Step>

      <Step title="Create the Google service account and grant log write access">
        ```bash wrap theme={null}
        gcloud iam service-accounts create "$GCP_SA_NAME" \
          --display-name="Houston Audit Log Writer" \
          --project="$GCP_PROJECT_ID"

        gcloud projects add-iam-policy-binding "$GCP_PROJECT_ID" \
          --member="serviceAccount:${GCP_SA_EMAIL}" \
          --role="roles/logging.logWriter"
        ```
      </Step>

      <Step title="Bind the Google service account to the APC API Kubernetes service account">
        ```bash wrap theme={null}
        gcloud iam service-accounts add-iam-policy-binding "$GCP_SA_EMAIL" \
          --role="roles/iam.workloadIdentityUser" \
          --member="serviceAccount:${GCP_PROJECT_ID}.svc.id.goog[${K8S_NAMESPACE}/${K8S_SA}]" \
          --project="$GCP_PROJECT_ID"
        ```
      </Step>

      <Step title="Apply the Helm values override">
        ```yaml wrap theme={null}
        astronomer:
          houston:
            serviceAccount:
              annotations:
                iam.gke.io/gcp-service-account: "<GCP_SA_EMAIL>"
            logging:
              loggingSidecar:
                enabled: true
                gcpCloudLogging:
                  enabled: true
                  projectId: "<GCP_PROJECT_ID>"
                  logId: "houston-audit"
                  resource:
                    type: "k8s_container"
                    location: "<GKE_LOCATION>"
                    clusterName: "<GKE_CLUSTER_NAME>"
                  severityKey: "level"
                  useWorkloadIdentity: true
        ```

        `projectId`, `resource.location`, and `resource.clusterName` are required when `gcpCloudLogging.enabled` is true. Leaving any of these empty causes a Helm validation error.
      </Step>
    </Steps>

    #### Configure a service account key (fallback)

    Use this configuration on GKE when Workload Identity isn't in use.

    <Steps>
      <Step title="Create a key for the Google service account">
        ```bash wrap theme={null}
        gcloud iam service-accounts keys create /tmp/houston-logging-key.json \
          --iam-account="$GCP_SA_EMAIL"
        ```
      </Step>

      <Step title="Create a Kubernetes secret from the key">
        ```bash wrap theme={null}
        kubectl create secret generic houston-gcp-logging-creds \
          --from-file=key.json=/tmp/houston-logging-key.json \
          --namespace "$K8S_NAMESPACE"

        rm /tmp/houston-logging-key.json
        ```
      </Step>

      <Step title="Apply the Helm values override">
        ```yaml wrap theme={null}
        astronomer:
          houston:
            logging:
              loggingSidecar:
                enabled: true
                gcpCloudLogging:
                  enabled: true
                  projectId: "<GCP_PROJECT_ID>"
                  logId: "houston-audit"
                  resource:
                    type: "k8s_container"
                    location: "<GKE_LOCATION>"
                    clusterName: "<GKE_CLUSTER_NAME>"
                  severityKey: "level"
                  useWorkloadIdentity: false
                  credentialsSecretName: "houston-gcp-logging-creds"
                  credentialsSecretKey: "key.json"
        ```
      </Step>
    </Steps>

    #### Verify

    ```bash wrap theme={null}
    HOUSTON_POD=$(kubectl get pods -n "$K8S_NAMESPACE" \
      -l component=houston \
      -o jsonpath='{.items[0].metadata.name}')

    kubectl logs -n "$K8S_NAMESPACE" "$HOUSTON_POD" -c vector --tail=20

    gcloud logging read \
      "logName=\"projects/${GCP_PROJECT_ID}/logs/houston-audit\"" \
      --project="$GCP_PROJECT_ID" \
      --limit=5 \
      --freshness=5m
    ```

    Perform any action that the APC API audits and confirm a matching entry appears in the log stream within a few seconds.
  </Tab>

  <Tab title="Elasticsearch">
    Use this sink to ship APC API audit events to an external Elasticsearch cluster. The Elasticsearch sink is supported when the Astro Private Cloud control plane runs on Amazon EKS, GKE, or Azure Kubernetes Service (AKS). This sink needs no cloud-provider IAM configuration.

    #### Prerequisites

    * A reachable Elasticsearch endpoint on version 8.
    * For basic auth, a username and password.
    * For custom CA trust, a PEM-formatted CA certificate.

    ##### Environment variables

    ```bash wrap theme={null}
    export K8S_NAMESPACE="astronomer"
    export HELM_RELEASE="astronomer"
    export ES_ENDPOINT="https://es.example.com:9200"
    export ES_USERNAME="elastic"
    export ES_PASSWORD="<password>"
    ```

    #### Configure basic auth

    <Steps>
      <Step title="Create a Kubernetes secret with the Elasticsearch credentials">
        ```bash wrap theme={null}
        kubectl create secret generic houston-elasticsearch-creds \
          --from-literal=username="$ES_USERNAME" \
          --from-literal=password="$ES_PASSWORD" \
          -n "$K8S_NAMESPACE"
        ```
      </Step>

      <Step title="Apply the Helm values override">
        ```yaml wrap theme={null}
        astronomer:
          houston:
            logging:
              loggingSidecar:
                enabled: true
                elasticsearch:
                  enabled: true
                  endpoint: "https://es.example.com:9200"
                  index: "houston-audit-%Y.%m.%d"
                  apiVersion: "v8"
                  auth:
                    strategy: "basic"
                    secretName: "houston-elasticsearch-creds"
        ```
      </Step>
    </Steps>

    #### Configure basic auth with a custom CA

    Use this variant when the Elasticsearch endpoint presents a certificate signed by a private or internal CA that isn't already trusted by the Vector sidecar.

    <Steps>
      <Step title="Create a Kubernetes secret with the credentials">
        ```bash wrap theme={null}
        kubectl create secret generic houston-elasticsearch-creds \
          --from-literal=username="$ES_USERNAME" \
          --from-literal=password="$ES_PASSWORD" \
          -n "$K8S_NAMESPACE"
        ```
      </Step>

      <Step title="Create a Kubernetes secret with the CA certificate">
        The secret key must be named `ca.pem`. The Vector sidecar expects the CA at `/etc/es-tls/ca.pem` and the chart mounts the referenced secret to that path.

        ```bash wrap theme={null}
        kubectl create secret generic houston-es-ca \
          --from-file=ca.pem=/path/to/ca-certificate.pem \
          -n "$K8S_NAMESPACE"
        ```
      </Step>

      <Step title="Apply the Helm values override">
        ```yaml wrap theme={null}
        astronomer:
          houston:
            logging:
              loggingSidecar:
                enabled: true
                elasticsearch:
                  enabled: true
                  endpoint: "https://es.example.com:9200"
                  index: "houston-audit-%Y.%m.%d"
                  apiVersion: "v8"
                  auth:
                    strategy: "basic"
                    secretName: "houston-elasticsearch-creds"
                  tls:
                    enabled: true
                    caSecretName: "houston-es-ca"
        ```
      </Step>
    </Steps>

    #### Configure anonymous auth

    Use this configuration when the Elasticsearch endpoint accepts unauthenticated requests. This configuration needs no credentials secret.

    ```yaml wrap theme={null}
    astronomer:
      houston:
        logging:
          loggingSidecar:
            enabled: true
            elasticsearch:
              enabled: true
              endpoint: "http://es.example.com:9200"
              index: "houston-audit-%Y.%m.%d"
              apiVersion: "v8"
              auth:
                strategy: "none"
    ```

    ##### Configure anonymous auth with a custom CA

    Use this variant when the Elasticsearch endpoint accepts unauthenticated requests over TLS and presents a certificate signed by a private or internal CA.

    <Steps>
      <Step title="Create a Kubernetes secret with the CA certificate">
        The secret key must be named `ca.pem`. The Vector sidecar expects the CA at `/etc/es-tls/ca.pem` and the chart mounts the referenced secret to that path.

        ```bash wrap theme={null}
        kubectl create secret generic houston-es-ca \
          --from-file=ca.pem=/path/to/ca-certificate.pem \
          -n "$K8S_NAMESPACE"
        ```
      </Step>

      <Step title="Apply the Helm values override">
        ```yaml wrap theme={null}
        astronomer:
          houston:
            logging:
              loggingSidecar:
                enabled: true
                elasticsearch:
                  enabled: true
                  endpoint: "https://es.example.com:9200"
                  index: "houston-audit-%Y.%m.%d"
                  apiVersion: "v8"
                  auth:
                    strategy: "none"
                  tls:
                    enabled: true
                    caSecretName: "houston-es-ca"
        ```
      </Step>
    </Steps>

    #### Verify

    Check the Vector sidecar logs for sink health:

    ```bash wrap theme={null}
    HOUSTON_POD=$(kubectl get pods -n "$K8S_NAMESPACE" \
      -l component=houston \
      -o jsonpath='{.items[0].metadata.name}')

    kubectl logs -n "$K8S_NAMESPACE" "$HOUSTON_POD" -c vector --tail=20
    ```

    Query the Elasticsearch endpoint for the `houston-audit-*` indices. If the endpoint requires authentication:

    ```bash wrap theme={null}
    curl --cacert /path/to/ca-certificate.pem -u "$ES_USERNAME:$ES_PASSWORD" \
      "$ES_ENDPOINT/_cat/indices/houston-audit-*?v"
    ```

    If the endpoint accepts anonymous requests:

    ```bash wrap theme={null}
    curl --cacert /path/to/ca-certificate.pem \
      "$ES_ENDPOINT/_cat/indices/houston-audit-*?v"
    ```

    Omit the `--cacert` flag when the endpoint uses a publicly trusted certificate. Perform any action that the APC API audits and confirm that a new document appears in the matching `houston-audit-*` index.
  </Tab>
</Tabs>

## Disable audit log shipping

To stop shipping audit events, set the sidecar to disabled and run `helm upgrade`:

```yaml wrap theme={null}
astronomer:
  houston:
    logging:
      loggingSidecar:
        enabled: false
```

`helm upgrade` removes the Vector sidecar from the next Pod restart. The APC API continues to emit audit events to standard output, so you can still inspect recent events with `kubectl logs` on the APC API and APC Worker Pods.

## Common issues

<AccordionGroup>
  <Accordion title="Helm reports 'supports exactly one sink at a time'">
    The sidecar is enabled with more than one sink. Edit the values file so that only one of `cloudwatch.enabled`, `gcpCloudLogging.enabled`, or `elasticsearch.enabled` is true, then run `helm upgrade` again.
  </Accordion>

  <Accordion title="Helm reports 'requires at least one supported sink'">
    The sidecar is enabled but no sink is selected. Set one of `cloudwatch.enabled`, `gcpCloudLogging.enabled`, or `elasticsearch.enabled` to true, or set `loggingSidecar.enabled` to false.
  </Accordion>

  <Accordion title="Helm reports a required GCP Cloud Logging value is missing">
    `gcpCloudLogging.projectId`, `gcpCloudLogging.resource.location`, and `gcpCloudLogging.resource.clusterName` are required when GCP Cloud Logging is enabled. Whitespace-only values are also rejected. Set all three to concrete values and run `helm upgrade` again.
  </Accordion>

  <Accordion title="Vector reports AccessDenied errors on CloudWatch">
    The IAM principal that Vector uses can't write to the target log group.

    * For IRSA, check that the IRSA role trust policy names the correct OIDC provider, namespace, and service account, and that `HoustonCloudWatchLogsPolicy` is attached to the role. Also verify that the `eks.amazonaws.com/role-arn` annotation on `houston.serviceAccount.annotations` points to the same role ARN.
    * For static credentials, check that `houston-cloudwatch-creds` contains valid `aws_access_key_id` and `aws_secret_access_key` values and that the corresponding IAM user or role is allowed to write to the target log group.
  </Accordion>

  <Accordion title="GCP Cloud Logging rejects entries with HTTP 400">
    The monitored resource is invalid. Confirm that `resource.type` is `k8s_container`, and that `resource.location` and `resource.clusterName` match the GKE cluster as it appears in Cloud Logging.
  </Accordion>

  <Accordion title="Vector can't connect to Elasticsearch">
    Check that `endpoint` is reachable from within the cluster and that, if TLS is in use, the CA certificate in `caSecretName` signs the server certificate presented by the endpoint. If basic auth is enabled, verify that the username and password in `houston-elasticsearch-creds` are correct.
  </Accordion>
</AccordionGroup>

## Next steps

* For a field-by-field description of each value under `houston.logging.loggingSidecar`, see [Audit logging configuration reference](/docs/astro-private-cloud/v-2-x/audit-logging-reference).
* For the shape of each audit event and the list of audited operations, see [Audit log schema and operations](/docs/astro-private-cloud/v-2-x/audit-log-schema).


# Bring your own Kubernetes service accounts
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/byo-service-accounts

Use a pre-existing Kubernetes service account definition for Astro Private Cloud.

In Astro Private Cloud, you can disable automatic creation of Service Accounts (SA), and use a pre-created service account. When you do this, you can either define service accounts manually, or use a service account creation template.

Using a pre-created service account, Organizations can create service accounts using a central authority or system, without granting Astro Private Cloud similarly elevated permissions.

## Step 1: Create a service account template

Use the [registry template](https://github.com/astronomer/astronomer/blob/master/charts/astronomer/templates/registry/registry-serviceaccount.yaml) to create a service account template. The following examples use a service account saved with the name, `custom-sa`.

## Step 2: Disable automatic service account creation

1. Disable Astronomer from creating Roles, RoleBindings, and other SAs in the namespace by setting the global config `rbac.enabled` and `serviceAccount.create` to `false` globally:

```yaml wrap theme={null}
global:
  rbac:
    enabled: false
  serviceAccount:
    create: false
```

2. You must also set `serviceAccount.create` to `false` for each component that will use a custom SA: `commander`, `configsyncer`, `houston`, and `houston-worker`.

```yaml expandable wrap theme={null}
global:
  deployMechanisms:
    dagOnlyDeployment:
      enabled: true
      serviceAccount:
        create: false
astronomer:
  airflowChartVersion: <your-airflow-chart-version>
  houston:
    config:
      deployments:
        helm:
          airflow:
            rbac:
              create: false

            scheduler:
              serviceAccount:
                create: false

            flower:
              serviceAccount:
                create: false

            apiServer:
              serviceAccount:
                create: false

            triggerer:
              serviceAccount:
                create: false

            pgbouncer:
              serviceAccount:
                create: false

            migrateDatabaseJob:
              serviceAccount:
                create: false

            statsd:
              serviceAccount:
                create: false
            redis:
              serviceAccount:
                create: false

            cleanup:
              serviceAccount:
                create: false

            workers:
              serviceAccount:
                create: false
```

## Step 3: Apply the config change

Then [apply the config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).


# Clean up and delete task metadata from Airflow DB
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/clean-up-task-metadata

Clean up task metadata from your Airflow metadata DB on Astro Private Cloud.

The APC API GraphQL query, `cleanupAirflowDb`, triggers the Airflow metadata cleanup job. You can run a cleanup job to automatically delete task and Dag metadata from your Deployment. This job runs an Astronomer custom cleanup script for all of your Deployments and exports the results in a CSV-formatted file structure to your configured external storage service.

You can enable this feature by setting the config flag in `astronomer.houston.cleanupAirflowDb.enabled` to `true` in your `values.yaml` file.

There are two ways to use this feature:

* Scheduled Cleanup: You can configure a Kubernetes CronJob to run the cleanup job at regular intervals by defining the schedule and job parameters in the `astronomer.houston.cleanupAirflowDb` section of your `values.yaml` file.
* Manual Cleanup: The APC API GraphQL query, `cleanupAirflowDb`, manually triggers the Airflow metadata cleanup job for immediate execution.

<Danger>The cleanup job deletes any data that's older than the number of days specified in your `olderThan` configuration. Ensure that none of your historical data is required to run current Dags or tasks before enabling this feature.</Danger>

## Prerequisites

* [System admin](/docs/astro-private-cloud/v-2-x/role-permission-reference#system-admin) user privileges
* External storage credentials that allow read/write permissions to your storage
* (AWS Cloud Provider) The [AWS CLI](https://aws.amazon.com/cli/?pg=developertools)

## Step 1: Configure your external storage credentials

<Tabs>
  <Tab title="Google Cloud Storage">
    1. You must provision a [GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) with appropriate read/write permissions to your bucket. Export these credentials as a JSON file.

    2. Create a Kubernetes secret in your Astronomer platform namespace with a name such as `astronomer-gcs-keyfile`. Then, run the following commands to update your environment:

    ```bash wrap theme={null}
    kubectl annotate secret astronomer-gcs-keyfile "astronomer.io/commander-sync"="platform=astronomer"

    kubectl run job --from=cronjob/astronomer-config-syncer runconfigsyncer-job-001
    ```

    You use this Kubernetes secret to configure `providerEnvSecretName` when you configure the cleanup job and `env.name` when you set the storage provider secret.
  </Tab>

  <Tab title="AWS">
    1. Create IAM policy called `s3-policy.json`.

    ```json wrap theme={null}
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject"
                ],
                "Resource": "arn:aws:s3:::my-bucket/*"
            }
        ]
    }
    ```

    2. Run the following commands to create an AWS access and secret key that grants read/write access to the [AWS S3 bucket](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_rw-bucket.html).

    ```bash wrap theme={null}
    aws iam create-user --user-name "s3-user"
    aws iam create-policy --policy-name "s3-user-policy" --policy-document file://s3-policy.json --description "S3 read/write access"
    aws iam create-access-key --user-name "s3-user"
    aws iam attach-user-policy --user-name "s3-user" --policy-arn "arn:aws:iam::your-account-id:policy/s3-user-policy"
    ```

    3. Create a Kubernetes secret in your Astronomer platform namespace with a name such as `aws-secret`. Then, run the following commands to update your environment so that every Deployment you create can use the AWS access credentials:

    ```bash wrap theme={null}
    kubectl create secret generic aws-secret --from-literal aws_access_key=“<aws-access-key-id>" --from-literal as_secret_key=“<aws-secret-access-key>"

    kubectl annotate secret aws-secret "astronomer.io/commander-sync"="platform=astronomer"

    kubectl run job --from=cronjob/astronomer-config-syncer runconfigsyncer-job-001
    ```
  </Tab>
</Tabs>

### (Optional) Configure a connection ID

If you want to run jobs for specific Deployments or within a Workspace or run manually triggered jobs using an API query, you can choose to configure an Airflow connection to your external storage service so that it can be stored as an environment variable. You must use the service account credentials to authenticate to your service when configuring your connection.

<Tabs>
  <Tab title="Google Cloud Storage">
    1. You must provision a [GCP Service Account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) with appropriate read/write permissions to your bucket. Export these credentials as a JSON file.

    2. Create an Airflow connection using these credentials. See [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#storing-connections-in-environment-variables) to learn how to configure your connection.
  </Tab>

  <Tab title="AWS">
    1. Create IAM policy called `s3-policy.json`.

    ```json wrap theme={null}
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject"
                ],
                "Resource": "arn:aws:s3:::my-bucket/*"
            }
        ]
    }
    ```

    Then, run the following commands to create an AWS access and secret key that grants read/write access to the [AWS S3 bucket](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_rw-bucket.html).

    ```bash wrap theme={null}
    aws iam create-user --user-name "s3-user"
    aws iam create-policy --policy-name "s3-user-policy" --policy-document file://s3-policy.json --description "S3 read/write access"
    aws iam create-access-key --user-name "s3-user"
    aws iam attach-user-policy --user-name "s3-user" --policy-arn "arn:aws:iam::your-account-id:policy/s3-user-policy"
    ```

    2. Create an Airflow connection using these credentials to configure the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. See [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#storing-connections-in-environment-variables) to learn how to configure your connection.
  </Tab>
</Tabs>

<Warning>This strategy isn't secure because the secret is in base64 encoded format in your `config.yaml`, which can be decoded.</Warning>

You can use this connection as your `connectionId` when you make API queries as the cleanup job trigger, but it isn't required.

## Step 3: Configure the cleanup job

The cronjob configuration provides the default values that your cleanup job uses whether you run a scheduled or manual cleanup job.

The following example shows the automatic cleanup job configuration that runs at 5:23AM and cleans up Deployments that are more than one year old.

```yaml expandable wrap theme={null}
astronomer:
  houston:
    cleanupAirflowDb:
      # Enable cleanup CronJob
      enabled: true

      # Default run is at 5:23 every morning https://crontab.guru/#23_5_*_*_*
      schedule: "23 5 * * *"

      # Cleanup deployments older than this many days
      olderThan: 365

      # Output path of archived data csv export
      outputPath: "/tmp"

      # Delete archived tables
      dropArchives: true

      # If set true, prints out the deployments that should be cleaned up and skip actual cleanup
      dryRun: false

      # Name of file storage provider, supported providers - gcp/azure/aws/local
      provider: <gcp/azure/aws/local>

      # Name of the provider bucket name / local file path
      bucketName: "/tmp"

      # The name of the Kubernetes Secret containing your cloud provider connection secret
      providerEnvSecretName: "<your-secret-name>"

      # Run cleanup on specific table or list of tables in a comma separated format
      tables: "callback_request,celery_taskmeta,celery_tasksetmeta,dag,dag_run,dataset_event,import_error,job,log,session,sla_miss,task_fail,task_instance,task_reschedule,trigger,xcom"

      # Number of rows to delete in each batch operation (default: 500000)
      batchSize: "500000"

      # Enable db archive data export to external storage.
      # Set to true to export cleaned-up records as CSV to your configured storage provider.
      # Default: false. Customers must explicitly enable this to export cleanup data.
      enableExport: false

      # Display count of records to be cleaned up during dry runs and actual runs (default: false)
      showRecordCount: false

```

## Step 4: Set the storage provider secret

In the APC API config section of your `values.yaml` file, set the storage provider secret that you configured in Step 1, so that the cleanup job can export your cleanup results to your cloud storage.

<Warning>You can configure the task metadata cleanup in different sections of the Helm chart, depending on your scope and use case. However, you can't have `cleanupAirflowDb.enabled: true` enabled at multiple levels. You can only have the job enabled one of the three scope levels.</Warning>

The `env.name` value must match the secret name that you configured for `providerEnvSecretName` in your `values.yaml` file.

### Configure the provider secret in APC

<Tabs>
  <Tab title="Google Cloud Storage">
    ```yaml wrap theme={null}
    astronomer:
      houston:
        cleanupAirflowDb:
          enabled: true
          extraVolumes:
            - name: dbcleanup
              secret:
                defaultMode: 420
                optional: true
                secretName: astronomer-gcs-keyfile
          extraVolumeMounts:
            - mountPath: /tmp/creds/astronomer-gcs-keyfile
              name: dbcleanup
              readOnly: false
              subPath: astronomer-gcs-keyfile
          extraEnv:
            - name: GCP_PASS
              value: /tmp/creds/astronomer-gcs-keyfile
    ```
  </Tab>

  <Tab title="AWS">
    ```yaml wrap theme={null}
    astronomer:
      houston:
        cleanupAirflowDb:
          enabled: true
          extraVolumes:
            - name: dbcleanup
              secret:
                defaultMode: 420
                optional: true
                secretName: aws-secret
          extraVolumeMounts:
            - mountPath: /tmp/creds/aws-secret
              name: dbcleanup
              subPath: aws-secret
          extraEnv:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: aws_access_key
                  name: aws-secret
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: aws_secret_key
                  name: aws-secret
    ```
  </Tab>
</Tabs>

### Configure the storage provider secret in a Deployment

<Tabs>
  <Tab title="Google Cloud Storage">
    ```yaml wrap theme={null}
    astronomer:
      houston:
       deployments:
          cleanupAirflowDb:
            enabled: true
            extraVolumes:
              - name: dbcleanup
                secret:
                  defaultMode: 420
                  optional: true
                  secretName: astronomer-gcs-keyfile
            extraVolumeMounts:
              - mountPath: /tmp/creds/astronomer-gcs-keyfile
                name: dbcleanup
                readOnly: false
                subPath: astronomer-gcs-keyfile
            extraEnv:
              - name: GCP_PASS
                value: /tmp/creds/astronomer-gcs-keyfile
    ```
  </Tab>

  <Tab title="AWS">
    ```yaml wrap theme={null}
    astronomer:
      houston:
       deployments:
          cleanupAirflowDb:
            enabled: true
            extraVolumes:
              - name: dbcleanup
                secret:
                  defaultMode: 420
                  optional: true
                  secretName: aws-secret
            extraVolumeMounts:
              - mountPath: /tmp/creds/aws-secret
                name: dbcleanup
                subPath: aws-secret
            extraEnv:
              - name: AWS_ACCESS_KEY_ID
                valueFrom:
                  secretKeyRef:
                    key: aws_access_key
                    name: aws-secret
              - name: AWS_SECRET_ACCESS_KEY
                valueFrom:
                  secretKeyRef:
                    key: aws_secret_key
                    name: aws-secret
    ```
  </Tab>
</Tabs>

## Step 5: (Optional) Set container CPU and memory limits or requests

You can set limits and requests for CPU and Memory of the cleanup container by adding the following to your `cleanupAirflowDb` configuration. These configurations become the new defaults for your cleanup job if you don't pass any additional configurations in your GraphQL mutation. Additionally, if you don't use the manual trigger and instead use the cleanup cronjob, these resources also become the new default used when scheduling cleanup jobs.

```yaml wrap theme={null}
cleanupAirflowDb:
  resources:
    requests:
      cpu: 200m
      memory: 786Mi
    limits:
      cpu: 500m
      memory: 1536Mi
```

<Tip>You can override these resource definitions, or configure resources if you don't define any, by using `resourceSpec` in an API query. See [Scenario 4: Configure custom Pod Resources](#configure-custom-pod-resources).</Tip>

## Step 6: Apply your configuration

Apply your [platform configuration changes](/docs/astro-private-cloud/v-2-x/apply-platform-config) to enable cleanup jobs and to set your cronjob schedule.

```bash wrap theme={null}

helm upgrade <your-platform-release-name> astronomer/astronomer -f <your-updated-config-yaml-file> -n <your-platform-namespace> --set astronomer.houston.upgradeDeployments.enabled=false

```

<Tip>If you want to upgrade all Deployments while updating your configuration, you can set `astronomer.houston.upgradeDeployments.enabled` to `true`.</Tip>

## Step 7: (Optional) Manually trigger the cleanup job

The following configuration enables you to trigger a cleanup job manually using an APC API query. When you use the cleanup job in this way, the values you include in the query are used instead of the defaults set in the `values.yaml` configuration. This means you must specify the Deployment or Workspace in your query that you want to clean up.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        cleanupAirflowDb:
          enabled: true
```

<Tip>
  **Restrict cleanup to manual-only triggers**

  In Step 3, you set an automatic schedule for your platform to clean up task metadata by setting the `astronomer.houston.cleanupAirflowDb.enabled` configuration to `true`. To enable only triggering cleanup jobs manually, you must instead set `astronomer.houston.cleanupAirflowDb.enabled` to `false`. Manually triggered cleanup jobs require you to use an APC API query and specify the Deployments where you want to archive metadata.
</Tip>

The following examples shows different mutations that you can use depending on your needs. See [APC API examples](/docs/astro-private-cloud/v-2-x/houston-api-example-queries) for all examples and scenarios that you can use to work with the APC API.

### APC API parameters

| Name                    | Type   | Description                                                                                                                                     |
| ----------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `olderThan`             | Int    | Clean up data in Deployments that are older than the number of days defined in this parameter.                                                  |
| `dryRun`                | Bool   | When set to `true`, the job doesn't make changes, it only logs which data would be cleaned up. If unspecified, default is `false`.              |
| `outputPath`            | String | The path in your storage bucket or local storage where the job saves the archived CSV data.                                                     |
| `dropArchives`          | Bool   | If `true`, deletes any previously archived tables after export. Use with caution. Set to `false` by default.                                    |
| `provider`              | String | The cloud provider you use for archiving. Supported values: `aws`, `azure`, `gcp`, `local`. If unspecified, defaults to `local`.                |
| `bucketName`            | String | Name of the cloud storage bucket or local directory where the job saves the archive CSV export.                                                 |
| `providerEnvSecretName` | String | Name of the Kubernetes Secret that contains the credentials or config for the storage provider, if you used a Kubernetes secret.                |
| `deploymentIds`         | String | List of the specific Deployment IDs to target for cleanup.                                                                                      |
| `workspaceId`           | String | Restricts cleanup to Deployments within the configured Workspace.                                                                               |
| `tables`                | String | Comma-separated list of tables to target for cleanup. If you don't configure this parameter, all supported tables will be cleaned.              |
| `batchSize`             | String | Number of rows to delete in each batch operation. Default is `500000`.                                                                          |
| `enableExport`          | Bool   | Set to `true` to export cleaned-up records as CSV to your configured external storage provider. Default is `false`.                             |
| `showRecordCount`       | Bool   | When set to `true`, displays the count of records to be cleaned up during both dry runs and actual runs. Default is `false`.                    |
| `resourceSpec`          | JSON   | (Optional configuration) A JSON object that allows you to define a Pod resource configuration.                                                  |
| `connectionId`          | String | (Optional configuration) Airflow connection ID used for accessing the underlying data warehouse. Can be left empty if no connection is defined. |

<Tip>Set `dryRun: true` to test this feature without deleting any data. When dry runs are enabled, the cleanup job will only print the data that it plans to modify in the serial output of the webserver Pod. To view the dryRun events of the cleanup job, check the logs of your webserver Pod for each Deployment.</Tip>

The following examples show different queries you can use depending on your needs. For the full parameter reference, see [Clean up and delete task metadata](/docs/astro-private-cloud/v-2-x/clean-up-task-metadata#apc-api-parameters).

### Clean up Deployments per Workspace

```graphql wrap theme={null}
query cleanupAirflowDb(
  $olderThan: Int!
  $dryRun: Boolean!
  $outputPath: String!
  $dropArchives: Boolean!
  $provider: String!
  $bucketName: String!
  $providerEnvSecretName: String!
  $deploymentIds: [Id]
  $workspaceId: Uuid
  $tables: String!
  $connectionId: String
) {
  cleanupAirflowDb(
    olderThan: $olderThan
    dryRun: $dryRun
    outputPath: $outputPath
    dropArchives: $dropArchives
    provider: $provider
    bucketName: $bucketName
    providerEnvSecretName: $providerEnvSecretName
    workspaceId: $workspaceId
    tables: $tables
    connectionId: $connectionId
  )
}
```

Query variables to clean up all Deployments older than 1 day within a Workspace that uses GCP as a cloud provider:

```graphql wrap theme={null}
{
	"olderThan": 1,
	"dryRun": true,
	"outputPath": "",
	"dropArchives": true,
	"provider":  "gcp",
	"bucketName" : "",
	"connectionId":  "",
	"tables": "callback_request,celery_taskmeta,celery_tasksetmeta,dag,dag_run,dataset_event,import_error,job,log,session,sla_miss,task_fail,task_instance,task_reschedule,trigger,xcom",
	"providerEnvSecretName": "GCP_PASS",
	"workspaceId": "cma40n66l000008l89nye86o1"
}
```

### Clean up specific Deployments

Query variables to clean up specific Deployments older than 1 day within a Workspace:

```graphql wrap theme={null}
{
	"olderThan": 1,
	"dryRun": true,
	"outputPath": "",
	"dropArchives": true,
	"provider":  "gcp",
	"bucketName" : "",
	"connectionId":  "",
	"tables": "callback_request,celery_taskmeta,celery_tasksetmeta,dag,dag_run,dataset_event,import_error,job,log,session,sla_miss,task_fail,task_instance,task_reschedule,trigger,xcom",
	"providerEnvSecretName": "GCP_PASS",
	"deploymentIds": ["cma42zc67000108l89eb37iy5","cma42zjdp000208l8g16ygm6m"],
	"workspaceId": "cma42z570000008l8f6rpc72f"
}
```

### Clean up using an Airflow connection ID

<Warning>Requires configuring an Airflow Connection ID, `connectionId`, from the Airflow UI or CLI.</Warning>

Query variables to clean up Deployments and export the cleanup logs to the storage provider configured in an [Airflow Connection](/docs/learn/connections):

```graphql wrap theme={null}
{
	"olderThan": 1,
	"dryRun": true,
	"outputPath": "",
	"dropArchives": true,
	"provider":  "gcp",
	"bucketName" : "",
	"connectionId":  "<airflow_connection_id>",
	"tables": "callback_request,celery_taskmeta,celery_tasksetmeta,dag,dag_run,dataset_event,import_error,job,log,session,sla_miss,task_fail,task_instance,task_reschedule,trigger,xcom",
	"deploymentIds": ["cm6q3jpn61741517mhonzgcgz7","cm6q3jpn61741517mhonzgcgz7"],
	"workspaceId": "cm5nj9wly007617iox80beute"
}
```

### Configure custom Pod resources

If you don't configure a default Pod CPU or memory resource amount, or want to override one, make a query that sets `resourceSpec`:

```graphql wrap theme={null}
query cleanupAirflowDb(
    $olderThan: Int!
    $dryRun: Boolean!
    $outputPath: String!
    $dropArchives: Boolean!
    $provider: String!
    $bucketName: String!
    $providerEnvSecretName: String!
    $tables: String!
    $resourceSpec: JSON
  ) {
    cleanupAirflowDb(
      olderThan: $olderThan
      dryRun: $dryRun
      outputPath: $outputPath
      dropArchives: $dropArchives
      provider: $provider
      bucketName: $bucketName
      providerEnvSecretName: $providerEnvSecretName
      tables: $tables
      resourceSpec: $resourceSpec
    )
  }
```

Query variables that configure resource requests and limits for the cleanup run:

```graphql wrap theme={null}
{
  "resourceSpec": {
    "requests": {
      "cpu": "100m",
      "memory": "5000Mi"
    },
  "limits": {
      "cpu": "100m",
      "memory": "5000Mi"
    }
  },
  "olderThan": 1,
  "dryRun": false,
  "outputPath": "/abc",
  "dropArchives": false,
  "provider": "aws",
  "bucketName": "test",
  "providerEnvSecretName": "test-secret",
  "tables": "dag"
}
```

You can also find Workspace IDs with the [`sysWorkspaces` APC API query](/docs/astro-private-cloud/v-2-x/houston-api-example-queries#sysWorkspaces).

## Access your cleanup logs

You can access your cleanup logs through the UI or with your Pod logs.

### Pod logs

You can access your Pod logs with vector sidecar logging or Fluentd with `<release-name>-meta-cleanup-job` in the Airflow namespace.

### UI access

Go to the **Logs** tab in your **Deployments** page and select the **AirflowMetaCleanup** tab to access the logs. Sidecar logging and DaemonSet logging are both supported.


# Configure cleanup jobs
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/cleanup-cronjobs

Configure automated cleanup jobs for database maintenance in Astro Private Cloud.

Configure automated cleanup jobs to maintain database health by removing old data. Astro Private Cloud (APC) includes several cleanup jobs that run as CronJobs on configurable schedules to manage storage growth and query performance.

## Cleanup jobs summary

| Job                      | Default Schedule | Default Retention | Purpose                                      |
| ------------------------ | ---------------- | ----------------- | -------------------------------------------- |
| `cleanupDeployments`     | Daily @ 00:00    | 14 days           | Removes soft-deleted deployments             |
| `cleanupDeployRevisions` | Daily @ 23:11    | 90 days           | Archive deploy history                       |
| `cleanupTaskUsageData`   | Daily @ 23:40    | 90 days           | Purge task metrics                           |
| `cleanupClusterAudits`   | Daily @ 23:49    | 90 days           | Remove cluster audit logs                    |
| `cleanupAirflowDb`       | Daily @ 05:23    | 365 days          | Clean Airflow metadata (disabled by default) |

## `cleanupDeployments`

Permanently removes deployments that have been soft-deleted after the retention period.

### What gets cleaned

* Deployment database records marked with `deletedAt`
* Associated Docker registry images
* Deployment metadata database

### Configuration

```yaml wrap theme={null}
houston:
  cleanupDeployments:
    enabled: true
    schedule: "0 0 * * *"      # Midnight daily
    olderThan: 14              # Days since deletion
    dryRun: false              # Set true to preview
```

### Manual trigger

Run this command from a machine with access to the underlying Kubernetes cluster:

```bash wrap theme={null}
kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-deployments --older-than=14 --dry-run=false
```

## `cleanupDeployRevisions`

Removes old deployment revision records to reduce database size.

### What gets cleaned

* `deployRevision` records older than retention period
* Historical deployment configuration snapshots

### Configuration

```yaml wrap theme={null}
houston:
  cleanupDeployRevisions:
    enabled: true
    schedule: "11 23 * * *"    # 23:11 daily
    olderThan: 90              # Days to retain
```

### Manual trigger

Run this command from a machine with access to the underlying Kubernetes cluster:

```bash wrap theme={null}
kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-deploy-revisions --older-than=90
```

### Per-deployment cleanup

Run this command from a machine with access to the underlying Kubernetes cluster to clean revisions for a specific deployment:

```bash wrap theme={null}
kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-deploy-revisions --older-than=90 --deploymentUuid=<uuid>
```

## `cleanupTaskUsageData`

Purges task usage metrics and audit logs.

### What gets cleaned

* `TaskUsage` records (daily aggregated metrics)
* `TaskUsageAuditLog` records (raw task data)

### Configuration

```yaml wrap theme={null}
houston:
  cleanupTaskUsageData:
    enabled: true
    schedule: "40 23 * * *"    # 23:40 daily
    olderThan: 90              # Minimum 90 days
    dryRun: false
```

### Manual trigger

Run this command from a machine with access to the underlying Kubernetes cluster:

```bash wrap theme={null}
kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-task-usage-data --older-than=90 --dry-run=false
```

### GraphQL trigger

Use the `cleanupTaskUsageDataJob` query to manually trigger a purge of task usage metrics and audit logs:

```graphql wrap theme={null}
query {
  cleanupTaskUsageDataJob(olderThan: 90)
}
```

<Note>
  Minimum retention is 90 days and can't be reduced.
</Note>

## `cleanupClusterAudits`

Removes cluster audit log entries.

### What gets cleaned

* `ClusterAudit` records tracking cluster configuration changes
* Historical cluster state snapshots

### Configuration

```yaml wrap theme={null}
houston:
  cleanupClusterAudits:
    enabled: true
    schedule: "49 23 * * *"    # 23:49 daily
    olderThan: 90              # Days to retain
```

### Manual trigger

Run this command from a machine with access to the underlying Kubernetes cluster:

```bash wrap theme={null}
kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-cluster-audit --older-than=90
```

### Filter by cluster

Run this command from a machine with access to the underlying Kubernetes cluster to clean audits for specific clusters:

```bash wrap theme={null}
kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-cluster-audit --older-than=90 --cluster-ids=<id1>,<id2>
```

## `cleanupAirflowDb`

Cleans Airflow metadata from individual Deployment databases.

<Warning>
  This job is disabled by default due to potential impact on running Deployments.
</Warning>

### What gets cleaned

Default tables:

* `callback_request` - Task callback requests
* `celery_taskmeta`, `celery_tasksetmeta` - Celery metadata
* `dag` - Dag definitions
* `dag_run` - Dag execution history
* `dataset_event` - Dataset events
* `import_error` - Import errors
* `job` - Job records
* `log` - Task execution logs
* `session` - Session data
* `sla_miss` - SLA violations
* `task_fail` - Task failures
* `task_instance` - Task execution records
* `task_reschedule` - Reschedule events
* `trigger` - Trigger records
* `xcom` - Cross-communication data

### Configuration

```yaml wrap theme={null}
houston:
  cleanupAirflowDb:
    enabled: false             # Must explicitly enable
    schedule: "23 5 * * *"     # 05:23 daily
    olderThan: 365             # Days to retain
    outputPath: "/tmp"         # Archive location
    dropArchives: true         # Delete after archiving
    dryRun: false
    provider: local            # Storage: local/aws/azure/gcp
    bucketName: "/tmp"         # Cloud bucket or local path
    tables: ""                 # Specific tables (empty = all)
```

### Cloud storage export

Export archived data to cloud storage:

```yaml wrap theme={null}
houston:
  cleanupAirflowDb:
    enabled: true
    provider: aws              # aws, azure, or gcp
    bucketName: "my-archive-bucket"
    providerEnvSecretName: "aws-credentials-secret"
```

### Specific tables only

Clean only specific tables:

```yaml wrap theme={null}
houston:
  cleanupAirflowDb:
    enabled: true
    tables: "log,task_instance,xcom"
```

### Manual trigger

Run this command from a machine with access to the underlying Kubernetes cluster:

```bash wrap theme={null}
kubectl -n <namespace> exec -it deploy/<release-name>-houston -- yarn cleanup-airflow-db-data \
  --older-than=365 \
  --provider=local \
  --bucket-name=/tmp \
  --tables="log,task_instance"
```

## Schedule reference

Default schedules are staggered to avoid simultaneous execution:

| Time  | Job                      |
| ----- | ------------------------ |
| 00:00 | `cleanupDeployments`     |
| 05:23 | `cleanupAirflowDb`       |
| 23:11 | `cleanupDeployRevisions` |
| 23:40 | `cleanupTaskUsageData`   |
| 23:49 | `cleanupClusterAudits`   |

## Common configuration options

All cleanup jobs share these options:

```yaml wrap theme={null}
houston:
  cleanup<JobName>:
    enabled: true/false        # Enable/disable the job
    schedule: "cron-expression"  # When to run
    olderThan: <days>          # Retention period
    dryRun: false              # Preview without deleting
    readinessProbe: {}         # Optional health probes
    livenessProbe: {}
```

## Kubernetes CronJob behavior

All cleanup CronJobs use:

* **Concurrency policy**: `Forbid` (prevents overlapping runs)
* **Backoff limit**: 1 retry on failure
* **Restart policy**: Never

## Monitor cleanup jobs

### Check job status

```bash wrap theme={null}
# List all cleanup CronJobs
kubectl get cronjobs -n astronomer | grep cleanup

# View recent job runs
kubectl get jobs -n astronomer | grep cleanup

# Check job logs
kubectl logs job/<job-name> -n astronomer

# Trigger individual jobs manually
kubectl create job --from=cronjobs/jobname jobname-hash -n astronomer
```

### Verify data cleanup

```sql wrap theme={null}
-- Check remaining records by date
SELECT DATE(created_at), COUNT(*)
FROM deploy_revision
GROUP BY DATE(created_at)
ORDER BY DATE(created_at) DESC;
```

## Troubleshooting

### Job not running

1. Check CronJob exists:

   ```bash wrap theme={null}
   kubectl get cronjob houston-cleanup-deployments -n astronomer
   ```

2. Check job is enabled in Helm values.

3. Verify schedule syntax is valid cron expression.

### Job failing

1. Check job logs:

   ```bash wrap theme={null}
   kubectl logs job/houston-cleanup-deployments-<timestamp> -n astronomer
   ```

2. Database connectivity: Ensure the APC API can reach the database.

3. Permissions: Verify service account has required database permissions.

### Data not being cleaned

1. Check retention period: Data younger than `olderThan` won't be deleted.
2. Verify timestamps: Check `createdAt`/`deletedAt` values in database.
3. Run with dry-run: Preview what would be deleted.

## Best practices

1. Monitor database size before and after cleanup jobs
2. Start with dry-run when adjusting retention periods
3. Stagger schedules if adding custom cleanup jobs
4. Archive before delete for cleanupAirflowDb in production
5. Set alerts for failed cleanup jobs

## Related documentation

* [Apply platform configuration](/docs/astro-private-cloud/v-2-x/apply-platform-config)


# Manage cluster status
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/cluster-status-management

Monitor and manage data plane cluster status in Astro Private Cloud.

Astro Private Cloud (APC) tracks the operational status of every data plane cluster so that workloads only run on healthy infrastructure. This page describes the cluster status values, how the APC API determines status, the GraphQL operations for querying and updating status, and how to troubleshoot unhealthy clusters.

## Authenticate to the APC API

Every operation on this page requires an APC API token sent as a bearer credential. Send your token in the `Authorization` header on each request to the APC GraphQL endpoint:

```bash wrap theme={null}
curl -X POST https://houston.<your-base-domain>/v1 \
  -H "Authorization: <your-token>" \
  -H "Content-Type: application/json" \
  -d '{"query": "query { self { user { username } } }"}'
```

For step-by-step instructions on obtaining a user token or creating a system service account token, see [Authenticate to the APC API](/docs/astro-private-cloud/v-2-x/houston-api-authenticate).

## Required roles and permissions

Cluster operations are gated by RBAC permissions. The following table maps each operation to the permission APC API checks and the default role that grants it.

| Operation                     | Required permission      | Default role that grants access |
| ----------------------------- | ------------------------ | ------------------------------- |
| `paginatedClusters`           | Authenticated user       | Any signed-in user              |
| `cluster`                     | `system.clusters.get`    | System Admin                    |
| `updateCluster`               | `system.clusters.update` | System Admin                    |
| `reconcileClusterMetadataJob` | `system.clusters.update` | System Admin                    |

The System Admin role inherits every `system.clusters.*` permission.

## Cluster status values

| Status     | Description                                             | Allows new deployments | Allows configuration updates |
| ---------- | ------------------------------------------------------- | ---------------------- | ---------------------------- |
| `ACTIVE`   | Cluster is healthy and reachable                        | Yes                    | Yes                          |
| `INACTIVE` | Cluster is unreachable or reporting an unhealthy status | No                     | No                           |

## Status determination

The APC API derives cluster status from the `healthStatus` field in the deployment orchestrator's `/metadata` response. The mapping is binary:

| Deployment orchestrator `healthStatus` | APC API cluster status |
| -------------------------------------- | ---------------------- |
| `HEALTHY`                              | `ACTIVE`               |
| Any other value                        | `INACTIVE`             |
| Fetch error or timeout                 | `INACTIVE`             |

A CronJob in the control plane reconciles cluster metadata by calling the deployment orchestrator's `/metadata` endpoint. The default schedule is `0 * * * *` (every hour at minute 0), and is configurable through the `houston.syncDataplaneClusters.schedule` value on the Astronomer Helm chart.

```mermaid actions={true} wrap theme={null}
sequenceDiagram
    participant H as the APC API (control plane)
    participant C as deployment orchestrator
    H->>C: GET /metadata
    C-->>H: { healthStatus: "HEALTHY", ... }
    Note over H: Map healthStatus → cluster status<br/>HEALTHY → ACTIVE, else → INACTIVE
```

You can list the reconcile CronJob and recent runs with the following command:

```bash wrap theme={null}
kubectl get cronjob,jobs -n astronomer | grep sync-dataplane-clusters
```

## Query cluster status

### List clusters

The `paginatedClusters` query returns clusters the caller has access to. Pagination uses the `take` argument, plus either `cursor` (a cluster UUID) or `pageNumber`. The response object contains a `clusters` list and a total `count`.

```graphql wrap theme={null}
query {
  paginatedClusters(
    take: 50
    status: ACTIVE
  ) {
    clusters {
      id
      name
      status
      statusReason
      healthStatus
      k8sVersion
      cloudProvider
      region
      createdAt
      updatedAt
    }
    count
  }
}
```

### Get a single cluster

```graphql wrap theme={null}
query {
  cluster(id: "<cluster-id>") {
    id
    name
    status
    statusReason
    healthStatus
    k8sVersion
    cloudProvider
    region
    dpChartVersion
    commanderVersion
    config
    configOverride
  }
}
```

<Note>
  The `healthStatus` field returns a JSON object containing the full health payload the APC API received from the deployment orchestrator, not a single string. The `statusReason` field is also a JSON object.
</Note>

### Filter by cloud provider and region

```graphql wrap theme={null}
query {
  paginatedClusters(
    status: INACTIVE
    cloudProvider: "aws"
    region: "us-east-1"
    take: 25
  ) {
    clusters {
      id
      name
      statusReason
    }
    count
  }
}
```

Other supported filter arguments include `searchPhrase`, `k8sVersion`, `id`, `sortBy`, and `sortDirection`.

See [Update cluster status](#update-cluster-status) for the shape the APC API writes to `statusReason`.

## Update cluster status

A user with permission to update clusters can change a cluster's status manually. The `statusReason` argument accepts a JSON object whose shape isn't enforced by the schema, but the APC API itself writes the value the deployment orchestrator returns in its `/metadata` response when reconciling. To stay consistent, use the same shape the APC API uses or include a descriptive `message` field.

```graphql wrap theme={null}
mutation {
  updateCluster(
    id: "<cluster-id>"
    status: INACTIVE
    statusReason: { message: "Maintenance window — cluster offline for upgrades" }
  ) {
    id
    status
    statusReason
  }
}
```

For status changes, supply `id` (required), `status`, and `statusReason`. The `updateCluster` mutation also accepts `name` and `deploymentsConfigOverride` for non-status changes; see [Update data plane cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster) for those workflows.

<Note>
  The APC API blocks configuration updates (`deploymentsConfigOverride`, `name`) while the cluster status is `INACTIVE` and returns the error `This operation is not allowed as the cluster is not active.` Status itself can still be updated in any state.
</Note>

To manually restore a cluster to `ACTIVE` after confirming it's healthy:

```graphql wrap theme={null}
mutation {
  updateCluster(
    id: "<cluster-id>"
    status: ACTIVE
    statusReason: { message: "Manually verified healthy" }
  ) {
    id
    status
  }
}
```

## Force a metadata reconciliation

Use the `reconcileClusterMetadataJob` query to make the APC API refetch metadata from the deployment orchestrator immediately, instead of waiting for the next CronJob run. The query accepts a list of cluster UUIDs; if you pass `null` or omit the argument, the APC API reconciles every cluster the caller is authorized to update.

```graphql wrap theme={null}
query {
  reconcileClusterMetadataJob(
    clusterIds: ["<cluster-id-1>", "<cluster-id-2>"]
  ) {
    successfulClusterIds
    failedClusterIds
    skippedClusterIds
  }
}
```

A cluster appears in `skippedClusterIds` when it lacks a data plane URL or when the caller isn't authorized to reconcile it.

Use this query in the following situations:

* After resolving a network or DNS issue between the control plane and a data plane.
* After restarting the deployment orchestrator.
* To verify cluster health after a maintenance window.
* When debugging connectivity from the control plane.

## Troubleshoot unhealthy clusters

<Steps>
  <Step title="Check the cluster's current status">
    ```graphql wrap theme={null}
    query {
      cluster(id: "<cluster-id>") {
        status
        statusReason
        healthStatus
        updatedAt
      }
    }
    ```
  </Step>

  <Step title="Verify the deployment orchestrator connectivity">
    From a Pod in the control plane namespace with network access to the deployment orchestrator, call the metadata endpoint:

    ```bash wrap theme={null}
    curl -s https://<commander-url>/metadata | jq .
    ```

    A healthy response includes (among other fields) the following:

    ```json wrap theme={null}
    {
      "kubernetesVersion": "<k8s-version>",
      "baseDomain": "<cluster-base-domain>",
      "healthStatus": "HEALTHY",
      "cloudProvider": "<provider>",
      "region": "<region>",
      "dataplaneChartVersion": "<chart-version>",
      "commander": {
        "version": "<commander-version>",
        "url": "<commander-grpc-url>",
        "status": "HEALTHY",
        "airflowChartVersion": "<airflow-chart-version>"
      }
    }
    ```

    The full response also includes `mode`, `dataplaneUrl`, `dataplaneId`, `releaseName`, `releaseNamespace`, `dbType`, `namespacePools`, and `registry`.
  </Step>

  <Step title="Check the deployment orchestrator health and pods">
    ```bash wrap theme={null}
    curl -s https://<commander-url>/healthz
    ```

    ```bash wrap theme={null}
    kubectl get pods -n astronomer -l app=commander
    ```
  </Step>

  <Step title="Force a metadata refresh">
    ```graphql wrap theme={null}
    query {
      reconcileClusterMetadataJob(clusterIds: ["<cluster-id>"]) {
        successfulClusterIds
        failedClusterIds
        skippedClusterIds
      }
    }
    ```
  </Step>

  <Step title="Review deployment orchestrator logs">
    Replace `<release-name>` with your Helm release name, which is `astronomer` by default:

    ```bash wrap theme={null}
    kubectl logs -n astronomer deployment/<release-name>-commander --tail=100
    ```
  </Step>
</Steps>

## Common issues and resolutions

### Cluster stuck in `INACTIVE`

Possible causes:

1. The deployment orchestrator Pod isn't running.
2. Network connectivity between APC API and the deployment orchestrator is broken (firewall, DNS, service mesh).
3. TLS certificate problems on the metadata endpoint.
4. The deployment orchestrator's `/metadata` endpoint returns a non-2xx response or a payload without `healthStatus: "HEALTHY"`.

Resolution steps:

```bash wrap theme={null}
kubectl get pods -n astronomer -l app=commander
```

```bash wrap theme={null}
kubectl describe pod <commander-pod> -n astronomer
```

```bash wrap theme={null}
kubectl logs -n astronomer deployment/<release-name>-commander
```

Test connectivity from APC API (replace `<release-name>` with your Helm release, default `astronomer`):

```bash wrap theme={null}
kubectl exec -it deployment/<release-name>-houston -n astronomer -- \
  curl -v https://<commander-url>/metadata
```

After the underlying issue is resolved, force a reconciliation through the `reconcileClusterMetadataJob` query.

### Configuration updates rejected

APC API returns this error when a configuration update is attempted on a non-`ACTIVE` cluster:

```text wrap theme={null}
This operation is not allowed as the cluster is not active.
```

Resolution:

1. Confirm the cluster is reachable and run `reconcileClusterMetadataJob` to refresh status.
2. If the cluster reports healthy but APC API hasn't yet reconciled, wait for the next reconcile cycle or trigger one manually.
3. As a last resort, a System Admin can manually set the cluster status back to `ACTIVE`:

```graphql wrap theme={null}
mutation {
  updateCluster(
    id: "<cluster-id>"
    status: ACTIVE
    statusReason: { message: "Manually verified healthy" }
  ) {
    id
    status
  }
}
```

## Best practices

* Monitor cluster status proactively. Configure alerts for clusters transitioning to `INACTIVE` and surface status on operations dashboards.
* Always provide a meaningful `statusReason` when manually changing status. The reason is preserved in the cluster record and is useful when diagnosing later incidents.
* Distribute Deployments across multiple clusters so that a single `INACTIVE` cluster doesn't affect every workload.
* Validate connectivity from the control plane Pod after firewall, DNS, or certificate changes; don't rely on the next scheduled reconciliation to surface the problem.

## Related documentation

* [Authenticate to the APC API](/docs/astro-private-cloud/v-2-x/houston-api-authenticate)
* [Use the APC API on Astro Private Cloud](/docs/astro-private-cloud/v-2-x/houston-api)
* [Register a data plane](/docs/astro-private-cloud/v-2-x/register-data-plane)
* [Deregister a data plane cluster](/docs/astro-private-cloud/v-2-x/deregister-data-plane)
* [Update data plane cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster)
* [Data plane architecture](/docs/astro-private-cloud/v-2-x/data-plane-architecture)


# Config governance
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/config-governance

How configuration is owned and changed—platform-wide settings via Helm, plus a four-tier layered merge for keys under deployments.

Astro Private Cloud (APC) 2.0 uses *config governance* to describe how platform operators and, where allowed, workspace and deployment admins control and change platform and deployment-level settings. If you are choosing which layer to edit first (Helm, cluster, workspace, or deployment), start with the shorter guide, [Configure Astro Private Cloud](/docs/astro-private-cloud/v-2-x/configure-astro-private-cloud), then return here for rules, blocklists, and API detail. For tabular defaults and allowed values under `astronomer.houston.config.deployments`, see [Helm configuration reference](/docs/astro-private-cloud/v-2-x/helm-config-reference#apc-api-configuration).

Config governance in Astro Private Cloud enables settings under the deployments section of the Helm values file to be overridden at the Cluster, Workspace, or Deployment level, allowing for fine-grained control over the configuration of each Apache Airflow Deployment.

Config governance uses a layered deep merge of deployments (cluster, workspace, and deployment layers on top of the platform default), so effective values for those keys resolve predictably. Settings outside the deployments section (such as the astronomer, global, webserver, and nats sections) can only be changed in the Helm values files, and not in the three deployment override layers.

## Overview

For settings under the `deployments` section, config governance resolves values by merging four tiers, where each subsequent tier overrides the previous for keys it sets:

```text wrap theme={null}
Platform (Helm / values.yaml default) → Cluster → Workspace → Deployment
```

| Tier       | Scope                                            | Who sets it                     | How it's set                                                                                                    |
| ---------- | ------------------------------------------------ | ------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Platform   | All Deployments across all clusters              | Platform operator               | `values.yaml` and a Helm upgrade                                                                                |
| Cluster    | All Deployments in a specific data plane cluster | System Admin                    | Astro UI or GraphQL API ([Override data plane cluster](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster)) |
| Workspace  | All Deployments in a specific workspace          | Workspace Admin or System Admin | Astro UI or GraphQL API                                                                                         |
| Deployment | A single Deployment                              | Workspace Admin or System Admin | Astro UI or GraphQL API                                                                                         |

When APC resolves a configuration value for a Deployment, it starts with the platform defaults, merges in any cluster-level overrides, then workspace-level overrides, and finally deployment-level overrides. Later tiers always win for any key they set.

<Note>
  In the Astro UI, configuration overrides at the workspace or deployment level show the override you saved for that scope, not a full preview of the effective merged deployments result. The platform computes the final values when a Deployment is created or updated.
</Note>

## What can be configured

Config governance applies to all settings under the `deployments.*` part of the platform config in your `values.yaml` (under `astronomer.houston.config`). This includes:

| Configuration domain           | Example settings                                                    |
| ------------------------------ | ------------------------------------------------------------------- |
| Runtime management             | Airflow 3 enablement, custom image SHA, minimum runtime versions    |
| Dag deployment mechanisms      | Dag-only deployment, git-sync, NFS mount                            |
| Airflow components             | Triggerer, Dag processor enablement                                 |
| Deployment lifecycle           | Deploy rollback, hard delete, Airflow DB cleanup                    |
| Resource management            | Executor configuration, component resources, max pod/extra capacity |
| Database management            | Manual connection strings, PgBouncer strategy                       |
| Metrics and reporting          | Grafana UI, task usage metrics                                      |
| Deployment images and registry | Docker webhook endpoint, update deployment image endpoint           |

<Note>
  The four override tiers in this model (platform default, then cluster, workspace, and deployment) apply only to configuration under the `deployments` section. Keys elsewhere in the platform configuration, for example `global`, `webserver`, and `nats`, aren't changed through those tiers; the platform operator updates them in the values file and with a Helm upgrade.
</Note>

## Blocklisted keys

Certain configuration keys can't be overridden at the workspace or deployment level because they are tightly coupled to platform or cluster infrastructure.

### Workspace-level blocklist

The following keys can't be set in workspace-level overrides:

| Blocklisted key | Reason                                                                        |
| --------------- | ----------------------------------------------------------------------------- |
| `authSideCar`   | Platform/cluster-level auth proxy; per-workspace override would break routing |
| `logging`       | Includes Vector sidecar configuration; platform-level dependency              |

### Deployment-level blocklist

The following keys can't be set in deployment-level overrides (superset of workspace blocklist):

| Blocklisted key       | Reason                                                         |
| --------------------- | -------------------------------------------------------------- |
| `authSideCar`         | Platform/cluster-level auth proxy                              |
| `logging`             | Platform-level Vector sidecar dependency                       |
| `namespaceManagement` | Namespace configuration is immutable after Deployment creation |

If you attempt to set a blocklisted key, the API returns a `BlocklistedDeploymentConfigKeysError` with the offending key names.

## `DELETE_KEY` and strict schema validation

The string `"DELETE_KEY"` is a merge sentinel for cluster, workspace, and deployment `deploymentsConfigOverride` objects. When you set a key’s value to `"DELETE_KEY"`, the control plane removes that key from the stored override (or a whole subtree, if you set a parent to `"DELETE_KEY"`), so the effective value falls back to the next tier. Use it with the `updateCluster`, `updateWorkspaceDeploymentsConfig`, and `updateDeploymentConfig` GraphQL mutations. See [Override data plane cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster) for the cluster case. You can't use `"DELETE_KEY"` to remove keys that are required in the platform’s default `deployments` object.

The `astronomer.houston.strictSchemaCheck.enabled` value in your platform values file controls whether deployment override payloads are validated against a JSON schema. When this is `true` (the default in the 2.0 [Astronomer Helm chart](https://github.com/astronomer/astronomer)), unknown top-level domain keys and invalid types are rejected. When `false`, that validation is skipped, which is useful in edge cases such as [Git-Sync relay metrics](/docs/astro-private-cloud/v-2-x/git-sync-relay-metrics) where you need `helm` keys the schema doesn't list yet. After you change the flag, apply it with a Helm upgrade of the control plane.

## Cluster-level configuration

Cluster-level overrides apply to all Deployments within a specific data plane cluster. This is useful for setting cluster-specific defaults like resource limits, executor configurations, or feature flags that should apply uniformly to all Deployments in a given cluster.

For instructions on configuring cluster-level overrides, see [Override data plane cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster).

## Workspace-level configuration

Workspace-level overrides apply to all Deployments within a specific workspace. This allows Workspace Admins to customize settings for their team without affecting other Workspaces.

### Add or update workspace configuration

<Frame>
  <img alt="Workspace Settings, Configuration Overrides tab, with the Workspace Deployments Configuration YAML editor open for add configuration." />
</Frame>

You can set workspace configuration overrides using the Astro UI or the APC API.

<Tabs>
  <Tab title="APC API">
    Use the `updateWorkspaceDeploymentsConfig` mutation to add or update workspace-level overrides:

    ```graphql wrap theme={null}
    mutation {
      updateWorkspaceDeploymentsConfig(
        workspaceUuid: "<workspace-id>"
        deploymentsConfigOverride: {
          deploymentLifecycle: {
            deployRollback: {
              enabled: true
            }
          }
        }
        reason: "Enable deploy rollback for all deployments in this workspace"
      ) {
        id
        config
      }
    }
    ```

    The `deploymentsConfigOverride` argument accepts a partial JSON object. Keys you provide are merged into the existing workspace override. Keys you omit are left unchanged. To remove a specific key from the stored override, set its value to the string `"DELETE_KEY"`:

    ```graphql wrap theme={null}
    mutation {
      updateWorkspaceDeploymentsConfig(
        workspaceUuid: "<workspace-id>"
        deploymentsConfigOverride: {
          deploymentLifecycle: {
            deployRollback: {
              enabled: "DELETE_KEY"
            }
          }
        }
        reason: "Remove deploy rollback override, revert to cluster/platform default"
      ) {
        id
        config
      }
    }
    ```
  </Tab>

  <Tab title="Astro UI">
    <Frame>
      <img alt="Workspace Settings, Configuration Overrides tab, with the Workspace Deployments Configuration YAML editor open when removing a specific key from the override." />
    </Frame>

    <Steps>
      <Step title="Select the workspace">
        In the Astro UI, navigate to the workspace where you want to configure overrides.
      </Step>

      <Step title="Open Deployment Configuration">
        Open the **Deployment Configuration** section.
      </Step>

      <Step title="Open the configuration editor">
        Click **Edit**.
      </Step>

      <Step title="Edit configuration keys">
        Add, modify, or remove the configuration keys you need.
      </Step>

      <Step title="Save changes">
        Click **Update**.
      </Step>
    </Steps>
  </Tab>
</Tabs>

### Delete workspace configuration

Deleting workspace configuration removes all workspace-level overrides, reverting all Deployments in the workspace to use cluster-level and platform-level defaults.

<Tabs>
  <Tab title="APC API">
    Use the `deleteWorkspaceDeploymentsConfig` mutation:

    ```graphql wrap theme={null}
    mutation {
      deleteWorkspaceDeploymentsConfig(
        workspaceUuid: "<workspace-id>"
        reason: "Revert workspace to cluster defaults"
      ) {
        id
        config
        deletedAt
      }
    }
    ```
  </Tab>

  <Tab title="Astro UI">
    <Frame>
      <img alt="Workspace Settings, Configuration Overrides, showing the action to remove all workspace-level deployment overrides." />
    </Frame>

    <Steps>
      <Step title="Open workspace configuration">
        In the Astro UI, navigate to the workspace where you want to remove overrides, then open the **Deployment Configuration** section.
      </Step>

      <Step title="Delete workspace overrides">
        Click **Delete** to remove all workspace-level overrides.
      </Step>

      <Step title="Confirm deletion" />
    </Steps>
  </Tab>
</Tabs>

## Deployment-level configuration

Deployment-level overrides apply to a single Deployment. This is the most granular tier and takes the highest precedence. Use this to customize a specific Deployment's behavior without affecting other Deployments in the same workspace.

### Add or update deployment configuration

<Frame>
  <img alt="Deployment, Configuration tab, Configuration Override section open in the add / editor view." />
</Frame>

You can set deployment configuration overrides using the Astro UI or the APC API.

<Tabs>
  <Tab title="APC API">
    Use the `updateDeploymentConfig` mutation to add or update deployment-level overrides:

    ```graphql wrap theme={null}
    mutation {
      updateDeploymentConfig(
        deploymentUuid: "<deployment-id>"
        deploymentsConfigOverride: {
          airflowComponents: {
            triggerer: {
              enabled: false
            }
          }
        }
        reason: "Disable triggerer for this specific deployment"
      ) {
        id
        config
      }
    }
    ```

    The same `DELETE_KEY` mechanism applies — set a key's value to `"DELETE_KEY"` to remove it from the stored override:

    ```graphql wrap theme={null}
    mutation {
      updateDeploymentConfig(
        deploymentUuid: "<deployment-id>"
        deploymentsConfigOverride: {
          airflowComponents: {
            triggerer: {
              enabled: "DELETE_KEY"
            }
          }
        }
        reason: "Remove triggerer override, revert to workspace/cluster default"
      ) {
        id
        config
      }
    }
    ```
  </Tab>

  <Tab title="Astro UI">
    <Frame>
      <img alt="Deployment, Configuration tab, Configuration Override editor when removing a key from the override." />
    </Frame>

    <Steps>
      <Step title="Select the Deployment">
        In the Astro UI, navigate to the Deployment you want to configure.
      </Step>

      <Step title="Open the Configuration tab">
        Open the **Configuration** tab.
      </Step>

      <Step title="Open the configuration editor">
        Click **Edit** in the **Configuration Override** section.
      </Step>

      <Step title="Edit configuration keys">
        Add, modify, or remove the configuration keys you need.
      </Step>

      <Step title="Save changes">
        Click **Update**.
      </Step>
    </Steps>
  </Tab>
</Tabs>

### Delete deployment configuration

Deleting deployment configuration removes all deployment-level overrides, reverting the Deployment to use workspace-level, cluster-level, and platform-level defaults.

<Tabs>
  <Tab title="APC API">
    Use the `deleteDeploymentConfig` mutation:

    ```graphql wrap theme={null}
    mutation {
      deleteDeploymentConfig(
        deploymentUuid: "<deployment-id>"
        reason: "Revert to workspace/cluster defaults"
      ) {
        id
        config
        deletedAt
      }
    }
    ```
  </Tab>

  <Tab title="Astro UI">
    <Frame>
      <img alt="Deployment, Configuration tab, Configuration Override after choosing to remove all deployment-level overrides." />
    </Frame>

    <Steps>
      <Step title="Open deployment configuration">
        In the Astro UI, navigate to the Deployment where you want to remove overrides, then open the **Configuration** tab.
      </Step>

      <Step title="Click Delete">
        Click **Delete** in the **Configuration Override** section to remove all deployment-level overrides.
      </Step>

      <Step title="Confirm deletion" />
    </Steps>
  </Tab>
</Tabs>

## Effective configuration

The *effective configuration* for a Deployment is the result of merging all four tiers. The Astro UI doesn't include a dedicated screen for the fully merged result or a per-key source tier readout (Platform, Cluster, Workspace, or Deployment). You retrieve that from the GraphQL API instead.

The `Deployment` type exposes the merged `deployments` object on `effectiveConfig` (the final value after the Platform through Deployment merge). It also exposes `configOverrides` for the deployment tier only, when you need the fourth layer in isolation. For cluster- and workspace-level override payloads, use the corresponding GraphQL queries or fields on those objects in your client.

The `Deployment` type exposes the merged `deployments` object on `effectiveConfig` (the final value after the Platform through Deployment merge). It also exposes `configOverrides` for the deployment tier only, when you need the fourth layer in isolation.

For example, `workspaceDeployment` can return the merged result alongside deployment-level overrides:

```graphql wrap theme={null}
query {
  workspaceDeployment(
    releaseName: "<release-name>"
    workspaceUuid: "<workspace-id>"
  ) {
    id
    label
    effectiveConfig
    configOverrides {
      config
    }
  }
}
```

To work out the source tier for a given key, compare the merged `effectiveConfig` to each tier’s contribution (platform defaults, cluster and workspace override records, and `configOverrides` as appropriate). The API doesn't label each key with a tier, but the underlying data is all available for inspection with GraphQL.

See also [Example GraphQL API queries](/docs/astro-private-cloud/v-2-x/houston-api-example-queries#query-deployment-details) for `workspaceDeployment` request parameters and related patterns.

## How merge works

Configuration merging uses a *deep merge* strategy:

1. Start with the platform’s default `deployments` object from your control plane `values.yaml` (`astronomer.houston.config`).
2. Deep-merge cluster-level overrides (if the Deployment's cluster has overrides).
3. Deep-merge workspace-level overrides (if the Deployment's workspace has overrides).
4. Deep-merge deployment-level overrides (if the Deployment has its own overrides).

For any key that appears at multiple tiers, the lowest tier wins (Deployment > Workspace > Cluster > Platform). For nested objects, the merge is recursive — you only need to specify the keys you want to override, and all other keys at that level are preserved from the parent tier.

### Example

Given the following configuration at each tier:

**Platform default**:

```yaml wrap theme={null}
deployments:
  deploymentLifecycle:
    deployRollback:
      enabled: false
      deployRevisionReportNumberOfDays: 90
      dagTarballVersionValidation:
        enabled: true
```

**Cluster override**:

```yaml wrap theme={null}
deploymentLifecycle:
  deployRollback:
    enabled: true
```

**Workspace override**:

```yaml wrap theme={null}
deploymentLifecycle:
  deployRollback:
    deployRevisionReportNumberOfDays: 30
```

The effective configuration for Deployments in this workspace and cluster would be:

```yaml wrap theme={null}
deploymentLifecycle:
  deployRollback:
    enabled: true                        # from Cluster
    deployRevisionReportNumberOfDays: 30 # from Workspace
    dagTarballVersionValidation:         # from Platform; not set at Cluster or Workspace
      enabled: true
```

## Configurable domains reference

The following table lists the top-level configuration domains under `deployments` and their availability at each override tier.

| Domain                     | Cluster | Workspace | Deployment | Description                                              |
| -------------------------- | ------- | --------- | ---------- | -------------------------------------------------------- |
| `runtimeManagement`        | Yes     | Yes       | Yes        | Airflow 3, custom image SHA, runtime version constraints |
| `deployMechanisms`         | Yes     | Yes       | Yes        | Dag-only, git-sync, NFS mount deployment                 |
| `airflowComponents`        | Yes     | Yes       | Yes        | Triggerer, Dag processor enablement                      |
| `deploymentLifecycle`      | Yes     | Yes       | Yes        | Deploy rollback, hard delete, Airflow DB cleanup         |
| `resourceManagement`       | Yes     | Yes       | Yes        | Executors, component resources, capacity limits          |
| `databaseManagement`       | Yes     | Yes       | Yes        | Database settings, manual connection strings             |
| `metricsReporting`         | Yes     | Yes       | Yes        | Grafana UI, task usage metrics                           |
| `deploymentImagesRegistry` | Yes     | Yes       | Yes        | Docker webhook, update image endpoint                    |
| `dagDeploy`                | Yes     | Yes       | Yes        | Dag deploy server settings                               |
| `performanceOptimization`  | Yes     | Yes       | Yes        | Performance optimization mode                            |
| `mode`                     | Yes     | Yes       | Yes        | Helm or operator deployment mode                         |
| `authSideCar`              | Yes     | No        | No         | Auth sidecar proxy (platform/cluster only)               |
| `logging`                  | Yes     | No        | No         | Logging sidecar configuration (platform/cluster only)    |
| `namespaceManagement`      | Yes     | Yes       | No         | Namespace settings (immutable at deployment level)       |

## Best practices

* **Keep overrides minimal**: Only set values that need to differ from the parent tier. Fewer overrides mean less configuration drift and easier debugging.
* **Use workspace-level overrides for team defaults**: If all Deployments in a workspace need a common setting (like enabling deploy rollback), set it at the workspace level rather than on each Deployment individually.
* **Use deployment-level overrides for exceptions**: Reserve deployment-level overrides for cases where a single Deployment needs to deviate from workspace defaults.
* **Use the `DELETE_KEY` mechanism**: Remove individual keys from an override without deleting the entire configuration. This is more precise than deleting and recreating the override.
* **Review effective configuration**: In the GraphQL API (for example `Deployment.effectiveConfig` on `workspaceDeployment`), compare the merged result after making changes to confirm that the merge produced the expected result.


# Configure Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-astro-private-cloud

Learn which settings belong in `values.yaml`, cluster configuration, workspace overrides, and deployment overrides in Astro Private Cloud 2.x.

Astro Private Cloud (APC) 2.x uses four customer-facing configuration layers:

* A platform-wide `values.yaml`, which you apply with Helm.
* Cluster configuration, which you set for one data plane cluster.
* Workspace overrides, which apply to all Deployments in one Workspace.
* Deployment overrides, which apply to one Deployment.

Use this document to decide which layer to change and what to expect after you apply the change.

## Use the right configuration layer

Use `values.yaml` when you want to change:

* Sign-in configuration, such as `auth`.
* Platform services, such as registry, ingress, email, and cleanup jobs.
* The default value for a Deployment setting across the platform.

Use cluster configuration when you want to change:

* A `deployments.*` setting for every Deployment in one data plane cluster.
* A cluster-specific exception to the platform default.

Use a Workspace override when you want to change:

* A `deployments.*` setting for every Deployment in one Workspace.
* One Workspace without affecting other Workspaces in the same cluster.

Use a Deployment override when you want to change:

* A `deployments.*` setting for one Deployment only.
* One Deployment without changing the rest of the Workspace.

If your goal is "change this everywhere," use `values.yaml`. If your goal is "change this for one cluster, one workspace, or one Deployment," use the most specific override that matches the scope you want.

## What happens when you run Helm

When you update `values.yaml` and run a Helm upgrade:

* Platform-level settings change after the Helm upgrade finishes.
* Platform defaults under `astronomer.houston.config.deployments` change.
* Saved cluster, Workspace, and Deployment overrides don't change.

If a cluster, Workspace, or Deployment already has its own saved value for the same `deployments.*` key, that saved value stays in effect until you update that specific override through Astro Private Cloud API or UI.

APC 2.x chooses the final value in this order:

1. Platform default
2. Cluster configuration
3. Workspace override
4. Deployment override

```mermaid actions={true} wrap theme={null}
flowchart TD
  platformHelm["Platform (Helm / values.yaml default)"] --> clusterConfig[Cluster configuration]
  clusterConfig --> workspaceConfig[Workspace override]
  workspaceConfig --> deploymentConfig[Deployment override]
```

Treat the preceding list as priority from top to bottom. If the same setting is set in more than one place, the lower line (closer to a single Deployment) is usually what applies. A few settings can be changed only in `values.yaml` or only at the cluster; for those rules, see [Config governance](/docs/astro-private-cloud/v-2-x/config-governance).

## In the Astro UI (2.0)

Use the Astro UI when you aren't using the APC API or your own automation:

* Data plane (cluster): In **Clusters**, open a cluster, then use **Configuration Override** (and related sections on the cluster page). This matches [Data plane overrides](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster).
* Workspace: In a Workspace, go to **Settings**, then **Configuration Overrides**, and edit the YAML for all Deployments in that Workspace in that cluster.
* Deployment: In a Deployment, open **Configuration Overrides** and edit the YAML for a single Deployment.

<Note>
  On a data plane cluster, you edit in **Configuration Override** and you read **Current Configuration** as read-only (defaults, or after you save, a read-only, git-style view of what was added or modified), as in [Data plane overrides](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster). In Workspace and Deployment settings, the **Configuration Overrides** editor is different: it shows the override stored for that workspace or that Deployment. In every case, the platform still combines the four layers in the order above. Who can open and edit these pages depends on your [role and permissions](/docs/astro-private-cloud/v-2-x/role-permission-reference). Platform administrators can [change role permissions](/docs/astro-private-cloud/v-2-x/manage-platform-users#customize-role-permissions) in `values.yaml`.
</Note>

## Decide where to change a setting

Use `values.yaml` for examples like these:

* Enable or disable an identity provider.
* Change a cleanup schedule.
* Change the default Deployment behavior for all clusters.

Use cluster configuration for examples like these:

* Enable a Deployment feature only for the `prod-us-east` data plane.
* Keep one regulated cluster on a different Deployment setting than the rest of the platform.

Use a Workspace override for examples like these:

* Keep one business unit on a different Deployment default than the rest of the cluster.
* Change one Workspace without affecting other Workspaces in the same cluster.

Use a Deployment override for examples like these:

* Test a different Deployment setting on one Deployment.
* Make a one-off exception for a single team or environment.

## Example: Same setting at all four levels

Assume you want to control whether Deployment rollbacks are enabled.

1. You set the platform default in `values.yaml`:

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        deploymentLifecycle:
          deployRollback:
            enabled: false
```

2. The `prod-us-east` cluster overrides the same key to `true`.
3. The `finance` Workspace overrides the same key back to `false`.
4. One Deployment in the `finance` Workspace overrides the same key to `true`.

Result:

* The specific Deployment uses `true`.
* Other Deployments in the `finance` Workspace use `false`.
* Other Deployments in `prod-us-east` use `true`.
* Clusters without a more specific override use the platform default of `false`.

If you run another Helm upgrade and leave the cluster, Workspace, or Deployment override in place, those more specific saved values still win.

## Use the highest scope that matches your goal

To keep configuration manageable:

* Use `values.yaml` for shared defaults.
* Use cluster configuration only when one data plane cluster truly needs a different value.
* Use Workspace overrides only when one Workspace truly needs a different value.
* Use Deployment overrides sparingly for exceptions.

## Important limits

Not every key under the platform `deployments` tree can be set in every override tier, and some keys are blocklisted at the Workspace or Deployment level. The Astro UI and API reject those writes.

* Workspace overrides: `authSideCar`, `logging` can't be set.
* Deployment overrides: `authSideCar`, `logging`, and `namespaceManagement` can't be set.

Rationale, API errors, `DELETE_KEY`, and `strictSchemaCheck` are covered starting at [Blocklisted keys](/docs/astro-private-cloud/v-2-x/config-governance#blocklisted-keys) in [Config governance](/docs/astro-private-cloud/v-2-x/config-governance).

To create or change overrides with automation, use the `updateCluster`, `updateWorkspaceDeploymentsConfig`, and `updateDeploymentConfig` GraphQL mutations, which are described in the same [Config governance](/docs/astro-private-cloud/v-2-x/config-governance) page.

## What APC can update separately

APC can refresh some data plane metadata separately from a Helm upgrade. For example, it can sync values such as data plane chart version or namespace pool metadata back into the cluster record.

This isn't the same as replacing saved cluster, Workspace, or Deployment overrides. If you want to change a saved value under the platform `deployments` tree, update the specific override at the scope where it is stored.

## Related documents

* [Config governance](/docs/astro-private-cloud/v-2-x/config-governance)
* [Helm configuration reference](/docs/astro-private-cloud/v-2-x/helm-config-reference)
* [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config)
* [Data plane overrides](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster)
* [Astro Private Cloud user role and permission reference](/docs/astro-private-cloud/v-2-x/role-permission-reference)


# Configure component size limits
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-component-size-limits

Learn the ways you can configure the maximum and minimum size of platform and Airflow components on Astro Private Cloud.

Astro Private Cloud allows you to customize the minimum and maximum sizes of most Astronomer platform and Airflow components.

You can configure the CPU and memory resources of Airflow components through the Astro UI or with the APC API.

<a />

## Configure Deployment-level limits for individual Pod sizes

You can use `astronomer.houston.config.deployments.maxPodCapacity` to configure the maximum size any individual pod can be.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        maxPodCapacity:
          cpu: 3500
          memory: 13440
```

<a />

## Configure Deployment-level limits for resource usage

Astro Private Cloud limits the amount of resources that can be used by all pods in a Deployment by creating and managing a `LimitRange` and `ResourceQuota` for the namespace associated with each Deployment.

These values are automatically adjusted to account for the resource requirements of various components.

You can add additional resources, beyond the standard amount allocated based on the resource-requirements of standing components, to the `LimitRange` and `ResourceQuota`. Add resources by configuring `astronomer.houston.config.deployments.maxExtraCapacity` to account for the requirements of KubernetesExecutor and `KubernetesPodOperator` tasks.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        maxExtraCapacity:
          cpu: 40000
          memory: 153600
```

<a />

## Configure the sizes of individual Deployment-level components

Components represent different parts of the Astro Private Cloud Deployment. You can customize the default configuration for a component by defining it in `astronomer.houston.config.deployments.components`.

A list of configurable components and options is provided in [Configurable Components](#configurable-components).

<Tip>KubernetesExecutor Task pod sizes are created on an as-needed basis, and don't have persisting resource requirements. Their resource requirements are [configured at the task level](#configure-kubernetes-task-pod-size).</Tip>

When defining components, you must include the full definition of the component in the list entry after the components key, instead of only the components you want to define.

For example, to increase the maximum size a Celery worker task from 3 Vcpu/11.5Gi to 3 Vcpu/192Gi, add the equivalent (in milli vCPU and Mi) full Celery worker component definition to `astronomer.houston.config.deployments.components` in your `values.yaml` with a higher limit:

<Tip>When increasing CPU or memory limits, ensure the [maximum pod size](#configure-max-pod-size) is large enough to avoid errors during pod creation.</Tip>

```yaml expandable wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        components:
        - name: workers
          resources:
            default:
              cpu: 1000
              memory: 3840
            minimum:
              cpu: 100
              memory: 384
            limit:
              cpu: 3000
              memory: 11520
            KubernetesExecutor:
              default:
                cpu: 100
                memory: 384
              minimum:
                cpu: 100
                memory: 384
              limit:
                cpu: 3000
                memory: 11520
          extra:
            - name: terminationGracePeriodSeconds
              default: 600
              minimum: 0
              limit: 36000
            - name: replicas
              default: 1
              minimum: 1
              limit: 10
        # any additional component configurations go here
        # - name: another-component
        #   resources:
        #     default: 10
        #     ...
```

### Configurable components

<Info>When defining components, you must include the full definition of the component in the list entry after the components key, instead of only the components you want to define.</Info>

<Tip>KubernetesExecutor task pod sizes are created on an as-needed basis and don't have persisting resource requirements. Their resource requirements are [configured at the task level](#configure-kubernetes-task-pod-size).</Tip>

Configurable components include:

#### Airflow scheduler

```yaml wrap theme={null}
- name: scheduler
  resources:
    default:
     cpu: 500
     memory: 1920
    minimum:
     cpu: 500
     memory: 1920
    limit:
     cpu: 3000
     memory: 11520
  extra:
    - name: replicas
      default: 1
      minimum: 1
      limit: 4
```

#### Airflow webserver

```yaml wrap theme={null}
- name: webserver
  resources:
    default:
     cpu: 500
     memory: 1920
    minimum:
     cpu: 500
     memory: 1920
    limit:
     cpu: 3000
     memory: 11520
```

#### Airflow `apiServer` (Airflow 3.0 and above)

```yaml wrap theme={null}
- name: apiServer
  resources:
    default:
     cpu: 1000
     memory: 3840
    minimum:
     cpu: 1000
     memory: 3840
    limit:
     cpu: 3000
     memory: 11520
   extra:
    - name: replicas
      default: 1
      minimum: 1
      limit: 4
```

#### StatsD

```yaml wrap theme={null}
- name: statsd
  resources:
    default:
     cpu: 200
     memory: 768
    minimum:
     cpu: 200
     memory: 768
    limit:
     cpu: 3000
     memory: 11520
```

#### Database connection pooler (PgBouncer)

```yaml wrap theme={null}
- name: pgbouncer
  resources:
    default:
     cpu: 200
     memory: 768
    minimum:
     cpu: 200
     memory: 768
    limit:
     cpu: 200
     memory: 768
```

#### Celery diagnostic web interface (Flower)

```yaml wrap theme={null}
- name: flower
  resources:
    default:
     cpu: 200
     memory: 768
    minimum:
     cpu: 200
     memory: 768
    limit:
     cpu: 200
     memory: 768
```

#### Redis

```yaml wrap theme={null}
- name: redis
  resources:
    default:
     cpu: 200
     memory: 768
    minimum:
     cpu: 200
     memory: 768
    limit:
     cpu: 200
     memory: 768
```

#### Celery workers

```yaml wrap theme={null}
- name: workers
  resources:
    default:
     cpu: 1000
     memory: 3840
    minimum:
     cpu: 100
     memory: 384
    limit:
     cpu: 3000
     memory: 11520
  extra:
    - name: terminationGracePeriodSeconds
      default: 600
      minimum: 0
      limit: 36000
    - name: replicas
      default: 1
      minimum: 1
      limit: 10
```

#### Triggerer

```yaml expandable wrap theme={null}
- name: triggerer
  resources:
    default:
     cpu: 500
     memory: 1920
    minimum:
     cpu: 500
     memory: 1920
    limit:
     cpu: 3000
     memory: 11520
  extra:
    - name: replicas
      default: 1
      minimum: 0
      limit: 2
```

#### Dag processor

```yaml wrap theme={null}
- name: dagProcessor
  resources:
    default:
     cpu: 500
     memory: 1920
    minimum:
     cpu: 500
     memory: 1920
    limit:
     cpu: 3000
     memory: 11520
  extra:
    - name: replicas
      default: 0
      minimum: 0
      limit: 3
```

<a />

##  Manage Kubernetes worker CPU and memory with global platform config

You can use `workers.resources.enabled` for the `KubernetesExecutor` to manage worker Pod CPU and memory allocations with a platform configuration. When set to `false`, Astro Private Cloud ignores worker Pod CPU or memory allocations set in the API or UI. This allows you to control Pod resources with your own Kubernetes policies or admission controllers.

### API/UI manages Pod resources (Default KubernetesExecutor configuration)

To configure the default behavior where the API and UI manage worker Pod CPU and memory resources for the KubernetesExecutor, use the following configuration:

```yaml wrap theme={null}
deployments:
  executors:
    - name: KubernetesExecutor
      enabled: true
      components:
        - scheduler
        - webserver
        - apiServer
        - statsd
        - pgbouncer
        - triggerer
        - dagProcessor
      defaultExtraCapacity:
        cpu: 1000
        memory: 3840
      workers:
        ephemeralStorage:
          disabled: true
        resources:
          enabled: true
```

### Disable API/UI resource configuration

If you configure your KubernetesExecutor so that CPU and memory requests and limits are set outside of Astro Private Cloud, you must disable resource configuration set with the Astro Private Cloud API/UI. To disable the Astro Private Cloud UI/API configuration, add the `resources.enabled: false` flag to your `values.yaml` file. This ensures that Astro Private Cloud applies the worker resource settings exclusively from your preferred source, such as Pod mutation hooks or Pod configuration files. The following configuration sets them as an empty `dict`.

```yaml wrap theme={null}
deployments:
  executors:
    - name: KubernetesExecutor
      enabled: true
      components:
        - scheduler
        - webserver
        - apiServer
        - statsd
        - pgbouncer
        - triggerer
        - dagProcessor
      defaultExtraCapacity:
        cpu: 1000
        memory: 3840
      workers:
        ephemeralStorage:
          disabled: true
        resources:
          enabled: false
```

Existing deployments are unchanged unless you update this setting.

<a />

### Configure the size of KubernetesExecutor task pods

Kubernetes Executor task pods are defined at the task level when the Dag passes resource requests as part of `executor_config` into the Operator. When not defined, these tasks default to using 0.1 Vcpu/384 Mi of memory. This means that when you define resource requests or limits for CPU and memory, ensure the [maximum pod size](#configure-max-pod-size) is large enough to avoid errors during pod creation.

<Danger>Astro Private Cloud doesn't automatically raise the namespace-level cumulative resource limits for pods created by the KubernetesExecutor. To avoid pod creation failures, increase the `maxExtraCapacity` to support your desired level of resourcing and concurrency.</Danger>

The following example demonstrates how to configure resource limits and requests:

```text expandable wrap theme={null}
# import kubernetes.client.models as k8s
from kubernetes.client import models as k8s
# define an executor_config with the desired resources
my_executor_config={
  "pod_override": k8s.V1Pod(
      spec=k8s.V1PodSpec(
          containers=[
              k8s.V1Container(
                  name="base",
                  resources=k8s.V1ResourceRequirements(
                      requests={
                          "cpu": "50m",
                          "memory": "384Mi"
                      },
                      limits={
                          "cpu": "1000m",
                          "memory": "1024Mi"
                      }
                  )
              )
          ]
      )
  )
}
# pass in executor_config=my_executor_config to any Operator
#@task(executor_config=my_executor_config)
#def some_task():
 #   ...
#task = PythonOperator(
#    task_id="another_task",
#    python_callable=my_fun,
#    executor_config=my_executor_config
#)
```

Note that KubernetesExecutor task Pods are limited to the `LimitRanges` and `quotas` defined within the pod namespace.


# Configure control plane reliability
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-control-plane-disaster-recovery

Set up an Astro Private Cloud control plane reliability group by installing and registering two or more control planes that share one database and one global domain.

This guide walks you through standing up a control plane reliability group: two or more control planes that share one database, one set of JSON Web Token (JWT) signing material, and one global domain behind weighted, health-checked DNS.

For a conceptual overview of the feature, see [Control plane reliability](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery). After the group is running, see [Manage a control plane reliability group](/docs/astro-private-cloud/v-2-x/manage-control-plane-disaster-recovery) for day-2 operations.

Complete the steps in order. The second control plane depends on secrets and registry records created by the first.

## Prerequisites

* An Astro Private Cloud (APC) 2.1.0 or later installation.
* The ability to install the Astronomer platform on two or more Kubernetes clusters, one per control plane. See [Install the control plane](/docs/astro-private-cloud/v-2-x/install-control-plane).
* A shared database server that every control plane can reach through an identical connection.
* A TLS certificate per control plane that covers both the global and per-control-plane (per-CP) admin names. See the following section.
* Permission to create DNS records for both the global domain and each per-CP admin hostname.
* `kubectl` access to each control plane cluster.

## Set up TLS and DNS prerequisites

Set these up before you install any control plane. Every control plane is reachable on two kinds of hostnames — the shared global hostname behind the load balancer, and its own per-CP admin hostname that bypasses the load balancer. Both must be covered by TLS and DNS.

### TLS certificates

For each control plane, provision a certificate whose Subject Alternative Names (SANs) cover both:

* The global names — `*.<global-domain-name>`, which cover `app.`, `houston.`, and the other global subdomains.
* That control plane's per-CP admin names — `*.<cpNN-domain>`, for example `*.cp01.<parent-domain>`.

A single certificate per control plane that covers both wildcards is the simplest arrangement.

<Warning>
  Use the DNS-01 Automatic Certificate Management Environment (ACME) challenge, not HTTP-01, for issued certificates. Under control plane reliability the global hostname resolves only to active-region control planes, so an HTTP-01 challenge for a global name can't be served by a standby control plane and won't validate reliably. DNS-01 doesn't require the hostname to be routable and succeeds from any control plane that has DNS-provider API credentials.
</Warning>

### DNS records

* *Per-CP admin hostnames*: add a static DNS record for each control plane's admin hostname (`cp01.<parent-domain>`, `cp02.<parent-domain>`) that points directly at that control plane and bypasses the load balancer. You need these to reach and register a specific control plane before it's in load-balancer rotation.
* *Global hostname*: `app.<global-domain-name>` and the other global names must resolve to all of the control plane load balancers, with a per-CP health check at `/controlplane/status`. This is the load-balanced, failover record set, configured in the last step of this guide. See [Configure the global DNS](#configure-the-global-dns).

## Install control plane 1 (the bootstrap cluster)

Install the Astronomer platform on the first cluster with control plane reliability enabled. CP 1 is the cluster that generates the shared JWT signing material every other control plane reuses.

Add the following to your Astronomer Helm values:

```yaml theme={null}
global:
  baseDomain: <cp01-domain>          # this control plane's own per-CP admin domain
  controlPlaneHA:
    enabled: true
    bootstrapJwks: true
    globalBaseDomain: <global-domain-name>
  dataPlaneFailover:
    enabled: true
```

Each setting does the following:

* `baseDomain` is this control plane's per-CP admin domain. Each control plane has its own `baseDomain` (`cp01.<parent-domain>`, `cp02.<parent-domain>`), distinct from the shared `globalBaseDomain`.
* `controlPlaneHA.enabled: true` turns on control plane reliability mode. The chart renders the global-domain ingresses and the APC API runs as part of a high availability (HA) group.
* `controlPlaneHA.bootstrapJwks: true` tells this control plane to generate the shared JWT signing key and certificate. Because this is the first cluster, it bootstraps the material itself, creating two secrets: `<release-name>-houston-jwt-signing-key` and `<release-name>-houston-jwt-signing-certificate`. This certificate also signs Docker Registry tokens; there is no separate registry keypair.
* `controlPlaneHA.globalBaseDomain` is the shared domain that all control planes serve.
* `dataPlaneFailover.enabled: true` is optional. It enables data plane failover so that Astro Deployments can be moved between clusters. It's independent of control plane reliability. See [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover).

Install or upgrade the release and wait for all Pods to come up. After the APC API is running, confirm that both JWT secrets exist in the Astronomer namespace:

```bash theme={null}
kubectl get secret <release-name>-houston-jwt-signing-key \
  <release-name>-houston-jwt-signing-certificate -n <astronomer-namespace>
```

CP 2 reuses these two secrets in the next step.

## Copy the JWT signing material to control plane 2

Both control planes must sign JWTs with the same key and certificate, so that a token minted on one control plane is accepted on the other. CP 1 generated this material in the previous step. CP 2 must reuse it rather than generate its own.

Copy these two secrets from CP 1's Astronomer namespace into CP 2's Astronomer namespace, keeping the same names and data keys:

* `<release-name>-houston-jwt-signing-key` holds data key `tls.key`.
* `<release-name>-houston-jwt-signing-certificate` holds data key `tls.crt`.

<Warning>
  The signing-certificate secret must carry the following annotation so that the certificate is propagated into the Airflow Deployment namespaces:

  ```text theme={null}
  astronomer.io/commander-sync: platform-release=<release-name>
  ```
</Warning>

Copy the secrets with whatever tooling you use for secrets — `kubectl get -o yaml` and `kubectl apply`, sealed secrets, GitOps, the External Secrets Operator, or a secrets manager. For example:

```bash theme={null}
# On CP 1, export the two secrets:
kubectl get secret <release-name>-houston-jwt-signing-key \
  <release-name>-houston-jwt-signing-certificate \
  -n <astronomer-namespace> -o yaml > jwt-secrets.yaml

# Remove cluster-specific metadata (resourceVersion, uid, creationTimestamp),
# keep the commander-sync annotation on the certificate secret, then on CP 2:
kubectl apply -f jwt-secrets.yaml -n <astronomer-namespace>
```

These secrets must exist on CP 2 before you install the Astronomer platform there. CP 2 runs with `bootstrapJwks: false` and consumes the pre-copied secrets. Verify that both secrets exist on CP 2 before you continue.

## Install control plane 2 (and any subsequent control planes)

Install the Astronomer platform on the second cluster. The values are almost identical to CP 1, with two differences: use this control plane's own `baseDomain`, and set `bootstrapJwks: false` so that CP 2 consumes the signing material you copied instead of generating its own.

```yaml theme={null}
global:
  baseDomain: <cp02-domain>          # THIS control plane's own per-CP admin domain
  controlPlaneHA:
    enabled: true
    bootstrapJwks: false             # consume the pre-copied JWT secrets
    globalBaseDomain: <global-domain-name>
  dataPlaneFailover:
    enabled: true
```

If CP 2 bootstrapped its own JSON Web Key Set (JWKS), it would sign tokens with a different key than CP 1, and tokens would fail validation when a user is routed to the other cluster. Setting `bootstrapJwks: false` makes CP 2 use the pre-copied secrets. You don't need to disable JWKS generation any other way — HA mode skips generation by default.

<Warning>
  Both control planes must use the same database. In practice, the `astronomer-bootstrap` secret that holds the database connection and credentials must be identical on both clusters. If the two control planes point at different databases, they don't share users, Astro Deployments, or the control plane registry, and control plane reliability doesn't work.

  Before you install CP 2, copy the `astronomer-bootstrap` secret from CP 1, strip its cluster-specific metadata, and apply it on CP 2:

  ```bash theme={null}
  kubectl get secret astronomer-bootstrap -n <astronomer-namespace> -o yaml
  # (strip cluster-specific metadata) then apply on CP 2
  ```
</Warning>

Install or upgrade the release on CP 2 and wait for all Pods to be healthy. Because it shares CP 1's database, CP 2 immediately sees the same users, Workspaces, and Astro Deployments.

Repeat this step and the previous JWT-copy step for every additional control plane you want in the group. Each one copies the same JWT material and points at the same database.

## Create the first admin user

Before you can register regions or control planes, you need an admin user to authenticate as. Because all control planes share one database, you create the admin only once and it works across every control plane.

Open `app.<global-domain-name>` in a browser and complete the first-admin sign-up flow, or use the `createUser` mutation. The `createRegion` and `registerControlPlane` mutations you run next require you to be authenticated as a system admin.

## Register the region and control plane 1

Tell the APC API that CP 1 exists and which region it belongs to. You can do this from the APC UI or with GraphQL mutations, authenticated as the admin from the previous step.

### Create a region

A region is the logical grouping that control planes attach to. All control planes in the same group share the same region.

Run the `createRegion` mutation:

```graphql theme={null}
mutation {
  createRegion(name: "us-east", cloudProvider: "aws") {
    region { id name cloudProvider }
  }
}
```

* `name` is a human-readable region name and must be unique.
* `cloudProvider` is the cloud provider, for example `aws`, `gcp`, or `azure`.

Save the returned `region.id`. You need it in the next call and again when you register CP 2. Only one region can be active at a time.

To create the region from the APC UI instead, open the **Regions** tab in the left sidebar (visible after you enable control plane reliability) and select **Create Region**. Enter a **Name** and choose a **Cloud Provider**, then save. The region appears in the list with an **Inactive** status. New regions start inactive, matching the mutation's default. Use the per-row **Activate** action when you're ready to make it the serving region.

<Frame>
  <img alt="Create Region dialog in the APC UI, with a Name entered and the Cloud Provider dropdown open showing AWS, GCP, Azure, and Local." />
</Frame>

<Frame>
  <img alt="Regions list in the APC UI showing the newly created region with an Inactive status badge." />
</Frame>

### Register the control plane

Register CP 1 into the control plane registry and attach it to the region:

```graphql theme={null}
mutation {
  registerControlPlane(cpId: "<cp1-uuid>", name: "cp01", regionId: "<region-id>") {
    id
    name
    ingressUrl
    chartVersion
    region { id name }
  }
}
```

* `cpId` is the stable UUID for this control plane. It comes from the `cp-identity` secret on the cluster. Read it from the control plane itself rather than inventing one:

  ```bash theme={null}
  kubectl -n <astronomer-namespace> get secret cp-identity \
    -o jsonpath='{.data.cp_id}' | base64 -d
  ```
* `name` is a human-readable identifier for this control plane and must be unique across the registry.
* `regionId` is the region ID from the previous step.

Don't pass `ingressUrl` or `chartVersion` — the APC API derives them from the control plane's own configuration. The mutation is idempotent: re-running it with the same `cpId` refreshes those fields, which is useful after a Helm upgrade. The region must already exist before you register a control plane against it.

To register from the APC UI instead, use that control plane's own admin URL (`cpNN.<parent-domain>`) so the identity pre-fills correctly. From CP 1's admin URL, open the **Control Planes** tab and select **Register Control Plane**. The **Control Plane ID** field is pre-filled from this control plane's `cp-identity` secret — leave it as-is. Enter a **Name** and select the **Region** you created, then save. The APC API populates **Ingress URL** and **Chart Version** automatically.

<Frame>
  <img alt="Register Control Plane dialog in the APC UI, with the Control Plane ID pre-filled, a Name entered, and the Region dropdown open." />
</Frame>

## Register control plane 2

CP 2 needs its own registry entry so that the APC API knows it's part of the group. Because both clusters share the same database, the region you created already exists, so you don't create a new one. You only register the new control plane against that existing region.

Run the `registerControlPlane` mutation with CP 2's own `cpId` and the same `regionId` from before:

```graphql theme={null}
mutation {
  registerControlPlane(cpId: "<cp2-uuid>", name: "cp02", regionId: "<region-id>") {
    id
    name
    ingressUrl
    region { id name }
  }
}
```

Get CP 2's `cpId` the same way as before, but from CP 2's cluster:

```bash theme={null}
kubectl -n <astronomer-namespace> get secret cp-identity \
  -o jsonpath='{.data.cp_id}' | base64 -d
```

To register from the APC UI instead, use CP 2's admin URL (`cp02.<parent-domain>`), open the **Control Planes** tab, and select **Register Control Plane**. The **Control Plane ID** pre-fills with CP 2's identity. Enter a **Name** and select the same **Region** as CP 1 — don't create a new one — then save.

After this, both control planes are registered against the same region and are part of the same group. List the registered control planes to confirm that both entries share the same region.

<Frame>
  <img alt="Control Planes list in the APC UI showing two registered control planes, cp-ha-01 and cp-ha-02, attached to the same region." />
</Frame>

## Configure the global DNS

This step load-balances user traffic across the active region's control planes and provides in-region high availability. Point the global names — `app.<global-domain-name>` and the rest of `*.<global-domain-name>` — at all of the control plane load balancers using weighted, health-checked DNS records, so that traffic is spread across the control planes and drained away from an unhealthy one automatically.

### Create the record sets

For the shared global domain, create a record set that covers both:

* `<global-domain-name>`: the apex.
* `*.<global-domain-name>`: the wildcard. This one record set covers every customer-facing subdomain (`app.`, `houston.`, `grafana.`, `prometheus.`, `alertmanager.`).

For each control plane, add an entry to both record sets that points at that control plane's ingress load balancer, with:

* *Weighted routing* that gives every control plane an equal weight, so DNS load-balances evenly. Each control plane's entry needs its own unique set identifier.
* *A health check* attached to each control plane's entry, so that an unhealthy control plane is removed from rotation automatically.

With two control planes, both the apex and the wildcard record sets contain two equal-weight entries, one aliasing each control plane's load balancer, each guarded by its own health check.

<Note>
  Keep this separate from the per-CP admin records. The static per-CP admin records (`cp01.<parent-domain>`, `cp02.<parent-domain>`) point directly at a single control plane and aren't part of this weighted set. They intentionally bypass the load balancer.
</Note>

### Configure the health check

Tie each control plane's DNS entry to a health check that targets that control plane's own APC API endpoint on its per-CP hostname, not the global one:

* Protocol `HTTPS`, port `443`.
* Path `/controlplane/status`, the APC API HA health endpoint. It reports the control plane as unhealthy if the control plane isn't registered, cordoned, decommissioned, running an outdated chart version, or attached to an inactive region.
* A reasonable cadence, for example a 30-second interval with a failure threshold of three.

When a control plane's `/controlplane/status` starts failing, its DNS entry is pulled from the record set and users are routed to the remaining healthy control planes. When it recovers, it's added back automatically. At least one control plane must be healthy and in the active region to serve traffic. If every control plane is unhealthy, the global name serves nothing.

<Warning>
  Use weighted, health-checked records for the global domain, not plain records. Plain records send users to a single control plane with no automatic failover, and multiple plain records for the same name conflict with each other. If you run DNS automation that would otherwise create plain records, make sure it doesn't manage `<global-domain-name>` or its subdomains, or it will fight with these weighted, health-checked records.
</Warning>

### Verify

After the records are in place:

```bash theme={null}
# The global names resolve, load-balancing across the control plane load balancers:
dig +short <global-domain-name>
dig +short app.<global-domain-name>

# Each control plane's health endpoint reports healthy:
curl -s https://houston.<cpNN-domain>/controlplane/status
```

In your DNS provider, the apex and `*` records for `<global-domain-name>` should show one weighted entry per control plane, each with its own health check. Take one control plane offline and confirm that its answer stops being served while the global domain stays reachable through the healthy control plane. After a control plane recovers, resolvers may keep serving a cached negative answer for the negative-cache TTL, so flush your local DNS cache and allow for upstream resolver TTLs when you verify recovery.

## Add existing Astro Deployments to the global domain

Astro Deployments that existed before you enabled control plane reliability have their Apache Airflow ingress host rules and auth annotations (`auth-url`, `auth-signin`) pointing at the per-CP URL of whichever control plane last upserted them. For those Deployments to work under control plane reliability and be reachable on the global domain with cross-control-plane single sign-on (SSO), their ingresses must be re-stamped. Re-stamping:

* Adds the global-domain alias hosts (`<release>-airflow.….<globalBaseDomain>`) alongside the existing per-CP hosts.
* Repoints `auth-signin` to `https://houston.<globalBaseDomain>/v1/auth/deployment-signin`.
* Regenerates `auth-url` (`…/v1/authorization`) against `globalBaseDomain`.

After re-stamping, a Deployment is reachable at both its per-CP URL and its new global URL, and the global session cookie (scoped to `.<globalBaseDomain>`) is sent to the Deployment's global Airflow subdomain, so SSO works across control planes.

For how the session cookie and Deployment URLs are scoped, see [Cookie and URL strategy](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery-reference#cookie-and-url-strategy). Two ways to perform the re-stamp are available.

### Automatic (default)

Because `astronomer.houston.upgradeDeployments.enabled` is `true` by default, the `houston-upgrade-deployments` hook runs on the same `helm upgrade` that enables control plane reliability, and on every control plane upgrade after that. It's a `post-upgrade` hook, so it runs after the new configuration (including `globalBaseDomain`) is applied. It therefore re-stamps against the live global domain and doesn't leave stale per-CP-only URLs. You don't need to run any manual steps in the default configuration.

<Warning>
  This hook re-templates every non-cordoned Astro Deployment on every control plane `helm upgrade`, so each upgrade triggers a fleet-wide re-upsert and can restart Airflow Pods. On a large production fleet this means a simultaneous restart of many Deployments, which can cause a load spike and brief disruption to running tasks, with no control over timing. The re-stamp is idempotent: it re-applies each Deployment's desired state without changing its version.
</Warning>

### Manual (when the hook is disabled)

To decouple the re-stamp from the upgrade so you can run it in a maintenance window and scope the rollout, set `astronomer.houston.upgradeDeployments.enabled: false`. Existing Deployments are then not re-stamped automatically, and you run the script yourself after `globalBaseDomain` is live:

```bash theme={null}
# Run inside an APC API (houston) Pod, using the same command the hook uses:
yarn upgrade-deployments                              # all non-cordoned Deployments
yarn upgrade-deployments -- --clusterId=<uuid>        # scope to one cluster
yarn upgrade-deployments -- --deploymentUuid=<id>     # scope to one Deployment
yarn upgrade-deployments -- --canary                  # scope to canary Deployments only
```

The script is idempotent and re-runnable, and scoping to zero matches is a safe no-op. Run it only after `globalBaseDomain` and the URL-helper and annotation support are live, so that it stamps the correct global URLs rather than per-CP ones.

<Note>
  Set `astronomer.houston.upgradeDeployments.enabled` under the `astronomer` subchart, as shown. A top-level `houston.upgradeDeployments.enabled` doesn't take effect and silently leaves the hook enabled.
</Note>

## Configure identity provider redirect URLs

If you integrate an external identity provider (IdP), the redirect and callback URLs you register with the IdP must point at the global domain, not a per-CP domain. Under control plane reliability the APC API templates all customer-facing URLs, including the OAuth `redirect_uri`, from `globalBaseDomain`.

If you're migrating an existing installation to control plane reliability and your IdP application was registered with a per-CP redirect URI, update it to the global-domain URI as part of enabling control plane reliability. For the exact values to register and the reason behind them, see [Identity provider authentication and the OAuth redirect URL](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery-reference#identity-provider-authentication-and-the-oauth-redirect-url).

## Related documentation

* [Control plane reliability](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery)
* [Manage a control plane reliability group](/docs/astro-private-cloud/v-2-x/manage-control-plane-disaster-recovery)
* [Control plane reliability reference](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery-reference)
* [Integrate an auth system](/docs/astro-private-cloud/v-2-x/integrate-auth-system)
* [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config)


# Configure a Deployment on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-deployment

Learn the ways you can configure individual Airflow Deployments on Astro Private Cloud.

An Airflow Deployment on Astro Private Cloud (APC) is an instance of Apache Airflow that runs in your APC cluster. Each APC Deployment is hosted on a dedicated Kubernetes namespace, has a dedicated set of resources, and operates with an isolated Postgres metadata database.

A Deployment typically encapsulates a single use case or context. Each Deployment has a number of settings that you can fine-tune so that you're running Airflow optimally for this context. Use the following topics to learn more about each available configuration option on an APC Deployment.

## Deployment creation and deletion

Creating and deleting Deployments can become an operational complexity when you run Airflow across many teams. See the following documentation to learn about all of the options for creating Deployments, deleting Deployments, and ensuring that Deployment resources are returned back to your cluster for future use.

## Deployment resources

The amount of CPU and memory available to your Deployment defines how many tasks it can run in parallel. See [Deployment resources](/docs/astro-private-cloud/v-2-x/scale-deployment-resources) to learn how to configure your Deployment's scheduler, executor, and webserver resources.

## Environment variables

Each Deployment has its own set of environment variables that you can use to define both Airflow-level configurations and Dag objects such as connections. See [Environment variables](/docs/astro-private-cloud/v-2-x/environment-variables) to learn more about setting these

## Deploy methods

APC supports several different methods for deploying Dags and code-based project configurations to Astro. See [Deploy methods overview](/docs/astro-private-cloud/v-2-x/deploy-code-overview).

When you're ready to automate a deploy mechanism at scale for your team, see [CI/CD](/docs/astro-private-cloud/v-2-x/ci-cd) to learn how to configure API credentials and examples of CI/CD pipelines in different version management tools.

## Upgrade Deployments

To take advantage of the latest features in Apache Airflow and stay in support, you must upgrade your Deployments over time. See [Upgrade Astro Runtime](/docs/runtime/manage-airflow-versions) for setup steps.

The Airflow [executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html) works closely with the Airflow scheduler to decide what resources will complete tasks as they're queued. The difference between executors comes down to their available resources and how they utilize those resources to distribute work.

APC supports 3 executors:

* [Local executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/local.html)
* [Celery executor](https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/index.html)
* [Kubernetes executor](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/kubernetes_executor.html)

Though it largely depends on your use case, we recommend the Local executor for development environments and the Celery or Kubernetes executors for production environments operating at scale.

For a detailed breakdown of each executor, see [Airflow executors explained](https://docs.astronomer.io/learn/airflow-executors-explained).

## Scale core resources

Apache Airflow requires two primary components:

* The Airflow webserver/ API server
* The Airflow Scheduler

To scale either resource, adjust the corresponding slider in the UI to increase its available computing resources.

Read the following sections to help you determine which core resources to scale and when.

### Airflow webserver and API server

In Airflow 2, the webserver is responsible for rendering the [Airflow UI](https://airflow.apache.org/docs/apache-airflow/stable/ui.html), where users can monitor Dags, view task logs, and set various non-code configurations.

In Airflow 3, the webserver has evolved into the API server. The API server serves both the UI and the internal Task API for communication between Airflow components. The API server acts as the single entry point for user interface access and task execution requests.

If the Airflow UI or API experiences slowness or is unavailable, increase the resources allocated to the webserver (Airflow 2) or API server (Airflow 3), depending on your Airflow version.

### Airflow scheduler

The [Airflow scheduler](https://airflow.apache.org/docs/apache-airflow/stable/scheduler.html) is responsible for monitoring task execution and triggering downstream tasks once dependencies have been met.

If you experience delays in task execution, which you can track using the [Gantt Chart](https://airflow.apache.org/docs/apache-airflow/stable/ui.html#gantt-chart) view of the Airflow UI, APC recommends increasing the resources allocated towards the scheduler.

<Tip> To set alerts that notify you via email when your Airflow scheduler is underprovisioned, refer to [Airflow alerts](/docs/astro-private-cloud/v-0-37/airflow-alerts).</Tip>

#### Scheduler count

Airflow 2.0 comes with the ability for users to run multiple schedulers concurrently to ensure high-availability, zero recovery time, and faster performance. By adjusting the **Scheduler Count** slider in the UI, you can provision up to 4 schedulers on any Deployment running Airflow 2.0+ on Astronomer.

Each individual scheduler will be provisioned with the resources specified in **Scheduler Resources**. For example, if you set the CPU figure in **Scheduler Resources** to 5 CPUs and set **Scheduler Count** to 2, your Airflow Deployment will run with 2 Airflow schedulers using 5 CPUs each for a total of 10 CPUs.

To increase the speed at which tasks are scheduled and ensure high-availability, Astronomer recommends provisioning 2 or more Airflow schedulers for production environments. For more information on the Airflow 2.0 scheduler, refer to Astronomer's ["The Airflow 2.0 Scheduler" blog post](https://www.astronomer.io/blog/airflow-2-scheduler).

### Triggerer

Airflow 2.2 introduces the triggerer, which is a component for running tasks with [deferrable operators](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html). Like the scheduler, the triggerer is highly available: If a triggerer shuts down unexpectedly, the tasks it was deferring can be recovered and moved to another triggerer.

By adjusting the **Triggerer** slider in the UI, you can provision up to 2 triggerers on any Deployment running Airflow 2.2+. To take advantage of the triggerers high availability, we recommend provisioning 2 triggerers for production Deployments.

## Kubernetes executor: Set extra capacity

On APC, resources required for the [`KubernetesPodOperator`](/docs/astro-private-cloud/v-2-x/kube-pod-operator) or the [Kubernetes Executor](/docs/astro-private-cloud/v-2-x/kubernetes-executor) are set as **Extra Capacity**.

The Kubernetes executor and `KubernetesPodOperator` each spin up an individual Kubernetes pod for each task that needs to be executed, then spin down the pod once that task is completed.

The amount of CPU and Memory allocated to **Extra Capacity** maps to [resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) on the [Kubernetes Namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) in which your Airflow Deployment lives on APC. More specifically, **Extra Capacity** represents the maximum possible resources that could be provisioned to a pod at any given time.

Resources allocated to **Extra Capacity** don't affect scheduler or webserver performance and don't represent actual usage.

## Celery executor: Configure workers

To optimize for flexibility and availability, the Celery executor works with a set of independent Celery workers across which it can delegate tasks. On APC, you're free to configure your Celery workers to fit your use case.

### Worker count

By adjusting the **Worker Count** slider, users can provision up to 20 Celery workers on any Airflow Deployment.

Each individual worker will be provisioned with the resources specified in **Worker Resources**. If you set the CPU figure in **Worker Resources** to 5 CPUs and set **Worker Count** to 3, for example, your Airflow Deployment will run with 3 Celery workers using 5 CPUs each for a total of 15 CPUs.

### Worker termination grace period

On APC, Celery workers restart following every code deploy to your Airflow Deployment. This is to make sure that workers are executing with the most up-to-date code. To minimize disruption during task execution, however, APC supports the ability to set a **Worker Termination Grace Period**.

If a deploy is triggered while a Celery worker is executing a task and **Worker Termination Grace Period** is set, the worker will continue to process that task up to a certain number of minutes before restarting itself. By default, the grace period is ten minutes.

<Tip>The **Worker Termination Grace Period** is an advantage to the Celery executor. If your Airflow Deployment runs on the Local executor, the scheduler will restart immediately upon every code deploy or configuration change and potentially interrupt task execution.</Tip>

## Set environment variables

Environment variables can be used to set [Airflow configurations](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html) and custom values, both of which can be applied to your Airflow Deployment either locally or on APC.

These can include setting Airflow Parallelism, an SMTP service for alerts, or a [secrets backend](/docs/astro-private-cloud/v-2-x/secrets-backend) to manage Airflow connections and variables.

Environment variables can be set for your Airflow Deployment either in the **Variables** tab of the UI or in your `Dockerfile`. If you're developing locally, they can also be added to a local `.env` file. For more information on configuring environment variables, read [Environment variables on APC](/docs/astro-private-cloud/v-2-x/environment-variables).

<Note> Environment variables are distinct from [Airflow variables](https://airflow.apache.org/docs/apache-airflow/stable/howto/variable.html?highlight=variables) and [XComs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html), which you can configure directly using the Airflow UI and are used for inter-task communication.</Note>

## Customize release names

An Airflow Deployment's release name on APC is a unique, immutable identifier for that Deployment that corresponds to its Kubernetes namespace and that renders in Grafana and other platform-level monitoring tools. By default, release names are randomly generated in the following format: `noun-noun-<4-digit-number>`. For example: `elementary-zenith-7243`.

To customize the release name for a Deployment as you're creating it, you first need to enable the feature on your data plane cluster. To do so:

1. In the APC UI, go to your **Clusters** page and select your cluster.

2. In the cluster details page, click **Edit**. In the **Cluster Deployments Configuration** YAML editor, click **Find** and search for the feature you want to override. Then add the following override in the appropriate YAML field:

   ```yaml wrap theme={null}
   namespaceManagement:
     manualReleaseNames:
       enabled: true # Allows you to set your release names
   ```

   For details on using the UI for configuration, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).

3. Click **Update Cluster**.

After applying this change, the **Release Name** field in the UI becomes configurable:

<Frame>
  <img alt="Custom Release Name Field" />
</Frame>

## Code deploy mechanisms

Deploying code is the process of applying code from your local machine to an APC Deployment. A code deploy can include an entire Astro project as a Docker image, or just the code in your Astro project `dags` directory. APC supports a few different methods for deploying code to a Deployment. You can:

* Deploy [project images](/docs/astro-private-cloud/v-2-x/deploy-code-overview#full-image-deploy) or [Dags only](/docs/astro-private-cloud/v-2-x/deploy-dags) using the Astro CLI. Deploying a project image is the only way to deploy Airflow-level configurations and dependencies to a Deployment.
* Deploy Dags using an [NFS volume](/docs/astro-private-cloud/v-2-x/deploy-nfs).
* Deploy Dags using [Git sync](/docs/astro-private-cloud/v-2-x/deploy-git-sync).

### Astro CLI deploys

By default, you can deploy code to an Airflow Deployment by building it into a Docker image and pushing that image to the Astronomer Registry via the CLI or API. This workflow is described in [Deploy code via the CLI](/docs/astro-private-cloud/v-2-x/deploy-code-overview#full-image-deploy).

This mechanism builds your Dags into a Docker image alongside all other files in your Astro project directory, including your Python and OS-level packages, your Dockerfile, and your plugins.

The resulting image is then used to generate a set of Docker containers for each of Airflow's core components. Every time you run `astro deploy` in the Astro CLI, your Dags are rebuilt into a new Docker image and all Docker containers are restarted.

If you have local, pre-built images, you can use the `--image-name` tag to deploy an existing image to the registry. This skips the image build step, but still restarts all Docker containers and deploys your Dags. See [Deploy a project image](/docs/astro-private-cloud/v-2-x/deploy-code-overview#full-image-deploy) for more information.

You can also enable [Dag-only deploys](/docs/astro-private-cloud/v-2-x/deploy-dags) to deploy only your `dags` directory without building a Docker image. Note that you will still need access to Docker to authenticate to APC before you can deploy Dags.

### NFS volume-based Dag deploys

For advanced teams who deploy Dag changes more frequently, APC also supports an [NFS volume-based](https://kubernetes.io/docs/concepts/storage/volumes/#nfs) Dag deploy mechanism.

Using this mechanism, you can deploy Dags to an Airflow Deployment on APC by adding the corresponding Python files to a shared file system on your network. Compared to image-based deploys, NFS volume-based deploys limit downtime and enable continuous deployment.

To deploy Dags to a Deployment using an NFS volume, you must first enable the feature at the platform level. For more information, read [Deploy Dags via NFS volume](/docs/astro-private-cloud/v-2-x/deploy-nfs).

### Git-sync Dag deploys

For teams using a Git-based workflow for Dag development, Astronomer supports a [git-sync](https://github.com/kubernetes/git-sync) deploy mechanism.

To deploy Dags using git-sync, you add Dags to a repository that has been configured to sync with your APC Deployment. Once the Deployment detects a change in the repository, your Dag code will automatically sync to your Deployment with no downtime. For more information on configuring this feature, read [Deploy Dags via git sync](/docs/astro-private-cloud/v-2-x/deploy-git-sync).

## Delete a Deployment

You can delete an Airflow Deployment using the **Delete Deployment** button at the bottom of the Deployment's **Settings** tab.

When you delete a Deployment, you delete your Airflow webserver, scheduler, metadata database, and deploy history, and you lose any configurations set in the Airflow UI.

By default, Astro performs a *soft delete* when you delete a Deployment. After you delete a Deployment, your APC database, the corresponding `Deployment` record receives a `deletedAt` value and continues to persist until permanently deleted through a *hard delete*. A hard delete includes both the Deployment's metadata database and the Deployment entry in your APC database. 14 days after your Deployment's soft delete, APC automatically runs a hard delete cronjob that deletes any values that remained after your soft delete.

<Tip>APC recommends regularly doing a database audit to confirm that you hard delete databases.</Tip>

### Automate Deployment deletion clean up

APC runs a cronjob to hard delete the deleted Deployment's metadata database and Deployment entry in your APC database at midnight on a specified day. You can enable whether this cronjob runs or not, how many days after your soft delete to run the cronjob, and what time of day to run the cronjob by editing `astronomer.houston.cleanupDeployments` in your APC Helm chart.

The following is an example of how you might configure the cronjob in your Helm chart:

```yaml wrap theme={null}
# Cleanup deployments that have been soft-deleted
  # This clean up runs as a CronJob
  cleanupDeployments:
    # Enable the clean up CronJob
    enabled: true

    # Default time for the CronJob to run https://crontab.guru/#0_0_*_*_*
    schedule: "0 0 * * *"

    # Number of days after the Deployment deletion to run the CronJob
    olderThan: 14
```

### Hard delete a Deployment

To reuse a custom release name given to an existing Deployment after a soft delete but before APC automatically cleans up any persisting Deployment records, you need to hard delete both the Deployment's metadata database and the Deployment's entry in your APC database.

Hard delete is enabled by default in APC 2.0 and doesn't require any platform-level configuration. Hard delete a Deployment with the UI or Astro CLI:

* **UI**: Go to the Deployment's **Settings** tab and select **Deprovision**. Then click **Delete Deployment**.

  <Frame>
    <img alt="Hard delete checkbox" />
  </Frame>

* **Astro CLI**: Run `astro deployment delete --hard`.

This action permanently deletes all data associated with a Deployment, including the database and underlying Kubernetes resources.

## Programmatically create or update Deployments

You can programmatically create or update Deployments with all possible configurations using the APC API `upsertDeployment` mutation. See [Create or update a Deployment with configurations](/docs/astro-private-cloud/v-2-x/houston-api).


# Configure External Secrets Operator security
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-external-secrets-operator-security

Set up the default, hardened, and customer-managed isolated External Secrets Operator modes for Astro Private Cloud data plane failover.

This document is the configuration reference and setup runbooks for External Secrets Operator (ESO) security in Astro Private Cloud (APC). Use it after you've chosen a mode in [External Secrets Operator security](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security). For the full Kubernetes manifests these runbooks apply, see [External Secrets Operator security manifests reference](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests).

<Note>
  Mode 1 (the default `ClusterSecretStore` setup) is available in APC 2.0 and later. Modes 2 and 3 and the other options in this document require APC 2.1 or later.
</Note>

The three modes are:

* *Mode 1 — Default shared identity:* unchanged from previous releases. ESO uses its standard chart RBAC. No extra setup.
* *Mode 2 — Hardened shared identity:* you set `external-secrets.rbac.create: false` to disable the ESO sub-chart's default RBAC, then apply Astronomer's minimal `ClusterRole` and `ClusterRoleBinding`.
* *Mode 3 — Customer-managed isolated identity:* namespace pools, with per-namespace RBAC and a per-namespace secret store that you provision.

All examples use AWS Secrets Manager. GCP Secret Manager and Hashicorp Vault (Kubernetes auth) follow the same shape with a different provider block. Throughout, `<release-name>` is the Helm release name you install the Astronomer chart with (for example `astronomer`).

<Note>
  The manifests and chart values in this document assume the platform is installed in the `astronomer` namespace. If you installed it into a different namespace, replace `astronomer` accordingly throughout.
</Note>

## Prerequisites

* An APC data plane cluster with data plane failover enabled (`global.dataPlaneFailover.enabled: true`). Mode 1 is available in APC 2.0 and later; Modes 2 and 3 require APC 2.1 or later.
* ESO custom resource definitions installed on the cluster — either by the chart (default) or by you. See [Install the ESO CRDs yourself](#install-the-eso-crds-yourself).
* Backend authentication ready — either a workload identity (AWS IRSA, GKE Workload Identity, or Vault Kubernetes auth) or a cloud credential pair.
* `kubectl` access to the cluster. For Mode 3, this can be a limited-privilege user.

## Common setup (Modes 1 and 2)

The shared-identity modes need a platform namespace, backend credentials, and one secret store. Full manifests are in the [manifests reference](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests).

<Steps>
  <Step title="Create the platform namespace">
    ```bash theme={null}
    kubectl create namespace astronomer
    ```
  </Step>

  <Step title="Create the credentials secret">
    Create the backend credentials in the `astronomer` namespace. Skip this if you use workload identity.

    ```yaml theme={null}
    apiVersion: v1
    kind: Secret
    metadata:
      name: secrets-backend-credentials
      namespace: astronomer
    type: Opaque
    data:
      access-key: <base64-aws-access-key-id>
      secret-access-key: <base64-aws-secret-access-key>
    ```
  </Step>

  <Step title="Create the secret store">
    Create a cluster-scoped `ClusterSecretStore`. In Mode 2 you can instead create a namespaced `SecretStore` annotated with `astronomer.io/commander-sync` that the platform syncs to all Deployment namespaces. See [Secret store manifests](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#secret-store-manifests-modes-1-and-2).

    <Note>
      If you use custom release names, set `forceDeleteWithoutRecovery: true` on the AWS provider in your store. AWS Secrets Manager soft-deletes secrets by default, with a 30-day recovery window, so reusing a release name for a new Deployment can collide with the still-recoverable secret from the one you deleted. Hard-deleting avoids the collision, at the cost of no recovery window. See [Force-delete secrets in AWS Secrets Manager](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#force-delete-secrets-in-aws-secrets-manager).
    </Note>
  </Step>
</Steps>

## Runbook: Mode 1 — Default shared identity

No ESO-specific configuration is required beyond the common setup for most backends. Install the platform as usual. ESO uses its standard chart-managed RBAC — with ServiceAccount token creation disabled by default — and Deployments authenticate as the shared identity through the secret store you created. If your backend is Hashicorp Vault, re-enable token creation (unpinned) by setting `external-secrets.rbac.serviceAccountTokenCreate: true`, because Vault's Kubernetes auth needs ESO to mint a ServiceAccount token.

```yaml theme={null}
global:
  dataPlaneFailover:
    enabled: true
    externalSecretManagerName: astronomer-secret-store
  plane:
    mode: "data"
external-secrets:
  enabled: true
```

## Runbook: Mode 2 — Hardened shared identity

Start from Mode 1, then create ESO's ServiceAccount and RBAC and disable the chart's ESO RBAC and ServiceAccount creation, so ESO uses the objects you provisioned. Provision the ServiceAccount and RBAC before you install the platform, so ESO starts with them already in place.

<Steps>
  <Step title="Complete the common setup">
    Create the credentials secret and a shared secret store.
  </Step>

  <Step title="Create the ESO ServiceAccount">
    Create the ESO controller ServiceAccount in the `astronomer` namespace, so it exists before ESO starts. See [ESO controller ServiceAccount](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#eso-controller-serviceaccount-modes-2-and-3).

    ```yaml theme={null}
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: <release-name>-external-secrets
      namespace: astronomer
    ```
  </Step>

  <Step title="Apply the provided ESO RBAC manifests">
    Apply the minimal `ClusterRole` and `ClusterRoleBinding` Astronomer provides, before you install the platform. They scope ESO's cluster access and pin ServiceAccount token creation to the ServiceAccount you created in the `astronomer` namespace. See [Mode 2 hardened ESO cluster RBAC](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#mode-2-hardened-eso-cluster-rbac).
  </Step>

  <Step title="Set the platform values">
    Disable the ESO sub-chart's default RBAC and ServiceAccount creation, point ESO at the ServiceAccount you created, and turn off the cluster-scoped secret processing the platform doesn't use.

    ```yaml theme={null}
    global:
      dataPlaneFailover:
        enabled: true
        externalSecretManagerName: astronomer-secret-store
      plane:
        mode: "data"
    external-secrets:
      enabled: true
      processClusterExternalSecret: false
      processClusterPushSecret: false
      rbac:
        create: false
      serviceAccount:
        create: false
        name: <release-name>-external-secrets
    ```

    <Note>
      With `external-secrets.rbac.create: false`, the chart skips its ESO RBAC entirely — including the `serviceAccountTokenCreate` token rule — so the pinned token creation comes from the manifests you applied in the previous step. With `serviceAccount.create: false`, the chart uses the ServiceAccount you created earlier instead of creating its own.
    </Note>
  </Step>

  <Step title="Install or upgrade the platform and verify">
    ```bash theme={null}
    helm upgrade <release-name> astronomer-internal/astronomer \
      --version 2.1.0 -f <platform-values>.yaml
    ```

    Confirm the chart's default ESO RBAC is gone and the applied minimal role is in place. See [Verify the RBAC state](#verify-the-rbac-state).
  </Step>
</Steps>

## Runbook: Mode 3 — Customer-managed isolated identity

The chart creates no cluster roles. You pre-provision the namespaces, per-namespace roles, and a per-namespace secret store, and install with a limited-privilege user. Use this for per-Deployment identity isolation, or when your organization requires you to provision all RBAC yourself. You can substitute your own role definitions for the generated ones.

Mode 3 builds on the [namespace pools](/docs/astro-private-cloud/v-2-x/namespace-pools) feature. Complete the standard namespace pools setup first. The ESO objects described here are in addition to it.

<Steps>
  <Step title="Pre-provision namespaces and the installer identity">
    Create the pool namespaces, then create a limited-privilege installer identity. Generate it with the provided script, or apply the reference installer `Role` from the [manifests reference](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#generate-the-platform-rbac-mode-3):

    ```bash theme={null}
    python bin/generate-namespace-pools-rbac.py \
      --installer-user <installer> \
      --namespaces astronomer | kubectl apply -f -
    ```
  </Step>

  <Step title="Create the ESO controller ServiceAccount">
    Create the ESO controller ServiceAccount `<release-name>-external-secrets` in the `astronomer` namespace, so it exists before ESO starts. The chart doesn't create it, because you set `serviceAccount.create: false` in the platform values (the next step). See [ESO controller ServiceAccount](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#eso-controller-serviceaccount-modes-2-and-3).

    ```yaml theme={null}
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: <release-name>-external-secrets
      namespace: astronomer
    ```
  </Step>

  <Step title="Pre-provision the per-namespace ESO chain">
    In each pool namespace, pre-provision the per-namespace ServiceAccount, namespaced `SecretStore`, ESO reconcile `Role` and `RoleBinding`, and the `resourceNames`-pinned token-creation `Role` and `RoleBinding`. You must also provision the RBAC for the other platform components — Commander, kube-state-metrics, the Houston DB bootstrapper hook, Prometheus, and NGINX — not just ESO. See [Component RBAC for restricted mode](/docs/astro-private-cloud/v-2-x/namespace-pools#component-rbac-for-restricted-mode). Every namespace's `SecretStore` must be named the same value as `global.dataPlaneFailover.externalSecretManagerName`, which you set per cluster when you enable data plane failover; only the name must match across namespaces, and each store's contents can differ.

    Apply the full set from the [manifests reference](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#mode-3-per-namespace-rbac), or generate the platform RBAC with:

    ```bash theme={null}
    python bin/generate-namespace-pools-rbac.py \
      --release-name <release-name> \
      --release-namespace astronomer \
      --namespaces astronomer,airflow-pool-001,airflow-pool-002,airflow-pool-003 | kubectl apply -f -
    ```

    For failover, pre-provision the same chain, with matching namespace names, on every cluster a Deployment can fail over to, and make sure your backend trust spans those clusters.
  </Step>

  <Step title="Set the platform values">
    With the ServiceAccount and RBAC in place, install with cluster roles disabled, namespace pools enabled, and ESO pointed at the ServiceAccount you created.

    ```yaml theme={null}
    global:
      clusterRoles: false
      namespaceManagement:
        namespacePools:
          enabled: true
          createRbac: false
          namespaces:
            create: false
            names:
              - airflow-pool-001
              - airflow-pool-002
              - airflow-pool-003
      dataPlaneFailover:
        enabled: true
        externalSecretManagerName: astronomer-secret-store
      plane:
        mode: "data"
    external-secrets:
      enabled: true
      processClusterExternalSecret: false
      processClusterPushSecret: false
      crd:
        create: true
      serviceAccount:
        create: false
        name: <release-name>-external-secrets
    ```
  </Step>

  <Step title="Install and verify">
    Run the `helm upgrade` with these values, then confirm the RBAC state. See [Verify the RBAC state](#verify-the-rbac-state).
  </Step>
</Steps>

## Install the ESO CRDs yourself

By default the APC chart installs the ESO CRDs. To install them yourself — for example, when a separate infrastructure team owns CRD installation — set the chart value that disables CRD creation and apply the CRDs before installing the platform. This is optional and independent of the mode, and it's most relevant to Modes 2 and 3.

```yaml theme={null}
external-secrets:
  enabled: true
  crd:
    create: false
```

For the published CRD bundle URL and the `kubectl apply` command, see [Install the ESO CRDs yourself](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#install-the-eso-crds-yourself).

## Use Vault as your backend

If your secret backend is Hashicorp Vault, ESO authenticates to Vault with the Kubernetes auth method — ESO presents its ServiceAccount token, and Vault validates it and issues a Vault token. AppRole and token auth aren't supported in 2.1, because they would require storing a static credential in the cluster. Because ESO must mint its own ServiceAccount token to authenticate, Vault backends require ServiceAccount token creation to be enabled: in Mode 1, set `external-secrets.rbac.serviceAccountTokenCreate: true` (unpinned); Modes 2 and 3 grant it through their applied manifests, pinned to the relevant ServiceAccount.

For the full setup, see [Configure Hashicorp Vault for data plane failover](/docs/astro-private-cloud/v-2-x/configure-vault-data-plane-failover): create a Vault policy, enable a Kubernetes auth mount per data plane cluster, create a Vault role, and reference it from your `SecretStore` or `ClusterSecretStore` with a `provider.vault` block that uses `auth.kubernetes`.

The Vault role maps a Kubernetes ServiceAccount to the policy that grants the secret paths: map the shared ESO ServiceAccount in Modes 1 and 2, or the per-namespace ServiceAccount in Mode 3.

## Run ESO with multiple replicas

For extra resiliency, run the ESO controller with more than one replica and enable leader election, so a single replica reconciles secrets at a time while the others stand by. This helps ESO survive Pod failures and restarts, which matters for disaster recovery. It's optional and works with any mode.

```yaml theme={null}
external-secrets:
  replicaCount: 2
  leaderElect: true
```

In Modes 2 and 3, where you manage ESO's RBAC, also create the leader-election `Role` and `RoleBinding` in the `astronomer` namespace. See [Leader-election RBAC](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#leader-election-rbac-optional-eso-high-availability). In Mode 1, the chart creates this RBAC for you.

## Verify the RBAC state

The expected state depends on the mode.

* *Mode 1:* ESO's default chart RBAC is present — a cluster role and binding for external-secrets.
* *Mode 2:* the chart's default ESO RBAC isn't created, and the minimal `ClusterRole` and `ClusterRoleBinding` you applied are present, with token creation pinned to ESO's ServiceAccount.
* *Mode 3:* no cluster-scoped ESO roles exist; only per-namespace roles and bindings.

```bash theme={null}
# ESO cluster-scoped roles: present in Modes 1 and 2, absent (0) in Mode 3
kubectl get clusterrole -o json | jq '[.items[].metadata.name | select(contains("external-secrets"))] | length'
kubectl get clusterrolebinding -o json | jq '[.items[].metadata.name | select(contains("external-secrets"))] | length'

# Per-namespace ESO roles: present in Mode 3
kubectl get role -o json | jq '[.items[].metadata.name | select(contains("external-secrets"))] | length'
kubectl get rolebinding -o json | jq '[.items[].metadata.name | select(contains("external-secrets"))] | length'
```

Confirm the secret store is valid and secrets are syncing:

```bash theme={null}
kubectl get secretstore,pushsecret -n airflow-pool-001
# secretstore ... Valid   ReadWrite   True
# pushsecret  ... Synced
```

## Upgrade from 2.0 to 2.1

* Upgrading to 2.1 requires no changes to keep your current configuration — Mode 1 is unchanged from previous releases.
* Modes 2 and 3 are opt-in. Adopt Mode 2 by setting the ESO chart flags and applying the provided manifests; adopt Mode 3 per cluster through namespace pools.
* If a later ESO version adds a new custom resource type the operator depends on, the Mode 2 minimal `ClusterRole` needs a corresponding update as a step in the data plane upgrade runbook.

## Configuration reference

| Setting                                                       | Default | Effect                                                                                                                                                                                                                                                                        |
| ------------------------------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `external-secrets.enabled`                                    | `false` | Installs the ESO controller through the APC chart.                                                                                                                                                                                                                            |
| `external-secrets.rbac.create`                                | `true`  | Whether the ESO sub-chart creates its default RBAC. Set `false` for Mode 2, then apply the provided minimal manifests.                                                                                                                                                        |
| `external-secrets.rbac.serviceAccountTokenCreate`             | `false` | Whether the ESO sub-chart grants the ServiceAccount token-create rule. Defaults off in 2.1 to close the gap from previous releases. Set `true` (unpinned) for Vault backends in Mode 1; in Mode 2 it's moot because `rbac.create: false` skips the chart's ESO RBAC entirely. |
| `external-secrets.crd.create`                                 | `true`  | Whether the chart installs the ESO CRDs. Set `false` to install them yourself.                                                                                                                                                                                                |
| `external-secrets.serviceAccount.create`                      | `true`  | Whether the ESO sub-chart creates its ServiceAccount. Set `false` in Modes 2 and 3 and create the ServiceAccount yourself.                                                                                                                                                    |
| `external-secrets.serviceAccount.name`                        | —       | The name of the ServiceAccount ESO runs as. Set to `<release-name>-external-secrets` in Modes 2 and 3.                                                                                                                                                                        |
| `external-secrets.processClusterExternalSecret`               | `true`  | Whether ESO reconciles cluster-scoped `ClusterExternalSecret` objects. Set `false` in Modes 2 and 3, which don't use them.                                                                                                                                                    |
| `external-secrets.processClusterPushSecret`                   | `true`  | Whether ESO reconciles cluster-scoped `ClusterPushSecret` objects. Set `false` in Modes 2 and 3, which don't use them.                                                                                                                                                        |
| `external-secrets.replicaCount`                               | `1`     | The number of ESO controller replicas. Set higher for resiliency, together with `leaderElect`.                                                                                                                                                                                |
| `external-secrets.leaderElect`                                | `false` | Enables leader election so only one replica reconciles at a time. Enable it when running multiple replicas.                                                                                                                                                                   |
| `global.clusterRoles`                                         | `true`  | When `false` (Mode 3), the chart creates no cluster-scoped roles and you provision per-namespace roles yourself.                                                                                                                                                              |
| `global.namespaceManagement.namespacePools.enabled`           | `false` | Enables namespace pools. Required for Mode 3.                                                                                                                                                                                                                                 |
| `global.namespaceManagement.namespacePools.createRbac`        | `true`  | Whether the chart generates the per-namespace platform RBAC. Set `false` in Mode 3 to provision it yourself.                                                                                                                                                                  |
| `global.namespaceManagement.namespacePools.namespaces.create` | `true`  | Whether the chart creates the pool namespaces. Set `false` when you pre-create them.                                                                                                                                                                                          |
| `global.namespaceManagement.namespacePools.namespaces.names`  | —       | The list of pool namespace names.                                                                                                                                                                                                                                             |
| `global.dataPlaneFailover.enabled`                            | `false` | Enables data plane failover, which ESO secret sync supports. Required for these features.                                                                                                                                                                                     |
| `global.dataPlaneFailover.externalSecretManagerName`          | —       | The name of the secret store the platform uses when creating `ExternalSecret` and `PushSecret` objects. In Mode 3, every per-namespace `SecretStore` must use this exact name.                                                                                                |

## Related documentation

* [External Secrets Operator security](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security)
* [External Secrets Operator security manifests reference](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests)
* [Configure Hashicorp Vault for data plane failover](/docs/astro-private-cloud/v-2-x/configure-vault-data-plane-failover)
* [Configure a Kubernetes namespace pool](/docs/astro-private-cloud/v-2-x/namespace-pools)


# Configure LDAP authentication
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-ldap-authentication

Enable LDAP authentication on Astro Private Cloud and connect Houston to your Active Directory or OpenLDAP server.

Astro Private Cloud can authenticate users directly against an LDAP directory server, so people who already have accounts in your Active Directory (AD) or OpenLDAP directory can sign in without an intermediate identity provider. When you enable LDAP, Houston binds to the directory on each sign-in, verifies the user's credentials, provisions the account on first sign-in, and optionally maps LDAP groups to Astro Private Cloud Teams and system roles.

Choose LDAP when your organization runs an on-premises AD or OpenLDAP directory that can't or shouldn't be exposed through an OIDC bridge, and when you want the simplest possible path from a directory account to a working Astro Private Cloud user.

<Note>
  **Astro Private Cloud 2.1**

  This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
</Note>

## Overview

LDAP authentication in Astro Private Cloud gives you:

* Username and password sign-in that goes directly to your directory. Houston never stores or caches the password.
* Group-to-team reconciliation. Users' directory groups can be turned into Astro Private Cloud Teams so team membership tracks the directory.
* Group-to-system-role assignment. Directory groups can grant `SYSTEM_ADMIN`, `SYSTEM_EDITOR`, or `SYSTEM_VIEWER` roles on Astro Private Cloud.
* Multiple group-resolution strategies to match how your directory represents membership. Houston supports four modes — direct `memberOf`, AD nested groups, sub-scoped search, and client-side recursive walk.

Supported directories: Active Directory Domain Services (AD DS), Microsoft Entra Domain Services, and OpenLDAP. Any RFC 4511 LDAP server works for basic authentication; nested-group behavior depends on which resolution mode the server supports.

<Info>The Astro CLI doesn't accept LDAP credentials through its username and password prompt. LDAP users obtain an OAuth token by signing in to the Astro Private Cloud UI and paste it into `astro login`. See [Sign in to the Astro CLI](/docs/astro-private-cloud/v-2-x/log-in-to-private-cloud#sign-in-to-the-astro-cli) for the exact CLI flow.</Info>

## Prerequisites

Before you enable LDAP, confirm:

* The LDAP or LDAPS host is reachable from the Astro Private Cloud control plane. Houston makes outbound TCP connections to the host and port defined by `auth.ldap.host` and `auth.ldap.port`.
* You have a service account in the directory with permission to search under the configured `searchBase`. Houston uses this account for the initial bind (the "service bind"). It doesn't need to modify entries — search-only permission is enough.
* You know the base distinguished name (DN) of your directory and the DN structure for users and groups. You use these when you set `bindDn`, `searchBase`, and the `groups.*` fields.
* You have write access to the Astro Private Cloud `values.yaml` file and can apply Helm-values changes. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

## Configure the APC API through Helm values

Add an `auth.ldap` block to your `astronomer.houston.config` section in `values.yaml`. The following example shows every field you can set, with production-safe defaults. Field-by-field details for TLS, attribute mapping, group resolution, team reconciliation, and system-role assignment follow in later sections.

```yaml wrap theme={null}
astronomer:
  houston:
    # Uncomment this block to source bindCredentials from a Kubernetes Secret
    # (recommended for production). See the "Secure bindCredentials with a
    # Kubernetes Secret" section of this document.
    # secret:
    #   - envName: "AUTH__LDAP__BIND_CREDENTIALS"
    #     secretName: "houston-ldap-bind"
    #     secretKey: "password"
    config:
      auth:
        ldap:
          enabled: true
          host: ldap.corp.example.com
          # port is optional. If unset, Houston derives it from tls.mode
          # (636 for ldaps, 389 for none and starttls).
          # port: 636
          tls:
            mode: ldaps          # one of: none, starttls, ldaps
            verifyServerCert: true
          bindDn: "cn=houston-svc,ou=svc,dc=corp,dc=example,dc=com"
          # For dev and test only. In production, source this from a Kubernetes
          # Secret through the astronomer.houston.secret block above.
          bindCredentials: "plaintext-only-for-dev"
          searchBase: "ou=users,dc=corp,dc=example,dc=com"
          searchFilter: "(uid={{username}})"
          attributes:
            email: mail
            name: cn
          groups:
            enabled: true
            reconcileTeams: true
            nestedGroups: false          # or "ad", "recursive", "search"
            searchBase: "ou=users,dc=corp,dc=example,dc=com"
            searchFilter: "(member={{dn}})"
            nameAttribute: cn
            maxDepth: 10
            teamFilterRegex: ""
            manageSystemPermissions:
              enabled: false
              systemAdmin: []
              systemEditor: []
              systemViewer: []
```

After you save `values.yaml`, apply the change to your platform through the standard Helm upgrade flow. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

## Secure `bindCredentials` with a Kubernetes Secret

The `bindCredentials` field accepts an inline value, but that inline value ends up on disk in `values.yaml` and in Helm release history. For production installs, source it from a Kubernetes Secret instead.

Astro Private Cloud uses the same `astronomer.houston.secret[]` mechanism that the OIDC guide uses for `clientSecret` and that the install guide uses for `EMAIL__SMTP_URL`. Each entry in the list becomes a `valueFrom.secretKeyRef` environment variable on the Houston pod, and Houston reads the LDAP bind password from that environment variable rather than from the config file.

<Steps>
  <Step title="Create a Kubernetes secret">
    ```bash wrap theme={null}
    kubectl create secret generic houston-ldap-bind \
      --from-literal=password='<the-bind-password>' \
      -n astronomer
    ```

    Replace `<the-bind-password>` with the password for the service account named in `bindDn`.
  </Step>

  <Step title="Reference the secret from values.yaml">
    Add the `secret:` block under `astronomer.houston` and remove the inline `bindCredentials` line:

    ```yaml wrap theme={null}
    astronomer:
      houston:
        secret:
          - envName: "AUTH__LDAP__BIND_CREDENTIALS"
            secretName: "houston-ldap-bind"
            secretKey: "password"
        config:
          auth:
            ldap:
              enabled: true
              host: ldap.corp.example.com
              bindDn: "cn=houston-svc,ou=svc,dc=corp,dc=example,dc=com"
              # bindCredentials is now sourced from AUTH__LDAP__BIND_CREDENTIALS,
              # which the Secret block above wires up. Omit the inline value.
              searchBase: "ou=users,dc=corp,dc=example,dc=com"
              searchFilter: "(uid={{username}})"
    ```

    The environment variable takes precedence when both an inline value and a `secretKeyRef` are present. `secretKey` is optional and defaults to `value`. Apply the change with `helm upgrade` and Houston restarts with the new environment variable bound.
  </Step>
</Steps>

<Tip>The same `astronomer.houston.secret[]` block already carries OIDC client secrets and the SMTP connection string. If you already use one of those, add the LDAP entry as another list item rather than creating a second block.</Tip>

## Configure TLS

LDAP transports credentials over the network. Any deployment outside a fully isolated test environment must use TLS. Houston supports three transport modes through `auth.ldap.tls.mode`.

| Mode       | Wire behavior                                                                              | Default port | When to use                                                                       |
| ---------- | ------------------------------------------------------------------------------------------ | ------------ | --------------------------------------------------------------------------------- |
| `none`     | Plain LDAP on 389, no encryption at any point.                                             | 389          | Isolated test environments only. Don't use in production.                         |
| `starttls` | Plain TCP on 389, upgraded to TLS through the StartTLS extended operation before any bind. | 389          | AD deployments that only expose 389 and require encryption before authentication. |
| `ldaps`    | TLS-wrapped from the first byte on 636.                                                    | 636          | Any deployment where 636 is reachable. Simplest, most common.                     |

Set `auth.ldap.tls.verifyServerCert: true` (the default) to have Houston validate the LDAP server's certificate chain against its trusted CAs. This applies to both `starttls` and `ldaps` modes. Set it to `false` only for dev environments where the server presents a self-signed certificate that isn't in Houston's trust store.

If your LDAP server presents a certificate signed by a private certificate authority, add the CA certificate to Houston's trust store through the platform configuration and keep `verifyServerCert: true`. See [Configure private CAs](/docs/astro-private-cloud/v-2-x/configure-private-cas).

`auth.ldap.port` is optional. When unset, Houston derives it from `tls.mode`: 636 for `ldaps`, 389 for `none` and `starttls`. Set the field explicitly only when your directory listens on a non-standard port.

## Attribute mapping

Houston reads two attributes from each user's LDAP entry to build the corresponding Astro Private Cloud user account:

* `attributes.email` (default `mail`) — the value used as the user's email address in Astro Private Cloud. Houston converts the value to lowercase before it stores the user. If the attribute is multi-valued in your directory, Houston takes the first value.
* `attributes.name` (default `cn`) — the value used as the user's full name in Astro Private Cloud.

For AD deployments, `displayName` produces a friendlier full name than `cn`:

```yaml wrap theme={null}
attributes:
  email: mail
  name: displayName
```

### Group name attribute

By default, when Houston resolves groups through the direct `memberOf` mode, it takes the group's name from the leftmost relative distinguished name (RDN) of the group's DN. For example, a `memberOf: cn=engineering,ou=groups,dc=example,dc=com` value produces a group name of `engineering`.

To use a different attribute for the group name — such as `description` — set `groups.nameAttribute`:

```yaml wrap theme={null}
groups:
  enabled: true
  nestedGroups: false
  nameAttribute: description
```

When `nameAttribute` is anything other than `cn`, Houston fetches each group's entry individually to read the configured attribute. This is one additional LDAP query per group, so the direct-`memberOf` mode is slower with a custom `nameAttribute`. The default `nameAttribute: cn` skips the per-group fetch and stays on the fast path.

## Configure group resolution

If you enable `groups.enabled: true`, Houston resolves the user's directory groups on each sign-in. Houston supports four resolution strategies through `groups.nestedGroups`.

| Mode              | `nestedGroups`    | How Houston finds groups                                                                                                                                   | Directory requirement                                                    | Cost per sign-in                                                  |
| ----------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------- |
| Direct `memberOf` | `false` (default) | Reads the `memberOf` attribute on the user's entry.                                                                                                        | AD, or OpenLDAP with the `memberof` overlay populated.                   | One round-trip, plus one per group if `nameAttribute` isn't `cn`. |
| Nested groups     | `"ad"`            | Runs a single sub-scoped search under `groups.searchBase` using AD's `LDAP_MATCHING_RULE_IN_CHAIN` operator (object identifier `1.2.840.113556.1.4.1941`). | Active Directory. The rule is AD-specific.                               | One round-trip regardless of nesting depth.                       |
| Search            | `"search"`        | Runs a sub-scoped search under `groups.searchBase` using `groups.searchFilter`, substituting the user's DN into every `{{dn}}` placeholder.                | Any RFC 4511 LDAP server.                                                | One round-trip.                                                   |
| Recursive         | `"recursive"`     | Walks the user's `memberOf` attribute, then each group's `memberOf`, up to `groups.maxDepth`.                                                              | Non-AD directory where `memberOf` is populated on both users and groups. | One round-trip per level, up to `maxDepth`.                       |

Choose the mode that matches how your directory represents nested membership. Every mode returns the same shape of result — a flat list of group names — so downstream reconciliation and role assignment don't depend on which mode you pick.

### Direct `memberOf` mode

The default. Use for Active Directory (where `memberOf` is native) or for OpenLDAP with the `memberof` overlay populated.

```yaml wrap theme={null}
groups:
  enabled: true
  reconcileTeams: true
  nestedGroups: false
```

Verify that `memberOf` is populated on your user entry:

```bash wrap theme={null}
ldapsearch -x -H ldap://ldap.corp.example.com -D "cn=houston-svc,..." -w "..." \
  -b "ou=users,dc=corp,dc=example,dc=com" \
  "(uid=alice)" memberOf
```

The response must include one `memberOf:` line per group the user belongs to. If it's missing, either enable the `memberof` overlay on your directory or switch to `nestedGroups: "search"`.

### Nested groups mode

Use for Active Directory when you need transitive group membership — for example, when your users are direct members of a role group that is nested inside a broader access group and you want Houston to see both.

```yaml wrap theme={null}
groups:
  enabled: true
  reconcileTeams: true
  nestedGroups: "ad"
  searchBase: "cn=Users,dc=corp,dc=example,dc=com"
```

This mode requires `groups.searchBase`. Houston constructs the transitive query internally; you don't need to configure the matching rule OID.

Verify with `ldapsearch`:

```bash wrap theme={null}
ldapsearch -x -H ldaps://ad.corp.example.com:636 -D "cn=houston-svc,..." -w "..." \
  -b "cn=Users,dc=corp,dc=example,dc=com" \
  "(member:1.2.840.113556.1.4.1941:=CN=alice,CN=Users,DC=corp,DC=example,DC=com)" cn
```

The response includes every group the user reaches through direct or nested membership.

### Search mode

Use when your directory doesn't populate `memberOf` but does index group `member` entries.

```yaml wrap theme={null}
groups:
  enabled: true
  reconcileTeams: true
  nestedGroups: "search"
  searchBase: "ou=groups,dc=corp,dc=example,dc=com"
  searchFilter: "(member={{dn}})"
```

`groups.searchBase` and `groups.searchFilter` are both required. Houston substitutes `{{dn}}` with the signed-in user's DN before it runs the search. Multiple `{{dn}}` placeholders are supported — for example, to match either `member` or `uniqueMember`:

```yaml wrap theme={null}
searchFilter: "(|(member={{dn}})(uniqueMember={{dn}}))"
```

Houston escapes the user's DN according to [RFC 4515](https://datatracker.ietf.org/doc/html/rfc4515) before substitution. DN values that contain filter special characters (`*`, `(`, `)`, `\`, `NUL`) are handled safely.

### Recursive mode

Use for non-AD directories where `memberOf` is populated on user entries and on group entries, and where you need transitive group traversal.

```yaml wrap theme={null}
groups:
  enabled: true
  reconcileTeams: true
  nestedGroups: "recursive"
  maxDepth: 10
```

`groups.maxDepth` caps the walk. Houston stops walking once it reaches the configured depth, even if more parent groups exist above that point. The default of `10` is enough for most nesting patterns.

Recursive mode starts from the user's `memberOf` attribute and walks up the group hierarchy. If your OpenLDAP directory doesn't populate `memberOf`, enable the `memberof` overlay on the directory server, or switch to `nestedGroups: "search"`, which reads group membership from group entries instead.

## Reconcile groups into Astro Private Cloud Teams

When `groups.reconcileTeams: true`, every resolved group becomes an Astro Private Cloud Team, and the signed-in user is added to the corresponding team.

```yaml wrap theme={null}
groups:
  enabled: true
  reconcileTeams: true
```

Teams that Houston creates through LDAP reconciliation are tagged with `provider='ldap'` in the Houston database. They don't collide with Teams from OIDC providers or Teams created manually — the two provider spaces are independent. For the extended cross-provider details, see [Import identity provider (IdP) groups](/docs/astro-private-cloud/v-2-x/import-idp-groups).

### Filter which groups become Teams

`groups.teamFilterRegex` restricts which directory groups become Teams. Only groups whose name matches the regular expression are reconciled; the rest are ignored.

```yaml wrap theme={null}
groups:
  reconcileTeams: true
  teamFilterRegex: "^astro-"
```

The regex is JavaScript syntax and is applied case-sensitively. It runs after group resolution completes, so it filters the exact names Houston sees — for direct-`memberOf` mode, those are the CN components of each `memberOf` DN. Only LDAP groups are affected; OIDC has its own separate filter.

## Assign system roles from directory groups

Houston can grant `SYSTEM_ADMIN`, `SYSTEM_EDITOR`, or `SYSTEM_VIEWER` platform roles to Teams that come from specific directory groups.

```yaml wrap theme={null}
groups:
  enabled: true
  reconcileTeams: true
  manageSystemPermissions:
    enabled: true
    systemAdmin: ["astro-platform-admins"]
    systemEditor: ["astro-workspace-editors"]
    systemViewer: ["astro-observers"]
```

Each list holds group names that map to that system role. Houston normalizes DN values in these lists to their CN components before matching, so you can list either short names (`astro-platform-admins`) or full DN values (`cn=astro-platform-admins,ou=groups,dc=corp,dc=example,dc=com`).

Priority is `SYSTEM_ADMIN` > `SYSTEM_EDITOR` > `SYSTEM_VIEWER`. If the same group appears in more than one list, the highest role wins. Roles apply to the Team that Houston creates from the LDAP group — individual users inherit the role through their team membership rather than getting a direct role binding.

Houston reconciles system roles on every sign-in. When a user is removed from a role-mapping group in the directory, they lose the corresponding system role the next time they sign in.

<Warning>
  Setting `manageSystemPermissions.enabled: false` stops future synchronization from directory groups but doesn't revoke roles that were previously auto-assigned. Existing system role assignments remain in place until you remove them manually in the Astro Private Cloud UI, in **Settings** > **Teams**. The OIDC equivalent (`auth.openidConnect.manageSystemPermissionsViaIdpGroups.enabled`) behaves the same way.
</Warning>

## Security guards

Houston enforces the following guards on every LDAP sign-in.

### Service and user bind are separate

Houston uses `bindDn` and `bindCredentials` only for the search that resolves the user's DN. The actual authentication is a second bind that Houston performs with the user's own DN and the password they submitted. The service account never authenticates users, and user passwords never travel outside the sign-in request.

### LDAP doesn't auto-link to existing accounts

If a user with the same email already exists on the platform through local auth or an OIDC provider, LDAP sign-in for that email is rejected. Linking an existing account to LDAP requires an explicit `OAuthCredential(provider='ldap', ...)` row. See [Sign-in migration paths](#sign-in-migration-paths).

### Deactivated users can't sign in

Only Astro Private Cloud users in the `ACTIVE` or `PENDING` status can complete an LDAP sign-in. Deactivating a user through Houston blocks their LDAP sign-in even if their directory account remains active.

### Last-admin protection

Houston prevents removing the last user with the `SYSTEM_ADMIN` role. This applies to system-role reconciliation as well as manual role changes.

### Error messages don't distinguish failure modes

A failed bind produces a generic "Invalid username or password" message regardless of whether the user doesn't exist, the password is wrong, or the account is deactivated. This prevents an attacker from probing the directory for valid usernames.

### Passwords are never stored or cached

Houston uses the user's password only during the sign-in bind. It doesn't persist the password to the database, log it, or hold it in memory beyond the request.

### Group-resolution failure isn't sign-in failure

If group resolution returns an error or no groups, Houston still signs the user in. Their teams and system roles aren't updated, but they can use the platform with whatever team and role state they already have. See [Diagnose configuration issues](#diagnose-configuration-issues) for the log lines that surface this case.

### LDAP filters escape user DN values

When Houston builds a filter that includes the user's DN — for example, `(member={{dn}})` in search mode — it escapes filter special characters (`*`, `(`, `)`, `\`, `NUL`) to their `\XX` hex-pair form according to RFC 4515. A crafted DN can't alter the semantics of the query.

## Diagnose configuration issues

Houston emits warn-level log lines that surface two silent-failure classes operators otherwise miss.

### Invalid `nestedGroups` value

If `groups.nestedGroups` holds anything other than `false`, `"ad"`, `"recursive"`, or `"search"` — for example the boolean `true`, an integer, or a typo like `"recursvie"` — Houston logs a warn identifying the invalid value and falls back to direct-`memberOf` mode. Example:

```text wrap theme={null}
Invalid auth.ldap.groups.nestedGroups value: true. Valid values are false, "ad", "recursive", "search". Falling back to direct-memberOf mode.
```

Sign-in continues to work under the fallback, but the mode you configured is silently overridden. Search the Houston pod logs for `Invalid auth.ldap.groups.nestedGroups` after a config change to confirm your value was accepted.

### Empty group resolution when reconciliation is expected

If you set `groups.reconcileTeams: true` or `groups.manageSystemPermissions.enabled: true` but group resolution returned zero groups for the signed-in user, Houston logs a warn that includes the user's DN and a mode-specific mitigation hint. Example in direct-`memberOf` mode:

```text wrap theme={null}
LDAP group resolution returned no groups for user cn=alice,ou=users,dc=corp,dc=example,dc=com — configured team reconciliation / role assignment cannot proceed. nestedGroups=false; check that memberOf is populated on the user's LDAP entry, or set nestedGroups='recursive', 'ad', or 'search' depending on your directory.
```

The hint text is mode-aware. In search mode, the hint suggests checking `searchBase` and `searchFilter`. In recursive mode, the hint points at the `memberof` overlay.

Filter the Houston pod logs for `LDAP group resolution returned no groups` when a user reports that their teams or system roles aren't being applied.

## Sign-in migration paths

When both local auth and LDAP are enabled on the same platform, Astro Private Cloud user accounts aren't automatically linked across providers. This is intentional — the account-takeover guard requires an explicit `OAuthCredential(provider='ldap', ...)` row before an LDAP sign-in can attach to an existing user. Choose one of three operator-driven paths to migrate.

### Backfill

Recommended for platforms with a working user base you want to preserve. For each existing user who is moving to LDAP, insert an `OAuthCredential(provider='ldap', oauthUserId=<identity>)` row where `<identity>` is the value that matches `auth.ldap.searchFilter` for that user (typically their `mail` or `uid`).

Backfill preserves `User.id`, existing team memberships, role bindings, and the audit trail across the migration. Users notice no change other than the sign-in flow itself.

### Re-create

Clean-slate path. The Astro Private Cloud administrator deletes existing `User` rows for the affected users; the deletion cascades through `Email`, `OAuthCredential`, `RoleBinding`, and `_TeamToUser` in the Houston database. Users then sign in through LDAP as new accounts.

Re-creation is simpler operationally but every affected user receives a new `User.id`, and existing audit entries in your security information and event management (SIEM) system continue to reference the old ID. Choose this path only if audit continuity across the migration isn't a requirement.

### Coexistence

Keep both providers active with no migration. Each user keeps whichever credential they were created with — local users continue to sign in with their email and password, LDAP users sign in through the directory. Astro Private Cloud doesn't attempt to unify accounts even when the emails match.

<Info>A self-serve, in-platform "link my LDAP account to this Astro Private Cloud user" flow isn't part of Astro Private Cloud 2.1.0. The three paths in the preceding section are the supported options.</Info>

## Verify the setup

After you apply the LDAP configuration and Houston restarts, verify end-to-end sign-in.

<Steps>
  <Step title="Sign in through the UI">
    Open the Astro Private Cloud UI and sign in with a directory user's credentials. Successful sign-in indicates that Houston can reach the directory, the service bind succeeded, the user search filter matched, and the user's bind succeeded.
  </Step>

  <Step title="Confirm the database state">
    For a successful first sign-in, Houston writes:

    * One row in `User` with `status = 'active'`
    * One row in `Email` for the user's directory email
    * One row in `OAuthCredential` with `provider = 'ldap'`
    * One row per resolved LDAP group in `Team` with `provider = 'ldap'` (when you enable `reconcileTeams`)
    * One `RoleBinding` per Team-to-system-role mapping (when you enable `manageSystemPermissions`)

    You can confirm from a Houston pod with `psql`:

    ```sql wrap theme={null}
    SELECT u.id, u.username, u."fullName", u.status, oc.provider
    FROM "User" u
    JOIN "OAuthCredential" oc ON oc."userId" = u.id
    WHERE oc.provider = 'ldap'
      AND u.username = 'alice@corp.example.com';
    ```
  </Step>

  <Step title="Verify the TLS mode on the wire">
    The LDAP server logs show which TLS transport Houston used:

    * `mode: none` — plain `BIND` lines with `ssf=0`.
    * `mode: starttls` — an `EXT oid=1.3.6.1.4.1.1466.20037` line and `TLS established tls_ssf=N` before the bind.
    * `mode: ldaps` — `TLS established tls_ssf=N` immediately after the `ACCEPT` on port 636.

    If the server-side log shows `ssf=0` when you expected TLS, review the `tls.mode` value and confirm the client and server ports agree.
  </Step>
</Steps>

## Audit logging

Every LDAP sign-in attempt produces an audit record with `action: auth.login`, identical in shape to sign-ins through local auth and OIDC. Existing SIEM rules that filter on `auth.login` catch LDAP sign-ins uniformly.

The record's `entity.identity` field holds the LDAP username submitted at sign-in, and the password field is redacted by the platform's existing redaction rules. Failed binds produce records with `outcome: failure` and the generic sign-in error message. Underlying cert or network errors go to the Houston pod log, not the audit record.

## Limitations

* The Astro CLI doesn't accept LDAP credentials through its username and password prompt. LDAP users authenticate to the CLI by signing in to the Astro Private Cloud UI, retrieving an OAuth token, and pasting it into `astro login`. See [Sign in to the Astro CLI](/docs/astro-private-cloud/v-2-x/log-in-to-private-cloud#sign-in-to-the-astro-cli) for the token flow.
* OIDC and LDAP can coexist, but accounts aren't automatically linked between providers even when the emails match. See [Sign-in migration paths](#sign-in-migration-paths).
* Airflow UI authentication is unchanged. LDAP configuration on Astro Private Cloud doesn't affect the Airflow security model. Airflow continues to use its Flask AppBuilder (FAB) authentication.
* A self-serve account-linking UI isn't available. Linking an existing Astro Private Cloud user to their LDAP identity requires the Backfill path described earlier.

## Reference: `auth.ldap` fields

| Field                                         | Environment variable                                           | Default                 | Description                                                                                                         |
| --------------------------------------------- | -------------------------------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `enabled`                                     | `AUTH__LDAP__ENABLED`                                          | `false`                 | Turn LDAP sign-in on or off.                                                                                        |
| `host`                                        | `AUTH__LDAP__HOST`                                             | —                       | LDAP or LDAPS server hostname.                                                                                      |
| `port`                                        | `AUTH__LDAP__PORT`                                             | derived from `tls.mode` | TCP port. When unset, `636` for `ldaps`, `389` otherwise.                                                           |
| `tls.mode`                                    | `AUTH__LDAP__TLS__MODE`                                        | `none`                  | One of `none`, `starttls`, `ldaps`.                                                                                 |
| `tls.verifyServerCert`                        | `AUTH__LDAP__TLS__VERIFY_SERVER_CERT`                          | `true`                  | Validate the server's TLS certificate chain when TLS is in use.                                                     |
| `bindDn`                                      | `AUTH__LDAP__BIND_DN`                                          | —                       | Service-account DN used for the user-search bind.                                                                   |
| `bindCredentials`                             | `AUTH__LDAP__BIND_CREDENTIALS`                                 | —                       | Service-account password. Source from a Kubernetes Secret in production.                                            |
| `searchBase`                                  | `AUTH__LDAP__SEARCH_BASE`                                      | —                       | Base DN under which Houston searches for user entries.                                                              |
| `searchFilter`                                | `AUTH__LDAP__SEARCH_FILTER`                                    | `(uid={{username}})`    | Filter with `{{username}}` substituted from the sign-in form.                                                       |
| `attributes.email`                            | `AUTH__LDAP__ATTRIBUTES__EMAIL`                                | `mail`                  | LDAP attribute Houston reads for the user's email.                                                                  |
| `attributes.name`                             | `AUTH__LDAP__ATTRIBUTES__NAME`                                 | `cn`                    | LDAP attribute Houston reads for the user's full name.                                                              |
| `groups.enabled`                              | `AUTH__LDAP__GROUPS__ENABLED`                                  | `false`                 | Turn group resolution on. Required for `reconcileTeams` and `manageSystemPermissions`.                              |
| `groups.reconcileTeams`                       | `AUTH__LDAP__GROUPS__RECONCILE_TEAMS`                          | `false`                 | Turn resolved LDAP groups into Astro Private Cloud Teams tagged `provider='ldap'`.                                  |
| `groups.nestedGroups`                         | `AUTH__LDAP__GROUPS__NESTED_GROUPS`                            | `false`                 | Group-resolution mode. One of `false`, `"ad"`, `"recursive"`, `"search"`.                                           |
| `groups.searchBase`                           | `AUTH__LDAP__GROUPS__SEARCH_BASE`                              | —                       | Base DN for group searches. Required for `"ad"` and `"search"` modes.                                               |
| `groups.searchFilter`                         | `AUTH__LDAP__GROUPS__SEARCH_FILTER`                            | `(member={{dn}})`       | Filter used in `"search"` mode. `{{dn}}` is RFC 4515-escaped before substitution.                                   |
| `groups.nameAttribute`                        | `AUTH__LDAP__GROUPS__NAME_ATTRIBUTE`                           | `cn`                    | Attribute Houston reads for each group's name. Non-`cn` values trigger a per-group fetch in direct-`memberOf` mode. |
| `groups.maxDepth`                             | `AUTH__LDAP__GROUPS__MAX_DEPTH`                                | `10`                    | Recursion depth cap for `"recursive"` mode.                                                                         |
| `groups.teamFilterRegex`                      | `AUTH__LDAP__GROUPS__TEAM_FILTER_REGEX`                        | —                       | Regex applied to group names; only matches become Teams.                                                            |
| `groups.manageSystemPermissions.enabled`      | `AUTH__LDAP__GROUPS__MANAGE_SYSTEM_PERMISSIONS__ENABLED`       | `false`                 | Turn group-to-system-role mapping on.                                                                               |
| `groups.manageSystemPermissions.systemAdmin`  | `AUTH__LDAP__GROUPS__MANAGE_SYSTEM_PERMISSIONS__SYSTEM_ADMIN`  | `[]`                    | Group names that grant `SYSTEM_ADMIN` on Astro Private Cloud.                                                       |
| `groups.manageSystemPermissions.systemEditor` | `AUTH__LDAP__GROUPS__MANAGE_SYSTEM_PERMISSIONS__SYSTEM_EDITOR` | `[]`                    | Group names that grant `SYSTEM_EDITOR`.                                                                             |
| `groups.manageSystemPermissions.systemViewer` | `AUTH__LDAP__GROUPS__MANAGE_SYSTEM_PERMISSIONS__SYSTEM_VIEWER` | `[]`                    | Group names that grant `SYSTEM_VIEWER`.                                                                             |


# Configure per-deployment migration
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-per-deployment-migration

Set up per-deployment migration on Astro Private Cloud by assigning regions to your failover-enabled data plane clusters.

Per-deployment migration moves one or more individual Astro Deployments from one data plane cluster to another, without failing over the whole cluster. It is available in Astro Private Cloud (APC) 2.1 and later.

Per-deployment migration is an extension of [data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover). It has no separate license, feature flag, or Helm value of its own: if you enabled data plane failover on your clusters, per-deployment migration is enabled too. Where the 2.0 feature only failed over an entire cluster at once, per-deployment migration lets you move a chosen subset of Deployments while everything else stays in place.

Two things typically drive its use:

* Rebalancing: move a single busy or unhealthy Deployment onto a neighboring cluster without disturbing anything else on either cluster.
* Per-team failover drills: when different teams own different Deployments on the same cluster, cluster-level failover forces every team into one coordinated window. Per-deployment migration lets each team validate failover on their own Deployments, on their own schedule.

Under the hood, a migration is the same machinery as a cluster failover: one mission per Deployment, moving through the same flight sequence. What differs is the entry point — you pick the Deployments — and one hard constraint.

<Warning>
  Migration is in-region only. A Deployment can only be migrated to a cluster in the same region as its current cluster, and this is enforced, not advisory.

  Per-deployment migration moves the Deployment, not its Airflow metadata database. A Deployment moved across a region boundary keeps talking to its metadata database in the original region, which adds cross-region database traffic on every scheduler tick. The latency cost is permanent and produces no error: the Deployment comes up healthy and simply runs slowly. To move workloads across regions, use full-cluster [data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover), which is sequenced with your database promotion.
</Warning>

Enforcing that constraint is what most of the configuration work is about. The migration gate compares the region assigned to the source cluster against the region assigned to the destination cluster, so before you can migrate anything, every participating cluster needs a real region assigned to it. That region model is new in 2.1 and is the main configuration task.

This guide covers the setup. For day-2 operations, see [Manage and observe per-deployment migration](/docs/astro-private-cloud/v-2-x/manage-per-deployment-migration). For the reference tables this guide links to — cluster failover states, configuration settings, and permissions — see [Per-deployment migration reference](/docs/astro-private-cloud/v-2-x/per-deployment-migration-reference).

## Prerequisites

Per-deployment migration inherits every prerequisite of data plane failover. None of them are new in 2.1: if you already run data plane failover, you already have them. Confirm the following before you configure regions.

| Prerequisite                                                                                                                                                                                                                                                                                       | Why migration needs it                                                                                                                                  |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Data plane failover enabled on both the source and destination clusters. See [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover).                                                                                                                                  | Migration reuses the failover mission and flight machinery end to end.                                                                                  |
| A shared Airflow metadata database instance, reachable from both clusters, with a stable write endpoint. See [Database requirements](/docs/astro-private-cloud/v-2-x/data-plane-failover#database-requirements).                                                                                        | The migrated Deployment keeps its existing database, so both clusters must reach it. This is why the move must stay in-region.                          |
| Per-Deployment, per-data-plane database users. See [Per-Deployment database users](/docs/astro-private-cloud/v-2-x/data-plane-failover#per-deployment-database-users).                                                                                                                                  | The destination flight fences the source database user and enables the destination one, guaranteeing at most one Airflow writer.                        |
| The External Secrets Operator (ESO) installed on every data plane cluster, with a `ClusterSecretStore` or `SecretStore` and a secrets backend both clusters can read. See [Configure External Secrets Operator security](/docs/astro-private-cloud/v-2-x/configure-external-secrets-operator-security). | Deployment secrets are pushed to the backend on creation and pulled onto the destination during the move.                                               |
| A container registry both clusters can pull from. See [Container registry requirements](/docs/astro-private-cloud/v-2-x/data-plane-failover#container-registry-requirements).                                                                                                                           | If the Deployment's image isn't available to the destination cluster, it can't start there.                                                             |
| Remote logging to object storage, with a stable log folder. See [Airflow log sink requirements](/docs/astro-private-cloud/v-2-x/data-plane-failover#airflow-log-sink-requirements).                                                                                                                     | Task logs must survive the move.                                                                                                                        |
| Deployments deployed by image (deploy revision type `image`).                                                                                                                                                                                                                                      | Git-sync, Dag-only, and in-cluster-registry Deployments aren't failover-compatible and are skipped.                                                     |
| Deployments upgraded for failover.                                                                                                                                                                                                                                                                 | Deployments created before the failover prerequisites existed must be retrofitted first. Un-upgraded Deployments are skipped with `NOT_FAILOVER_READY`. |

## Step 1: Confirm data plane failover is enabled

Per-deployment migration ships with data plane failover, so there is no additional switch. On each participating data plane cluster, your APC Helm values must contain:

```yaml theme={null}
external-secrets:
  # Installs the ESO CRDs that data plane failover requires
  enabled: true

global:
  dataPlaneFailover:
    enabled: true
    # Name of the secret store you created; must be identical on every cluster
    externalSecretManagerName: astronomer-secret-store
```

On the control plane:

```yaml theme={null}
global:
  dataPlaneFailover:
    enabled: true
```

Data plane failover requires split mode — a separate control plane (`global.plane.mode: control`) and data plane (`global.plane.mode: data`). It isn't supported in unified mode (`global.plane.mode: unified`). `global.dataPlaneFailover.enabled` is the primary switch, and what it turns on depends on the plane's mode:

* On a data plane, it enables the data plane execution components (Pilot and Flightdeck) and the data plane internal API's `StartFlight` remote procedure call.
* On a control plane, it enables Navigator and the APC API dispatcher.

For the full enablement procedure, see [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover).

<Warning>
  Enabling the Helm value is necessary but not sufficient. Setting `dataPlaneFailover.enabled` makes a cluster *failover-capable*. The control plane then independently evaluates whether the cluster is actually *failover-enabled* by checking ESO configuration, Deployment types, and whether you upgraded every Deployment for failover. A cluster can stay capable-but-not-enabled indefinitely.

  Check the **Failover** field on the cluster detail page, or query `Cluster.failoverEnabled`, to see the effective state. Only failover-enabled clusters can be chosen as a migration destination. See [Cluster failover states](/docs/astro-private-cloud/v-2-x/per-deployment-migration-reference#cluster-failover-states).
</Warning>

## Step 2: Create your regions

A region is a control plane record — a name and a cloud provider — that clusters are assigned to. Regions are what the migration gate compares. You manage them on the **Regions** admin page in the APC UI, or through the GraphQL API. Regions are new in 2.1.

### Seeded regions

The 2.1 schema migration creates two region rows automatically:

| Name     | Cloud provider | Purpose                                                                                                                                                                   |
| -------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `onprem` | `local`        | A ready-to-use default region for on-premises clusters.                                                                                                                   |
| `unset`  | `local`        | A sentinel. Every existing cluster that had no region is backfilled to point here. It is designed never to match, so migration is blocked until you assign a real region. |

You do not need to take any action for the upgrade to succeed — existing clusters are backfilled automatically and nothing breaks. But until you replace `unset` with a real region, every migration attempt involving that cluster is skipped with `CLUSTER_REGION_NOT_SET`.

### Create a region in the APC UI

In the APC UI, go to **Regions** in the admin sidebar and select **Create Region**. A region needs two values:

* **Name** must be unique across the installation. Use whatever names match how you reason about your topology, for example `us-east-1`, `eu-west`, or `dc-chicago`. The control plane doesn't interpret the name — the gate compares region identity, not name strings.
* **Cloud Provider**, for example `aws`, `gcp`, `azure`, or `local`. This isn't cosmetic: a cluster can only be assigned to a region whose cloud provider matches the cluster's own. See [Step 3](#step-3-assign-a-region-to-every-failover-enabled-cluster).

The page lists every region with its cloud provider and creation date, and offers **Edit**, **Delete**, and **Activate** per row.

<Frame>
  <img alt="The Regions admin page listing the two seeded regions" />
</Frame>

<Frame>
  <img alt="The Create Region dialog with the Name and Cloud Provider fields" />
</Frame>

<Frame>
  <img alt="The Cloud Provider dropdown in the Create Region dialog" />
</Frame>

<Note>
  The **Regions** page is shared with control plane high availability, where exactly one region is active at a time and activating a region switches control plane traffic. That has no bearing on data plane migration. The migration gate compares region assignment only — it never reads a region's active state. The two seeded regions ship inactive, which is correct: an inactive region is a valid home for data plane clusters. If you aren't running control plane high availability, ignore the **Active** column.
</Note>

### Create a region with the API

```graphql theme={null}
mutation {
  createRegion(name: "us-east-1", cloudProvider: "aws") {
    region { id name cloudProvider }
  }
}
```

Save the returned `region.id` — you need it to assign clusters in [Step 3](#step-3-assign-a-region-to-every-failover-enabled-cluster). For the full region query and mutation surface, see [Per-deployment migration reference](/docs/astro-private-cloud/v-2-x/per-deployment-migration-reference#graphql-surface).

<Warning>
  A region's `cloudProvider` is immutable while clusters are assigned to it. `updateRegion` rejects a `cloudProvider` change while any cluster references the region, and `deleteRegion` fails while any cluster is still assigned — reassign those clusters first. This prevents a region edit from silently orphaning its clusters. Name-only edits, and re-saving the same provider, are unaffected.
</Warning>

## Step 3: Assign a region to every failover-enabled cluster

This is the step that unlocks migration. Every cluster you want to migrate a Deployment from, and every cluster you want to migrate one to, needs a real, non-`unset` region.

### Assign a region to an existing cluster

1. In the APC UI, go to **Clusters** in the admin sidebar and open the cluster you want to assign.

   <Frame>
     <img alt="The data plane clusters list in the APC UI" />
   </Frame>

2. Turn on **Edit**. The cluster form includes an **Assigned Region** dropdown next to **Cloud Provider**, and a read-only **Failover** field.

   <Frame>
     <img alt="The cluster page in edit mode, showing the Assigned Region dropdown and the Failover status field" />
   </Frame>

3. (Optional) If the cluster's **Cloud Provider** is incorrect, correct it first. **Assigned Region** lists only the regions whose cloud provider matches the cluster's.

   <Frame>
     <img alt="Editing the Cloud Provider field on the cluster edit form" />
   </Frame>

   <Frame>
     <img alt="The Assigned Region dropdown filtered to regions that match the cluster's cloud provider" />
   </Frame>

4. In **Assigned Region**, select the region the cluster physically lives in.

   <Frame>
     <img alt="A cluster with its cloud provider and assigned region both updated" />
   </Frame>

5. Select **Update Cluster**. Review the old and new values in the **Confirm cluster configuration update** dialog, then select **Confirm update**.

   <Frame>
     <img alt="The Confirm cluster configuration update dialog listing the old and new values for Assigned Region and Cloud Provider" />
   </Frame>

To do the same with the API:

```graphql theme={null}
mutation {
  updateCluster(id: "<cluster-id>", regionId: "<region-id>") {
    id
    name
  }
}
```

Passing `regionId: null` clears the assignment.

### Assign a region to a new cluster

Cluster registration accepts `regionId` directly, and the registration form includes the same region selector. At registration, the cluster inherits the assigned region's `cloudProvider`, so there is no separate provider to set and no mismatch to resolve. With no region assigned, the provider comes from the data plane metadata, defaulting to `local`.

### Cloud-provider consistency

When you edit an existing cluster's assigned region, the region must share the cluster's own `cloudProvider`. The cluster edit form's **Cloud Provider** field is editable and filters the **Assigned Region** dropdown to matching regions; to switch provider, set the provider and a matching region together. The comparison is case-insensitive, so `aws` and `AWS` match.

If the providers don't match, the save is rejected:

```text theme={null}
Region cloud provider "gcp" does not match cluster cloud provider "aws"
```

If the region ID doesn't exist:

```text theme={null}
Region <region-id> not found
```

Because every region belongs to exactly one cloud provider, region equality implies provider equality — the migration gate gets cross-provider protection without a second check. Keeping `cluster.cloudProvider` and `region.cloudProvider` in agreement is what makes that inference sound.

<Note>
  If a cluster's own `cloudProvider` is empty, the consistency check is skipped and any region is accepted. This is intentional, so that on-premises and older clusters aren't blocked, but it means the check is a guardrail rather than a guarantee.
</Note>

<Warning>
  Nothing forces a failover-enabled cluster to have a real region. A cluster can be fully failover-enabled, pass every health check, and still sit on the `unset` sentinel: registration succeeds, edits succeed, and cluster-level failover works normally. The consequence surfaces only at migration time, as a per-Deployment skip with `CLUSTER_REGION_NOT_SET`.

  Audit your clusters after upgrading to 2.1. Any failover-enabled cluster still on `unset` is a migration that silently does nothing the first time someone tries it.
</Warning>

## Step 4: Verify the configuration

Run these checks before you declare the setup done.

1. Confirm every failover-enabled cluster has a real region. In the APC UI, check the region on each cluster, or list regions with the API and confirm no participating cluster is assigned to `unset`:

   ```graphql theme={null}
   query {
     regions { id name cloudProvider }
   }
   ```

2. Confirm the destination cluster is failover-enabled and healthy. On the cluster detail page, the **Failover** field should read **Enabled** and the health dot should be green. With the API, `Cluster.failoverEnabled` should be `true`.

3. Ask the control plane which destinations it considers valid:

   ```graphql theme={null}
   query {
     failoverTargetClusters(sourceClusterId: "<source-cluster-id>") {
       id
       name
       status
       healthStatus
       failoverEnabled
     }
   }
   ```

4. Do a canary migration. Migrate one low-stakes Deployment to the destination, and back, before you move anything that matters. See [Manage and observe per-deployment migration](/docs/astro-private-cloud/v-2-x/manage-per-deployment-migration#trigger-a-migration).

<Warning>
  `failoverTargetClusters` is broader than the migration gate. It filters on active status, `failoverEnabled`, data plane health, and no in-flight failover already targeting the cluster — but it doesn't filter by region and does not exclude cordoned clusters. A cluster can appear in this list and still be rejected by `migrateDeployments` with `DESTINATION_CLUSTER_CORDONED`, or have every Deployment skipped with `CROSS_REGION_MIGRATION_NOT_ALLOWED`. The APC UI destination picker draws from this same query, so it inherits the same gaps: each option shows the cluster's region so you can check it, but a cross-region pick fails at submit. Treat both the query result and the picker as a candidate list, not a guarantee.
</Warning>

## Why migration is in-region only

A cluster-level failover and a per-deployment migration look similar but move different things:

| Operation                | What moves                      | What happens to the Airflow metadata database                                                                                                                                    |
| ------------------------ | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Cluster failover         | Every Deployment on the cluster | You promote the database replica in the destination region first, with your managed-database tooling, then trigger the failover. The Deployments land next to their new primary. |
| Per-deployment migration | Only the Deployments you select | Nothing. The Deployment keeps using the same database, at the same endpoint.                                                                                                     |

Because migration doesn't touch the database, a cross-region migration would leave the Deployment's scheduler, workers, and triggerer in one region while its metadata database stays in another. The Airflow scheduler queries its metadata database on every scheduling loop, so this adds cross-region round-trip latency to the hottest path in the system. Throughput drops, and nothing in the platform reports an error, because from Airflow's point of view everything is working.

To move across regions, use full-cluster data plane failover, sequenced after your database promotion. Cross-provider moves at the Deployment level are out of scope.

## Backward compatibility and rollout

The move from "region is optional and ignored" to "region is required for migration" is phased so that upgrading to 2.1 breaks nothing.

The 2.1 upgrade does the following automatically:

* Adds `cloudProvider` to the region record.
* Seeds the `onprem` and `unset` regions.
* Backfills every cluster with no region to `unset`.
* Ships the **Regions** admin page and the cluster region selector.
* Makes `migrateDeployments` available, skipping any Deployment whose source or destination cluster is still on `unset`, with `CLUSTER_REGION_NOT_SET`.

After upgrading, you should:

1. Create regions that reflect your real topology.
2. Reassign every failover-enabled cluster off `unset`.
3. Canary-migrate one Deployment per region pair to validate.

## Related documentation

* [Manage and observe per-deployment migration](/docs/astro-private-cloud/v-2-x/manage-per-deployment-migration)
* [Per-deployment migration reference](/docs/astro-private-cloud/v-2-x/per-deployment-migration-reference)
* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)
* [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover)
* [Trigger a data plane failover](/docs/astro-private-cloud/v-2-x/trigger-data-plane-failover)


# Configure platform resources
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-platform-resources

Configure CPU and memory resources for APC control plane and data plane components.

Astro Private Cloud (APC) runs on Kubernetes and requires careful resource planning for both control plane and data plane components. This guide covers resource configuration for all platform components.

## Architecture overview

APC uses a control plane/data plane architecture:

* **Control Plane**: APC API, Astro UI, Registry, Config Syncer.
* **Data Plane**: the deployment orchestrator, Registry, Airflow deployments.
* **Unified Mode**: All components in a single cluster.

## Control plane components

### APC API

APC API is the GraphQL API that manages platform operations.

```yaml wrap theme={null}
houston:
  replicas: 2
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "1000m"
      memory: "2Gi"
```

#### Scaling recommendations

| Platform Size              | Replicas | CPU Request | Memory Request |
| -------------------------- | -------- | ----------- | -------------- |
| Small (\< 10 deployments)  | 2        | 250m        | 512Mi          |
| Medium (10-50 deployments) | 2        | 500m        | 1Gi            |
| Large (50+ deployments)    | 3        | 1000m       | 2Gi            |

### APC worker

Background job processor for asynchronous operations. The worker uses the same `houston.resources` as the APC API.

```yaml wrap theme={null}
houston:
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "1000m"
      memory: "2Gi"
  worker:
    replicas: 2
```

### Astro UI

Web interface for platform management.

```yaml wrap theme={null}
astroUI:
  replicas: 2
  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
    limits:
      cpu: "500m"
      memory: "512Mi"
```

## Data plane components

### The deployment orchestrator

Manages Airflow deployment provisioning and Kubernetes operations.

```yaml wrap theme={null}
commander:
  replicas: 2
  resources:
    requests:
      cpu: "250m"
      memory: "512Mi"
    limits:
      cpu: "500m"
      memory: "1Gi"
```

### Registry

Docker image registry for Airflow deployments.

```yaml wrap theme={null}
registry:
  replicas: 1
  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
    limits:
      cpu: "500m"
      memory: "512Mi"
  persistence:
    enabled: true
    size: 100Gi
```

#### Storage backend options

* Local PersistentVolume (default)
* Google Cloud Storage (GCS)
* Azure Blob Storage
* Amazon S3

## Ingress and networking

### NGINX ingress controller

```yaml wrap theme={null}
nginx:
  replicas: 2
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "1000m"
      memory: "2Gi"
  serviceType: LoadBalancer
```

## Database

### PostgreSQL

APC API metadata database.

```yaml wrap theme={null}
postgresql:
  resources:
    requests:
      cpu: "250m"
      memory: "256Mi"
    limits:
      cpu: "1000m"
      memory: "1Gi"
  persistence:
    enabled: true
    size: 8Gi
```

#### Production configuration

```yaml wrap theme={null}
postgresql:
  replication:
    enabled: true
    slaveReplicas: 2
    synchronousCommit: "on"
```

## Resource sizing examples

### Development environment

```yaml wrap theme={null}
houston:
  replicas: 1
  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
    limits:
      cpu: "500m"
      memory: "1Gi"

astroUI:
  replicas: 1
  resources:
    requests:
      cpu: "50m"
      memory: "128Mi"

commander:
  replicas: 1
  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"

nginx:
  replicas: 1
  resources:
    requests:
      cpu: "100m"
      memory: "256Mi"
```

### Production environment

```yaml expandable wrap theme={null}
houston:
  replicas: 3
  resources:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "2000m"
      memory: "4Gi"

astroUI:
  replicas: 2
  resources:
    requests:
      cpu: "250m"
      memory: "512Mi"
    limits:
      cpu: "500m"
      memory: "1Gi"

commander:
  replicas: 2
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"

nginx:
  replicas: 3
  resources:
    requests:
      cpu: "1000m"
      memory: "2Gi"

postgresql:
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
  persistence:
    size: 50Gi
```

### High availability configuration

```yaml wrap theme={null}
houston:
  replicas: 3
  podDisruptionBudget:
    enabled: true
    maxUnavailable: 1

astroUI:
  replicas: 3
  podDisruptionBudget:
    enabled: true
    maxUnavailable: 1

commander:
  replicas: 3
  podDisruptionBudget:
    enabled: true
    maxUnavailable: 1
```

## Monitor resource usage

```bash wrap theme={null}
# View pod resource usage
kubectl top pods -n astronomer

# View node resource usage
kubectl top nodes
```

## Troubleshooting

### Out of memory (OOMKilled)

**Symptom**: Pods restart with OOMKilled status.

**Solution**: Increase memory limits:

```yaml wrap theme={null}
houston:
  resources:
    limits:
      memory: "4Gi"
```

### CPU throttling

**Symptom**: Slow response times, high latency.

**Solution**: Increase CPU limits or add replicas:

```yaml wrap theme={null}
houston:
  replicas: 3
  resources:
    limits:
      cpu: "2000m"
```

### Pending pods

**Symptom**: Pods stuck in Pending state.

**Solution**:

1. Check node resources: `kubectl describe nodes`.
2. Reduce resource requests or add nodes.
3. Check for taints/tolerations mismatches.

## Best practices

* Set both requests and limits for predictable scheduling.
* Use Pod Disruption Budgets for high availability.
* Monitor resource usage before scaling.
* Size based on workload not just component count.
* Plan for growth with 20-30% headroom.
* Use separate node pools for platform components.


# Configure Pod security contexts
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-securitycontext

Customize the podSecurityContext or containerSecurityContext on Astro Private Cloud.

Astro Private Cloud platform components support granular security contexts by using the component Helm values. These configurations apply security policies, like `runAsNonRoot` or dropping capabilities, across an entire Pod or in a specific container. When you configure these settings to meet your organization's security requirements, it allows you to add elevated permissions to particular components. Or, you can use validation tooling like Gatekeeper or Kyverno without manual overrides.

The Pod security context provides the defaults for the component's containers. If you set a container security context, it overrides the defaults you configure for the Pod-level, allowing you to customize the behavior of your components.

<Warning>
  Kubernetes security contexts have different fields depending on whether you apply them at the container-level, with the Astronomer `securityContext`, or at the Pod-level, with the Astro Private Cloud `podSecurityContext`. If you configure *both* the `securityContext` and `podSecurityContext`, the container-level definitions take precedence.

  Read more about [`SecurityContext`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.33/#securitycontext-v1-core) and [`podSecurityContext`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.33/#podsecuritycontext-v1-core) in the Kubernetes documentation.
</Warning>

## Prerequisites

* [System Admin](/docs/astro-private-cloud/v-2-x/role-permission-reference#system-admin) permission level, to make changes to the Astronomer Helm chart

<Warning>For OpenShift users, some configurations for security contexts are incompatible with OpenShift operations. When you configure container and Pod security contexts, you must ensure that the configurations you select are compatible with OpenShift. For example, if you have `global.openshift.enabled: true` configured, you can't use a configuration like `fsGroup: 1000` because OpenShift doesn't allow users to run non-standard UIDs.</Warning>

## Configure `securityContext`

To see possible configuration options per component, refer to the [Helm chart config reference](/docs/astro-private-cloud/v-2-x/helm-config-reference). These configurations must be made per-component, and not globally. However there is a sub-set of components that can be configured simultaneously.

<a />

### `astro-ui`, `commander`, `houston`, `configsyncer`, and registry container configuration

You can simultaneously configure the security context for containers in the `astro-ui`, `commander`, `houston`, `configsyncer`, and registry components by updating the `astronomer.securityContext` parameter. The following example configures all containers in these components to run as a non-root user.

```yaml wrap theme={null}
astronomer:
    securityContext:
        runAsNonRoot: true
```

### Component configuration

For all other Astro Private Cloud components, you must define the container security context using the `securityContext` parameter.

```yaml wrap theme={null}
<software-component>:
    securityContext:
```

For example, the following code example configures containers in the Grafana component to run as non-root users:

```yaml wrap theme={null}
grafana:
    securityContext:
        runAsNonRoot: true
```

## Configure `podSecurityContext`

To see possible configuration options per component, refer to the [Helm chart config reference](/docs/astro-private-cloud/v-2-x/helm-config-reference). `podSecurityContext` configurations must be made per-component, and not globally. In general, the configuration structure follows the pattern in this example:

```yaml wrap theme={null}
<software-component>:
    podSecurityContext: {}
```

For example, you can configure the `podSecurityContext` for the Astro UI (`astro-ui`) with the following:

```yaml wrap theme={null}
astronomer:
    astro-ui:
        podSecurityContext:
            fsGroup: 50000
```

## Test security context presence

You can confirm which components have security contexts applied by running the following query in your environment to produce a list of components and their security context status:

```bash wrap theme={null}
kubectl -n astronomer get pods -o json | jq -r \
  '.items[] | "\(.metadata.name)\t\(
    if .spec.securityContext then "HAS_SECURITY_CONTEXT" else "MISSING_SECURITY_CONTEXT" end
  )"' | column -t
```

## Best practice recommendation

Astronomer recommends that you set security contexts only when you need to comply with security policies and requirements. If you apply security contexts, Astronomer recommends using pod-level security context for common settings, and container-level for specific overrides. For example, the following configuration shows how to configure Pod-level and container-level security contexts for the deployment orchestrator. In this example, the container-level `runAsUser` setting defined in Astronomer’s `values.yaml` (under `astronomer.securityContexts.container`) overrides the deployment orchestrator’s Pod-level `runAsNonRoot` setting.

```yaml wrap theme={null}
astronomer:
    securityContexts:
        container:
            runAsUser: 1001
    commander:
        podSecurityContext:
            runAsNonRoot: true
            fsGroup: 50000

```

This produces a Pod manifest for the deployment orchestrator with the following definitions:

```yaml wrap theme={null}
spec:
    securityContext:           # Pod level - defaults for all containers
        runAsNonRoot: true
        fsGroup: 50000
    containers:
    - name: commander
        securityContext:         # Container level - overrides pod defaults
            runAsUser: 1001        # Overrides pod setting
```

## Configure Pod security admission

You can use Kubernetes [Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/) (PSA) to enforce [Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/) at the namespace level and prevent privileged Pod creation in your Apache Airflow Deployment namespaces.

Astro Private Cloud supports the `baseline` Pod Security Standard. Astro Private Cloud doesn't support the `restricted` standard.

<Warning>
  Before you enable PSA, verify that your Dags and custom workloads don't require privileged containers or elevated permissions. Kubernetes rejects Pods that violate the security standard.
</Warning>

### Enable Pod security admission

To enable PSA for Deployment namespaces, add the following label configuration to your `values.yaml` file:

```yaml wrap theme={null}
global:
  namespaceLabels:
    pod-security.kubernetes.io/enforce: "baseline"
```

This configuration adds PSA labels to all namespaces that Astro Private Cloud creates, enforcing the `baseline` security standard. For information about PSA modes and gradual rollout strategies, see [Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/) in the Kubernetes documentation.

## Example: Add privileged access

By default, the Astro Private Cloud install doesn't require elevated privileges. This means that the software container ports are limited to `1024` or greater by default. If you need your install to have exceptions for privileged access, you can update the `controller` settings in the Helm `configs.yaml` file by using the following container security context definition.

```yaml wrap theme={null}
nginx:
    securityContext:
        capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
        allowPrivilegeEscalation: true
```


# Configure Hashicorp Vault for data plane failover
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-vault-data-plane-failover

Configure Hashicorp Vault as the external secrets store for Astro Private Cloud data plane failover, including the Kubernetes auth mount, policy, and permissions each data plane cluster needs.

This document explains how to configure [Hashicorp Vault](https://www.vaultproject.io/) as the external secrets store for Astro Private Cloud (APC) data plane failover, which is supported starting in Astro Private Cloud 2.1. Use it after you've read [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover) and before you complete [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover).

APC authenticates to Vault with Vault's Kubernetes auth method: ESO presents its Kubernetes service account token, and Vault validates it and issues a Vault token. AppRole and token auth aren't supported in Astro Private Cloud 2.1.

<Note>
  This document covers Vault as the `ClusterSecretStore` or `SecretStore` backend that the External Secrets Operator (ESO) uses to replicate Airflow secrets between data planes during failover. It's a different integration from using Vault as an [Airflow secrets backend](/docs/astro-private-cloud/v-2-x/secrets-backend-hashicorp) for Airflow variables and connections within a single Deployment. You can use either integration independently, or both together.
</Note>

## Prerequisites

* A Vault server reachable from every data plane cluster that will use it.
* Permission to create policies, auth mounts, and roles in Vault.
* The Vault CLI, or equivalent API access, authenticated with an admin or operator token.
* `kubectl` access to each data plane cluster. Depending on your ESO mode, you need permission to create either a cluster-scoped `ClusterSecretStore` or a namespaced `SecretStore`. See [Which secret store to create](#which-secret-store-to-create).

<Note>
  The manifests and commands here assume the platform is installed in the `astronomer` namespace. If you installed it into a different namespace, replace `astronomer` throughout, including `bound_service_account_namespaces` and the `serviceAccountRef.namespace`.
</Note>

## Which secret store to create

Vault works behind either an `external-secrets.io` `ClusterSecretStore` or a namespaced `SecretStore`. Which kind you create — a single shared store, a platform-synced store, or one per pool namespace — is determined by your External Secrets Operator mode, not by Vault. See [External Secrets Operator security](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security) to choose a mode, and the [manifests reference](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#secret-store-manifests-modes-1-and-2) for the store manifests. This page covers only the Vault-specific configuration you add to that store.

## Grant Vault access for data plane failover

APC currently offers ESO integration with Vault only for data plane failover, not as a standalone mechanism for syncing other secrets. The policy in this document grants read and write access unconditionally:

* ESO creates `ExternalSecret` resources on every failover-enabled cluster, which read the replicated secrets from Vault.
* The deployment orchestrator on the source cluster also creates `PushSecret` resources, which write the fernet key, environment variables, and database credentials to Vault so the destination cluster can pull them.

<Note>
  Enabling data plane failover (`global.dataPlaneFailover.enabled: true`) doesn't enable ESO by itself. ESO installs from its own Helm subchart, gated separately by `external-secrets.enabled`. See [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover).
</Note>

## Step 1: Create a Vault policy

[Create a Vault policy](https://developer.hashicorp.com/vault/docs/concepts/policies) that grants read and write access:

```hcl wrap theme={null}
path "secret/data/*" {
  capabilities = ["create", "read", "update", "list"]
}

path "secret/metadata/*" {
  capabilities = ["create", "read", "update", "list"]
}
```

This requires capabilities on both the `secret/data/*` and `secret/metadata/*` paths. `PushSecret` uses the metadata path to manage secret versions, so a policy that grants access only to `secret/data/*` still causes `PushSecret` writes to fail.

Write the policy to Vault:

```bash wrap theme={null}
vault policy write <policy-name> <path-to-policy-file>.hcl
```

## Step 2: Enable a Kubernetes auth mount for each data plane cluster

Vault's `auth.kubernetes` method validates a service account token against one specific Kubernetes API server per auth mount. If more than one data plane cluster authenticates to the same Vault instance, each cluster needs its own auth mount — a single mount can't validate tokens from more than one cluster.

1. Enable a dedicated mount for the cluster:

   ```bash wrap theme={null}
   vault auth enable -path=<data-plane-name>-kubernetes kubernetes
   ```

2. Configure the mount with that cluster's API server details:

   ```bash wrap theme={null}
   vault write auth/<data-plane-name>-kubernetes/config \
     kubernetes_host="<cluster-api-server-url>" \
     kubernetes_ca_cert=@<path-to-ca-cert> \
     token_reviewer_jwt=@<path-to-reviewer-jwt>
   ```

   Retrieve `kubernetes_host` from the cluster's API server endpoint, and retrieve `kubernetes_ca_cert` and `token_reviewer_jwt` from a service account token that Vault uses to validate other tokens. See [Vault's Kubernetes auth method documentation](https://developer.hashicorp.com/vault/docs/auth/kubernetes) for how to generate the reviewer JWT for your Kubernetes version.

Repeat this step for every data plane cluster that authenticates to this Vault instance, substituting a unique `-path` value each time.

## Step 3: Create a Vault role

Create a role that binds the policy from Step 1 to the auth mount from Step 2, scoped to the service account ESO runs as:

```bash wrap theme={null}
vault write auth/<data-plane-name>-kubernetes/role/<role-name> \
  bound_service_account_names=<eso-service-account> \
  bound_service_account_namespaces=astronomer \
  policies=<policy-name> \
  ttl=1h
```

`<eso-service-account>` is the Kubernetes service account that the ESO Pod runs as on this data plane cluster. If you installed ESO through the APC Helm chart's bundled subchart (`external-secrets.enabled: true`), this is the release's ESO service account in the `astronomer` namespace.

<Note>
  If the cluster uses [External Secrets Operator security](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security) Mode 3 (customer-managed isolated identity), map the per-namespace service account here instead of the shared ESO service account, so each Deployment authenticates to Vault as its own identity.
</Note>

## Step 4: Add the Vault provider to your secret store

Reference the auth mount and role you created from the `spec.provider.vault` block of your ESO secret store. The example below is a `ClusterSecretStore` (the shared-identity form). For a namespaced `SecretStore` — the platform-synced store in Mode 2 or a per-pool-namespace store in Mode 3 — use the identical `spec.provider.vault` block in the `SecretStore` shape shown in the [manifests reference](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#secret-store-manifests-modes-1-and-2).

```yaml wrap theme={null}
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: astronomer-secret-store
spec:
  provider:
    vault:
      server: <vault-server-url>
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: <data-plane-name>-kubernetes
          role: <role-name>
          serviceAccountRef:
            name: <eso-service-account>
            namespace: astronomer
```

Use the same value for `metadata.name` on every data plane cluster that shares this Vault instance — this is the value you provide for `global.dataPlaneFailover.externalSecretManagerName` in your Helm values. Use the same `path` on every cluster as well, since it identifies the Vault secrets engine that holds the replicated secrets, and the destination cluster reads from the same location the source cluster writes to. The `auth.kubernetes.mountPath` and `auth.kubernetes.role` values differ per cluster.

## Network requirements

If your data plane clusters run in separate networks, Vault must be reachable from each of them over the network, not just from its own cluster's in-cluster DNS.

* Expose Vault through a network load balancer reachable from every data plane cluster that needs it.
* Open firewall or security group rules that allow traffic from each data plane cluster's CIDR range to the Vault endpoint.
* Confirm connectivity from each cluster before you rely on the configuration — see the verification steps in the following section.

## Verify the configuration

Before applying Helm values that depend on this `ClusterSecretStore` or `SecretStore`, confirm the setup works end to end.

1. Write a test secret to Vault:

   ```bash wrap theme={null}
   vault kv put secret/variables/test-variable value=test-value
   ```

2. Create a test `ExternalSecret` in the data plane cluster's `astronomer` namespace that references the store and reads the test secret. This example uses a `ClusterSecretStore`; on a namespace-pools cluster, set `secretStoreRef.kind` to `SecretStore` instead.

   ```yaml wrap theme={null}
   apiVersion: external-secrets.io/v1
   kind: ExternalSecret
   metadata:
     name: <test-secret-name>
     namespace: astronomer
   spec:
     refreshInterval: 10s
     secretStoreRef:
       name: astronomer-secret-store
       kind: ClusterSecretStore
     target:
       name: <test-secret-name>
       creationPolicy: Owner
     data:
       - secretKey: value
         remoteRef:
           key: variables/test-variable
           property: value
   ```

3. Confirm it syncs:

   ```bash wrap theme={null}
   kubectl get externalsecret <test-secret-name> -n astronomer
   ```

   The `STATUS` column should show `SecretSynced`. If it doesn't, check the ESO Pod logs for authentication or permission errors before continuing.

Repeat this test from each data plane cluster that shares the Vault instance, since each cluster authenticates through its own auth mount.

After you enable data plane failover (`dataPlaneFailover.enabled: true`), also complete the `PushSecret` and connection-secret checks in [Verify secret replication before a failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover#verify-secret-replication-before-a-failover). Those checks confirm write access, which the preceding `ExternalSecret` test doesn't cover.

## Troubleshooting

* **`PushSecret` reports a failure, or Airflow secrets don't appear on the destination cluster after a failover**: The Vault policy likely grants read-only access on a cluster where `dataPlaneFailover.enabled` is `true`. Update the policy to include `create` and `update` on both `secret/data/*` and `secret/metadata/*`, then retry.
* **`ExternalSecret` or `PushSecret` fails to authenticate from one data plane cluster but succeeds from another**: Confirm the failing cluster has its own Kubernetes auth mount and that the mount's `kubernetes_host` and reviewer JWT match that cluster, not another cluster's.
* **A cluster in a different network can't reach Vault**: Confirm Vault is exposed beyond in-cluster DNS for that network, and that firewall or security group rules allow traffic from the cluster's CIDR range. See [Network requirements](#network-requirements).

## Related documentation

* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)
* [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover)
* [External Secrets Operator security](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security)
* [Run a failover upgrade](/docs/astro-private-cloud/v-2-x/run-failover-upgrade)
* [Trigger a data plane failover](/docs/astro-private-cloud/v-2-x/trigger-data-plane-failover)
* [Hashicorp Vault as an Airflow secrets backend](/docs/astro-private-cloud/v-2-x/secrets-backend-hashicorp)
* [External Secrets Operator Vault provider documentation](https://external-secrets.io/latest/provider/hashicorp-vault/#kubernetes-authentication)


# Control plane reliability
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery

Understand how Astro Private Cloud control plane reliability provides in-region high availability and manual cross-region disaster recovery for the control plane.

Control plane reliability runs two or more control planes that share one database and serve one customer-facing domain. It works at two levels:

* *In-region high availability (HA)*: if you stand up more than one control plane in the active region, they serve traffic together behind weighted, health-checked DNS. When one becomes unhealthy, DNS drains it from rotation and routes users to the remaining healthy control planes in that region automatically. A region with a single control plane has no in-region redundancy.
* *Manual cross-region disaster recovery*: you place control planes in a second, standby region and, during a regional outage or a planned migration, an admin fails the platform over to that region. Cross-region failover is always an explicit admin action — there is no automatic region failover, so the control plane isn't automatically highly available across regions.

Control plane reliability is available on Astro Private Cloud (APC) 2.1.0 and later.

<Note>
  Control plane reliability and data plane failover are independent features that solve different problems. Control plane reliability keeps the *control plane* (the APC UI and API) available: highly available within a region, and manually recoverable across regions. [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover) moves *Apache Airflow Deployments* between data plane clusters. You can run either feature on its own, or both together. This document covers only the control plane.
</Note>

## How it works

A control plane reliability group is a set of control planes that share three things and are fronted by one load-balanced domain:

* *A shared database*: every control plane connects to the same APC API database, so they present the same users, Workspaces, and Astro Deployments. Both clusters must use an identical database connection, supplied through the same `astronomer-bootstrap` secret.
* *Shared JWT signing material*: every control plane signs JSON Web Tokens (JWTs) with the same key and certificate, so a token minted on one control plane is trusted by every other control plane. The first control plane generates this material; every subsequent control plane reuses a copy of it.
* *A shared global domain*: every control plane serves the same customer-facing domain, `app.<global-domain-name>`. Weighted, health-checked DNS records spread traffic across all control planes and drain an unhealthy one automatically.

Each control plane is also reachable on its own per-control-plane (per-CP) admin hostname (`cpNN.<parent-domain>`) that bypasses the load balancer, so you can reach and administer a specific control plane directly even when it isn't in DNS rotation.

### Regions and the control plane registry

When you enable control plane reliability, the APC API tracks two kinds of records that you manage from the APC UI or through GraphQL mutations:

* A *region* is a logical grouping that control planes attach to. Every control plane in a control plane reliability group belongs to a region, and *only one region is active at a time*. Activating a region atomically deactivates every other region.
* A *control plane registry* entry records each control plane, the region it belongs to, its ingress URL, and its chart version. You register each control plane once against a region.

<Frame>
  <img alt="Regions page in the APC UI, listing each region with its cloud provider, an Active or Inactive status badge, and creation time." />
</Frame>

<Frame>
  <img alt="Control Planes page in the APC UI, listing each registered control plane with its health, region, status, chart version, and ingress URL." />
</Frame>

### The health endpoint

Each control plane exposes an HA health endpoint at `/controlplane/status`. Your global DNS load balancer polls this endpoint per control plane and serves traffic only to control planes that report healthy. A control plane reports unhealthy (HTTP `503`) when it isn't registered or when it is cordoned, decommissioned, running a chart version behind its region's maximum, or attached to an inactive region. At least one control plane must be healthy and in the active region to serve traffic.

### Chart-version eligibility

Only control planes running the highest registered chart version within their own region are eligible to serve traffic. The APC API computes this maximum dynamically across the active control planes in each region. A control plane that lags behind its region's maximum is drained until it catches up. This gate governs both rolling upgrades and cross-region failover. For the upgrade procedure, see [Manage a control plane reliability group](/docs/astro-private-cloud/v-2-x/manage-control-plane-disaster-recovery#upgrade-the-chart-version).

## What control plane reliability does and doesn't own

Control plane reliability owns the control plane side of availability: which control planes are eligible to serve, which region is active, and the health signal the DNS load balancer polls.

Control plane reliability doesn't fail over the APC API database, and it doesn't rotate DNS. Both are external to the platform:

* *Database failover* is handled by your managed-database tooling (for example, Amazon RDS or Google Cloud SQL). During a cross-region failover you must promote the destination-region database and repoint the connection before you activate the destination region.
* *DNS load balancing and failover* is handled by your DNS provider through the weighted, health-checked records you configure. Control plane reliability only flips each control plane's health signal.

This distinction matters most during [cross-region failover](/docs/astro-private-cloud/v-2-x/manage-control-plane-disaster-recovery#cross-region-failover).

## Terminology

| Term                     | Meaning                                                                                                                                                |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `<global-domain-name>`   | The shared customer-facing domain that every control plane in the group serves. Users reach the platform at `app.<global-domain-name>`.                |
| `<parent-domain>`        | The parent domain you host the per-CP admin hostnames under.                                                                                           |
| `<cpNN-domain>`          | A control plane's per-CP admin domain (for example, `cp01.<parent-domain>`). Reaches one specific control plane directly, bypassing the load balancer. |
| `<release-name>`         | The Helm release name of the Astronomer platform (for example, `astronomer`).                                                                          |
| `<astronomer-namespace>` | The Kubernetes namespace the Astronomer platform is installed into.                                                                                    |
| CP                       | Control plane. Used in the *per-CP* modifier and in the `cpNN` hostnames.                                                                              |
| CP 1                     | The first, bootstrap control plane. It generates the shared JWT signing material.                                                                      |
| CP 2                     | The second and any subsequent control plane. Each reuses CP 1's JWT signing material.                                                                  |

## Requirements

Before you configure control plane reliability, confirm the following:

* A TLS certificate for each control plane whose Subject Alternative Names cover both the global names (`*.<global-domain-name>`) and that control plane's per-CP admin names (`*.<cpNN-domain>`). Use the DNS-01 Automatic Certificate Management Environment (ACME) challenge, not HTTP-01. For details, see [Configure control plane disaster recovery](/docs/astro-private-cloud/v-2-x/configure-control-plane-disaster-recovery#set-up-tls-and-dns-prerequisites).
* A single shared database that every control plane connects to through an identical `astronomer-bootstrap` secret.
* The ability to create weighted, health-checked DNS records for the global domain in your DNS provider.
* A plan for sizing the shared database's connection limit as you add control planes. See [Control plane reliability reference](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery-reference#database-connection-sizing).

## Limitations

* *Admin-driven region failover only*: there is no automatic cross-region failover. Moving the active region is always an explicit admin action.
* *No in-flight work drain*: cordoning a control plane doesn't drain in-flight work. Any messages still queued in a source region's NATS at deactivation stop being consumed.
* *Database and DNS are external*: control plane reliability doesn't fail over the database or rotate DNS. You are responsible for both. See [What control plane reliability does and doesn't own](#what-control-plane-reliability-does-and-doesn’t-own).

## Related documentation

* [Configure control plane reliability](/docs/astro-private-cloud/v-2-x/configure-control-plane-disaster-recovery)
* [Manage a control plane reliability group](/docs/astro-private-cloud/v-2-x/manage-control-plane-disaster-recovery)
* [Control plane reliability reference](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery-reference)
* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)
* [Control plane architecture](/docs/astro-private-cloud/v-2-x/control-plane-architecture)


# Control plane reliability reference
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery-reference

Reference for control plane reliability: Helm values, cookie and URL strategies, identity provider redirect URLs, database connection sizing, statuses, and error codes.

This page is the reference companion to [Control plane reliability](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery). It documents the Helm values, cookie and URL strategies, identity provider (IdP) redirect URLs, database connection sizing, control plane statuses, and error codes for control plane reliability. For the setup procedure, see [Configure control plane reliability](/docs/astro-private-cloud/v-2-x/configure-control-plane-disaster-recovery). For day-2 operations, see [Manage a control plane reliability group](/docs/astro-private-cloud/v-2-x/manage-control-plane-disaster-recovery).

## Helm values

Set these values in your Astronomer Helm values. `controlPlaneHA.enabled` and `dataPlaneFailover.enabled` are independent switches, so you can run control plane reliability without data plane failover, or the other way around.

| Value                                           | Default                     | Effect                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| ----------------------------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `global.controlPlaneHA.enabled`                 | `false`                     | Set to `true` on every control plane in the group. Turns on control plane reliability: renders the global-domain ingresses, mounts the per-control-plane (per-CP) identity (`cp-identity` secret), enables the control plane registry, the `/controlplane/status` health endpoint, and chart-version-eligibility gating, scopes session cookies and URLs to `globalBaseDomain`, and surfaces the **Regions** and **Control Planes** admin tabs. Requires `globalBaseDomain`, a shared database, and a shared JSON Web Token (JWT) keypair. |
| `global.controlPlaneHA.bootstrapJwks`           | `false`                     | Set to `true` on the first control plane only, and `false` on all others. On the first control plane, generates the shared JWT signing key and certificate. On subsequent control planes, leave it `false` so that they consume the copied keypair — a fresh key would break cross-control-plane token validation.                                                                                                                                                                                                                         |
| `global.controlPlaneHA.globalBaseDomain`        | Unset                       | The shared customer domain that every control plane serves (`app.<globalBaseDomain>`, `houston.<globalBaseDomain>`). Drives the global ingress hosts and the session-cookie scope (`.<globalBaseDomain>`).                                                                                                                                                                                                                                                                                                                                 |
| `global.baseDomain`                             | Required, per control plane | This control plane's own admin hostname (`cp01.<parent-domain>`), distinct from `globalBaseDomain`. Used to reach a specific control plane directly, bypassing the global load balancer.                                                                                                                                                                                                                                                                                                                                                   |
| `global.dataPlaneFailover.enabled`              | `false`                     | Independent of `controlPlaneHA`. On a control plane, enables the components that let Astro Deployments be failed over between clusters. On a data plane, enables the data plane execution components. See [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover).                                                                                                                                                                                                                                                           |
| `astronomer.houston.upgradeDeployments.enabled` | `true`                      | The `houston-upgrade-deployments` Helm hook (`post-upgrade`, `post-install`). When enabled, every control plane `helm upgrade` re-stamps all non-cordoned Astro Deployment ingresses. Set to `false` to skip the automatic re-stamp and run it manually. See [Add existing Astro Deployments to the global domain](/docs/astro-private-cloud/v-2-x/configure-control-plane-disaster-recovery#add-existing-astro-deployments-to-the-global-domain).                                                                                              |

<Note>
  Set `astronomer.houston.upgradeDeployments.enabled` under the `astronomer` subchart, as shown. A top-level `houston.upgradeDeployments.enabled` doesn't take effect and silently leaves the hook enabled.
</Note>

## Cookie and URL strategy

When you bring existing Astro Deployments under control plane reliability, you choose how their URLs and the session cookie are scoped. APC supports three strategies, all selected through Helm and DNS configuration on an APC 2.1.0 or later build. Pick one per installation.

### Alias existing URLs

Option A is the recommended default for existing installations. Keep every Astro Deployment's existing per-CP URL and add a parallel global-domain URL as an alias. Both resolve to the same Airflow webserver.

* *Enable*: the default path. Set `controlPlaneHA.enabled: true` and `globalBaseDomain`. The [ingress re-stamp](/docs/astro-private-cloud/v-2-x/configure-control-plane-disaster-recovery#add-existing-astro-deployments-to-the-global-domain) adds the global-alias hosts alongside the per-CP ones.
* *Cookie*: the session cookie is scoped to `.<globalBaseDomain>`, so it's sent to Deployment Airflow subdomains under the global domain and single sign-on (SSO) works there. Per-CP hosts keep their own per-CP cookie.
* *Customer impact*: none. Old per-CP URLs keep working, and global URLs are added. This is the safe default for existing customers.

### Widen the cookie scope

With Option B, scope one session cookie to a common ancestor domain that covers both the global and per-CP host families, so a single cookie is sent to every host under that ancestor.

* *Enable*: set `helm.cookieDomain` to the shared parent of your global and per-CP domains. The platform then runs in single-cookie mode, in which per-CP hosts mint the same wide cookie.

<Warning>
  The cookie is sent to every host under the ancestor, including anything else hosted there. There is no platform guard against too broad an ancestor. You're responsible for choosing one that covers only trusted Astronomer hosts, and for TLS covering those hosts. Choose Option B only if that broader exposure is acceptable.
</Warning>

### Re-align URLs

Option C is for fresh installations only. Regenerate all Astro Deployment URLs under a single new shared parent domain, replacing the per-CP URLs rather than aliasing them.

* *Enable*: install with the cluster base domain set to the shared parent so that every URL templates under it.

<Warning>
  Because per-CP URLs are replaced rather than kept, any tooling that uses the old hardcoded URLs breaks. For that reason, Option C is supported only for fresh installations, never as a migration for existing customers.
</Warning>

## Identity provider authentication and the OAuth redirect URL

To configure an external identity provider on a standard single-control-plane installation, see [Integrate an authentication system](/docs/astro-private-cloud/v-2-x/integrate-auth-system). This section covers only what changes when you enable control plane reliability with `controlPlaneHA.enabled: true`.

If you integrate an external identity provider (IdP), the redirect and callback URL you register with the IdP must point at the global domain, not a per-CP domain. Whenever `controlPlaneHA.enabled: true`, the APC API templates all customer-facing URLs — including the OAuth `redirect_uri` it hands to your IdP — from `globalBaseDomain` instead of the per-CP `baseDomain`.

Register the values that use the global domain:

* `https://houston.<global-domain-name>/v1/oauth/redirect/` for the implicit flow, which is the default.
* `https://houston.<global-domain-name>/v1/oauth/callback/` for the code flow (`auth.openidConnect.flow: "code"`).

Register whichever matches your configured flow, or both if you're unsure, and keep the trailing `/` as the APC API emits it, because most IdPs match the redirect URI exactly.

Customer sign-in and the IdP round trip run on the global hostname so that APC sets the session cookie, scoped to `.<global-domain-name>`, on a host that's allowed to set it. A per-CP host is a sibling of the global domain, not a descendant, so a global-scoped cookie set there is dropped and the user is bounced back to the sign-in page. To enforce this, the APC API doesn't serve customer auth paths on the per-CP admin hostnames: any OAuth start, IdP callback, or Deployment sign-in request that lands on a per-CP host is redirected to `app.<global-domain-name>/login`. The per-CP hostnames are for direct admin access only.

There is no separate Helm value for the redirect URL. It follows the same `globalBaseDomain` resolution as every other customer-facing URL. The session cookie's domain defaults to `.<global-domain-name>` and is overridable through `helm.cookieDomain`. See [Widen the cookie scope](#widen-the-cookie-scope).

<Warning>
  When you migrate an existing installation to control plane reliability, if your IdP application was registered with a per-CP redirect URI, update it to the global-domain URI as part of enabling control plane reliability. Otherwise the IdP rejects the callback after sign-in once the APC API starts sending the global `redirect_uri`.
</Warning>

<Note>
  This section applies to bring-your-own OIDC and IdP setups. If you use the default shared Auth0 tenant that Astronomer provides, rather than your own external IdP, the redirect is fixed to `https://redirect.astronomer.io` regardless of domain, and this section doesn't apply.
</Note>

## Database connection sizing

Every control plane shares one database, and each long-lived APC API Pod holds its own connection pool against it. As you add control planes or scale replica counts, total connections grow. Size the database's `max_connections`, or cap the pools, accordingly, or you risk connection exhaustion.

### Measured consumption

Measured on a two-control-plane installation with `astronomer.houston.prismaConnectionLimit` unset, so each Pod uses the default pool of about `num_cpus × 2 + 1`:

* About four connections per long-lived Pod.
* Long-lived, database-connected Pods per control plane at default replica counts, with control plane reliability and data plane failover enabled: two APC API, two APC API worker, three DP-Link, and three Navigator Pods, for 10 Pods and about 40 connections per control plane.
* Observed total for two control planes: 84 connections (about 42 per control plane), almost all idle, against `max_connections = 400` — about 21% utilized.
* Short-lived hook and cron jobs (database migration, `upgrade-deployments`, control plane refresh, and cleanup) open a few more connections briefly while they run. Budget a small margin for these.

### Capacity-planning formula

```text theme={null}
required_connections ≈ N_CPs × pods_per_CP × pool_per_pod
                      + margin for hook and cron jobs
                      + connections used by any other database on the instance
```

At default replica counts and default pool, `pods_per_CP × pool_per_pod` is about 40, so as a rule of thumb:

* Budget about 40 connections per control plane, and set `max_connections ≥ N_CPs × 40 × 1.25` for about 25% headroom.
* At `max_connections = 400`, that leaves headroom for roughly five to six control planes at default sizing before you must intervene.

Recompute if you change replica counts or increase Pod CPU, because the default pool scales with CPU (`num_cpus × 2 + 1` per Pod), so larger Pods open larger pools.

### Levers for more headroom

* *Cap the per-Pod pool*: set `astronomer.houston.prismaConnectionLimit` to a fixed, smaller value so that each Pod's pool is bounded regardless of Pod CPU. This makes the total exactly predictable: `N_CPs × pods_per_CP × prismaConnectionLimit`.
* *Front the database with PgBouncer* in transaction-pooling mode, so that many APC API Pods multiplex onto far fewer server-side connections. Astronomer recommends this once the control plane count or replica counts push you toward `max_connections`.
* *Raise `max_connections`* on the managed database if the instance class allows it. Each connection costs memory, so scale the instance accordingly.

Before you add a control plane, check current utilization and keep it comfortably below the limit, aiming for 75% or less:

```sql theme={null}
SELECT count(*) AS current,
       (SELECT setting::int FROM pg_settings WHERE name = 'max_connections') AS max_connections
FROM pg_stat_activity WHERE datname = current_database();
```

## Control plane status reference

| Effective status | Badge color | `/controlplane/status`                                      | Mutations                    |
| ---------------- | ----------- | ----------------------------------------------------------- | ---------------------------- |
| `ACTIVE`         | Green       | `200`, if registered, at maximum version, and region active | Accepted                     |
| `STANDBY`        | Blue        | `503` (`REGION_INACTIVE`)                                   | Rejected `REGION_INACTIVE`   |
| `CORDONED`       | Yellow      | `503` (`CORDONED`)                                          | Rejected `CP_CORDONED`       |
| `DECOMMISSIONED` | Gray        | `503` (`DECOMMISSIONED`)                                    | Rejected `CP_DECOMMISSIONED` |

## Health endpoint unhealthy reasons

The `/controlplane/status` endpoint evaluates the following reasons in order, and the first match wins:

1. Not registered (`NOT_REGISTERED`).
2. Cordoned (`CORDONED`).
3. Decommissioned (`DECOMMISSIONED`).
4. Region inactive (`REGION_INACTIVE`).
5. Chart version behind the region maximum (`VERSION_OUTDATED`).

A transient database error also returns unhealthy for that single poll, and this result isn't cached. Otherwise the endpoint returns `200`. Results are cached for about 30 seconds. The endpoint doesn't consider the last heartbeat — heartbeat staleness affects only the UI health dot, never routing.

## Error codes

These are the GraphQL error codes the APC API returns when it rejects a mutation. They're a separate set from the health-endpoint reasons in the previous section, which is why some names differ — for example, the health endpoint reports `VERSION_OUTDATED` while the matching mutation error is `CP_VERSION_OUTDATED`.

| Code                           | When it occurs                                                                                                                            |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `INVALID_CP_STATUS_TRANSITION` | You attempted an illegal status change, for example `ACTIVE` to `DECOMMISSIONED` directly.                                                |
| `CP_CORDONED`                  | You sent a mutation to a cordoned control plane.                                                                                          |
| `CP_DECOMMISSIONED`            | You sent a mutation to a decommissioned control plane.                                                                                    |
| `CP_VERSION_OUTDATED`          | You sent a mutation to a control plane behind its region's maximum chart version.                                                         |
| `REGION_INACTIVE`              | You sent a mutation to a control plane whose region isn't the active one.                                                                 |
| `TARGET_REGION_NOT_UPGRADED`   | `activateRegion` refused because no control plane in the target region is at the group-wide maximum version. Override with `force: true`. |
| `NOT_REGISTERED`               | The control plane has no registry row. Register it, or it was deregistered.                                                               |

## Related documentation

* [Control plane reliability](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery)
* [Configure control plane reliability](/docs/astro-private-cloud/v-2-x/configure-control-plane-disaster-recovery)
* [Manage a control plane reliability group](/docs/astro-private-cloud/v-2-x/manage-control-plane-disaster-recovery)
* [Helm configuration reference](/docs/astro-private-cloud/v-2-x/helm-config-reference)


# Pause and resume APC management of a Deployment
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/cordon-deployment

Cordon a Deployment so Astro Private Cloud stops applying changes to it, and uncordon it to resume. The Deployment keeps running either way.

Cordoning a Deployment tells Astro Private Cloud (APC) to stop applying changes to it. The Deployment keeps running exactly as it is: its schedulers, workers, and other components stay up, and Airflow keeps scheduling and running tasks. Only APC's ability to change the Deployment is paused.

Uncordon it to resume normal management.

Cordoning applies to any Deployment, whether it is Helm-managed or operator-managed, and whether APC created it or [adopted](/docs/astro-private-cloud/v-2-x/adopt-operator-deployments) it.

## When to cordon

* **During maintenance on the Deployment**, when you are changing something out of band and don't want APC applying changes at the same time.
* **During a change freeze**, to guarantee that nobody's configuration edit reaches a production Deployment.
* **Before a platform upgrade**, for any Deployment you don't want the platform to act on while it upgrades.
* **While the operator is being replaced**, so that nothing is applied during the window when no operator is reconciling. See [Move the operator under APC](/docs/astro-private-cloud/v-2-x/transition-operator-to-apc).

## What a cordon blocks

While a Deployment is cordoned, APC refuses these operations on it:

* Deployment configuration updates, including resources and configuration overrides
* Environment variable changes
* KEDA autoscaling configuration changes
* Code deploys and image updates
* Deploy rollbacks
* Migrating the Deployment to another cluster
* Deleting the Deployment. In the Astro UI the delete action is disabled and labeled **uncordon to delete**

Attempts through the Astro UI or the APC API are rejected with an error saying the Deployment is cordoned. Background work is skipped rather than queued, so nothing you attempt during a cordon is applied later when you uncordon: reapply it yourself afterwards.

Everything else is unaffected. Your Airflow keeps running, Dags keep being scheduled, tasks keep executing, and logs and metrics keep flowing.

<Note>
  A whole cluster can also be cordoned, which blocks every Deployment on it regardless of each Deployment's own cordon state. If a Deployment refuses changes and its own cordon is off, check whether its cluster is cordoned.
</Note>

## Cordon a Deployment

<Tabs>
  <Tab title="Astro UI">
    In the Deployments list, open the Deployment's actions menu and select **Cordon Deployment**. You can optionally give a reason.

    <Frame>
      <img alt="The Deployments list with a Deployment's actions menu open, showing the Cordon Deployment option." />
    </Frame>

    The Deployment then shows a **Cordoned** badge in the **Cordoned** column of the Deployments list, and its editable pages show a banner explaining that configuration changes can't be applied until it's uncordoned.
  </Tab>

  <Tab title="APC API">
    ```graphql theme={null}
    mutation {
      cordonDeployment(deploymentUuid: "<deployment-id>", reason: "Maintenance window") {
        id
        isCordoned
        cordonedAt
        cordonedReason
      }
    }
    ```

    `reason` is optional and is recorded with the cordon so other people can see why it was paused.
  </Tab>
</Tabs>

Cordoning is idempotent. Cordoning a Deployment that is already cordoned changes nothing and is not recorded again.

There is no Astro CLI command for cordoning. Use the Astro UI or the APC API.

## Uncordon a Deployment

<Tabs>
  <Tab title="Astro UI">
    In the Deployments list, open the Deployment's actions menu and select **Uncordon Deployment**.

    <Frame>
      <img alt="The Deployments list showing a Deployment marked Cordoned, with its actions menu open on the Uncordon Deployment option and Delete Deployment disabled." />
    </Frame>
  </Tab>

  <Tab title="APC API">
    ```graphql theme={null}
    mutation {
      uncordonDeployment(deploymentUuid: "<deployment-id>") {
        id
        isCordoned
      }
    }
    ```
  </Tab>
</Tabs>

Uncordoning is also idempotent, and APC resumes applying changes immediately. Changes you attempted while it was cordoned are not replayed, so make them again.

## Who can cordon and uncordon

Cordoning and uncordoning require permission to update the Deployment, so anyone who can change a Deployment's configuration can also pause and resume its management. See [Manage permissions](/docs/astro-private-cloud/v-2-x/manage-permissions) and the [role and permission reference](/docs/astro-private-cloud/v-2-x/role-permission-reference).

## Cordons set by Astronomer

A cordon records why it was applied. Most cordons are user-initiated, and appear as such. Astronomer tooling can also cordon a Deployment as part of remediating an adoption, in which case the in-product banner says so. Treat an Astronomer-initiated cordon as deliberate and check with Astronomer before uncordoning it.

## Related documentation

* [Configure a Deployment](/docs/astro-private-cloud/v-2-x/configure-deployment)
* [Adopt operator Deployments](/docs/astro-private-cloud/v-2-x/adopt-operator-deployments)
* [Move the operator under APC](/docs/astro-private-cloud/v-2-x/transition-operator-to-apc)
* [Manage permissions](/docs/astro-private-cloud/v-2-x/manage-permissions)


# Programmatically create or update Deployments on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/create-deployment-programmatic

Programmatically create or update Deployments on Astro Private Cloud using the APC API.

You can programmatically create or update Deployments with all possible configurations using the APC API `upsertDeployment` mutation.

<Warning>
  When you make upsert updates to your Airflow Deployments, you must explicitly specify all existing environment variables, otherwise, the upsert overwrites them.
</Warning>

For a complete example of the `upsertDeployment` mutation, see [Upsert a Deployment with the APC API](/docs/astro-private-cloud/v-2-x/houston-upsert-deployment).


# Create Deployments using Astro Runtime SHA256 digest
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/create-deployment-sha

Create Deployments using an Astro Runtime SHA256 digest for reproducible, immutable builds.

By default, Deployment creation references an Astro Runtime image by its tag in the Kubernetes spec, like `quay.io/astronomer/astro-runtime:9.3.0`. However, image tags are mutable and can lead to non-reproducible builds if the image associated with the tag changes. Instead of using the Runtime image tag, you can configure Astro Private Cloud to reference a Runtime image's immutable `sha256` digest, such as `quay.io/astronomer/astro-runtime@sha256:<digest>`. Using the `sha256` digest ensures secure, immutable, and reproducible Deployments, which prevents unexpected behavior caused by tag reassignments.

After you enable using the `sha256` digest, when users create or update Deployments that include a SHA version, they still see the same Runtime Image tag view as before in the UI or CLI, but the system resolves the build using the `sha256` digest in the Kubernetes spec.

## Step 1: Enable configuration

[Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config) to your APC API `values.yaml` file to enable `customImageShaEnabled`.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        customImageShaEnabled: true
```

## Step 2: (Optional) Correlate SHA256 with Runtime versions

By default, Astro Private Cloud checks for Airflow updates, which are included in the Astro Runtime, once per day at midnight by querying `https://updates.astronomer.io/astronomer-runtime`. This returns a JSON file with details about the latest available Astro Runtime versions.

You can store this information in the cluster itself by completing the following steps:

1. Download the JSON files and store them in a Kubernetes configmap by running the following commands:

```sh wrap theme={null}
curl -XGET https://updates.astronomer.io/astronomer-runtime -o astro_runtime_releases.json

kubectl -n <astronomer platform namespace> create configmap astro-runtime-base-images --from-file=astro_runtime_releases.json
```

2. Open the `astro_runtime_release.json` file and manually add the SHA256 values that you want Deployments to use for each Runtime version. For example, the following code example shows

```json wrap theme={null}
"13.0.0": {
  "metadata": {
    "airflowVersion": "2.11.0",
    "channel": "stable",
    "releaseDate": "2025-05-20",
    "endOfSupport": "2026-11-30",
    "LTS": true
  },
  "migrations": {
    "airflowDatabase": false,
    "stellarDatabase": false
  }
}
```

Add the Tag and SHA256 value and save:

```json wrap theme={null}
"13.0.0": {
  "metadata": {
    "airflowVersion": "2.11.0",
    "channel": "stable",
    "releaseDate": "2025-05-20",
    "endOfSupport": "2026-11-30",
     "LTS": true
  },
  "migrations": {
    "airflowDatabase": false,
    "stellarDatabase": false
  },
  "sha256": "82dc7efe0b16acc74e96a82bc8f1fd1db35a76a5a8c32f581d171d9765c02326"
}
```

3. Add your configmap name, `astro-runtime-base-images` to your APC API configuration using the `runtimeReleasesConfigMapName` configuration:

```yaml wrap theme={null}
astronomer:
  houston:
    runtimeReleasesConfigMapName: astro-runtime-base-images
    config:
      airgapped:
        enabled: true
```

## Step 3: (Optional) Specify default Runtime

If you want to configure your platform to create Deployments with a single, specific Runtime version, you can add the `defaultRuntimeRepository` configuration to specify the Runtime:

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        customImageShaEnabled: true
      helm:
        defaultRuntimeRepository: quay.io/astronomer/astro-runtime@sha256
```


# Create a Deployment on Astro Private Cloud from the UI
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/create-deployment-ui

Create a new Airflow Deployment on Astro Private Cloud from the UI.

Use this document to learn how to create a Deployment on Astro Private Cloud (APC).

## Prerequisites

* [The Astro CLI](/docs/cli/v1.43/install-cli)
* A [Workspace](/docs/astro-private-cloud/v-2-x/manage-workspaces)

## Create a Deployment in the Astro Private Cloud UI

To create an Airflow Deployment on APC:

1. Sign in to your APC platform at `app.BASEDOMAIN`, select a Workspace, and then click **+ Deployment**.

2. Complete the following fields:

   * **Name**: Enter a descriptive name for the Deployment.
   * **Description**: (Optional) Enter a description for your Deployment.
   * **Cluster**: Select the cluster for your Deployment.
   * **Release Name**: (Optional) If enabled, enter a custom release name.
   * **Image Version**: Select the Astro Runtime.
   * **Dag Deployment Method**: Choose from Image, Git Sync, or Dag Only Deployment. Additional fields may appear based on your selection.
   * **Executor and resources**: Select an executor, Local, Celery, or Kubernetes, and set resource options as needed for your environment.

3. Click **Create Deployment** and wait a few moments. After the Deployment is created, you can access the **Settings** page of your new Deployment.

On this tab you can modify resources for your Deployment. Specifically, you can:

* Choose a strategy for how you configure resources to Airflow components. See [Customize resource usage](/docs/astro-private-cloud/v-2-x/customize-resource-usage).
* Select an Airflow executor.
* Allocate resources to your Airflow scheduler, API server, triggerer, and data processor.
* Set scheduler count
* Add extra capacity (*Kubernetes only*)
* Set worker count (*Celery only*)
* Adjust your worker termination grace period (*Celery only*)


# Configure a custom registry for Deployment images
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/custom-image-registry

Replace Astronomer's built-in container image registry with your own.

Astro Private Cloud (APC) includes access to a Docker image registry that is installed by the Astronomer Helm chart, for data plane and unified mode. Every time a user deploys to APC, a Docker image is generated and pushed to this registry. Depending on your deploy method, these Docker images can include OS and Python dependencies, Dag code, and the Airflow service.

Using the Astronomer-provided container image registry shipped with APC is recommended when you're getting started and your team is comfortable deploying code. However, the Astronomer registry might not meet your organization's security requirements.

If your organization can't support the Astronomer default internal registry, you can configure a custom container image registry. This option is best suited for organizations who require additional control for security and governance reasons. Using a custom registry provides your organization with the opportunity to scan images for CVEs, malicious code, and unapproved Python and OS-level packages contained in Docker images.

<Info>A custom registry can still connect to public networks or internet. Therefore, this procedure is different if you're installing Astronomer in an air gapped environment. If you need to create a custom registry for a system that can't connect to the public networks or internet, follow the configuration steps in the **Air gapped** tabs.</Info>

<Warning>
  These instructions don't apply to images hosted on Amazon Elastic Container Registry (ECR). [Credentials for ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/security_iam_service-with-iam.html) have a limited lifespan and are unsuitable for using on APC. To use AWS ECR to serve images for Astro Private Cloud, you must grant permissions for the following actions to the Kubernetes Nodes IAM Role.

  ```json wrap theme={null}

    "ecr:GetDownloadUrlForLayer",
    "ecr:BatchGetImage"

  ```
</Warning>

## Prerequisites

* Helm.
* kubectl.
* Astro CLI version 1.3.0+.
* A custom container image registry.
* A process for building and pushing your Astro projects as images to your custom registry.

## Setup

<Tabs>
  <Tab title="Standard">
    1. Create a secret for the container repository credentials in your Astronomer namespace:

       ```bash wrap theme={null}
       kubectl -n <astronomer-platform-namespace> create secret docker-registry <name-of-secret> \
         --docker-server=<your-registry-server> \
         --docker-username=<your-name> \
         --docker-password=<your-password> \
         --docker-email=<your-email>
       ```

       To have Astro Private Cloud sync the registry credentials to all Deployment namespaces, add the following annotation:

       ```bash wrap theme={null}
       kubectl -n <astronomer-platform-namespace> annotate secret <name-of-secret> "astronomer.io/commander-sync"="platform=astronomer"
       ```

    <Info>
      To use different registries for each Deployment, create the same secret in each Deployment namespace instead of your Astronomer namespace. Make sure to specify different custom registries using `--docker-server`. If you don't need to synch your secrets between Deployments, you don't need to add the secret annotation.
    </Info>

    2. In the APC UI, go to your **Clusters** page and select your cluster.
    3. In the cluster details page, click **Edit**. In the **Cluster Deployments Configuration** YAML editor, click **Find** and search for the feature you want to override. Then add the following override in the appropriate YAML field:

       ```yaml wrap theme={null}
       astronomer:
         houston:
           config:
             deployments:
               deploymentImagesRegistry:
                 updateDeploymentImageEndpoint:
                   enabled: true
             registry:
               protectedCustomRegistry:
                 enabled: true
                 updateRegistry:
                   enabled: true
                   host: <airflow-image-repo>
                   secretName: <name-of-secret>
       ```

       For details on using the UI for configuration, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).

    <Info>
      To use different registries for each Deployment, omit the `astronomer.houston.config.deployments.registry.protectedCustomRegistry.updateRegistry.host` setting.

      If you do set the host, use the format `<registry>/<repo-name>/<subdirectory>`, where `<repo-name>/<subdirectory>` includes the repository name and optionally a subdirectory within that repository.
    </Info>

    4. Save and apply your changes in the UI.
    5. For any existing Deployments, run the following command to sync the registry credentials.

       ```bash wrap theme={null}
       kubectl create job -n <astronomer-platform-namespace> --from=cronjob/<platform-release-name>-config-syncer upgrade-config-synchronization
       ```

       <Info>
         If you're using different registries for each Deployment, skip this step.
       </Info>
  </Tab>

  <Tab title="Air gapped">
    ### Air gapped

    1. Create a secret for the container repository credentials in your Astronomer namespace:

       ```bash wrap theme={null}
       kubectl -n <your-namespace> create secret docker-registry <name-of-secret> \
         --docker-server=<your-registry-server> \
         --docker-username=<your-name> \
         --docker-password=<your-password> \
         --docker-email=<your-email>
       ```

       To have Astro Private Cloud sync the registry credentials to all Deployment namespaces, add the following annotation:

       ```bash wrap theme={null}
       kubectl -n <astronomer-platform-namespace> annotate secret <name-of-secret> "astronomer.io/commander-sync"="platform=astronomer"
       ```

       <Info>
         To use different registries for each Deployment, create the same secret in each Deployment namespace instead of your Astronomer namespace. Make sure to specify different custom registries using `--docker-server`. You don't need to add the annotation if you're not syncing secrets between Deployments.
       </Info>

    2. In the APC UI, go to your **Clusters** page and select your cluster.

    3. In the cluster details page, click **Edit**. In the **Cluster Deployments Configuration** YAML editor, click **Find** and search for the feature you want to override. Then add the following override in the appropriate YAML field:

       ```yaml wrap theme={null}
       astronomer:
         houston:
           config:
             deployments:
               deploymentImagesRegistry:
                 updateDeploymentImageEndpoint:
                   enabled: true
               helm:
                 airflow:
                   defaultAirflowRepository: <airflow-image-repo>
                   images:
                     airflow:
                       repository: <airflow-image-repo>
             registry:
               protectedCustomRegistry:
                 enabled: true
                 baseRegistry:
                   enabled: true
                   host: <airflow-image-repo>
                   secretName: <name-of-secret-containing-image-repo-creds>
                 updateRegistry:
                   enabled: true
                   host: <airflow-image-repo>
                   secretName: <name-of-secret-containing-image-repo-creds>
       ```

       For details on using the UI for configuration, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).

    <Info>
      To use different registries for each Deployment, omit the `astronomer.houston.config.deployments.registry.protectedCustomRegistry.updateRegistry.host` setting.

      If you do set the host, use the format `<registry>/<repo-name>/<subdirectory>`, where `<repo-name>/<subdirectory>` includes the repository name and optionally a subdirectory within that repository.
    </Info>

    4. Save and apply your changes in the UI.
    5. For any existing Deployments, run the following command to sync the registry credentials. If you're using different registries for each Deployment, you can skip this step.

       ```bash wrap theme={null}
       kubectl create job -n <astronomer-platform-namespace> --from=cronjob/<platform-release-name>-config-syncer upgrade-config-synchronization
       ```
  </Tab>
</Tabs>

## Push code to a custom registry

You can use the Astro CLI to build and push images to your custom registry. Based on the Helm configurations in your Astronomer cluster, the Astro CLI automatically detects your custom image registry and pushes your image to it. It then calls the APC API to update your Deployment to pull the new image from the registry.

After you configure your custom registry, open your Astro project and run:

```sh wrap theme={null}
astro deploy
```

Alternatively, you can run a GraphQL mutation to update the image in your Deployment after manually pushing the image to the custom registry. This can be useful for automating code deploys using CI/CD.

You can run a GraphQL mutation to update the image in your Deployment after manually pushing the image to a custom registry. This can be useful for automating code deploys using CI/CD.

At a minimum, your mutation has to include the following:

```graphql wrap theme={null}
mutation updateDeploymentImage {
        updateDeploymentImage(
                releaseName: "<deployment-release-name>", # for example "analytics-dev"
                image: "<host>/<image-name>:<tag>",  # for example docker.io/cmart123/ap-airflow:test4
                runtimeVersion: "<runtime-version-number>" # for example "5.0.6"
        )
        {
                id
        }
}
```

Alternatively, you can run this same mutation using cURL:

```bash wrap theme={null}
curl 'https://houston.BASEDOMAIN/v1' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: <your-token>' \
  --data-binary '{"query":"mutation updateDeploymentImage {updateDeploymentImage(releaseName: \"<deployment-release-name>\", image: \"<host>/<image-name>:<tag>\",runtimeVersion: \"<runtime-version-number>\"){id}}"}'
```


# Create custom roles on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/custom-roles

Create, assign, and manage custom roles with fine-grained permissions on Astro Private Cloud.

In addition to the built-in System, Workspace, and Deployment roles described in [User roles and permissions](/docs/astro-private-cloud/v-2-x/role-permission-reference), Astro Private Cloud supports custom roles. A custom role is a role you define yourself, made up of any combination of permissions from the [permission catalog](/docs/astro-private-cloud/v-2-x/role-permission-reference#custom-role-permission-catalog), and assigned to users, Teams, or service accounts through the Astro Private Cloud UI.

Custom roles differ from the built-in roles in a few ways:

* You define them through the Astro Private Cloud UI, not your `values.yaml` file.
* They're stored in the platform's database, not your Helm chart configuration.
* You can create any number of them.
* Each one applies at a single scope: System, Cluster, Workspace, or Deployment.

Enabling custom roles doesn't change the built-in roles. Custom roles are an additional option for organizations that need permission sets the three built-in tiers (Viewer, Editor, Admin) don't cover.

Custom roles are turned off by default. A System Admin enables them at the platform level before anyone can create or assign a custom role. See [Enable custom roles](#enable-custom-roles).

<Note>
  **Astro Private Cloud 2.1**

  This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
</Note>

## Prerequisites

* System Admin access to Astro Private Cloud.
* Custom roles enabled in your `values.yaml` file. See [Enable custom roles](#enable-custom-roles).

## Enable custom roles

Custom roles are disabled by default. To enable them, add the following to your `values.yaml` file:

```yaml theme={null}
astronomer:
  houston:
    config:
      customRBAC:
        enabled: true
```

Then, push the configuration change to your platform. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

After the change takes effect, a **Roles and Permissions** page appears in the platform's navigation for System Admins.

## Understand role scope

Every custom role has a scope, set when you create it:

* **System**: Applies across the entire platform. A System-scoped role can grant permissions at the System, Cluster, Workspace, or Deployment level.
* **Cluster**: Applies to a single cluster. A cluster-scoped role can only grant cluster-level permissions.
* **Workspace**: Applies to a single Workspace. A Workspace-scoped role can grant Workspace-level permissions, Deployment-level permissions, or both.
* **Deployment**: Applies to a single Deployment. A Deployment-scoped role can only grant Deployment-level permissions.

Scope determines two things: which permissions a role can include, and what happens when you assign it.

<Warning>
  If a Workspace-scoped role includes Deployment-level permissions, assigning that role to a user grants them those permissions on all Deployments in the Workspace, including Deployments created after the assignment. Assign Workspace-scoped roles with this in mind.
</Warning>

A role is a reusable template, not tied to a specific Workspace or Deployment. For example, you can create a single Deployment-scoped role named `Pipeline Developer` and assign it to different users across different Deployments.

## Create a custom role

<Steps>
  <Step title="Open the role builder">
    In the Astro Private Cloud UI, click **Roles and Permissions** in the navigation, then click **Create Role**.
  </Step>

  <Step title="Name the role and set its scope">
    Enter a **Role name** and, optionally, a **Description**. Select a **Scope** from the dropdown. The scope determines which permissions you can select in the following step.
  </Step>

  <Step title="(Optional) Start from an existing role">
    To use another role's permissions as a starting point, select it from **Start from an existing role**. You can select any built-in or custom role at the same scope. Its permissions populate the permission picker, where you can add or remove permissions. Select **Start from blank** to begin with no permissions selected.
  </Step>

  <Step title="Select permissions">
    Select the permissions this role grants. Permissions are grouped by category (for example, Dag Operations, Variables and Connections). Hover over the info icon next to a permission to see what it does.

    Some permissions depend on others. For example, selecting a permission to update or delete a resource automatically selects the permission to view that resource, since a user can't act on something they can't see. Automatically selected permissions display an **auto** badge and can't be individually deselected. To remove one, deselect the permission that depends on it first.

    <Note>
      Dag-level permissions (for example, in the Dag Operations category) apply on both Airflow 2 and Airflow 3. Airflow 2's security model uses the Dag-read permission as the parent check for Dag runs, task instances, logs, and more, so it isn't optional there.
    </Note>
  </Step>

  <Step title="Create the role">
    Click **Create Role**.
  </Step>
</Steps>

## Clone a role

Cloning creates a new, independent role with the same permissions as the source. You can clone a built-in role or an existing custom role, and edit the clone afterward without affecting the original.

<Steps>
  <Step title="Open the source role">
    Open the role you want to clone, then click **Clone Role**.
  </Step>

  <Step title="Name the new role">
    Enter a name for the new role, then click **Clone Role** to confirm.
  </Step>
</Steps>

Cloning copies only the role's permissions. It doesn't copy any assignments from the source role.

<Tip>
  Cloning a built-in role is the only way to customize it beyond what `values.yaml` allows, since built-in roles themselves can't be edited directly. See [User roles and permissions](/docs/astro-private-cloud/v-2-x/role-permission-reference) for the default permissions of each built-in role.
</Tip>

## Edit a custom role

Open the role and change its name, description, or permissions, then click **Save changes**. Built-in roles are read-only. To customize a built-in role's permissions, clone it first.

Editing a role doesn't change who is assigned to it. Adding or removing permissions updates the effective permissions of everyone currently assigned to that role.

## Delete a custom role

Open the role and click **Delete role**. If the role has active assignments, you must revoke them first. Deleting a role can't be undone.

Built-in roles can't be deleted.

## Assign a role to a user, Team, or service account

You can assign both built-in and custom roles to users, [Teams](/docs/astro-private-cloud/v-2-x/manage-permissions), and service accounts from the same interface. On the Users, Teams, or Service Accounts list for a System, Workspace, or Deployment, click **Edit roles** next to a principal to open the role assignment panel.

The panel shows the roles currently assigned at that scope, plus a preview of the principal's effective permissions. Select or deselect roles, then click **Save changes**.

<Warning>
  You can only assign a role that grants permissions you hold yourself. If a role includes a permission you don't have, assigning it fails. This applies only to assigning a role, not to creating or editing one: a System Admin can build a role with any available permissions, but can only assign it up to their own level of access.
</Warning>

<Note>
  Creating, updating, or deleting a custom role always requires System-scope role-management permissions, regardless of what scope the role itself targets. A Workspace or Deployment Admin can be *assigned* a Workspace- or Deployment-scoped role, but authoring the role definition itself is a System Admin action.
</Note>

## Revoke a role assignment

From the same role assignment panel, deselect the role and click **Save changes**. You can't revoke an assignment if doing so leaves a System, Workspace, Deployment, or cluster without an admin.

<Warning>
  This guard applies to revoking a role assignment. It doesn't apply to editing a role's permissions. If you remove an admin-granting permission from a role that's currently assigned to someone, nothing stops you from leaving that scope without an admin. Always confirm another admin exists before narrowing a role that grants admin-level access, especially at System scope: if no one retains System Admin permissions, you can lock your organization out of platform administration entirely.
</Warning>

## Known issues and limitations

* **On Airflow 2, a custom `webserver_config.py` file can be silently overwritten.** If you maintain your own `webserver_config.py` on an Airflow 2 Deployment — for example, to connect the Airflow UI to your company's LDAP or single sign-on system — those settings can be overwritten without warning. This can happen whenever a Deployment is deployed or brought under Astronomer's management (adopted), whether or not custom roles are turned on. If you maintain a custom `webserver_config.py` on an Airflow 2 Deployment, check after any deploy or adoption that its settings are still in effect. A fix that lets your file and ours coexist is planned for a future release.

* **On Airflow 2, there's currently no way to confirm that a custom role is actually being enforced.** In rare cases, the platform can fail to fully set up enforcement for a custom role on an Airflow 2 Deployment. Today, that failure isn't shown anywhere — not during deploy, not in the platform. If it happens, the person just gets the broader default access level (Viewer, Editor, or Admin) instead of the narrower access their custom role was meant to give them, with nothing to indicate that's what happened. We're planning a fix that makes this visible instead of silent. Until then, after assigning a custom role on an Airflow 2 Deployment, it's worth confirming the person actually has only the access the role grants, not more.

* **Not all custom roles are visible on the Roles and Permissions screen when you have many of them.** If you've created a large number of custom roles, some of them can be cut off and hidden on the Roles and Permissions screen. Until this is fixed, use the **Scope** dropdown to filter roles by scope (for example, **Workspace**) to bring the missing roles into view.

* **Editing a custom role's permissions may not update the UI right away.** After you change a custom role's permissions, other views that show that role — user, Team, and service account lists, role assignments, and admin views — can keep showing the old permissions until you refresh or navigate away and back. The change is saved; only the display is stale. Refresh the page to see the current permissions.

* **Role assignments may not appear in the Show all view right away.** On the Roles and Permissions screen, newly assigned roles can be missing from the **Show all** view until you refresh the page or toggle the **Hide others** button. The assignment is saved; only the display is stale.

These are known and already scoped for a future release.

## What's next

* To see who has access to what, or what a specific user, Team, or service account can do, see [Permission audit](/docs/astro-private-cloud/v-2-x/permission-audit).
* For the full list of permissions custom roles can grant, see [Custom role permission catalog](/docs/astro-private-cloud/v-2-x/role-permission-reference#custom-role-permission-catalog).


# Customer-created database users
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/customer-created-db-users

SQL setup required when you create per-Deployment database login roles yourself for Astro Private Cloud data plane failover.

This page covers the database setup required when you choose the *Customer manages users* model for [data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover#per-deployment-database-users). In this model, you create the per-Deployment login roles yourself, and Astro Private Cloud (APC) still needs the privileges it uses to fence those roles during failover.

For the high-level overview and the alternative *APC manages users* model, see [Per-deployment database users](/docs/astro-private-cloud/v-2-x/data-plane-failover#per-deployment-database-users).

## Roles APC needs

For every Apache Airflow Deployment, APC requires the following on the Deployment's metadata database:

* An *owner role* on the metadata database, so APC can manage the schema and switch `CONNECT` privileges between the two login roles during failover.
* A *connection-terminator role* that can terminate active sessions for the two login roles, so APC can fence the source data plane during failover without needing full access to those login roles.
* *Membership* in both of these roles for the deployment orchestrator database user.

You create the two login roles per Deployment (one per data plane cluster) and grant them `CONNECT` on the metadata database.

## PostgreSQL example

For a Deployment whose metadata database is `airflow_db_<deployment>` and whose per-data-plane login roles are `dp1_user_<deployment>` and `dp2_user_<deployment>`, the full setup is:

```sql wrap theme={null}
CREATE ROLE airflow_db_<deployment>_owner NOLOGIN;
CREATE ROLE airflow_conn_killer_<deployment> NOLOGIN NOINHERIT;

ALTER DATABASE airflow_db_<deployment> OWNER TO airflow_db_<deployment>_owner;

REVOKE CONNECT ON DATABASE airflow_db_<deployment> FROM PUBLIC;
GRANT CONNECT ON DATABASE airflow_db_<deployment> TO dp1_user_<deployment>;
GRANT CONNECT ON DATABASE airflow_db_<deployment> TO dp2_user_<deployment>;

GRANT dp1_user_<deployment> TO airflow_conn_killer_<deployment> WITH INHERIT FALSE, SET FALSE;
GRANT dp2_user_<deployment> TO airflow_conn_killer_<deployment> WITH INHERIT FALSE, SET FALSE;

GRANT airflow_db_<deployment>_owner TO commander_user;
GRANT airflow_conn_killer_<deployment> TO commander_user;
```

Replace `commander_user` with the deployment orchestrator database user configured for your APC installation, and `<deployment>` with the identifier you use for each Airflow Deployment.

## Related documentation

* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)
* [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover)
* [Trigger a data plane failover](/docs/astro-private-cloud/v-2-x/trigger-data-plane-failover)


# Customize Deployment release names
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/customize-deployment-releases-name

Customize the release name for an Airflow Deployment on Astro Private Cloud.

An Airflow Deployment's release name on Astro Private Cloud is a unique, immutable identifier for that Deployment. The release name corresponds to its Kubernetes namespace and that renders in Grafana and other platform-level monitoring tools.

By default, release names are randomly generated in the following format: `noun-noun-<4-digit-number>`. For example: `elementary-zenith-7243`. Alternatively, you can customize the release name for a Deployment if you want all namespaces in your cluster to follow a specific style.

To customize the release name for a Deployment as you're creating it, you first need to enable the feature on your data plane cluster. To do so:

1. In the APC UI, go to your **Clusters** page and select your cluster.

2. In the cluster details page, click **Edit**. In the **Cluster Deployments Configuration** YAML editor, click **Find** and search for the feature you want to override. Then add the following override in the appropriate YAML field:

   ```yaml wrap theme={null}
   namespaceManagement:
     manualReleaseNames:
       enabled: true # Allows you to set your release names
   ```

   For details on using the UI for configuration, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).

3. Save and apply your changes in the UI.

After applying this change, the **Release Name** field in the Astro Private Cloud UI becomes configurable.


# Customize Deployment CPU and management resources per component
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/customize-resource-usage

Scale Deployments directly using non-proportional CPU and memory specifications.

<Warning>
  **Astronomer Units (AUs) Removed**

  **Astronomer Units (AUs)** are no longer supported.
  All Deployments must now specify **CPU and memory directly** when configuring resources.
  This change provides more clarity, flexibility, and aligns with Kubernetes-native resource management.
</Warning>

When you create a new Astro Private Cloud (APC) Deployment, you now specify the **exact amount of CPU and memory** that power its core components.

For example:

* You might need to allocate **significantly more memory than CPU** to your worker Pods if you run memory-intensive tasks.
* At the same time, your scheduler may need more memory than CPU.

Deployment Admins can:

* Assign **exact CPU and memory values** to each component.
* Ensure that Pods use these values as both **requests** and **limits**, providing predictable scheduling and resource enforcement.

In Astro Private Cloud, you configure resources by setting **CPU** and **memory** directly for each Apache Airflow component in the UI or with the [APC API](/docs/astro-private-cloud/v-2-x/houston-api).

## Set CPU and memory resources in the Astro Private Cloud UI

### Configure worker resources

1. Navigate to your **Deployment** in the APC UI.
2. Open the **Settings** page for that Deployment.
3. In the **Execution Environment** section, under **Workers**, set the desired **CPU** and **memory** values. These values will be applied to all worker Pods.
4. Click **Deploy Changes**.

### Configure core component resources

1. Navigate to your **Deployment** in the APC UI.
2. Open the **Settings** page for that Deployment.
3. In the **Core Resources** section, set CPU and memory for the following components individually:
   * **Webserver / API Server (Airflow 3 only)**
   * **Scheduler**
   * **Triggerer**
   * **Dag Processor**

<Warning>
  When setting CPU resources for the **Scheduler** or **Workers** using milliCPU (mCPU) values, the values **must be multiples of 100** (such as `100`, `200`, `300`, etc).
  Non-multiples of 100 may cause Deployment failures.
</Warning>

4. Click **Deploy Changes**.

<Note>All values set here are applied as both **requests** and **limits** for Pods, ensuring consistent scheduling.</Note>

<Info>For memory-intensive or CPU-intensive workloads, adjust individual components accordingly. For example, you can give workers more memory for data-heavy tasks.</Info>

## Set Deployment component resources with APC API

You can also set resource specifications with the [APC API](/docs/astro-private-cloud/v-2-x/houston-api).

### Set defaults in the config

You can set custom defaults, limits, and minimums for Airflow components by defining them in the config file. When you define a component using the following example, Deployments are created with those resource values, and the UI reflects the same defaults.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      components:
        - name: scheduler
          custom:
            default:
              cpu: 1000
              memory: 2000
            minimum:
              cpu: 500
              memory: 1000
            limit:
              cpu: 6000
              memory: 12000
          extra:
            - name: replicas
              default: 1
              minimum: 1
              limit: 4
              minAirflowVersion: "2.0.0"
```

## Set resource quotas per Deployment

Astro Private Cloud lets you override default quota calculations by specifying resource quotas directly in the APC API’s Deployment payload. Use the `quotas` object to set custom CPU and memory requests/limits for your Deployment. If you don't provide `quotas`, Astronomer uses the platform’s default quota logic.

### Default quotas config

Include the requests and limits in your `quotas` parameter of the `deployment upsert` API payload. If you set quotas, make sure your values aren't less than required platform minimums and don't exceed the allowed platform maximums.

The following example shows the `quotas` object and the platform defaults for CPU and memory requests and limits.

```json wrap theme={null}
"quotas": {
  "requests": {
    "cpu": 1,
    "memory": "1920Mi"
  },
  "limits": {
    "cpu": 2,
    "memory": "1920Mi"
  }
}
```

### Configuration options

Use custom quotas if you need to guarantee or constrain Deployment resources beyond the system-determined logic. If you don't set quotas, the Deployment will use platform default resource constraints.

<Warning>
  Deployments will fail due to insufficient quotas if you set resource quotas to less than or greater than the Astronomer platform-provided minimum or maximum limits.

  **Typical platform defaults**:

  * CPU: 10 vCPU
  * Memory: 28272Mi (\~28Gi)
</Warning>

| Configuration Name       | Component  | Description                                      | Default Value (if not set)      | Accepted Values                       |
| ------------------------ | ---------- | ------------------------------------------------ | ------------------------------- | ------------------------------------- |
| `quotas`                 | Deployment | Optional JSON object with custom resource quotas | Not set; platform logic is used | JSON object (`requests` and `limits`) |
| `quotas.requests.cpu`    | Deployment | CPU quota guaranteed (requested)                 | Platform default                | Number (integer or float)             |
| `quotas.requests.memory` | Deployment | Memory quota guaranteed (requested)              | Platform default                | String (`"1920Mi"`, `"2Gi"`)          |
| `quotas.limits.cpu`      | Deployment | CPU quota maximum (limit)                        | Platform default                | Number (integer or float)             |
| `quotas.limits.memory`   | Deployment | Memory quota maximum (limit)                     | Platform default                | String (`"1920Mi"`, `"28272Mi"`)      |

## Configure Deployment-level limits for resource usage

Astro Private Cloud limits the amount of resources that can be used by all Pods in a Deployment by creating and managing a `LimitRange` and `ResourceQuota` for the namespace associated with each Deployment.

These values are automatically adjusted to account for the resource requirements of various components.

You can add additional resources, beyond the standard amount allocated based on the resource requirements of standing components, to the `LimitRange` and `ResourceQuota`. Add resources by configuring `astronomer.houston.config.deployments.maxExtraCapacity` and `astronomer.houston.config.deployments.maxExtraPodCapacity` to account for the requirements of KubernetesExecutor and `KubernetesPodOperator` tasks.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        maxExtraCapacity:
          cpu: 40000 # in milliCPUs (m)
          memory: 153600 # in MiB (Mi)
        maxPodCapacity:
          cpu: 3500 # in milliCPUs (m)
          memory: 13440 # in MiB (Mi)
```

### Configurable components

<Tip>KubernetesExecutor task Pod sizes are created on an as-needed basis and don't have persisting resource requirements. Their resource requirements are [configured at the task level](/docs/astro-private-cloud/v-2-x/kubernetes-executor#configure-the-worker-pod-for-a-specific-task).</Tip>

Configurable components include:

#### Airflow scheduler

```yaml wrap theme={null}
- name: scheduler
  custom:
    default:
      cpu: 1000
      memory: 2000
    minimum:
      cpu: 500
      memory: 1000
    limit:
      cpu: 6000
      memory: 12000
  extra:
    - name: replicas
      default: 1
      minimum: 1
      limit: 4
      minAirflowVersion: "2.0.0"
```

#### Airflow Dag processor

<Note>
  To enable a standalone Dag processor, set the `airflowComponents.dagProcessor.enabled` feature flag to `true` at the cluster level. In the APC UI, go to your **Clusters** page, select your cluster, click **Edit** in the **Deployment Configuration** section, and add the following override to the **Configuration Override** field:

  ```yaml wrap theme={null}
  astronomer:
    houston:
      config:
        deployments:
          airflowComponents:
            dagProcessor:
              enabled: true
  ```

  For details on using the UI for configuration, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).
</Note>

<Info>
  You can configure extra containers for the Dag processor in the `values.yaml` file. For example:

  ```yaml wrap theme={null}
  houston:
    config:
      deployments:
        helm:
          airflow:
            dagProcessor:
              extraContainers:
                - name: <extra-container>
                  env:
                    - name: DAGS_LOCATION
                      value: /dags
                    - name: SECRET_LOCATION
                      value: /secrets
                    - name: SYNC_INTERVAL
                      value: "15"
                  image: ubuntu:latest
                  command: ["/bin/bash", "-c", "--"]
                  args: ["<arg-1>; <arg-2>; <arg-3>;"]
  ```
</Info>

```yaml wrap theme={null}
- name: dagProcessor
  custom:
    default:
      cpu: 1000
      memory: 3840
    minimum:
      cpu: 1000
      memory: 3840
    limit:
      cpu: 3000
      memory: 11520
  extra:
    - name: replicas
      default: 0
      minimum: 0
      limit: 3
```

#### Airflow webserver

```yaml wrap theme={null}
- name: webserver
 custom:
   default:
     cpu: 1000
     memory: 2000
   minimum:
     cpu: 500
     memory: 1000
   limit:
     cpu: 6000
     memory: 12000
```

#### StatsD

```yaml wrap theme={null}
- name: statsd
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 3000
      memory: 11520
```

#### Database connection pooler (PgBouncer)

```yaml wrap theme={null}
- name: pgbouncer
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 200
      memory: 768
```

#### Celery diagnostic web interface (Flower)

```yaml wrap theme={null}
- name: flower
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 200
      memory: 768
```

#### Redis

```yaml wrap theme={null}
- name: redis
 custom:
   default:
     cpu: 200
     memory: 768
   minimum:
     cpu: 200
     memory: 768
   limit:
     cpu: 200
     memory: 768
```

#### Celery workers

```yaml wrap theme={null}
- name: workers
 custom:
   default:
     cpu: 1000
     memory: 3840
   minimum:
     cpu: 100
     memory: 384
   limit:
     cpu: 3000
     memory: 11520
 extra:
   - name: terminationGracePeriodSeconds
     default: 600
     minimum: 0
     limit: 36000
   - name: replicas
     default: 1
     minimum: 1
     limit: 20
```

#### Triggerer

```yaml wrap theme={null}
- name: triggerer
  custom:
    default:
      cpu: 1000
      memory: 2000
    minimum:
      cpu: 500
      memory: 1000
    limit:
      cpu: 6000
      memory: 12000
  extra:
    - name: replicas
      default: 1
      minimum: 0
      limit: 4
      minAirflowVersion: "2.2.0"
```


# Data plane failover
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/data-plane-failover

Understand how Astro Private Cloud data plane failover works and what components it requires.

Data plane failover is a resiliency feature that moves all Apache Airflow Deployments from a source data plane cluster to a destination data plane cluster. When you trigger a failover, Astro Private Cloud (APC) applies each Deployment's configuration and secrets to the destination cluster, gets it up and running there, and cleans up the source side with minimal manual intervention.

Failover is a full-cluster operation — every Deployment on the source cluster is included. It is asynchronous: after you submit a request, the platform drives execution through a state machine until every Deployment is running on the destination cluster or has failed with an error that requires operator attention.

## How it works

A failover request moves through the following stages:

1. You submit a failover request from the APC UI, specifying a source cluster, a destination cluster, and a failover mode.
2. APC creates a `FailoverRequest` record and transitions it to `IN_PROGRESS`.
3. Navigator (the control plane component that orchestrates the failover) creates one *mission* per Deployment included in the failover request, along with a pair of *flights* for each mission — one targeting the source cluster and one targeting the destination cluster.
4. Dispatcher workers dispatch each flight to its target cluster, where Pilot (the data plane execution agent) picks it up and runs it.
5. Pilot executes each flight plan — the series of steps that make up a flight — to either bring the Deployment up on the destination cluster or drain and delete the Deployment on the source cluster.
6. After all missions complete, Navigator marks the `FailoverRequest` as `SUCCEEDED` or `FAILED`.

### Failover modes

| Mode           | Behavior                                                                                                                                                                                                                                      |
| -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Controlled** | Drains Airflow Deployment components on the source cluster and waits for in-flight tasks to finish, up to a configured timeout, before promoting the destination. Use for planned maintenance or migrations where task loss isn't acceptable. |
| **Forced**     | Promotes the destination cluster immediately without waiting for source Deployments to drain. Use when the source cluster is unreachable or when speed is the priority.                                                                       |

### Cluster eligibility

The **Trigger Failover** button in the UI is active only when `failoverEnabled` is `true` on the source cluster, when `external-secrets.enabled` is `true` in both the source and destination clusters, `global.dataPlaneFailover.externalSecretManagerName` is set, and a valid, authenticated `ClusterSecretStore` exists.

The destination cluster dropdown shows only clusters that APC considers schedulable targets for the selected source. A cluster appears as a valid target when it is registered, healthy, and has no pending failover operations targeting it as a destination.

APC doesn't compare APC versions between the source and destination data planes before a failover. You are responsible for keeping the source and destination clusters on compatible APC versions.

## Components

Data plane failover adds several components that aren't deployed in a standard APC installation. Each component runs on either the control plane or the data plane, as described in the following table.

| Component      | Plane   | Description                                                                                                                                                                                                     |
| -------------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Navigator**  | Control | Control plane component that decides when and where each Airflow Deployment moves by creating and managing `FailoverRequest` and `Mission` records and producing the flights that carry out each mission.       |
| **DP-Link**    | Control | Maintains persistent gRPC streams to each registered data plane. Monitors heartbeats and updates cluster health status (`HEALTHY`, `DEGRADED`, `UNREACHABLE`).                                                  |
| **Dispatcher** | Control | APC Worker process that issues `StartFlight` remote procedure calls to the deployment orchestrator on the data plane.                                                                                           |
| **Pilot**      | Data    | Data plane component that claims flights from the Flightdeck queue and executes flight plans: namespace creation, secret application, Deployment upsert, database fencing, drain Deployment, delete Deployment. |
| **Flightdeck** | Data    | PostgreSQL or MySQL-backed queue table (`dp_flights`) shared by the deployment orchestrator and Pilot. The deployment orchestrator writes flights; Pilot claims and executes them.                              |

### Secret replication

APC uses the [External Secrets Operator (ESO)](https://external-secrets.io) to replicate Airflow secrets between data planes. When failover is enabled:

* The deployment orchestrator on the source cluster creates `PushSecret` custom resources that write Airflow secrets (fernet key, environment variables, and database credentials) into an external secrets store through a `ClusterSecretStore`.
* The deployment orchestrator on the destination cluster creates `ExternalSecret` custom resources that pull those secrets from the same store into the destination namespace.

Both the source and destination data plane clusters must be able to reach the same external secrets store.

<Note>
  Hashicorp Vault support for data plane failover is introduced in Astro Private Cloud 2.1. If your external secrets store is Vault, each data plane cluster needs its own Kubernetes auth mount in Vault. Vault's `auth.kubernetes` method validates a service account token against one specific Kubernetes API server per mount, so a single mount can't authenticate more than one cluster. AWS Secrets Manager and Google Cloud Secret Manager don't have this constraint. See [Configure Hashicorp Vault for data plane failover](/docs/astro-private-cloud/v-2-x/configure-vault-data-plane-failover) for setup steps.
</Note>

## Database requirements

APC provisions logical databases and database users automatically when you create a Deployment, but it doesn't provision database servers. You must provide a database server hostname that is network-accessible from both the source and destination data plane clusters.

### Supported topologies

Two database server topologies are supported:

* *Shared server*: both clusters connect to the same database server hostname. This is typically a cloud-managed database server (for example, AWS RDS) reachable from both cluster networks.
* *Synchronized servers*: a primary database server is kept in sync with one or more replicas through a customer-managed replication mechanism. APC must connect through a single stable hostname or endpoint that always resolves to the active primary — the source and destination clusters don't point at their own per-cluster endpoints. During failover, you are responsible for promoting the replica and updating that endpoint to point at the new primary before initiating the APC data plane failover.

In either topology, you are responsible for setting up network access from your data plane clusters to the database server, and for managing replication, primary promotion, and endpoint cutover if you use the synchronized topology.

### Expected failover order

When performing regional failover with synchronized database servers, the expected sequence is:

1. Database replica promotion.
2. Endpoint updated to point to the new primary.
3. APC data plane failover initiated.

### How APC uses the database server

When a Deployment is created with failover enabled, APC provisions two sets of logical databases and credentials on the database server: an *active* set used by the source cluster and an *inactive* set used by the destination cluster. APC immediately blocks the inactive credentials from connecting after provisioning.

During failover, the deployment orchestrator performs *database fencing* for each Deployment: it revokes connect access from the active credentials and grants it to the inactive credentials. This ensures only one cluster writes to a given Deployment's logical databases at a time. Fencing is per Deployment, so individual Deployments can be migrated at different times during a single failover request.

### Per-Deployment database users

To prevent split-brain writes during failover, APC fences each Airflow Deployment at the database level using two database users per Deployment. APC supports two ownership models:

* *APC manages users*: You grant APC permission to create and manage Airflow database users (for example, `ALTER ROLE`), and APC handles the rest. You don't take any further action.
* *Customer manages users*: You create the two login roles per Airflow Deployment yourself (one per data plane cluster). APC still owns each Deployment's Airflow metadata database and fences those login roles during failover, so you grant the deployment orchestrator database user an owner role and a connection-terminator role for every Deployment. For the exact SQL setup, see [Customer-created database users](/docs/astro-private-cloud/v-2-x/customer-created-db-users).

## Container registry requirements

APC doesn't replicate container images between regions. You configure a single container registry endpoint per APC installation on the control plane, and every data plane in the installation pulls Airflow Deployment images from that endpoint. APC currently doesn't support different registry endpoints per region.

### Required capabilities

* You configure one externally managed container registry endpoint on the control plane, and every data plane uses it.
* The registry serves the same repository paths and tags from every region where a data plane may run, whether through a globally routed endpoint, customer-managed cross-region replication behind a single hostname, or another mechanism. An image reference like `registry.example.com/my-org/airflow:1.2.3` must resolve to the same image regardless of which data plane pulls it.
* Every data plane cluster has network access and credentials to pull from that endpoint.

### Replication and failover eligibility

If you back the registry endpoint with cross-region replication, replication latency determines when a Deployment is eligible to run on a destination data plane. If a Deployment's image hasn't yet replicated to the region serving the destination data plane, APC can't start that Deployment there — Pilot's upsert step fails until the image becomes available. Size your registry replication SLA to be faster than your expected failover window for the Deployments that must be able to fail over.

## Airflow log sink requirements

APC ships Airflow task logs from each data plane to an external Elasticsearch sink. After a Deployment moves between data planes, you still need to be able to read its task logs through the same UI, so the log sink topology matters for failover.

For details on configuring Vector and the Elasticsearch sink itself, see [Configure task log collection and exporting to ElasticSearch](/docs/astro-private-cloud/v-2-x/export-task-logs).

### Supported topologies

APC supports two Elasticsearch topologies for failover:

* *Single shared Elasticsearch endpoint*: Every data plane in the APC installation ships logs to the same Elasticsearch endpoint. Logs from the source and destination clusters land in the same backend, so post-failover log lookups continue to work without any further configuration. This is typically a multi-region or globally routed Elasticsearch service that all data planes can reach over the network.
* *Active-active Elasticsearch per region*: You run an Elasticsearch cluster in each region with bidirectional replication between them. You configure each regional data plane with its own regional Elasticsearch endpoint, and you manage the cross-region replication so that logs written by either side are visible from both. Each data plane writes to the closest endpoint, and queries from the control plane resolve against any region.

In either topology, you are responsible for sizing, securing, and operating the Elasticsearch infrastructure, and (in the active-active topology) for managing replication between regions.

## Prerequisites

Before enabling data plane failover, confirm the following:

* You have an APC installation with separate control and data plane clusters (`global.plane.mode: control` on the control plane, `global.plane.mode: data` on each data plane). Failover isn't supported in unified mode.
* You have an external secrets store supported for APC data plane failover. APC currently supports AWS Secrets Manager and Google Cloud Secret Manager through ESO. Hashicorp Vault support is introduced in Astro Private Cloud 2.1.
* You have configured a `ClusterSecretStore` custom resource in your data plane clusters that points to that secrets store.
* Your source and destination data plane clusters have network access to the external secrets store.
* You have an external sink for Airflow logs — an external Elasticsearch instance — that is reachable from every data plane cluster. Shipping logs to an external sink is required so that Airflow task logs remain accessible after a Deployment moves between data planes. For supported topologies, see [Airflow log sink requirements](#airflow-log-sink-requirements).
* You have a destination cluster that is registered with the APC control plane and is healthy.
* You have a single externally managed container registry endpoint, configured on the control plane, that every data plane in the APC installation can pull Airflow Deployment images from. For requirements, see [Container registry requirements](#container-registry-requirements). To configure the registry backend, see [Use a registry backend](/docs/astro-private-cloud/v-2-x/registry-backend).

## Limitations

The initial APC 2.0 release of data plane failover has the following limitations:

* **Image-based Deployments only**: Failover is supported only for image-based Airflow Deployments. Deployments that use git-sync or Dag-only deploy mechanisms aren't supported and can't be failed over.
* **New clusters and Deployments only**: Failover can't be enabled on a data plane cluster that already has Airflow Deployments on it. The feature is supported only on newly registered failover-enabled data plane clusters and on Deployments created on those clusters after you enable failover. In APC 2.0, existing Deployments on a pre-existing cluster can't be retroactively brought under failover management. Starting in APC 2.1, you can retrofit existing Deployments on a failover-enabled cluster with the connection failover requires instead of recreating them — see [Run a failover upgrade](/docs/astro-private-cloud/v-2-x/run-failover-upgrade).

## Related documentation

* [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover)
* [Configure Hashicorp Vault for data plane failover](/docs/astro-private-cloud/v-2-x/configure-vault-data-plane-failover)
* [Run a failover upgrade](/docs/astro-private-cloud/v-2-x/run-failover-upgrade)
* [Trigger a data plane failover](/docs/astro-private-cloud/v-2-x/trigger-data-plane-failover)
* [Customer-created database users](/docs/astro-private-cloud/v-2-x/customer-created-db-users)
* [Configure task log collection and exporting to ElasticSearch](/docs/astro-private-cloud/v-2-x/export-task-logs)
* [Use a registry backend](/docs/astro-private-cloud/v-2-x/registry-backend)
* [Data plane architecture](/docs/astro-private-cloud/v-2-x/data-plane-architecture)
* [Control plane architecture](/docs/astro-private-cloud/v-2-x/control-plane-architecture)


# Deregister a data plane cluster
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/deregister-data-plane

Safely remove a data plane cluster from your Astro Private Cloud control plane.

Use this guide to remove a data plane cluster from your Astro Private Cloud (APC) control plane.

<Info>Only a *System Admin* can deregister a data plane cluster.</Info>

## Prerequisites

* Make sure there are no running Airflow Deployments in the data plane cluster.
* Check that your automation configurations and scripts, such as your CI/CD or IaC, don't reference this cluster.

## Deregister using the UI

1. In the Astro Private Cloud UI, go to the **Clusters** page.
2. Click **See More \[...]** on the cluster that you want to deregister.
3. Click **Deregister Cluster**.
4. Click **Deregister**.

<Frame>
  <img alt="Deregister a cluster" />
</Frame>

This removes the control plane association. Now, platform services no longer target this data plane cluster.

<Note>
  Deregistering a data plane cluster doesn't delete existing platform resources such as namespaces, services, or persistent volumes in your cloud environment. If you want to fully remove these resources, you must manually delete the Helm release.
</Note>

## Deregister through the APC API

You can also deregister a data plane cluster by calling the `deregisterCluster` mutation on the APC API.

Send the request to `https://houston.<your-base-domain>/v1` with a system service account or System Admin user token in the `Authorization` header. For details, see [Authenticate to the APC API](/docs/astro-private-cloud/v-2-x/houston-api-authenticate).

```graphql wrap theme={null}
mutation {
  deregisterCluster(id: "<cluster-id>") {
    id
    name
  }
}
```

Argument reference:

| Argument | Type   | Required | Description                                                                                     |
| -------- | ------ | -------- | ----------------------------------------------------------------------------------------------- |
| `id`     | `Uuid` | Yes      | The unique identifier of the cluster to deregister. Find it with the `paginatedClusters` query. |

<Warning>
  This permanently deletes the cluster record from the control plane. Deregistration doesn't delete platform resources running in the data plane Kubernetes cluster (namespaces, services, persistent volumes). Remove those by uninstalling the data plane Helm release.
</Warning>

## Re-register a cluster

If you deregistered your data plane cluster by mistake, you can [register the same data plane cluster](/docs/astro-private-cloud/v-2-x/register-data-plane) again.

## Related

* [Clusters overview](/docs/astro-private-cloud/v-2-x/overview-data-plane-cluster)
* [Register a cluster](/docs/astro-private-cloud/v-2-x/register-data-plane)
* [Update cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster)
* [Manage cluster status](/docs/astro-private-cloud/v-2-x/cluster-status-management)
* [Authenticate to the APC API](/docs/astro-private-cloud/v-2-x/houston-api-authenticate)


# Enable data plane failover
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/enable-data-plane-failover

Configure the Helm values required to enable data plane failover on your Astro Private Cloud control plane and data plane clusters.

This guide walks you through enabling data plane failover on an existing Astro Private Cloud (APC) installation. You configure the control plane and each participating data plane cluster separately.

For a conceptual overview of the feature and its components, see [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover).

## Prerequisites

* A working APC installation with at least one control plane cluster (`global.plane.mode: control`) and at least two data plane clusters (`global.plane.mode: data`).
* A database server hostname that is network-accessible from both the source and destination data plane clusters. APC provisions the logical databases automatically, but the server itself must be reachable from both clusters. For supported topologies, see [Database requirements](/docs/astro-private-cloud/v-2-x/data-plane-failover#database-requirements).
* An external secrets store supported for APC data plane failover. APC currently supports AWS Secrets Manager and Google Cloud Secret Manager through the External Secrets Operator (ESO). Hashicorp Vault support is introduced in Astro Private Cloud 2.1.
* The External Secrets Operator (ESO) secret store configured on each data plane cluster — the `ClusterSecretStore`, or a synced `SecretStore` when you use namespace pools. See [Configure External Secrets Operator security](/docs/astro-private-cloud/v-2-x/configure-external-secrets-operator-security). The name you give the secret store is the value you provide for `global.dataPlaneFailover.externalSecretManagerName`.
* A single externally managed container registry endpoint, configured on the control plane, that serves the Apache Airflow Deployment images used by your Deployments to every region where a data plane may run, with the same repository paths and tags in each region. Every data plane cluster must be able to pull from this endpoint. APC currently supports only one registry endpoint per APC installation. For details, see [Container registry requirements](/docs/astro-private-cloud/v-2-x/data-plane-failover#container-registry-requirements). To configure the registry backend, see [Use a registry backend](/docs/astro-private-cloud/v-2-x/registry-backend).
* An external sink for Airflow logs — an external Elasticsearch instance — that is reachable from every data plane cluster, so that task logs remain accessible after a Deployment moves between data planes. For supported topologies, see [Airflow log sink requirements](/docs/astro-private-cloud/v-2-x/data-plane-failover#airflow-log-sink-requirements). For details on how Astro Private Cloud collects and exports task logs to Elasticsearch, see [Configure task log collection and exporting to ElasticSearch](/docs/astro-private-cloud/v-2-x/export-task-logs).
* Helm 3.6 or later.
* A Kubernetes `ClusterRole` for the identity running `helm install` or `helm upgrade` on each data plane cluster. ESO installs cluster-scoped CRDs, which require cluster-level permissions.
* Access to your APC Helm values files.

## Configure the External Secrets Operator secret store

Data plane failover uses the External Secrets Operator (ESO) to replicate each Deployment's Airflow secrets between data plane clusters. Before you apply the failover Helm values, set up ESO and its secret store on each data plane cluster: the backend credentials or workload identity, the `ClusterSecretStore` (or a synced `SecretStore` when you use namespace pools), and — for Hashicorp Vault — the auth mount, policy, and role.

See [Configure External Secrets Operator security](/docs/astro-private-cloud/v-2-x/configure-external-secrets-operator-security) for the full setup and the available modes, and [Configure Hashicorp Vault for data plane failover](/docs/astro-private-cloud/v-2-x/configure-vault-data-plane-failover) if your backend is Vault. The name you give the secret store is the value you set for `global.dataPlaneFailover.externalSecretManagerName` in the following steps.

## Step 1: Configure the control plane

Add the following values to your control plane `values.yaml`. Setting `global.dataPlaneFailover.enabled: true` activates Navigator, DP-Link, and the APC API dispatcher when `global.plane.mode` is `control`.

```yaml wrap theme={null}
global:
  dataPlaneFailover:
    enabled: true
    externalSecretManagerName: astronomer-secret-store
external-secrets:
  enabled: false
```

Set `externalSecretManagerName` to the name of the secret store you created in the ESO setup (`astronomer-secret-store` in these examples). It must be identical on the control plane and every data plane cluster.

<Note>
  ESO isn't required on the control plane. Don't set `external-secrets.enabled: true` in your control plane values.
</Note>

## Step 2: Configure each data plane

Add the following values to each data plane `values.yaml`. Setting `global.dataPlaneFailover.enabled: true` activates Pilot and the Flightdeck database bootstrap when `global.plane.mode` is `data`.

```yaml wrap theme={null}
global:
  dataPlaneFailover:
    enabled: true
    externalSecretManagerName: astronomer-secret-store
external-secrets:
  enabled: true
```

Use the same value for `externalSecretManagerName` as on the control plane. Both clusters must reference the same `ClusterSecretStore`.

<Note>
  The `external-secrets` key enables the bundled ESO subchart, which installs cluster-scoped CRDs. The identity running `helm upgrade` must have a `ClusterRole` on the data plane cluster. If you already run ESO separately, set `external-secrets.enabled: false` and ensure your existing ESO installation recognizes the `ClusterSecretStore` that APC expects.
</Note>

<Warning>
  The deployment orchestrator bootstraps the Flightdeck database as an init container during startup. If the bootstrap fails, the deployment orchestrator Pod doesn't start. Check the `flightdeck-bootstrapper` and `flightdeck-db-migrations` init container logs if the deployment orchestrator fails to come up after enabling this feature.
</Warning>

## Step 3: Apply the changes

Apply the updated values to each cluster using `helm upgrade`. Upgrade the control plane first.

```bash wrap theme={null}
helm upgrade <release-name> astronomer/astronomer \
  --namespace <namespace> \
  --values values.yaml \
  --version <chart-version>
```

Run the same command for each data plane cluster, substituting the appropriate release name, namespace, and values file.

## Step 4: Verify the deployment

After the upgrade completes, confirm that the new components are running on each cluster.

On the control plane, verify that the following Pods are running:

```bash wrap theme={null}
kubectl get pods -n <namespace> | grep -E "navigator|dp-link|houston"
```

On each data plane, verify that the deployment orchestrator started successfully and Pilot is running:

```bash wrap theme={null}
kubectl get pods -n <namespace> | grep -E "commander|pilot"
```

Check deployment orchestrator logs to confirm Flightdeck initialized correctly:

```bash wrap theme={null}
kubectl logs -n <namespace> deployment/<release-name>-commander \
  -c flightdeck-bootstrapper
```

## Advanced configuration

<Warning>
  Changing any of the values in this section can meaningfully affect resource
  usage on your Kubernetes clusters and may adversely affect failover
  functionality. Change and test these values in a non-production environment
  before applying them to production.
</Warning>

### Tune Pilot behavior

Pilot's claim, retry, and circuit breaker behavior is configurable via environment variables. Set these under `astronomer.pilot.env` in your data plane `values.yaml`.

| Environment variable            | Default | Description                                                                           |
| ------------------------------- | ------- | ------------------------------------------------------------------------------------- |
| `PILOT_MAX_INFLIGHT_PER_WORKER` | `5`     | Maximum number of flights Pilot executes concurrently per worker.                     |
| `PILOT_CLAIM_POLL_INTERVAL_MS`  | `5000`  | How often Pilot polls for new flights, in milliseconds.                               |
| `PILOT_LEASE_TTL_SECONDS`       | `60`    | How long a claimed flight lease is valid before expiring.                             |
| `PILOT_MAX_ATTEMPTS_PER_FLIGHT` | `15`    | Maximum number of execution attempts before a flight is marked as failed.             |
| `PILOT_RETRY_BASE_INTERVAL_MS`  | `250`   | Base delay between retry attempts, in milliseconds.                                   |
| `PILOT_RETRY_MAX_INTERVAL_MS`   | `5000`  | Maximum delay between retry attempts, in milliseconds.                                |
| `PILOT_CB_FAILURE_THRESHOLD`    | `10`    | Number of consecutive failures before the circuit breaker opens.                      |
| `PILOT_CB_COOLOFF_SECONDS`      | `30`    | How long the circuit breaker remains open before allowing probe attempts, in seconds. |

For data planes with a larger number of Airflow Deployments (roughly 50 or more), or for cross-region failovers where each Deployment takes longer to come up because the deployment orchestrator has to pull container images from a remote-region registry endpoint or fetch secrets from a remote-region secrets backend, consider raising `PILOT_MAX_INFLIGHT_PER_WORKER` above the default of `5`. A higher value lets Pilot bring more Deployments up on the destination cluster in parallel, which reduces overall failover time and helps amortize cross-region latency. Each in-flight flight runs additional work on the data plane cluster (secret syncs, Helm installs, and database operations) and consumes additional bandwidth to the registry and secrets store, so only raise this value if your data plane cluster has spare CPU, memory, and API server headroom and your registry/secrets backends can handle the extra concurrent traffic. Validate the new value in a non-production environment first.

### Tune Navigator behavior

Navigator's reconcile loop timing is configurable via environment variables. Set these under `astronomer.navigator.env` in your control plane `values.yaml`.

| Environment variable                           | Default | Description                                                                      |
| ---------------------------------------------- | ------- | -------------------------------------------------------------------------------- |
| `FAILOVER_REQUEST_RECONCILER_INTERVAL_SECONDS` | `10`    | How often Navigator checks for new or in-progress failover requests, in seconds. |
| `MISSION_CLAIM_MIN_BATCH_SIZE`                 | `10`    | Minimum number of missions Navigator claims per reconcile cycle.                 |
| `MISSION_PLAN_BATCH_SIZE`                      | `5`     | Number of missions Navigator plans concurrently.                                 |
| `MISSION_RECONCILE_BATCH_SIZE`                 | `5`     | Number of missions Navigator reconciles concurrently.                            |
| `CLAIM_INTERVAL_SECONDS`                       | `10`    | Interval between mission claim cycles, in seconds.                               |

### Tune DP-Link health thresholds

DP-Link determines cluster health based on heartbeat age. Adjust these thresholds under `astronomer.dpLink.env` in your control plane `values.yaml`.

| Environment variable            | Default | Description                                                                           |
| ------------------------------- | ------- | ------------------------------------------------------------------------------------- |
| `UNREACHABLE_THRESHOLD_SECONDS` | `90`    | Age of the last heartbeat, in seconds, after which a cluster is marked `UNREACHABLE`. |
| `DEGRADED_THRESHOLD_SECONDS`    | `30`    | Age of the last heartbeat, in seconds, after which a cluster is marked `DEGRADED`.    |

### Tune the APC API dispatcher behavior

The APC API dispatcher dispatches flights from the control plane to the deployment orchestrator on each data plane. Its loop timing, concurrency, retry, and circuit breaker behavior are configurable through environment variables. Set these under `astronomer.houston.env` in your control plane `values.yaml`.

#### Dispatcher loop

| Environment variable                   | Default | Description                                                                                                   |
| -------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------- |
| `DISPATCH_LEASE_TTL_SECONDS`           | `30`    | How long a dispatcher lease on a flight is valid before expiring, in seconds.                                 |
| `DISPATCH_BATCH_SIZE`                  | `50`    | Maximum number of flights the dispatcher claims per poll cycle.                                               |
| `DISPATCH_MAX_INFLIGHT`                | `50`    | Maximum number of flights the dispatcher executes concurrently across all data planes.                        |
| `DISPATCH_MAX_INFLIGHT_PER_DP`         | `5`     | Maximum number of flights the dispatcher executes concurrently against a single data plane.                   |
| `DISPATCH_POLL_SECONDS`                | `10`    | How often the dispatcher polls for new flights, in seconds.                                                   |
| `DISPATCH_MAX_ATTEMPTS_PER_LEASE`      | `5`     | Maximum number of in-process retries for a flight while the dispatcher holds its lease.                       |
| `DISPATCH_MAX_ATTEMPTS_PER_FLIGHT`     | `25`    | Maximum durable retries for a flight across all dispatcher processes before it is marked as failed.           |
| `DISPATCH_RETRY_COOLOFF_PERIOD`        | `60`    | Cool-off, in seconds, applied after a flight crosses `DISPATCH_MAX_ATTEMPTS_PER_FLIGHT`.                      |
| `IN_REGION_STARTFLIGHT_RPC_TIMEOUT`    | `5000`  | Timeout for `StartFlight` RPCs to a data plane in the same region as the control plane, in milliseconds.      |
| `CROSS_REGION_STARTFLIGHT_RPC_TIMEOUT` | `12000` | Timeout for `StartFlight` RPCs to a data plane in a different region from the control plane, in milliseconds. |

#### Circuit breaker

| Environment variable    | Default | Description                                                                                    |
| ----------------------- | ------- | ---------------------------------------------------------------------------------------------- |
| `CB_FAILURE_THRESHOLD`  | `10`    | Number of consecutive `StartFlight` failures to a data plane before the circuit breaker opens. |
| `CB_COOLOFF_SECONDS`    | `30`    | How long the circuit breaker remains open before allowing probe attempts, in seconds.          |
| `CB_PROBE_MAX_INFLIGHT` | `1`     | Maximum number of probe `StartFlight` calls allowed while the circuit breaker is half-open.    |

## Verify secret replication before a failover

A `ClusterSecretStore` (or `SecretStore`, if your cluster uses namespace pools) reporting `READY=True` confirms that ESO can reach and authenticate to the secrets store. It doesn't confirm that ESO can write to it. If the secrets store policy is missing write permissions, the `PushSecret` write performed by the deployment orchestrator fails, but this failure is logged as a warning and doesn't block Deployment provisioning. Nothing surfaces the problem until a failover is triggered and the destination cluster can't find the secrets it needs.

Before triggering a failover, confirm that `PushSecret` resources are succeeding on the source cluster:

```bash wrap theme={null}
kubectl get pushsecrets -n <namespace>
```

Each `PushSecret` should report a `Synced` status. If any report a failure, review the secrets store's write permissions before triggering a failover.

Also confirm that the following connection secrets resolve successfully. Data plane failover fencing reads these directly, and a failover for a Deployment fails if any of them are missing or hold an invalid database connection string:

* `<release-name>-active-metadata`
* `<release-name>-active-result-backend`
* `<release-name>-inactive-metadata`
* `<release-name>-inactive-result-backend`

## Related documentation

* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)
* [Configure Hashicorp Vault for data plane failover](/docs/astro-private-cloud/v-2-x/configure-vault-data-plane-failover)
* [Trigger a data plane failover](/docs/astro-private-cloud/v-2-x/trigger-data-plane-failover)
* [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config)


# Ephemeral storage configuration
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/ephemeral-storage

Configure temporary storage for Airflow components using Kubernetes emptyDir volumes.

Ephemeral storage in Astro Private Cloud (APC) uses Kubernetes emptyDir volumes for temporary data that doesn't need to persist across Pod restarts. This guide covers configuring ephemeral storage for Apache Airflow components including Dags, logs, and Redis.

## Ephemeral storage overview

Ephemeral storage (emptyDir volumes) provides:

* Temporary storage that exists for the lifetime of a Pod.
* Faster I/O if using memory-backed storage.
* No persistence across Pod restarts.
* Shared access between containers in the same Pod.

<Note>
  APC runs every Airflow container with `readOnlyRootFilesystem: true`. To keep Airflow writable where it needs to be, the chart mounts `emptyDir` volumes at specific subpaths under `/usr/local/airflow`, such as `/usr/local/airflow/logs`, and at `/usr/local/airflow/dags` when using git-sync or the emptyDir Dag mode. Depending on log volume and Dag size, these volumes can consume significant ephemeral storage and exceed namespace limits. Factor this into your ephemeral storage sizing. For details on adding more writable directories, see [Read-only root filesystem](/docs/astro-private-cloud/v-2-x/read-only-root-filesystem).
</Note>

## Configurable volumes

### Dags volume (gitSync)

When using git-sync for Dag deployment, Dags are stored in an emptyDir volume.

```yaml wrap theme={null}
dags:
  gitSync:
    enabled: true
    emptyDirConfig:
      sizeLimit: 2Gi
      medium: Memory  # Optional: use RAM instead of disk
```

Parameters:

* `sizeLimit`: Maximum storage size (for example, `1Gi`, `2Gi`).
* `medium`: Storage medium.
  * `""` (empty): Use node's default storage (disk).
  * `"Memory"`: Use RAM (tmpfs) for faster access.

### Logs volume

Task logs can be stored in ephemeral storage when persistence is disabled.

```yaml wrap theme={null}
logs:
  persistence:
    enabled: false
  emptyDirConfig:
    sizeLimit: 10Gi
    medium: ""  # Use disk for logs (recommended)
```

Recommendations:

* Use larger `sizeLimit` for high-volume task execution.
* Avoid `medium: Memory` for logs unless you have aggressive cleanup.
* Enable log groomer sidecar to prevent storage exhaustion.

### Redis volume

When Redis persistence is disabled, it uses ephemeral storage.

```yaml wrap theme={null}
redis:
  persistence:
    enabled: false
  emptyDirConfig:
    sizeLimit: 1Gi
    medium: Memory  # RAM-backed for performance
```

## Configuration examples

### Development environment

```yaml wrap theme={null}
dags:
  gitSync:
    enabled: true
    emptyDirConfig:
      sizeLimit: 1Gi

logs:
  emptyDirConfig:
    sizeLimit: 5Gi

redis:
  emptyDirConfig:
    sizeLimit: 512Mi
    medium: Memory
```

### Production environment

```yaml wrap theme={null}
dags:
  gitSync:
    enabled: true
    emptyDirConfig:
      sizeLimit: 5Gi
      medium: ""  # Use disk storage for durability

logs:
  persistence:
    enabled: true  # Use persistent storage in production
    size: 100Gi

redis:
  persistence:
    enabled: true  # Use persistent storage in production
    size: 8Gi
```

### High-performance configuration

For latency-sensitive workloads:

```yaml wrap theme={null}
dags:
  gitSync:
    enabled: true
    emptyDirConfig:
      sizeLimit: 2Gi
      medium: Memory  # Faster Dag parsing

redis:
  emptyDirConfig:
    sizeLimit: 2Gi
    medium: Memory  # Faster task queue operations
```

## Storage medium comparison

| Medium      | Speed    | Persistence  | Memory Impact | Use Case          |
| ----------- | -------- | ------------ | ------------- | ----------------- |
| `""` (disk) | Moderate | Pod lifetime | None          | Logs, large Dags  |
| `"Memory"`  | Fast     | Pod lifetime | Consumes RAM  | Redis, small Dags |

<Warning>
  Memory-backed volumes (`medium: Memory`) count against container memory limits. If the volume grows too large, Pods may be OOMKilled. Size memory limits accordingly or use disk-backed storage.
</Warning>

## Sizing guidelines

### Dags volume

| Dag count | Recommended size |
| --------- | ---------------- |
| \< 50     | 1Gi              |
| 50-200    | 2Gi              |
| 200-500   | 5Gi              |
| 500+      | 10Gi             |

### Logs volume

| Task Volume          | Recommended Size       |
| -------------------- | ---------------------- |
| \< 100 tasks/day     | 5Gi                    |
| 100-1000 tasks/day   | 10Gi                   |
| 1000-10000 tasks/day | 50Gi                   |
| 10000+ tasks/day     | Use persistent storage |

### Redis volume

| Worker Count  | Recommended Size |
| ------------- | ---------------- |
| 1-5 workers   | 512Mi            |
| 5-20 workers  | 1Gi              |
| 20-50 workers | 2Gi              |
| 50+ workers   | 4Gi              |

## Monitor storage usage

### Check volume usage

```bash wrap theme={null}
kubectl exec -n <namespace> <pod-name> -- df -h
```

### Check memory-backed volume

```bash wrap theme={null}
kubectl exec -n <namespace> <pod-name> -- mount | grep tmpfs
```

## Troubleshooting

### Pod eviction due to storage

**Symptom**: Pods evicted with `DiskPressure` or ephemeral storage exceeded.

**Cause**: emptyDir volume exceeded node's ephemeral storage limits.

**Solution**:

1. Increase `sizeLimit` in `emptyDirConfig`.
2. Enable log groomer with shorter retention.
3. Switch to persistent storage.

### Out of memory with memory-backed volumes

**Symptom**: Pods OOMKilled when using `medium: Memory`.

**Cause**: Memory-backed emptyDir counts against container memory limits.

**Solution**:

1. Increase container memory limits.
2. Reduce `sizeLimit` on memory-backed volumes.
3. Switch to disk-backed storage.

### Slow Dag parsing

**Symptom**: Dag processing takes too long.

**Cause**: Disk I/O latency on Dag volume.

**Solution**:

1. Use `medium: Memory` for Dag volume.
2. Ensure sufficient `sizeLimit`.
3. Consider SSD-backed nodes.

## Best practices

* Set explicit `sizeLimit` to prevent unbounded storage growth.
* Use memory sparingly for performance-critical, small volumes only.
* Monitor usage and set alerts for storage utilization.
* Use persistent storage for production logs since ephemeral storage loses logs on restart.
* Size for peak usage to account for burst workloads.
* Enable log groomer to prevent log accumulation.


# External Secrets Operator security
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/external-secrets-operator-security

Astro Private Cloud 2.1 gives you more control over how the External Secrets Operator is granted access to Kubernetes, with opt-in modes that tighten its identity, RBAC, and secret store scope.

The External Secrets Operator (ESO) is the component that [data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover) uses to synchronize each Deployment's Kubernetes secrets to and from your external secret backend — for example AWS Secrets Manager, GCP Secret Manager, or Hashicorp Vault — so that a Deployment's secrets move with it between data plane clusters. Data plane failover has used ESO since APC 2.0, and APC 2.1 gives you more control over how ESO is granted access to Kubernetes.

<Note>
  The default ESO configuration (Mode 1) is available in every release that supports data plane failover, including APC 2.0. The hardened and isolated modes (Modes 2 and 3) and the other options described here — workload identity, self-managed CRDs, and pinned or disabled ServiceAccount token creation — require APC 2.1 or later.
</Note>

By default, 2.1 closes the one significant gap from previous releases — it disables ESO's unconditional ServiceAccount token creation — while otherwise keeping ESO's standard chart RBAC, so upgrading requires no action for most installations. On top of that, 2.1 adds two opt-in modes that progressively tighten ESO's Kubernetes access and identity, so you can match your organization's security and compliance requirements. These changes ship as part of Data Plane Failover v2 and build on the data plane failover feature introduced in 2.0.

This document explains what's new and the modes you can choose from. For step-by-step setup, see [Configure External Secrets Operator security](/docs/astro-private-cloud/v-2-x/configure-external-secrets-operator-security). For the reference Kubernetes manifests, see [External Secrets Operator security manifests reference](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests).

## What's new in 2.1

* Two opt-in modes for tighter ESO access: in addition to the default configuration, you can choose a hardened shared-identity mode or a customer-managed isolated-identity mode. See [Choose a mode](#choose-a-mode).
* ServiceAccount token creation is off by default: the chart now disables ESO's unconditional ServiceAccount token creation by default, closing the main gap from previous releases. The hardened and isolated modes go further, pinning token creation with Kubernetes `resourceNames` to a specific ServiceAccount. Hashicorp Vault backends are the exception and need it re-enabled — see Mode 1.
* Workload identity support: you can authenticate ESO to your backend with workload identity such as AWS IAM Roles for Service Accounts (IRSA), GKE Workload Identity, or Vault Kubernetes auth, so no static credentials need to live in the cluster. Static credentials remain fully supported.
* Customer-managed CRD installation: a chart value lets you install the ESO custom resource definitions (CRDs) yourself, for organizations where CRD installation is owned by a separate infrastructure team. See [ESO CRD installation](#eso-crd-installation).

## Backend authentication

Independent of the mode you choose, you authenticate ESO to your secret backend in one of two ways:

* *Static credentials.* A Kubernetes `Secret` holding a cloud credential pair, as in previous releases.
* *Workload identity.* AWS IRSA, GKE Workload Identity, or Vault Kubernetes auth, so no long-lived credentials are stored in the cluster.

Workload identity is the recommended option where your cloud supports it, but it isn't required.

## Choose a mode

APC supports three modes. Mode 1 is available in APC 2.0 and later; Modes 2 and 3 require APC 2.1 or later. A cluster runs entirely in one mode, and Deployments inherit the cluster's mode.

|                         | Mode 1: Default shared identity                                          | Mode 2: Hardened shared identity                                                                                                                                             | Mode 3: Customer-managed isolated identity                                                     |
| ----------------------- | ------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| **Backend identity**    | One shared identity                                                      | One shared identity                                                                                                                                                          | A separate identity per Deployment                                                             |
| **ESO Kubernetes RBAC** | Chart default, created automatically, with token creation off by default | You disable the chart's ESO RBAC and token-create, then apply Astronomer's minimal `ClusterRole` and `ClusterRoleBinding` with token creation pinned to ESO's ServiceAccount | You pre-provision all per-namespace roles                                                      |
| **Secret store**        | `ClusterSecretStore`                                                     | `ClusterSecretStore`, or a namespaced `SecretStore` synced by the platform                                                                                                   | A per-namespace `SecretStore` in each Deployment namespace                                     |
| **Namespace pools**     | Not required                                                             | Not required                                                                                                                                                                 | Required                                                                                       |
| **Extra setup**         | None                                                                     | Two chart flags plus applying the provided manifests                                                                                                                         | Pre-provision namespaces, per-namespace RBAC, and per-namespace stores                         |
| **Best for**            | Existing installs and the simplest setup                                 | A single shared identity with tightened, reviewable ESO cluster access                                                                                                       | Per-Deployment identity isolation, or clusters that require you to provision all RBAC yourself |

### Mode 1: Default shared identity

The default. ESO uses its standard chart-managed RBAC, with one change from previous releases: its unconditional ServiceAccount token creation is disabled by default, closing the main gap. All Deployments authenticate to the backend as one shared identity. No extra setup is required on upgrade, with one exception: if your backend is Hashicorp Vault, re-enable ESO's ServiceAccount token creation (unpinned) with `external-secrets.rbac.serviceAccountTokenCreate: true`, because Vault's Kubernetes auth needs ESO to mint a ServiceAccount token. You provision a single cluster-scoped `ClusterSecretStore`. See [Secret store options](#secret-store-options).

### Mode 2: Hardened shared identity

The same shared-identity model as Mode 1, with ESO's Kubernetes access tightened. To use it, you:

1. Set the two ESO sub-chart flags that disable its default RBAC and its ServiceAccount token-create rule.
2. Apply the minimal `ClusterRole` and `ClusterRoleBinding` that Astronomer provides. These scope ESO's cluster access and restrict ServiceAccount token creation, with `resourceNames`, to ESO's own ServiceAccount in the `astronomer` namespace.

Mode 2 uses a shared secret store — either a `ClusterSecretStore` or a synced namespaced `SecretStore`. See [Secret store options](#secret-store-options). Choose Mode 2 when you want a single shared backend identity but need ESO's cluster access to be least-privilege and reviewed by your team.

### Mode 3: Customer-managed isolated identity

Each Deployment gets its own backend identity, and you provision all of the required Kubernetes RBAC yourself. This mode builds on the [namespace pools](/docs/astro-private-cloud/v-2-x/namespace-pools) feature: you complete the standard namespace pools setup to pre-provision a fixed set of Deployment namespaces, then, in addition, pre-provision the ESO chain in each namespace — a per-namespace ServiceAccount (named after the pre-created namespace, since no Deployment exists yet), a namespaced `SecretStore`, and the per-namespace roles that grant ESO scoped access. The platform is installed without cluster-wide roles and can run with a limited-privilege user. You can substitute your own role definitions for the generated ones to match an internal RBAC standard.

The `SecretStore` in every pool namespace must use the same name, which is the value set for `global.dataPlaneFailover.externalSecretManagerName` when you enable data plane failover per cluster. Only the name must match across namespaces. Each store's contents — provider, region, and identity — can differ, which is what gives each Deployment its own backend identity.

<Warning>
  **Isolation boundary**

  Kubernetes namespaces and RBAC are defense in depth. The authoritative isolation boundary is your backend policy: a per-Deployment IAM role or Vault policy that grants only that Deployment's secret paths. If the backend policy is broad, ESO can read across Deployments regardless of namespace scoping.
</Warning>

## Secret store options

Both shared-identity modes use a single secret store. Mode 1 uses a cluster-scoped `ClusterSecretStore`. Mode 2 can use either shape:

* *Cluster-scoped `ClusterSecretStore`.* One store that every Deployment namespace references by name.
* *Namespaced `SecretStore`, synced by the platform.* You create one `SecretStore` and its credentials `Secret` in the `astronomer` namespace with the `astronomer.io/commander-sync` annotation, and the platform syncs them into every Deployment namespace. This keeps a single shared backend identity while using namespaced stores instead of a cluster-wide one.

Both use the same shared backend identity, so they grant identical backend access. The difference is how the store is scoped in Kubernetes and which authentication methods it supports, not what it can reach.

|                            | `ClusterSecretStore`                                                                         | Synced `SecretStore`                                                                                                             |
| -------------------------- | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| **Object scope**           | A single cluster-scoped object                                                               | A namespaced object in each Deployment namespace, created and kept in sync by the platform                                       |
| **Backend authentication** | Static credentials or workload identity (IRSA, GKE Workload Identity, Vault Kubernetes auth) | Static credentials only                                                                                                          |
| **RBAC to manage it**      | Requires cluster-scoped permissions                                                          | Namespace-scoped only — no cluster-scoped secret-store object exists                                                             |
| **Setup and upkeep**       | Lowest: one object, created once                                                             | You author one store in the `astronomer` namespace, and the platform replicates it to each Deployment namespace                  |
| **Choose when**            | Cluster-scoped resources are acceptable and you want the simplest setup                      | Your security or compliance policy limits or reviews cluster-scoped objects, or you want the secret store governed per namespace |

If your organization restricts cluster-scoped Kubernetes resources, choose the synced `SecretStore` — it keeps everything namespace-scoped while you still author and rotate a single store. The synced `SecretStore` supports static credentials only: if you need workload identity, use the `ClusterSecretStore`, which is also the simpler choice when cluster-scoped resources are acceptable.

Mode 3 doesn't use a shared store; each Deployment namespace has its own `SecretStore`.

## ESO CRD installation

By default, the APC chart installs the ESO custom resource definitions. If CRD installation in your environment is owned by a separate infrastructure team, you can set the chart value that disables CRD creation and install the ESO CRDs yourself before installing the platform. This option is independent of the mode you choose and is most relevant to Modes 2 and 3. For the CRD bundle and command, see [Install the ESO CRDs yourself](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#install-the-eso-crds-yourself).

## How this interacts with data plane failover

* A cluster runs entirely in one mode. Per-Deployment mixing within a cluster isn't supported in 2.1.
* In Mode 3, the objects you pre-provision — namespace, ServiceAccount, `SecretStore`, and roles — are preserved across a failover. The platform doesn't delete the pool namespace during a move, and the pool slot is freed only when the Deployment is deleted.
* At Deployment creation, and again on the destination cluster during a failover, the platform validates that the per-namespace ESO chain exists and reports any gaps.
* For Mode 3, pre-provision the same chain, with matching namespace names, on every cluster a Deployment can fail over to, and make sure your backend trust spans those clusters.
* For extra resiliency, you can run the ESO controller with multiple replicas and leader election, so it survives Pod failures and restarts. See [Run ESO with multiple replicas](/docs/astro-private-cloud/v-2-x/configure-external-secrets-operator-security#run-eso-with-multiple-replicas).

## Terminology

| Term                                 | Meaning                                                                                                                                                           |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| External Secrets Operator (ESO)      | The Kubernetes operator that syncs secrets between the cluster and an external secret backend.                                                                    |
| `ClusterSecretStore` / `SecretStore` | ESO custom resources that describe where to connect and which identity to authenticate as. `ClusterSecretStore` is cluster-scoped; `SecretStore` is namespaced.   |
| Namespace pools                      | An APC feature (`global.namespaceManagement.namespacePools`) that pre-provisions a fixed set of Deployment namespaces. Mode 3 is built on it.                     |
| Workload identity                    | Authenticating to a cloud or Vault without static credentials (AWS IRSA, GKE Workload Identity, Vault Kubernetes auth).                                           |
| `<release-name>`                     | The Helm release name you install the Astronomer chart with (for example `astronomer`). It prefixes the ESO ServiceAccount and RBAC object names in these guides. |

## Related documentation

* [Configure External Secrets Operator security](/docs/astro-private-cloud/v-2-x/configure-external-secrets-operator-security)
* [External Secrets Operator security manifests reference](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests)
* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)
* [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover)
* [Configure a Kubernetes namespace pool](/docs/astro-private-cloud/v-2-x/namespace-pools)


# External Secrets Operator security manifests reference
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests

Reference Kubernetes manifests for External Secrets Operator security in Astro Private Cloud 2.1: CRDs, backend credentials, secret stores, and the Mode 2 and Mode 3 RBAC.

This document collects the reference Kubernetes manifests for External Secrets Operator (ESO) security in Astro Private Cloud (APC): the backend credentials secret, the secret store options for the shared-identity modes, the Mode 2 hardened cluster RBAC, and the Mode 3 per-namespace RBAC.

<Note>
  The backend-credentials secret and the `ClusterSecretStore` (Mode 1) apply to APC 2.0 and later. The ESO controller ServiceAccount, the Mode 2 cluster RBAC, the Mode 3 per-namespace RBAC, and the leader-election RBAC require APC 2.1 or later.
</Note>

Use it alongside [External Secrets Operator security](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security), which explains the modes and how to choose, and [Configure External Secrets Operator security](/docs/astro-private-cloud/v-2-x/configure-external-secrets-operator-security), the step-by-step setup that applies these manifests.

All examples use AWS Secrets Manager. GCP Secret Manager and Hashicorp Vault (Kubernetes auth) use the same object shapes with a different `provider` block; for the Vault provider block and Vault-side setup, see [Configure Hashicorp Vault for data plane failover](/docs/astro-private-cloud/v-2-x/configure-vault-data-plane-failover). The manifests use two placeholders: `<release-name>` is the Helm release name you install the Astronomer chart with (for example `astronomer`), and `<pool-namespace>` is one of your pre-created namespace pool namespaces.

<Note>
  These manifests assume the platform is installed in the `astronomer` namespace. If you installed it into a different namespace, replace `astronomer` in the manifests and the chart values accordingly.
</Note>

## Install the ESO CRDs yourself

By default the APC chart installs the External Secrets Operator CRDs (`external-secrets.crd.create: true`). If you set `external-secrets.crd.create: false` — for example, when a separate infrastructure team owns CRD installation — install the CRDs yourself before you install or upgrade the platform.

The CRDs are published by the external-secrets project as a single [CRD bundle](https://raw.githubusercontent.com/external-secrets/external-secrets/refs/tags/v2.7.0/deploy/crds/bundle.yaml). Apply the bundle whose tag matches the ESO version the APC chart ships, currently `v2.7.0`:

```bash theme={null}
kubectl apply -f https://raw.githubusercontent.com/external-secrets/external-secrets/refs/tags/v2.7.0/deploy/crds/bundle.yaml --server-side
```

Match the bundle tag to the ESO version your APC chart ships — applying CRDs from a different version can break ESO reconciliation. Apply the bundle before the platform install so the CRDs exist when ESO starts.

## Backend credentials secret (Modes 1 and 2)

Create the backend credentials in the platform (`astronomer`) namespace. Skip this if you authenticate with workload identity instead of a credential pair.

```yaml theme={null}
apiVersion: v1
kind: Secret
metadata:
  name: secrets-backend-credentials
  namespace: astronomer
type: Opaque
data:
  access-key: <base64-aws-access-key-id>
  secret-access-key: <base64-aws-secret-access-key>
```

## ESO controller ServiceAccount (Modes 2 and 3)

In Modes 2 and 3, you create the ESO controller ServiceAccount yourself and disable the chart's ServiceAccount creation, so the ServiceAccount and its RBAC exist before ESO starts. In [Configure External Secrets Operator security](/docs/astro-private-cloud/v-2-x/configure-external-secrets-operator-security), you set `external-secrets.serviceAccount.create: false` and `external-secrets.serviceAccount.name` to this ServiceAccount. Create it in the `astronomer` namespace:

```yaml theme={null}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: <release-name>-external-secrets
  namespace: astronomer
```

For workload identity, annotate this ServiceAccount for your provider (`eks.amazonaws.com/role-arn` for AWS, `iam.gke.io/gcp-service-account` for GCP).

## Secret store manifests (Modes 1 and 2)

### Option A — cluster-scoped ClusterSecretStore (Modes 1 and 2)

```yaml theme={null}
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: astronomer-secret-store
spec:
  provider:
    aws:
      secretsManager:
        forceDeleteWithoutRecovery: true
      service: SecretsManager
      region: us-east-2
      auth:
        secretRef:
          accessKeyIDSecretRef:
            key: access-key
            name: secrets-backend-credentials
            namespace: astronomer
          secretAccessKeySecretRef:
            key: secret-access-key
            name: secrets-backend-credentials
            namespace: astronomer
```

The manifest above authenticates with static credentials. To use workload identity instead, reference the ESO controller ServiceAccount from the store's `auth` block. AWS IRSA is shown. GCP and Vault use their provider's equivalent auth:

```yaml theme={null}
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: astronomer-secret-store
spec:
  provider:
    aws:
      secretsManager:
        forceDeleteWithoutRecovery: true
      service: SecretsManager
      region: us-east-2
      auth:
        jwt:
          serviceAccountRef:
            name: <release-name>-external-secrets
            namespace: astronomer
```

The ESO controller ServiceAccount (`<release-name>-external-secrets`) is created during ESO setup — by the chart in Mode 1, or by you in Modes 2 and 3 (see [ESO controller ServiceAccount](#eso-controller-serviceaccount-modes-2-and-3)). For workload identity, annotate it for your provider (`eks.amazonaws.com/role-arn` for AWS, `iam.gke.io/gcp-service-account` for GCP) — in Mode 1 through the chart's `external-secrets.serviceAccount.annotations` value, or in Modes 2 and 3 directly on the ServiceAccount you create.

### Option B — namespaced SecretStore synced by the platform (Mode 2 only)

Both the credentials secret and the `SecretStore` carry the `astronomer.io/commander-sync` annotation, which tells the platform to sync them into every Deployment namespace.

<Note>
  **Static credentials only**

  This synced-`SecretStore` option supports static credentials only. The platform copies the credentials Secret and the `SecretStore` into each Deployment namespace, so it can't carry per-namespace workload-identity references. To use workload identity (AWS IRSA, GKE Workload Identity, or Vault Kubernetes auth) in Mode 2, use the cluster-scoped `ClusterSecretStore` (Option A) instead.
</Note>

```yaml theme={null}
apiVersion: v1
kind: Secret
metadata:
  name: secrets-backend-credentials
  namespace: astronomer
  annotations:
    astronomer.io/commander-sync: platform-release=<platform-release-name>
type: Opaque
data:
  access-key: <base64-aws-access-key-id>
  secret-access-key: <base64-aws-secret-access-key>
---
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: astronomer-secret-store
  namespace: astronomer
  annotations:
    astronomer.io/commander-sync: platform-release=<platform-release-name>
spec:
  provider:
    aws:
      secretsManager:
        forceDeleteWithoutRecovery: true
      service: SecretsManager
      region: us-east-2
      auth:
        secretRef:
          accessKeyIDSecretRef:
            key: access-key
            name: secrets-backend-credentials
          secretAccessKeySecretRef:
            key: secret-access-key
            name: secrets-backend-credentials
```

## Force-delete secrets in AWS Secrets Manager

AWS Secrets Manager soft-deletes secrets by default, keeping them recoverable for a 30-day window during which the secret name stays reserved. If the platform then tries to create a secret with the same name, the create conflicts with the still-recoverable secret. This can happen when you use custom release names and reuse a release name for a new Deployment after deleting the previous one.

To make ESO hard-delete secrets instead, set `forceDeleteWithoutRecovery` on the AWS provider in your `SecretStore` or `ClusterSecretStore`:

```yaml theme={null}
spec:
  provider:
    aws:
      secretsManager:
        forceDeleteWithoutRecovery: true
```

With this set, deleted secrets are removed immediately with no recovery window, so a new Deployment can reuse the name right away. If you use custom release names, set this, because reusing a release name after deleting a Deployment is exactly when the name collision occurs. The trade-off is that deleted secrets can't be restored, since there is no recovery window.

This option applies to AWS Secrets Manager only. Add it to whichever store your mode uses — the shared `ClusterSecretStore` or `SecretStore` in Modes 1 and 2, or the per-namespace `SecretStore` in Mode 3. The ESO GCP Secret Manager and Azure Key Vault providers don't expose an equivalent option: GCP Secret Manager deletes secrets immediately, so the conflict doesn't arise, and Azure Key Vault's soft-delete can't be overridden from the store — on Azure you purge a deleted secret through Azure itself.

## Mode 2: hardened ESO cluster RBAC

In Mode 2, you disable the ESO sub-chart's default RBAC (`external-secrets.rbac.create: false`), then apply the `ClusterRole` and `ClusterRoleBinding` below. They grant ESO the same reconcile rules as the Mode 3 per-namespace Role, but cluster-scoped, and pin ServiceAccount token creation with `resourceNames` to ESO's own ServiceAccount in the `astronomer` namespace. That pinned token-create serves the same purpose as in Mode 3: ESO assumes the shared identity by minting its own ServiceAccount's token (the `serviceAccountRef` path) — required for Vault, and used by AWS or GCP workload identity through `serviceAccountRef`. It goes unused if you authenticate with static credentials.

<Note>
  In Mode 2 you apply this `ClusterRole` and `ClusterRoleBinding` yourself. The chart doesn't ship them, since `external-secrets.rbac.create: false`. The rules mirror the external-secrets Role, expressed as a cluster-scoped role with token creation pinned to the ESO ServiceAccount.
</Note>

The `clustersecretstores` rules let ESO read the cluster-scoped store; include them when Mode 2 uses a `ClusterSecretStore`. If you use only the synced `SecretStore` (Option B), you can omit them.

```yaml theme={null}
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: <release-name>-external-secrets
rules:
  - apiGroups: ["external-secrets.io"]
    resources: ["secretstores", "externalsecrets", "pushsecrets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["external-secrets.io"]
    resources: ["externalsecrets", "externalsecrets/status", "secretstores", "secretstores/status", "pushsecrets", "pushsecrets/status"]
    verbs: ["get", "update", "patch"]
  - apiGroups: ["external-secrets.io"]
    resources: ["externalsecrets"]
    verbs: ["create", "update", "delete"]
  - apiGroups: ["external-secrets.io"]
    resources: ["pushsecrets"]
    verbs: ["create", "update", "delete"]
  - apiGroups: ["generators.external-secrets.io"]
    resources: ["generatorstates"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"]
  # ClusterSecretStore access (Mode 2 with a ClusterSecretStore)
  - apiGroups: ["external-secrets.io"]
    resources: ["clustersecretstores"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["external-secrets.io"]
    resources: ["clustersecretstores/status"]
    verbs: ["get", "update", "patch"]
  - apiGroups: [""]
    resources: ["serviceaccounts", "namespaces"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"]
  # Token creation pinned to ESO's own ServiceAccount (Mode 2 hardening)
  - apiGroups: [""]
    resources: ["serviceaccounts/token"]
    verbs: ["create"]
    resourceNames: ["<release-name>-external-secrets"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: <release-name>-external-secrets
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: <release-name>-external-secrets
subjects:
  - kind: ServiceAccount
    name: <release-name>-external-secrets
    namespace: astronomer
```

### config-syncer RBAC (Option B synced SecretStore)

The platform's config-syncer component performs the Option B sync: it copies the `commander-sync`-annotated credentials Secret and `SecretStore` from the `astronomer` namespace into each Deployment namespace, so it needs write access to `secrets` and `secretstores` in every target namespace. Mode 3 doesn't use this component — there, each namespace has its own pre-provisioned `SecretStore` and nothing is synced.

The chart creates this RBAC automatically in a standard install. Provision the Role and RoleBinding below yourself only if you run with cluster-scoped roles disabled, so config-syncer isn't otherwise granted namespace access. Create them in each Deployment namespace, binding the `<release-name>-config-syncer` ServiceAccount in `astronomer`.

```yaml theme={null}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: <release-name>-config-syncer
  namespace: <deployment-namespace>
  labels:
    app.kubernetes.io/name: config-syncer
    app.kubernetes.io/instance: <release-name>
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "list", "patch", "update"]
  - apiGroups: ["external-secrets.io"]
    resources: ["secretstores"]
    verbs: ["create", "get", "list", "patch", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: <release-name>-config-syncer
  namespace: <deployment-namespace>
  labels:
    app.kubernetes.io/name: config-syncer
    app.kubernetes.io/instance: <release-name>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: <release-name>-config-syncer
subjects:
  - kind: ServiceAccount
    name: <release-name>-config-syncer
    namespace: astronomer
```

## Mode 3: per-namespace RBAC

In Mode 3 (customer-managed isolated identity), each Deployment gets its own backend identity, and you pre-provision per-namespace RBAC for every platform component that operates in the pool namespaces. This document covers the ESO and Commander RBAC. The remaining platform components — kube-state-metrics, the Houston DB bootstrapper hook, Prometheus, and NGINX — are covered in [Component RBAC for restricted mode](/docs/astro-private-cloud/v-2-x/namespace-pools#component-rbac-for-restricted-mode). Create these objects in the `astronomer` namespace and in each pool namespace, in addition to the standard [namespace pools](/docs/astro-private-cloud/v-2-x/namespace-pools) setup, and replicate them with matching namespace names on every cluster a Deployment can fail over to.

Each `RoleBinding` names the component's ServiceAccount in the `astronomer` namespace as a cross-namespace subject. The chart installs those ServiceAccounts release-prefixed: `<release-name>-external-secrets` and `<release-name>-commander`.

Rather than hand-writing every object, you can generate the full set with the provided script. See [Generate the platform RBAC](#generate-the-platform-rbac-mode-3). The manifests below are the reference for what that script produces.

### Per-namespace ServiceAccount and SecretStore (isolated identity)

For isolated backend identity, pre-provision a ServiceAccount and a namespaced `SecretStore` that authenticates as it through workload identity in each pool namespace. In namespace pools you provision this chain when the namespaces are created, before any Deployment exists, so name the ServiceAccount after the pre-created pool namespace (shown here as `<pool-namespace>-eso`) rather than a Deployment ID, which isn't yet known. Each pool namespace hosts one Deployment at a time, so a per-namespace identity is effectively per-Deployment. Annotate the ServiceAccount for your provider. Omit the annotation for Vault Kubernetes auth.

```yaml theme={null}
# AWS IRSA
apiVersion: v1
kind: ServiceAccount
metadata:
  name: <pool-namespace>-eso
  namespace: <pool-namespace>
  annotations:
    eks.amazonaws.com/role-arn: <per-namespace-iam-role-arn>
---
# GKE Workload Identity (alternative)
# metadata.annotations:
#   iam.gke.io/gcp-service-account: <per-namespace-gcp-service-account>
```

Name the `SecretStore` the same in every pool namespace — the value of `global.dataPlaneFailover.externalSecretManagerName` (shown here as `astronomer-secret-store`). The name must be identical across namespaces. The `provider` block and identity can differ per namespace.

```yaml theme={null}
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: astronomer-secret-store
  namespace: <pool-namespace>
spec:
  provider:
    aws:
      secretsManager:
        forceDeleteWithoutRecovery: true
      service: SecretsManager
      region: us-east-2
      auth:
        jwt:
          serviceAccountRef:
            name: <pool-namespace>-eso
```

### ESO Role and RoleBinding

Grants the ESO controller ServiceAccount the reconcile verbs it needs inside the namespace. Create it in the `astronomer` namespace and in each pool namespace.

```yaml theme={null}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: <release-name>-external-secrets
  namespace: <pool-namespace>
rules:
  - apiGroups: ["external-secrets.io"]
    resources: ["secretstores", "externalsecrets", "pushsecrets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["external-secrets.io"]
    resources: ["externalsecrets", "externalsecrets/status", "secretstores", "secretstores/status", "pushsecrets", "pushsecrets/status"]
    verbs: ["get", "update", "patch"]
  - apiGroups: ["external-secrets.io"]
    resources: ["externalsecrets"]
    verbs: ["create", "update", "delete"]
  - apiGroups: ["external-secrets.io"]
    resources: ["pushsecrets"]
    verbs: ["create", "update", "delete"]
  - apiGroups: ["generators.external-secrets.io"]
    resources: ["generatorstates"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"]
  - apiGroups: [""]
    resources: ["serviceaccounts", "namespaces"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: <release-name>-external-secrets
  namespace: <pool-namespace>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: <release-name>-external-secrets
subjects:
  - kind: ServiceAccount
    name: <release-name>-external-secrets
    namespace: astronomer
```

### Token-creation Role and RoleBinding

Required in Mode 3, for any backend: the per-namespace `SecretStore` authenticates as the per-namespace ServiceAccount (`auth.jwt.serviceAccountRef`), so ESO must mint that ServiceAccount's token to assume its identity — whether the backend is AWS (IRSA), GCP (Workload Identity), or Vault (Kubernetes auth). More broadly, ESO needs ServiceAccount-token creation whenever a store authenticates through a `serviceAccountRef`: always for Vault, and for AWS or GCP only when you authenticate with a ServiceAccount token (workload identity) rather than static credentials or the ESO controller's own IRSA or GKE Workload Identity.

The `RoleBinding` subject is the ESO controller ServiceAccount (`<release-name>-external-secrets` in `astronomer`), because the controller is the process that calls the Kubernetes `TokenRequest` API. The rule's `resourceNames` pins which ServiceAccount the controller may mint a token for — the per-namespace SA `<pool-namespace>-eso` — so it can assume that one identity and nothing else. Create the Role and RoleBinding alongside the ESO Role in each pool namespace.

```yaml theme={null}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: <release-name>-external-secrets-token
  namespace: <pool-namespace>
rules:
  - apiGroups: [""]
    resources: ["serviceaccounts/token"]
    verbs: ["create"]
    resourceNames: ["<pool-namespace>-eso"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: <release-name>-external-secrets-token
  namespace: <pool-namespace>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: <release-name>-external-secrets-token
subjects:
  - kind: ServiceAccount
    name: <release-name>-external-secrets
    namespace: astronomer
```

### Commander Role and RoleBinding

Create in the `astronomer` namespace and in each pool namespace, binding the `<release-name>-commander` ServiceAccount.

```yaml theme={null}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: <release-name>-commander
  namespace: <pool-namespace>
  labels:
    tier: houston
    release: <release-name>
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["serviceaccounts"]
    verbs: ["create", "delete", "get", "patch", "list", "watch"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["roles"]
    verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["create", "delete", "deletecollection", "get", "list", "update", "watch", "patch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "deletecollection", "get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "get"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["create", "delete", "get", "list", "update", "watch"]
  - apiGroups: [""]
    resources: ["limitranges"]
    verbs: ["create", "delete", "get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["nodes/proxy"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["create", "delete", "get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["replicationcontrollers"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["resourcequotas"]
    verbs: ["create", "delete", "get", "list", "patch", "watch"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: ["apps"]
    resources: ["statefulsets"]
    verbs: ["create", "delete", "get", "list", "patch", "watch"]
  - apiGroups: ["apps"]
    resources: ["daemonsets"]
    verbs: ["create", "delete", "get", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
  - apiGroups: ["autoscaling"]
    resources: ["horizontalpodautoscalers"]
    verbs: ["list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["list", "watch", "create", "delete", "get", "deletecollection"]
  - apiGroups: ["batch"]
    resources: ["cronjobs"]
    verbs: ["create", "delete", "get", "list", "patch", "watch", "deletecollection"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "delete", "patch", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "create", "delete", "patch", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses/status"]
    verbs: ["update", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["networkpolicies"]
    verbs: ["create", "delete", "get", "patch", "list", "watch"]
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["rolebindings"]
    verbs: ["create", "delete", "get", "patch", "list", "watch"]
  - apiGroups: ["authentication.k8s.io"]
    resources: ["tokenreviews"]
    verbs: ["create", "delete", "list", "watch"]
  - apiGroups: ["authorization.k8s.io"]
    resources: ["subjectaccessreviews"]
    verbs: ["create", "delete", "list", "watch"]
  - apiGroups: ["policy"]
    resources: ["poddisruptionbudgets"]
    verbs: ["create", "delete", "get", "list", "patch", "watch"]
  - apiGroups: ["external-secrets.io"]
    resources: ["externalsecrets", "pushsecrets", "secretstores"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["external-secrets.io"]
    resources: ["generatorstates"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: <release-name>-commander
  namespace: <pool-namespace>
  labels:
    tier: houston
    release: <release-name>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: <release-name>-commander
subjects:
  - kind: ServiceAccount
    name: <release-name>-commander
    namespace: astronomer
```

## Generate the platform RBAC (Mode 3)

Mode 3 needs RBAC for every platform component that runs in the pool namespaces, not just ESO. Rather than hand-writing each object, generate the complete set with the provided script. It emits a single bundle that covers both the ESO-specific RBAC in this document and the namespace pools platform-component RBAC in [Component RBAC for restricted mode](/docs/astro-private-cloud/v-2-x/namespace-pools#component-rbac-for-restricted-mode):

* *ESO* — the reconcile `Role` and `RoleBinding` and the `resourceNames`-pinned token-creation `Role` and `RoleBinding`.
* *Commander* — the per-namespace `Role` and `RoleBinding`.
* *kube-state-metrics* — the per-namespace `Role` and `RoleBinding`.
* *Houston DB bootstrapper hook* — a `Role` and `RoleBinding` created in the `astronomer` namespace only.
* *Prometheus* — a cluster-scoped `ClusterRole` and `ClusterRoleBinding` for scrape access.
* *NGINX* — a cluster-scoped `ClusterRole` and `ClusterRoleBinding`, plus its namespaced config `Role` and `RoleBinding`.

Generate and apply the full set across the platform and pool namespaces:

```bash theme={null}
python bin/generate-namespace-pools-rbac.py \
  --release-name <release-name> \
  --release-namespace astronomer \
  --namespaces astronomer,<pool-namespace-1>,<pool-namespace-2> | kubectl apply -f -
```

The options are:

* `--release-name` — the Helm release name of the platform. The script uses it to prefix every generated ServiceAccount, `Role`, and `ClusterRole` (for example `<release-name>-commander` and `<release-name>-external-secrets`).
* `--release-namespace` — the platform namespace (`astronomer`), where the component ServiceAccounts live. Every generated `RoleBinding` names them here as a cross-namespace subject.
* `--namespaces` — a comma-separated list of the namespaces to emit per-namespace RBAC for. Include the platform namespace and every pool namespace.

To generate only the limited-privilege installer identity used to run the platform install, rather than the full component set, pass `--installer-user` instead:

```bash theme={null}
python bin/generate-namespace-pools-rbac.py \
  --installer-user <installer> \
  --namespaces astronomer | kubectl apply -f -
```

The per-namespace ESO ServiceAccount and `SecretStore` carry provider-specific identity, so the script doesn't generate them — create those as shown in [Per-namespace ServiceAccount and SecretStore (isolated identity)](#per-namespace-serviceaccount-and-secretstore-isolated-identity).

## Leader-election RBAC (optional ESO high availability)

To run the ESO controller with more than one replica for resiliency, enable leader election so only one replica reconciles secrets at a time. See [Run ESO with multiple replicas](/docs/astro-private-cloud/v-2-x/configure-external-secrets-operator-security#run-eso-with-multiple-replicas). In Modes 2 and 3, where you manage ESO's RBAC, create the leader-election Role and RoleBinding in the `astronomer` namespace, binding the ESO ServiceAccount:

```yaml theme={null}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: <release-name>-external-secrets-leaderelection
  namespace: astronomer
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["external-secrets-controller"]
    verbs: ["get", "update", "patch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["create"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: <release-name>-external-secrets-leaderelection
  namespace: astronomer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: <release-name>-external-secrets-leaderelection
subjects:
  - kind: ServiceAccount
    name: <release-name>-external-secrets
    namespace: astronomer
```

## Related documentation

* [External Secrets Operator security](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security)
* [Configure External Secrets Operator security](/docs/astro-private-cloud/v-2-x/configure-external-secrets-operator-security)
* [Component RBAC for restricted mode](/docs/astro-private-cloud/v-2-x/namespace-pools#component-rbac-for-restricted-mode)


# Add a Deployment user role with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-add-deployment-user-role

Grant a user a role on a Deployment using the APC API deploymentAddUserRole mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

Use the `deploymentAddUserRole` GraphQL mutation to add a user, or `deploymentUpdateUserRole` to change an existing user's role.

```graphql wrap theme={null}
mutation {
  deploymentAddUserRole(
    deploymentId: "<deployment-id>"
    email: "<user-email>"
    role: DEPLOYMENT_EDITOR
  ) {
    id
  }
}
```

Use `role: DEPLOYMENT_VIEWER` to restrict a user to read-only access instead.

For the full role-assignment flow, see [Manage users on Astro Private Cloud](/docs/astro-private-cloud/v-2-x/manage-platform-users).


# Add a System Admin with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-add-system-admin

Add a user as a System Admin using the APC API.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

## Add a System Admin

To add a user as a System Admin through the APC API, you need the following values:

* The user's ID. To retrieve this, request the `id` value in a `users` query or run `astro workspace user list`.
* System Admin permissions.

You can then run the following query to add the user as a System Admin.

```graphql wrap theme={null}
mutation createSystemRoleBinding (
    $userId: ID! = "<user-id>"
    $role: Role! = SYSTEM_ADMIN
) {
  createSystemRoleBinding(
    userId: $userId
    role: $role
  ) {
    id
  }
}
```


# Use the APC API on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-api

Learn how to make requests to your Astro Private Cloud installation using the APC API.

The APC API allows you to build applications that provision and manage resources on Astro Private Cloud (APC). You can use the APC API to perform CRUD operations on entities scoped to the Astronomer platform, including Airflow Deployments, Workspaces, and users.

For example, you can:

* Create, update, or delete a Workspace
* Create, update, or delete a Deployment
* Look up a Deployment's resource config
* Add a user to a Workspace or change their role
* Make a user a System Administrator

Anything you can do with the Astro Private Cloud UI, you can do programmatically with the APC API.

Clusters correspond to registered data planes, which you add and configure outside the APC API. To manage clusters, see [Register a data plane](/docs/astro-private-cloud/v-2-x/register-data-plane).

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

## Schema overview

The APC API uses the [GraphQL API](https://graphql.org/learn/) query language. You can make GraphQL requests using standard REST API tools such as [HTTP](https://www.apollographql.com/blog/making-graphql-requests-using-http-methods) and cURL. GraphQL APIs support two main request types: *queries* and *mutations*. You can run these requests against *objects* which are similar to endpoints in REST APIs.

A *query* is a request for specific information and is similar to a `GET` request in a REST API. The primary difference between GraphQL queries and REST API queries is that GraphQL queries only return the fields that you specify within your request. For example, consider the following query to retrieve information about a user:

```graphql wrap theme={null}
query User {
  users(user: { email: "name@mycompany.com"} )
  {
    id
    roleBindings {role}
  }
}
```

The APC API would only return the ID and role bindings for the user because those are the only fields specified in the query.

A *mutation* is a request to update data for a specific object. Different mutations exist for creating, updating, and deleting objects. When you write a mutation request, it's best practice to use [variables](https://graphql.org/learn/queries/#variables) to store the data you want to update. For example, consider the following example mutation:

```graphql wrap theme={null}
mutation workspaceAddUser(
  $workspaceUuid: Uuid = "<your-workspace-uuid>"
  $email: String! = "<user-email-address>"
  $role: Role! = <user-workspace-role>
  $bypassInvite: Boolean! = true
) {
  workspaceAddUser(
    workspaceUuid: $workspaceUuid
    email: $email
    role: $role
    bypassInvite: $bypassInvite
  ) {
    id
  }
}
```

In this mutation, the values to update are formatted as variables in the first part of the request, then applied in the second half. Variables marked with a `!` are required in order for the query to complete. Lastly, the mutation requests the APC API to return `id` of the added user to confirm that the mutation was successful. In this way, it's possible to make a mutation and a query in a single request.

For more basic GraphQL usage rules and examples, see the [GraphQL documentation](https://graphql.org/learn/queries/).


# Authenticate to the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-api-authenticate

Authenticate to the APC API to make requests to your Astro Private Cloud installation.

An APC API request requires a token so that the request can be authenticated and authorized with a specific level of access on Astro Private Cloud.

You can retrieve a token in one of the following ways:

* Create a [service account](/docs/astro-private-cloud/v-2-x/ci-cd#service-account-authentication) and note its **API Key**. This is recommended for most production workflows.
* Go to `https://app.<your-base-domain>/token`, sign in with your user account, and copy the API token that appears. This is recommended if you want to test API requests with your own user credentials. API tokens retrieved this way expire after 24 hours or when an Admin users changes your level of access.

You then need to add your token as an `Authorization` header to your APC API requests. For example, the following cURL command uses `'Authorization: <my-api-token>'` to authenticate a request to add a new Workspace user:

```bash wrap theme={null}
curl 'https://houston.<your-base-domain>/v1' -H 'Accept-Encoding: gzip, deflate, br' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'Connection: keep-alive' -H 'DNT: 1' -H 'Origin: https://houston.<your-base-domain>' -H 'authorization: <my-api-token>' --data-binary '{"query":"mutation workspaceAddUser(\n    $workspaceUuid: Uuid = \"<your-workspace-uuid>\"\n    $email: String! = \"<user-email-address>\"\n    $role: Role! = <user-workspace-role>\n    $bypassInvite: Boolean! = true\n  ) {\n    workspaceAddUser(\n      workspaceUuid: $workspaceUuid\n      email: $email\n      role: $role\n      bypassInvite: $bypassInvite\n    ) {\n      id\n    }\n  }"}' --compressed
```


# Develop and test APC API queries
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-api-develop-test

Use the GraphQL playground to develop and test APC API queries for Astro Private Cloud.

The Astronomer APC API is available in a [GraphQL playground](https://www.apollographql.com/docs/apollo-server/v2/testing/graphql-playground/) where you can view the APC API schema and documentation, as well as model, test, and export requests. To test requests in the playground, you must have a valid [authentication token](/docs/astro-private-cloud/v-2-x/houston-api-authenticate).

1. Access the Astronomer APC API GraphQL playground at `https://houston.<your-base-domain>/v1`.
2. Click **HTTP Headers**.
3. Add your authentication token in the following format:

   ```graphql wrap theme={null}
   {"authorization": "<your-api-token>"}
   ```

You can now run APC API queries directly in your browser using the **Play** button in the GraphQL playground.


# Example APC API queries
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-api-example-queries

Example APC API queries for common Astro Private Cloud operations.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

You can retrieve common information for specific Astronomer objects by using the following sample queries.

<a />

## Find the ID of a workspace you belong to

Use the `workspaces` query to find the ID of a Workspace that you belong to. Optionally, you can choose to filter by Deployment label.

```graphql wrap theme={null}
query {
  workspaces(label:"example-workspace") {
    id,
    label
  }
}
```

<a />

### Find the ID of all workspaces

System administrators can use the `sysWorkspaces` query to perform a bulk-fetch of all Workspaces and their respective IDs using the `sysWorkspaces` query. After retrieving the full list, you can filter the Workspaces within your application to locate the ID corresponding to the label of interest.

```graphql wrap theme={null}
query {
  sysWorkspaces {
    id,
    label
  }
}
```

### Query Deployment details

You can use the `workspaceDeployment` query to retrieve details about a Deployment in a given Workspace. It requires the following inputs:

* **Workspace ID**: To retrieve this value, use the [`sysWorkspaces`](#sysWorkspaces) or [`workspaces`](#workspaces) query, or run `astro workspace list`. Alternatively, open a Workspace in the Astro Private Cloud UI and copy the value after `/w/` in your Workspace URL, for example, `https://app.basedomain/w/<workspace-id>`.

* **Deployment release name**: To retrieve this value, run `astro deployment list` in your Workspace. Alternatively, you can copy the **Release name** from your Deployment's **Settings** tab in the Astro Private Cloud UI.

  The `workspaceDeployment` query can return also any of the fields under `Type Details`, such as:

* `config`

* `uuid`

* `status`

* `createdAt`

* `updatedAt`

* `roleBindings`

For example, you can run the following query to retrieve the Deployment's:

* ID
* Health status
* Creation time
* Update time
* Users

```graphql wrap theme={null}
query workspaceDeployment {
  workspaceDeployment(
    releaseName: "mathematical-probe-2087"
    workspaceUuid: "ck35y9uf44y8l0a19cmwd1x8x"
  )
  {
    id
    status
    createdAt
    updatedAt
    roleBindings {
      id,
      role,
      user {
        username,
        emails {
          primary
        }
      }
    }
  }
}
```

<a />

### Query a single Workspace

Use the `workspace` query to retrieve details about a single Workspace, including its role bindings. Provide the `workspaceUuid`, which you can retrieve with the [`workspaces`](#workspaces) query or by running `astro workspace list`.

```graphql wrap theme={null}
query GetWorkspace {
  workspace(workspaceUuid: "<workspace-id>") {
    id
    label
    description
    createdAt
    updatedAt
    roleBindings {
      role
      user {
        id
        username
      }
    }
  }
}
```

<a />

### List Workspace users

Use the `workspaceUsers` query to list the users in a Workspace and their roles.

```graphql wrap theme={null}
query ListWorkspaceUsers {
  workspaceUsers(workspaceUuid: "<workspace-id>") {
    id
    username
    fullName
    emails {
      address
    }
    roleBindings {
      workspace {
        id
      }
      role
    }
  }
}
```

<a />

### List Deployments in a Workspace

Use the `workspaceDeployments` query to list all Deployments in a Workspace. Provide the `workspaceUuid`, and optionally a `releaseName` to filter to a single Deployment.

```graphql wrap theme={null}
query GetDeployments {
  workspaceDeployments(workspaceUuid: "<workspace-id>") {
    id
    label
    releaseName
    version
    runtimeVersion
    clusterId
    createdAt
    updatedAt
  }
}
```

<Note>
  The `clusterId` field identifies the cluster, or registered data plane, that hosts a Deployment. Clusters aren't created or updated through the APC API. To add a cluster, register a data plane. See [Register a data plane](/docs/astro-private-cloud/v-2-x/register-data-plane).
</Note>

### Query user details

A common query is `users`, which lets you retrieve information about multiple users at once. To use this query, you must provide:

* At least one of the following `userSearch` values:

  * `userId` (String): The user's ID
  * `userUuid`(String): The user's unique ID
  * `username` (String): The user's username
  * `email` (String): The user's email
  * `fullName` (String): The user's full name
  * `createdAt`(DateTime): When the user was created
  * `updatedAt`(DateTime): When the user was updated

The query returns the requested details for all users who exactly match the values provided for the `userSearch`. For example, the following query would retrieve the requested values for any user accounts with the email `name@mycompany.com`:

```graphql wrap theme={null}
query User {
  users(user: { email: "name@mycompany.com"} )
  {
    id
    roleBindings {role}
    status
    createdAt
  }
}
```

### Query a Deployment's effective configuration

The `Deployment` type exposes the merged `deployments` object on `effectiveConfig` (the final value after the Platform through Deployment merge). It also exposes `configOverrides` for the deployment tier only, when you need the fourth layer in isolation.

For example, `workspaceDeployment` can return the merged result alongside deployment-level overrides:

```graphql wrap theme={null}
query {
  workspaceDeployment(
    releaseName: "<release-name>"
    workspaceUuid: "<workspace-id>"
  ) {
    id
    label
    effectiveConfig
    configOverrides {
      config
    }
  }
}
```

For the full config governance model, see [Config governance](/docs/astro-private-cloud/v-2-x/config-governance).

### Query teams

### Get single team

```graphql wrap theme={null}
query {
  team(teamUuid: "<team-uuid>") {
    id
    name
    provider
    description
    createdAt
    updatedAt
    users {
      id
      username
      emails {
        address
      }
    }
    roleBindings {
      role
      workspace {
        id
        label
      }
      deployment {
        id
        label
      }
    }
  }
}
```

### List teams with search

<Note>
  `searchPhrase` requires a minimum of three characters.
</Note>

```graphql wrap theme={null}
query {
  paginatedTeams(
    take: 20
    pageNumber: 1
    searchPhrase: "engineering"
  ) {
    teams {
      id
      name
      provider
      users {
        id
      }
    }
    count
  }
}
```

### List workspace teams

```graphql wrap theme={null}
query {
  workspaceTeams(workspaceUuid: "<workspace-uuid>") {
    id
    name
    roleBindings {
      role
    }
  }
}
```

### List deployment teams

```graphql wrap theme={null}
query {
  deploymentTeams(deploymentUuid: "<deployment-uuid>") {
    id
    name
    roleBindings {
      role
    }
  }
}
```

For the full teams model, roles, and error reference, see [team management reference](/docs/astro-private-cloud/v-2-x/team-management-api).

### Query clusters

### List clusters

The `paginatedClusters` query returns clusters the caller has access to. Pagination uses the `take` argument, plus either `cursor` (a cluster UUID) or `pageNumber`. The response object contains a `clusters` list and a total `count`.

```graphql wrap theme={null}
query {
  paginatedClusters(
    take: 50
    status: ACTIVE
  ) {
    clusters {
      id
      name
      status
      statusReason
      healthStatus
      k8sVersion
      cloudProvider
      region
      createdAt
      updatedAt
    }
    count
  }
}
```

### Get a single cluster

```graphql wrap theme={null}
query {
  cluster(id: "<cluster-id>") {
    id
    name
    status
    statusReason
    healthStatus
    k8sVersion
    cloudProvider
    region
    dpChartVersion
    commanderVersion
    config
    configOverride
  }
}
```

<Note>
  The `healthStatus` field returns a JSON object containing the full health payload the APC API received from the deployment orchestrator, not a single string. The `statusReason` field is also a JSON object.
</Note>

### Filter by cloud provider and region

```graphql wrap theme={null}
query {
  paginatedClusters(
    status: INACTIVE
    cloudProvider: "aws"
    region: "us-east-1"
    take: 25
  ) {
    clusters {
      id
      name
      statusReason
    }
    count
  }
}
```

Other supported filter arguments include `searchPhrase`, `k8sVersion`, `id`, `sortBy`, and `sortDirection`.

For status values and troubleshooting, see [Manage cluster status](/docs/astro-private-cloud/v-2-x/cluster-status-management).

<a />

### Force a cluster metadata reconciliation

Use the `reconcileClusterMetadataJob` query to make the APC API refetch metadata from the deployment orchestrator immediately, instead of waiting for the next CronJob run. The query accepts a list of cluster UUIDs; if you pass `null` or omit the argument, the APC API reconciles every cluster the caller is authorized to update.

```graphql wrap theme={null}
query {
  reconcileClusterMetadataJob(
    clusterIds: ["<cluster-id-1>", "<cluster-id-2>"]
  ) {
    successfulClusterIds
    failedClusterIds
    skippedClusterIds
  }
}
```

A cluster appears in `skippedClusterIds` when it lacks a data plane URL or when the caller isn't authorized to reconcile it.

See [Manage cluster status](/docs/astro-private-cloud/v-2-x/cluster-status-management) for when to use this query.

### Clean up Airflow metadata

The following examples show different queries you can use depending on your needs. For the full parameter reference, see [Clean up and delete task metadata](/docs/astro-private-cloud/v-2-x/clean-up-task-metadata#apc-api-parameters).

### Clean up Deployments per Workspace

```graphql wrap theme={null}
query cleanupAirflowDb(
  $olderThan: Int!
  $dryRun: Boolean!
  $outputPath: String!
  $dropArchives: Boolean!
  $provider: String!
  $bucketName: String!
  $providerEnvSecretName: String!
  $deploymentIds: [Id]
  $workspaceId: Uuid
  $tables: String!
  $connectionId: String
) {
  cleanupAirflowDb(
    olderThan: $olderThan
    dryRun: $dryRun
    outputPath: $outputPath
    dropArchives: $dropArchives
    provider: $provider
    bucketName: $bucketName
    providerEnvSecretName: $providerEnvSecretName
    workspaceId: $workspaceId
    tables: $tables
    connectionId: $connectionId
  )
}
```

Query variables to clean up all Deployments older than 1 day within a Workspace that uses GCP as a cloud provider:

```graphql wrap theme={null}
{
	"olderThan": 1,
	"dryRun": true,
	"outputPath": "",
	"dropArchives": true,
	"provider":  "gcp",
	"bucketName" : "",
	"connectionId":  "",
	"tables": "callback_request,celery_taskmeta,celery_tasksetmeta,dag,dag_run,dataset_event,import_error,job,log,session,sla_miss,task_fail,task_instance,task_reschedule,trigger,xcom",
	"providerEnvSecretName": "GCP_PASS",
	"workspaceId": "cma40n66l000008l89nye86o1"
}
```

### Clean up specific Deployments

Query variables to clean up specific Deployments older than 1 day within a Workspace:

```graphql wrap theme={null}
{
	"olderThan": 1,
	"dryRun": true,
	"outputPath": "",
	"dropArchives": true,
	"provider":  "gcp",
	"bucketName" : "",
	"connectionId":  "",
	"tables": "callback_request,celery_taskmeta,celery_tasksetmeta,dag,dag_run,dataset_event,import_error,job,log,session,sla_miss,task_fail,task_instance,task_reschedule,trigger,xcom",
	"providerEnvSecretName": "GCP_PASS",
	"deploymentIds": ["cma42zc67000108l89eb37iy5","cma42zjdp000208l8g16ygm6m"],
	"workspaceId": "cma42z570000008l8f6rpc72f"
}
```

### Clean up using an Airflow connection ID

<Warning>Requires configuring an Airflow Connection ID, `connectionId`, from the Airflow UI or CLI.</Warning>

Query variables to clean up Deployments and export the cleanup logs to the storage provider configured in an [Airflow Connection](/docs/learn/connections):

```graphql wrap theme={null}
{
	"olderThan": 1,
	"dryRun": true,
	"outputPath": "",
	"dropArchives": true,
	"provider":  "gcp",
	"bucketName" : "",
	"connectionId":  "<airflow_connection_id>",
	"tables": "callback_request,celery_taskmeta,celery_tasksetmeta,dag,dag_run,dataset_event,import_error,job,log,session,sla_miss,task_fail,task_instance,task_reschedule,trigger,xcom",
	"deploymentIds": ["cm6q3jpn61741517mhonzgcgz7","cm6q3jpn61741517mhonzgcgz7"],
	"workspaceId": "cm5nj9wly007617iox80beute"
}
```

### Configure custom Pod resources

If you don't configure a default Pod CPU or memory resource amount, or want to override one, make a query that sets `resourceSpec`:

```graphql wrap theme={null}
query cleanupAirflowDb(
    $olderThan: Int!
    $dryRun: Boolean!
    $outputPath: String!
    $dropArchives: Boolean!
    $provider: String!
    $bucketName: String!
    $providerEnvSecretName: String!
    $tables: String!
    $resourceSpec: JSON
  ) {
    cleanupAirflowDb(
      olderThan: $olderThan
      dryRun: $dryRun
      outputPath: $outputPath
      dropArchives: $dropArchives
      provider: $provider
      bucketName: $bucketName
      providerEnvSecretName: $providerEnvSecretName
      tables: $tables
      resourceSpec: $resourceSpec
    )
  }
```

Query variables that configure resource requests and limits for the cleanup run:

```graphql wrap theme={null}
{
  "resourceSpec": {
    "requests": {
      "cpu": "100m",
      "memory": "5000Mi"
    },
  "limits": {
      "cpu": "100m",
      "memory": "5000Mi"
    }
  },
  "olderThan": 1,
  "dryRun": false,
  "outputPath": "/abc",
  "dropArchives": false,
  "provider": "aws",
  "bucketName": "test",
  "providerEnvSecretName": "test-secret",
  "tables": "dag"
}
```

For the full parameter reference, see [Clean up and delete task metadata](/docs/astro-private-cloud/v-2-x/clean-up-task-metadata#apc-api-parameters).

### Trigger task usage data cleanup

Use the `cleanupTaskUsageDataJob` query to manually trigger a purge of task usage metrics and audit logs:

```graphql wrap theme={null}
query {
  cleanupTaskUsageDataJob(olderThan: 90)
}
```

<Note>
  Minimum retention is 90 days and can't be reduced.
</Note>

For the full cleanup job reference, see [Configure cleanup jobs](/docs/astro-private-cloud/v-2-x/cleanup-cronjobs).


# Bypass user email verification with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-bypass-user-email-verification

Bypass email verification for users joining a Workspace using the APC API.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

If you don't need certain users to verify their email before they join a Workspace, you can configure a bypass when you add them to a Workspace. This can be useful for minimizing friction when programmatically inviting many users to your platform.

To run this mutation, you need:

* Workspace Admin permissions
* A Workspace ID. To retrieve this value, run `astro workspace list`. Alternatively, open a Workspace in the Astro Private Cloud UI and copy the value after `/w/` in your Workspace URL (for example `https://app.basedomain/w/<workspace-id>`).
* The user's email address.
* The user's desired role in the Workspace (`WORKSPACE_VIEWER`, `WORKSPACE_EDITOR`, `WORKSPACE_ADMIN`).

The following example mutation can be run to add a user to a Workspace as a `WORKSPACE_VIEWER`.

```graphql wrap theme={null}
mutation workspaceAddUser(
    $workspaceUuid: Uuid = "<your-workspace-uuid>"
    $email: String! = "<user-email-address>"
    $role: Role! = WORKSPACE_VIEWER
    $bypassInvite: Boolean! = true
  ) {
    workspaceAddUser(
      workspaceUuid: $workspaceUuid
      email: $email
      role: $role
      bypassInvite: $bypassInvite
    ) {
      id
    }
  }
```


# Create a Deployment service account with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-create-deployment-service-account

Create a Deployment-scoped service account for CI/CD using the APC API createDeploymentServiceAccount mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

You can also create a service account using the GraphQL API. The `deploymentUuid` field is the same Deployment ID (UUID) returned by `astro deployment list`.

```graphql wrap theme={null}
mutation {
  createDeploymentServiceAccount(
    deploymentUuid: "<deployment-id>"
    label: "CI/CD Pipeline"
    role: DEPLOYMENT_ADMIN
  ) {
    id
    apiKey
  }
}
```

Set in CI/CD environment:

```bash wrap theme={null}
export ASTRONOMER_KEY_ID=<service-account-id>
export ASTRONOMER_KEY_SECRET=<api-key>
```

For the full CI/CD setup, including Workspace-level service accounts and platform examples, see [Configure CI/CD on Astro Private Cloud](/docs/astro-private-cloud/v-2-x/ci-cd).


# Create a Deployment user with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-create-deployment-user

Add an existing Astro Private Cloud user to a Deployment using the APC API.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

To add an existing Astro Private Cloud user to a Deployment, you need:

* Workspace Admin privileges
* A Deployment ID. To retrieve this value, run `astro deployment list` or request the `id` value in the `workspaceDeployment` query.
* The ID of the user to add. To retrieve this, request the `id` value in a `users` query or run `astro workspace user list`.
* The role to add the user as. Can be `DEPLOYMENT_ADMIN`, `DEPLOYMENT_EDITOR`, or `DEPLOYMENT_VIEWER`.

The following query adds a user to a Deployment as a Deployment viewer, then returns the user and Deployment information back to the requester.

```graphql wrap theme={null}
mutation AddDeploymentUser(
  $userId: Id! = "<user-id>",
  $email: String! = "usertoadd@mycompany.com",
  $deploymentId: Id! = "<some_id>",
  $role: Role! = DEPLOYMENT_VIEWER
)
{
		deploymentAddUserRole(
			userId: $userId
			email: $email
			deploymentId: $deploymentId
			role: $role
		) {
			id
			user {
				username
			}
			role
			deployment {
				id
				releaseName
			}
		}
	}
```


# Create a System Service Account Token with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-create-system-service-account-token

Create a system service account token for external integrations using the APC API.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

System administrators can create a global system service account token to integrate with external systems, such as CI/CD pipelines. Use the following example mutation to create the service account token:

```graphql wrap theme={null}
mutation systemSACreate {
  createSystemServiceAccount(
    label: "your-sa"
    role: SYSTEM_ADMIN
  ) {
    id
    apiKey
  }
}
```

Save a copy of the returned API key value in a secure place as it won't be displayed again.


# Create a team with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-create-team

Create a local or IdP-synced team using the APC API createTeam mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

### Create local team

```graphql wrap theme={null}
mutation {
  createTeam(
    name: "Data Engineering"
    description: "Data engineering team"
    provider: "local"
    userIds: ["<user-uuid-1>", "<user-uuid-2>"]
  ) {
    team {
      id
      name
      provider
      description
      users {
        id
        username
      }
    }
    message
  }
}
```

### Create IdP team

[IdP group sync](/docs/astro-private-cloud/v-2-x/import-idp-groups) automatically creates IdP teams, but you can also create them manually:

```graphql wrap theme={null}
mutation {
  createTeam(
    name: "engineering-group"
    description: "Synced from Okta"
    provider: "okta"
  ) {
    team {
      id
      name
      provider
    }
    message
  }
}
```

<Note>
  You can't assign users to IdP teams at creation time. The IdP syncs users to the team.
</Note>

For the full teams model, roles, and error reference, see [team management reference](/docs/astro-private-cloud/v-2-x/team-management-api).


# Delete a Deployment with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-delete-deployment

Delete a Deployment on Astro Private Cloud using the APC API.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

To delete a Deployment, you need:

* Either System Admin or Workspace Admin permissions
* A Deployment ID. To retrieve this value, run `astro deployment list` or request the `id` value in the `workspaceDeployment` query.

The following example mutation deletes a Deployment, then returns the ID of the Deployment to confirm that it was successfully deleted.

```graphql wrap theme={null}
mutation DeleteDeployment {
  deleteDeployment (
    deploymentUuid: "<deployment-id>"
  ) {
    id
  }
}
```


# Delete Deployment configuration with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-delete-deployment-config

Remove all Deployment-level configuration overrides using the APC API deleteDeploymentConfig mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

Deleting deployment configuration removes all deployment-level overrides, reverting the Deployment to use workspace-level, cluster-level, and platform-level defaults.

Use the `deleteDeploymentConfig` mutation:

```graphql wrap theme={null}
mutation {
  deleteDeploymentConfig(
    deploymentUuid: "<deployment-id>"
    reason: "Revert to workspace/cluster defaults"
  ) {
    id
    config
    deletedAt
  }
}
```

For the full config governance model these overrides participate in, see [Config governance](/docs/astro-private-cloud/v-2-x/config-governance).


# Delete a user with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-delete-user

Delete a user from Astro Private Cloud using the APC API.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

To delete a user from Astro Private Cloud, you need:

* System Admin permissions
* The ID of the user to delete. To retrieve this, request the `id` value in a `users` query or run `astro workspace user list`.

The following query removes a user, then returns information about the deleted user.

```graphql wrap theme={null}
mutation removeUser {
	removeUser (
    id: "<user-id>"
  ) {
    uuid
    emails {address}
    status
  }
}
```


# Example mutations overview
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-example-mutations-overview

Overview of the APC API example mutations, grouped by the resource each mutation acts on.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

The following pages show example GraphQL mutations for common APC API operations, grouped by the resource they act on.

## Deployment lifecycle

* [Upsert Deployment](/docs/astro-private-cloud/v-2-x/houston-upsert-deployment)
* [Delete Deployment](/docs/astro-private-cloud/v-2-x/houston-delete-deployment)

## Deployment configuration

* [Update Deployment configuration](/docs/astro-private-cloud/v-2-x/houston-update-deployment-config)
* [Delete Deployment configuration](/docs/astro-private-cloud/v-2-x/houston-delete-deployment-config)
* [Update a Deployment image](/docs/astro-private-cloud/v-2-x/houston-update-deployment-image)
* [Update a Deployment KEDA config](/docs/astro-private-cloud/v-2-x/houston-update-deployment-keda-config)
* [Update environment variables](/docs/astro-private-cloud/v-2-x/houston-update-environment-variables)

## User accounts

* [Create Deployment user](/docs/astro-private-cloud/v-2-x/houston-create-deployment-user)
* [Delete user](/docs/astro-private-cloud/v-2-x/houston-delete-user)
* [Verify user email](/docs/astro-private-cloud/v-2-x/houston-verify-user-email)
* [Bypass user email verification](/docs/astro-private-cloud/v-2-x/houston-bypass-user-email-verification)
* [Add a System Admin](/docs/astro-private-cloud/v-2-x/houston-add-system-admin)

## Service accounts and roles

* [Create System Service Account Token](/docs/astro-private-cloud/v-2-x/houston-create-system-service-account-token)
* [Create a Deployment service account](/docs/astro-private-cloud/v-2-x/houston-create-deployment-service-account)
* [Add a Deployment user role](/docs/astro-private-cloud/v-2-x/houston-add-deployment-user-role)
* [Remove a Deployment user role](/docs/astro-private-cloud/v-2-x/houston-remove-deployment-user-role)
* [Update a Deployment user's role](/docs/astro-private-cloud/v-2-x/houston-update-deployment-user-role)

## Teams

* [Create a team](/docs/astro-private-cloud/v-2-x/houston-create-team)
* [Update a team](/docs/astro-private-cloud/v-2-x/houston-update-team)
* [Remove a team](/docs/astro-private-cloud/v-2-x/houston-remove-team)
* [Assign a team role](/docs/astro-private-cloud/v-2-x/houston-assign-team-role)
* [Update a team's role](/docs/astro-private-cloud/v-2-x/houston-update-team-role)
* [Remove a team's role](/docs/astro-private-cloud/v-2-x/houston-remove-team-role)

## Workspace configuration

* [Update workspace configuration](/docs/astro-private-cloud/v-2-x/houston-update-workspace-config)
* [Delete workspace configuration](/docs/astro-private-cloud/v-2-x/houston-delete-workspace-config)

## Clusters

* [Update a cluster](/docs/astro-private-cloud/v-2-x/houston-update-cluster)
* [Deregister a cluster](/docs/astro-private-cloud/v-2-x/houston-deregister-cluster)

For read-only operations, see [Example Queries](/docs/astro-private-cloud/v-2-x/houston-api-example-queries).


# Remove a Deployment user role with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-remove-deployment-user-role

Remove a user's role from a Deployment using the APC API deploymentRemoveUserRole mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

Use the `deploymentRemoveUserRole` mutation to remove a user from a Deployment entirely:

```graphql wrap theme={null}
mutation {
  deploymentRemoveUserRole(
    deploymentId: "<deployment-id>"
    email: "<user-email>"
  ) {
    id
  }
}
```

After the role binding is removed, the user retains their previous access until their JWT expires (up to 24 hours by default).

For the full role-assignment flow, see [Manage users on Astro Private Cloud](/docs/astro-private-cloud/v-2-x/manage-platform-users).


# Update Deployment configuration with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-update-deployment-config

Add or update Deployment-level configuration overrides using the APC API updateDeploymentConfig mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

Use the `updateDeploymentConfig` mutation to add or update deployment-level overrides:

```graphql wrap theme={null}
mutation {
  updateDeploymentConfig(
    deploymentUuid: "<deployment-id>"
    deploymentsConfigOverride: {
      airflowComponents: {
        triggerer: {
          enabled: false
        }
      }
    }
    reason: "Disable triggerer for this specific deployment"
  ) {
    id
    config
  }
}
```

The same `DELETE_KEY` mechanism applies — set a key's value to `"DELETE_KEY"` to remove it from the stored override:

```graphql wrap theme={null}
mutation {
  updateDeploymentConfig(
    deploymentUuid: "<deployment-id>"
    deploymentsConfigOverride: {
      airflowComponents: {
        triggerer: {
          enabled: "DELETE_KEY"
        }
      }
    }
    reason: "Remove triggerer override, revert to workspace/cluster default"
  ) {
    id
    config
  }
}
```

For the full config governance model these overrides participate in, see [Config governance](/docs/astro-private-cloud/v-2-x/config-governance).


# Update a Deployment image with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-update-deployment-image

Point a Deployment at a newly pushed custom registry image using the APC API updateDeploymentImage mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

You can run a GraphQL mutation to update the image in your Deployment after manually pushing the image to a custom registry. This can be useful for automating code deploys using CI/CD.

APC disables this endpoint by default. To turn it on, see [Enable the update deployment image endpoint](/docs/astro-private-cloud/v-2-x/deploy-git-sync#enable-the-update-deployment-image-endpoint).

At a minimum, your mutation has to include the following:

```graphql wrap theme={null}
mutation updateDeploymentImage {
        updateDeploymentImage(
                releaseName: "<deployment-release-name>", # for example "analytics-dev"
                image: "<host>/<image-name>:<tag>",  # for example docker.io/cmart123/ap-airflow:test4
                runtimeVersion: "<runtime-version-number>" # for example "5.0.6"
        )
        {
                id
        }
}
```

Alternatively, you can run this same mutation using cURL:

```bash wrap theme={null}
curl 'https://houston.BASEDOMAIN/v1' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: <your-token>' \
  --data-binary '{"query":"mutation updateDeploymentImage {updateDeploymentImage(releaseName: \"<deployment-release-name>\", image: \"<host>/<image-name>:<tag>\",runtimeVersion: \"<runtime-version-number>\"){id}}"}'
```

For the full custom registry setup, see [Configure a custom registry for Deployment images](/docs/astro-private-cloud/v-2-x/custom-image-registry).


# Update a Deployment KEDA config with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-update-deployment-keda-config

Enable or disable KEDA autoscaling for a Deployment using the APC API updateDeploymentKedaConfig mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

Kubernetes Event-driven Autoscaling (KEDA) scales Celery workers based on task queue depth. Enable KEDA for a Deployment using the `updateDeploymentKedaConfig` mutation:

```graphql wrap theme={null}
mutation {
  updateDeploymentKedaConfig(
    deploymentUuid: "<deployment-uuid>"
    state: true
  ) {
    id
    label
  }
}
```

For sizing guidance and other Airflow resource settings, see [Scale Airflow resources](/docs/astro-private-cloud/v-2-x/scale-airflow-resources).


# Update a Deployment user's role with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-update-deployment-user-role

Change a user's role on a Deployment using the APC API deploymentUpdateUserRole mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

Use the `deploymentUpdateUserRole` mutation to change an existing user's role on a Deployment, for example to promote a `DEPLOYMENT_VIEWER` to `DEPLOYMENT_EDITOR`:

```graphql wrap theme={null}
mutation {
  deploymentUpdateUserRole(
    deploymentId: "<deployment-id>"
    email: "<user-email>"
    role: DEPLOYMENT_EDITOR
  ) {
    id
  }
}
```

For the full role-assignment flow, see [Manage users on Astro Private Cloud](/docs/astro-private-cloud/v-2-x/manage-platform-users).


# Update environment variables with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-update-environment-variables

Update environment variables for a Deployment on Astro Private Cloud using the APC API.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

To programmatically update environment variables, you need:

* A Deployment ID. To retrieve this value, run `astro deployment list` or request the `id` value in the `workspaceDeployment` query.
* A Deployment release name: To retrieve this value, run `astro deployment list` in your Workspace. Alternatively, you can copy the **Release name** from your Deployment's **Settings** tab in the Astro Private Cloud UI.

Then, in your GraphQL Playground, run the following:

```graphql wrap theme={null}
mutation UpdateDeploymentVariables {
  updateDeploymentVariables(
    deploymentUuid: ID! = "<deployment-id>",
  	releaseName: String! = "<deployment-release-name>",
    environmentVariables: [
      {key: "<environment-variable-1>",
      value: "<environment-variable-value-1>",
      isSecret: <true-or-false>},
      {key: "<environment-variable-2>",
      value: "<environment-variable-value-2>",
      isSecret: <true-or-false>}
    ]
  ) {
    key
    value
    isSecret
  }
}
```


# Upsert a Deployment with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-upsert-deployment

Create or update a Deployment with all possible configurations using the APC API upsertDeployment mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

You can use the `upsertDeployment` mutation to both create and update Deployments with all possible Deployment configurations. If you query `upsertDeployment` without a `deploymentUuid`, the APC API creates a new Deployment according to your specifications. If you specify an existing `deploymentUuid`, the APC API updates the Deployment with that ID. All queries to create a Deployment require specifying a `workspaceUuid`.

<Warning>When you make upsert updates to your Airflow Deployments, you must explicitly specify all existing environment variables, otherwise, the upsert overwrites them.</Warning>

The following query creates a new Deployment in a custom namespace `test-new-dep` and configures a Deployment environment variable `AIRFLOW__CORE__COLORED_LOG_FORMAT`.

```graphql expandable wrap theme={null}
mutation upsertDeployment(
  $workspaceUuid: Uuid,
  $deploymentUuid: Uuid,
  $label: String,
  $description: String,
  $releaseName: String,
  $namespace: String,
  $environmentVariables: [InputEnvironmentVariable],
  $image: String,
  $dockerconfigjson: JSON,
  $version: String,
  $airflowVersion: String,
  $runtimeVersion: String,
  $executor: ExecutorType,
  $workers: Workers,
  $webserver: Webserver,
  $scheduler: Scheduler,
  $triggerer: Triggerer,
  $dagProcessor: DagProcessor,
  $dagDeployment: DagDeployment,
  $properties: JSON,
  $cloudRole: String
) {
  upsertDeployment(
    workspaceUuid: $workspaceUuid,
    deploymentUuid: $deploymentUuid,
    label: $label,
    description: $description,
    releaseName: $releaseName,
    namespace: $namespace,
    environmentVariables: $environmentVariables,
    image: $image,
    dockerconfigjson: $dockerconfigjson,
    version: $version,
    airflowVersion: $airflowVersion,
    runtimeVersion: $runtimeVersion,
    executor: $executor,
    workers: $workers,
    webserver: $webserver,
    scheduler: $scheduler,
    triggerer: $triggerer,
    dagProcessor: $dagProcessor,
    dagDeployment: $dagDeployment,
    properties: $properties,
    cloudRole: $cloudRole
) {
    id
    config
    urls {
      type
      url
      __typename
    }
    properties
    description
    label
    releaseName
    namespace
    status
    type
    version
    workspace {
      id
      label
      __typename
    }
    airflowVersion
    runtimeVersion
    upsertedEnvironmentVariables {
      key
      value
      isSecret
      __typename
    }
    dagDeployment {
      type
      nfsLocation
      repositoryUrl
      branchName
      syncInterval
      syncTimeout
      ephemeralStorage
      dagDirectoryLocation
      rev
      sshKey
      knownHosts
      __typename
    }
    createdAt
    updatedAt
    __typename
  }
}
{
  "workspaceUuid": "cldemxl9502454yxe6vjlxy23",
	"environmentVariables": [
    {
      "key": "AIRFLOW__CORE__COLORED_LOG_FORMAT",
      "value": "test",
      "isSecret": false
    }
  ],
  "releaseName": "",
  "namespace": "test-new-dep",
  "executor": "CeleryExecutor",
  "workers": {},
  "webserver": {},
  "scheduler": {
    "replicas": 1
  },
  "dagProcessor": {},
  "label": "test-new-dep",
  "description": "",
  "runtimeVersion": "7.2.0",
  "properties": {
    "extra_au": 0
  },
  "dagDeployment": {
    "type": "image",
    "nfsLocation": "",
    "repositoryUrl": "",
    "branchName": "",
    "syncInterval": 1,
    "syncTimeout": 120,
    "ephemeralStorage": 2,
    "dagDirectoryLocation": "",
    "rev": "",
    "sshKey": "",
    "knownHosts": ""
  }
}
```

## More upsertDeployment examples

The following examples show `upsertDeployment` used for a few other common, narrower use cases.

### Deploy a pre-built image from CI/CD

This approach is useful when you need to integrate with systems that can't use the Astro CLI directly.

```graphql wrap theme={null}
mutation {
  upsertDeployment(
    workspaceUuid: "<workspace-uuid>"
    clusterId: "<cluster-id>"
    releaseName: "my-deployment"
    image: "quay.io/myorg/airflow:v1.2.3"
    runtimeVersion: "12.1.0"
    deployRevisionDescription: "CI/CD Pipeline Deploy"
  ) {
    id
    status
  }
}
```

The mutation accepts the following fields:

* `workspaceUuid`: The ID of the Workspace that contains the Deployment. You can provide `workspaceLabel` instead. One of the two is required.
* `clusterId`: The ID of the cluster that hosts the Deployment.
* `releaseName`: The release name of your Deployment, following the pattern `spaceyword-spaceyword-4digits`. For example, `infrared-photon-7780`.
* `image`: The full image path including registry, repository, and tag. The image must be accessible from your Astro Private Cloud data plane.
* `runtimeVersion`: The Astro Runtime version that the image is based on. For example, `12.1.0`.
* `deployRevisionDescription`: An optional description for the deploy revision, useful for tracking deploys in the APC UI.

For more information about deploying custom images with the APC API, see [Configure a custom image registry](/docs/astro-private-cloud/v-2-x/custom-image-registry).

### Configure NFS Dag deployment

Use `upsertDeployment` to configure a Deployment's Dag deployment mechanism as an NFS volume mount:

```graphql wrap theme={null}
mutation {
  upsertDeployment(
    workspaceUuid: "<workspace-uuid>"
    label: "my-deployment"
    dagDeployment: {
      type: volume
      nfsLocation: "192.168.0.1:/dags"
    }
  ) {
    id
    releaseName
  }
}
```

For the full NFS setup, see [Deploy Dags with NFS](/docs/astro-private-cloud/v-2-x/deploy-nfs).

### Skip Airflow database provisioning

To use pre-existing or managed databases, set `skipAirflowDatabaseProvisioning` to `true` in the `upsertDeployment` mutation:

```graphql wrap theme={null}
mutation {
  upsertDeployment(
    workspaceUuid: "<workspace-uuid>"
    label: "<my-deployment-label>"
    skipAirflowDatabaseProvisioning: true
  ) {
    id
  }
}
```

When using external databases, provide the connection string in your Deployment configuration. For complete setup steps with connection string examples, see [Bring your own Airflow database](/docs/astro-private-cloud/v-2-x/multi-db).


# Verify user email with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-verify-user-email

Manually verify a user's email address on Astro Private Cloud using the APC API.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

If a user on the platform has trouble verifying their email address, you can use the APC API to manually verify it for them.

To run this mutation, you'll need:

* System Admin Permissions
* The user's email address. This needs to be the email address that the user provided when they began creating an account on the platform. They must have signed up for an account, and Astro Private Cloud must already have generated an invite token for the user.

The following request verifies the email and returns `true` or `false` based on whether the mutation was successful.

```graphql wrap theme={null}
mutation verifyEmail {
	verifyEmail (
    email: "<user-email>"
  )
}
```


# Import identity provider groups into Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/import-idp-groups

Import your identity provider's organization structure into Astro Private Cloud.

You can import existing identity provider (IdP) groups into Astro Private Cloud as *Teams*, which are groups of Astro users that have the same set of permissions for a specific Workspace or Deployment. Configuring Teams lets you quickly onboard staff to APC and provides better control of user permissions.

Astro Private Cloud supports importing groups from OpenID Connect (OIDC) identity providers. Starting in Astro Private Cloud 2.1, you can also import groups from LDAP directories (OpenLDAP or Active Directory). This document covers both. For the initial LDAP authentication setup that provisions the LDAP connection itself, see [Configure LDAP authentication](/docs/astro-private-cloud/v-2-x/configure-ldap-authentication).

APC Teams function similar to users. You can:

* Assign Teams to both Workspaces and Deployments.
* Assign Viewer, Editor, or Admin roles to a Team.
* View information about users and permissions from the UI.

After you configure [SCIM](/docs/astro-private-cloud/v-2-x/integrate-auth-system#manage-users-and-teams-with-scim), you can use templates to add or remove Teams on Astro Private Cloud and manage groups of users directly from your IdP. However, if SCIM isn't available, you can pre-populate groups through the UI or API following the procedure for [Create local teams](#create-local-teams). With either process, when you create new user groups in the future, you can automatically apply a batch of permissions that they need to access Astro Private Cloud.

## Implementation considerations

Before you implement Teams, consider the following:

* By default, the first user to sign in to your Astronomer platform is automatically granted `SYSTEM ADMIN` permissions. If you configure Teams for a new Astronomer installation, Astronomer recommends signing in first as the user responsible for importing your IdP groups using the default Astronomer sign-in flow.
* Teams are based solely on the IdP group they are configured from, which means that you can't configure Team membership from Astronomer.
* To remove a Team from your installation, you have to delete it from the Astro Private Cloud UI or the APC API. Deleting an IdP group from your IdP UI doesn't automatically delete the associated Team.
* If a user is added or removed from your original IdP group, that change applies to the related Astronomer Team only after the user logs back in to Astronomer.

<Warning>
  **Most Permissive Role Priority**

  Astronomer user roles function on a *most permissive* policy: If a user has roles defined at both the Workspace and the Team level, then that user will continue to have the most permissive role between the two contexts. This policy has a few implications for implementing Team:

  * If a user's most permissive role comes from a Workspace configuration, there is no way to override/ remove this permission from a Team configuration.
  * If a user's most permissive role comes from a Team configuration, then there is no way to override/ remove this permission from a Workspace configuration.
  * Importing a Team from an IdP has no effect on existing Astronomer user roles. Users continue to have permissions from both contexts, with the most permissive role defining how they interact with a given Workspace or Deployment.

  For example, consider a user with **Workspace Editor** permission in a `Production Workspace` through Astronomer's default authentication for the last year. Your organization begins using Okta as your authentication system for Astronomer and adds this user to a Team with **Workspace Viewer** permissions in `Production Workspace`. Because the user still has **Workspace Editor** permissions from their original account, they continue to have **Workspace Editor** permissions in `Production Workspace`. The only way to remove their Editor permissions is to have a **Workspace Admin** remove them through Workspace settings.
</Warning>

## Prerequisites

To complete this setup, you need:

* A configured identity provider — an OIDC provider integrated through [Integrate an auth system](/docs/astro-private-cloud/v-2-x/integrate-auth-system), or an LDAP directory integrated through [Configure LDAP authentication](/docs/astro-private-cloud/v-2-x/configure-ldap-authentication).
* System Admin permissions for configuring the feature.
* Workspace or Deployment Admin permissions for managing Teams.
* For OIDC providers only: an OAuth authorization code flow. See [Configure a custom OAuth flow](/docs/astro-private-cloud/v-2-x/integrate-auth-system#configure-a-custom-oauth-flow). LDAP doesn't use OAuth; groups are resolved through LDAP queries at each sign-in.
* An IdP group (OIDC) or a directory group (LDAP).

Astronomer also recommends setting up [SCIM](/docs/astro-private-cloud/v-2-x/integrate-auth-system#manage-users-and-teams-with-scim) so that you can manage user groups as Teams directly from your IdP.

## Step 1: Enable APC Teams

Add the following values to your `values.yaml` file. The exact shape depends on whether you provision Teams from an OIDC provider or from an LDAP directory. Save the configuration and push it to your platform as described in [Apply a Platform Config Change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

<Tabs>
  <Tab title="OIDC">
    ```yaml wrap theme={null}
    # Auth configuration.
    auth:
      openidConnect:
        idpGroupsImport:
          enabled: true
        # Optional. Set to assign system-level permissions from IdP groups.
        manageSystemPermissionsViaIdpGroups:
          enabled: true
          systemAdmin: ["<your-system-admin-groups>"]     # Only these groups get SYSTEM_ADMIN
          systemEditor: ["<your-system-editor-groups>"]
          systemViewer: ["<your-system-viewer-groups>"]
        # Optional. Restrict which IdP groups become Teams.
        teamFilterRegex: "^astro-"
    ```
  </Tab>

  <Tab title="LDAP (OpenLDAP)">
    <Note>
      **Astro Private Cloud 2.1**

      This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
    </Note>

    ```yaml wrap theme={null}
    # Auth configuration.
    auth:
      ldap:
        enabled: true
        # The rest of the LDAP connection (host, bindDn, searchBase, etc.) is set up
        # in Configure LDAP authentication. This block only adds the group-import
        # settings on top of a working LDAP configuration.
        groups:
          enabled: true
          reconcileTeams: true                             # Turn resolved LDAP groups into Teams
          searchBase: "ou=groups,dc=corp,dc=example,dc=com"
          searchFilter: "(member={{dn}})"
          # Optional. Restrict which LDAP groups become Teams. LDAP-scoped —
          # separate from auth.openidConnect.teamFilterRegex.
          teamFilterRegex: "^astro-"
          # Optional. Assign system-level permissions from LDAP groups.
          manageSystemPermissions:
            enabled: true
            systemAdmin: ["astro-platform-admins"]
            systemEditor: ["astro-workspace-editors"]
            systemViewer: ["astro-observers"]
    ```
  </Tab>

  <Tab title="LDAP (Active Directory)">
    <Note>
      **Astro Private Cloud 2.1**

      This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
    </Note>

    ```yaml wrap theme={null}
    # Auth configuration.
    auth:
      ldap:
        enabled: true
        # The rest of the LDAP connection (host, bindDn, searchBase, etc.) is set up
        # in Configure LDAP authentication. This block only adds the group-import
        # settings on top of a working LDAP configuration.
        groups:
          enabled: true
          reconcileTeams: true                             # Turn resolved AD groups into Teams
          nestedGroups: "ad"                               # Use LDAP_MATCHING_RULE_IN_CHAIN for transitive membership
          searchBase: "CN=Users,DC=corp,DC=example,DC=com"
          # Optional. Restrict which AD groups become Teams. LDAP-scoped —
          # separate from auth.openidConnect.teamFilterRegex.
          teamFilterRegex: "^Astro-"
          # Optional. Assign system-level permissions from AD groups.
          manageSystemPermissions:
            enabled: true
            systemAdmin: ["Astro Platform Admins"]
            systemEditor: ["Astro Workspace Editors"]
            systemViewer: ["Astro Observers"]
    ```
  </Tab>
</Tabs>

<Note>The `auth.ldap.groups.teamFilterRegex` field is applied only to LDAP-resolved groups. It has no effect on OIDC-provisioned Teams, which use `auth.openidConnect.teamFilterRegex` — the two filters are independent. For the full LDAP-specific behavior see [Configure LDAP authentication](/docs/astro-private-cloud/v-2-x/configure-ldap-authentication#configure-group-resolution).</Note>

<Warning>
  Disabling `manageSystemPermissionsViaIdpGroups.enabled` (OIDC) or `groups.manageSystemPermissions.enabled` (LDAP) stops future synchronization from directory groups but doesn't revoke roles that were previously auto-assigned. Existing system role assignments remain in place until you remove them manually in the Astro Private Cloud UI, in **Settings** > **Teams**.
</Warning>

## Step 2: Add a group claim to your IdP group (OIDC only)

<Note>This step applies only when you provision Teams from an OIDC identity provider. LDAP directories don't use token-based group claims — Houston resolves LDAP groups through a direct LDAP query on every sign-in based on the `auth.ldap.groups.*` configuration you set in Step 1. Skip to Step 3 if you provision Teams from LDAP.</Note>

To add your IdP group to Astronomer as a **Team**, Astronomer needs to be able to recognize the IdP group through a group claim and assign members from the group through tokens.

If you haven't already, add group claims to the IdP groups that you're importing to Astronomer through your configured [third party identity provider](/docs/astro-private-cloud/v-2-x/integrate-auth-system). Refer to your IdP's documentation for information on how to complete this step. For example, for Okta you can refer to [Customize tokens returned from Okta with a Groups claim](https://developer.okta.com/docs/guides/customize-tokens-groups-claim/main).

<Tip>
  By default, Astronomer assumes that the name of your group claim is `groups`. If you named your group claim something other than `groups`, complete the following setup:

  1. In your `values.yaml` file, set `houston.config.auth.openidConnect.<idp-provider>.claimsMapping` to the custom name of your group claim.
  2. Save this configuration and push it to your platform. See [Apply a Platform Config Change](/docs/astro-private-cloud/v-2-x/apply-platform-config).
</Tip>

## Step 3: Add Teams to Workspaces and Deployments

After you complete Step 1, Astro Private Cloud provisions Teams automatically the first time an eligible user signs in:

* **OIDC:** Houston reads the group claim from the user's OIDC token and creates one Team per matching group, tagged with the OIDC provider name (for example, `google`, `microsoft`, `okta`).
* **LDAP:** Houston resolves the user's directory groups according to the `auth.ldap.groups.nestedGroups` mode you set in Step 1 and creates one Team per resolved group, tagged with `provider='ldap'`. See [Configure LDAP authentication](/docs/astro-private-cloud/v-2-x/configure-ldap-authentication#configure-group-resolution) for the resolution modes.

You don't need to pre-create Teams — the sign-in flow provisions them. After a Team appears on the **System Admin** > **Teams** page, Workspace Admins and Deployment Admins can assign it to Workspaces and Deployments through the UI in the same way they assign individual users.

<Note>
  You can use the **System Admin** > **Teams** > **Create Team** dialog to pre-create Teams before any group member signs in, but the **Provider** dropdown in that dialog lists only `local` (when [local Teams](#create-local-teams) are enabled) and OIDC providers. LDAP doesn't appear in the dropdown. For LDAP-provisioned Teams, wait for the first sign-in from a member of the directory group — the Team appears automatically after Houston reconciles it.
</Note>

## Create local teams

If you want to create a Team of Astronomer users, and the team doesn't map to a group in the IdP, you can enable local team creation. This means that if SCIM sync isn't available, you don't have to wait for a user to sign in to Astro Private Cloud for their IdP user groups to sync. You can instead create teams using a locally available source of users.

To enable the feature, add the following configuration to your `values.yaml` file and [apply the change to your installation](/docs/astro-private-cloud/v-2-x/apply-platform-config).

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      # Auth configuration.
      auth:
      # Local database (user/pass) configuration.
        local:
          enabled: true
          teams:
            enabled: true
```

Then, to create a local Team:

1. In the UI, open the **System Admin** menu, then click **Teams**.
2. Click **Create Team**.
3. Give the team a name and a description, then select all the users that you want in the Team.
4. (Optional) Grant the Team a **System Level Role** if the Team needs system-level permissions.

You can now add the local Team to a Workspace or Deployment as you would with an IdP Team or an individual user.

## Disable individual user management

To use Teams as the only user management system on Astro Private Cloud, add the following entry to your `values.yaml` file:

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      userManagement:
        enabled: false
```

Save this configuration and push it to your platform. See [Apply a Platform Config Change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

After you apply the configuration, individual users can't be invited or assigned Workspace or Deployment-level roles in Astro Private Cloud. Users must be invited through a Team by a System Admin, and only Teams can be assigned roles for Workspaces and Deployments. You can still create individual service accounts with Workspace and Deployment permissions.


# Configure authentication and configure an identity provider on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/integrate-auth-system

Integrate your authentication system with Astro Private Cloud.

An auth system determines how users can sign in to Astro Private Cloud. By default, Astro Private Cloud allows users to create an account and authenticate using one of the following methods:

* Google OAuth
* GitHub OAuth
* Local username/password

Integrating an external identity provider (IdP) greatly increases the security of your platform. When you integrate your IdP into Astro Private Cloud:

* Users no longer need to repeatedly sign in and remember credentials for their account.
* You have complete ownership over credential configuration and management on Astro Private Cloud.
* You can enforce multi-factor authentication (MFA) for users.

In addition to the default methods, Astronomer provides the option to integrate any IdP that follows the [Open Id Connect (OIDC)](https://openid.net/connect/) protocol. This includes (but isn't limited to):

* [Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/v2-protocols-oidc)
* [Okta](https://www.okta.com)
* IdPs managed through [Auth0](https://auth0.com/)
* [Amazon Cognito](https://aws.amazon.com/cognito/)

After you integrate your IdP, you can invite users that already have an account on your IdP to Astro Private Cloud. For a more advanced integration, you can configure [SCIM](#manage-users-and-teams-with-scim) so that you can manage users directly from your IdP and import batches of users into Astro Private Cloud as [Teams](/docs/astro-private-cloud/v-2-x/import-idp-groups).

<Info>The following setups assume that you are using the default Astronomer [implicit flow](https://datatracker.ietf.org/doc/html/rfc6749#section-4.2) as your authorization flow. To implement a custom authorization flow, see [Configure a Custom OAuth Flow](/docs/astro-private-cloud/v-2-x/integrate-auth-system#configure-a-custom-oauth-flow).</Info>

<Note>
  If control plane reliability is enabled on your installation with `controlPlaneHA.enabled: true`, register the OAuth redirect and callback URLs against your global domain, `houston.<global-domain-name>`, instead of the per-control-plane base domain shown in the following steps. When you migrate an existing installation to control plane reliability, update the redirect URI on your IdP application to the global domain, or the IdP rejects the callback after sign-in. For the redirect URL behavior under control plane reliability, see [Identity provider authentication and the OAuth redirect URL](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery-reference#identity-provider-authentication-and-the-oauth-redirect-url).
</Note>

<Tabs>
  <Tab title="Microsoft Entra ID">
    ## Step 1: Register an application using `App Registrations` on Azure

    1. In Microsoft Entra ID, click **App registrations** > **New registration**.

    2. Complete the following sections:

       * **Name**: Any
       * **Supported account types**: Accounts in this organizational directory only (Astronomer only - single tenant)
       * **Redirect URIs**:

         * Web / `https://houston.BASEDOMAIN/v1/oauth/redirect/`.
         * Web / `https://houston.BASEDOMAIN/v1/oauth/callback/`.

         Replace `BASEDOMAIN` with your own. For example, if your base domain is `example.com`, your redirect URIs should be `https://houston.example.com/v1/oauth/redirect/` and `https://houston.example.com/v1/oauth/callback/`.

    3. Click **Register**.

    4. Click **Authentication** in the left menu.

    5. In the **Web** area, confirm the redirect URI is correct.

    6. In the **Implicit grant and hybrid flows** area, select **Access tokens** and **ID tokens**.

    7. Click **Save**.

    <Frame>
      <img alt="authentication.png" />
    </Frame>

    ### Step 2: (Optional) Create a client secret

    Complete this setup only if you want to import Microsoft Entra ID groups to Astro Private Cloud as [Teams](/docs/astro-private-cloud/v-2-x/import-idp-groups).

    1. In your Microsoft Entra ID application management left menu, click **Certificates & secrets**.

    2. Click **New client secret**.

    3. Enter a description in the **Description** field and then select an expiry period in the **Expires** list.

    4. Click **Add**.

    5. Copy the values in the **Value** and **Secret ID** columns.

    6. Click **API permissions** in the left menu.

    7. Click **Microsoft Graph** and add the following minimum permissions for Microsoft Graph:

       * `email`
       * `Group.Read.All`
       * `openid`
       * `profile`
       * `User.Read`

         For each of these permissions, select **Grant Admin Consent for Astronomer Data**. Your Microsoft Graph permissions should look similar to the following image:

         <Frame>
           <img alt="Completed permissions page in Azure" />
         </Frame>

    8. Click **Token configuration** in the left menu.

    9. Click **Add groups claim** and select the following options:

       * In the **Select group types to include in Access, ID, and SAML tokens** area, select every option.
       * In **Customize token properties by type** area, expand **ID**, **Access**, and **SAML** and then select **Group ID** for each type.

    10. Click **Add**.

    11. Base64 encode the client secret retrieved from Microsoft with Linux or macOS terminal with the following command:

        ```bash wrap theme={null}
        echo '<oauth-client-secret-value> | base64
        ```

    12. Create a Kubernetes Secret in the `astronomer` Namespace, or whichever Namespace where the APC API is deployed, using the following YAML as an example:

    ```yaml wrap theme={null}
    # Required configuration for all secrets
    kind: Secret
    apiVersion: v1
    metadata:
         name: oauth-client-secret
    type: Opaque
    # Specify a key and value for the data you want to encrypt
    data:
        client_secret:  "<encoded-client-secret-value>"
    ```

    13. Apply the Kubernetes Secret with `kubectl apply -n astronomer -f <filename>.yaml`

    #### Step 3: Enable Microsoft Entra ID in your `values.yaml` file

    Add the following values to your `values.yaml` file:

    ```yaml wrap theme={null}
    astronomer:
      houston:
        # Add/extend astronomer.houston.secret section if using IDP Group Sync
        #secret:
        #  - envName: "AUTH__OPENID_CONNECT__MICROSOFT__CLIENT_SECRET"
        #    secretName: "oauth-client-secret"
        #    secretKey: "client_secret"
        config:
          auth:
            openidConnect:
              flow: "code"
              google:
                enabled: false
              microsoft:
                enabled: true
                clientId: <your-client-id>
                discoveryUrl: https://login.microsoftonline.com/<tenant-id>/v2.0/.well-known/openid-configuration
                baseDomain: login.microsoftonline.com
                authUrlParams:
                  audience: <your-client-id>
            github:
              enabled: false
    ```

    Then, push the configuration change to your platform. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).
  </Tab>

  <Tab title="Okta">
    #### Step 1: Configure Okta

    1. If you haven't already, create an [Okta account](https://www.okta.com/).

    2. In your Okta account, create a new web app for Astronomer.

    3. In Okta, under **General Settings** > **Application**, set `Login redirect URIs` to `https://houston.BASEDOMAIN/v1/oauth/redirect/`, where `BASEDOMAIN` is the domain where you're hosting your APC installation.

    4. Under **Allowed grant types**, select `Implicit (Hybrid)`.

    5. Save the `Client ID` generated for this Okta app for use in the next steps.

    6. Optional. To ensure that an Okta tile appears for Astronomer, set `Initiate Login URI` to `https://houston.BASEDOMAIN/v1/oauth/start?provider=okta`.

    #### Step 2: Integrate Okta with Astro Private Cloud

    Add the following to your `values.yaml` file in your `astronomer` directory:

    ```yaml wrap theme={null}
    astronomer:
      houston:
        # Add/extend astronomer.houston.secret section if using IDP Group Sync
        #secret:
        #  - envName: "AUTH__OPENID_CONNECT__OKTA__CLIENT_SECRET"
        #    secretName: "oauth-client-secret"
        #    secretKey: "client_secret"
        config:
          auth:
            openidConnect:
              okta:
                enabled: true
                clientId: "<okta-client-id>"
                discoveryUrl: "https://<okta-base-domain>/.well-known/openid-configuration"
    ```

    Then, push the configuration change to your platform. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

    <Info>`okta-base-domain` is different from the base domain of your APC installation. See [Okta documentation for finding your domain](https://developer.okta.com/docs/api/getting_started/finding_your_domain/) if you are unsure what this value should be.</Info>
  </Tab>

  <Tab title="Auth0">
    If you manage your identity provider through Auth0, follow these steps to configure the identity provider for Astro.

    #### Step 1: Create an Auth0 tenant domain

    Follow the Auth0 documentation to [create a tenant](https://auth0.com/docs/get-started/auth0-overview/create-tenants). You can use the default domain name or your own unique `tenant-name`. Your full tenant domain looks something like `astronomer.auth0.com`.

    <Info>Your full tenant domain may differ if you've created it outside of the United States.</Info>

    #### Step 2: Create a connection between Auth0 and your identity management provider

    Follow steps in the Auth0 [connection guide](https://auth0.com/docs/identityproviders) for your identity provider to create an integration between your tenant and identity provider.

    #### Step 3: Configure Auth0 application settings

    1. Go to `https://manage.auth0.com/dashboard/us/<tenant-name>/applications`.
    2. Under **Applications**, select **Default App**.
    3. Open the **Connections** tab. You should see your new connection here. Enable your new connection, and disable any connections that you won't be using.
    4. Open the **Settings** tab.
    5. Under **Allowed Callback URLs**, add `https://houston.<your-astronomer-base-domain>/v1/oauth/redirect/`.
    6. Under **Allowed Logout URLs**, add `https://app.<your-astronomer-base-domain>/logout`.
    7. Under **Allowed Origins (CORS)**, add `https://*.<your-astronomer-base-domain>`.
    8. Go to `https://manage.auth0.com/dashboard/us/<tenant-name>/apis`.
    9. Click **+ Create API**.
    10. Under **Name**, enter `astronomer-ee`.
    11. Under **Identifier**, enter `astronomer-ee`.
    12. Leave the value under **Signing Algorithm** as `RS256`.

    #### Step 4: Enable Auth0 in your `values.yaml` file

    Add the following to your `values.yaml` file in your `astronomer` directory:

    ```yaml wrap theme={null}
    astronomer:
      houston:
        # Add/extend astronomer.houston.secret section if using IDP Group Sync
        #secret:
        #  - envName: "AUTH__OPENID_CONNECT__AUTH0__CLIENT_SECRET"
        #    secretName: "oauth-client-secret"
        #    secretKey: "client_secret"
        config:
          auth:
            openidConnect:
              auth0:
                enabled: true
                clientId: "<default-app-client-id>"
                discoveryUrl: https://<tenant-name>.auth0.com
    ```

    Then, push the configuration change to your platform as described in [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

    <Info>You can find your `clientID` value at `https://manage.auth0.com/dashboard/us/<tenant-name>/applications` listed next to 'Default App'.</Info>
  </Tab>

  <Tab title="AWS Cognito">
    #### Step 1: Create a user pool in Cognito

    Start by creating a user pool in Cognito. You can either review the default settings or step through them to customize.

    Make sure that you create an `App client`, which is the OpenID client configuration that Astro Private Cloud uses to authenticate against. You don't need to generate a client secret, as Astro Private Cloud is a public client that uses implicit flow.

    After Auth0 creates the pool and app client, open `App integration` >`App client settings` and configure the following settings:

    * Select an identity provider to use (either the built-in Cognito user pool or a federated identity provider).
    * Set the callback URL parameter to `https://houston.BASEDOMAIN/v1/oauth/redirect/`.
    * Enable `Implicit grant` in `Allowed OAuth Flows`. Leave the other settings disabled.
    * Enable `email`, `openid`, and `profile` in `Allowed OAuth Scopes`.

    Then, switch to the **Domain name** tab and select a unique domain name to use for your hosted Cognito components.

    #### Step 2: Edit your Astro Private Cloud configuration

    Add the following values to your `values.yaml` file in the `astronomer/` directory:

    ```yaml wrap theme={null}
    astronomer:
      houston:
        config:
          auth:
            openidConnect:
              cognito:
                enabled: true
                clientId: <client_id>
                discoveryUrl: https://cognito-idp.<AWS-REGION>.amazonaws.com/<COGNITO-POOL-ID>/.well-known/openid-configuration
                authUrlParams:
                  response_type: token
    ```

    Your Cognito pool ID can be found in the `General settings` tab of the Cognito portal. Your client ID is found in the `App clients` tab.

    After you save your `values.yaml` file with these values, push it to your platform. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).
  </Tab>

  <Tab title="Local auth">
    To let users authenticate to Astro Private Cloud with a local username and password, follow the steps below.

    1. Enable `local auth` in your `values.yaml` file:

       ```yaml wrap theme={null}
       astronomer:
         houston:
           config:
             auth:
               local:
                 enabled: true
       ```

    2. Push the configuration change to your platform. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).
  </Tab>

  <Tab title="General OIDC">
    Astro Private Cloud supports a generic OIDC configuration to accommodate all OIDC-compliant providers. If there are no specific setup instructions for your OIDC provider in this document, you can add the following configuration to your `values.yaml` file.

    For example:

    ```yaml wrap theme={null}
    astronomer:
      houston:
        # Add/extend astronomer.houston.secret section if using IDP Group Sync
        #secret:
        #  - envName: "AUTH__OPENID_CONNECT__CUSTOM__CLIENT_SECRET"
        #    secretName: "custom-oauth-secret"
        #    secretKey: "client_secret"
        config:
          auth:
            openidConnect:
              clockTolerance: 0 # A field that can optionally be set to adjust for clock skew on the server.
              custom:
                enabled: true
                discoveryUrl: <provider-discovery-url> # Note this must be a URL that with an https:// prefix
                clientId: <provider-client-id>
                authUrlParams: # Additional required params set on case-by-case basis
    ```

    Then, push the configuration change to your platform. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).
  </Tab>
</Tabs>

## Run behind an HTTPS proxy

Integrating an external identity provider with Astro Private Cloud requires that the platform's APC API component is able to make outbound HTTPS requests to those identity providers in order to fetch discovery documents, sign keys, and ask for user profile information upon sign-in or sign-up.

If your install is configured *without* a direct connection to the internet you will need to configure an HTTPS proxy server for the APC API.

### Configure an HTTPS proxy server for the APC API

To configure the proxy server used we need to set the `GLOBAL_AGENT_HTTPS_PROXY` Environment Variable for the APC API deployment.

To do so, add the following to the APC API section of the `values.yaml` file in your `astronomer` directory:

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      auth:
        openidConnect:
          custom:
            enabled: true
            displayName: My OAuth Provider
            clientId: ...
            discoveryUrl: ...
    env:
      - name: GLOBAL_AGENT_HTTPS_PROXY
        value: http://my-proxy:3129
```

Then, push the configuration change to your platform as described in [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

## Configure a custom OAuth flow

You can set up a custom OAuth authorization flow as an alternative to Astro Private Cloud's default [implicit flow](https://datatracker.ietf.org/doc/html/rfc6749#section-4.2). You can customize Astronomer's existing Okta, Google, and GitHub OAuth flows or import an entirely custom OAuth flow.

<Danger>This setup must be completed only during a scheduled maintenance window. There should be no active users on your installation until the setup has been finalized.</Danger>

### Step 1: Configure your authorization flow on Astro Private Cloud

To use a custom OAuth authorization code flow:

1. In your `values.yaml` file, set the `astronomer.houston.config.auth.openidConnect.flow` value to `"code"`:

   ```yaml wrap theme={null}
   astronomer:
     houston:
       config:
         auth:
           # Local database (user/pass) configuration.
           local:
             enabled: true
           openidConnect:
             # Valid values are "code" and "implicit"
             flow: "code"
   ```

2. Configure the section of your `values.yaml` file specific to your identity provider with each of the following values:

   * `enabled`: Set this value to `true` under the section for your own identity provider.
   * `clientId`: Your [Client ID and Client secret](https://www.oauth.com/oauth2-servers/client-registration/client-id-secret/)
   * `discoveryURL`: Your base [Discovery URL](https://www.oauth.com/oauth2-servers/indieauth/discovery/)
   * `authUrlParams`: Additional [parameters](https://developer.okta.com/docs/guides/add-an-external-idp/saml2/main/#use-the-authorize-url-to-simulate-the-authorization-flow) to append to your discovery URL. At a minimum, you must configure `audience`. Refer to your identity provider's documentation for information on how to find this value (Auth0 maintains this information in their [glossary](https://auth0.com/docs/glossary), for example).

     For example, a custom configuration of Okta might look like the following.

     ```yaml wrap theme={null}
     astronomer:
      houston:
        secret:
          - envName: "AUTH__OPENID_CONNECT__OKTA__CLIENT_SECRET"
            secretName: "oauth-client-secret"
            secretKey: "client_secret"
        config:
          auth:
            okta:
              enabled: true
              clientId: ffhsdf78f734h2fsd
              discoveryUrl: "https://<your-idp-url>.okta.com/oauth2/default/.well-known/openid-configuration"
              authUrlParams:
                audience: "GYHWEYHTHR443fFEW"
     ```

   1. Push your configuration changes to your platform as described in [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

### Step 2: Configure your identity provider

To finalize your configuration, configure the following key values in your identity provider's settings:

* **Grant Code**: Set to "Code" or "Auth Code" depending on your identity provider.
* **Sign-in Redirect URI**: Set to `https://houston.<BASE_DOMAIN>/v1/oauth/callback/`. Be sure to include the trailing `/`.

### Step 3: Confirm your installation

When you complete this setup, you should be able to see the differences in sign-in flow when signing in at `<BASE_DOMAIN>.astronomer.io`:

<Frame>
  <img alt="Custom sign-in button on the Astronomer sign-in screen" />
</Frame>

You can see the name you configured in `AUTH__OPENID_CONNECT__CUSTOM__DISPLAY_NAME` when authenticating using the Astro CLI.

## Manage users and Teams with SCIM

Astro Private Cloud supports integration with the open standard System for Cross-Domain Identity Management (SCIM). Using the SCIM protocol with Astro Private Cloud allows you to automatically provision and deprovision users and Teams based on templates that define permission and accesses. It also centralizes user management so that you can configure Astro Private Cloud user permissions directly from your identity provider (IdP).

<Info>SCIM works because the IdP pushes updates about users and teams to Astro Private Cloud. This means your Astro Private Cloud platform must be connected to the internet and exposed on port 443 to receive those updates. If you run Astro Private Cloud without exposing it to the internet, there might be solutions with [Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity/app-provisioning/on-premises-scim-provisioning) and [Okta](https://help.okta.com/en-us/content/topics/provisioning/opp/opp-create-scim-connectors.htm) for routing SCIM traffic, depending on your combination of cloud provider and IdP. Contact [Astronomer support](https://support.astronomer.io) for more information.</Info>

<Tabs>
  <Tab title="Okta">
    1. In Okta Admin dashboard, go to **Applications** > **Applications**.

    2. Click **Browse App catalog**

    3. Search for `SCIM 2.0`, then select the option that includes **Basic Auth**. The configuration page for the SCIM integration appears.

    4. Complete the **General Settings** page, then click **Next**.

    5. Complete the **Sign-On Options** page and click **Done**.

    6. Return to the **Applications** menu and search for the integration you just created. Click the integration to open its settings.

    7. Click **Provisioning**, then click **Configure API integration**.

    8. Tick the **Enable API integration** checkbox, then configure the following values:

       * **SCIM connector base URL**: `https://astro-apc-host/v1/scim/v2/okta`
       * **Authentication mode**: Basic Auth
         * Username: `<your-provisioning-account-username>`
         * Password: `<your-provisioning-account-password>`

    9. Click **General**, then click **Edit**. Give your application a name and configure any other required general settings.

    10. Go to **Push Groups** page and create a rule for Group Push. See [Group Push](https://help.okta.com/en-us/Content/Topics/users-groups-profiles/usgp-about-group-push.htm).

    11. On the **Assignments** tab, ensure that the right users and groups in your org are assigned to the app integration. See [Use the Assign Users to App action](https://help.okta.com/en-us/Content/Topics/Apps/apps-assign-applications.htm?cshid=ext_Apps_Apps_Page-assign).

    12. Follow the steps in [Store and encrypt identity provider secrets](#step-2-configure-your-identity-provider) to store your provisioning account credentials as a Kubernetes secret. Make sure to Base64 encode the credentials, as it is a requirement for Kubernetes secrets. This can be done using the Linux or macOS terminal with  `echo '<your-provisioning-account-username>:<your-provisioning-account-password> | base64`. Your secret configuration should look similar to the following:

        ```yaml wrap theme={null}
        # Required configuration for all secrets
        kind: Secret
        apiVersion: v1
        metadata:
             name: okta-provisioning-secret
        type: Opaque
        # Specify a key and value for the data you want to encrypt
        data:
            okta_provisioning_account_secret: "<encoded-oauth-client-secret-value>"
        ```

    13. Add the following lines to your `values.yaml` file:

        ```yaml wrap theme={null}
        astronomer:
          houston:
            secret:
              - envName: "SCIM_AUTH_CODE_OKTA"
                secretName: "okta-provisioning-secret"
                secretKey: "okta_provisioning_account_secret"
        ```

    14. Push the configuration change. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

    See [Add SCIM provisioning to app integrations](https://help.okta.com/en-us/Content/Topics/Apps/Apps_App_Integration_Wizard_SCIM.htm?cshid=ext_Apps_App_Integration_Wizard-scim) for more information about configuring SCIM within Okta.
  </Tab>

  <Tab title="Microsoft Entra ID">
    1. Generate a random string to use as an authentication secret. See [random.org](https://www.random.org/strings/) for accessible randomization tools.

    2. Follow the steps in [Store and encrypt identity provider secrets](#step-2-configure-your-identity-provider) to store your string as a Kubernetes secret. Your secret configuration should look similar to the following:

       ```yaml wrap theme={null}
       # Required configuration for all secrets
       kind: Secret
       apiVersion: v1
       metadata:
            name: scim-provisioning-secret
       type: Opaque
       # Specify a key and value for the data you want to encrypt
       data:
           scim_provisioning_secret: "<base64-encoded-client-secret-value>"
       ```

    3. In your `values.yaml` file, add the following configuration:

       ```yaml wrap theme={null}
       astronomer:
         houston:
           secret:
             - envName: "SCIM_AUTH_CODE_MICROSOFT"
               secretName: "scim-provisioning-secret"
               secretKey: "scim_provisioning_secret"
       ```

    4. Push the configuration change. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

    5. Sign in to the [Microsoft Entra ID portal](https://aad.portal.azure.com/).

    6. In the left menu, select **Enterprise applications**, then click **New application** > **Create your own application**.

    7. Enter a name for your application and select **Integrate any other application you don't find in the gallery**.

    8. Click **Create** to create an app object. Microsoft Entra ID opens the application management menu for your new application.

    9. In the application management menu for your new application, go to **Manage** > **Provisioning** and click **Get Started**.

    10. Click **Provisioning Mode** > **Automatic**.

    11. In the **Tenant URL** field, enter `https://houston.BASEDOMAIN/v1/scim/v2/microsoft`. This is the Astro Private Cloud SCIM endpoint URL.

    12. Paste the `scimAuthCode` that you generated in Step 1 into the **Secret Token** field.

    13. Click **Test connection** in the Microsoft Entra ID application management menu to confirm your connection to the SCIM endpoint.

    14. Create mappings for your Astronomer users and roles. See [Tutorial - Customize user provisioning attribute-mappings for SaaS applications in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity/app-provisioning/customize-application-attributes).

    15. Click **Manage** > **Provisioning** > **Settings**.

    16. In the **Scope** setting list, select **Sync only assigned users and groups**.

    17. Click the **Provisioning status** toggle to turn provisioning status on.

    18. Click **Save**.
  </Tab>
</Tabs>


# Integrate IAM roles on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/integrate-iam

Append IAM roles to an Airflow Deployment on Astro Private Cloud.

On Astro Private Cloud, IAM roles can be appended to the webserver, scheduler and worker pods within any individual Airflow Deployment on the platform.

IAM roles on [AWS](https://aws.amazon.com/iam/faqs/) and other platforms are often used to manage the level of access a specific user (or object, or group of users) has to some resource (or set of resources). The resource in question could be an S3 bucket or Secret Backend, both of which are commonly used in tandem with Airflow and Astronomer and can now be configured to be accessible only to a subset of Kubernetes pods within your wider Astronomer cluster.

## Implementation considerations

Consider the following when you integrate IAM roles:

* All pods within your Airflow Deployment assume the IAM role.
* There is currently no way to use more than one IAM role per Deployment.
* If you’d like your IAM role to apply to more than one Deployment, you must annotate each Deployment.
* You must use the Astro CLI to pass IAM role annotations.
* Only Workspace Admins can pass IAM role annotations.
* Once a Deployment is created or updated with an IAM role, the annotation can't be deleted.
* When using XCom or secrets backends that store values in resources governed by your cloud-provider's IAM solution, grant [the set of service-accounts used by that namespace](#sas-that-use-backends) access to the associated cloud-resources.

## Prerequisites

* [The Astro CLI](/docs/cli/v1.43/install-cli)
* Admin access on an Astronomer Workspace
* Direct access to your Kubernetes cluster (for example, permission to run `kubectl describe po`)
* A compatible version of Kubernetes as described in Astronomer's [Version compatibility reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference)

## AWS

Before you can integrate IAM with an Airflow Deployment on Astronomer, you'll need to do the following within AWS:

* Create an IAM OIDC Identity Provider
* Create an IAM Policy
* Create an IAM role
* Create a Trust Relationship

### Step 1: Create an IAM OIDC identity provider

1. Retrieve your EKS cluster with the following AWS CLI command:

   ```bash wrap theme={null}
   aws eks list-clusters
   ```

   The output of this command should look something like this:

   ```json wrap theme={null}
   {
     "clusters": ["<your-cluster>"]
   }
   ```

2. Retrieve and make note of your cluster's OIDC issuer URL with the following AWS CLI command:

   ```bash wrap theme={null}
   aws eks describe-cluster --name <your-cluster> --query "cluster.identity.oidc.issuer" --output text
   ```

   The output of this command should be a URL with the format `https://oidc.eks.[region].amazonaws.com/id/[id]`.

3. Open the [IAM console](https://console.aws.amazon.com/iam/).

4. In the navigation pane, click **Identity Providers** > **Create Provider**.

5. For **Provider Type**, click **Choose a provider type** > **OpenID Connect**.

6. For **Provider URL**, use the OIDC issuer URL for your cluster.

7. For **Audience**, use `sts.amazonaws.com`.

8. Verify that the provider information is correct, and then click **Add provider** to create your identity provider.

For additional information, refer to [Enable IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html).

### Step 2: Create an IAM policy

1. Open the [IAM console](https://console.aws.amazon.com/iam/).

2. In the navigation panel, click **Policies** > **Create Policy**.

3. Open the **JSON** tab.

4. In the Policy Document field, specify the permissions you'd like to apply (or restrict) to the resource in question (for example, read / write access to an AWS S3 bucket). You can also use the visual editor to construct your own policy.

   The following example will grant your IAM role read/write permissions to an S3 bucket named `astronomer-bucket`:

   ```json wrap theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "s3:ListBucket",
           "s3:GetBucketLocation",
           "s3:ListBucketMultipartUploads"
         ],
         "Resource": "arn:aws:s3:::astronomer-bucket"
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:PutObject",
           "s3:GetObject",
           "s3:DeleteObject",
           "s3:ListMultipartUploadParts",
           "s3:AbortMultipartUpload"
         ],
         "Resource": "arn:aws:s3:::astronomer-bucket/*"
       }
     ]
   }
   ```

5. Review and create your policy.

### Step 3: Create an IAM role

1. Open the [IAM console](https://console.aws.amazon.com/iam/).
2. In the navigation panel, go to **Roles** > **Create Role**.
3. In the **Select trusted entity** section, choose **AWS service** and **EC2**. Choose **Next**.
4. In the **Add permissions** section, select your policy created in the previous section. Choose **Next.**
5. In the **Name, review, and create** section, enter a name for your role and click **Create role**.

For additional information, refer to [Create service account IAM Policy and Role](https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html).

### Step 4: Create a trust relationship

To create a trust relationship between your IAM role and OIDC identity provider:

1. Open the [IAM console](https://console.aws.amazon.com/iam/).
2. In the navigation panel, choose **Roles** and open your role created in the previous section.
3. Select the **Trust relationships** tab and choose **Edit trust policy**.
4. Create a trust relationship between your IAM role and OIDC identity provider with the following format:

   ```json wrap theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "ec2.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       },
       {
         "Effect": "Allow",
         "Principal": {
           "Federated": "arn:aws:iam::AWS_ACCOUNT_ID:oidc-provider/OIDC_PROVIDER"
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
           "StringLike": {
             "OIDC_PROVIDER:sub": "system:serviceaccount:SERVICE_ACCOUNT_NAMESPACE:SERVICE_ACCOUNT_NAME"
           }
         }
       }
     ]
   }
   ```

   Example:

   ```json wrap theme={null}
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "ec2.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       },
       {
         "Effect": "Allow",
         "Principal": {
           "Federated": "arn:aws:iam::<your-iam-id>:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/EXAMPLEA829F4B2854D8DAE63782CE90"
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
           "StringLike": {
             "oidc.eks.us-west-2.amazonaws.com/id/EXAMPLEA829F4B2854D8DAE63782CE90:sub": "system:serviceaccount:astronomer-*:*"
           }
         }
       }
     ]
   }
   ```

For additional information, refer to [IAM role Configuration](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#iam-role-configuration).

### Step 5: Integrate your IAM role with Astro Private Cloud

In order to apply your IAM role to any Airflow Deployment on Astro Private Cloud, you'll need to explicitly pass an annotation key to the platform. To do so:

1. In the APC UI, go to your **Clusters** page and select your cluster.

2. In the cluster details, click **Edit** in the **Deployment Configuration** section and add the following override to the **Configuration Override** field:

   ```yaml wrap theme={null}
   deploymentImagesRegistry:
     serviceAccountAnnotationKey: eks.amazonaws.com/role-arn
   ```

   For details on using the UI for configuration, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).

3. Save and apply your changes in the UI.

### Step 6: Create or update an Airflow Deployment with an attached IAM role

1. To create a new Airflow Deployment with your IAM role attached, run the following Astro CLI command:

   ```sh wrap theme={null}
   astro deployment create <deployment-id> --executor=celery --cloud-role=arn:aws:iam::<your-iam-id>:role/<your-role>
   ```

   Alternatively, to update an existing Airflow Deployment with your IAM role attached, run the following:

   ```sh wrap theme={null}
   astro deployment update <deployment-id> --cloud-role=arn:aws:iam::<your-iam-id>:role/<your-role>
   ```

2. Confirm the role was passed successfully to all webserver, scheduler and worker pods within your Airflow Deployment by running the following command:

   ```bash wrap theme={null}
   kubectl describe po <pod-name> -n <airflow-namespace>
   ```

   You should see the following in your output:

   ```yaml wrap theme={null}
   AWS_ROLE_ARN: arn:aws:iam::<your-iam-id>:role/<your-role>
   AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
   ```

<Note>If using Airflow `1.10.5`, you'll need to add `boto3 >=1.9` and `botocore >= 1.12` to your `requirements.txt` file.</Note>

## GCP

[Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) is a secure and manageable way to access Google Cloud services from applications running on GKE. This guide walks through the necessary steps for integrating IAM roles with Airflow Deployments running on GKE using Workload Identity.

### Step 1: Enable workload identity on your GKE cluster

1. Create a new cluster with Workload Identity enabled by running the following command:

   ```bash wrap theme={null}
   gcloud container clusters create <cluster-name> \
     --workload-pool=<project-id>.svc.id.goog
   ```

   Alternatively, run the following to enable Workload Identity on an existing cluster:

   ```bash wrap theme={null}
   gcloud container clusters update <cluster-name> \
     --workload-pool=<project-id>.svc.id.goog
   ```

2. Configure your node pool to use Workload Identity by running the following command:

   ```bash wrap theme={null}
   gcloud container node-pools update <nodepool-name> \
     --cluster=<cluster-name> \
     --workload-metadata=GKE_METADATA
   ```

### Step 2: Create a GCP service account

To create a GCP service account, run the following command:

```bash wrap theme={null}
gcloud iam service-accounts create <gsa-name>
```

### Step 3: Configure Astro Private Cloud

In the APC UI, go to your **Clusters** page and select your cluster.

In the cluster details, click **Edit** in the **Deployment Configuration** section and add the following override to the **Configuration Override** field:

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        deploymentImagesRegistry:
          serviceAccountAnnotationKey: iam.gke.io/gcp-service-account
```

For details on using the UI for configuration, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).

### Step 4: Create an Airflow Deployment

1. Create an Airflow Deployment with your GCP service account attached by running the following command:

   ```bash wrap theme={null}
   astro deployment create <deployment-name> --executor=celery --cloud-role=<gsa-name>@<project-id>.iam.gserviceaccount.com
   ```

2. Note the name of the worker, triggerer, scheduler, webserver, cleanup, and migrate-database-job service accounts that appear when you run the following command:

   ```bash wrap theme={null}
   kubectl get sa -n <your-airflow-namespace>
   ```

3. Create an IAM policy binding your Google and GKE service accounts by running the following command for both the worker and scheduler GKE service accounts you noted:

   ```bash wrap theme={null}
   gcloud iam service-accounts add-iam-policy-binding \
   --role roles/iam.workloadIdentityUser \
   --member "serviceAccount:<project-id>.svc.id.goog[<your-airflow-namespace>/<airflow-worker-service-account-name>]" \
   <gsa-name>@<project-id>.iam.gserviceaccount.com
   ```

### Step 5: Confirm Workload Identity is working

1. Create an interactive session by running the following command:

   ```bash wrap theme={null}
     kubectl run -it \
     --image google/cloud-sdk:slim \
     --overrides='{ "spec": { "serviceAccount": "<airflow-worker-service-account-name>" } }' \
     --namespace <your-airflow-namespace> \
     workload-identity-test
   ```

2. In the interactive session, confirm you're able to authenticate successfully via Workload Identity by running the following command:

   ```bash wrap theme={null}
   gcloud auth list
   ```

   If Workload Identity is working, you should see a list of credentialed accounts related to your GCP service account.

<a />

## Grant access to Airflow components using XCom and secrets backends

Astro Private Cloud creates a set of service accounts for each Airflow instance it manages. The following roles can require access to Airflow XCom backends or secrets backends to function:

| Component                             | Rationale                                                                                                                                                                                                                                                             |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `<release name>-cleanup`              | Accesses task instances and other data that might include serialized references to values stored in secrets backends or XCom backends.                                                                                                                                |
| `<release name>-dag-processor`        | The standalone Dag Processor is a dedicated Pod that runs separately from the scheduler. It continuously parses and processes Dag files, detects any changes, and updates the Airflow metadata database — allowing the scheduler Pod to only work on task scheduling. |
| `<release name>-migrate-database-job` | Analyzes serialized Dag models that might include serialized references to values stored in secrets backends or XCom backends.                                                                                                                                        |
| `<release name>-scheduler`            | Regularly interprets your Dag code by the scheduler to determine which tasks are part of Dags and might incorporate references to values stored in secret storage or XCom.                                                                                            |
| `<release name>-triggerer`            | Regularly interprets your Dag code by the triggerer at task run-time and might incorporate references to values stored in secrets storage or XCom.                                                                                                                    |
| `<release name>-webserver`            | Provides a mechanism for you to view XCom entries and to view and set secrets.                                                                                                                                                                                        |
| `<release name>-worker`               | Regularly interprets your Dag code by the worker at task runtime and might incorporate references to values stored in secret storage or XCom.                                                                                                                         |


# Configure Kerberos authentication for Airflow databases
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/kerberos-database-setup

Configure Astro Private Cloud to use Kerberos authentication for Airflow Deployment databases.

Astro Private Cloud supports Kerberos authentication for Airflow deployment databases, allowing you to connect to Kerberized PostgreSQL databases.

## Overview

Kerberos is an authentication protocol that uses tickets to allow secure authentication in network environments. In enterprise environments with strict security requirements, databases are often configured to use Kerberos authentication instead of traditional username and password authentication.

With Kerberos database support in Astro Private Cloud, you can:

* Connect Airflow deployments to Kerberized PostgreSQL databases
* Maintain compliance with enterprise security policies that require Kerberos authentication
* Use existing Kerberos infrastructure for database authentication
* Support both unified and control plane/data plane deployment modes

## How it works

Astro Private Cloud uses PgBouncer as a proxy between Airflow components and the Kerberized database. When you enable Kerberos for a deployment:

1. You provide labels and environment variables for PgBouncer Pods via the APC API when creating or updating a deployment.
2. Your Kerberos credential injection mechanism (such as a mutation webhook) uses these labels to inject Kerberos credentials (keytabs or credential refresh sidecars) into the PgBouncer Pods.
3. PgBouncer authenticates to the PostgreSQL database using GSSAPI (Kerberos protocol).
4. Airflow components connect to PgBouncer using standard authentication, and PgBouncer proxies the connection to the Kerberized database.

### Architecture

Astro Private Cloud supports two deployment modes with different Kerberos configurations:

#### Unified mode

In unified mode, the control plane and data plane are installed in the same Kubernetes cluster.

<Frame>
  <img alt="Kerberos architecture in Unified mode" />
</Frame>

#### Control plane/data plane mode

In control plane/data plane mode, you can configure separate Kerberos authentication for the control plane database and data plane (Airflow) databases:

<Frame>
  <img alt="Kerberos architecture in CP/DP mode" />
</Frame>

<Note>
  In control plane/data plane mode, you can use separate Active Directory instances and Kerberos realms for the control plane and data plane. This allows for greater security isolation between control plane and Airflow deployment databases.
</Note>

## Prerequisites

Before configuring Kerberos authentication, ensure you have:

* Astro Private Cloud
* A Kerberized PostgreSQL database (PostgreSQL 18 has known issues)
* Kerberos infrastructure:
  * Active Directory or MIT Kerberos KDC (Key Distribution Center)
  * Network connectivity between your Kubernetes cluster and the KDC
  * For CP/DP mode: Optionally, separate Active Directory instances for control plane and data plane
* A mechanism to inject Kerberos credentials into PgBouncer Pods. See [Kerberos credential injection](#kerberos-credential-injection).
* A Kerberos user principal created in your Active Directory
* APC API access to create deployments

## Responsibility model

Kerberos database authentication in Astro Private Cloud follows a shared responsibility model:

### Astronomer responsibilities

* Providing PgBouncer images with Kerberos (GSSAPI) support
* Supporting labels and environment variables on PgBouncer Pods via the APC API
* Maintaining deployment stability during updates

### Customer responsibilities

* Setting up and managing Kerberos infrastructure (Active Directory, KDC, etc.)
* Creating and managing Kerberos user principals and keytabs
* Implementing a mechanism to inject Kerberos credentials into PgBouncer Pods (such as a mutation webhook)
* Configuring appropriate labels and environment variables via the APC API to trigger credential injection
* Creating Kerberos users in the PostgreSQL database with appropriate permissions
* Pre-creating Airflow databases for deployments
* Managing Kerberos ticket lifecycle (renewal, rotation)

## Scope and limitations

The following are supported in Astro Private Cloud with Kerberos authentication:

* **Executor**: Kubernetes executor only
* **Deployment Type**: Image-based deployments only
* **Database**: PostgreSQL with Kerberos authentication
* **Deployment Modes**: Both unified and control plane/data plane modes

Future releases may expand support to other executors, deployment types, and databases.

## Kerberos credential injection

You are responsible for implementing a mechanism to inject Kerberos credentials into PgBouncer Pods. One common approach is using a Kubernetes mutation webhook.

### Mutation webhook approach (recommended)

A mutation webhook can automatically inject Kerberos credentials when PgBouncer Pods are created. The webhook typically:

1. Watches for Pod creation requests with specific labels that you configure via the APC API
2. Injects a sidecar container that manages Kerberos ticket renewal
3. Mounts Kerberos configuration files (`krb5.conf`) and keytabs as volumes
4. Configures environment variables for Kerberos authentication

When creating a deployment, you'll specify labels in the `pgbouncerConfig` section that trigger your webhook to inject the necessary credentials.

### Alternative approaches

Other methods for credential injection include:

* Init containers that fetch credentials from a secret management system
* Direct volume mounts of Kerberos keytabs from Kubernetes secrets
* Service mesh sidecars

Choose the approach that best fits your organization's security requirements and infrastructure.

For implementation guidance, contact Astronomer support or your Astronomer representative.

## Step 1: Configure cluster settings

Before creating Kerberos-enabled deployments, you must enable manual connection strings in your cluster configuration.

### For unified mode

1. Open your control plane `values.yaml` file.

2. Add the following configuration:

```yaml wrap theme={null}
global:
  airflow:
    images:
      pgbouncer:
        repository: "quay.io/astronomer/ap-pgbouncer-krb"
        tag: "1.25.0-2"
```

3. Push the configuration change. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

### For control plane/data plane mode

1. Enable manual connection strings on the data plane cluster by adding the following to its **Configuration Override**. Cluster overrides apply to `deployments.*` values, so don't include the `deployments.` prefix. See [Update data plane cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster) for instructions on updating cluster-specific settings.

   ```yaml wrap theme={null}
   databaseManagement:
     manualConnectionStrings:
       enabled: true
   ```

2. Add the following to your control plane `values.yaml` to set the PgBouncer image used by Deployments on the data plane:

   ```yaml wrap theme={null}
   global:
     airflow:
       images:
         pgbouncer:
           repository: "quay.io/astronomer/ap-pgbouncer-krb"
           tag: "1.25.0-2"
   ```

   Push the configuration change. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

## Step 2: Create the Kerberos database user

You must create a Kerberos user in your PostgreSQL database with the appropriate permissions.

<Note>
  In control plane/data plane mode, create separate Kerberos users for:

  * The control plane database (if using a Kerberized control plane database)
  * Each data plane's Airflow databases
</Note>

1. Connect to your PostgreSQL database using a superuser account.

2. Create the Kerberos user. The username must be in the format `<username>@<REALM>`:

```sql wrap theme={null}
CREATE USER "astro_user@APC.ASTRONOMER.IO" WITH LOGIN;
```

3. Grant the necessary permissions:

```sql wrap theme={null}
-- For AWS RDS with Kerberos
GRANT rds_ad TO "astro_user@APC.ASTRONOMER.IO";

-- Database and schema permissions
GRANT ALL PRIVILEGES ON DATABASE postgres TO "astro_user@APC.ASTRONOMER.IO";
GRANT ALL PRIVILEGES ON SCHEMA public TO "astro_user@APC.ASTRONOMER.IO";

-- Allow database creation
ALTER ROLE "astro_user@APC.ASTRONOMER.IO" CREATEDB;
```

Replace `astro_user` with your Kerberos username and `APC.ASTRONOMER.IO` with your Kerberos realm.

## Step 3: (Optional) Configure PgBouncer in the control plane

If you need to use a Kerberized database for the control plane (the APC API's database), you must configure PgBouncer in the control plane namespace.

### Create PgBouncer configuration

1. Create a `pgbouncer.ini` file with the following contents. Replace the placeholders with your actual values:

```ini expandable wrap theme={null}
[databases]
* = host=<rds-hostname> port=5432 user=<kerberos_user>@<kerberos_realm>

[pgbouncer]
pool_mode = transaction
listen_addr = 0.0.0.0
listen_port = 6543
idle_transaction_timeout = 60
transaction_timeout = 60
autodb_idle_timeout = 60
admin_users = postgres
client_idle_timeout = 60
server_idle_timeout = 60
track_extra_parameters = search_path
stats_users = postgres

# Authentication settings
auth_type = md5
auth_file = /etc/pgbouncer/users.txt
ignore_startup_parameters = extra_float_digits

# Kerberos settings
server_gssauth_negotiate = allow
server_krb_spn = postgres/<rds-hostname>@<kerberos_realm>

max_client_conn = 200
verbose = 2
log_disconnections = 0
log_connections = 0

# Strongly recommended for RDS
server_tls_sslmode = prefer
```

2. Generate a password hash for the PgBouncer `users.txt` file:

```bash wrap theme={null}
echo -n "md5"; echo -n "<password><user>" | md5sum | awk '{print $1}'
```

3. Create a `users.txt` file with the password hashes:

```text wrap theme={null}
"<kerberos_user>" "<output_from_above_command>"
"postgres" "<output_from_above_command>"
```

<Note>
  The password authentication in `users.txt` is used for the APC API to connect to PgBouncer. PgBouncer then authenticates to the PostgreSQL database using Kerberos (GSSAPI).
</Note>

4. Create the Kubernetes secret:

```bash wrap theme={null}
kubectl -n astronomer create secret generic astronomer-pgbouncer-config \
  --from-file=pgbouncer.ini=./pgbouncer.ini \
  --from-file=users.txt=./users.txt \
  --dry-run=client -o yaml | kubectl apply -f -
```

### Update control plane configuration

1. Open your control plane `values.yaml` file.

2. Add the following PgBouncer configuration:

```yaml wrap theme={null}
pgbouncer:
  enabled: true
  repository: "quay.io/astronomer/ap-pgbouncer-krb"
  tag: "1.25.0-2"
  securityContext:
    runAsGroup: 65534
    runAsNonRoot: true
    runAsUser: 65534

global:
  pgbouncer:
    enabled: true
    extraEnv: []
    extraLabels: []
    gssSupport: true
    secretName: astronomer-pgbouncer-config
    securityContext:
      runAsGroup: 65534
      runAsUser: 65534
    servicePort: "6543"
    username: postgres
```

3. Update the Astronomer bootstrap secret to point to the PgBouncer service:

```text wrap theme={null}
postgres://<username>:<password>@astronomer-pgbouncer.astronomer.svc.cluster.local:6543?pgbouncer=true&connection_limit=100&pool_timeout=60&prisma_connection_limit=100
```

The username and password are the user and password from the `users.txt` file above.

<Warning>
  The query parameters `pgbouncer=true&connection_limit=100&pool_timeout=60&prisma_connection_limit=100` are critical for Prisma to work correctly with PgBouncer. Without these parameters, the control plane may experience connection issues.
</Warning>

4. Upgrade the control plane installation. You must perform the upgrade in two steps:

   **Step 1**: First, upgrade with the `--no-hooks` flag. This installs PgBouncer in the control plane without running database migration jobs:

```bash wrap theme={null}
helm upgrade astronomer astronomer/astronomer \
  --namespace astronomer \
  -f <values_file>.yaml \
  --version <version> \
  --no-hooks
```

The `<version>` is the APC version you want to upgrade to (for example, 1.1.0).

**Step 2**: After the upgrade completes and PgBouncer is running, run the upgrade again without the `--no-hooks` flag. This runs the database migration jobs:

```bash wrap theme={null}
helm upgrade astronomer astronomer/astronomer \
  --namespace astronomer \
  -f <values_file>.yaml \
  --version <version>
```

The `<version>` is the APC version you want to upgrade to (for example, 1.1.0).

<Note>
  The two-step upgrade process is necessary because:

  1. The first upgrade installs PgBouncer, which is required for database connectivity when using a Kerberized database.
  2. The second upgrade runs database migration hooks that depend on PgBouncer being available.
</Note>

## Step 4: Create an Airflow deployment database

Before creating a Kerberos-enabled deployment, you must manually create the Airflow database.

Follow these steps to create the database:

1. Create a PostgreSQL client Pod for database operations:

```yaml wrap theme={null}
apiVersion: v1
kind: Pod
metadata:
  labels:
    release: astronomer
    tier: astronomer
  name: postgres-debug-client
  namespace: astronomer
spec:
  containers:
  - command:
    - /bin/bash
    - -c
    - sleep infinity
    image: postgres:16
    imagePullPolicy: Always
    name: psql-client
```

2. Apply the Pod:

```bash wrap theme={null}
kubectl apply -f postgres-debug-client.yaml
```

3. Exec into the Pod and connect to the database:

```bash wrap theme={null}
kubectl -n astronomer exec -it postgres-debug-client -- bash
```

4. Connect to your database. If using PgBouncer in the control plane:

```bash wrap theme={null}
psql "postgres://<username>:<password>@astronomer-pgbouncer.astronomer.svc.cluster.local:6543"
```

<Note>
  The username and password in the connection string should match the credentials configured in your `users.txt` file from [Step 3](#step-3-optional-configure-pgbouncer-in-the-control-plane).
</Note>

Or connect directly to your Kerberized database (ensure you have appropriate credentials).

5. Create the Airflow database:

```sql wrap theme={null}
CREATE DATABASE <deployment_name>_airflow OWNER "<kerberos_user>@<kerberos_realm>";
```

For example:

```sql wrap theme={null}
CREATE DATABASE mydeployment_airflow OWNER "astro_user@APC.ASTRONOMER.IO";
```

## Step 5: Create a Kerberos-enabled deployment

Kerberos-enabled deployments must be created using the APC API. They can't be created from the Astro UI.

### Use the `upsertDeployment` mutation

1. Compose your mutation payload. The following example shows the required fields for a Kerberos-enabled deployment:

```json expandable wrap theme={null}
{
  "cloudRole": "",
  "dagDeployment": {
    "nfsLocation": "",
    "type": "image"
  },
  "dagProcessor": {
    "replicas": 1
  },
  "kerberosEnabled": true,
  "deploymentUuid": "",
  "deployRevisionDescription": "",
  "description": "",
  "dockerconfigjson": null,
  "environmentVariables": [],
  "executor": "KubernetesExecutor",
  "image": "",
  "label": "my-kerberos-deployment",
  "metadataConnection": "",
  "skipAirflowDatabaseProvisioning": true,
  "metadataConnectionJson": {
    "protocol": "postgresql",
    "user": "astro_user@APC.ASTRONOMER.IO",
    "pass": "no-pass",
    "host": "<airflow-db-host>",
    "port": 5432,
    "db": "mydeployment_airflow"
  },
  "mode": "helm",
  "namespace": "",
  "properties": {
    "extra_capacity": {
      "cpu": 0,
      "memory": 0
    }
  },
  "releaseName": "mydeployment",
  "resultBackendConnection": "",
  "resultBackendConnectionJson": {
    "protocol": "postgresql",
    "user": "astro_user@APC.ASTRONOMER.IO",
    "pass": "no-pass",
    "host": "<airflow-db-host>",
    "port": 5432,
    "db": "mydeployment_airflow"
  },
  "rollbackEnabled": false,
  "runtimeVersion": "12.0.0",
  "scheduler": {
    "replicas": 1
  },
  "triggerer": {},
  "webserver": {},
  "workers": {},
  "pgbouncerConfig": {
    "labels": {
      "key": "value"
    },
    "env": [
      {"name": "KERBEROS_USER", "value": "astro_user@APC.ASTRONOMER.IO"},
      {"name": "KERBEROS_PASSWORD", "value": "YourKerberosPassword"}
    ],
    "extraIniResultBackend": "user=astro_user@APC.ASTRONOMER.IO",
    "extraIniMetadata": "user=astro_user@APC.ASTRONOMER.IO",
    "extraIni": "server_gssauth_negotiate = allow\\nserver_krb_spn = postgres/<airflow-db-host>@APC.ASTRONOMER.IO",
    "sslmode": "prefer"
  },
  "workspaceLabel": "my-workspace",
  "workspaceUuid": "<workspace-uuid>"
}
```

### Important configuration fields

* `kerberosEnabled`: Must be set to `true`. This enables the APC API to perform Kerberos-specific validation.
* `skipAirflowDatabaseProvisioning`: Must be set to `true` because you manually create the Airflow database.
* `metadataConnectionJson` and `resultBackendConnectionJson`:
  * `user`: Must be in the format `<username>@<REALM>`
  * `pass`: Can be any value (for example, `"no-pass"`) since PgBouncer uses Kerberos for database authentication
  * `host`: The hostname of your Kerberized PostgreSQL database
  * `db`: The database name you created (for example, `mydeployment_airflow`)
* `pgbouncerConfig`:
  * `labels`: Custom labels for the PgBouncer Pod. Use these labels to trigger your Kerberos credential injection mechanism (for example, `"krb-inject": "enabled"` or `"component": "pgbouncer"`).
  * `env`: Environment variables for the PgBouncer Pod. Your credential injection mechanism can use these to configure Kerberos authentication (for example, `KERBEROS_USER`, `KERBEROS_PASSWORD`).
  * `extraIniMetadata` and `extraIniResultBackend`: Must specify the Kerberos user.
  * `extraIni`: Must include Kerberos/GSS settings:
    * `server_gssauth_negotiate = allow`
    * `server_krb_spn = postgres/<airflow-db-host>@<kerberos_realm>`
  * `sslmode`: Set to `prefer` for RDS or other TLS-enabled databases

<Note>
  In control plane/data plane mode, ensure the Airflow database host, Kerberos realm, and Kerberos user you specify are for the data plane, not the control plane.
</Note>

2. Execute the mutation using the [APC API](/docs/astro-private-cloud/v-2-x/houston-api).

## Step 6: Verify Kerberos authentication

After creating your deployment, verify that Kerberos authentication is working correctly.

### Verify deployment creation

1. List the Pods in the deployment namespace:

```bash wrap theme={null}
kubectl -n <airflow-namespace> get pods
```

2. Verify that all Airflow Pods are running:

```bash wrap theme={null}
kubectl -n <airflow-namespace> get pods -l component=scheduler
kubectl -n <airflow-namespace> get pods -l component=webserver
kubectl -n <airflow-namespace> get pods -l component=pgbouncer
```

### Verify database connectivity

1. Check the PgBouncer logs for successful connections:

```bash wrap theme={null}
kubectl -n <airflow-namespace> logs <pgbouncer-pod-name>
```

Look for log entries showing connections using the Kerberos principal.

2. Verify that the Airflow UI loads successfully:

   a. Sign in to the Astro UI.

   b. Navigate to your deployment.

   c. Click **Airflow UI**.

If the Airflow UI loads without errors, PgBouncer is successfully connecting to the database using Kerberos authentication.

## Troubleshooting

### PgBouncer Pod fails to start

If the PgBouncer Pod fails to start, check the following:

* Verify that your Kerberos credential injection mechanism is configured correctly.

* Check the Pod events for error messages:

  ```bash wrap theme={null}
  kubectl -n <airflow-namespace> describe pod <pgbouncer-pod-name>
  ```

* Verify that the labels in `pgbouncerConfig` match what your credential injection mechanism expects.

### Kerberos authentication failures

If authentication fails, verify:

* The Kerberos user exists in your Active Directory and PostgreSQL database.
* The `server_krb_spn` in the PgBouncer configuration matches your database hostname and realm.
* Network connectivity between the Kubernetes cluster and the KDC.
* Your Kerberos credential injection mechanism is working correctly.
* In CP/DP mode, you're using the correct Kerberos realm and credentials for the data plane (not the control plane).

### Airflow UI fails to load

If the Airflow UI fails to load:

* Check the webserver and scheduler logs for database connection errors.
* Verify that the database was created with the correct owner.
* Ensure the `metadataConnectionJson` and `resultBackendConnectionJson` are correctly configured.
* Verify the `pgbouncerConfig` settings, especially the `extraIni` configuration.

## Additional resources

* [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config)
* [Update data plane cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster)
* [APC API](/docs/astro-private-cloud/v-2-x/houston-api)
* [APC API example queries](/docs/astro-private-cloud/v-2-x/houston-api-example-queries)
* [Manage permissions](/docs/astro-private-cloud/v-2-x/manage-permissions)


# Use kubectl to administer Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/kubectl

Deploy Astro Private Cloud Airflow instances with kubectl and Helm.

## kubectl and Helm

[kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) and [Helm](https://helm.sh/docs/using_helm/) are the two primary ways DevOps users will interact/administer the Astronomer Platform.

### Setup

Both these tools are needed to deploy Astro Private Cloud onto a Kubernetes cluster. We also recommend using [`kubectx`](https://github.com/ahmetb/kubectx) to simplify commands (the rest of this guide will use `kubectx`).

### Base namespace and release

The initial `helm install` command to deploy Astro Private Cloud requires a namespace to deploy the base platform pods into.

```text wrap theme={null}
helm install -f dataplane-values.yaml . -n datarouter
```

This deploys a randomly named [release](https://helm.sh/docs/glossary/#release) of our Helm charts into the `datarouter` [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/).

All [pods](https://kubernetes.io/docs/concepts/workloads/pods/) can be listed in the namespace with kubectl.

```bash expandable wrap theme={null}
root@orbiter: datarouter
Context "gke_astronomer-dev-190903_us-east4-a_astronomer-dev-lybjumoigv" modified.
Active namespace is "datarouter".

root@orbiter: kubectl get pods
NAME                                                     READY   STATUS    RESTARTS   AGE
cautious-seal-postgresql-0                                1/1     Running     0          2d
singed-chimp-commander-789767877f-jdqrf                     1/1     Running     0          15h
singed-chimp-commander-789767877f-zvsfj                     1/1     Running     0          15h
singed-chimp-config-syncer-29317458-4pbnw                   0/1     Completed   0          27h
singed-chimp-config-syncer-29318898-mmjf5                   0/1     Completed   0          3h27m
singed-chimp-dp-nginx-77c89d7676-gbbdh                      1/1     Running     0          43h
singed-chimp-dp-nginx-77c89d7676-zsl8z                      1/1     Running     0          43h
singed-chimp-elasticsearch-client-6b6c56f976-rppqx          1/1     Running     0          43h
singed-chimp-elasticsearch-client-6b6c56f976-wjgpw          1/1     Running     0          43h
singed-chimp-elasticsearch-curator-29317020-cglck           0/1     Completed   0          34h
singed-chimp-elasticsearch-curator-29318460-nqwfm           0/1     Completed   0          10h
singed-chimp-elasticsearch-data-0                           1/1     Running     0          43h
singed-chimp-elasticsearch-data-1                           1/1     Running     0          43h
singed-chimp-elasticsearch-exporter-798976cc85-wjl7z        1/1     Running     0          43h
singed-chimp-elasticsearch-master-0                         1/1     Running     0          43h
singed-chimp-elasticsearch-master-1                         1/1     Running     0          43h
singed-chimp-elasticsearch-master-2                         1/1     Running     0          43h
singed-chimp-elasticsearch-nginx-674b5d74b-vnl62            1/1     Running     0          43h
singed-chimp-external-es-proxy-544456d54b-5fw8j             1/1     Running     0          43h
singed-chimp-kube-state-85f6887d6f-lfm7x                    1/1     Running     0          43h
singed-chimp-nginx-default-backend-75fd84bc4b-rtkqm         1/1     Running     0          43h
singed-chimp-nginx-default-backend-75fd84bc4b-tngw2         1/1     Running     0          43h
singed-chimp-prometheus-0                                   3/3     Running     0          43h
singed-chimp-prometheus-federation-auth-967f5d9f9-dbgq7     1/1     Running     0          15h
singed-chimp-prometheus-federation-auth-967f5d9f9-f6mcj     1/1     Running     0          15h
singed-chimp-prometheus-postgres-exporter-5bd4964ff-jcvph   1/1     Running     0          43h
singed-chimp-prometheus-postgres-exporter-5bd4964ff-mcr7z   1/1     Running     0          43h
singed-chimp-registry-745f59599-bhz42                       1/1     Running     0          43h
```

Run `helm ls -A` to list all releases:

```bash wrap theme={null}
root@orbiter: helm ls -A
NAME                             	REVISION	UPDATED                 	STATUS  	CHART                   	NAMESPACE
cautious-seal                    	1       	Mon May  6 14:37:34 2019	DEPLOYED	postgresql-0.18.1       	datarouter
singed-chimp                     	3       	Thu May  9 12:08:36 2019	DEPLOYED	astronomer-0.8.2        	datarouter
```

There is a release for Astronomer (`singed-chimp`) and Postgres (`cautious-seal`) in the `datarouter` namespace.

### Create new Airflow deployments

If you navigate to the Astro Private Cloud UI and create a new deployment, it creates a new Helm release in a new namespace.

```bash wrap theme={null}
root@orbiter helm ls -A
NAME                             	REVISION	UPDATED                 	STATUS  	CHART                   	NAMESPACE
accurate-bolide-9914             	2       	Thu May  9 12:06:06 2019	DEPLOYED	airflow-0.8.2           	datarouter-accurate-bolide-9914
cautious-seal                    	1       	Mon May  6 14:37:34 2019	DEPLOYED	postgresql-0.18.1       	datarouter
singed-chimp                     	3       	Thu May  9 12:08:36 2019	DEPLOYED	astronomer-0.8.2        	datarouter
```

The `accurate-bolide-9914` Helm release now lives in the a namespace generated by Astronomer named `$basenamespace-release_name` (datarouter-accurate-bolide-9914) and lives on the URL `$release-name-airflow.<BASEDOMAIN>`.

Switching into the `datarouter-accurate-bolide-9914` namespace reveals the Airflow pods.

```bash wrap theme={null}
root@orbiter: kubectl get namespaces
NAME                                     STATUS   AGE
datarouter                               Active   2d
datarouter-accurate-bolide-9914          Active   1h
default                                  Active   222d

root@orbiter: kubens datarouter-accurate-bolide-9914
root@orbiter: kubectl get pods

NAME                                                        READY   STATUS    RESTARTS   AGE
accurate-bolide-9914-pgbouncer-57b46d67bb-b4tpw             2/2     Running   0          1h
accurate-bolide-9914-scheduler-7d4f5596b5-r457d             2/2     Running   0          1h
accurate-bolide-9914-statsd-84d76854fb-wz45c                1/1     Running   0          1h
accurate-bolide-9914-webserver-86bcbc48b-5xcwr              1/1     Running   0          1h
accurate-bolide-9914-worker-0                               1/1     Running   0          1m
accurate-bolide-9914-redis-0                                1/1     Running   0          1m
accurate-bolide-9914-flower-84f7d67fbd-q5sd7                1/1     Running   0          1m
```

Since this deployment is running the Celery executor with 1 worker, there are pods for Redis and Flower.

### Manage Pods

Airflow logs can be fetched directly from the underlying pods:

```bash wrap theme={null}
root@orbiter: kubectl logs accurate-bolide-9914-scheduler-7d4f5596b5-r457d

Waiting for host: accurate-bolide-9914-pgbouncer 6543
Initializing airflow database...
[2019-05-09 17:53:22,865] {settings.py:182} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=18
[2019-05-09 17:53:24,345] {__init__.py:51} INFO - Using executor KubernetesExecutor
DB: postgresql://accurate_bolide_9914_airflow**:*@accurate-bolide-9914-pgbouncer:6543/accurate-bolide-9914-metadata
[2019-05-09 17:53:24,969] {db.py:350} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Done.
[2019-05-09 17:53:28,159] {settings.py:182} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=8
[2019-05-09 17:53:29,663] {__init__.py:51} INFO - Using executor KubernetesExecutor
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2019-05-09 17:53:30,338] {jobs.py:1500} INFO - Starting the scheduler
[2019-05-09 17:53:30,339] {jobs.py:1508} INFO - Running execute loop for -1 seconds
[2019-05-09 17:53:30,340] {jobs.py:1509} INFO - Processing each file at most -1 times
[2019-05-09 17:53:30,340] {jobs.py:1512} INFO - Searching for files in /usr/local/airflow/dags
[2019-05-09 17:53:30,341] {jobs.py:1514} INFO - There are 2 files in /usr/local/airflow/dags
[2019-05-09 17:53:30,343] {kubernetes_executor.py:691} INFO - Start Kubernetes executor
```

A full description of a pods status, resource request, and other data can be found with the `describe` command

```bash wrap theme={null}
root@orbiter: kubectl describe po/accurate-bolide-9914-scheduler-7d4f5596b5-g72pg
Name:               accurate-bolide-9914-scheduler-7d4f5596b5-g72pg
Namespace:          datarouter-accurate-bolide-9914
Priority:           0
PriorityClassName:  <none>
Node:               gke-astronomer-dev-l-astronomer-dev-n-1c6fc689-s8pg/10.150.0.77
Start Time:         Thu, 09 May 2019 13:53:20 -0400
Labels:             component=scheduler
                    platform=singed-chimp
                    pod-template-hash=3809115261
                    release=accurate-bolide-9914
                    tier=airflow
                    workspace=cjvgu6b2d00110b785tfxyqp0

```

Pods can also be deleted as a way to restart any Airflow component.

```text wrap theme={null}
root@orbiter: kubectl delete po/accurate-bolide-9914-scheduler-7d4f5596b5-r457d
```

This will delete that copy of the pod and spin up a new one. All pods in an Airflow deployment are meant to be stateless, so deleting one and letting it recreate shouldn't cause any harm.

### Other resources

For additional `kubectl` commands, check out this [Kubernetes Cheat Sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/).


# Manage a control plane reliability group
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/manage-control-plane-disaster-recovery

Day-2 operations for a control plane reliability group: read the admin UI, cordon and decommission control planes, roll a chart-version upgrade, and perform a cross-region failover.

This guide covers day-2 operations for a control plane reliability group: how to read the admin UI, take a control plane in or out of service, roll a chart-version upgrade across your control planes, and perform a cross-region failover.

It assumes you have already stood up a control plane reliability group by following [Configure control plane reliability](/docs/astro-private-cloud/v-2-x/configure-control-plane-disaster-recovery) — two or more control planes sharing one database and one global domain, with weighted, health-checked DNS in front of them. For the terms used here (`<global-domain-name>`, the per-control-plane (per-CP) admin hostname `<cpNN-domain>`, `/controlplane/status`, and the shared JSON Web Token (JWT) keypair), see [Control plane reliability](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery).

<Note>
  Cross-region failover in this guide means moving the *active region* for the control planes. It's unrelated to [data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover), which moves Apache Airflow Deployments between data plane clusters and has its own components and runbook.
</Note>

## The admin UI

When you enable control plane reliability, two admin tabs appear in the left sidebar, along with a small indicator showing which control plane you're connected to. All three are hidden on single-control-plane installations.

* *Control Planes tab*: visible to system admins only. Lists every registered control plane. This is where you cordon, decommission, register, and deregister.
* *Regions tab*: visible to any viewer, but its actions are admin-only. Lists regions. This is where you activate a region.
* *Connected-control-plane indicator*: a sidebar chip, admin-only, that shows the region and status of the control plane your browser session is currently connected to.

<Frame>
  <img alt="APC admin UI showing the Control Planes tab, the Regions, Control Planes, and System entries in the left sidebar, and the connected-control-plane indicator at the bottom of the sidebar." />
</Frame>

### Control plane list

Each row on the **Control Planes** tab shows the following columns:

| Column             | Description                                                                                                                                                                                                                            |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Health**         | A colored dot. See [The health dot](#the-health-dot).                                                                                                                                                                                  |
| **Name**           | The human-readable name you registered the control plane under, for example `cp01`.                                                                                                                                                    |
| **Region**         | The region this control plane is attached to.                                                                                                                                                                                          |
| **Status**         | The control plane's effective status: `ACTIVE`, `STANDBY`, `CORDONED`, or `DECOMMISSIONED`. See [Control plane status reference](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery-reference#control-plane-status-reference). |
| **Chart Version**  | The platform chart version this control plane is running.                                                                                                                                                                              |
| **Ingress URL**    | The control plane's own APC API ingress URL.                                                                                                                                                                                           |
| **Registered**     | When the control plane was first registered.                                                                                                                                                                                           |
| **Last Heartbeat** | When the APC API last recorded a liveness tick from this control plane.                                                                                                                                                                |

The available actions are **Register Control Plane**, and per row **Edit** (a dialog with **Name**, **Region**, and a **Status** dropdown) and **Deregister** (which removes the control plane from the registry after a typed confirmation).

<Note>
  There is no dedicated cordon or decommission action. You change a control plane's state through **Edit** > **Status**. The dropdown offers only legal transitions. See [Manage control plane status](#manage-control-plane-status).
</Note>

### The health dot

The health dot is a three-state readiness indicator, separate from the **Status** badge. It's computed in the browser from the control plane's last heartbeat and its chart version relative to the group-wide maximum. It isn't the `/controlplane/status` routing signal.

| Dot    | Meaning                                                                                                                                                           |
| ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Green  | The heartbeat is fresh and the control plane is at the group-wide maximum chart version, so it's ready to serve.                                                  |
| Orange | The heartbeat is fresh, but the chart version is behind the group-wide maximum, so the control plane is running but not eligible to serve at the current version. |
| Red    | The heartbeat is stale, so the control plane's APC API process is likely down or unreachable.                                                                     |

Staleness is checked first, because a control plane that isn't running has a meaningless version. Hover the dot for the exact reason.

<Warning>
  A red dot doesn't, by itself, pull a control plane out of DNS rotation. Routing eligibility is driven entirely by `/controlplane/status`. The heartbeat and the dot are observability signals only. That said, if a control plane's APC API is truly down, its `/controlplane/status` also fails at the connection level, so DNS drains it anyway.
</Warning>

### Heartbeat

Every APC API replica writes a periodic liveness tick. The default cadence is 30 seconds, and a heartbeat is considered stale after about 90 seconds (three missed ticks). This threshold is server-authoritative.

The **Control Planes** list refetches on navigation and after any control plane or region change. It doesn't poll on a fixed timer, so the health dot's freshness advances as the page re-renders rather than ticking live.

### Region list

The **Regions** tab lists **Name**, **Cloud Provider**, **Status** (an **Active** or **Inactive** badge), and **Created**. The available actions are **Create Region**, and per row **Edit**, **Delete**, and **Activate**. **Activate** appears only on regions that aren't currently active.

<Frame>
  <img alt="Regions tab in the APC UI, listing regions with Name, Cloud Provider, Status, and Created columns, and per-row Activate, Edit, and Delete actions." />
</Frame>

<Note>
  There is deliberately no deactivate action. Activating one region atomically deactivates all others, so switching the active region is a single action. See [Cross-region failover](#cross-region-failover).
</Note>

## Manage control plane status

A control plane has one of three stored statuses, plus a derived fourth status that you only ever see and never set:

| Status           | Set by you | Serving                      | Mutations                    | Heartbeat | Meaning                                                                                                                                                                         |
| ---------------- | ---------- | ---------------------------- | ---------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ACTIVE`         | Yes        | Yes, if the region is active | Accepted                     | Yes       | Normal operation.                                                                                                                                                               |
| `STANDBY`        | Derived    | No, routed away              | Rejected `REGION_INACTIVE`   | Yes       | An active control plane whose region is inactive — a hot standby waiting for its region to be activated. Computed from status and region state, not stored.                     |
| `CORDONED`       | Yes        | No, routed away              | Rejected `CP_CORDONED`       | Yes       | Temporarily out of service and reversible. Used for maintenance, node work, or as a prelude to decommissioning. Still heartbeats, so it stays in the tracked fleet.             |
| `DECOMMISSIONED` | Yes        | No, routed away              | Rejected `CP_DECOMMISSIONED` | No        | Removed from the fleet. No heartbeat, and no participation in version or activation math. Recoverable to `ACTIVE` while its Pods are still running; otherwise treat it as gone. |

### Change a control plane's status

From the APC UI, open the **Control Planes** tab, select **Edit** on the control plane, pick the new value from the **Status** dropdown, and save.

<Frame>
  <img alt="Edit Control Plane dialog in the APC UI with the Status dropdown open, showing the CORDONED option available from ACTIVE." />
</Frame>

Through the API, use the `updateControlPlane` mutation, which requires system-admin access. Cordon and decommission aren't separate mutations — they're status values on this one call:

```graphql theme={null}
mutation {
  updateControlPlane(cpId: "<cp-uuid>", status: CORDONED) {
    id name status
  }
}
```

Legal transitions are enforced server-side, and the UI dropdown offers only legal ones:

```text theme={null}
ACTIVE          → CORDONED
CORDONED        → ACTIVE | DECOMMISSIONED
DECOMMISSIONED  → ACTIVE
```

The forward path is `ACTIVE` → `CORDONED` → `DECOMMISSIONED`, and both non-active states are reversible to `ACTIVE`. You can't jump straight from `ACTIVE` to `DECOMMISSIONED`; you must cordon first. An illegal transition is rejected with `INVALID_CP_STATUS_TRANSITION`.

### What happens when you cordon

Setting `CORDONED` writes one database field, and two independent consumers act on it:

* *The health endpoint fails*. `/controlplane/status` on that control plane starts returning `503`. Your global DNS load balancer, which health-checks that path, drains the control plane from rotation and sends customers to the remaining healthy control planes. Control plane reliability doesn't touch DNS itself; it only flips the health signal.
* *Mutations are rejected*. The cordoned control plane still serves read queries, but every state-changing mutation is rejected with `CP_CORDONED`, except the exempt admin mutations, so you can always un-cordon it. A cordoned control plane is also excluded from the chart-version-maximum computation, which is what makes it safe to use as a maintenance state during a rolling upgrade.

<Frame>
  <img alt="Control Planes list in the APC UI showing a control plane with a CORDONED status badge." />
</Frame>

Decommissioning behaves the same way for serving and mutations, rejecting mutations with `CP_DECOMMISSIONED`, but it's the terminal state: the control plane stops heartbeating and is excluded from all version and activation math, so a stale or dead control plane can never hold back the rest of the fleet.

<Frame>
  <img alt="Edit Control Plane dialog in the APC UI with the Status dropdown open, showing the DECOMMISSIONED option available from CORDONED." />
</Frame>

<Frame>
  <img alt="Control Planes list in the APC UI showing a control plane with a DECOMMISSIONED status badge." />
</Frame>

### Recover or remove a control plane

* *Un-cordon or recover*: **Edit** > **Status** > `ACTIVE`. This works from both `CORDONED` and `DECOMMISSIONED` as long as the control plane's Pods are still running. These are exempt mutations that bypass the status gate.

<Frame>
  <img alt="Edit Control Plane dialog in the APC UI with the Status dropdown open, showing the ACTIVE option available to recover a decommissioned control plane." />
</Frame>

* *Deregister*: the **Deregister** action, or `deregisterControlPlane(cpId)`, removes the control plane's registry row entirely. Use this once a control plane is truly gone. After you deregister, that control plane's `/controlplane/status` reports `NOT_REGISTERED` (`503`) until it's registered again.

The typical workflow is to cordon, do the maintenance, then un-cordon. To retire a control plane permanently: cordon, confirm it has drained, decommission, tear down the infrastructure, then deregister.

## Upgrade the chart version

Only control planes running the highest registered chart version within their own region are eligible to serve traffic. The APC API computes the maximum chart version across the active control planes in a region and marks any control plane below it as outdated (`503`, and mutations rejected with `CP_VERSION_OUTDATED`). There is no separately stored expected version — it's computed dynamically as the peer maximum, using semantic-version comparison. Cordoned and decommissioned control planes are excluded from this maximum, so they can't drag the bar up or down.

A control plane's reported version refreshes automatically right after `helm upgrade` and again on each heartbeat tick.

### Upgrade procedure

Upgrade one control plane at a time, or several in parallel:

1. Run `helm upgrade` on the first control plane. Its chart version becomes the new maximum in its region.
2. Every not-yet-upgraded control plane in that region immediately becomes ineligible: its `/controlplane/status` returns `503` and it stops accepting mutations until it catches up.
3. During this window, customer traffic flows only to the already-upgraded control planes.
4. Upgrade the next control plane. It rejoins the eligible set. Repeat until every control plane in the region is upgraded.
5. Verify that each control plane's health dot is green and `/controlplane/status` returns `200`.

### Availability during an upgrade

Within an active region, moving one control plane ahead makes the lagging control planes temporarily unhealthy, so the region runs at reduced capacity — potentially a single serving control plane — until the others catch up. To minimize that window, run the `helm upgrade` commands in parallel across the region's control planes rather than strictly one after another.

The version-maximum check is scoped to each control plane's own region, so upgrading control planes in a standby (inactive) region has no effect on the active region's serving. A common pattern is to fully upgrade the standby region first with zero customer impact, activate it, then upgrade the now-standby former-active region.

<Warning>
  Roll back by running `helm upgrade` to the target older version, never `helm rollback`. `helm rollback` reverts the APC API Deployment but doesn't re-render the `astronomer-houston-config` ConfigMap, which is a keep-policy pre-upgrade-hook resource. This leaves the APC API reporting a stale chart version, so the **Control Planes** UI shows the wrong version and the eligibility gate mis-ranks the control plane. Running `helm upgrade` to the older version re-fires the pre-upgrade hook and re-renders the config.

  Rollback is symmetric with upgrade: downgrading a single control plane makes it the lowest version and therefore ineligible, so to actually roll back you must downgrade all control planes in the region.
</Warning>

## Cross-region failover

Cross-region failover moves the active region from one set of control planes to another. It's always admin-driven; there is no automatic region failover.

<Warning>
  Control plane reliability doesn't fail over your database. Your managed-database tooling (for example, Amazon RDS or Google Cloud SQL) promotes a database replica in the destination region and repoints the connection, and you must complete this before you activate the destination region. Region activation is only the control plane half of the cutover — it flips which region serves; it doesn't move data. Activating a region before its database is the writable primary is a known failure mode.
</Warning>

Control plane reliability needs only two outcomes from your database failover, and your database tooling is responsible for delivering both:

1. The database is failed over, so the destination region's database is the writable primary.
2. APC always sees a single, stable endpoint — the hostname the APC API connects through (in `astronomer-bootstrap`) resolves to that primary at all times.

How you achieve those outcomes — replica promotion, DNS or CNAME switching, a single auto-switching endpoint, or another mechanism — is your choice. Any specific database steps in this section are suggestions from Astronomer's internal testing, not requirements. The one ordering constraint is yours to meet: complete the database failover before you call `activateRegion`.

### Planned region cutover

Use this graceful procedure for an intentional move — maintenance, migration, or cost — rather than an outage. Run every admin action against the per-CP admin hostnames (`cpNN.<parent-domain>`), not the global URL. During the cutover window the global URL is intentionally unavailable.

<Steps>
  <Step title="Cordon every control plane in the source region">
    Set **Status** to `CORDONED` on each source control plane, from the UI or with `updateControlPlane(status: CORDONED)`. This blocks subsequent mutations on the source control planes, which quiesces writes so that database replication can catch up before the promotion in the next step.

    Within about 30 seconds — the health-check TTL — the global load balancer stops serving every cordoned control plane, so APC becomes unavailable through the global DNS name and is reachable only through the per-CP admin URLs, which is what admins should use to orchestrate the failover. This has no effect on the data plane or running Airflow Deployments; it only takes the APC UI and API offline on the global URL for the cutover window.

    <Note>
      There is currently no in-flight-work drain. Any messages still queued in the source region's NATS at deactivation stop being consumed.
    </Note>
  </Step>

  <Step title="Fail over the database to the destination region">
    Control plane reliability doesn't do this; your managed-database tooling handles it, and the mechanism is your choice. Before you activate the region, both required outcomes must hold: the destination database is the writable primary, and APC sees a single, stable endpoint resolving to it. The following are suggestions from Astronomer's internal testing, not a required procedure:

    * With the source control planes cordoned, writes are quiesced, so for a planned failover you can wait for replication lag to reach zero before promoting, to avoid data loss.
    * If you promote a read replica, wait until promotion is fully complete — for example, the instance reports `available` and is no longer attached to a replication source. Some engines briefly report `available` while still attached.
    * If you keep APC's endpoint stable through a DNS or CNAME record, a low TTL (for example, 60 seconds) makes the switch propagate quickly, and the `astronomer-bootstrap` secret stays unchanged because only the record moves. If you use a single auto-switching endpoint, there's nothing to repoint.
    * You may need to recycle the APC API's database connections on the destination control planes (roll the APC API Kubernetes Deployment) if their Pods were pinned to the old endpoint, so that they reconnect to the new primary.
    * Plan failback as its own cutover. After a promotion, cross-region database replication is typically broken, and re-establishing it in the reverse direction is a separate rebuild.
  </Step>

  <Step title="Activate the destination region">
    Call `activateRegion` against a control plane in the destination region, using that control plane's per-CP admin hostname. The global load balancer won't route to a control plane whose region is still inactive. Within about 30 seconds every control plane's `/controlplane/status` reflects the new region-active state, and the global load balancer begins serving the destination control planes. See [Activate a region](#activate-a-region).
  </Step>

  <Step title="Restart the ingress controller on the source control planes">
    Scale the nginx ingress controller Deployment on the source (now-inactive) control planes to `0` and back up. New browser connections already route to the newly active region through the global load balancer, but existing keep-alive connections may still be held open against a source control plane and keep hitting the now-inactive region. Bouncing the ingress controller breaks those connections and forces browsers to reconnect, re-resolve the global DNS name, and land on the correct active control plane.
  </Step>
</Steps>

### Outage cutover

If the source region's control planes are down or unreachable, you can't cordon them and there's nothing to quiesce, so skip the cordon step. Fail over the database, accepting data loss up to the last replicated transaction (skip the zero-lag wait), then activate the destination region. Because the source control planes are already unreachable, existing browser connections to them are already broken, so the ingress-controller restart is unnecessary. Expect that some in-flight customer operations may need to be retried once traffic lands in the destination region.

### Activate a region

From the APC UI, open the **Regions** tab, select **Activate** on the destination region, and type the region name to confirm. The confirmation dialog warns that activation shifts customer traffic to this region, deactivates the currently active one, and can't be undone automatically.

<Frame>
  <img alt="Activate region dialog in the APC UI, warning that activation shifts customer traffic and deactivates the currently active region, with a field to type the region name to confirm." />
</Frame>

After activation, the **Control Planes** tab reflects the new active region: the control plane in the newly active region shows `ACTIVE`, and the control plane in the now-inactive region shows `STANDBY`.

<Frame>
  <img alt="Control Planes list in the APC UI after a region activation, with the control plane in the newly active region showing ACTIVE and the control plane in the now-inactive region showing STANDBY." />
</Frame>

Through the API, use the `activateRegion` mutation:

```graphql theme={null}
mutation {
  activateRegion(regionId: "<region-id>", force: false) {
    id name active
  }
}
```

* It runs in a single transaction that deactivates every currently active region and activates the target, so the "exactly one region active" invariant holds with no window of zero or two active regions.
* It's an exempt mutation, so you can run it from a control plane whose region is currently inactive — otherwise failover would be impossible. Run it through the destination control plane's per-CP admin hostname.
* An inactive region's control planes report `REGION_INACTIVE` (`503`, and mutations rejected), but they stay alive, heartbeating and upgradeable, serving no customer traffic until their region is activated. This is the `STANDBY` state.

### The activation version gate

To prevent a silent downgrade, `activateRegion` refuses to activate a region unless at least one control plane in it is at the group-wide maximum chart version (the maximum across all active control planes in every region, with cordoned and decommissioned control planes excluded). Otherwise it fails with `TARGET_REGION_NOT_UPGRADED`, whose payload lists each target control plane's version against the group-wide maximum, so you can see exactly what to upgrade. In the UI this surfaces as a **Cannot activate region** dialog listing the lagging control planes.

<Frame>
  <img alt="Cannot activate region dialog in the APC UI, explaining that no control plane in the target region is at the fleet-wide maximum chart version, with Understood and Force activate anyway actions." />
</Frame>

<Warning>
  `force: true` bypasses the version gate. Use it only for an emergency rollback where a knowing downgrade is acceptable and you're sure the schema is backward-compatible. Forced activations are written to the audit log. In the UI this is gated behind a second confirmation.
</Warning>

<Frame>
  <img alt="Force activation downgrade-risk dialog in the APC UI, warning that activating a region whose control planes are behind the fleet-wide maximum downgrades serving, with Back and Force activate actions." />
</Frame>

If you activate a region that has no control plane — for example, an empty standby region — every control plane in the group reports `STANDBY` and no control plane serves customer traffic until a control plane in the active region is available.

<Frame>
  <img alt="Control Planes list in the APC UI after activating a region with no control plane, showing every control plane with a STANDBY status so none is serving traffic." />
</Frame>

### Why the customer session survives a cutover

Mid-session users aren't signed out across a cutover because the session cookie is scoped to `.<global-domain-name>`, the JWT signing key is shared across all control planes (so a token minted on the old region is trusted on the new one), and the database is the same instance, already failed over by your database tooling before the cutover.

<Warning>
  Before any failover, re-check that the global DNS records have health checks against `/controlplane/status`. Without them, DNS keeps sending customers to control planes in the now-inactive region. Those control planes correctly reject mutations with `REGION_INACTIVE`, but customers see errors instead of an uninterrupted cutover.
</Warning>

## Related documentation

* [Control plane reliability](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery)
* [Configure control plane reliability](/docs/astro-private-cloud/v-2-x/configure-control-plane-disaster-recovery)
* [Control plane reliability reference](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery-reference)
* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)


# Manage and observe per-deployment migration
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/manage-per-deployment-migration

Day-2 operations for per-deployment migration on Astro Private Cloud: trigger a migration, watch it run, understand what it blocks, and recover a failed move.

This guide covers day-2 operations for per-deployment migration: reading the new UI surfaces, triggering a migration, watching one run, understanding what gets blocked while it runs, and recovering when one fails. Per-deployment migration is available in Astro Private Cloud (APC) 2.1 and later.

It assumes you have already completed the setup in [Configure per-deployment migration](/docs/astro-private-cloud/v-2-x/configure-per-deployment-migration): data plane failover enabled on both clusters, real regions created, and every participating cluster assigned to one. For the lookup tables this guide links to — mission and flight states, skip reasons, and the GraphQL surface — see [Per-deployment migration reference](/docs/astro-private-cloud/v-2-x/per-deployment-migration-reference).

Selecting N Deployments and migrating them creates N independent missions, one per Deployment. There is no batch object on the server. Each mission moves through the same state machine as a cluster failover, and each succeeds or fails on its own. Partial success is normal and expected.

<Warning>
  Read this before your first migration: there is no rollback. Per-deployment migration always runs in controlled mode, which means the Deployment is fully removed from the source cluster before it is created on the destination. There is no operation to put it back.

  If the destination then fails — insufficient capacity, quota, or an image-pull failure — the mission ends `FAILED`, the Deployment's `clusterId` stays pointed at the source, and the Deployment is installed nowhere. Its database, secrets, and Dag storage are intact, so no data is lost, but the Deployment is down until you act.

  Recovery is retry-forward only: re-run the migration. Because the failed mission releases its per-Deployment lock, the Deployment is immediately eligible to migrate again. Fix whatever failed on the destination, then re-run the migration for the same Deployment to the same destination — or to another in-region cluster. Don't try to hand-restore the Deployment on the source; its release there was already purged. See [Recover a failed migration](#recover-a-failed-migration).

  Verify destination capacity and image availability before you migrate, and canary a single Deployment before you move a batch.
</Warning>

<Note>
  Migration moves the Deployments you pick, within a region, from a healthy source. Cluster failover moves everything on a cluster, can run in forced mode against a dead source, and is sequenced with your database promotion. If your source cluster is down, migration is the wrong tool — use [data plane failover](/docs/astro-private-cloud/v-2-x/trigger-data-plane-failover).
</Note>

## Read the migration UI

### Regions page

The **Regions** admin page lists every region with its **Name**, **Cloud Provider**, **Status**, and **Created** date, with **Create Region**, **Edit**, and **Delete** actions. For per-deployment migration, only **Name** and **Cloud Provider** matter. The **Status** badge and the **Activate** action belong to control plane high availability and have no effect on data plane migration — the migration gate never reads them. Deleting a region is blocked while any cluster still references it.

<Frame>
  <img alt="The Regions page listing regions with their cloud provider and status" />
</Frame>

### Clusters list

A **Failover** column shows a cluster-level failover-readiness shield that mirrors the cluster's **Failover** status on its detail page, separate from the existing health dot:

* Grey **Pending upgrade**: failover-capable but not yet upgraded.
* Amber **Upgrading**: a failover upgrade is in flight.
* Green **Ready**: the cluster is **Enabled**.
* Red **Issues**: failover-capable but not enabled, for example an External Secrets Operator (ESO) secret sync failing on one of its Deployments.

The column is empty only when the cluster isn't failover-capable. For the same status in words, open the cluster detail page's **Failover** field.

### Cluster detail

Two elements matter here:

* **Failover** is a status field that replaces the old binary "Failover Enabled": **Not Capable**, **Pending Upgrade**, **Upgrading**, **Enabled**, or **Issues Detected**. Only a cluster reading **Enabled** can be a migration destination. For the full table, see [Cluster failover states](/docs/astro-private-cloud/v-2-x/per-deployment-migration-reference#cluster-failover-states).
* **Trigger Failover** is enabled only when the status is **Enabled**. In any other state it is disabled, with a message explaining why. This triggers a cluster-level failover, not a migration.

For a cluster failover, this page also shows progress across the request's missions and the skipped-Deployments list at reconcile finalization — which is where you find Deployments that a cluster failover passed over because they were already in a migration.

<Frame>
  <img alt="The cluster detail page, where the Failover status field appears" />
</Frame>

### Deployments list

Four elements are new or changed:

| Element                   | What it does                                                                                                                                                                                                                                            |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Checkbox column           | A persistent leading column, one checkbox per Deployment. Selection is the entry point for every Deployment-scoped bulk action. A checkbox is disabled when the Deployment has an active mission of any kind, or when you lack update permission on it. |
| Bulk-actions menu         | Appears once one or more Deployments are selected. Contains **Migrate to cluster** and **Upgrade for failover**.                                                                                                                                        |
| Failover-readiness shield | A small shield icon next to the Deployment name, shown only on failover-capable clusters. See the following shield states.                                                                                                                              |
| Mission Status column     | A per-Deployment phase indicator reflecting any in-flight mission — migration, failover, or upgrade — rendered as, for example, "Migration - In progress." This is where you watch a migration from the list.                                           |

<Frame>
  <img alt="The Deployments list with the checkbox column, bulk-actions menu, failover-readiness shield, and Mission Status column" />
</Frame>

The shield uses one outline icon, distinguished by color, plus an exclamation badge for the **Attention** state:

* No shield: failover-incapable cluster.
* Grey **Pending upgrade**: failover-capable but not yet upgraded for failover.
* Amber **In progress**: an upgrade mission is in flight.
* Green **Ready**: upgraded and currently failover-eligible.
* Red **Attention**: the upgrade failed, or the Deployment is upgraded but now ineligible.

The API field behind the shield is `Deployment.failoverReadiness`. See [Per-deployment migration reference](/docs/astro-private-cloud/v-2-x/per-deployment-migration-reference#relevant-fields).

<Warning>
  The shield doesn't track migrations. It tracks the one-time failover-upgrade lifecycle only. A Deployment that is mid-migration keeps a green **Ready** shield the entire time — the shield means "this Deployment is failover-capable," not "this Deployment is idle." Use the **Mission Status** column, or `Deployment.activeMission`, to see an in-flight migration.
</Warning>

### Deployment detail

* **Migrate to cluster** is a per-Deployment action in the Deployment header's **Actions** menu, alongside **Upgrade for failover**. It calls the same `migrateDeployments` mutation with a single ID. Use it for canary moves, retries, and one-offs.
* There is no dedicated mission or flight view in the APC UI in 2.1. The durable, per-mission detail is available through the `missionProgress` GraphQL query. The **Mission Status** column on the Deployments list is the at-a-glance UI surface.

<Frame>
  <img alt="The Deployment detail page" />
</Frame>

<Note>
  There is deliberately no batch progress page. If you reload after starting a 20-Deployment migration, you haven't lost anything: you can query each Deployment's status individually through its **Mission Status** column or `missionProgress`.
</Note>

## Trigger a migration

### From the Deployments list

1. Select the Deployments with the checkbox column. Deployments with an active mission aren't selectable.
2. Open the bulk-actions menu and select **Migrate to cluster**.
3. Select the destination cluster from the picker.
4. Confirm.

<Frame>
  <img alt="The Migrate to cluster destination picker opened from the Deployments list bulk-actions menu" />
</Frame>

Results come back immediately: successfully claimed Deployments start their missions, and the outcome appears inline as a result banner, for example "47 claimed, 3 skipped (2 current mission conflict, 1 cross region migration not allowed)." The banner is orange when anything was skipped and green when everything was claimed.

Read the skip banner. Skipping is normal and non-fatal, but it means those Deployments didn't move. Every skip reason and its remedy is in [Per-deployment migration reference](/docs/astro-private-cloud/v-2-x/per-deployment-migration-reference#per-deployment-skips).

### From the Deployment detail page

Open the Deployment and select **Migrate to cluster** from the header **Actions** menu, then select a destination. It's the same mutation, the same rules, and one ID. Unlike the list, the result here is a toast: a "Migration started" toast on success, or a "Migration not started" error toast whose text is the raw skip reason when the Deployment is skipped.

<Frame>
  <img alt="The Actions menu on the Deployment detail page, showing Migrate to cluster" />
</Frame>

<Frame>
  <img alt="The Migrate to cluster destination picker opened from the Deployment detail page" />
</Frame>

### With the API

```graphql theme={null}
mutation {
  migrateDeployments(
    deploymentIds: ["<deployment-id-1>", "<deployment-id-2>"]
    destinationClusterId: "<destination-cluster-id>"
  ) {
    missions { deploymentId missionId }
    skipped  { deploymentId reason }
  }
}
```

* `deploymentIds` takes one or more IDs. Duplicates are de-duplicated. A single-element list is the trivial case; there is no separate single-Deployment mutation.
* `destinationClusterId` is the one cluster all listed Deployments move to. They may come from different source clusters, as long as each source is in the same region as the destination.
* Mode is always `CONTROLLED`. It isn't a parameter. Migration drains a healthy source before bringing the Deployment up elsewhere; if the source is unavailable, you need a cluster failover, not a migration.

Save the returned `missionId` values — they are the handle for `missionProgress`.

<Note>
  The response is synchronous in acceptance, asynchronous in execution. A returned `missionId` means the mission was durably created, not that the Deployment has moved. Watch progress in the APC UI or with `missionProgress`.
</Note>

### What migration isn't for

* Draining a whole cluster. Use full-cluster failover to decommission a cluster.
* Cross-region or cross-provider moves. The region gate blocks them.
* Moving a Deployment off a dead cluster. Migration requires a healthy source. Use cluster failover.
* Automatic destination selection. You always specify the destination.

## What happens during a migration

Each mission runs the controlled-move sequence:

```text theme={null}
Mission (kind = MOVE, mode = CONTROLLED)
  │
  ├─▶ Scavenger flight  — on the SOURCE cluster
  │     DRAIN   scale down components
  │     DELETE  uninstall the Helm release and purge its resources. The
  │             namespace itself is deleted only when namespace pools
  │             are disabled
  │
  └─▶ Hyperjump flight  — on the DESTINATION cluster
        (dispatched only after the Scavenger flight SUCCEEDS)
        NAMESPACE   create the destination namespace
        SECRETS     ESO pulls the Deployment's secrets from your backend
        FENCE       disable the source database users and terminate their
                    connections, enable the destination database users
        DEPLOYMENT  install the Helm release on the destination
        WAIT        wait for the Airflow components to become healthy
```

The phase names — `DRAIN`, `DELETE`, `NAMESPACE`, `SECRETS`, `FENCE`, `DEPLOYMENT`, and `WAIT` — are the literal values recorded against each flight and emitted in the data plane logs, which is useful when correlating a stuck migration with data plane output.

Controlled mode drains and deletes the source first, so that no Airflow instance is writing to the metadata database while another is starting. That is what makes a controlled move zero-data-loss — and also what makes it non-reversible, because by the time the Hyperjump flight runs, the source release is gone.

The migration doesn't touch the Airflow metadata database, your secrets backend, Dag storage, or the Deployment's record in the control plane. Only the Kubernetes-side installation moves.

* On success, the mission reaches `COMPLETED`, the Deployment's `clusterId` flips from source to destination, and its active-mission reference clears.
* On failure, the mission reaches `FAILED`, `clusterId` stays on the source, and the active-mission reference clears. See [Recover a failed migration](#recover-a-failed-migration).

## Observe a migration

### In the APC UI

The **Mission Status** column on the Deployments list gives a one-glance phase per Deployment, for example "Migration - In progress." There is no mission or flight detail view in the APC UI in 2.1; for the full per-mission detail, use the `missionProgress` API, which is the durable source of truth.

<Frame>
  <img alt="A Deployment with a newly claimed migration mission in the Mission Status column" />
</Frame>

<Frame>
  <img alt="A Deployment showing a migration in progress in the Mission Status column" />
</Frame>

### With the API

Query the mission IDs the mutation returned:

```graphql theme={null}
query {
  missionProgress(missionIds: ["<mission-id-1>", "<mission-id-2>"]) {
    id
    kind                  # MOVE for a migration
    state
    originClusterId
    destinationClusterId
    failoverRequestId     # always null for a migration
    flights {
      id
      role                # SCAVENGER | HYPERJUMP
      state
    }
  }
}
```

Or ask a Deployment what it is currently doing:

```graphql theme={null}
query {
  deployment(where: { id: "<deployment-id>" }) {
    isBeingFailedOver     # true while any mission is active
    failoverReadiness     # the shield state (upgrade lifecycle, not migration)
    activeMission {
      id kind state
      flights { role state }
    }
  }
}
```

`activeMission` and `isBeingFailedOver` clear to `null` and `false` as soon as the Hyperjump flight succeeds — for a per-deployment migration the mission is then `COMPLETED` — and on any terminal state. An in-progress cutover has a non-null `activeMission`; a completed one is `null` plus a flipped `clusterId`.

For the mission-state and flight-state enumerations, see [Per-deployment migration reference](/docs/astro-private-cloud/v-2-x/per-deployment-migration-reference#mission-states).

<Warning>
  Two API caveats. First, `missionProgress` silently drops missions you lack permission to see rather than returning an error, so a short result list may mean a permissions gap, not a missing mission. Second, `missionProgress` carries no timestamps, no error text, and no percentage — it tells you which flight is where in its state machine, not how long it has been there or why it failed. For failure detail, check the control plane logs for the mission ID.
</Warning>

<Note>
  `missionProgress` requires cluster-admin access — `system.clusters.get` or a cluster-admin role — the same authority needed to run a migration, so the operator who triggers migrations can observe them. A workspace member can't call it. The `deployment` query is available to anyone who can view the Deployment.
</Note>

## What is blocked during a migration

While a Deployment has an active mission, mutations that would reach the data plane for that Deployment are rejected, because they would tear down or alter resources the Hyperjump flight is provisioning. The blocked mutations are `upsertDeployment` updates, `deleteDeployment`, `updateDeploymentConfig`, `deleteDeploymentConfig`, `updateDeploymentVariables`, `updateDeploymentImage`, `updateDeploymentKedaConfig`, `createDeployRevision`, and `deployRollback`. Creating a new Deployment isn't blocked by an active mission, though it is still blocked by an active cluster failover or a cordoned target cluster.

`updateDeploymentsResources` is gated too, but it silently skips the affected Deployments instead of returning an error — check its response rather than assuming it applied to everything you selected.

The error is explicit:

```text theme={null}
deleteDeployment is disabled while deployment <deployment-id> is part of
active mission <mission-id>.
```

The block is per Deployment — other Deployments on either cluster stay fully mutable — and it lifts automatically when the mission reaches a terminal state. There is nothing to unlock manually. In practice, code deploys to a migrating Deployment fail while the move is in flight. Migrations are usually short, but tell the owning team before you start one, and avoid migrating during a team's deploy window.

## Recover a failed migration

A mission in `FAILED` needs your attention. What to do depends on where it failed. Query `missionProgress`, or open the Deployment detail page, and look at the flights:

| Symptom                                        | What it means                                                               | State of the Deployment                          |
| ---------------------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------------ |
| Scavenger `FAILED`, Hyperjump never dispatched | The source drain or uninstall failed.                                       | Still on the source, possibly partially drained. |
| Scavenger `SUCCEEDED`, Hyperjump `FAILED`      | The source was purged and the destination couldn't bring the Deployment up. | Installed nowhere. Data intact, workload down.   |
| Any flight `AWAITING_DP_HEALTH`                | Not a failure — a wait.                                                     | Unchanged. It resumes automatically.             |

If the Hyperjump flight failed, the Deployment is installed nowhere and needs the following recovery:

1. Diagnose the destination. The usual causes are node capacity, resource quota, and image-pull failure. The destination cluster's events and the control plane logs for that mission ID will say which.
2. Fix the underlying cause on the destination cluster.
3. Retry forward. Re-run `migrateDeployments` for the same Deployment and the same destination. The active-mission reference cleared when the mission reached its terminal state, so the Deployment is claimable again.
4. If the destination can't be fixed quickly, retry forward to a different in-region destination rather than trying to restore the source.

<Warning>
  Don't try to hand-restore the Deployment on the source cluster. The Helm release, namespace, and Kubernetes secrets there were deliberately purged. Recreating them out of band bypasses the fencing model and risks two Airflow instances writing to the same metadata database. Always recover by retrying forward through `migrateDeployments`.
</Warning>

The Airflow metadata database, your secrets backend, and Dag storage are untouched by a failed migration. The workload is down, but nothing is lost. Dag runs that were active during the move may need attention once the Deployment is back up.

To avoid this entirely:

* Confirm destination headroom — nodes and quota — before migrating.
* Confirm the Deployment's image is present in a registry the destination can pull from. Replication lag between regional registries is a real cause of Hyperjump failure.
* Canary one Deployment before moving a batch.
* Migrate outside the owning team's deploy window.

## Related documentation

* [Configure per-deployment migration](/docs/astro-private-cloud/v-2-x/configure-per-deployment-migration)
* [Per-deployment migration reference](/docs/astro-private-cloud/v-2-x/per-deployment-migration-reference)
* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)
* [Trigger a data plane failover](/docs/astro-private-cloud/v-2-x/trigger-data-plane-failover)


# Manage user permissions on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/manage-permissions

Manage user roles and permissions on any Astronomer Workspace and all Airflow Deployments within it.

Astro Private Cloud supports a permissions and role-based access control (RBAC) framework that allows users to configure varying levels of access both at the Workspace and Airflow Deployment levels.

Workspace and Deployment-level access can each be configured with three user roles (*Admin*, *Editor*, *Viewer*), all of which can be set and changed using the Astro Private Cloud UI and CLI. Each role maps to a combination of permissions for both Astro Private Cloud and Apache Airflow itself.

This guide includes:

1. How to invite users to an Astro Private Cloud Workspace and Deployment
2. How to view, set, and modify user roles
3. Deployment and Workspace Permissions Reference

## Invite users

Workspace and Deployment Admins can invite and otherwise manage users both via the Astro Private Cloud UI and CLI. All users who have access to a Workspace must be assigned one of three Workspace roles, though Deployment-level roles aren't required.

Read below for guidelines.

### Invite to Workspace

The ability to invite users to an Astronomer Workspace is limited to Workspace Admins, who can also grant the Admin role to other users. Workspace Editors and Viewers can't invite or otherwise manage other Workspace users, though they may do so at the Deployment level depending on their Deployment-level role.

A user who creates a Workspace is automatically granted the Admin role for the Workspace and has the ability to create any number of Airflow Deployments within it. Every Workspace must have at least one Workspace Admin.

#### Use the Astro Private Cloud UI

To invite a user to a Workspace with the Astro Private Cloud UI, select your Workspace from the Workspace list dropdown on the side navigation bar and navigate to **Workspace Settings** > **Users** > **Invite User**.

When a Workspace Admin invites a user to a Workspace in which one or more Airflow Deployments exist, they'll have the opportunity to set that user's Deployment-level roles as well, though it isn't required.

If a Workspace Admin invites a user to a Workspace that has no Airflow Deployments, the **Deployment Roles** dialog won't appear.

#### Use the Astro CLI

To invite a user to a Workspace with the Astro CLI, run:

```bash wrap theme={null}
astro workspace user add <email-address> --workspace-id <workspace-id> --role <workspace-role>
```

Only Workspace *Admins* can invite other users and set their permissions.

To find **Workspace ID**, you can:

* Run `$ astro workspace list`
* Find it in the Workspace URL from your browser after the `/w/` (for example, `https://app.basedomain/w/<workspace-id>`)

To set a **Role**, add a flag in the following format:

* `--WORKSPACE_EDITOR`
* `--WORKSPACE_VIEWER`
* `--WORKSPACE_ADMIN`

If you do *not* specify a role in this command, `WORKSPACE_VIEWER` will be set by default. In all cases where a user is invited to a Workspace and Deployment-level role isn't specified, no Deployment-level role will be assumed.

#### Use Teams

You can invite a group of users from a configured third party identity provider (IdP) as a Team to your Workspace. A Team is an IdP-defined group of users who all share the same permissions to a given Deployment or Workspace.

Note that to use Teams, a System Admin must first complete the setup in [Integrate an auth system](/docs/astro-private-cloud/v-2-x/integrate-auth-system) and configure user groups as described in [Import IdP Groups](/docs/astro-private-cloud/v-2-x/import-idp-groups).

To add a Team to a Workspace:

1. In the Control Plane UI, go to your **Workspace Settings** page and open the **Teams** tab.

2. Click **+Team**.

3. Under **Team Name**, enter the name of your IdP group.

4. Select a **Workspace Role** for the Team. If your Workspace has existing Deployments, you can also configure the Team's permissions to those Deployments on this page:

   <Frame>
     <img alt="Screen for adding a Team to a Workspace" />
   </Frame>

5. Click **Add**.

<Warning>If a user already exists on a Workspace before being invited via a Team, the user context with the most permissive role will be applied to the Workspace. For more information, read [Import IdP Groups](/docs/astro-private-cloud/v-2-x/import-idp-groups).</Warning>

### Invite to Deployment

The ability to invite Workspace users to an Airflow Deployment within it is limited to Deployment *Admins*, who can also grant the *Admin* role to other users. Deployment *Editors* and *Viewers* can't invite or otherwise manage users. A user who creates a Deployment is automatically granted the *Admin* role within it.

<Note>
  In order for a user to be granted access to an Airflow Deployment, they must *first* be invited to and assigned a role within the Workspace. A user can be a part of a Workspace but have no access or role to any Airflow Deployments within it.
</Note>

#### Use the Astro Private Cloud UI

To invite a Workspace user to an Airflow Deployment via the Astro Private Cloud UI:

1. Select your **Workspace** and then navigate to **Deployment** > **Users**.
2. Type the Workspace user's name in the search bar on top or click **Show All** to view all users.
3. Select a Deployment role from the drop-down menu to the right of the selected user.
4. Click the `+` symbol.

#### Use the Astro CLI

To invite a Workspace user to an Airflow Deployment using the Astro CLI, run:

```text wrap theme={null}
astro deployment user add --email=<email-address> --deployment-id=<deployment-id> --role=<deployment-role>
```

Only Deployment *Admins* can invite other users and set their permissions.

To find **Deployment ID**, you can:

* Run `$ astro deployment list`

To set a **Role**, add a flag in the following format:

* `--DEPLOYMENT_EDITOR`
* `--DEPLOYMENT_VIEWER`
* `--DEPLOYMENT_ADMIN`

If you do *not* specify a role in this command, `DEPLOYMENT_VIEWER` will be set by default.

#### Use Teams

You can invite a group of users from a configured third party identity provider (IdP) as a Team on your Deployment. A Team is an IdP-defined group of users who all share the same permissions to a given Deployment or Workspace.

Note that to use Teams, a System Admin must first complete the setup in [Integrate an auth system](/docs/astro-private-cloud/v-2-x/integrate-auth-system) and configure user groups as described in [Import IdP Groups](/docs/astro-private-cloud/v-2-x/import-idp-groups).

To add a team to a Deployment:

1. In the Control Plane UI, go to your Deployment and open the **Teams** tab.
2. In the search bar that appears, search for your Team's name.
3. When your Team appears, select a Deployment-level role for the Team and click the **+** button:

   <Frame>
     <img alt="Screen for adding a Team to a Deployment" />
   </Frame>

<Warning>If a user already exists on a Deployment before being invited via a Team, the user context with the most permissive role will be applied to the Deployment. For more information, read [Import IdP Groups](/docs/astro-private-cloud/v-2-x/import-idp-groups).</Warning>

## View and edit user roles

### Workspace

#### View Workspace users

To view roles within a Workspace via the Astro Private Cloud UI, select your Workspace from the left sidebar and navigate to **Workspace Settings** > **Users**. All Workspace users have access to this view and can see the roles of other users.

To list Workspace users using the Astro CLI, run:

```bash wrap theme={null}
astro workspace user list
```

This command will output the email addresses of all users in the Workspace alongside their ID and Workspace Role.

#### Edit Workspace user role

If you're a Workspace *Admin*, you can edit both Workspace and Deployment-level permissions by selecting your Workspace from the left sidebar and navigating to **Workspace Settings** > **Users** and clicking into an individual user.

To edit a user's role using the Astro CLI, run:

```bash wrap theme={null}
astro workspace user update <email> --workspace-id=<workspace-id> --role=<workspace-role>
```

Only Workspace *Admins* can modify the role of another user in the Workspace.

#### Remove Workspace user

Workspace *Admins* can remove users from a Workspace by selecting your Workspace from the left sidebar and navigating to: **Workspace Settings** > **Users** > **Individual User** > **Remove User**.

<Frame>
  <img alt="Remove Workspace User" />
</Frame>

To remove a user from a Workspace with the Astro CLI, make sure you're first operating in that Workspace. Then, run:

```bash wrap theme={null}
astro workspace user remove <email>
```

Only Workspace *Admins* can remove other Workspace users.

### Deployment

#### View Deployment users

To list all users within a Deployment and their corresponding roles, select your Workspace from the left sidebar and navigate to **Deployments** > **Individual Deployment** > **Users**. All Deployment users have access to this view and can see the roles of other users.

To list Deployment users with the Astro CLI, run:

```bash wrap theme={null}
astro deployment user list --deployment-id=<deployment-id>
```

#### Edit Deployment user role

Deployment *Admins* can edit permissions using the dropdown menu in the **Access** tab in the Astro Private Cloud UI.

To edit a user's role with the Astro CLI, run:

```bash wrap theme={null}
astro deployment user update <email> --deployment-id=<deployment-id> --role=<deployment-role>
```

<Note>
  A Deployment-level role can't be edited while a Workspace invitation to that user is pending. If you invite a user to a Workspace, you won't be able to modify their permissions until they accept the Workspace invite.
</Note>

#### Remove Deployment user

To delete a user from an Airflow Deployment with the Astro Private Cloud UI, Deployment *Admins* can click the wastebasket icon within the **Access** tab shown in the image above.

To delete a user from an Airflow Deployment with the Astro CLI, run:

```bash wrap theme={null}
astro deployment user remove <email> --deployment-id=<deployment-id>
```

## User permissions reference

### Deployment

#### Deployment Viewer

Deployment *Viewers* are limited to read-only mode. They can only:

* View Deployment users
* View the **Metrics** and **Logs** tabs of the Astro Private Cloud UI
* View information about Dags and tasks in the Airflow UI

Deployment Viewers can't deploy to, modify, or delete anything within an Airflow Deployment. Additionally, they can't create or use service accounts to do so. Attempts to modify a Deployment in any way will result in a `403` and an `Access is Denied` message.

<Frame>
  <img alt="Access Denied" />
</Frame>

#### Deployment Editor

With fewer permissions than *Admins*, a Deployment *Editor*:

* Can access and make changes to the Deployment on Astronomer, such as modify resources, add environment variables, or push code
* Can't delete the Deployment
* Can perform CRUD operations on any service account in the Deployment
* Can't manage other users in the Deployment
* Has full access to modify and interact with Dags in the Airflow UI
* doesn't have access to the **Admin** menu in Airflow, which includes:
  * Pools
  * Configuration
  * Users
  * Connections
  * Variables
  * XComs

<Frame>
  <img alt="No Admin Tab" />
</Frame>

#### Deployment Admin

Deployment *Admins* are the highest-tiered role. Admins:

* Can perform CRUD (create, read, update, delete) Astronomer operations on the Deployments, such as modify resources, add environment variables, push code, or delete the Deployment
* Can manage users and their permissions in the Deployment
* Can perform CRUD operations on any service account in the Workspace
* Can perform CRUD Airflow operations (push code, add Connections, clear tasks, delete Dags etc.)
* Have full access to the **Admin** menu in the Airflow UI
* Have full access to modify and interact with Dags in the Airflow UI

Every Deployment must have at least one Deployment *Admin*.

### Workspace

#### Workspace Viewer

A Workspace *Viewer* is limited to read-only mode. *Viewers*:

* Can list users in a Workspace
* Can view all service accounts in the Workspace
* Can't delete or modify the Workspace or its users

<Note>
  If a role isn't set, newly invited users are Workspace *Viewers* by default.
</Note>

#### Workspace Editor

Below a Workspace *Admin*, an *Editor*:

* Can access and make changes to the Workspace in the **Settings** tab
* Can perform CRUD operations on any service account in the Workspace
* Can create Airflow Deployments in the Workspace
* Can't manage other users in the Workspace
* Can't delete the Workspace

#### Workspace Admin

Workspace *Admins* are the highest-tiered role at the Workspace level. Admins:

* Can manage users and their permissions in a Workspace.
* Can perform CRUD (create, read, update, delete) operations on the Workspace (for example, delete the Workspace, change its name).
* Can create Airflow Deployments in the Workspace.
* Can perform CRUD operations on any Airflow Deployment within the Workspace.
* Can perform CRUD operations on any service account in the Workspace.

Every Workspace must have at least one Workspace *Admin*.

A Workspace *Admin* always has these permissions for any Deployment in the Workspace. Even if a Workspace Admin also has a defined role with lower permissions like Deployment *Viewer*, Astronomer uses the permissions configured for the user at the Workspace level.

## System roles

### System Viewer

System *Viewers* have read-only access across the entire platform. They:

* Can view the Airflow UI and configuration for any Deployment
* Can view environment variables and settings for any Deployment
* Can view all Workspaces, users, and service accounts
* Can view system monitoring dashboards
* Can view pending user invites and the current Astronomer release version

***

### System Editor

System *Editors* have write access to most configurations but not full admin control. They:

* Inherit all System Viewer permissions
* Can modify environment variables and IAM roles for any Deployment
* Can create, update, and delete service accounts for any Workspace or Deployment
* Can view system admin users
* Can push or modify base Docker images used by Airflow

***

### System Admin

System *Admins* have complete administrative access across the Astronomer platform. They:

* Inherit all System Viewer and Editor permissions
* Can create, modify, or delete any Deployment or Workspace
* Can manage all users, roles, and service accounts globally
* Can view and manage logs and metrics for all Deployments
* Can push images and deploy code to any Deployment
* Can invite, delete, or force-delete users (including IdP-managed users)
* Can manually verify user emails
* Can perform system-level Airflow administration (pools, connections, variables, etc.)
* Can perform system cleanup operations (for example, purge Airflow metadata)

## What's next

As an Astro Private Cloud user, you can customize all user permissions at the platform-level, or create additional roles with their own custom sets of permissions. For more information, read:

* [Manage Users on Astro Private Cloud](/docs/astro-private-cloud/v-2-x/manage-platform-users#customize-role-permissions)
* [Create custom roles](/docs/astro-private-cloud/v-2-x/custom-roles)
* [Integrate an auth system](/docs/astro-private-cloud/v-2-x/integrate-auth-system)


# Manage users on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/manage-platform-users

Add and customize user permissions on Astro Private Cloud.

Astro Private Cloud allows you to adjust permissions for each user role and define how new users join your organization.

This document covers customizing user signups and user roles, and using Astro Private Cloud system-level permissions. For a list of the default permissions for each role, see [User roles and permissions](/docs/astro-private-cloud/v-2-x/role-permission-reference).

To learn more about managing users through your identity provider (IdP), see [Import IdP groups](/docs/astro-private-cloud/v-2-x/import-idp-groups).

#### Prerequisites

* System Admin access to Astro Private Cloud.
* Access to your platform's `values.yaml` file.

## Add users to Astro Private Cloud

When you first deploy Astro Private Cloud, the first user to sign in receives System Admin permissions by default. After that, Astro Private Cloud creates a user in any of the following ways:

* A Workspace Admin invites them to a Workspace
* A System Admin invites them to the platform
* They sign up through the Astro Private Cloud UI without an invitation, which requires `publicSignups`
* Astro Private Cloud imports them through an [IdP group](/docs/astro-private-cloud/v-2-x/import-idp-groups)

As a System Admin, you can open the platform to public signups, limit account creation to users invited by others, or make it so that users can only join the platform as part of an [IdP-based Team](/docs/astro-private-cloud/v-2-x/import-idp-groups).

<Note>
  New users appear under a System Admin's **Users** tab only after the new user signs in for the first time.
</Note>

<Note>
  You can bypass the email verification process for new users through an Astro Private Cloud (APC) API mutation. For the format of this mutation, see [Use the APC API](/docs/astro-private-cloud/v-2-x/houston-api).
</Note>

### Enable public signups

Public signups allow any user with access to your base domain to create an account. If you disable public signups, users who try to access Astro Private Cloud without an invitation from another user receive an error.

Enabling public signups can simplify initial setup when SMTP credentials are difficult to acquire, because disabling public signups requires that a user accept an email invitation. `publicSignups` is an APC API configuration that you set in the `values.yaml` file of your Helm chart.

To enable public signups, add the following YAML snippet to your `values.yaml` file:

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      publicSignups: true
      emailConfirmation: false # Set to false if you also want to disable other SMTP-dependent features
```

An example `values.yaml` file would look like:

```yaml wrap theme={null}
global:
  baseDomain: mybasedomain
  tlsSecret: astronomer-tls
nginx:
  loadBalancerIP: 0.0.0.0
  preserveSourceIP: true

astronomer:
  houston:
    config:
      publicSignups: true
      emailConfirmation: false
```

Then, push the configuration change to your platform as described in [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

## System permissions on Astro Private Cloud

The System Admin role grants full permissions across all clusters, Workspaces, and Deployment entities. Users with this role can monitor and take action on Workspaces, Deployments, and users throughout all clusters.

On Astro Private Cloud, System Admins specifically can:

* Register a data plane
* List all data planes
* Manage data plane-wide config
* De-register a data plane
* List and search all users
* List and search all Deployments
* Access the Airflow UI for all Deployments
* Delete a user
* Delete a Deployment
* Access Grafana for cluster-level monitoring
* Add other System Admins

Astro Private Cloud also supports a System Editor and a System Viewer permission set.

No user holds the System Editor or System Viewer role by default. System Admins assign these roles using the APC API. After you assign the System Viewer role, for example, that user can access Grafana but can't delete a Workspace they don't belong to.

You can customize all three Astro Private Cloud permission sets to meet your requirements. For more information about the default configurations for the System Admin, Editor, and Viewer roles, see the [APC API default config](https://github.com/astronomer/astronomer-docs-resources/blob/main/software/software_configs/0.35/default.yaml).

### Assign users System Admin roles

Use the **System Admin** tab in the Astro Private Cloud UI to add System Admins.

Keep in mind that:

* Only existing System Admins can grant the System Admin role to another user.
* The user must have a verified email address and already exist in the system.

<Note>
  To assign a user a different system-level role, either `SYSTEM_VIEWER` or `SYSTEM_EDITOR`, use an API call from your platform's GraphQL playground. For guidelines, see [Use the APC API](/docs/astro-private-cloud/v-2-x/houston-api).
</Note>

#### Verify System Admin access

To verify a user was successfully granted the System Admin role, ensure they can do the following:

* Go to `grafana.<BASEDOMAIN>`
* Access the **System Admin** tab from the top left menu of the Astro Private Cloud UI

## User roles on Astro Private Cloud

Administrators can customize permissions across your installation. On Astro Private Cloud, you can assign users roles at four levels:

* Deployment level (Viewer, Editor, Admin)
* Workspace level (Viewer, Editor, Admin)
* System level (Viewer, Editor, Admin)
* Cluster level (Admin)

Deployment roles apply to a Deployment within a single Workspace. Workspace roles apply to all Airflow Deployments within a single Workspace. Cluster roles apply to all Workspaces across a single data plane. System roles apply across all data planes. For more information about the three Workspace-level roles on Astro Private Cloud (Viewer, Editor, and Admin), see [Manage user permissions on an Astro Private Cloud Workspace](/docs/astro-private-cloud/v-2-x/manage-permissions).

## Customize role permissions

Astro Private Cloud defines permissions as `scope.entity.action`, where:

* `scope`: The layer of the platform to which the permission applies
* `entity`: The object or role being operated on
* `action`: The verb describing the operation being performed on the `entity`

For example, the `deployment.serviceAccounts.create` permission translates to the ability for a user to create a Deployment-level service account in any Deployment to which they belong. For the available platform permissions and default role configurations, see [User roles and permissions](/docs/astro-private-cloud/v-2-x/role-permission-reference).

A permission for a given `scope` applies only to the parts of the scope a user was invited to. For example, a user with a role including the `workspace.serviceAccounts.get` permission can view service accounts only in the Workspaces they belong to.

The steps in this section modify the permissions of the built-in System, Workspace, and Deployment roles through your `values.yaml` file. To instead create additional roles with their own custom sets of permissions and assign them through the Astro Private Cloud UI, see [Create custom roles](/docs/astro-private-cloud/v-2-x/custom-roles).

### Role permission inheritance

In addition to their own permissions, roles inherit permissions from other roles.

There are several chains of inheritance in the Astro Private Cloud RBAC system. In the following list, `>` represents "inherits from":

* System Admin > System Editor > System Viewer > User
* Deployment Admin > Deployment Editor > Deployment Viewer > User
* Workspace Admin > Workspace Editor > Workspace Viewer > User

### Modify built-in role permissions

<Steps>
  <Step title="Identify a permission change">
    Review the default roles and permissions in the [default APC API configuration](https://github.com/astronomer/astronomer-docs-resources/blob/main/software/software_configs/0.37/default.yaml) and determine the following:

    * What role you want to configure. For example, `DEPLOYMENT_EDITOR`.
    * What permissions you want to add to or remove from the role. For example, `deployment.images.push`.

    For example, you might want to block a `DEPLOYMENT_EDITOR` (and therefore `WORKSPACE_EDITOR`) from deploying code to all Airflow Deployments within a Workspace and instead limit that action to users assigned the `DEPLOYMENT_ADMIN` role.
  </Step>

  <Step title="Modify your values.yaml file">
    Apply the role and permission changes to your organization's `values.yaml` file. For example:

    ```yaml wrap theme={null}
    astronomer:
      houston:
        config:
          roles:
            DEPLOYMENT_EDITOR:
              permissions:
                deployment.images.push: false
    ```

    In the same way you can remove permissions from a particular role by setting a permission to `:false`, you can add permissions to a role at any time by setting a permission to `:true`.

    For example, if you want to allow any `DEPLOYMENT_VIEWER` (and therefore `WORKSPACE_VIEWER`) to push code directly to any Airflow Deployment within a Workspace, you'd specify the following:

    ```yaml wrap theme={null}
    astronomer:
      houston:
        config:
          roles:
            DEPLOYMENT_VIEWER:
              permissions:
                deployment.images.push: true
    ```

    Push the configuration change to your platform. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).
  </Step>
</Steps>

### Example customization: limit Workspace creation

Unless otherwise configured, a user who creates a Workspace on Astro Private Cloud is automatically granted the `WORKSPACE_ADMIN` role and can create an unlimited number of Airflow Deployments within that Workspace. For organizations looking to more strictly control resources, Astro Private Cloud supports limiting the Workspace creation function through the `USER` role.

Astro Private Cloud includes a `USER` role that is synthetically bound to all users within the control plane. By default, this role includes the `system.workspace.create` permission.

If you're a System Admin who wants to limit Workspace creation, you can:

* Set the `system.workspace.create` permission for the `USER` role to `false`.
* Attach the `system.workspace.create` permission to a separate role of your choice.

You might want to limit this permission to the `SYSTEM_ADMIN` role on the platform, because System Admins can be responsible for managing cluster-level resources and costs. To reassign this permission to System Admins, your `values.yaml` would appear similar to the following example:

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      roles:
        SYSTEM_ADMIN:
          permissions:
            system.workspace.create: true
        USER:
          permissions:
            system.workspace.create: false
```


# Manage Workspaces and Deployments on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/manage-workspaces

Manage Astro Private Cloud Workspaces, Airflow Deployments, users, teams, and service accounts from the Astro Private Cloud UI.

A Workspace is the highest level of organization in Astro Private Cloud. From a Workspace, you can manage a collection of Clusters (which function as data planes).

With the separated control plane (CP)/data plane (DP) architecture in Astro Private Cloud:

* The CP hosts the Astro Private Cloud UI, API, and global services.
* A Cluster represents a DP and is where Airflow Deployments are created and run.
* Each Workspace can span multiple Clusters, giving admins the flexibility to organize Deployments across environments such as dev, staging, and production.

If you’re not a member of any Workspaces, you’ll be prompted to create one the first time you sign in to the Astro Private Cloud UI. If you already have access to at least one Workspace, you can create a new one at any time using the **New Workspace** button in the sidebar.

This page covers creating and managing Workspaces as a Workspace Admin. It’s organized by the four tabs available from a Workspace’s menu in the Astro Private Cloud UI:

* **Workspace Settings** > **General**
* **Workspace Settings** > **Users**
* **Workspace Settings** > **Teams**
* **Workspace Settings** > **Service Accounts**

<Frame>
  <img alt="Workspace configuration tab location" />
</Frame>

## Deployments

The most important function of Workspaces is to create and manage access to one or more Clusters and their associated Airflow Deployments.

An Airflow Deployment is an instance of Apache Airflow running within a Cluster (data plane). Each Deployment includes a scheduler, webserver, and one or more workers if you’re running the Celery Executor or Kubernetes executor.

To create a new Deployment, go to the **Deployments** tab in your Workspace and select **New Deployment**. You can also create Deployments using the Astro CLI as described in the CLI Quickstart.

<Frame>
  <img alt="The Deployments tab in the Astro Private Cloud UI" />
</Frame>

You can't use or share Deployments across Workspaces. You can push local Dags and code to any Deployment at any time, but there is no way to move an existing Airflow Deployment from one Workspace to another once created.

## Settings

You can rename your Workspace or update its description in the **Workspace Settings** tab. While these fields have no effect on how tasks are executed, Astronomer recommends configuring them to give users an idea of the Workspace's purpose and scope.

## Users

You can see who has access to the Workspace in the **Users** tab of the **Workspace Settings**.

If you want to share access to other members of your organization, invite them to a Workspace you're a part of. Once your team members are part of your Workspace, Deployment Admins can grant them varying levels of access to Airflow Deployments within the Workspace. Likewise, Workspace Admins can grant them varying levels of access to the entire Workspace.

For a full breakdown of user roles and permissions, see [Astro Private Cloud user role and permission reference](/docs/astro-private-cloud/v-2-x/role-permission-reference).

In addition, Astro Private Cloud system admins can add or remove specific permissions for each type of user role. For more information on this feature, read [Customize Permissions](/docs/astro-private-cloud/v-2-x/manage-platform-users#customize-role-permissions).

## Teams

Use the **Teams** tab in **Workspace Settings** to manage group-based access controls. Teams allow you to assign permissions to groups of users.

The **Teams** tab displays a list of all teams with access to the Workspace, including their provider, description, and Workspace role. You can search for teams within the Workspace using the search bar.

To add a team to the Workspace, select the **+ Team** button. In the **Add Team to Workspace** panel, search for and select an existing team. Assign the appropriate Workspace role, **Viewer**, **Editor**, or **Admin**, to define the team's access level. Optionally, assign Deployment-level roles for individual Airflow Deployments by selecting Deployments and choosing the required role for each.

You can also apply Deployment roles in bulk by using the **Select All** option and applying the desired role to multiple Deployments simultaneously.

## Service accounts

Use the **Service Accounts** tab of the **Workspace Settings** to create a Workspace-level service account. Service accounts generate a permanent API key that you can use to automate any action at the Workspace level, such as deploying to your Workspace's Airflow Deployments using a CI/CD tool of your choice.

To automate actions at the Deployment level, create a Deployment service account. For more information on this feature, read [Deploy via CI/CD](/docs/astro-private-cloud/v-2-x/ci-cd).


# Migrate from unified mode to a split control plane and data plane
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/migrate-unified-to-split

Migrate an existing unified-mode Astro Private Cloud installation to a split control plane and data plane in the same cluster, with no Airflow downtime.

Astro Private Cloud (APC) supports three provisioning modes, set by `global.plane.mode` in the APC Helm chart:

* **unified** — the control plane and data plane run in the same cluster and namespace. This is the legacy default for existing APC installations.
* **control** — control plane components only, managing one or more remote data planes.
* **data** — data plane components only, registered with and managed by a remote control plane.

A split control plane and data plane (CP/DP) is a prerequisite for capabilities such as [data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover), multi-cluster management, and dedicated Airflow clusters. This document describes how to migrate an existing unified-mode installation to split mode within the same cluster, with no Airflow downtime and no data loss.

The migration is a two-phase process:

1. **Control plane migration** — Put the cluster into maintenance mode, then run a Helm upgrade that switches `global.plane.mode` from `unified` to `control`. The existing cluster becomes a standalone control plane. Airflow Deployments keep running throughout, but the platform accepts no updates from the API or UI during the transition.
2. **Data plane provisioning and re-association** — Provision a new data plane in the same cluster, verify its endpoints, and bind the new data plane back to the control plane.

Deployment domains don't change during the migration. The split data plane reuses the existing `global.baseDomain` and ingress controller, so Airflow Deployment URLs stay identical to the prior unified installation.

## Migration steps at a glance

1. Upgrade the cluster to APC 2.1, the minimum version eligible for the CP/DP split.
2. Enable maintenance mode on the cluster to block Deployment provisioning calls during the migration.
3. Upgrade the platform from unified to control mode.
4. Provision a new data plane in the same namespace.
5. Re-bind the control plane to the data plane's internal gRPC and metadata endpoints.
6. Disable maintenance mode and resume Airflow Deployment operations.

## Prerequisites

* An APC platform running in unified mode, upgraded to APC 2.1 or later. APC 2.1 is the minimum version eligible for the CP/DP split.
* The Cluster Admin or System Admin role. Only these roles can put a cluster into maintenance mode.
* Access to the platform Helm release and `kubectl` access to the cluster.
* A backup of the platform metadata database and your current Helm values, taken before you begin.
* A record of your existing unified-mode Helm values. You reuse most of them and change only the keys shown in this document.

<Note>
  If your unified installation uses in-cluster Elasticsearch for logging, you must enable shared Elasticsearch during the migration (shown in Phase 1) so that Airflow task and component logs survive the mode change.
</Note>

## Phase 1: Migrate the control plane

```mermaid theme={null}
sequenceDiagram
    actor User as UI / API
    participant API as APC API
    participant Cluster as Unified cluster
    participant AF as Airflow deployments

    Note over User,AF: 1. Enable maintenance mode
    User->>API: cordonCluster mutation
    API->>Cluster: set maintenance mode on
    Cluster-->>API: cordoned
    User--xCluster: deployment mutations blocked

    Note over User,AF: 2. Helm upgrade — unified to control
    User->>Cluster: helm upgrade (plane.mode: control)
    Cluster->>Cluster: data plane components removed
    Cluster-->>API: control plane ready

    Note over User,AF: 3. Airflow workloads keep running
    AF->>AF: Dag runs, triggerer, and schedulers continue
```

<Steps>
  <Step title="Enable maintenance mode">
    Put the cluster into maintenance mode so that Deployment mutations are blocked during the cutover. This also cordons existing Airflow Deployments so that the Helm upgrade doesn't re-roll them.

    ```graphql theme={null}
    mutation {
      cordonCluster(clusterId: "<cluster-id>", reason: "maintenance") {
        id
        isCordoned
        cordonedAt
      }
    }
    ```

    While maintenance mode is active, the platform blocks REST API calls, GraphQL mutations, and Astro CLI operations that change Airflow Deployment state on the cluster, and returns `cluster is under maintenance`. Existing Airflow Deployments keep running: Dag runs, task execution, the triggerer, and schedulers are unaffected.
  </Step>

  <Step title="Update the Helm values for control mode">
    Change the following keys in your existing platform values. Keep all other unified-mode values unchanged.

    ```yaml theme={null}
    global:
      baseDomain: <base-domain>  # unchanged from your unified install
      plane:
        mode: "control"          # switched from "unified"
      # Enable only if your unified install uses in-cluster Elasticsearch logging.
      # Keeps Elasticsearch running in control mode as a shared log store.
      sharedElasticsearch:
        enabled: true
    astronomer:
      houston:
        upgradeDeployments:
          enabled: false         # disables the upgrade deployments job
    ```

    <Accordion title="(Optional) Bring Your Own ingress controller or OpenShift">
      If you run your own ingress controller or Red Hat OpenShift, enable the auth sidecar and set your ingress annotations instead. Adjust the annotations to match your ingress controller class.

      ```yaml theme={null}
      global:
        baseDomain: <base-domain>  # unchanged from your unified install
        plane:
          mode: "control"          # switched from "unified"
        # Enable only if your unified install uses in-cluster Elasticsearch logging.
        # Keeps Elasticsearch running in control mode as a shared log store.
        sharedElasticsearch:
          enabled: true
        authSidecar:
          enabled: true
        extraAnnotations:
          # set according to your ingress controller class configuration
          openshift-default.kubernetes.io/ingress.class: openshift-default
          route.openshift.io/termination: edge
      astronomer:
        houston:
          upgradeDeployments:
            enabled: false         # disables the upgrade deployments job
      ```
    </Accordion>

    <Warning>
      Applying these values runs a Helm upgrade, which upgrades every Airflow Deployment on an uncordoned cluster. Confirm the cluster is in maintenance mode (the previous step) before you apply, and set `--set astronomer.houston.upgradeDeployments.enabled=false`.
    </Warning>

    <Warning>
      If your unified install uses in-cluster Elasticsearch, you must set `global.sharedElasticsearch.enabled: true`. Without this flag, Elasticsearch is torn down when the cluster switches to control mode and historical logs become unavailable.
    </Warning>
  </Step>

  <Step title="Run the Helm upgrade">
    Apply the updated values with a Helm upgrade and wait for the release to reach a deployed state. The data plane components are removed and the cluster is reconfigured as a standalone control plane.

    After the upgrade, Airflow Deployments are no longer explorable in the UI. This is expected: the data plane internal API isn't available in control-only mode, so calls to it return an error until you provision and bind the new data plane in Phase 2. Existing Airflow workloads continue to run.
  </Step>
</Steps>

## Phase 2: Provision the data plane and re-associate

```mermaid theme={null}
sequenceDiagram
    actor User as UI / API
    participant API as APC API
    participant CP as Control plane
    participant DP as New data plane

    Note over User,DP: 1. Install the data plane (plane.mode: data)
    User->>DP: helm install (same baseDomain, nginx disabled)
    DP-->>User: data plane ready

    Note over User,DP: 2. Re-bind endpoints (updateCluster)
    User->>API: updateCluster mutation
    API->>CP: update metadata and internal API endpoints
    CP-->>DP: bind data plane endpoints

    Note over User,DP: 3. Disable maintenance mode
    User->>API: uncordonCluster mutation
    API-->>User: normal operations restored
```

<Steps>
  <Step title="Provision the data plane">
    Install a new data plane release in the same namespace. For a same-cluster data plane:

    * Leave `global.plane.domainPrefix` empty so that the data plane reuses the existing `global.baseDomain`.
    * Set `global.nginx.enabled: false` so that the data plane reuses the existing ingress controller instead of installing its own.
    * Set the ingress class annotation so that traffic routes to the existing controller.

    ```yaml theme={null}
    global:
      baseDomain: <base-domain>                   # same as the control plane
      plane:
        mode: "data"
        domainPrefix: ""                          # reuse the existing baseDomain
      sharedElasticsearch:
        enabled: true                             # match the control plane setting
      extraAnnotations:
        kubernetes.io/ingress.class: "<control-plane-release-name>-nginx"
      nginx:
        enabled: false                            # reuse the control plane's ingress controller
      postgresql:
        enabled: false
      tlsSecret: astronomer-tls
      ssl:
        enabled: true
        mode: "require"
    ```

    <Accordion title="(Optional) Bring Your Own ingress controller or OpenShift">
      If you run your own ingress controller or Red Hat OpenShift, enable the auth sidecar and set your ingress annotations instead. Adjust the annotations to match your ingress controller class.

      ```yaml theme={null}
      global:
        baseDomain: <base-domain>                   # same as the control plane
        plane:
          mode: "data"
          domainPrefix: ""                          # reuse the existing baseDomain
        sharedElasticsearch:
          enabled: true                             # match the control plane setting
        authSidecar:
          enabled: true
        extraAnnotations:
          # set according to your ingress controller class configuration
          openshift-default.kubernetes.io/ingress.class: openshift-default
          route.openshift.io/termination: edge
      ```
    </Accordion>

    <Accordion title="(Optional) In-cluster hosted registry with a StatefulSet">
      If your data plane uses an in-cluster registry hosted with a StatefulSet, include the following block in the data plane values so it reuses the control plane's registry.

      ```yaml theme={null}
      astronomer:
        registry:
          enabled: true
          fullnameOverride: <control-plane-release-name>-registry
      ```
    </Accordion>

    <Warning>
      Don't set `global.plane.domainPrefix`. Setting it generates a new set of Deployment base domains, which causes widespread downtime, forces a Deployment upgrade, and changes user-facing URLs.
    </Warning>

    <Warning>
      The data plane and control plane share one ingress controller in split mode. If you use the default ingress controller, set the `global.extraAnnotations` ingress class to the control plane's ingress class (`<control-plane-release-name>-nginx`) so that traffic routes to the existing controller.
    </Warning>

    Use a different Helm release name from the original unified release. Wait for the data plane Pods to become ready.
  </Step>

  <Step title="Re-bind the control plane to the data plane">
    Point the control plane at the new data plane's internal API (gRPC) and metadata endpoints with the `updateCluster` mutation.

    ```graphql theme={null}
    mutation {
      updateCluster(
        id: "<cluster-id>"
        commanderEndpoint: "commander.<global-baseDomain>:443"
        metadataEndpoint: "http://<dataplane-release-name>-commander.<namespace>.svc.cluster.local.:8880"
      ) {
        id
      }
    }
    ```

    Cluster metadata reconciles automatically after `updateCluster`, so the split-mode configuration is refreshed without a separate manual sync.

    <Info>
      The metadata and gRPC endpoint values appear in the post-install output of the data plane Helm release. Copy those values into the `updateCluster` mutation.
    </Info>
  </Step>

  <Step title="Disable maintenance mode">
    Take the cluster out of maintenance mode to restore normal operations.

    ```graphql theme={null}
    mutation {
      uncordonCluster(clusterId: "<cluster-id>") {
        id
        isCordoned
        cordonedAt
      }
    }
    ```
  </Step>
</Steps>

## Verify the migration

Confirm the following after the cutover:

* The control plane and data plane Pods are running and healthy.
* Each Airflow Deployment is running and reachable at its existing domain. The same-cluster data plane reuses `baseDomain`, so URLs are unchanged.
* Airflow task logs and component logs are continuous across the migration with no gap. This confirms the shared Elasticsearch configuration.
* Metrics history is intact.
* The cluster is out of maintenance mode and Deployment mutations succeed again. Create or update a test Deployment to confirm.

## Roll back

You can roll back a CP/DP split setup to a unified installation with the following steps.

<Steps>
  <Step title="Enable maintenance mode">
    Put the cluster into maintenance mode so that Deployment mutations are blocked during the cutover. This also cordons existing Airflow Deployments so that the Helm upgrade doesn't re-roll them.

    ```graphql theme={null}
    mutation {
      cordonCluster(clusterId: "<cluster-id>", reason: "maintenance") {
        id
        isCordoned
        cordonedAt
      }
    }
    ```
  </Step>

  <Step title="Uninstall the split data plane Helm release">
    ```bash theme={null}
    helm uninstall <dataplane-release-name> --debug
    ```
  </Step>

  <Step title="Upgrade the control plane back to unified">
    Change the following keys in your control plane platform values to switch from split back to unified mode.

    ```yaml theme={null}
    global:
      baseDomain: <base-domain>  # unchanged from your split install
      plane:
        mode: "unified"          # switched from "control"
    astronomer:
      houston:
        upgradeDeployments:
          enabled: false         # disables the upgrade deployments job
    ```

    Apply the values with a Helm upgrade:

    ```bash theme={null}
    helm upgrade <control-plane-release-name> -f config.yaml --no-hooks
    ```

    <Info>
      Running with `--no-hooks` skips the blocking jobs for the missing components. After the Helm upgrade completes, all components return to normal.
    </Info>
  </Step>

  <Step title="Re-bind the metadata and gRPC endpoints">
    Point the control plane at the data plane's internal API (gRPC) and metadata endpoints with the `updateCluster` mutation.

    ```graphql theme={null}
    mutation {
      updateCluster(
        id: "<cluster-id>"
        commanderEndpoint: "<control-plane-release-name>-commander.<namespace>.svc.cluster.local:50051"
        metadataEndpoint: "http://<control-plane-release-name>-commander.<namespace>.svc.cluster.local.:8880"
      ) {
        id
      }
    }
    ```
  </Step>

  <Step title="Disable maintenance mode">
    Take the cluster out of maintenance mode to restore normal operations.

    ```graphql theme={null}
    mutation {
      uncordonCluster(clusterId: "<cluster-id>") {
        id
        isCordoned
        cordonedAt
      }
    }
    ```
  </Step>
</Steps>

### Verify the rollback

Confirm that the return to unified mode is complete:

* The control plane and data plane components are healthy.
* Airflow Deployments are accessible and you can update them.
* Airflow task logs and component logs are continuous across the migration with no gap. This confirms the shared Elasticsearch configuration.
* Metrics history is intact.
* The cluster is out of maintenance mode and Deployment mutations succeed again. Create or update a test Deployment to confirm.

## After migration

A same-cluster split satisfies the prerequisite for disaster recovery: the Deployments are now eligible for failover. Enabling failover is a separate procedure and requires a separate destination cluster; it isn't provided by this migration. See [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover).


# Configure a Kubernetes namespace pool for Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/namespace-pools

Manually create Kubernetes namespaces for new Deployments.

Every Deployment within your Astro Private Cloud installation requires an individual [Kubernetes namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/). You can configure a pool of pre-created namespaces to limit Astro Private Cloud access to these namespaces.

When you configure a pool of pre-created namespaces, Astronomer users are required to select a namespace from the pool whenever they create a new Deployment. Once the Deployment is created, Astro Private Cloud marks its corresponding namespace as unavailable. If a Deployment is deleted, its namespace is returned to the pool and is available.

## Benefits of pre-created namespaces

A pre-created namespace pool provides the following benefits:

* It limits the cluster-level permissions your organization needs to give to Astro Private Cloud. Astro Private Cloud requires permissions only for the individual namespaces you configure. Pre-created namespaces are recommended when your organization doesn't want to give Astro Private Cloud cluster-level permissions in a multi-tenant cluster.
* It can reduce costs and resource consumption. By default, Astro Private Cloud allows users to create Deployments until there is no more unreserved space in your cluster. If you use a pool, you can limit the number of active Deployments running at a time. This is especially important if you run other Elastic workloads on the same cluster where Astro Private Cloud runs and need to prevent users from accidentally claiming your entire pool of unallocated resources.

<Info>
  **Technical Details**

  When a user creates a new Deployment with the UI or CLI, Astro Private Cloud creates the necessary Airflow components and isolates them in a dedicated Kubernetes namespace. These Airflow components depend on Kubernetes resources, some of which are stored as secrets.

  To protect your Deployment Kubernetes secrets, Astronomer uses dedicated service accounts for parts of your Deployment that need to interact with external components. To enable this interaction, Astro Private Cloud needs extensive cluster-level permissions for all namespaces running in a cluster.

  In Kubernetes, you can grant service account permissions for an entire cluster, or you can grant permissions for existing namespaces. Astronomer uses cluster-level permissions because, by default, the amount of Deployments to manage is unknown. This level of permissions is appropriate when Astro Private Cloud runs in its own dedicated cluster, but it poses security risks when other applications share the cluster.

  For example, consider the deployment orchestrator service, which controls creating, updating, and deleting Deployments. By default, the deployment orchestrator has permissions to interact with secrets, roles, and service accounts for all applications in your cluster. The only way to mitigate this risk is by implementing pre-created namespaces.
</Info>

## Setup

To create a namespace pool you have the following options:

* Delegate the creation of each namespace, including roles and rolebindings, to the Astronomer Helm chart. This option is suitable for most organizations.
* Create each namespace manually, including roles and rolebindings. This option is suitable if you need to further restrict Astronomer Kubernetes resource permissions. However, using this methodology to limit permissions can prevent Deployments from functioning as expected.

<Note> If you have a separate control plane and data plane(s), following changes in the `values.yaml` file must be done for the data plane(s) and namespaces must be created in the data plane cluster(s). Each data plane can have their dedicated namespaces.</Note>

## Prerequisites

* [Helm](https://helm.sh/).
* [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) with access to the cluster hosting Astro Private Cloud.

## Option 1: Use the Astronomer Helm chart

1. Set the following configuration in your `values.yaml` file:

```yaml wrap theme={null}
global:
  # Set global.manageClusterScopedResources.enabled to false if you don't use ClusterRoles.
  # manageClusterScopedResources:
  #   enabled: false
  namespaceManagement:
    # Configure Vector to gather logs from all available namespaces.
    manualNamespaceNames:
      enabled: false
    # Prevent users from entering namespace names outside the configured pool.
    namespaceFreeFormEntry:
      enabled: false
    namespacePools:
      enabled: true
      # Create roles and rolebindings for the namespaces in the pool.
      createRbac: true
      namespaces:
        # Create the namespaces listed in names.
        create: true
        names:
          - <your-namespace-1>
          - <your-namespace-2>
```

2. Save the changes in your `values.yaml` file and update your Astro Private Cloud. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

Based on the namespace names that you specified, Astronomer creates the necessary namespaces and Kubernetes resources. These resources have permissions scoped appropriately for most use-cases.

Once you apply your configuration, you should be able to create new Deployments using a namespace from your pre-created namespace pool. You should also be able to see the namespaces you specified inside your cluster resources.

## Option 2: Manually create namespaces, roles, and rolebindings

Complete this setup if you want to further limit the namespace permissions that Astronomer provides by default.

### Step 1: Configure namespaces

For every namespace you want to add to a pool, you must create a [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/), [role](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#role-and-clusterrole), and [rolebinding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) for Astro Private Cloud to access the namespace with. The `rolebinding` must be scoped to the `astronomer-commander` service account and the `namespace` you are creating.

1. Create a new manifest file for each namespace you want to add to the pool. Replace `<your-namespace-name>` with the name of the namespace.

   ```yaml expandable wrap theme={null}
   apiVersion: v1
   kind: Namespace
   metadata:
     name: <your-namespace-name>
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: astronomer-commander
     namespace: <your-namespace-name>
   rules:
   - apiGroups: ["*"]
     resources: ["*"]
     verbs: ["list", "watch"]
   - apiGroups: [""]
     resources: ["configmaps"]
     verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
   - apiGroups: ["keda.k8s.io"]
     resources: ["scaledobjects"]
     verbs: ["get", "create", "delete"]
   - apiGroups: [""]
     resources: ["secrets"]
     verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
   - apiGroups: [""]
     resources: ["namespaces"]
     verbs: ["get", "list", "patch", "update", "watch"]
   - apiGroups: [""]
     resources: ["serviceaccounts"]
     verbs: ["create", "delete", "get", "patch"]
   - apiGroups: ["rbac.authorization.k8s.io"]
     resources: ["roles"]
     verbs: ["*"]
   - apiGroups: [""]
     resources: ["persistentvolumeclaims"]
     verbs: ["create", "delete", "deletecollection", "get", "list", "update", "watch", "patch"]
   - apiGroups: [""]
     resources: ["pods"]
     verbs: ["get", "list", "watch", "delete", "create", "patch"]
   - apiGroups: [""]
     resources: ["endpoints"]
     verbs: ["create", "delete", "get", "list", "update", "watch"]
   - apiGroups: [""]
     resources: ["limitranges"]
     verbs: ["create", "delete", "get", "list", "watch", "patch"]
   - apiGroups: [""]
     resources: ["nodes"]
     verbs: ["get", "list", "watch"]
   - apiGroups: [""]
     resources: ["nodes/proxy"]
     verbs: ["get"]
   - apiGroups: [""]
     resources: ["persistentvolumes"]
     verbs: ["create", "delete", "get", "list", "watch", "patch"]
   - apiGroups: [""]
     resources: ["replicationcontrollers"]
     verbs: ["list", "watch"]
   - apiGroups: [""]
     resources: ["resourcequotas"]
     verbs: ["create", "delete", "get", "list", "patch", "watch"]
   - apiGroups: [""]
     resources: ["services"]
     verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
   - apiGroups: ["apps"]
     resources: ["statefulsets"]
     verbs: ["create", "delete", "get", "list", "patch", "watch"]
   - apiGroups: ["apps"]
     resources: ["deployments"]
     verbs: ["create", "delete", "get", "patch","update"]
   - apiGroups: ["autoscaling"]
     resources: ["horizontalpodautoscalers"]
     verbs: ["list", "watch"]
   - apiGroups: ["batch"]
     resources: ["jobs"]
     verbs: ["list", "watch", "create", "delete", "get"]
   - apiGroups: ["batch"]
     resources: ["cronjobs"]
     verbs: ["create", "delete", "get", "list", "patch", "watch"]
   - apiGroups: ["extensions"]
     resources: ["daemonsets", "replicasets"]
     verbs: ["list", "watch"]
   - apiGroups: ["extensions"]
     resources: ["deployments"]
     verbs: ["create", "delete", "get", "list", "patch", "update", "watch"]
   - apiGroups: [""]
     resources: ["events"]
     verbs: ["create", "delete", "patch", "list"]
   - apiGroups: ["extensions"]
     resources: ["ingresses"]
     verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
   - apiGroups: ["extensions"]
     resources: ["ingresses/status"]
     verbs: ["update"]
   - apiGroups: ["networking.k8s.io"]
     resources: ["ingresses"]
     verbs: ["get", "create", "delete", "patch"]
   - apiGroups: ["networking.k8s.io"]
     resources: ["ingresses/status"]
     verbs: ["update"]
   - apiGroups: ["networking.k8s.io"]
     resources: ["networkpolicies"]
     verbs: ["create", "delete", "get", "patch"]
   - apiGroups: ["rbac.authorization.k8s.io"]
     resources: ["rolebindings"]
     verbs: ["create", "delete", "get", "patch"]
   - apiGroups: ["authentication.k8s.io"]
     resources: ["tokenreviews"]
     verbs: ["create", "delete"]
   - apiGroups: ["authorization.k8s.io"]
     resources: ["subjectaccessreviews"]
     verbs: ["create", "delete"]
   - apiGroups: ["policy"]
     resources: ["poddisruptionbudgets"]
     verbs: ["create", "delete", "get"]
   - apiGroups: [""]
     resources: ["pods/log"]
     verbs: ["get", "list"]
   - apiGroups: [""]
     resources: ["pods/exec"]
     verbs: ["get", "create"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: deployment-commander-rolebinding
     namespace: <your-namespace-name>
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: astronomer-commander # Should match name of Role
   subjects:
   - namespace: astronomer # Should match namespace where SA lives
     kind: ServiceAccount
     name: astronomer-commander # Should match service account name
   ```

2. Save this file and name it `<your-namespace-name>.yaml`.

3. For each namespace you will configure in `global.namespaceManagement.namespacePools.namespaces.names`, run `kubectl apply -f <your-namespace-name>.yaml`.

### Step 2: Configure a namespace pool in Astronomer

1. Set the following values in your `values.yaml` file, making sure to specify all of the namespaces you created in the `namespaces.names` object:

   ```yaml wrap theme={null}
   global:
     namespaceManagement:
       # Configure Vector to gather logs from all available namespaces.
       manualNamespaceNames:
         enabled: false
       # Prevent users from entering namespace names outside the configured pool.
       namespaceFreeFormEntry:
         enabled: false
       namespacePools:
         enabled: true
         # Don't create roles or rolebindings for the namespaces in the pool.
         createRbac: false
         namespaces:
           # Don't create namespaces because you created them manually.
           create: false
           names:
             - <your-namespace-1>
             - <your-namespace-2>
   ```

2. Save the changes in your `values.yaml` and update Astro Private Cloud. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

## Create Deployments in pre-created namespaces

After you enable the pre-created namespace pool, the UI shows the namespaces you registered as options when you create a new Deployment.

<Frame>
  <img alt="Kubernetes namespace option in the UI" />
</Frame>

When you create Deployments with the CLI, you are prompted to select one of the available namespaces for your new Deployment.

If no namespaces are available, the UI and CLI show an error message when you try to create a Deployment. Delete the Deployment associated with the namespace to return the namespace to the pool.

## Component RBAC for restricted mode

When you run namespace pools with cluster-scoped resources disabled — `global.clusterRoles: false` and `global.manageClusterScopedResources.enabled: false` — Astronomer creates no cluster-scoped RBAC, so you must provision the per-namespace RBAC that the other platform components need, in addition to the `astronomer-commander` Role from [Option 2](#option-2-manually-create-namespaces-roles-and-rolebindings). Create each Role and RoleBinding in the `astronomer` namespace and in each pool namespace, binding the component's service account in `astronomer`. Prometheus and NGINX still require cluster-scoped `ClusterRole`s.

<Note>
  These manifests are required when your cluster uses [External Secrets Operator security](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security) Mode 3, which runs namespace pools with cluster-scoped resources disabled.
</Note>

### kube-state-metrics

```yaml expandable wrap theme={null}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: astronomer-kube-state
  namespace: <your-namespace-name>
  labels:
    tier: monitoring
    component: kube-state
    release: astronomer
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["list", "watch"]
  - apiGroups: ["batch"]
    resources: ["cronjobs"]
    verbs: ["list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["limitranges"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["resourcequotas"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["list", "watch"]
  - apiGroups: ["apps"]
    resources: ["statefulsets"]
    verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: astronomer-kube-state
  namespace: <your-namespace-name>
  labels:
    tier: monitoring
    component: kube-state
    release: astronomer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: astronomer-kube-state
subjects:
  - kind: ServiceAccount
    name: astronomer-kube-state
    namespace: astronomer
```

### Houston DB bootstrapper

The Houston DB bootstrapper hook job runs in the platform (`astronomer`) namespace, so this Role is namespaced to `astronomer` only — you don't create it per pool namespace.

```yaml wrap theme={null}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: astronomer-houston-bootstrapper
  namespace: astronomer
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: astronomer-houston-bootstrapper-role
  namespace: astronomer
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "create", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: astronomer-houston-bootstrapper-role-binding
  namespace: astronomer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: astronomer-houston-bootstrapper-role
subjects:
  - kind: ServiceAccount
    name: astronomer-houston-bootstrapper
    namespace: astronomer
```

### Prometheus

Prometheus in data plane mode needs a cluster-scoped `ClusterRole` to reach its scrape targets.

```yaml wrap theme={null}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: astronomer-prometheus
  namespace: astronomer
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: astronomer-prometheus
rules:
  - apiGroups: [""]
    resources: ["services", "endpoints", "pods", "nodes", "nodes/proxy"]
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: astronomer-prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: astronomer-prometheus
subjects:
  - kind: ServiceAccount
    name: astronomer-prometheus
    namespace: astronomer
```

### NGINX

NGINX in restricted mode needs a cluster-scoped `ClusterRole` and `ClusterRoleBinding`:

```yaml expandable wrap theme={null}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: astronomer-nginx
  namespace: astronomer
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: astronomer-nginx
rules:
  - apiGroups: [""]
    resources: ["endpoints", "pods", "secrets", "configmaps", "nodes"]
    verbs: ["list", "watch", "get"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get", "list", "update", "watch"]
  - apiGroups: ["extensions"]
    resources: ["ingresses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses", "ingressclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["list", "watch", "get"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"]
  - apiGroups: ["extensions"]
    resources: ["ingresses/status"]
    verbs: ["update"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses/status"]
    verbs: ["update"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: astronomer-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: astronomer-nginx
subjects:
  - kind: ServiceAccount
    name: astronomer-nginx
    namespace: astronomer
```

NGINX also needs a namespaced `Role` in the `astronomer` namespace for its leader-election config:

```yaml expandable wrap theme={null}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: astronomer-nginx-config
  namespace: astronomer
rules:
  - apiGroups: [""]
    resources: ["configmaps", "namespaces", "pods", "secrets"]
    verbs: ["get"]
  - apiGroups: [""]
    resourceNames: ["ingress-controller-leader-astronomer-nginx"]
    resources: ["configmaps"]
    verbs: ["get", "update"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "create", "update"]
  - apiGroups: ["coordination.k8s.io"]
    resourceNames: ["ingress-controller-leader-astronomer-nginx"]
    resources: ["leases"]
    verbs: ["get", "update"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["create"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: astronomer-nginx-config
  namespace: astronomer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: astronomer-nginx-config
subjects:
  - kind: ServiceAccount
    name: astronomer-nginx
    namespace: astronomer
```

## Advanced settings

If your namespace configurations require more granularity, use the following settings in your `values.yaml` file.

Mixing global and advanced settings might result in unexpected behavior. If you use the advanced settings, Astronomer recommends that you set `global.namespaceManagement.namespacePools.enabled` to `false`.

<Info>
  The `astronomer.houston.config.deployments.namespaceManagement.*` settings shown below only seed the initial cluster configuration the first time a cluster is registered. To change `manualNamespaceNames` or `preCreatedNamespaces` for a cluster that's already running, use the **Configuration Override** section of the Astro UI or the `updateCluster` API instead of updating `values.yaml` and re-running Helm — see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).
</Info>

| Setting                                                                                  | Description                                                                                                                                                                                                                                                                                                                        |
| ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `global.namespaceManagement.manualNamespaceNames.enabled`                                | Expands the Vector log collector rule to look for Deployment component logs in all namespaces.                                                                                                                                                                                                                                     |
| `global.clusterRoles`                                                                    | When set to `false`, Astronomer doesn't create a ClusterRole for the deployment orchestrator.                                                                                                                                                                                                                                      |
| `global.manageClusterScopedResources.enabled`                                            | When set to `false`, Astronomer doesn't create or update cluster-scoped resources.                                                                                                                                                                                                                                                 |
| `global.rbac.enabled`                                                                    | When set to `false`, the platform doesn't create roles, rolebindings, or service accounts. You must define default roles for the default Kubernetes service account to continue with the platform installation. See [Bring your own Kubernetes service accounts](/docs/astro-private-cloud/v-2-x/byo-service-accounts) for setup steps. |
| `astronomer.houston.config.deployments.namespaceManagement.manualNamespaceNames.enabled` | When set to `true`, the Astro Private Cloud UI adds a dropdown field for namespace selection to the Deployment settings.                                                                                                                                                                                                           |
| `astronomer.houston.config.deployments.namespaceManagement.preCreatedNamespaces`         | Lists the namespaces that you manually created for the namespace pool.                                                                                                                                                                                                                                                             |
| `astronomer.commander.env`                                                               | Injects an environment variable for the deployment orchestrator that prevents namespace creation when a new Deployment starts. Set the name to `COMMANDER_MANUAL_NAMESPACE_NAMES` and the value to `"true"`.                                                                                                                       |

When using these settings, Astronomer recommends enabling [hard deletion](/docs/astro-private-cloud/v-2-x/configure-deployment#hard-delete-a-deployment) for Deployments.

In the following example `values.yaml` file, these settings are configured so that you don't configure namespace pools at a global level:

```yaml expandable wrap theme={null}
global:
  namespaceManagement:
    # Configure Vector to gather logs from all available namespaces.
    manualNamespaceNames:
      enabled: true
    namespacePools:
      enabled: false
  clusterRoles: false

astronomer:
  houston:
    config:
      deployments:
        namespaceManagement:
          # Enable manual namespace names.
          manualNamespaceNames:
            enabled: true
          # Specify pre-created namespace names.
          preCreatedNamespaces:
            - name: <namespace-name-1>
            - name: <namespace-name-2>
            - name: <namespace-name-x>

  commander:
    env:
      - name: "COMMANDER_MANUAL_NAMESPACE_NAMES"
        value: "true"
```

## Troubleshoot namespace pools

### My Deployment is in an unknown state

If a Deployment isn’t active, check the deployment orchestrator Pods to confirm they executed the Deployment commands successfully. When using a pre-created namespace pool with scoped roles, it’s possible that the `astronomer-commander` service account doesn't have the permissions necessary to perform a required action. When the deployment orchestrator succeeds, it shows the following notification:

```text wrap theme={null}
time="2021-07-21T16:30:23Z" level=info msg="releaseName <release-name>, chart astronomer-ee/airflow,     chartVersion 0.20.0, namespace <your-namespace-name>" function=InstallRelease package=helm
time="2021-07-21T16:30:23Z" level=info msg="CHART PATH: /home/commander/.cache/helm/repository/airflow-    0.20.0.tgz\n" function=InstallRelease package=helm
```

When the deployment orchestrator fails, it shows messages like the following:

```text wrap theme={null}
time="2022-02-17T22:52:40Z" level=error msg="serviceaccounts is forbidden: User \"system:serviceaccount:astronomer:astronomer-commander\" cannot create resource \"serviceaccounts\" in API group \"\" in the namespace <your-namespace>" function=InstallDeployment package=kubernetes
```

This error shows that the deployment orchestrator couldn’t create the service accounts in the pre-created namespaces, so you need to update the roles.

### My namespace isn't returning to the pool

If you're not using hard deletion, it can take several days for pre-created namespaces to become available after the associated Deployment is deleted. To enable hard deletes, see [Delete a Deployment](/docs/astro-private-cloud/v-2-x/configure-deployment#delete-a-deployment).

### My Deployments using NFS deploys stopped working

[NFS deploys](/docs/astro-private-cloud/v-2-x/deploy-nfs) don't work if you both use namespace pools and set `global.clusterRoles` to `false` in your `values.yaml` file.


# Network configuration
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/network-configuration

Configure networking, ingress, egress, and network policies for APC.

This guide covers network configuration for Astro Private Cloud (APC), including ingress, proxy settings, private networking, and network policies.

## Architecture

```mermaid actions={true} theme={null}
flowchart TD
    A["Internet"]
    B["Ingress (NGINX)"]
    C["APC API"]
    D["Airflow Deployments"]
    E["Registry"]

    A --> B
    B --> C
    B --> D
    B --> E
```

APC uses NGINX as its default ingress controller. For TLS certificate configuration, see [TLS certificate management](/docs/astro-private-cloud/v-2-x/tls-certificates). To use a pre-existing ingress controller, see [Use a third-party ingress controller](/docs/astro-private-cloud/v-2-x/third-party-ingress-controllers).

## DNS requirements

Before configuring ingress, create a wildcard DNS record pointing to the load balancer IP or hostname that NGINX will provision:

```text wrap theme={null}
*.<baseDomain>  →  <load-balancer-IP-or-hostname>
```

APC creates the following endpoints from `global.baseDomain`. All endpoints are served over HTTPS and covered by a single wildcard TLS certificate for `*.<baseDomain>`.

### Control plane endpoints

| Endpoint                    | Purpose                           | Accessible from |
| --------------------------- | --------------------------------- | --------------- |
| `app.<baseDomain>`          | Astro UI — primary user interface | Public          |
| `houston.<baseDomain>`      | APC API (GraphQL/REST)            | Public          |
| `registry.<baseDomain>`     | Container image registry          | Public          |
| `grafana.<baseDomain>`      | Grafana dashboards                | Public          |
| `prometheus.<baseDomain>`   | Prometheus UI                     | Public          |
| `alertmanager.<baseDomain>` | Alertmanager UI                   | Public          |

### Airflow Deployment endpoints

Each Airflow Deployment gets a path-based URL under the `deployments` subdomain:

| URL pattern                                               | Purpose                                 |
| --------------------------------------------------------- | --------------------------------------- |
| `https://deployments.<baseDomain>/<release-name>/airflow` | Airflow UI                              |
| `https://deployments.<baseDomain>/<release-name>/flower/` | Celery Flower UI (Celery executor only) |

### Data plane endpoints

In a split-plane deployment, both planes share the same `global.baseDomain`. The data plane uses `global.plane.domainPrefix` (a unique cluster identifier, for example `dp01`) to scope its endpoints under `<domainPrefix>.<baseDomain>`:

```yaml wrap theme={null}
# Control plane values.yaml
global:
  baseDomain: localtest.me
  plane:
    mode: control

# Data plane values.yaml
global:
  baseDomain: localtest.me          # Must match control plane
  plane:
    mode: data
    domainPrefix: dp01              # Unique identifier for this data plane cluster
```

<Warning>
  Both planes must use the same `global.baseDomain`. Auth tokens issued by the control plane the APC API are scoped to `.<baseDomain>`, so a mismatched base domain causes authentication failures across planes.
</Warning>

Create a separate wildcard DNS record for the data plane:

```text wrap theme={null}
*.<domainPrefix>.<baseDomain>  →  <data-plane-load-balancer>
```

The data plane creates the following endpoints:

| Endpoint                                                         | Purpose                                                                                                   |
| ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| `registry.<domainPrefix>.<baseDomain>`                           | Container image registry                                                                                  |
| `commander.<domainPrefix>.<baseDomain>`                          | Deployment orchestrator gRPC — used by the control plane to manage Airflow Deployments on this data plane |
| `<domainPrefix>.<baseDomain>/metadata`                           | Namespace label metadata served by the deployment orchestrator (read by the control plane)                |
| `prometheus.<domainPrefix>.<baseDomain>`                         | Prometheus UI                                                                                             |
| `prom-proxy.<domainPrefix>.<baseDomain>`                         | Prometheus federation auth proxy — the control plane scrapes this endpoint to aggregate metrics           |
| `elasticsearch.<domainPrefix>.<baseDomain>`                      | Elasticsearch                                                                                             |
| `deployments.<domainPrefix>.<baseDomain>/<release-name>/airflow` | Airflow UI for each Deployment on this data plane                                                         |

#### Metadata endpoint

The `/metadata` path on `<domainPrefix>.<baseDomain>` is served by the deployment orchestrator and returns a JSON document. The control plane reads this endpoint to fetch data plane details, including:

* Kubernetes version and health status
* The deployment orchestrator and registry URLs
* Data plane chart version, cloud provider, and region
* Namespace pools configuration and namespace labels
* Elasticsearch configuration
* External secret manager configuration

This endpoint is required for the data plane to function correctly with the control plane.

<Note>
  Retrieve your load balancer IP or hostname after installing APC. For the data plane, run `kubectl get svc -n <astronomer-namespace> -l component=dp-ingress-controller`. For the control plane, run `kubectl get svc -n <astronomer-namespace> -l component=cp-ingress-controller`. Create the wildcard DNS record pointing to that address before users try to access the platform.
</Note>

## Ingress configuration

### Load balancer

To expose the platform through a cloud load balancer, add the following to your `values.yaml`:

```yaml wrap theme={null}
nginx:
  serviceType: LoadBalancer
  loadBalancerIP: ""  # Optional: specific IP
  loadBalancerSourceRanges:
    - 10.0.0.0/8
    - 192.168.0.0/16
```

### Private load balancer

To provision an internal (non-public) load balancer, set `privateLoadBalancer: true`. APC automatically applies the appropriate annotation for AWS, GCP, and Azure:

```yaml wrap theme={null}
nginx:
  privateLoadBalancer: true
```

To add custom annotations to the ingress service beyond the auto-applied cloud annotations, use `ingressAnnotations`:

```yaml wrap theme={null}
nginx:
  privateLoadBalancer: true
  ingressAnnotations:
    service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-abc123"
```

### Custom ingress annotations

`global.extraAnnotations` applies annotations to all APC ingress resources — the platform UI, APC API, registry, Grafana, Prometheus, Alertmanager, and Elasticsearch ingresses. Its behavior depends on whether you're using the auth sidecar or the default NGINX ingress controller.

#### With `global.authSidecar.enabled: true`

All annotations under `global.extraAnnotations` are applied to every platform and Airflow ingress object, giving full control for environments using a bring-your-own ingress controller:

```yaml wrap theme={null}
global:
  authSidecar:
    enabled: true
  extraAnnotations:
    kubernetes.io/ingress.class: "my-ingress-controller"
```

For OpenShift route passthrough TLS:

```yaml wrap theme={null}
global:
  extraAnnotations:
    route.openshift.io/termination: "passthrough"  # Requires authSidecar.enabled: true
```

#### With default NGINX ingress

`global.extraAnnotations` is still respected, but you can't override annotations that APC manages. Use this to add custom annotations globally without conflicting with platform defaults.

The following annotations are protected and can't be overridden via `global.extraAnnotations`:

* `kubernetes.io/ingress.class`
* `nginx.ingress.kubernetes.io/custom-http-errors`

<Note>
  `global.extraAnnotations` applies to all ingress resources cluster-wide. `nginx.ingressAnnotations` applies only to the NGINX load balancer service. Use `global.extraAnnotations` for ingress routing behavior and `nginx.ingressAnnotations` for cloud load balancer configuration.
</Note>

### NodePort

```yaml wrap theme={null}
nginx:
  serviceType: NodePort
  httpNodePort: 30080
  httpsNodePort: 30443
```

### ClusterIP

<Note>
  **Astro Private Cloud 2.1**

  This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
</Note>

Some cluster policies require every Service, including ingress, to stay `ClusterIP` (no `LoadBalancer` or `NodePort`). Set:

```yaml wrap theme={null}
nginx:
  serviceType: ClusterIP
```

The default `serviceType` is `LoadBalancer`, since the ingress Service needs external exposure to receive traffic — APC doesn't change this default. If you set `serviceType: ClusterIP`, you're responsible for fronting the NGINX Service yourself, for example with your own cloud load balancer, a Kubernetes `Ingress` resource, or an external reverse proxy pointed at the Service.

### Source IP preservation

By default, `nginx.preserveSourceIP: false` sets `externalTrafficPolicy: Cluster` on the NGINX service, which distributes traffic evenly across all nodes but replaces the original client IP with the node IP through SNAT.

Set `preserveSourceIP: true` to use `externalTrafficPolicy: Local`, which preserves the original client IP in Airflow logs and Astro UI audit logs:

```yaml wrap theme={null}
nginx:
  preserveSourceIP: true
```

<Warning>
  With `externalTrafficPolicy: Local`, traffic is only routed to nodes running an NGINX pod. This can cause uneven load distribution if NGINX pods aren't spread across all nodes.
</Warning>

### NGINX request limits

Adjust these values for environments with slow upstreams, large Dag bundles, or long-running API calls:

```yaml wrap theme={null}
nginx:
  proxyConnectTimeout: 15    # seconds to establish an upstream connection
  proxyReadTimeout: 600      # seconds to wait for a response from upstream
  proxySendTimeout: 600      # seconds to transmit a request to upstream
  proxyBodySize: "1024m"     # maximum allowed request body size
```

The default `proxyBodySize` of `1024m` limits Dag bundle upload sizes. Increase this value if you see `413 Request Entity Too Large` errors.

## Connect to public endpoints

### Egress configuration

By default, Airflow Deployments can reach public endpoints. When network policies are enabled, you can restrict outbound traffic using Kubernetes NetworkPolicy resources. See [Network policies](#network-policies).

### Configure a proxy

APC's API reads proxy settings from environment variables on the APC API pod. Configure proxy settings using the `houston.env` Helm value in the `astronomer` subchart:

```yaml wrap theme={null}
astronomer:
  houston:
    env:
      - name: HTTPS_PROXY
        value: "http://proxy.example.com:8080"
      - name: HTTP_PROXY
        value: "http://proxy.example.com:8080"
      - name: NO_PROXY
        value: "localhost,127.0.0.1,kubernetes.default.svc,.svc.cluster.local,.<baseDomain>"
```

The APC API reads the proxy URL from the following environment variables, checked in order of precedence:

1. `HTTPS_PROXY` / `https_proxy` / `GLOBAL_AGENT_HTTPS_PROXY` — for HTTPS requests
2. `HTTP_PROXY` / `http_proxy` / `GLOBAL_AGENT_HTTP_PROXY` — for HTTP requests

`NO_PROXY` is respected automatically by the proxy agent. Include `.svc.cluster.local` to prevent internal Kubernetes service traffic from routing through the proxy, `kubernetes.default.svc` to ensure the deployment orchestrator can reach the Kubernetes API server without proxying, and `.<baseDomain>` if any platform components resolve each other using external DNS rather than cluster-internal DNS.

<Note>
  The `GLOBAL_AGENT_*` variants apply to HTTP libraries that use `global-agent` for proxy bootstrapping, in addition to the standard `HTTP_PROXY` and `HTTPS_PROXY` variables used by axios.
</Note>

<Note>
  In a proxy environment, apply the same environment variables to every platform pod that makes outbound calls. Set the same variables on the deployment orchestrator and Astro UI using `astronomer.commander.env` and `astronomer.astroUI.env` respectively.
</Note>

To verify that the APC API is using the proxy, check the APC API pod logs for the following line:

```text wrap theme={null}
Configuring Axios to use proxy: http://proxy.example.com:8080
```

#### Configure proxy for Airflow Deployments

Configure proxy settings for Airflow tasks by setting environment variables on each Airflow Deployment:

```yaml wrap theme={null}
env:
  - name: HTTP_PROXY
    value: "http://proxy.example.com:8080"
  - name: HTTPS_PROXY
    value: "http://proxy.example.com:8080"
  - name: NO_PROXY
    value: "localhost,127.0.0.1,.svc.cluster.local,.<baseDomain>"
```

## Private networking

### VPC/private subnet access

To resolve private hostnames from pods, add `hostAliases` to the relevant component. Configuration differs between platform components and Airflow Deployment components.

#### Platform components

The following platform components support `hostAliases`: Prometheus, registry, the APC API, APC Worker, and the deployment orchestrator:

```yaml expandable wrap theme={null}
prometheus:
  hostAliases:
    - ip: "10.0.0.100"
      hostnames:
        - "database.internal"
        - "api.internal"
astronomer:
  registry:
    hostAliases:
      - ip: "10.0.0.100"
        hostnames:
          - "database.internal"
          - "api.internal"
  commander:
    hostAliases:
      - ip: "10.0.0.100"
        hostnames:
          - "database.internal"
          - "api.internal"
  houston:
    hostAliases:
      - ip: "10.0.0.100"
        hostnames:
          - "database.internal"
          - "api.internal"
    worker:
      hostAliases:
        - ip: "10.0.0.100"
          hostnames:
            - "database.internal"
            - "api.internal"
```

#### Airflow Deployment components

The following Airflow components support `hostAliases`: scheduler, API server, webserver, triggerer, and workers:

```yaml wrap theme={null}
airflow:
  scheduler:
    hostAliases: []
  apiServer:
    hostAliases: []
  webserver:
    hostAliases: []
  triggerer:
    hostAliases: []
  workers:
    hostAliases: []
```

### VPN/direct connect

For AWS PrivateLink, Azure Private Link, or GCP Private Service Connect:

1. Configure the private endpoint in your cloud provider.
2. Create Kubernetes Service and Endpoints resources pointing to the private IP.
3. Reference the service name in your Airflow connections.

```yaml wrap theme={null}
apiVersion: v1
kind: Endpoints
metadata:
  name: private-database
subsets:
  - addresses:
      - ip: 10.0.0.100
    ports:
      - port: 5432
---
apiVersion: v1
kind: Service
metadata:
  name: private-database
spec:
  ports:
    - port: 5432
```

### Service mesh integration

#### Istio

APC is compatible with Istio, but APC doesn't configure or manage Istio. You must install and configure Istio on your cluster yourself before enabling this feature in APC.

To enable Istio compatibility mode:

```yaml wrap theme={null}
global:
  istio:
    enabled: true
    rootNamespace: "istio-config"  # Must match your Istio installation's root namespace
```

The `rootNamespace` must match the root config namespace of your Istio installation. In most Istio installations this is `istio-system` or `istio-config`. If this value is incorrect, the default Sidecar egress policy won't apply to Airflow Deployment namespaces.

<Warning>
  APC uses the `networking.istio.io/v1alpha3` API for Sidecar resources. This API is deprecated in Istio 1.22 and later. Clusters running Istio 1.22+ will see deprecation warnings in the Istio control plane logs.
</Warning>

**What APC creates when Istio is enabled**

APC creates three Istio `Sidecar` resources to control egress traffic:

| Resource                                        | Namespace          | Applies to                                  | Allowed egress                                           |
| ----------------------------------------------- | ------------------ | ------------------------------------------- | -------------------------------------------------------- |
| `default-sidecar-config`                        | `rootNamespace`    | All Airflow Deployment namespaces (default) | Same namespace, `istio-system`, Elasticsearch, SQL proxy |
| `<release-namespace>-sidecar-config`            | Platform namespace | All platform pods                           | Same namespace, `istio-system`, `kube-system`            |
| `<release-namespace>-prometheus-sidecar-config` | Platform namespace | Prometheus only                             | Everywhere (`*/*`) — required for metrics scraping       |

The `default-sidecar-config` resource in the `rootNamespace` acts as a cluster-wide default for any namespace that doesn't have its own Sidecar resource, which covers all Airflow Deployment namespaces.

**Namespace labeling for sidecar injection**

APC doesn't automatically label namespaces for Istio sidecar injection. You must label the platform namespace and any Airflow Deployment namespaces yourself for injection to take effect on long-running pods (the APC API, the deployment orchestrator, Astro UI, Registry):

```bash wrap theme={null}
kubectl label namespace <astronomer-namespace> istio-injection=enabled
kubectl label namespace <airflow-namespace> istio-injection=enabled
```

If you use namespace pools, apply the label automatically to all Airflow namespaces using `global.namespaceLabels`:

```yaml wrap theme={null}
global:
  namespaceLabels:
    istio-injection: "enabled"
```

**Airflow Deployment egress under Istio**

The `default-sidecar-config` restricts Airflow pod egress to the same namespace, `istio-system`, Elasticsearch, and SQL proxy only. Dag tasks that connect to external databases or APIs through the service mesh will be blocked.

To allow additional egress for a specific Airflow namespace, create a custom `Sidecar` resource in that namespace that overrides the default:

```yaml wrap theme={null}
apiVersion: networking.istio.io/v1alpha3
kind: Sidecar
metadata:
  name: allow-external-db
  namespace: <airflow-namespace>
spec:
  egress:
  - hosts:
    - "./*"
    - "istio-system/*"
    - "*/my-external-database.example.com"
```

<Note>
  Istio Sidecar resources only restrict traffic to services registered in the mesh. Egress to external public IPs not registered as Kubernetes services remains reachable by default unless you set `outboundTrafficPolicy: REGISTRY_ONLY` on the mesh.
</Note>

**Sidecar injection behavior**

APC disables Istio sidecar injection on pods that shouldn't have a proxy:

* **NGINX** (control plane and data plane) — injection disabled; also has `traffic.sidecar.istio.io/includeInboundPorts: ""` set unconditionally on all NGINX pods to prevent accidental inbound port interception even if Istio is installed but not yet enabled in values
* **The APC API cronjobs and Helm hooks** — injection disabled; short-lived batch jobs don't need a sidecar proxy.
* **Config syncer, deployment orchestrator JWKS hook, NATS JetStream job** — injection disabled for the same reason

Prometheus is the only platform component that receives the Istio sidecar, with explicit resource allocation:

```yaml wrap theme={null}
sidecar.istio.io/proxyCPU: "500m"
sidecar.istio.io/proxyMemory: "400Mi"
```

**What APC doesn't configure**

APC doesn't create VirtualServices, DestinationRules, Gateways, AuthorizationPolicy, or PeerAuthentication resources. Configure mTLS mode and traffic management policies directly in Istio. See the [Istio mTLS documentation](https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/) for details.

## Network policies

### Enable network policies

```yaml wrap theme={null}
global:
  networkPolicy:
    enabled: true
  defaultDenyNetworkPolicy: true  # default: true
```

When both `networkPolicy.enabled` and `defaultDenyNetworkPolicy` are `true`, APC creates a NetworkPolicy named `<release-name>-default-deny-ingress` in the platform namespace that denies all inbound traffic by default. Per-component network policies then selectively allow the required connections.

<Warning>
  The default-deny policy covers ingress only. Egress from platform and Airflow pods is unrestricted by default. Apply custom NetworkPolicy resources to restrict outbound traffic.
</Warning>

Set `defaultDenyNetworkPolicy: false` if your cluster already has a deny-all policy in place, or if other services running in the platform namespace would be disrupted.

### Default policies

When enabled, APC creates network policies for:

* Platform component communication
* Airflow Deployment isolation
* Ingress traffic

APC's API network policy allows ingress from any pod labeled `tier: airflow` across all namespaces, using `namespaceSelector: {}`. Isolation between Airflow Deployments is enforced through Kubernetes RBAC, not network policies.

### Custom policies

Apply custom Kubernetes NetworkPolicy resources to extend or restrict traffic beyond the default APC-managed policies.

#### Allow specific egress

```yaml wrap theme={null}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-database
  namespace: deployment-namespace
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 10.0.0.0/24
      ports:
        - protocol: TCP
          port: 5432
```

#### Restrict cross-namespace

```yaml wrap theme={null}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-cross-namespace
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector: {}
```

### Namespace labeling

<Note>
  Any namespace management requires cluster-scoped permissions. These settings only take effect if APC is granted a cluster role to manage namespace resources.
</Note>

#### global.networkNSLabels

Set `global.networkNSLabels.enabled: true` to label the platform namespace with `platform: <release-name>`. This also adds namespace label selectors to network policies, restricting traffic to pods in namespaces that carry the matching label.

```yaml wrap theme={null}
global:
  networkNSLabels:
    enabled: true
```

For example, if APC is installed with release name `apc-private-cloud` in namespace `astro-cloud`, enabling this feature patches the namespace with `platform=apc-private-cloud`, allowing you to filter it with:

```bash wrap theme={null}
kubectl get ns -l platform=apc-private-cloud
```

#### global.namespaceLabels

`global.namespaceLabels` applies custom labels to Airflow Deployment namespaces. Its behavior depends on who owns namespace provisioning:

**With customer-managed namespaces (namespace pools)**: When customers create and own their namespaces, APC no longer has access to patch them and skips namespace label updates. Customers are responsible for applying the required labels to their own namespaces.

**With Astronomer-owned provisioning (cluster role)**: APC manages namespace creation and applies both its own required labels and any labels specified under `global.namespaceLabels`:

```yaml wrap theme={null}
global:
  networkNSLabels:
    enabled: true
  namespaceLabels:
    platform: "<release-name>"
```

## TLS scope

`global.ssl.enabled` controls TLS for database connections only — the connection from platform components to PostgreSQL. It doesn't enable mTLS between platform services or TLS on internal cluster communication.

| Setting                    | Scope                                                          |
| -------------------------- | -------------------------------------------------------------- |
| `global.ssl.enabled: true` | Database connection TLS (platform → PostgreSQL)                |
| Ingress TLS                | Handled by `global.tlsSecret` and the NGINX ingress controller |

Don't rely on `global.ssl.enabled` to secure inter-service traffic. APC doesn't configure Istio — if you need service mesh mTLS, install and configure Istio directly in your cluster.

## DNS and service discovery

### Internal DNS

Airflow Deployments use Kubernetes DNS for internal service resolution:

* Services: `<service>.<namespace>.svc.cluster.local`
* Pods: `<pod-ip>.<namespace>.pod.cluster.local`

## Firewall requirements

### Ingress ports

| Port | Protocol | Purpose           |
| ---- | -------- | ----------------- |
| 443  | HTTPS    | Platform UI/API   |
| 80   | HTTP     | Redirect to HTTPS |

### Egress requirements

| Destination        | Port    | Purpose         |
| ------------------ | ------- | --------------- |
| Container registry | 443     | Image pulls     |
| PostgreSQL         | 5432    | Database        |
| Redis              | 6379    | Celery broker   |
| Your data sources  | Various | Dag connections |

### Control plane and data plane connectivity

In a split-plane deployment, the control plane reaches the data plane over HTTPS, and the data plane reaches the control plane APC API:

| Source        | Destination                              | Port | Purpose                                                      |
| ------------- | ---------------------------------------- | ---- | ------------------------------------------------------------ |
| Control plane | `commander.<domainPrefix>.<baseDomain>`  | 443  | Control plane manages Airflow Deployments on the data plane  |
| Control plane | `prom-proxy.<domainPrefix>.<baseDomain>` | 443  | Control plane scrapes metrics from the data plane            |
| Data plane    | `houston.<baseDomain>`                   | 443  | Data plane registers and communicates with the control plane |

Ensure firewall rules between the two cluster load balancers permit outbound HTTPS on port 443 in both directions.

### Internal cluster ports

| Port  | Purpose                                                         |
| ----- | --------------------------------------------------------------- |
| 8871  | APC API                                                         |
| 8880  | Deployment orchestrator (HTTP)                                  |
| 50051 | Deployment orchestrator (gRPC)                                  |
| 443   | Kubernetes API server (required by the deployment orchestrator) |
| 8080  | Airflow webserver                                               |
| 5555  | Flower                                                          |
| 6379  | Redis                                                           |
| 9200  | Elasticsearch (HTTP)                                            |
| 9300  | Elasticsearch (transport)                                       |
| 4222  | NATS (messaging)                                                |
| 5000  | Container registry                                              |

## Troubleshoot

### Connectivity issues

```bash wrap theme={null}
# Test DNS resolution
kubectl exec -n <namespace> <pod> -- nslookup api.example.com

# Test connectivity
kubectl exec -n <namespace> <pod> -- curl -v https://api.example.com

# Check network policies
kubectl get networkpolicies -n <namespace>
```

### Proxy issues

```bash wrap theme={null}
# Verify proxy settings on the Houston pod
kubectl exec -n <astronomer-namespace> <houston-pod> -- env | grep -i proxy

# Verify proxy settings on an Airflow Deployment pod
kubectl exec -n <deployment-namespace> <pod> -- env | grep -i proxy

# Test connectivity through proxy
kubectl exec -n <namespace> <pod> -- curl -v --proxy http://proxy:8080 https://api.example.com
```

### DNS issues

```bash wrap theme={null}
# Check CoreDNS
kubectl logs -n kube-system -l k8s-app=kube-dns

# Test internal DNS resolution
kubectl exec -n <namespace> <pod> -- nslookup houston.<astronomer-namespace>.svc.cluster.local
```

## Best practices

* Enable network policies to enforce least-privilege access.
* Include `.svc.cluster.local` and `kubernetes.default.svc` in `NO_PROXY` to prevent internal Kubernetes traffic from routing through a proxy.
* Use private load balancers for internal-only access.
* Document all egress requirements for firewall teams.
* Test connectivity before deploying Dags.
* Use Kubernetes services for internal connections.


# Overprovision Airflow components
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/overprovision-components

Configure cluster-level overprovisioning so Airflow component pods request less than their limits for better bin-packing and scaling.

Overprovisioning in Astro Private Cloud (APC) is a *cluster-level* setting that sets resource *requests* for selected Airflow components to a fraction of their *limits*. This allows the scheduler to pack more pods on nodes (bin-packing) while still allowing bursts up to the limit when needed.

## How overprovisioning works

You configure three values in your cluster's deployment config:

* `overProvisioningFactorCPU`: Fraction of the CPU limit used as the CPU request (0 \< value ≤ 1).
* `overProvisioningFactorMem`: Fraction of the memory limit used as the memory request (0 \< value ≤ 1).
* `overProvisioningComponents`: List of component names that receive the overprovisioning factor.

For each component in the list, requests are set as:

* `requests.cpu = limits.cpu × overProvisioningFactorCPU`
* `requests.memory = limits.memory × overProvisioningFactorMem`

For example, if a component has `limits: { cpu: "2000m", memory: "4Gi" }` and both factors are `0.5`, then `requests` become `{ cpu: "1000m", memory: "2Gi" }`. Default factors of `1` mean no change (requests equal limits).

## Configure overprovisioning

Set these keys in your cluster config (for example, in the cluster's deployment config or the values used when registering the cluster). Values must be greater than 0 and less than or equal to 1.

```yaml wrap theme={null}
overProvisioningFactorCPU: 0.5
overProvisioningFactorMem: 0.5
overProvisioningComponents:
  - scheduler
  - webserver
  - apiServer
  - workers
  - triggerer
  - flower
  - pgbouncer
  - statsd
  - dagProcessor
```

Only components that have resources defined in the deployment config are affected. Components not listed in `overProvisioningComponents` keep their existing request/limit values.

## Supported components

You can include any of these in `overProvisioningComponents`:

| Component      | Description        |
| -------------- | ------------------ |
| `scheduler`    | Airflow scheduler  |
| `apiServer`    | Airflow API server |
| `webserver`    | Airflow webserver  |
| `workers`      | Celery workers     |
| `triggerer`    | Airflow triggerer  |
| `flower`       | Flower (Celery UI) |
| `pgbouncer`    | PgBouncer          |
| `statsd`       | StatsD exporter    |
| `dagProcessor` | Dag processor      |

## Choose factor values

* `1` (default): No overprovisioning; requests equal limits. Use when you want predictable capacity and no bin-packing.
* `0.5`: Requests are half of limits. Common choice for better bin-packing while keeping headroom.
* **Lower values** (for example, `0.25`): More aggressive bin-packing; ensure your workloads can tolerate less guaranteed CPU/memory.

Start with `0.5` for CPU and memory and adjust based on utilization and scheduling behavior.

## Best practices

* **Apply factors per cluster**: Overprovisioning applies at the cluster level; all deployments on the cluster use the same factors for the listed components.
* **Include only components that have resources**: Only components with `resources.limits` (and optionally `resources.requests`) in the deployment config receive modifications; listing others has no effect.
* **Monitor utilization**: After enabling, watch pod scheduling and resource usage to confirm the factors match your workload and node capacity.


# Override data plane cluster configurations
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/override-data-plane-cluster

Override and update Deployment configurations for a data plane cluster.

Cluster overrides let you customize Deployment configuration for one data plane cluster. They don't change platform-level control plane settings.

Use [Configure Astro Private Cloud](/docs/astro-private-cloud/v-2-x/configure-astro-private-cloud) if you need to decide whether a setting belongs in platform config, a cluster override, a Workspace override, or a Deployment override.

Astro Private Cloud (APC) has two types of Deployment configurations that you can set at the data plane cluster level.

* **Default configuration**: The base, system-provided configuration. You can override default values, but you can't delete default keys.
* **Custom configuration**: Your own additional keys and values. You can add, modify, or delete these keys.

## Prerequisites

* An APC control plane is installed and reachable.
* You have System Admin permissions in the APC UI or access to the APC API.
* The data plane cluster is healthy and accessible from the control plane.

## Update your cluster

You can define cluster configurations when you first [register a cluster](/docs/astro-private-cloud/v-2-x/register-data-plane) or later by updating the cluster through the UI or APC API.

* **Registration**: The form includes an optional **Cluster Override** field. You can supply initial overrides when registering.
* **Edit**: The cluster page contains your overall cluster information and a **Deployment Configuration** section.
  * **Cluster Information**: View and edit the cluster name.
  * **Current Configuration**: Read-only view of the default configurations.
  * **Configuration Override**: Editable area where you provide override values for base keys and/or add custom keys.
  * After you add overrides, the **Current Configuration** shows a read-only, git-style diff of added/modified values.
* **APC API**: If you automate cluster management, use the APC API to update cluster metadata and connection secrets. Refer to your internal API client or see platform API docs.

## Update cluster information

Only the **Cluster name** is editable as a cluster property in the **Cluster Information** section. You must use **Configuration Override** to make changes to Deployment configurations by overriding the base Deployment configuration.

## Override base configuration

Add your own keys and values to tailor behavior per cluster. You can't delete default keys, but you can override their values to make custom configurations.

<Steps>
  <Step title="Select cluster">
    In the **Clusters** page, select the cluster where you want to override a configuration.
  </Step>

  <Step title="Find configuration to review">
    Search for the configuration that you want to edit in the **Current Configuration**.

    <Tip>Use `ctrl+F` or `cmd+F` on Mac to search for the config.</Tip>
  </Step>

  <Step title="Edit configuration override">
    Click **Edit** to unlock **Configuration Override**. You can now make override edits.
  </Step>

  <Step title="Add new key and value">
    Add your new key and value to the **Configuration Override**.

    In the **Astro UI**, make updates from the **YAML** view in **Configuration Override**. The UI sends the correct values to the APC API when you save, including when you clear an override for a key. Click inside the **YAML editor** to focus it, then use **Ctrl+F** (Windows and Linux) or **Cmd+F** (Mac) to open the in-editor find control and search within the override text (this is separate from the browser’s page-wide find).

    <Frame>
      <img alt="Configuration Override YAML editor focused, with the in-editor find control used to search the YAML." />
    </Frame>

    * **APC API `updateCluster` (GraphQL)**: To remove a key from the stored `deployments` override, pass a `deploymentsConfigOverride` object and set that key's value to the string `"DELETE_KEY"`. The merge rules are the same as for workspace- and deployment-level overrides; see [Config governance](/docs/astro-private-cloud/v-2-x/config-governance).
    * **Base configuration keys** can only have their *values* overridden, not removed from the underlying defaults, and you can't use `"DELETE_KEY"` on paths that are *required* in the platform's default `deployments` object.
    * **Schema validation**: When the APC API has `strictSchemaCheck.enabled` set to `true` (from `astronomer.houston.strictSchemaCheck` in the control plane's `values.yaml`, `true` by default in 2.0), unknown keys or bad types in `deployments` overrides are rejected. If you get validation errors for acceptable `helm` or other allow-listed subtrees, see [Disable the strict schema check](/docs/astro-private-cloud/v-2-x/config-governance#delete_key-and-strict-schema-validation) for the pattern to relax the check, or adjust overrides to match the [config governance](/docs/astro-private-cloud/v-2-x/config-governance) schema.

    <Warning>
      Carefully check your configuration updates before you update your cluster, as these configuration changes can impact all cluster users.
    </Warning>
  </Step>

  <Step title="Apply changes">
    Click **Update Cluster** to apply your changes.

    <Note>
      Deployment config updates aren't applied until each Deployment is individually updated.
    </Note>
  </Step>

  <Step title="Verify changes">
    * Confirm cluster status is **Healthy** on the **Clusters** page in the Astro UI.
    * For Airflow Deployments in your updated cluster, trigger a small change and confirm the deployment orchestrator applies it as expected.
    * In Prometheus on the control plane, verify the cluster appears in federated targets.
  </Step>
</Steps>

## Additional notes

* Cluster overrides apply to `deployments.*` values for the selected data plane cluster.
* In APC 2.x, Workspace and Deployment overrides can further override eligible `deployments.*` values after the cluster layer.
* Settings outside `deployments.*` stay in `values.yaml`, typically under `astronomer.houston.config`, and require a Helm upgrade.
* Use [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config) for platform-level values and [Configure Astro Private Cloud](/docs/astro-private-cloud/v-2-x/configure-astro-private-cloud) for the full hierarchy.

## Best practices

* Keep overrides minimal. Prefer **Base Config** unless a cluster truly deviates.
* Use consistent naming and comments for custom keys.
* Review the diff in **Current Configuration** to validate the final effective settings before you **Update Cluster**.


# Data plane clusters in Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/overview-data-plane-cluster

Understand what a data plane cluster is in Astro Private Cloud, why it matters, and how to register, update, and deregister data plane clusters.

In Astro Private Cloud (APC), a *cluster* is a [data plane](/docs/astro-private-cloud/v-2-x/data-plane-architecture) Kubernetes cluster, that runs Airflow Deployments and their runtime components. You can attach multiple data plane clusters to a single [control plane](/docs/astro-private-cloud/v-2-x/control-plane-architecture), which allows you to centralize management of users, workspaces, configuration, Astro UI/API, metrics, and orchestration services.

APC can run in *split mode*, where the data plane and control plane use separate Kubernetes clusters. Or, APC can also run in *unified mode*, where a single Kubernetes cluster performs both roles.

Using specific Kubernetes environments for control and data plane functions provides the following performance and administration benefits:

* **Isolation and compliance**: Keep teams, environments, or regions separate to meet security and governance requirements.
* **Performance and scale**: Allocate resources independently and scale data planes without affecting control-plane availability.
* **Cost and ownership**: Attribute spend to the teams or business units that own each data plane.
* **Reliability boundaries**: Limit the affected area of failures and define clear service level objectives (SLO) per cluster.
* **Network boundaries and residency**: Keep data within specific networks or geographies.

## Example

A company operates a single APC control plane and two data plane clusters at the same organization:

* **Finance data plane cluster**: Runs Airflow for data reconciliation and reporting with restricted network access and stricter compliance controls.
* **Engineering data plane cluster**: Runs Airflow for product analytics and machine learning feature pipelines with more flexible networking.

Both data planes are registered to the same control plane. Platform administrators manage users, Workspaces, and Deployments centrally, while teams operate their Dags within their isolated data planes.

## Manage clusters

* [Register a cluster](/docs/astro-private-cloud/v-2-x/register-data-plane): Register a data plane cluster.
* [Override cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster): Update data plane cluster details, like name, labels, connection settings, or credentials, and override cluster default Deployment configurations.
* [Deregister a cluster](/docs/astro-private-cloud/v-2-x/deregister-data-plane): Deregister a data plane cluster you no longer need.
* [Troubleshoot registration](/docs/astro-private-cloud/v-2-x/troubleshoot-data-plane-registration): Diagnose and resolve errors during data plane registration.

If you are starting fresh, [set up your control plane](/docs/astro-private-cloud/v-2-x/install-control-plane) first, then register one or more data plane clusters before creating Airflow Deployments.


# Per-deployment migration reference
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/per-deployment-migration-reference

Reference for per-deployment migration on Astro Private Cloud: configuration settings, cluster and mission states, skip reasons, the GraphQL surface, and permissions.

This page is the reference companion to [Configure per-deployment migration](/docs/astro-private-cloud/v-2-x/configure-per-deployment-migration) and [Manage and observe per-deployment migration](/docs/astro-private-cloud/v-2-x/manage-per-deployment-migration). It documents the configuration settings, cluster failover states, mission and flight states, migration errors and skips, the GraphQL surface, and the permissions for per-deployment migration on Astro Private Cloud (APC) 2.1 and later.

## Configuration settings

There is no per-deployment migration feature flag, no separate mutation to enable, and no per-cluster opt-in beyond `failoverEnabled`. The settings that govern it are the data plane failover settings plus the new region model.

| Setting                                              | Default                                                                                     | Set to                                                     | Effect                                                                                                                                                                                                                                                                                                                                                |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `global.dataPlaneFailover.enabled`                   | `false`                                                                                     | `true` on every participating control plane and data plane | Master switch for data plane failover and per-deployment migration. Requires split mode — not supported in unified mode. On a data plane, enables the data plane execution components (Pilot and Flightdeck) and the data plane internal API's `StartFlight` remote procedure call. On a control plane, enables Navigator and the APC API dispatcher. |
| `global.dataPlaneFailover.externalSecretManagerName` | Unset                                                                                       | The name of your `ClusterSecretStore`                      | Tells the data plane internal API which External Secrets Operator (ESO) `ClusterSecretStore` to use when pushing and pulling Deployment secrets. Required on data planes.                                                                                                                                                                             |
| `external-secrets.enabled`                           | `false`                                                                                     | `true` on every data plane                                 | Installs the ESO custom resource definitions that data plane failover depends on.                                                                                                                                                                                                                                                                     |
| `Region.name`                                        | —                                                                                           | Unique, human-meaningful                                   | Region identity. Compared by ID, not by name — the name is for you. Must be unique.                                                                                                                                                                                                                                                                   |
| `Region.cloudProvider`                               | —                                                                                           | `aws`, `gcp`, `azure`, or `local`                          | Every region has exactly one cloud provider. A cluster can only be assigned to a region whose provider matches the cluster's own, so region equality implies provider equality.                                                                                                                                                                       |
| `Cluster.regionId`                                   | Backfilled to `unset`                                                                       | The region the cluster physically lives in                 | The value the migration gate compares. Not enforced non-null, but migration is impossible while it points at `unset`.                                                                                                                                                                                                                                 |
| `Cluster.cloudProvider`                              | Inherited from the assigned region at registration; from data plane metadata when no region | Editable on the cluster                                    | Editable on the cluster edit form, and no longer overwritten by the data plane metadata reconcile. When you assign or change a region on an existing cluster, it is checked case-insensitively against `region.cloudProvider`, and skipped if the cluster's provider is empty.                                                                        |

## Cluster failover states

The cluster detail page shows a **Failover** field, which replaces the older binary "Failover Enabled." Reading it is how you tell whether a cluster can participate in migration:

| Status          | Meaning                                                                                                                                                       | Can it be a migration destination? |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
| Not Capable     | The data plane doesn't have `dataPlaneFailover.enabled` set.                                                                                                  | No                                 |
| Pending Upgrade | Failover-capable, but Deployments still need the one-time failover upgrade, and no upgrade missions are running.                                              | No                                 |
| Upgrading (N/M) | At least one upgrade mission is in flight. N of M Deployments upgraded.                                                                                       | No                                 |
| Enabled         | All eligible Deployments are upgraded and the cluster passes the eligibility check.                                                                           | Yes                                |
| Issues Detected | The cluster has upgraded Deployments that are individually ineligible, for example a Deployment type change or ESO config drift. Hover for the affected list. | No                                 |

The underlying API fields are `Cluster.failoverEnabled` (the control-plane-effective state) and `Cluster.failoverUpgradeStatus` (`not_started`, `in_progress`, `complete`, or `null`).

<Note>
  `Cluster.failoverEnabled` starts from what the data plane reports, but the control plane force-downgrades it to `false` whenever an eligibility check fails. It reflects reality, not intent. If you enabled the Helm value and the field is still `false`, something in the eligibility check is failing, and the **Failover** status tells you which.
</Note>

## Failover-readiness shield states

The Deployments list shows a failover-readiness shield next to each Deployment name, on failover-capable clusters only. It uses one outline icon, distinguished by color, plus an exclamation badge for **Attention**. The API field behind it is `Deployment.failoverReadiness`.

| State           | Color     | `failoverReadiness` value | Meaning                                                               |
| --------------- | --------- | ------------------------- | --------------------------------------------------------------------- |
| Not applicable  | No shield | `not_capable`             | Failover-incapable cluster.                                           |
| Pending upgrade | Grey      | `pending_upgrade`         | Failover-capable but not yet upgraded for failover.                   |
| In progress     | Amber     | `in_progress`             | An upgrade mission is in flight.                                      |
| Ready           | Green     | `ready`                   | Upgraded and currently failover-eligible.                             |
| Attention       | Red       | `attention`               | The upgrade failed, or the Deployment is upgraded but now ineligible. |

<Warning>
  The shield tracks the one-time failover-upgrade lifecycle, not migrations. A Deployment that is mid-migration keeps a green **Ready** shield the entire time. Use the **Mission Status** column, or `Deployment.activeMission`, to see an in-flight migration.
</Warning>

## Mission states

Selecting N Deployments creates N independent missions, one per Deployment. Each moves through the following states:

| State             | Meaning                                                                                                                                                           |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CREATED`         | Deployment claimed and the mission row exists. Flights may not exist yet.                                                                                         |
| `PLANNING`        | Navigator has claimed the mission for planning.                                                                                                                   |
| `PLANNED`         | The required control plane flights are created and initialized.                                                                                                   |
| `IN_PROGRESS`     | At least one flight has been dispatched to a data plane.                                                                                                          |
| `CLEANUP_PENDING` | The Hyperjump flight succeeded, but source-side cleanup is still pending or failed. A per-deployment migration never reaches this state — see the following note. |
| `COMPLETED`       | Fully complete: cutover done and cleanup done.                                                                                                                    |
| `FAILED`          | The mission failed. See [Recover a failed migration](/docs/astro-private-cloud/v-2-x/manage-per-deployment-migration#recover-a-failed-migration).                      |

<Note>
  `CLEANUP_PENDING` doesn't occur in per-deployment migration. Migrations always run in controlled mode, so the Scavenger flight (source cleanup) must succeed before the Hyperjump flight is dispatched. Once the Hyperjump succeeds, the source is already cleaned and the mission goes straight to `COMPLETED`. If the Scavenger fails, the Hyperjump never runs and the mission is `FAILED`. This state arises only in a forced cluster failover, where the Hyperjump can cut over while source-side cleanup is still deferred or has failed — and even there it isn't an error.
</Note>

## Flight states

Each flight within a mission has its own state:

| State                | Meaning                                                                                                    |
| -------------------- | ---------------------------------------------------------------------------------------------------------- |
| `READY_TO_DISPATCH`  | Queued, waiting for the dispatcher.                                                                        |
| `HOLDING`            | Held — a dependency isn't satisfied yet, for example the Hyperjump flight waiting on the Scavenger flight. |
| `DISPATCHING`        | The dispatcher holds a claim and is attempting `StartFlight`.                                              |
| `DISPATCHED`         | The data plane internal API acknowledged, and the data plane has durably recorded the flight.              |
| `AWAITING_DP_HEALTH` | The data plane is unhealthy or unreachable. Dispatch is paused and resumes automatically when it recovers. |
| `FAILED_DISPATCH`    | Dispatch failed after the configured retries.                                                              |
| `SUCCEEDED`          | The flight completed successfully.                                                                         |
| `FAILED`             | The flight failed.                                                                                         |
| `CANCELED`           | The flight was canceled.                                                                                   |

`AWAITING_DP_HEALTH` is a wait, not a failure: the circuit breaker parked the flight because the data plane was unreachable. Fix the data plane and it resumes on its own.

## Migration errors and skips

`migrateDeployments` has two failure modes. Whole-mutation errors are about the destination and reject the entire call, so nothing moves. Per-deployment skips pass over one Deployment and let the rest proceed.

### Whole-mutation errors

These are evaluated in order, and the first match wins. Because cordon and in-flight-failover are checked before authorization, an unauthorized caller may see a health or cordon error first.

| Error                              | Cause                                                                          | Fix                                                                                                                                            |
| ---------------------------------- | ------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `DESTINATION_NOT_FOUND`            | The destination cluster ID doesn't exist, or the cluster is deleted.           | Check the ID.                                                                                                                                  |
| `DESTINATION_NOT_FAILOVER_ENABLED` | The destination's `failoverEnabled` is `false`.                                | Check the **Failover** field on the cluster detail page and resolve what is holding it back, usually Deployments pending the failover upgrade. |
| `DESTINATION_CLUSTER_UNHEALTHY`    | The destination's data plane health isn't healthy.                             | Restore data plane health, then retry.                                                                                                         |
| `DESTINATION_CLUSTER_CORDONED`     | The destination cluster is in maintenance mode.                                | Uncordon it, or pick another destination.                                                                                                      |
| `DESTINATION_FAILOVER_IN_PROGRESS` | A cluster failover is already targeting the destination.                       | Wait for it to finish and clean up.                                                                                                            |
| `DESTINATION_NOT_AUTHORIZED`       | You aren't a System Admin and lack `cluster.config.update` on the destination. | See [Permissions](#permissions).                                                                                                               |

### Per-deployment skips

When a Deployment is skipped, the rest of the batch still migrates. Skips are safe to retry: re-issuing `migrateDeployments` with just the skipped IDs is a normal workflow, because a skipped Deployment was never claimed and nothing is left in a partial state.

| Reason                               | Cause                                                                                    | Fix                                                                                                                                                                                                                                          |
| ------------------------------------ | ---------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DEPLOYMENT_NOT_FOUND`               | The Deployment ID doesn't exist or the Deployment is deleted.                            | Check the ID.                                                                                                                                                                                                                                |
| `MISSING_CLUSTER_ID`                 | The Deployment's source cluster is missing or deleted.                                   | Data integrity issue — contact [Astronomer support](https://support.astronomer.io).                                                                                                                                                          |
| `SOURCE_CLUSTER_NOT_AUTHORIZED`      | You lack `cluster.config.update` on this Deployment's source cluster.                    | Migration is a cluster-admin operation on both clusters. See [Permissions](#permissions).                                                                                                                                                    |
| `CLUSTER_REGION_NOT_SET`             | The source or destination cluster is still on the `unset` sentinel region.               | Assign a real region. See [Configure per-deployment migration](/docs/astro-private-cloud/v-2-x/configure-per-deployment-migration#step-3-assign-a-region-to-every-failover-enabled-cluster). This is the most common skip after upgrading to 2.1. |
| `CROSS_REGION_MIGRATION_NOT_ALLOWED` | The source and destination are in different regions.                                     | You can't override this. Pick an in-region destination, or use cluster failover.                                                                                                                                                             |
| `SAME_CLUSTER`                       | The destination is the Deployment's current cluster.                                     | Nothing to do — it's already there.                                                                                                                                                                                                          |
| `CURRENT_MISSION_CONFLICT`           | The Deployment already has an active mission.                                            | Wait for it to reach a terminal state, then retry. The conflicting mission ID isn't returned in the skip payload — query `Deployment.activeMission` to find it.                                                                              |
| `CLUSTER_FAILOVER_IN_PROGRESS`       | A cluster failover is active on the Deployment's source cluster.                         | Wait for the failover to finish. Cluster failover and Deployment-scoped missions are mutually exclusive per Deployment.                                                                                                                      |
| `DEPLOYMENT_CORDONED`                | The Deployment is cordoned.                                                              | Uncordon it first.                                                                                                                                                                                                                           |
| `CLUSTER_CORDONED`                   | The Deployment's source cluster is in maintenance mode.                                  | Uncordon the cluster first.                                                                                                                                                                                                                  |
| `NOT_FAILOVER_READY`                 | The Deployment predates the failover prerequisites and hasn't been retrofitted.          | Run the failover upgrade on it, then retry.                                                                                                                                                                                                  |
| `UNSUPPORTED_DEPLOY_TYPE`            | The Deployment's latest deploy revision isn't `image`, for example git-sync or Dag-only. | This can't be migrated. Convert to image-based deploys.                                                                                                                                                                                      |
| `INTERNAL_ERROR` / `DB_ERROR`        | Transient failure during the claim transaction.                                          | Retry. If it persists, check the control plane logs.                                                                                                                                                                                         |

### Cluster failover skipping a migrating Deployment

The exclusion runs both ways. If a cluster failover starts while one of your migration missions is in flight, the failover silently skips that Deployment rather than aborting, and records it in `FailoverRequest.skippedDeployments`. Each entry carries the Deployment ID, release name, mission ID, and reason. The skipped list is surfaced on the cluster detail page at reconcile finalization. Those Deployments weren't failed over — once the migration mission completes, retry them explicitly.

## GraphQL surface

```graphql theme={null}
# Trigger
migrateDeployments(deploymentIds: [Id!]!, destinationClusterId: Id!): MigrateDeploymentsPayload!

type MigrateDeploymentsPayload {
  missions: [MigrationResult]!    # successfully claimed
  skipped:  [SkippedDeployment]!  # failed a per-deployment check
}
type MigrationResult   { deploymentId: Id, missionId: Id }
type SkippedDeployment { deploymentId: Id, reason: MigrationSkipReason }

# Observe
missionProgress(missionIds: [Id!]!): [MissionProgress]

type MissionProgress {
  id: Id
  kind: MissionKind                 # MOVE for migration and cluster failover; UPGRADE for retrofit
  state: MissionState
  originClusterId: Id
  destinationClusterId: Id          # null for upgrade missions
  failoverRequestId: Id             # null for migration and upgrade missions
  flights: [MissionFlightProgress!]
}
type MissionFlightProgress { id: Id, role: FlightRole, state: CpFlightState }

# Candidate destinations
failoverTargetClusters(sourceClusterId: Uuid!): [Cluster]

# Regions
regions: [Region]!
createRegion(name: String!, cloudProvider: String!): CreateRegionPayload!
updateRegion(regionId: Id!, name: String, cloudProvider: String): UpdateRegionPayload!
deleteRegion(regionId: Id!): DeleteRegionPayload!
```

## Relevant fields

| Field                                | Type              | Meaning                                                                                      |
| ------------------------------------ | ----------------- | -------------------------------------------------------------------------------------------- |
| `Deployment.isBeingFailedOver`       | `Boolean!`        | True while any mission is active. Clients should disable destructive actions.                |
| `Deployment.activeMission`           | `MissionProgress` | The current mission with its flights, or `null`.                                             |
| `Deployment.failoverReadiness`       | `String`          | Shield state: `not_capable`, `pending_upgrade`, `in_progress`, `ready`, or `attention`.      |
| `Deployment.isCordoned`              | `Boolean!`        | Cordoned Deployments are skipped by migration.                                               |
| `Deployment.secretsSynced`           | `Boolean`         | Whether the ESO push-secret sync succeeded. A failure surfaces as an **Attention** shield.   |
| `Cluster.failoverEnabled`            | `Boolean`         | Control-plane-effective: can this cluster be a migration destination?                        |
| `Cluster.failoverUpgradeStatus`      | `String`          | `not_started`, `in_progress`, or `complete`; `null` when the cluster isn't failover-capable. |
| `Cluster.isCordoned`                 | `Boolean`         | Cordoned clusters are rejected as destinations and block their Deployments as sources.       |
| `FailoverRequest.skippedDeployments` | `JSON`            | Deployments a cluster failover skipped because they were already in a mission.               |

<Note>
  The data plane health signal the migration gate reads is an internal control plane column and isn't exposed on the `Cluster` GraphQL type. To confirm it from the API, use `failoverTargetClusters`, which already filters on it.
</Note>

## Enumerations

```text theme={null}
MissionKind    MOVE | UPGRADE
MissionState   CREATED | PLANNING | PLANNED | IN_PROGRESS | CLEANUP_PENDING | COMPLETED | FAILED
FlightRole     HYPERJUMP | SCAVENGER | RETROFIT
CpFlightState  READY_TO_DISPATCH | AWAITING_DP_HEALTH | FAILED_DISPATCH | HOLDING |
               DISPATCHING | DISPATCHED | SUCCEEDED | FAILED | CANCELED
FailoverMode   CONTROLLED | FORCED        (migration is always CONTROLLED)

MigrationSkipReason
  DEPLOYMENT_NOT_FOUND | MISSING_CLUSTER_ID | SOURCE_CLUSTER_NOT_AUTHORIZED |
  CLUSTER_REGION_NOT_SET | CROSS_REGION_MIGRATION_NOT_ALLOWED | SAME_CLUSTER |
  CLUSTER_FAILOVER_IN_PROGRESS | CLUSTER_CORDONED | DEPLOYMENT_CORDONED |
  CURRENT_MISSION_CONFLICT | NOT_FAILOVER_READY | UNSUPPORTED_DEPLOY_TYPE |
  INTERNAL_ERROR | DB_ERROR
```

## Permissions

| Operation                                        | Required permission                                                                     |
| ------------------------------------------------ | --------------------------------------------------------------------------------------- |
| `regions` (list)                                 | `cluster.config.get` or `system.clusters.get`                                           |
| `createRegion` / `updateRegion` / `deleteRegion` | `cluster.config.update` or `system.clusters.update`                                     |
| `updateCluster` (assign region)                  | A cluster-admin role on that cluster, or System Admin                                   |
| `failoverTargetClusters`                         | `cluster.config.get` or `system.clusters.get`                                           |
| `migrateDeployments`                             | System Admin, or `cluster.config.update` on both the source and the destination cluster |
| `missionProgress`                                | `system.clusters.get`, or a cluster-admin role on the cluster                           |

<Warning>
  Migration is a cluster-admin operation. To migrate Deployments you must be a System Admin, or hold `cluster.config.update` on both the source and the destination cluster. Deployment-level permissions alone aren't enough. If you lack a cluster-admin role on the destination, the whole call is rejected with `DESTINATION_NOT_AUTHORIZED`. If you have it on the destination but not on a given Deployment's source cluster, that Deployment is skipped with `SOURCE_CLUSTER_NOT_AUTHORIZED` while the rest proceed.
</Warning>

## Troubleshooting index

| Symptom                                                    | Look at                                                                                                                                                                              |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Every Deployment was skipped with `CLUSTER_REGION_NOT_SET` | [Configure per-deployment migration](/docs/astro-private-cloud/v-2-x/configure-per-deployment-migration#step-3-assign-a-region-to-every-failover-enabled-cluster) — assign a real region. |
| The destination cluster isn't in the picker                | `Cluster.failoverEnabled` and the **Failover** field on cluster detail. You can't query data plane health directly — use `failoverTargetClusters`, which filters on it.              |
| The picker offered a cluster that then failed              | `failoverTargetClusters` doesn't filter by region or cordon. Confirm both manually.                                                                                                  |
| You can't deploy code to a Deployment                      | It probably has an active mission — check `Deployment.activeMission`.                                                                                                                |
| The shield is green but the Deployment is migrating        | Expected. The shield tracks the upgrade lifecycle; use the **Mission Status** column.                                                                                                |
| A mission is stuck in `CLEANUP_PENDING`                    | Not stuck — cutover succeeded, only cleanup remains. This applies to cluster failover only; a per-deployment migration never enters this state.                                      |
| A flight is `AWAITING_DP_HEALTH`                           | A wait, not a failure. Restore data plane health and it resumes automatically.                                                                                                       |
| A mission `FAILED` and the Deployment is gone              | See [Recover a failed migration](/docs/astro-private-cloud/v-2-x/manage-per-deployment-migration#recover-a-failed-migration). Retry forward — never hand-restore the source.              |
| A cluster failover didn't move one of your Deployments     | Check `FailoverRequest.skippedDeployments` on the cluster detail page.                                                                                                               |

## Related documentation

* [Configure per-deployment migration](/docs/astro-private-cloud/v-2-x/configure-per-deployment-migration)
* [Manage and observe per-deployment migration](/docs/astro-private-cloud/v-2-x/manage-per-deployment-migration)
* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)
* [Trigger a data plane failover](/docs/astro-private-cloud/v-2-x/trigger-data-plane-failover)


# Audit roles and permissions on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/permission-audit

Review who has access to what on Astro Private Cloud, and what a specific user, Team, or service account can do.

The Permission Audit page shows the current state of role assignments across your platform. Use it to answer two kinds of questions:

* Who has access to a given System, Workspace, or Deployment, and through which role?
* What can a specific user, Team, or service account do, and where does that access come from?

To open it, go to **Roles and Permissions** and click **Permission Audit**. The page has two modes: **By entity** and **By principal**.

<Note>
  **Astro Private Cloud 2.1**

  This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
</Note>

<Note>
  Permission Audit shows the current state of role assignments. It doesn't show a history of past changes, such as who created or edited a role.
</Note>

## Prerequisites

* System Admin access to Astro Private Cloud.
* Custom roles enabled in your `values.yaml` file. See [Enable custom roles](/docs/astro-private-cloud/v-2-x/custom-roles#enable-custom-roles).

## Audit by entity

Use **By entity** to see everyone with access to a specific System, Workspace, or Deployment.

<Steps>
  <Step title="Select a scope">
    Select **System**, **Workspace**, or **Deployment**.
  </Step>

  <Step title="Select an entity">
    If you selected **Workspace** or **Deployment**, select the specific entity.
  </Step>
</Steps>

Principals with access appear grouped under **Users**, **Teams**, and **Service accounts**. Each row shows the role a principal holds and its source:

* **Direct**: The role is assigned directly at this entity.
* **Workspace role**: For a Deployment, the role comes from a Workspace-scoped assignment on the Deployment's parent Workspace.
* **System role**: The role comes from a System-scoped assignment, which applies everywhere.

Click a row to expand its effective permissions.

## Audit by principal

Use **By principal** to see everything a specific user, Team, or service account can do.

<Steps>
  <Step title="Select a principal type">
    Select **Users**, **Teams**, or **Service accounts**.
  </Step>

  <Step title="Select a principal">
    Search for the principal, then select it.
  </Step>
</Steps>

Every entity the principal holds a direct role assignment on appears as a row, with its scope and role. Click a row to expand its effective permissions.

<Note>
  **By principal** lists a principal's direct role assignments only. If a user holds a role only through Team membership, that assignment doesn't appear as its own row; it surfaces in the effective permissions panel instead. See [Effective permissions](#effective-permissions).
</Note>

## Effective permissions

Expanding a row shows:

* **Granted by**: The roles contributing permissions at this scope and entity. A role held through Team membership is labeled **via team `<team-name>`**.
* **Effective permissions**: The full set of permissions the principal has at this scope and entity, combining every contributing role.

By default, only granted permissions appear. Turn off **Granted only** to also show the permissions the principal doesn't have at this scope, displayed struck through. Use this view to check for gaps before assigning a role, or to confirm a permission change had the effect you expected.

## Export to CSV

Click **Export CSV** to download the current view as a comma-separated values (CSV) file. In **By entity** mode, the export includes one row per principal with access to the selected entity. In **By principal** mode, it includes one row per entity the selected principal has access to. Each row includes the principal, its type, scope, entity, roles, and effective permissions.

## What's next

* To create a role with a specific set of permissions, see [Create custom roles](/docs/astro-private-cloud/v-2-x/custom-roles).
* For the full list of built-in roles and the permissions they grant, see [User roles and permissions](/docs/astro-private-cloud/v-2-x/role-permission-reference).


# Read-only root filesystem
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/read-only-root-filesystem

How Astro Private Cloud enforces readOnlyRootFilesystem on containers, and how to add writable directories with extraVolumes and extraVolumeMounts.

In Astro Private Cloud (APC), the root filesystem of every platform container and every Airflow Deployment container is read-only. Kubernetes enforces this by setting `readOnlyRootFilesystem: true` on each container's `securityContext` which reduces the container attack surface.

This document explains the behavior, lists the locations that are writable by default, and shows how to add additional writable directories when an application needs to write outside of those paths.

## Default behavior

* Every container in the APC platform Helm chart, the Airflow Helm chart, and the Airflow operator runs with `readOnlyRootFilesystem: true`.
* This setting is enforced by the chart templates and can't be disabled through Helm values.
* For Airflow Deployment Pods, `/usr/local/airflow` and `/tmp` are mounted as `emptyDir` volumes so that Airflow, your Dags, and provider packages can continue to write to these standard locations.
* Other paths inside the container image, such as `/`, `/etc`, `/var`, and `/home`, are read-only. Any process that tries to create or modify a file outside of a mounted writable volume fails with a `Read-only file system` error.

<Tip>
  If your application or Dag needs to write to disk, write to `/usr/local/airflow/logs` if the data is log output, or to a directory you have explicitly mounted with `extraVolumes` and `extraVolumeMounts`. Writing to other paths under `/usr/local/airflow` fails because the root filesystem is read-only.
</Tip>

## Add a writable directory with `extraVolumes` and `extraVolumeMounts`

When an application must write to a path that isn't writable by default, mount an `emptyDir` (or other ephemeral volume) at that path using the `extraVolumes` and `extraVolumeMounts` keys on the relevant component.

The Airflow Helm chart exposes `extraVolumes` and `extraVolumeMounts` on each Airflow component, including:

* `airflow.scheduler`
* `airflow.workers`
* `airflow.triggerer`
* `airflow.dagProcessor`
* `airflow.apiServer`
* `airflow.webserver`
* `airflow.pgbouncer`
* `airflow.redis`
* `airflow.statsd`
* `airflow.flower`
* `dagDeploy`

The following example mounts an `emptyDir` at `/opt/cache` on Airflow workers so a provider package can write its cache:

```yaml wrap theme={null}
airflow:
  workers:
    extraVolumes:
      - name: provider-cache
        emptyDir: {}
    extraVolumeMounts:
      - name: provider-cache
        mountPath: /opt/cache
```

Apply the same pattern to other components by setting `extraVolumes` and `extraVolumeMounts` under the appropriate component key.

### Size the volume

By default, `emptyDir` volumes can grow until they exhaust the node's ephemeral storage. Set `sizeLimit` to bound the volume:

```yaml wrap theme={null}
airflow:
  workers:
    extraVolumes:
      - name: provider-cache
        emptyDir:
          sizeLimit: 1Gi
    extraVolumeMounts:
      - name: provider-cache
        mountPath: /opt/cache
```

For more guidance on sizing ephemeral storage, see [Ephemeral storage configuration](/docs/astro-private-cloud/v-2-x/ephemeral-storage).

## Seed a writable directory with image contents

`extraVolumes` mounts an empty directory at the target path, which hides any files that were baked into the container image at that location. If your application reads files from a path it also needs to write to, use an init container to copy the image contents into the writable volume before the main container starts.

The following example makes `/opt/app` writable on Airflow workers while preserving files from the image:

```yaml wrap theme={null}
airflow:
  workers:
    extraVolumes:
      - name: app-data
        emptyDir: {}
    extraVolumeMounts:
      - name: app-data
        mountPath: /opt/app
    extraInitContainers:
      - name: seed-app-data
        image: "{{ .Values.images.airflow.repository }}:{{ .Values.images.airflow.tag }}"
        command:
          - sh
          - -c
          - cp -a /opt/app/. /seed/
        volumeMounts:
          - name: app-data
            mountPath: /seed
        securityContext:
          readOnlyRootFilesystem: true
          allowPrivilegeEscalation: false
```

The init container shares the `emptyDir` volume with the main container. When the main container starts, the volume already contains the files from the image, and the main container can also write to it.

## Redirect application write paths with environment variables

Some applications can be configured to write to a different location through environment variables. When that option exists, mount a writable volume with `extraVolumes` and `extraVolumeMounts` as shown earlier, and then point the application at that mount path.

For example, mount an `emptyDir` at `/tmp` on Airflow workers and set `TMPDIR` so libraries that respect it write there:

```yaml wrap theme={null}
airflow:
  workers:
    extraVolumes:
      - name: tmp
        emptyDir: {}
    extraVolumeMounts:
      - name: tmp
        mountPath: /tmp
    env:
      - name: TMPDIR
        value: /tmp
```

Check each application's documentation for the variables it supports, such as `TMPDIR`, `HOME`, `XDG_CACHE_HOME`, or vendor-specific cache directory variables. Combining a single writable volume with an environment variable is often cheaper than mounting volumes at each path the application touches.

## Troubleshoot read-only filesystem errors

If a Pod or task fails with an error such as `Read-only file system`, `Permission denied`, or `Errno 30`, the application is attempting to write to a path that isn't backed by a writable volume.

1. Identify the path the application is writing to from the error message or task logs.
2. If the path should already be writable, such as `/usr/local/airflow/logs` or a directory you mounted with `extraVolumes`, verify that the Pod has the expected volumes with `kubectl describe pod <pod-name>`.
3. If the path isn't writable, either:
   * Add `extraVolumes` and `extraVolumeMounts` for the component that runs the application, as shown in the previous sections.
   * Configure the application with an environment variable to write to a path you have mounted as writable.
4. Apply the change with a Helm upgrade and re-run the failing task.


# Register a data plane cluster
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/register-data-plane

Register a data plane cluster to your Astro Private Cloud control plane.

After you install a data plane cluster, you must register it with the Astro Private Cloud control plane so that the APC API can schedule Airflow Deployments.

<Warning>
  Only a system admin can register a data plane cluster.
</Warning>

## Prerequisites

* The data plane Kubernetes cluster is up and reachable from the control plane network.
* Astronomer data plane components are installed and healthy on the data plane, including the deployment orchestrator.
* You know the base domain for the data plane ingress. For example, `finance-dataplane-us-east-1.example.company.com`.

## Required fields

* **Name**: A unique identifier for the data plane cluster across the control plane. Names must be unique; pick a stable, human‑readable value. For example, `finance-dataplane-us-east-1`.
* **Base domain**: The base DNS domain served by the data plane ingress, in the format `https://<domainPrefix>.<base_domain>`. For example, `finance-dataplane-us-east-1.example.company.com`. This domain resolves to the data plane’s ingress endpoints.
* **Cluster override**: (Optional) Provide initial overrides to customize the cluster’s Deployment configuration. See [Cluster override](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster).

## Register in the UI

<Frame>
  <img alt="Shows the process of registering a cluster through the UI" />
</Frame>

<Steps>
  <Step title="Open Clusters page">
    In the APC UI, open the **Clusters** page.
  </Step>

  <Step title="Add a new cluster">
    Click **+ Cluster** to start registering your cluster.
  </Step>

  <Step title="Enter cluster details">
    Enter a unique **Name**, the data plane **Base domain** in the `https://<domainPrefix>.<base_domain>` format, and optionally set **Cluster override**.
  </Step>

  <Step title="Register the cluster">
    Click **Register Cluster** to save your cluster information. The control plane validates connectivity to the deployment orchestrator.
  </Step>

  <Step title="Verify">
    Check the list of clusters on the **Clusters** page. If you successfully registered your cluster, it appears in the list with a Healthy status.
  </Step>
</Steps>

Now you can proceed to create Airflow Deployments in this data plane.

## Related

* [Clusters overview](/docs/astro-private-cloud/v-2-x/overview-data-plane-cluster)
* [Deregister a cluster](/docs/astro-private-cloud/v-2-x/deregister-data-plane)
* [Update cluster configs with overrides](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster)


# Using external registry backends in Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/registry-backend

Configure an external registry backend to work with the Astronomer platform.

Astro Private Cloud requires a Docker Registry to store the Docker Images generated every time a user pushes code or makes a configuration change to an Airflow Deployment on Astronomer.

The default storage backend for this Docker Registry is a [Kubernetes Persistent Volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/). While this may be sufficient for teams just getting started on Astronomer, Astronomer recommends backing the registry with an external storage solution for any team running in production.

The following registry backends are supported by Astronomer:

* [Google Cloud Storage](https://cloud.google.com/storage/)
* [AWS S3](https://aws.amazon.com/s3/)
* [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/)

<Info>This document explains only how to set up a registry for hosting your Deployment images in external cloud object storage. To create a custom registry for Deployment images within your Astro Private Cloud cluster, see [Configure a custom image registry for Deployment images](/docs/astro-private-cloud/v-2-x/custom-image-registry). Or, to host all images in a high-security environment with no connections to public networks or internet, use the air gapped configurations during your [APC Installation](/docs/astro-private-cloud/v-2-x/install-overview).</Info>

## Google Cloud Storage

If you're running Astro Private Cloud on Google Cloud Platform (GCP) Google Kubernetes Engine (GKE), Astronomer recommends using Google Cloud Storage (GCS) as a registry backend solution.

### Prerequisites

To use GCS as a registry backend solution, you'll need:

* An existing GCS Bucket
* Your Google Cloud Platform service account JSON Key
* Permissions to create a Kubernetes Secret in your cluster

### Update your `values.yaml` file

1. Download your GCP service account JSON key from the [Google Console](https://console.cloud.google.com/apis/credentials/serviceaccountkey). Make sure the service account you use has both the `Storage Legacy Bucket Owner` and `Storage Object Admin` roles.

2. Create a [Kubernetes Secret](https://kubernetes.io/docs/concepts/configuration/secret/) using the downloaded key:

```text wrap theme={null}
kubectl create secret generic astronomer-gcs-keyfile --from-file astronomer-gcs-keyfile=/path/to/key.json -n <your-namespace>
```

3. Add the following to your `values.yaml` file:

```yaml wrap theme={null}
astronomer:
  registry:
    gcs:
      enabled: true
      bucket: my-gcs-bucket
```

Example:

```yaml expandable wrap theme={null}
#################################
## Astronomer global configuration
#################################
global:
  # Base domain for all subdomains exposed through ingress
  baseDomain: astro.mydomain.com

  # Name of secret containing TLS certificate
  tlsSecret: astronomer-tls

#################################
## Nginx configuration
#################################
nginx:
  # IP address the nginx ingress should bind to
  loadBalancerIP: 0.0.0.0
  preserveSourceIP: true

#################################
## SMTP configuration
#################################
astronomer:
  houston:
    config:
      email:
        enabled: true
        smtpUrl: YOUR_URI_HERE

#################################
## Registry configuration
#################################
  registry:
    gcs:
      enabled: true
      bucket: my-gcs-bucket
```

4. Push the configuration change to your platform as described in [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

## AWS S3

If you're running Astro Private Cloud on the Amazon Elastic Kubernetes Service (EKS), Astronomer recommends using AWS S3 as a registry backend solution.

### Prerequisites

To use AWS S3 as a registry backend solution, you'll need:

* An S3 bucket
* Your AWS Access Key
* Your AWS Secret Key
* Ability to create a Kubernetes Secret in your cluster

### Create S3 IAM policy and user

1. Use the following definition to create a new AWS IAM policy, making sure to replace `S3_BUCKET_NAME` with your own S3 bucket's name:

```text wrap theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": "arn:aws:s3:::S3_BUCKET_NAME"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::S3_BUCKET_NAME/*"
    }
  ]
}
```

2. Create a new IAM User and attach the Policy. Your access key and secret key are generated and displayed after you create the user.

3. Create Kubernetes secrets for your key credentials in your Astronomer installation:

   ```sh wrap theme={null}
   kubectl create secret generic astronomer-s3-access-key --from-literal=accesskey=<your-access-key> -n <your-namespace>
   kubectl create secret generic astronomer-s3-secret-key --from-literal=secretkey=<your-secret-key> -n <your-namespace>
   ```

4. Select one of the following options:

   * To authenticate to AWS with your registry credentials, add this entry to the `values.yaml` file:

   ```yaml wrap theme={null}
    astronomer:
      registry:
        s3:
          enabled: true
          region: us-east-1
          regionendpoint: <your-region-endpoint>
          bucket: <your-bucket-name>
        extraEnvVars:
          - name: REGISTRY_STORAGE_S3_REGION
            value: <your-s3-region>
          - name: REGISTRY_STORAGE_S3_ACCESSKEY
              valueFrom:
                secretKeyRef:
                  name: astronomer-s3-access-key
                  key: AWS_ACCESS_KEY_ID
          - name: REGISTRY_STORAGE_S3_SECRETKEY
              valueFrom:
                secretKeyRef:
                  name: astronomer-s3-secret-key
                  key: AWS_ACCESS_SECRET_ACCESS_KEY
   ```

   * To authenticate to AWS without providing your registry credentials, add this entry to the `values.yaml` file:

   ```yaml wrap theme={null}
    astronomer:
      registry:
        s3:
          enabled: true
          region: us-east-1
          regionendpoint: <your-region-endpoint>
          bucket: <your-bucket-name>
        extraEnvVars:
          - name: REGISTRY_STORAGE_S3_REGION
            value: <your-s3-region>
   ```

   4. Push the configuration change to your platform. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

### Enable encryption (*Optional*)

1. Create a key in AWS Key Management Service (KMS). During the key creation process you'll be asked to add "key users". Add the user created above as a "key user".

2. Create Kubernetes secrets for your key credentials:

   ```sh wrap theme={null}
   kubectl create secret generic astronomer-s3-access-key --from-literal=accesskey=<your-access-key> -n <your-namespace>
   kubectl create secret generic astronomer-s3-secret-key --from-literal=secretkey=<your-secret-key> -n <your-namespace>
   ```

3. Add the following values to your `values.yaml` file to enable encryption:

```yaml wrap theme={null}
astronomer:
registry:
 s3:
   enabled: true
   region: us-east-1
   bucket: my-s3-bucket
   encrypt: true
   keyid: my-kms-key-id
 extraEnvVars:
   - name: REGISTRY_STORAGE_S3_REGION
     value: <your-s3-region>
   - name: REGISTRY_STORAGE_S3_ACCESSKEY
       valueFrom:
         secretKeyRef:
           name: astronomer-s3-access-key
           key: accesskey
   - name: REGISTRY_STORAGE_S3_SECRETKEY
       valueFrom:
         secretKeyRef:
           name: astronomer-s3-secret-key
           key: secretkey
```

3. Push the configuration change to your platform. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

### Authorize Astronomer to a registry backend using IAM roles (Optional)

To avoid hardcoding credentials for your registry backend, add the following configuration to your `values.yaml` file:

```yaml wrap theme={null}
registry:
    serviceAccount:
      # Specifies whether a service account should be created
      create: true
      # Annotations to add to the service account
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxxxx:role/<your-iam-role>
    s3:
      enabled: true
      region: <your-region>
      bucket: <your-registry-backend>
```

Then, push the configuration change to your platform. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

## Azure Blob Storage

If you're running Astro Private Cloud on Azure Kubernetes Service (AKS), Astronomer recommends using Azure Blob Storage as a registry backend solution.

### Prerequisites

To use Azure Blog Storage as a registry backend solution, you'll need:

* Azure Storage Account Name
* Azure Account Access Key
* Azure Container Name

### Configure the registry backend

1. Create Kubernetes secrets for your key credentials:

   ```sh wrap theme={null}
   kubectl create secret generic astronomer-azure-access-key --from-literal=accountname=<your-account-name> -n <your-namespace>
   kubectl create secret generic astronomer-azure-secret-key --from-literal=accountkey=<your-account-key> -n <your-namespace>
   ```

2. Add the following to your `values.yaml` file:

```yaml wrap theme={null}
astronomer:
  registry:
    azure:
      enabled: true
      accountname: my-account-name
      accountkey: my-account-key
      container: my-container-name
      realm: core.windows.net
    extraEnvVars:
      - name: REGISTRY_STORAGE_AZURE_REGION
        value: <your-azure-region>
      - name: REGISTRY_STORAGE_AZURE_ACCOUNTNAME
          valueFrom:
            secretKeyRef:
              name: astronomer-azure-access-key
              key: accountname
      - name: REGISTRY_STORAGE_AZURE_ACCOUNTKEY
          valueFrom:
            secretKeyRef:
              name: astronomer-azure-secret-key
              key: accountkey
```

<Warning>
  If you use Astro Private Cloud version 0.37.2 and above, and have to force push images to use Registry V3 with Azure blob storage, you can rollback to Registry V2 to resolve the issue with the following configurations in your `values.yaml` file.

  The APC API registry configuration:

  ```yaml wrap theme={null}
  astronomer:
    houston:
      config:
        registry:
          version: 2
  ```

  Images registry configuration:

  ```yaml wrap theme={null}
  astronomer:
     images:
         registry:
            repository: quay.io/astronomer/ap-registry
            tag: 3.21.3-2
  ```
</Warning>

3. Push the configuration change to your platform as described in [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).


# Run a failover upgrade
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/run-failover-upgrade

Retrofit existing Astro Private Cloud Deployments so they become eligible for data plane failover, from the APC UI or the APC API.

Deployments created before you [enabled data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover) on a cluster don't have the inactive database connection that failover needs — only Deployments created after you enable it get one automatically. Running a failover upgrade retrofits existing Deployments with that missing connection so they become eligible for failover, without you needing to recreate them.

<Note>
  For a conceptual overview of failover, see [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover).
</Note>

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

## Prerequisites

* You've enabled data plane failover on the cluster. See [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover).
* The Deployments you want to upgrade use image-based deploys. Deployments that use git-sync or Dag-only deploy mechanisms aren't eligible.
* The Deployments aren't cordoned, and the cluster doesn't have a failover in progress.

<Warning>
  If you hard-delete a Deployment that used a manually specified (custom) release name on a failover-enabled cluster, the secrets pushed to your external secrets store aren't removed immediately — they enter that provider's soft-delete window first (for AWS Secrets Manager, 30 days by default, configurable down to a minimum of 7 days). If you recreate a Deployment with the **same** release name during that window, every other component comes up normally, but the `PushSecret` for that Deployment fails, because the secret name is still reserved by the pending deletion. Either wait out the provider's recovery window, force-delete the secrets from the store to skip it, or use a different release name when recreating the Deployment.
</Warning>

## Upgrade one Deployment

### From the APC UI

1. Open the Deployment you want to upgrade.
2. Click **Failover Upgrade**.

### Using the APC API

Use the `upgradeDeploymentsForFailover` mutation with `deploymentIds` set to the Deployment's ID:

```graphql wrap theme={null}
mutation upgradeDeploymentsForFailover(
  $deploymentIds: [Uuid]
) {
  upgradeDeploymentsForFailover(
    deploymentIds: $deploymentIds
  ) {
    missions {
      deploymentId
      missionId
      __typename
    }
    skipped {
      deploymentId
      reason
      __typename
    }
    __typename
  }
}
{
  "deploymentIds": ["deployment-id-1"]
}
```

## Upgrade multiple Deployments

### From the APC UI

1. Go to the cluster's Deployments list.
2. Select the Deployments you want to upgrade, or select all of them to upgrade every eligible Deployment on the cluster at once.
3. Click **Failover Upgrade**.

### Using the APC API

<Warning>Provide exactly one of `clusterId` or `deploymentIds`. Providing both, or neither, returns an error.</Warning>

To upgrade every eligible Deployment on a cluster at once, pass `clusterId`:

```graphql wrap theme={null}
mutation upgradeDeploymentsForFailover(
  $clusterId: Uuid
) {
  upgradeDeploymentsForFailover(
    clusterId: $clusterId
  ) {
    missions {
      deploymentId
      missionId
      __typename
    }
    skipped {
      deploymentId
      reason
      __typename
    }
    __typename
  }
}
{
  "clusterId": "cluster-id"
}
```

To upgrade a specific set of Deployments instead, pass `deploymentIds` and omit `clusterId`:

```graphql wrap theme={null}
mutation upgradeDeploymentsForFailover(
  $deploymentIds: [Uuid]
) {
  upgradeDeploymentsForFailover(
    deploymentIds: $deploymentIds
  ) {
    missions {
      deploymentId
      missionId
      __typename
    }
    skipped {
      deploymentId
      reason
      __typename
    }
    __typename
  }
}
{
  "deploymentIds": ["deployment-id-1", "deployment-id-2"]
}
```

## What happens

APC upgrades each selected Deployment individually. A Deployment that's already upgraded, cordoned, or using an unsupported deploy type is skipped automatically — it doesn't block the other Deployments in the same batch.

If you used the API, the mutation response lists each Deployment APC started an upgrade for (`missions`, with a `missionId` you can use to track that upgrade) and each Deployment APC didn't upgrade (`skipped`, with a `reason`):

| Reason                         | Meaning                                                               |
| ------------------------------ | --------------------------------------------------------------------- |
| `ALREADY_UPGRADED`             | The Deployment already has the inactive database connection it needs. |
| `UNSUPPORTED_DEPLOY_TYPE`      | The Deployment doesn't use an image-based deploy.                     |
| `CLUSTER_NOT_FAILOVER_CAPABLE` | The Deployment's cluster doesn't have data plane failover enabled.    |
| `CLUSTER_FAILOVER_IN_PROGRESS` | A cluster-level failover is currently in progress.                    |
| `DEPLOYMENT_CORDONED`          | The Deployment is cordoned.                                           |
| `CLUSTER_CORDONED`             | The Deployment's cluster is cordoned.                                 |
| `CURRENT_MISSION_CONFLICT`     | The Deployment already has another migration in progress.             |

If you used the UI, the cluster shows an overall failover upgrade status so you can confirm when every eligible Deployment has finished:

* **Not started**: You haven't upgraded any eligible Deployments yet.
* **In progress**: An upgrade is running for at least one Deployment.
* **Complete**: Every eligible Deployment on the cluster has the inactive database connection it needs for failover.

## Check failover readiness

Each Deployment on a failover-enabled cluster shows a failover-readiness shield in the Deployments list. The shield tells you, at a glance, whether that Deployment can fail over right now, still needs a failover upgrade, or needs your attention first:

* **No shield**: Failover isn't enabled on the Deployment's cluster, so readiness doesn't apply.
* **Gray — Needs upgrade**: The Deployment is on a failover-enabled cluster but hasn't been upgraded yet. Run a failover upgrade to make it eligible.
* **Yellow — Upgrading**: A failover upgrade is currently running for the Deployment.
* **Green — Ready**: The Deployment has everything it needs for failover — the inactive database connection, an image-based deploy, and its secrets synced to the external secrets store.
* **Red — Needs attention**: Either the Deployment's last failover upgrade failed, it's no longer eligible after being upgraded, or its secrets failed to sync to the external secrets store (a `PushSecret` failure). A `PushSecret` failure can also happen if you recently recreated the Deployment with the same manually specified release name as a hard-deleted one — see the note under [Prerequisites](#prerequisites). See [Verify secret replication before a failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover#verify-secret-replication-before-a-failover).

### Debug a red shield

Start by checking the `PushSecret` resources in the Deployment's namespace:

```bash wrap theme={null}
kubectl -n <deployment-namespace> get pushsecrets
```

A healthy Deployment shows every `PushSecret` as `Synced`:

```text wrap theme={null}
NAME                                             AGE   STATUS   LAST SYNC
<deployment-namespace>-active-metadata           41s   Synced   40s
<deployment-namespace>-active-result-backend     41s   Synced   40s
<deployment-namespace>-elasticsearch             41s   Synced   41s
<deployment-namespace>-env                       41s   Synced   41s
<deployment-namespace>-fernet-key                41s   Synced   41s
<deployment-namespace>-inactive-metadata         41s   Synced   40s
<deployment-namespace>-inactive-result-backend   40s   Synced   39s
```

If one instead shows an error, or no status at all, describe it to see why:

```bash wrap theme={null}
kubectl -n <namespace> describe pushsecret <pushsecret-name>
```

Check the `Events` section at the bottom of the output for the sync failure reason — typically a permission error from the external secrets store rejecting the write, or (per the note under [Prerequisites](#prerequisites)) a secret name still reserved by a pending soft-deletion.

## Related documentation

* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)
* [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover)
* [Trigger a data plane failover](/docs/astro-private-cloud/v-2-x/trigger-data-plane-failover)


# Scale Airflow resources
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/scale-airflow-resources

Configure CPU, memory, and replica settings for Airflow deployment components.

Configure CPU, memory, and replica settings for Airflow Deployment components including scheduler, webserver, workers, triggerer, Dag processor, and API server.

Resource values are plain integers: CPU in millicpu and memory in MiB.

## Component resources

### Scheduler

The scheduler orchestrates Dag runs and task scheduling.

```yaml wrap theme={null}
scheduler:
  replicas: 1
  resources:
    requests:
      cpu: 500
      memory: 1920
    limits:
      cpu: 1000
      memory: 3840
```

### Scaling considerations

* Add additional replicas for high availability.
* Increase memory for complex Dag dependencies.
* Set `safeToEvict: false` to prevent cluster autoscaler eviction.

### Webserver

```yaml wrap theme={null}
webserver:
  resources:
    requests:
      cpu: 500
      memory: 1920
    limits:
      cpu: 1000
      memory: 3840
```

### API server (Airflow 3+)

```yaml wrap theme={null}
apiServer:
  replicas: 1
  resources:
    requests:
      cpu: 1000
      memory: 3840
    limits:
      cpu: 2000
      memory: 7680
```

### Dag processor (Airflow 2.3+, required in Airflow 3)

```yaml wrap theme={null}
dagProcessor:
  enabled: true
  replicas: 1
  resources:
    requests:
      cpu: 1000
      memory: 3840
    limits:
      cpu: 2000
      memory: 7680
```

<Note>
  In Airflow 2, the Dag processor defaults to 0 replicas and must be explicitly enabled. In Airflow 3, the APC API automatically sets `dagProcessor.enabled: true` and enforces a minimum of 1 replica regardless of configuration.
</Note>

### Triggerer

```yaml wrap theme={null}
triggerer:
  replicas: 1
  resources:
    requests:
      cpu: 500
      memory: 1920
    limits:
      cpu: 1000
      memory: 3840
```

### Workers (Celery executor)

```yaml wrap theme={null}
workers:
  replicas: 2
  resources:
    requests:
      cpu: 1000
      memory: 3840
    limits:
      cpu: 2000
      memory: 7680
  terminationGracePeriodSeconds: 600
```

## Sizing recommendations

### Small workloads (fewer than 50 Dags)

```yaml wrap theme={null}
scheduler:
  resources:
    requests: { cpu: 500, memory: 1920 }
    limits: { cpu: 1000, memory: 3840 }
workers:
  replicas: 1
  resources:
    requests: { cpu: 1000, memory: 3840 }
```

### Medium workloads (50–200 Dags)

```yaml wrap theme={null}
scheduler:
  resources:
    requests: { cpu: 500, memory: 1920 }
    limits: { cpu: 1000, memory: 3840 }
dagProcessor:
  enabled: true
  resources:
    requests: { cpu: 1000, memory: 3840 }
workers:
  replicas: 3
  resources:
    requests: { cpu: 1000, memory: 3840 }
```

### Large workloads (more than 200 Dags)

```yaml wrap theme={null}
scheduler:
  replicas: 2
  resources:
    requests: { cpu: 1000, memory: 3840 }
    limits: { cpu: 2000, memory: 7680 }
dagProcessor:
  enabled: true
  replicas: 2
  resources:
    requests: { cpu: 1000, memory: 3840 }
workers:
  replicas: 10
```

## Autoscale workers with KEDA

Kubernetes Event-driven Autoscaling (KEDA) scales Celery workers based on task queue depth. Enable KEDA for a Deployment using the `updateDeploymentKedaConfig` mutation:

```graphql wrap theme={null}
mutation {
  updateDeploymentKedaConfig(
    deploymentUuid: "<deployment-uuid>"
    state: true
  ) {
    id
    label
  }
}
```

## Monitor resources

```bash wrap theme={null}
# View current resource usage
kubectl top pods -n <deployment-namespace>

# Check resource limits
kubectl describe pod <pod-name> -n <deployment-namespace>
```


# Configure resources for Airflow components on Astro Private Cloud Deployments
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/scale-deployment-resources

Configure Deployment components to use the right amount of computational resources for your use case.

Use this document to configure resource usage for a Deployment's executor, webserver/apiserver, scheduler and triggerer components.

## Select an executor

The Airflow [executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html) works closely with the Airflow scheduler to determine what resources complete tasks as they queue. The main difference between executors is their available resources and how they utilize those resources to distribute work.

Astro Private Cloud supports three executors:

* [Local executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/local.html)
* [Celery executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/celery.html)
* [Kubernetes executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html)

Though it largely depends on your use case, Astronomer recommends the Local executor for development environments and the Celery or Kubernetes executors for production environments operating at scale.

For a detailed description of each executor, see [Airflow executors explained](https://docs.astronomer.io/learn/airflow-executors-explained).

## Scale core resources

Apache Airflow requires four primary components:

* The Webserver or `apiServer` (Airflow 3)
* The Scheduler
* The Executor (and the workers it runs)
* The Triggerer

To scale these resources, adjust the corresponding slider in the Astro Private Cloud UI to increase its available resources.

Read the following sections to help you determine which core resources to scale and when.

### Airflow webserver and API server

In Airflow 2, the **webserver** component renders the [Airflow UI](https://airflow.apache.org/docs/apache-airflow/stable/ui.html), providing access to Dag monitoring, task logs, and configuration.

Starting with Airflow 3, the **`apiServer`** replaces the webserver for serving the Airflow UI.

If loading pages or functions in the Airflow UI is slow or unresponsive, increase the resources allocated to the webserver (Airflow 2) or `apiServer` (Airflow 3).

### Scheduler

The [Airflow scheduler](https://airflow.apache.org/docs/apache-airflow/stable/scheduler.html) is responsible for monitoring task execution and triggering downstream tasks once dependencies have been met.

If you experience delays in task execution, which you can track via the [Gantt Chart](https://airflow.apache.org/docs/apache-airflow/stable/ui.html#gantt-chart) view of the Airflow UI, Astronomer recommends increasing the resources allocated towards the scheduler.

#### Scheduler count

Airflow 2.0 comes with the ability for users to run multiple schedulers concurrently to ensure high-availability, zero recovery time, and faster performance. You can provision up to 4 schedulers on any Deployment.

Each individual scheduler will be provisioned with the resources specified in **Scheduler Resources**. For example, if you set the CPU figure in **Scheduler Resources** to 5 CPUs and set **Scheduler Count** to 2, your Airflow Deployment will run with 2 Airflow schedulers using 5 CPUs each for a total of 10 CPUs.

To increase the speed at which tasks are scheduled and ensure high-availability, Astronomer recommends provisioning 2 or more Airflow schedulers for production environments.

#### Dag processor

Complex, dynamically generated Dags, sub-optimal Dag parsing practices, or a growing business that requires a larger data pipeline can strain Dag processing and threaten your Airflow scheduler's availability. Deployments can support high-scale environments more reliably by separating the Dag processor from the scheduler. You can now configure the number of Dag processors for the Deployment from the UI and the APC API.

If you want to enable and provision resources for standalone Dag processors, you can set the `airflowComponents.dagProcessor.enabled` feature flag to `true` at the cluster level. In the APC UI, go to your **Clusters** page, select your cluster, click **Edit** in the **Deployment Configuration** section, and add the following override to the **Configuration Override** field:

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        airflowComponents:
          dagProcessor:
            enabled: true
```

For details on using the UI for configuration, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).

By adjusting the **Dag Processor** slider in the Astro Private Cloud UI, you can provision up to 3 Dag Processors on any Deployment. You can also configure CPU and Memory with the **Dag Processor Resources** sliders.

<Warning>
  Airflow 3 requires the Dag Processor component to be enabled. Set `airflowComponents.dagProcessor.enabled: true` in your Deployment configuration. If this flag is disabled, Airflow 3 Deployments will fail to start or process Dags correctly.
</Warning>

### Triggerer

Airflow 2.2 introduces the triggerer, which is a component for running tasks with [deferrable operators](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html). Like the scheduler, the triggerer is highly available: If a triggerer shuts down unexpectedly, the tasks it was deferring can be recovered and moved to another triggerer.

By adjusting the **Triggerer** slider in the Astro Private Cloud UI, you can provision up to 2 triggerers on any Deployment running Airflow 2.2+. To take advantage of the Triggerer's high availability, we recommend provisioning 2 triggerers for production Deployments.

### (Kubernetes executor only) Set extra capacity

On Astronomer, resources required for the [`KubernetesPodOperator`](/docs/astro-private-cloud/v-2-x/kube-pod-operator) or the [Kubernetes Executor](/docs/astro-private-cloud/v-2-x/kubernetes-executor) are set as **Extra Capacity**.

The Kubernetes executor and `KubernetesPodOperator` each spin up an individual Kubernetes pod for each task that needs to be executed, then spin down the pod after that task is completed.

The amount of CPU and Memory allocated to **Extra Capacity** maps to [resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) on the [Kubernetes Namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) in which your Airflow Deployment lives on Astro Private Cloud. More specifically, **Extra Capacity** represents the maximum possible resources that could be provisioned to a pod at any given time.

Resources allocated to **Extra Capacity** don't affect scheduler or webserver/apiserver performance and don't represent actual usage.

### (Celery executor only) Configure workers

To optimize for flexibility and availability, the Celery executor works with a set of independent Celery workers across that it can delegate tasks. On Astro Private Cloud, you can configure your Celery workers to fit your use case.

#### Worker count

By adjusting the **Worker Count** slider, users can provision up to 20 Celery workers on any Airflow Deployment. If you would like users to have the ability to set more than 20 workers on any Deployment, you can change this limit by applying a [config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

Each individual worker will be provisioned with the resources specified in **Worker Resources**. If you set the CPU figure in **Worker Resources** to 5 CPUs and set **Worker Count** to 3, for example, your Airflow Deployment will run with 3 Celery workers using 5 CPUs each for a total of 15 CPUs.

#### Worker termination grace period

On Astro Private Cloud, Celery workers restart after every code deploy to your Airflow Deployment. This makes sure that workers execute with the most up-to-date code. To minimize disruption during task execution, however, APC supports the ability to set a **Worker Termination Grace Period**.

If a deploy is triggered while a Celery worker is executing a task and **Worker Termination Grace Period** is set, the worker will continue to process that task up to a certain number of minutes before restarting itself. By default, the grace period is ten minutes.


# Configure an external secrets backend on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/secrets-backend

Configure a secrets backend on Astro Private Cloud to store Airflow variables and connections in a centralized place.

Configure a secrets backend on Astro Private Cloud to centrally and securely manage Airflow variables and connections using your preferred secrets management tool.

Astro Private Cloud supports integration with the following external secrets backends:

<CardGroup>
  <Card title="Hashicorp Vault" href="/astro-private-cloud/v-2-x/secrets-backend-hashicorp" icon="hashicorp">
    Use your own Hashicorp Vault instance for storing Airflow variables and connections.
  </Card>

  <Card title="AWS Secrets Manager" href="/astro-private-cloud/v-2-x/secrets-backend-aws-secrets-manager" icon="aws">
    Manage secrets and credentials with AWS Secrets Manager.
  </Card>

  <Card title="AWS Parameter Store" href="/astro-private-cloud/v-2-x/secrets-backend-aws-parameter-store" icon="aws">
    Use AWS Systems Manager Parameter Store for secret storage integration.
  </Card>

  <Card title="Google Cloud Secret Manager" href="/astro-private-cloud/v-2-x/secrets-backend-gcp" icon="google">
    Integrate Google Cloud Secret Manager with your Astro Private Cloud Deployments.
  </Card>

  <Card title="Azure Key Vault" href="/astro-private-cloud/v-2-x/secrets-backend-azure" icon="microsoft">
    Securely store and retrieve Airflow secrets with Azure Key Vault.
  </Card>
</CardGroup>

## Why integrate a secrets backend?

* Store Airflow secrets in a centralized place, keeping them outside your Airflow metadata database.
* Meet your organization's security and compliance requirements.
* Enable easier rotation and management of connection and variable secrets.

For detailed setup instructions for each backend, select your provider above.

<Info>
  You can continue to manage Airflow variables and connections via the Airflow UI or as environment variables if desired. When a secrets backend is configured, Airflow will check the external backend for secret values before falling back to environment variables and then to the UI.
</Info>


# Configure AWS Parameter Store as a secrets backend on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/secrets-backend-aws-parameter-store

Use AWS Systems Manager Parameter Store as a secrets backend for storing Airflow variables and connections with Astro Private Cloud.

In this section, you'll learn how to use [AWS Systems Manager (SSM) Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) as a secrets backend on Astro Private Cloud.

## Prerequisites

* A [Deployment](/docs/astro-private-cloud/v-2-x/configure-deployment).
* The [Astro CLI](/docs/cli/v1.43/overview).
* An Astro project initialized with `astro dev init`.
* Access to AWS SSM Parameter Store.
* A valid AWS Access Key ID and Secret Access Key.

### Step 1: Write an Airflow variable or connection to AWS Parameter Store

To start, add an Airflow variable or connection as a secret to Parameter Store for testing. For instructions, see the AWS documentation on how to do so using the [AWS Systems Manager Console](https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-create-console.html), the [AWS CLI](https://docs.aws.amazon.com/systems-manager/latest/userguide/param-create-cli.html), or [Tools for Windows PowerShell](https://docs.aws.amazon.com/systems-manager/latest/userguide/param-create-ps.html).

Variables and connections should live at `/airflow/variables` and `/airflow/connections`, respectively. For example, if you're setting a secret variable with the key `my_secret`, it should exist at `/airflow/connections/my_secret`.

#### Step 2: Set up AWS Parameter Store locally

To test AWS Parameter Store locally, configure it as a secrets backend in your Astro project.

First, install the [Airflow provider for Amazon](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/index.html) by adding the following to your project's `requirements.txt` file:

```text wrap theme={null}
apache-airflow-providers-amazon
```

Then, add the following environment variables to your project's `Dockerfile`:

```docker wrap theme={null}
# Make sure to replace `<your-aws-key>` and `<your-aws-secret-key>` with your own values.
ENV AWS_ACCESS_KEY_ID="<your-aws-key>"
ENV AWS_SECRET_ACCESS_KEY="<your-aws-secret-key>"
ENV AWS_DEFAULT_REGION="<your-aws-region>"
ENV AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
ENV AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "/airflow/connections", "variables_prefix": "/airflow/variables"}
```

In the next step, you'll test that this configuration is valid locally.

<Danger>If you want to deploy your project to a hosted Git repository before deploying to Astro Private Cloud, be sure to save `<your-aws-key>` and `<your-aws-secret-key>` in a secure manner. When you deploy to Astro Private Cloud, use the UI to set these values as secrets.</Danger>

<Tip>
  If you'd like to reference an AWS profile, you can also add the `profile` param to `ENV AIRFLOW__SECRETS__BACKEND_KWARGS`.

  To further customize the integration between Airflow and AWS SSM Parameter Store, reference Airflow documentation with the [full list of available kwargs](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/secrets/systems_manager/index.html).
</Tip>

#### Step 3: Run an example Dag to test AWS Parameter Store locally

To test Parameter Store, write a simple Dag which calls your secret and add this Dag to your Astro project's `dags` directory.

For example, you can use the following Dag to print the value of an Airflow variable to your task logs:

```python wrap theme={null}
from datetime import datetime

from airflow import DAG
from airflow.models import Variable
from airflow.operators.python import PythonOperator

def print_var():
    my_var = Variable.get("<your-variable-key>")
    print(f'My variable is: {my_var}')

with DAG('example_secrets_dags', start_date=datetime(2022, 1, 1), schedule=None) as dag:

  test_task = PythonOperator(
      task_id='test-task',
      python_callable=print_var,
)
```

You can do the same for any Airflow connection.

To test your changes:

1. Run `astro dev restart` to push your changes to your local Airflow environment.
2. In the Airflow UI (`http://localhost:8080/admin/`), trigger your new Dag.
3. Click **test-task** > **View Logs**. If you ran the example Dag above, you should see the contents of your secret in the task logs:

   ```text wrap theme={null}
   {logging_mixin.py:109} INFO - My variable is: my-test-variable
   ```

#### Step 4: Deploy to Astro Private Cloud

Once you've confirmed that the integration with AWS SSM Parameter Store works locally, you can complete a similar set up with a Deployment on Astro Private Cloud.

1. In the Astro Private Cloud UI, add the same environment variables found in your `Dockerfile` to your Deployment [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables). Specify both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` as **secret** ensure that your credentials are stored securely.
2. In your Astro project, delete the environment variables from your `Dockerfile`.
3. [Deploy your changes](/docs/astro-private-cloud/v-2-x/deploy-code-overview#full-image-deploy) to Astro Private Cloud.

Now, any Airflow variable or connection that you write to AWS SSM Parameter Store can be automatically pulled by any Dag in your Deployment on Astro Private Cloud.


# Configure AWS Secrets Manager as a secrets backend on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/secrets-backend-aws-secrets-manager

Use AWS Secrets Manager as a secrets backend for storing Airflow variables and connections with Astro Private Cloud.

## Prerequisites

* A [Deployment](/docs/astro-private-cloud/v-2-x/configure-deployment).
* The [Astro CLI](/docs/cli/v1.43/overview).
* An Astro project initialized with `astro dev init`.
* Access to AWS Secrets Manager.
* A valid AWS Access Key ID and Secret Access Key.

### Step 1: Write an Airflow variable or connection to AWS Secrets Manager

To start, add an Airflow variable or connection as a secret to Secrets Manager for testing. For instructions, see the [AWS documentation](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) on how to do so using the AWS Secrets Manager Console, CLI or SDK.

Variables and connections should live at `/airflow/variables` and `/airflow/connections`, respectively. For example, if you're setting a secret variable with the key `my_secret`, it should exist at `/airflow/connections/my_secret`.

#### Step 2: Set up AWS Secrets Manager locally

To test AWS Secrets Manager locally, configure it as a secrets backend in your Astro project.

First, install the [Airflow provider for Amazon](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/index.html) by adding the following to your project's `requirements.txt` file:

```text wrap theme={null}
apache-airflow-providers-amazon
```

Then, add the following environment variables to your project's `Dockerfile`:

```docker wrap theme={null}
# Make sure to replace `<your-aws-key>` and `<your-aws-secret-key>` with your own values.
ENV AWS_ACCESS_KEY_ID="<your-aws-key>"
ENV AWS_SECRET_ACCESS_KEY="<your-aws-secret-key>"
ENV AWS_DEFAULT_REGION="<your-aws-region>"
ENV AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
ENV AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}
```

In the next step, you'll test that this configuration is valid locally.

<Danger>If you want to deploy your project to a hosted Git repository before deploying to Astro Private Cloud, be sure to save `<your-aws-key>` and `<your-aws-secret-key>` in a secure manner. When you deploy to Astro Private Cloud, use the UI to set these values as secrets.</Danger>

<Tip>
  If you'd like to reference an AWS profile, you can also add the `profile` param to `ENV AIRFLOW__SECRETS__BACKEND_KWARGS`.

  To further customize the integration between Airflow and AWS Secrets Manager, reference Airflow documentation with the [full list of available kwargs](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/secrets/secrets_manager/index.html).
</Tip>

#### Step 3: Run an example Dag to test AWS Secrets Manager locally

To test Secrets Manager, write a simple Dag which calls your secret and add this Dag to your Astro project's `dags` directory.

For example, you can use the following Dag to print the value of an Airflow variable to your task logs:

```python wrap theme={null}
from datetime import datetime

from airflow import DAG
from airflow.models import Variable
from airflow.operators.python import PythonOperator

def print_var():
    my_var = Variable.get("<your-variable-key>")
    print(f'My variable is: {my_var}')

with DAG('example_secrets_dags', start_date=datetime(2022, 1, 1), schedule=None) as dag:

  test_task = PythonOperator(
      task_id='test-task',
      python_callable=print_var,
)
```

To test your changes:

1. Run `astro dev restart` to push your changes to your local Airflow environment.
2. In the Airflow UI (`http://localhost:8080/admin/`), trigger your new Dag.
3. Click **test-task** > **View Logs**. If you ran the example Dag above, you should see the contents of your secret in the task logs:

   ```text wrap theme={null}
   {logging_mixin.py:109} INFO - My variable is: my-test-variable
   ```

#### Step 4: Deploy to Astro Private Cloud

Once you've confirmed that the integration with AWS Secrets Manager works locally, you can complete a similar set-up with a Deployment on Astro Private Cloud.

1. In the Astro Private Cloud UI, add the same environment variables found in your `Dockerfile` to your Deployment [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables). Specify both `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` as **secret** ensure that your credentials are stored securely.
2. In your Astro project, delete the environment variables from your `Dockerfile`.
3. [Deploy your changes](/docs/astro-private-cloud/v-2-x/deploy-code-overview#full-image-deploy) to Astro Private Cloud.

Now, any Airflow variable or connection that you write to AWS Secrets Manager can be automatically pulled by any Dag in your Deployment on Astro Private Cloud.


# Configure Azure Key Vault as a secrets backend on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/secrets-backend-azure

Use Azure Key Vault as a secrets backend for storing Airflow variables and connections with Astro Private Cloud.

In this section, you'll learn how to use [Azure Key Vault](https://cloud.google.com/secret-manager/docs/configuring-secret-manager) as a secrets backend on Astro Private Cloud.

## Prerequisites

* A [Deployment](/docs/astro-private-cloud/v-2-x/configure-deployment).
* The [Astro CLI](/docs/cli/v1.43/overview).
* An Astro project initialized with `astro dev init`.
* An existing Azure Key Vault linked to a resource group.
* Your Key Vault URL. To find this, go to your Key Vault overview page > **Vault URI**.

If you don't already have Key Vault configured, see the [Microsoft Azure documentation](https://docs.microsoft.com/en-us/azure/key-vault/general/quick-create-portal).

### Step 1: Register Astro Private Cloud as an app on Azure

Follow the [Microsoft Azure documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-credentials) to register a new application for Astro Private Cloud.

At a minimum, you need to add a [secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-credentials) that Astro Private Cloud can use to authenticate to Key Vault.

Note the value of the application's client ID and secret for Step 3.

#### Step 2: Create an access policy

Follow the [Microsoft documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-credentials) to create a new access policy for the application that you just registered. The settings you need to configure for your policy are:

* **Configure from template**: Select `Key, Secret, & Certificate Management`.
* **Select principal**: Select the name of the application that you registered in Step 1.

#### Step 3: Set up Key Vault locally

In your Astro project, add the following line to your `requirements.txt` file:

```text wrap theme={null}
apache-airflow-providers-microsoft-azure
```

In your `Dockerfile`, add the following environment variables with your own values:

```docker wrap theme={null}
ENV AZURE_CLIENT_ID="<your-client-id>" # Found on App Registration page > 'Application (Client) ID'
ENV AZURE_TENANT_ID="<your-tenant-id>" # Found on App Registration page > 'Directory (tenant) ID'
ENV AZURE_CLIENT_SECRET="<your-client-secret>" # Found on App Registration Page > Certificates and Secrets > Client Secrets > 'Value'
ENV AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
ENV AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>"}
```

This tells Airflow to look for variable information at the `airflow-variables-*` path in Azure Key Vault and connection information at the `airflow-connections-*` path. In the next step, you'll run an example Dag to test this configuration locally.

<Tip>By default, this setup requires that you prefix any secret names in Key Vault with `airflow-connections` or `airflow-variables`. If you don't want to use prefixes in your Key Vault secret names, set the values for `sep`, `"connections_prefix"`, and `"variables_prefix"` to `""` within `AIRFLOW__SECRETS__BACKEND_KWARGS`.</Tip>

<Warning>
  If you want to deploy your project to a hosted Git repository before deploying to Astronomer, be sure to save `<your-client-id>`, `<your-tenant-id>`, and `<your-client-secret>` in a secure manner. When you deploy to Astronomer, you should set these values as secrets with the Astro Private Cloud UI.
</Warning>

#### Step 4: Test Key Vault locally

To test your Key Vault setup on Astro Private Cloud locally, [create a new secret](https://docs.microsoft.com/en-us/azure/key-vault/secrets/quick-create-portal#add-a-secret-to-key-vault) in Key Vault containing either a variable or a connection.

Once you create a test secret, write a simple Dag which calls the secret and add this Dag to your project's `dags` directory. For example, you can use the following Dag to print the value of a variable to your task logs:

```python wrap theme={null}
from datetime import datetime

from airflow import DAG
from airflow.models import Variable
from airflow.operators.python import PythonOperator

def print_var():
    my_var = Variable.get("<your-variable-key>")
    print(f'My variable is: {my_var}')

with DAG('example_secrets_dags', start_date=datetime(2022, 1, 1), schedule=None) as dag:

  test_task = PythonOperator(
      task_id='test-task',
      python_callable=print_var,
)
```

To test your changes:

1. Run `astro dev stop` followed by `astro dev start` to push your changes to your local Airflow environment.
2. In the Airflow UI (`http://localhost:8080/admin/`), trigger your new Dag.
3. Click **test-task** > **View Logs**. If you ran the example Dag above, you should see the contents of your secret in the task logs:

   ```text wrap theme={null}
   {logging_mixin.py:109} INFO - My variable is: my-test-variable
   ```

Once you confirm that the setup was successful, you can delete this Dag.

#### Step 5: Push changes to Astro Private Cloud

Once you've confirmed that your secrets are being imported correctly to your local environment, you're ready to configure the same feature in a Deployment on Astro Private Cloud.

1. In the Astro Private Cloud UI, add the same environment variables found in your `Dockerfile` to your Deployment [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables). Specify the `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, and `AZURE_CLIENT_SECRET` variables as **Secret** to ensure that your credentials are stored securely.
2. In your Astro project, delete the environment variables from your `Dockerfile`.
3. [Deploy your changes](/docs/astro-private-cloud/v-2-x/deploy-code-overview#full-image-deploy) to Astro Private Cloud.

From here, you can store any Airflow variables or connections as secrets on Key Vault and use them in your project.


# Configure Google Cloud Secret Manager as a secrets backend on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/secrets-backend-gcp

Use Google Cloud Secret Manager as a secrets backend for storing Airflow variables and connections with Astro Private Cloud.

In this section, you'll learn how to use [Google Cloud Secret Manager](https://cloud.google.com/secret-manager/docs/configuring-secret-manager) as a secrets backend on Astro Private Cloud.

## Prerequisites

* A [Deployment](/docs/astro-private-cloud/v-2-x/configure-deployment).
* The [Astro CLI](/docs/cli/v1.43/overview).
* An Astro project initialized with `astro dev init`.
* [Cloud SDK](https://cloud.google.com/sdk/gcloud).
* A Google Cloud environment with [Secret Manager](https://cloud.google.com/secret-manager/docs/configuring-secret-manager) configured.
* A [service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) with the [Secret Manager Secret Accessor](https://cloud.google.com/secret-manager/docs/access-control) role on Google Cloud.
* A [JSON service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating_service_account_keys) for the service account.

### Step 1: Write an Airflow variable or connection to Google Cloud Secret Manager

To start, add an Airflow variable or connection as a secret to Google Cloud Secret Manager. You can do so in the Cloud Console or the `gcloud` CLI.

Secrets must be formatted such that:

* Airflow variables are set as `airflow-variables-<variable-key>`.
* Airflow connections are set as `airflow-connections-<connection-id>`.

For example, to add an Airflow variable with a key `my-secret-variable`, you would run the following `gcloud` CLI command:

```sh wrap theme={null}
gcloud secrets create airflow-variables-<my-secret-variable> \
    --replication-policy="automatic"
```

For more information on creating secrets in Google Cloud Secret Manager, see the [Google Cloud documentation](https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#create).

#### Step 2: Set up Secret Manager locally

To test Google Secret Manager locally, configure it as a secrets backend in your Astro project.

First, install the [Airflow provider for Google](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/index.html) by adding the following to your project's `requirements.txt` file:

```text wrap theme={null}
apache-airflow-providers-google
```

Then, add the following environment variables to your project's Dockerfile:

```docker wrap theme={null}
ENV AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
ENV AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "gcp_keyfile_dict": <your-key-file>}
```

Make sure to paste your entire JSON service account key in place of `<your-key-file>`. In the next step, you'll test that this configuration is valid locally.

<Danger>If you want to deploy your project to a hosted Git repository before deploying to Astronomer, be sure to save `<your-key-file>` securely. Astronomer recommends adding it to your project's [`.env` file](/docs/cli/v1.43/private-python-packages) and specifying this file in `.gitignore`. When you deploy to Astronomer, you should set these values as secrets in the Astro Private Cloud UI.</Danger>

#### Step 3: Run an example Dag to test Secret Manager locally

To test Secret Manager, [create a secret](https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#create) containing either an Airflow variable or connection for testing.

Once you create a test secret, write a simple Dag which calls the secret and add this Dag to your project's `dags` directory. For example, you can use the following Dag to print the value of a variable to your task logs:

```python wrap theme={null}
from datetime import datetime

from airflow import DAG
from airflow.models import Variable
from airflow.operators.python import PythonOperator

def print_var():
    my_var = Variable.get("<your-variable-key>")
    print(f'My variable is: {my_var}')

with DAG('example_secrets_dags', start_date=datetime(2022, 1, 1), schedule=None) as dag:

  test_task = PythonOperator(

      task_id='test-task',
      python_callable=print_var,
)
```

To test your changes:

1. Run `astro dev stop` followed by `astro dev start` to push your changes to your local Airflow environment.
2. In the Airflow UI (`http://localhost:8080/admin/`), trigger your new Dag.
3. Click **test-task** > **View Logs**. If you ran the example Dag above, you should see the contents of your secret in the task logs:

   ```text wrap theme={null}
   {logging_mixin.py:109} INFO - My variable is: my-test-variable
   ```

Once you confirm that the setup was successful, you can delete this Dag.

#### Step 4: Deploy to Astro Private Cloud

Once you've confirmed that the integration with Google Cloud Secret Manager works locally, you can complete a similar set up with a Deployment on Astro Private Cloud.

1. In the Astro Private Cloud UI, add the same environment variables found in your `Dockerfile` to your Deployment [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables). Specify both `AIRFLOW__SECRETS__BACKEND` and `AIRFLOW__SECRETS__BACKEND_KWARGS` as **Secret** to ensure that your credentials are stored securely.
2. In your Astro project, delete the environment variables from your `Dockerfile`.
3. [Deploy your changes](/docs/astro-private-cloud/v-2-x/deploy-code-overview#full-image-deploy) to Astro Private Cloud.

You now should be able to see your secret information being pulled from Secret Manager on Astronomer. From here, you can store any Airflow variables or connections as secrets on Secret Manager and use them in your project.


# Configure a Hashicorp Vault secrets backend on Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/secrets-backend-hashicorp

Use Hashicorp Vault as a secrets backend for storing Airflow variables and connections with Astro Private Cloud.

In this section, you'll learn how to use [Hashicorp Vault](https://www.vaultproject.io/) as a secrets backend for both local development and on Astro Private Cloud. To do this, you will:

* Create an AppRole in Vault which grants Astronomer minimal required permissions.
* Write a test Airflow variable or connection as a secret to your Vault server.
* Configure your Astro project to pull the secret from Vault.
* Test the backend in a local environment.
* Deploy your changes to Astro Private Cloud.

## Prerequisites

* A [Deployment](/docs/astro-private-cloud/v-2-x/configure-deployment) on Astronomer.
* [The Astro CLI](/docs/cli/v1.43/install-cli).
* A [Hashicorp Vault server](https://learn.hashicorp.com/tutorials/vault/getting-started-dev-server?in=vault/getting-started).
* An Astro project initialized with `astro dev init`.
* [The Vault CLI](https://www.vaultproject.io/docs/install).
* Your Vault Server's URL. If you're using a local server, this should be `http://127.0.0.1:8200/`.

If you don't already have a Vault server deployed but would like to test this feature, Astronomer recommends that you either:

* Sign up for a Vault trial on [Hashicorp Cloud Platform (HCP)](https://cloud.hashicorp.com/products/vault) or
* Deploy a local Vault server. See [Starting the server](https://learn.hashicorp.com/tutorials/vault/getting-started-dev-server?in=vault/getting-started) in Hashicorp documentation.

### Step 1: Create a policy and AppRole in Vault

To use Vault as a secrets backend, Astronomer recommends configuring a Vault AppRole with a policy that grants only the minimum necessary permissions for Astro Private Cloud. To do this:

1. [Create a Vault policy](https://www.vaultproject.io/docs/concepts/policies) with the following permissions:

   ```hcl wrap theme={null}
   path "secret/data/variables/*" {
     capabilities = ["read", "list"]
   }

   path "secret/data/connections/*" {
     capabilities = ["read", "list"]
   }
   ```

2. [Create a Vault AppRole](https://www.vaultproject.io/docs/auth/approle) and attach the policy you just created to it.

3. Retrieve the `role-id` and `secret-id` for your AppRole by running the following commands:

   ```sh wrap theme={null}
   vault read auth/approle/role/<your-approle>/role-id
   vault write -f auth/approle/role/<your-approle>/secret-id
   ```

   Save these values for Step 3.

#### Step 2: Write an Airflow variable or connection to Vault

To test whether your Vault server is set up properly, create a test Airflow variable or connection to store as a secret.

To store an Airflow variable in Vault as a secret, run the following Vault CLI command with your own values:

```sh wrap theme={null}
vault kv put secret/variables/<your-variable-key> value=<your-variable-value>
```

To store a connection in Vault as a secret, run the following Vault CLI command with your own values:

```sh wrap theme={null}
vault kv put secret/connections/<your-connection-id> conn_uri=<connection-type>://<connection-login>:<connection-password>@<connection-host>:5432
```

To confirm that your secret was written to Vault successfully, run:

```sh wrap theme={null}
# For variables
vault kv get secret/variables/<your-variable-key>
# For connections
vault kv get secret/connections/<your-connection-id>
```

#### Step 3: Set up Vault locally

In your Astro project, add the [Hashicorp Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-hashicorp/stable/index.html) to your project by adding the following to your `requirements.txt` file:

```text wrap theme={null}
apache-airflow-providers-hashicorp
```

Then, add the following environment variables to your `Dockerfile`:

```docker wrap theme={null}
# Make sure to replace `<your-approle-id>` and `<your-approle-secret>` with your own values.
ENV AIRFLOW__SECRETS__BACKEND=airflow.providers.hashicorp.secrets.vault.VaultBackend
ENV AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_path": "connections", "variables_path": "variables", "config_path": null, "url": "http://host.docker.internal:8200", "auth_type": "approle", "role_id":"<your-approle-id>", "secret_id":"<your-approle-secret>"}
```

This tells Airflow to look for variable and connection information at the `secret/variables/*` and `secret/connections/*` paths in your Vault server. In the next step, you'll test this configuration in a local Airflow environment.

<Danger>
  If you want to deploy your project to a hosted Git repository before deploying to Astro Private Cloud, be sure to save `<your-approle-id>` and `<your-approle-secret>` securely. Astronomer recommends adding them to your project's [`.env` file](/docs/cli/v1.43/private-python-packages) and specifying this file in `.gitignore`.

  When you deploy to Astro Private Cloud in Step 4, you can set these values as secrets in the UI.
</Danger>

<Info>By default, Airflow uses `"kv_engine_version": 2`, but this secret was written using v1. You can change this to accommodate how you write and read your secrets.</Info>

For more information on the Airflow provider for Hashicorp Vault and how to further customize your integration, see the [Apache Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers-hashicorp/stable/_api/airflow/providers/hashicorp/hooks/vault/index.html).

#### Step 4: Run an example Dag to test Vault locally

To test Vault, write a simple Dag which calls your test secret and add this Dag to your project's `dags` directory. For example, you can use the following Dag to print the value of a variable to your task logs:

```python wrap theme={null}
from airflow import DAG
from airflow.hooks.base import BaseHook
from airflow.models import Variable
from airflow.operators.python import PythonOperator
from datetime import datetime

def print_var():
    my_var = Variable.get("<your-variable-key>")
    print(f'My variable is: {my_var}')

with DAG('example_secrets_dags', start_date=datetime(2022, 1, 1), schedule=None) as dag:

  test_task = PythonOperator(
      task_id='test-task',
      python_callable=print_var,
)
```

Once you've added this Dag to your project:

1. Run `astro dev restart` to push your changes to your local Airflow environment.
2. In the Airflow UI (`http://localhost:8080/admin/`), trigger your new Dag.
3. Click **test-task** > **View Logs**. If you ran the example Dag above, you should see the contents of your secret in the task logs:

   ```text wrap theme={null}
   {logging_mixin.py:109} INFO - My variable is: my-test-variable
   ```

Once you confirm that the setup was successful, you can delete this example Dag.

#### Step 5: Deploy on Astro Private Cloud

Once you've confirmed that the integration with Vault works locally, you can complete a similar set up with a Deployment on Astro Private Cloud.

1. In the Astro Private Cloud UI, add the same environment variables found in your `Dockerfile` to your Deployment [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables). Specify `AIRFLOW__SECRETS__BACKEND_KWARGS` as **secret** to ensure that your Vault credentials are stored securely.
2. In your Astro project, delete the environment variables from your `Dockerfile`.
3. [Deploy your changes](/docs/astro-private-cloud/v-2-x/deploy-code-overview#full-image-deploy) to Astro Private Cloud.

Now, any Airflow variable or connection that you write to your Vault server can be successfully accessed and pulled by any Dag in your Deployment on Astro Private Cloud.


# Manage teams via API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/team-management-api

Create and manage teams programmatically using the APC API GraphQL API.

Teams in Astro Private Cloud let you group users and assign permissions collectively. You can manage teams locally or sync them from an Identity Provider (IdP). To configure IdP group sync, see [Import identity provider (IdP) groups](/docs/astro-private-cloud/v-2-x/import-idp-groups).

## Prerequisites

* Access to the APC API GraphQL API endpoint for your Astro Private Cloud installation.
* A valid authentication token. See [Authenticate to the APC API](/docs/astro-private-cloud/v-2-x/houston-api-authenticate).
* The UUIDs of any users, workspaces, or Deployments you want to reference.

## Team types

| Type        | Provider                                    | User Management      | Use Case                               |
| ----------- | ------------------------------------------- | -------------------- | -------------------------------------- |
| Local teams | `local`                                     | Manual add/remove    | Local authentication, custom groupings |
| IdP teams   | `okta`, `auth0`, `microsoft`, `ida`, `adfs` | Auto-synced from IdP | Enterprise SSO integration             |

## Create a team

### Create local team

```graphql wrap theme={null}
mutation {
  createTeam(
    name: "Data Engineering"
    description: "Data engineering team"
    provider: "local"
    userIds: ["<user-uuid-1>", "<user-uuid-2>"]
  ) {
    team {
      id
      name
      provider
      description
      users {
        id
        username
      }
    }
    message
  }
}
```

### Create IdP team

[IdP group sync](/docs/astro-private-cloud/v-2-x/import-idp-groups) automatically creates IdP teams, but you can also create them manually:

```graphql wrap theme={null}
mutation {
  createTeam(
    name: "engineering-group"
    description: "Synced from Okta"
    provider: "okta"
  ) {
    team {
      id
      name
      provider
    }
    message
  }
}
```

<Note>
  You can't assign users to IdP teams at creation time. The IdP syncs users to the team.
</Note>

### Parameters

| Parameter     | Type   | Required | Description                                                    |
| ------------- | ------ | -------- | -------------------------------------------------------------- |
| `name`        | String | Yes      | Team name (unique per provider)                                |
| `description` | String | No       | Team description                                               |
| `provider`    | String | No       | `local` (default), `okta`, `auth0`, `microsoft`, `ida`, `adfs` |
| `userIds`     | \[ID]  | No       | User UUIDs (local teams only)                                  |

## Update a team

### Update team details

```graphql wrap theme={null}
mutation {
  updateTeam(
    id: "<team-uuid>"
    newName: "Platform Engineering"
    description: "Updated description"
  ) {
    team {
      id
      name
      description
    }
    message
  }
}
```

### Add users to local team

```graphql wrap theme={null}
mutation {
  updateTeam(
    id: "<team-uuid>"
    addUserIds: ["<user-uuid-3>", "<user-uuid-4>"]
  ) {
    team {
      id
      users {
        id
        username
      }
    }
  }
}
```

### Remove users from local team

```graphql wrap theme={null}
mutation {
  updateTeam(
    id: "<team-uuid>"
    removeUserIds: ["<user-uuid-1>"]
  ) {
    team {
      id
      users {
        id
        username
      }
    }
  }
}
```

### Replace all users

```graphql wrap theme={null}
mutation {
  updateTeam(
    id: "<team-uuid>"
    teamUserIds: ["<user-uuid-5>", "<user-uuid-6>"]
  ) {
    team {
      users {
        id
        username
      }
    }
  }
}
```

### Update by name (alternative)

Team names are unique per provider, not globally. You must include `provider` alongside `name` to uniquely identify a team.

```graphql wrap theme={null}
mutation {
  updateTeam(
    name: "Data Engineering"
    provider: "local"
    newName: "Data Platform"
  ) {
    team {
      id
      name
    }
  }
}
```

## Remove a team

### Remove by UUID

```graphql wrap theme={null}
mutation {
  removeTeam(teamUuid: "<team-uuid>") {
    id
    name
  }
}
```

### Remove by name and provider

```graphql wrap theme={null}
mutation {
  removeTeam(
    name: "Data Engineering"
    provider: "local"
  ) {
    id
    name
  }
}
```

<Note>
  You can only remove IdP teams that have no attached users.
</Note>

## Query teams

### Get single team

```graphql wrap theme={null}
query {
  team(teamUuid: "<team-uuid>") {
    id
    name
    provider
    description
    createdAt
    updatedAt
    users {
      id
      username
      emails {
        address
      }
    }
    roleBindings {
      role
      workspace {
        id
        label
      }
      deployment {
        id
        label
      }
    }
  }
}
```

### List teams with search

<Note>
  `searchPhrase` requires a minimum of three characters.
</Note>

```graphql wrap theme={null}
query {
  paginatedTeams(
    take: 20
    pageNumber: 1
    searchPhrase: "engineering"
  ) {
    teams {
      id
      name
      provider
      users {
        id
      }
    }
    count
  }
}
```

### List workspace teams

```graphql wrap theme={null}
query {
  workspaceTeams(workspaceUuid: "<workspace-uuid>") {
    id
    name
    roleBindings {
      role
    }
  }
}
```

### List deployment teams

```graphql wrap theme={null}
query {
  deploymentTeams(deploymentUuid: "<deployment-uuid>") {
    id
    name
    roleBindings {
      role
    }
  }
}
```

## Assign team roles

### Assign a team role

### Assign a team to a workspace

<Note>
  If you omit `role`, the team defaults to `WORKSPACE_VIEWER`.
</Note>

```graphql wrap theme={null}
mutation {
  workspaceAddTeam(
    teamUuid: "<team-uuid>"
    workspaceUuid: "<workspace-uuid>"
    role: WORKSPACE_EDITOR
  ) {
    id
    label
  }
}
```

Assign workspace and deployment roles in a single mutation:

```graphql wrap theme={null}
mutation {
  workspaceAddTeam(
    teamUuid: "<team-uuid>"
    workspaceUuid: "<workspace-uuid>"
    role: WORKSPACE_VIEWER
    deploymentRoles: [
      { deploymentId: "<deployment-uuid-1>", role: DEPLOYMENT_ADMIN }
      { deploymentId: "<deployment-uuid-2>", role: DEPLOYMENT_EDITOR }
    ]
  ) {
    id
  }
}
```

### Assign a team to a Deployment

```graphql wrap theme={null}
mutation {
  deploymentAddTeamRole(
    teamUuid: "<team-uuid>"
    deploymentUuid: "<deployment-uuid>"
    role: DEPLOYMENT_EDITOR
  ) {
    id
    role
  }
}
```

### Update a team's role

### Update a team's workspace role

```graphql wrap theme={null}
mutation {
  workspaceUpdateTeamRole(
    teamUuid: "<team-uuid>"
    workspaceUuid: "<workspace-uuid>"
    role: WORKSPACE_ADMIN
  )
}
```

### Update a team's Deployment role

```graphql wrap theme={null}
mutation {
  deploymentUpdateTeamRole(
    teamUuid: "<team-uuid>"
    deploymentUuid: "<deployment-uuid>"
    role: DEPLOYMENT_ADMIN
  ) {
    id
    role
  }
}
```

### Remove a team's role

### Remove a team from a workspace

```graphql wrap theme={null}
mutation {
  workspaceRemoveTeam(
    teamUuid: "<team-uuid>"
    workspaceUuid: "<workspace-uuid>"
  ) {
    id
  }
}
```

### Remove a team from a Deployment

```graphql wrap theme={null}
mutation {
  deploymentRemoveTeamRole(
    teamUuid: "<team-uuid>"
    deploymentUuid: "<deployment-uuid>"
  ) {
    id
  }
}
```

## Available roles

### Workspace roles

| Role               | Permissions                                 |
| ------------------ | ------------------------------------------- |
| `WORKSPACE_ADMIN`  | Full Workspace control, manage users/teams  |
| `WORKSPACE_EDITOR` | Create/manage Deployments, service accounts |
| `WORKSPACE_VIEWER` | View Workspace and Deployment details       |

### Deployment roles

| Role                | Permissions                            |
| ------------------- | -------------------------------------- |
| `DEPLOYMENT_ADMIN`  | Full Deployment control, manage access |
| `DEPLOYMENT_EDITOR` | Deploy code, manage configuration      |
| `DEPLOYMENT_VIEWER` | View Deployment details                |

## Configuration

### Enable local teams

```yaml wrap theme={null}
auth:
  local:
    teams:
      enabled: true
```

### Enable IdP group sync

For full setup instructions, see [Import identity provider (IdP) groups](/docs/astro-private-cloud/v-2-x/import-idp-groups).

```yaml wrap theme={null}
auth:
  openidConnect:
    idpGroupsImportEnabled: true
```

## Error handling

| Error                              | Cause                         | Resolution                                                  |
| ---------------------------------- | ----------------------------- | ----------------------------------------------------------- |
| `LocalTeamManagementDisabledError` | Local teams not enabled       | Enable in Helm values                                       |
| `IDPTeamManagementDisabledError`   | IdP groups import disabled    | Enable IdP group sync                                       |
| `DuplicateTeamError`               | Team name exists for provider | Use unique name                                             |
| `DuplicateRoleBindingError`        | Team already has role         | Update existing role instead                                |
| `InvalidTeamProviderError`         | Unsupported provider value    | Use `local`, `okta`, `auth0`, `microsoft`, `ida`, or `adfs` |
| `ResourceNotFoundError`            | Team/user not found           | Verify UUIDs                                                |

## Best practices

* Use IdP teams for enterprise SSO environments.
* Use local teams for custom access groups.
* Assign Workspace roles before Deployment roles.
* Use Viewer roles as default and escalate as needed.
* Audit team membership regularly.

## Related documentation

* [Import IdP groups](/docs/astro-private-cloud/v-2-x/import-idp-groups)
* [Manage user permissions](/docs/astro-private-cloud/v-2-x/manage-permissions)
* [APC API](/docs/astro-private-cloud/v-2-x/houston-api)


# Use a third-party ingress controller
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/third-party-ingress-controllers

Use a pre-existing ingress controller on Astro Private Cloud.

By default, Astro Private Cloud comes built-in with a default ingress controller to provide access to Astro services running in the cluster. The default ingress controller can co-exist with any other ingress controllers on the cluster.

Using the default ingress controller is the best choice for most organizations, but you might need to use a pre-existing ingress controller exclusively due to regulatory and compliance requirements. This guide provides steps for configuring your own ingress controller to use with the Astronomer platform.

## Step 1: Review general requirements for third-party ingress controllers

To use a third-party ingress-controller with Astro Private Cloud:

* Your ingress controller must service ingresses from the Astronomer Platform namespace, as well as all namespaces that host Airflow.
* Ingresses should work in newly created namespaces prior to installing Astro Private Cloud.
* Your third-party ingress controller must be able to resolve the DNS entries associated with your APC installation.
* Your third-party ingress controller must support SSL connections on port 443 and must present a certificate valid for the relevant hostnames. For a Unified mode installation, this should include a certificate with a wildcard entry for `*.{BASE_DOMAIN}` or the following specific hostnames:

Control Plane components

* `app.BASE_DOMAIN`
* `deployments.BASE_DOMAIN`
* `houston.BASE_DOMAIN`
* `grafana.BASE_DOMAIN`
* `prometheus.BASE_DOMAIN`
* `alertmanager.BASE_DOMAIN`

Data Plane components

* `<Dataplane_Prefix>.BASE_DOMAIN`
* `registry.<Dataplane_Prefix>.BASE_DOMAIN`
* `commander.<Dataplane_Prefix>.BASE_DOMAIN`
* `prometheus.<Dataplane_Prefix>.BASE_DOMAIN`
* `prom-proxy.<Dataplane_Prefix>.BASE_DOMAIN`
* `elasticsearch.<Dataplane_Prefix>.BASE_DOMAIN`

For a `control` mode installation:

* `app.BASE_DOMAIN`
* `houston.BASE_DOMAIN`
* `grafana.BASE_DOMAIN`
* `prometheus.BASE_DOMAIN`

For a `data` mode installation:

* `deployments.DOMAIN_PREFIX.BASE_DOMAIN`
* `registry.DOMAIN_PREFIX.BASE_DOMAIN`
* `prometheus.DOMAIN_PREFIX.BASE_DOMAIN`
* `elasticsearch.DOMAIN_PREFIX.BASE_DOMAIN`

If the certificates of the third-party ingress controller are signed by a private certificate authority:

* The third-party ingress controller must be configured to trust your private CA (as per the documentation of your ingress controller).
* The CA's public certificate must be stored as a Kubernetes secret in the Astronomer namespace.

If a private certificate authority is used to sign the certificate contained in the `global.tlsSecret` value in your `values.yaml` file, the third-party ingress controller must recognize the CA signing `global.tlsSecret` as valid. Typically, this is done by either:

* Signing the secret used in `global.tlsSecret` with a private CA that's already trusted by the custom ingress controller (typically the same CA used to sign the certificates being used by the ingress controller).
* Explicitly configuring your custom ingress controller to trust the CA used when generating the certificate contained in `global.tlsSecret`.

## Step 2: Verify your ingress controller is supported

To complete this setup, you need to supply your own ingress controller. Astronomer fully supports the following ingress controllers:

* OpenShift
* Kong
* HAProxy
* Ingress-nginx
* Traefik
* Contour

If you want to use an ingress controller that isn't listed here, contact your Astronomer representative.

<a />

## Step 3: (OpenShift Only) Perform required configuration to your Kubernetes environment

If not using OpenShift, skip this step.

OpenShift's standard ingress controller restricts hostname use to a single namespace, which isn't a compatible setting with Astro Private Cloud. You can disable this setting for the default IngressController instance using the following command:

```sh wrap theme={null}
kubectl -n openshift-ingress-operator patch ingresscontroller/default --patch '{"spec":{"routeAdmission":{"namespaceOwnership":"InterNamespaceAllowed"}}}' --type=merge
```

Alternatively, see [Use OpenShift Ingress Sharding](https://docs.openshift.com/container-platform/4.15/networking/ingress-sharding.html) to create an additional Ingress instance with the required `routeAdmission` policy.
For more information, including information about security implications for multi-tenant clusters, see the [OpenShift Ingress operator documentation](https://docs.openshift.com/container-platform/4.15/networking/ingress-operator.html).

OpenShift clusters with multi-tenant isolation enabled need to explicitly allow traffic from the ingress controller's namespace to services associated with ingresses in other namespaces.

Additionally, you must configure Astronomer to explicitly declare the SSL termination policy for the ingress resources it manages.

To do so, add the following configuration to your `values.yaml` file:

```text wrap theme={null}
global:
  extraAnnotations:
    kubernetes.io/ingress.class: openshift-default
    route.openshift.io/termination: "edge"
```

Don't deploy this change at this time as later steps of this document walk you through making additional required changes to `values.yaml`.

<Info>
  Only Ingress objects with the annotation `route.openshift.io/termination: "edge"` are supported for generating routes in OpenShift 4.11 and later.
  Other termination types are no longer supported for automatic route generation.

  If you're on an older version of OpenShift, route creation should be done manually.
</Info>

For more information, see the [OpenShift documentation](https://docs.openshift.com/container-platform/4.15/networking/network_policy/about-network-policy.html) on configuring network policy.

## Step 4: Mark the `astronomer-tls` secret for replication

Most third-party ingress controllers require the secret name to be replicated into each Airflow namespace. This name can be custom-set in your global configs, but the following examples use the `secretName`, `astronomer-tls`.

Annotate the secret and set `"astronomer.io/commander-sync"` to `platform=<astronomer platform release name>`. For example:

```sh wrap theme={null}
kubectl -n <astronomer platform namespace> annotate secret astronomer-tls "astronomer.io/commander-sync"="platform=astronomer"
```

## Step 5: Set required settings in `values.yaml`

Enable `authSidecar` and disable the Astronomer integrated ingress controller.

```yaml wrap theme={null}
global:
  nginx:
    enabled: false

  enabled: true
  # You can custom-name the tlsSecret when using a third-party ingress controller. The following example uses astronomer-tls.
  tlsSecret: astronomer-tls
```

## Step 6: Perform required configuration for your specific ingress controller

### Required configuration for nginx

If you're using an nginx ingress controller, add the following configuration to your `values.yaml` file:

```yaml wrap theme={null}
global:
  extraAnnotations:
    nginx.ingress.kubernetes.io/proxy-body-size: 0
```

This setting disables Nginx's maximum allowed upload size, which prevents HTTP 413 (Request Entity Too Large) error and allows the Astro CLI to properly deploy Dags to Astro Private Cloud's internal registry.

### Required configuration for Traefik

If you're using a Traefik ingress controller, add the following configuration to your `values.yaml` file:

```yaml wrap theme={null}
global:
  extraAnnotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls: "true"
```

<Note>
  Depending on the version of Traefik, upgrading from using the default ingress controller to a Traefik controller might cause issues. If you are upgrading a platform that used the built-in ingress controller, manually delete the Astronomer Platform Ingress objects in the Astronomer Platform namespace before updating your `values.yaml` file. You can do so using the following commands:

  ```bash wrap theme={null}
  kubectl -n <your-platform-namespace> delete ingress -l release=<your-platform-release-name>
  helm upgrade --install -f values.yaml --version=<your-platform-version> --namespace=<your-platform-namespace> <your-platform-release-name> astronomer/astronomer
  ```
</Note>

### Required configuration for Contour

Contour ships with support for WebSockets disabled by default. To use a Contour ingress controller, explicitly enable WebSocket support for the APC API's `/ws` prefix by creating an `HTTPProxy` object in the Astronomer platform namespace. To do so:

1. Create a file named `proxy.yaml` and add the following to it:

   ```yaml wrap theme={null}
   apiVersion: projectcontour.io/v1
   kind: HTTPProxy
   metadata:
     name: houston
     annotations:
       kubernetes.io/ingress.class: contour
   spec:
     virtualhost:
       fqdn: houston.<base-domain>
       tls:
         secretName: astronomer-tls
     routes:
       - conditions:
         - prefix: /ws
         enableWebsockets: true
         services:
           - name: astronomer-houston
             port: 8871
   ```

2. Apply the file to your platform namespace:

   ```bash wrap theme={null}
   kubectl apply -n <your-platform-namespace> -f proxy.yaml
   ```

<Info>
  Depending on the version of Contour, upgrading from using the default ingress controller to a Contour controller might cause issues. If you are upgrading a platform that used the built-in ingress controller, manually delete the Astronomer Platform Ingress objects in the Astronomer Platform namespace before updating your `values.yaml` file. You can do so using the following commands:

  ```sh wrap theme={null}
  kubectl -n <your-platform-namespace> delete ingress -l release=<your-platform-release-name>
  helm upgrade --install -f values.yaml --version=<your-platform-version> --namespace=<your-platform-namespace> <your-platform-release-name> astronomer/astronomer
  ```
</Info>

### Required configuration OpenShift ingress controller

See [Required Environment Configuration for OpenShift](#required-environment-configuration-openshift).

## Step 7: Apply changes with Helm

If performing a new installation, skip this step and don't apply changes until the install guide instructs you to do so.

If this is an existing installation, apply your updated configuration using the following command:

```bash wrap theme={null}
helm upgrade --install -f values.yaml --version=<your-platform-version> --namespace=<your-platform-namespace> <your-platform-release-name> astronomer/astronomer
```


# TLS certificate management
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/tls-certificates

Configure and manage TLS certificates for the APC platform and Deployments.

Astro Private Cloud (APC) uses TLS certificates for secure communication between components and for JWT token signing.

## Self-signed certificate generation

```bash wrap theme={null}
# Generate CA
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 \
  -out ca.crt -subj "/CN=APC-CA"

# Generate server certificate
openssl genrsa -out tls.key 4096
openssl req -new -key tls.key -out tls.csr \
  -subj "/CN=*.your-domain.com"

# Sign certificate with SAN
openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out tls.crt -days 365
```

### Create Kubernetes secret

```bash wrap theme={null}
kubectl create secret tls platform-tls \
  --cert=tls.crt \
  --key=tls.key \
  -n astronomer
```

## APC API JWT certificates

The APC API uses auto-generated JWT certificates to sign and verify authentication tokens. These certificates are created during installation.

The `regenerateCaEachUpgrade` flag controls whether APC regenerates the APC API certificate authority (CA) on each platform upgrade. This flag defaults to `false`:

```yaml wrap theme={null}
houston:
  regenerateCaEachUpgrade: false
```

<Warning>
  Setting `regenerateCaEachUpgrade` to `true` regenerates the CA on every upgrade, which invalidates all existing JWT tokens and forces all users and service accounts to re-authenticate.
</Warning>

Astronomer recommends keeping this value set to `false` unless you have a specific security requirement to rotate the CA regularly.

## Certificate sync

Certificate syncing in APC operates at two levels:

### Control plane to data plane

Control plane to data plane certificate sync occurs only during data plane install or upgrade. During this process, the platform calls the APC API endpoint to decode the certificates and annotates them with the Config Syncer label to propagate the necessary secrets to Airflow namespaces.

### Within a cluster (Config Syncer)

Config Syncer is a CronJob that propagates annotated secrets from the platform namespace to Airflow Deployment namespaces within the same cluster. It runs on a configurable schedule to keep secret contents in sync across namespaces.

```yaml wrap theme={null}
astronomer:
  configSyncer:
    enabled: true
    schedule: "*/5 * * * *"
```

## Ingress TLS

### Use an existing certificate

```yaml wrap theme={null}
global:
  tlsSecret: platform-tls
```

## Certificate renewal

### Renew certificates manually

```bash wrap theme={null}
# Update secret
kubectl create secret tls platform-tls \
  --cert=new-tls.crt \
  --key=new-tls.key \
  -n astronomer \
  --dry-run=client -o yaml | kubectl apply -f -

# Restart ingress
kubectl rollout restart deployment nginx -n astronomer
```

### Check certificate expiry

```bash wrap theme={null}
kubectl get secret platform-tls -n astronomer \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -enddate
```

## Best practices

* Use cert-manager for automatic renewal.
* Monitor certificate expiration with alerts.
* Keep `regenerateCaEachUpgrade: false` to preserve sessions.
* Use strong key sizes (4096-bit RSA).


# Move the Astro Runtime Operator under APC management (Preview)
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/transition-operator-to-apc

Hand the Astro Runtime Operator you installed yourself over to Astro Private Cloud, so you no longer install and upgrade it separately.

If you adopted your Airflow Deployments into Astro Private Cloud (APC) but still install and upgrade the Astro Runtime Operator yourself, you can hand the operator over to APC. After that, the platform installs and upgrades it with the rest of APC, and you stop maintaining it separately.

<Note>
  **Astro Private Cloud 2.1**

  This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
</Note>

<Info>
  **Preview**

  Moving the operator is in Preview. APC does not yet manage everything the Astro Runtime Operator can express, and operator mode is still missing some Helm-mode features. See [Feature support](/docs/astro-private-cloud/v-2-x/airflow-operator-mode#feature-support) for what it does not cover yet.
</Info>

This is a different change from adopting Deployments. Adoption brings your Airflow Deployments under APC management and leaves the operator alone. See [Adopt operator Deployments](/docs/astro-private-cloud/v-2-x/adopt-operator-deployments).

## Before you begin

You need:

* Your self-installed operator upgraded to version 1.6.1 or above before you start. This is required so that Helm marks the Airflow CRDs to be kept and does not delete them during this procedure. Deleting a CRD deletes every Airflow Deployment on the cluster.
* Airflow Deployments running under an operator you installed yourself, on a cluster registered as an APC data plane.
* Permission to run `helm` against both the operator's release and the APC platform release, and cluster-level permission to annotate CRDs.
* A maintenance window. This is a cutover rather than a gradual migration, and it includes a period during which changes to Airflow Deployments are not applied.

Collect these values before you start, because every step uses them:

| Value                                         | How to find it                                                                                     |
| --------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| Operator release name and namespace           | `helm list -A \| grep airflow-operator`                                                            |
| APC platform release name and namespace       | `helm list -A \| grep astronomer`                                                                  |
| Whether the operator release created the CRDs | `helm get manifest <operator-release> -n <operator-namespace> \| grep -c CustomResourceDefinition` |

Airflow keeps running throughout. The operator reconciles your Airflow custom resources into Kubernetes workloads, and those workloads are ordinary Deployments and StatefulSets that Kubernetes keeps running without it.

## Step 1: Confirm your Airflow CRDs are protected

The `keep` resource policy is required so that no `helm` command can delete the Airflow CRDs during this procedure. Deleting a CRD deletes every custom resource of that kind, which is every Airflow Deployment on the cluster. Operator 1.6.1 and above sets this policy for you. Confirm all 13 CRDs carry the `keep` annotation:

```bash theme={null}
kubectl get crd -o name | grep airflow.apache.org \
  | xargs kubectl get -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.helm\.sh/resource-policy}{"\n"}{end}'
```

Every row should end in `keep`. Helm honors the annotation both when an upgrade would remove a resource and when an uninstall would delete it. The annotation is metadata only: it does not restart anything and does not change how the operator behaves.

If any row is missing `keep`, stop and contact [Astronomer support](/docs/astro-private-cloud/v-2-x/support) before going further.

## Step 2: Record what you have

You need this to verify the result, and to spot immediately if anything is lost:

```bash theme={null}
kubectl get airflows.airflow.apache.org -A
kubectl get crd -o name | grep airflow.apache.org | wc -l
```

Keep the output. The Deployment list and the CRD count must be identical at the end.

## Step 3: Remove your operator installation

```bash theme={null}
helm uninstall <operator-release> -n <operator-namespace>
```

Confirm the CRDs and your Deployments survived, using the output you kept in Step 2:

```bash theme={null}
kubectl get airflows.airflow.apache.org -A
kubectl get crd -o name | grep airflow.apache.org | wc -l
```

## Step 4: Enable APC's operator and transfer CRD ownership

Turn APC's operator on and let the platform release take ownership of the Airflow CRDs in a single upgrade. The APC platform chart also ships these CRDs, and Helm does not take ownership of objects that belong to another release, so pass `--take-ownership` to have the platform release adopt them. CRD updates then arrive with the platform chart, like every other APC resource.

For the configuration that enables operator support, see [Enable operator support](/docs/astro-private-cloud/v-2-x/airflow-operator-mode#enable-operator-support). Apply that configuration and upgrade the platform release with `--take-ownership`:

```bash theme={null}
helm upgrade <platform-release> <platform-chart> \
  -n <platform-namespace> \
  --take-ownership \
  -f values.yaml
```

Use the platform release name, namespace, and chart you collected in [Before you begin](#before-you-begin). See [Apply platform configuration](/docs/astro-private-cloud/v-2-x/apply-platform-config) for how to apply the upgrade.

## Step 5: Verify

Exactly one operator is running:

```bash theme={null}
kubectl get deployment -A | grep -E 'aocm|airflow-operator-controller-manager'
```

One mutating and one validating webhook configuration are registered:

```bash theme={null}
kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration | grep airflow
```

Your Deployments and CRDs are all still present, matching Step 2:

```bash theme={null}
kubectl get airflows.airflow.apache.org -A
kubectl get crd -o name | grep airflow.apache.org | wc -l
```

## Step 6: Adopt Deployments and confirm a change applies

[Adopt the Deployments](/docs/astro-private-cloud/v-2-x/adopt-operator-deployments), then confirm a change reaches the running Airflow. That proves the new operator is reconciling.

## If something goes wrong

* **The platform upgrade fails with an ownership error on the CRDs.** The upgrade ran without `--take-ownership`. Re-run Step 4 with the flag, checking that the release name and namespace match the platform release exactly.
* **Changes to Airflow Deployments are rejected.** A webhook configuration is registered with no operator behind it. Confirm Step 4 completed and the operator pod is running.
* **No operator is running and the upgrade failed.** Airflow keeps running. Fix the upgrade and re-run it. Nothing reconciles in the meantime, so avoid making changes to Airflow Deployments until it succeeds.
* **An Airflow Deployment is missing.** Stop and contact [Astronomer support](/docs/astro-private-cloud/v-2-x/support). Do not run further `helm` commands.

## Limitations

* **It is a cutover, not a phased migration.** Every Airflow Deployment on the cluster changes operator at the same time. There is no supported way to move one namespace at a time.
* **There is a window with no operator.** Between Step 3 and Step 4, nothing reconciles your Airflow custom resources. Running Airflow is unaffected, but changes are not applied.
* **Moving back is not supported.** Once CRD ownership sits with the platform release, returning to a self-managed operator is not a documented path.

## Related documentation

* [Adopt operator Deployments](/docs/astro-private-cloud/v-2-x/adopt-operator-deployments)
* [Airflow Operator mode](/docs/astro-private-cloud/v-2-x/airflow-operator-mode)
* [Apply platform configuration](/docs/astro-private-cloud/v-2-x/apply-platform-config)
* [Install a data plane cluster](/docs/astro-private-cloud/v-2-x/install-data-plane)
* [Astronomer support](/docs/astro-private-cloud/v-2-x/support)


# Trigger a data plane failover
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/trigger-data-plane-failover

Use the Astro Private Cloud UI to trigger a failover that moves Airflow Deployments from one data plane cluster to another.

Use this guide to trigger a data plane failover from the Astro Private Cloud (APC) UI. A failover moves all Airflow Deployments from a source cluster to a destination cluster. The process runs asynchronously — after you submit the request, APC handles execution without further input.

For a conceptual overview of how failover works, see [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover). To enable the feature before triggering it, see [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover).

## Prerequisites

* Data plane failover is enabled on your APC installation. See [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover).
* The source cluster has `failoverEnabled` set to `true`. If the **Trigger Failover** button appears dimmed with the message "Failover isn't enabled for this cluster", this means one of the feature configuration requirements hasn't been satisfied. Double check the configuration instructions, running pods, and pod logs.
* At least one healthy destination cluster is registered with the same APC control plane as the source cluster.
* `PushSecret` resources on the source cluster report a `Synced` status, and the active and inactive connection secrets resolve successfully. See [Verify secret replication before a failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover#verify-secret-replication-before-a-failover).
* You have permission to update clusters in the APC UI.

## Trigger a failover

<Steps>
  <Step title="Open cluster details">
    In the APC UI, go to the cluster list and click the source cluster — the cluster you want to fail over *from*.
  </Step>

  <Step title="Click trigger failover">
    In the top-right corner of the cluster details page, click **Trigger Failover**.

    If the button appears dimmed, the source cluster isn't eligible for failover. See the prerequisites above.
  </Step>

  <Step title="Select a destination cluster">
    In the **Destination Cluster** dropdown, select the cluster you want to fail over *to*. The dropdown lists only clusters that APC considers valid targets for the source cluster.

    If no destinations appear, verify that at least one other cluster is registered, healthy, and reachable from the control plane.
  </Step>

  <Step title="Select a failover mode">
    In the **Mode** dropdown, select one of the following:

    * **Controlled**: Drains Airflow Deployment components on the source cluster and waits for in-flight tasks to finish, up to a configured timeout, before promoting the destination. Use for planned maintenance or migrations where task loss isn't acceptable.
    * **Forced**: Promotes the destination cluster immediately without waiting for source Deployments to drain. Use this mode when the source cluster is unreachable or when speed takes priority over task completion.
  </Step>

  <Step title="Submit the request">
    Click **Trigger Failover** to submit the request.

    APC queues the request and begins execution asynchronously. A success notification confirms the request was accepted. If the request fails immediately, an error message describes the cause.
  </Step>
</Steps>

## Monitor failover progress

After you submit a failover request, APC transitions each Deployment on the source cluster through its own migration state machine. You can monitor progress by checking cluster health status in the APC UI.

The source cluster status transitions to `FAILING_OVER` while the request is in progress. If any Deployment migration fails, the cluster status transitions to `FAILOVER_FAILED`. This status is only visible directly within the Postgres database.

<Note>
  Failover execution continues in the background even if you close the browser
  or navigate away from the cluster details page.
</Note>

## What happens during a failover

For each Deployment on the source cluster, APC:

1. Replicates Airflow secrets — including the fernet key, environment variables, and database credentials — to the destination cluster using the External Secrets Operator.
2. Creates the Airflow namespace on the destination cluster.
3. In **Controlled** mode, drains and deletes Deployments on the source cluster before continuing. In **Forced** mode, this step is skipped.
4. Fences the source database connection to prevent split-brain writes.
5. Creates the Airflow Deployment on the destination cluster with the same configuration, name, namespace, and ID as the source Deployment.

After all Deployments migrate successfully, the failover request transitions to `SUCCEEDED`.

<Note>
  If you want a Dag run to resume after a failover, make sure the Dag sets
  `retries` greater than `0` and that the tasks within it are idempotent.
  Airflow reschedules on the destination cluster any tasks that were running on the source cluster when failover started. Airflow only retries them if the task's retry count allows it.
</Note>

## Related documentation

* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)
* [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover)
* [Configure Hashicorp Vault for data plane failover](/docs/astro-private-cloud/v-2-x/configure-vault-data-plane-failover)
* [Run a failover upgrade](/docs/astro-private-cloud/v-2-x/run-failover-upgrade)
* [Data plane architecture](/docs/astro-private-cloud/v-2-x/data-plane-architecture)


# Troubleshoot data plane registration
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/troubleshoot-data-plane-registration

Resolve common errors when registering a data plane cluster in Astro Private Cloud.

Use this guide to diagnose and resolve errors that occur when you register a data plane cluster in Astro Private Cloud (APC) 2.x.

## "Commander metadata service unavailable"

```text wrap theme={null}
Registration Failed
Cannot fetch dataplane metadata details. Reason: Commander Metadata service unavailable.
```

This error occurs when the control plane APC API can't reach the deployment orchestrator's `/metadata` HTTP endpoint on the data plane during the registration handshake. Registration can't complete until the APC API successfully fetches and validates this metadata.

### How registration works

When you register a data plane, the APC API makes an outbound HTTPS `GET` request to the deployment orchestrator's metadata endpoint and validates the JSON response before creating the cluster record. The failure can occur at any point along that path: DNS resolution, network connectivity, TLS trust, or ingress routing.

### Correct metadata URL format

In APC 2.x, the chart creates two separate ingresses for the deployment orchestrator on the data plane:

| Ingress                      | Hostname                                | Port | Protocol | Purpose                                         |
| ---------------------------- | --------------------------------------- | ---- | -------- | ----------------------------------------------- |
| `commander-api-ingress`      | `commander.<domainPrefix>.<baseDomain>` | 443  | gRPC     | APC API–deployment orchestrator control channel |
| `commander-metadata-ingress` | `<domainPrefix>.<baseDomain>`           | 443  | HTTPS    | Registration metadata endpoint                  |

The metadata URL you enter during registration must point to the metadata ingress (the bare data plane domain). the APC API appends `/metadata` to this URL internally when it makes the fetch call, so don't include the path in the registration form:

```text wrap theme={null}
https://<domainPrefix>.<baseDomain>
```

**Example**: If your control plane base domain is `apc.example.com` and your data plane `domainPrefix` is `dp-01`:

* **Correct**: `https://dp-01.apc.example.com`
* **Incorrect**: `https://commander.dp-01.apc.example.com` (this is the gRPC API ingress and returns a 404)

### Diagnose the error

Work through the following steps in order to isolate the cause.

#### Step 1: Verify the deployment orchestrator is running

On the data plane cluster, confirm the deployment orchestrator is healthy:

```bash wrap theme={null}
kubectl get pods -n <data-plane-namespace> -l component=commander
```

Expected output: `1/1 Running`. If the deployment orchestrator is crashlooping, check its logs:

```bash wrap theme={null}
kubectl logs -n <data-plane-namespace> -l component=commander
```

#### Step 2: Verify DNS resolves to the correct IP

The data plane has its own NGINX ingress controller with its own load balancer IP, separate from the control plane's load balancer. Confirm that the DNS for the data plane domain resolves to the data plane's load balancer, not the control plane's.

Find the data plane's load balancer IP:

```bash wrap theme={null}
kubectl get svc -n <data-plane-namespace> -l component=nginx
```

Look for the `EXTERNAL-IP` on the `LoadBalancer`-type service. In a co-located setup (control plane and data plane on the same cluster), the service name typically includes `-dp-nginx` and has a different IP from the control plane's NGINX service.

Then verify DNS:

```bash wrap theme={null}
dig +short <domainPrefix>.<baseDomain>
```

The returned IP must match the data plane's load balancer `EXTERNAL-IP`. A mismatch (for example, the data plane subdomain pointing to the control plane load balancer IP) causes all requests to return 404 because the control plane's NGINX has no ingress rules for the data plane's hostnames.

<Note>
  **Co-located deployments**

  When the control plane and data plane share a single cluster, they use separate namespaces and separate NGINX load balancers. DNS for `*.<domainPrefix>.<baseDomain>` must be a distinct wildcard record pointing to the data plane's load balancer IP. Verify this record exists and doesn't share the control plane's IP.
</Note>

#### Step 3: Test the metadata endpoint directly

After DNS resolves correctly, confirm the endpoint returns a valid 200 response:

```bash wrap theme={null}
curl -sk -w "\nHTTP Status: %{http_code}\n" \
  "https://<domainPrefix>.<baseDomain>/metadata"
```

A healthy response returns JSON similar to the following:

```json wrap theme={null}
{
  "kubernetesVersion": "v1.32.x",
  "mode": "data",
  "dataplaneChartVersion": "2.x.x",
  "cloudProvider": "GCP",
  "healthStatus": "HEALTHY",
  "baseDomain": "dp-01.apc.example.com",
  "commander": {
    "version": "2.x.x",
    "url": "commander.dp-01.apc.example.com:443",
    "status": "HEALTHY",
    "airflowChartVersion": "2.x.x"
  }
}
```

<Note>
  The `cloudProvider` field may return `"local"` on Kubernetes clusters and certain co-located environments. This is expected and doesn't affect registration.
</Note>

If this request returns a 404, continue to Step 4. If it times out or the connection is refused, skip to Step 5.

#### Step 4: Verify the metadata ingress exists and is configured correctly

Check that the metadata ingress exists in the data plane namespace:

```bash wrap theme={null}
kubectl get ingress -n <data-plane-namespace> | grep metadata
```

Describe it to verify the hostname and path:

```bash wrap theme={null}
kubectl describe ingress -n <data-plane-namespace> <release-name>-commander-metadata-ingress
```

Confirm the following:

* **Host** matches `<domainPrefix>.<baseDomain>` exactly.
* **Path** is `/metadata`.
* **Backend** points to the deployment orchestrator service on port `8880` (the HTTP port, not the gRPC port `50051`).
* **Ingress class** annotation (`kubernetes.io/ingress.class`) matches the data plane's NGINX class name.

If the ingress is missing or misconfigured, re-run `helm upgrade` on the data plane release with the correct `global.plane.domainPrefix` and `global.baseDomain` values.

#### Step 5: Verify network connectivity from the control plane

Even if the endpoint works from your workstation, the APC API on the control plane must also reach it. Test from inside the APC Pod:

```bash wrap theme={null}
kubectl exec -it -n <cp-namespace> deploy/<cp-release>-houston -- sh
```

Inside the Pod, run:

```bash wrap theme={null}
wget -qO- https://<domainPrefix>.<baseDomain>/metadata
```

If this fails, check for firewall rules or network policies blocking HTTPS egress from the control plane to the data plane's load balancer IP.

#### Step 6: Verify TLS certificate trust

If the data plane uses a private certificate authority (CA) or a certificate chain that the APC API doesn't trust, the HTTPS request fails with a TLS error. Test with certificate verification disabled:

```bash wrap theme={null}
# Inside the Houston Pod
wget --no-check-certificate -qO- https://<domainPrefix>.<baseDomain>/metadata
```

If this succeeds but the unmodified request fails, add the data plane's CA certificate to the control plane's `global.privateCaCerts` Helm values and re-deploy.

### Summary checklist

Before retrying registration, verify all of the following:

* deployment orchestrator Pod is `1/1 Running` in the data plane namespace.
* `dig <domainPrefix>.<baseDomain>` returns the data plane's NGINX load balancer IP (not the control plane's).
* `curl https://<domainPrefix>.<baseDomain>/metadata` returns HTTP 200 with valid JSON.
* The metadata URL entered in the registration form is `https://<domainPrefix>.<baseDomain>` (no `/metadata` suffix, the APC API appends that internally).
* The metadata ingress exists and its backend targets the deployment orchestrator's HTTP port (`8880`).
* The APC API on the control plane can reach the data plane's load balancer over HTTPS.

## Related

* [Register a data plane cluster](/docs/astro-private-cloud/v-2-x/register-data-plane)
* [Data plane cluster overview](/docs/astro-private-cloud/v-2-x/overview-data-plane-cluster)
* [Debug an installation](/docs/astro-private-cloud/v-2-x/debug-install)


# Airflow chart compatibility reference for Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/airflow-chart-compatibility

A reference of all adjacent tooling required to run Astro Private Cloud and corresponding version compatibility.

Astro Private Cloud (APC) Deployments use the [Astronomer-distributed Helm chart for Apache Airflow](https://github.com/astronomer/airflow-chart). A Deployment's Airflow chart defines how a Deployment interacts with other components in your cluster.

Use the following table to see the Airflow Helm chart version for each supported version of APC. To view the Airflow Helm chart for an unsupported version of APC, open the default Astronomer Helm chart in the [`astronomer/astronomer` repository](https://github.com/astronomer/astronome/blob/release-2.0/charts/astronomer/values.yaml) and select the **Tag** that corresponds to the unsupported version. The value of `airflowChartVersion` is the Airflow Helm chart version.

| Astro Private Cloud version | Astronomer Airflow Helm chart version |
| --------------------------- | ------------------------------------- |
| 2.0.0                       | TBD                                   |


# Configure metrics
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-metrics

Configure StatsD, OpenTelemetry, and Prometheus metrics for Airflow monitoring.

Astro Private Cloud (APC) provides multiple options for collecting and exporting Airflow metrics including StatsD, OpenTelemetry (OTEL), and Prometheus integration.

## StatsD configuration (default)

StatsD resource limits are managed at the API level via `componentsConfig` and apply to all components — they can't be configured independently per component.

```yaml wrap theme={null}
resources:
  requests:
    cpu: "100m"
    memory: "384Mi"
  limits:
    cpu: "100m"
    memory: "384Mi"
```

### Airflow configuration

```ini wrap theme={null}
[metrics]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
```

## Prometheus integration

```yaml wrap theme={null}
prometheus:
  enabled: true
  retention: 15d
  persistence:
    enabled: true
    size: 100Gi
```

## Grafana dashboards

Access Grafana at:

```text wrap theme={null}
https://grafana.<platform-domain>
```

Pre-built dashboards include:

* Airflow Dag performance
* Task execution metrics
* Scheduler health
* Worker utilization

## Alerting

```yaml wrap theme={null}
alertmanager:
  enabled: true
  config:
    route:
      receiver: 'platform'
    receivers:
      - name: 'platform'
        webhook_configs:
          - url: 'http://houston:8871/v1/alerts'
```

### Built-in alerts

* `AirflowDeploymentUnhealthy`
* `AirflowSchedulerUnhealthy`
* `AirflowTasksPendingIncreasing`

## Key metrics

| Metric                            | Description               |
| --------------------------------- | ------------------------- |
| `airflow_dagrun_duration_seconds` | Dag run duration          |
| `airflow_ti_successes`            | Successful task instances |
| `airflow_ti_failures`             | Failed task instances     |
| `airflow_scheduler_heartbeat`     | Scheduler health          |
| `airflow_executor_queued_tasks`   | Queued task count         |

## Best practices

* Set appropriate retention based on storage capacity.
* Use OTEL for multi-backend export.
* Configure alerts for critical health metrics.
* Monitor task queue depth for scaling needs.


# Configure probes
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-probes

Configure liveness and readiness probes for Astronomer and Airflow components.

In Astro Private Cloud (APC), you can create liveness and readiness probes to assess whether your Kubernetes Pods or network are healthy and can process requests.

Some components in APC include liveness and readiness probes by default, but all components support adding and configuring them. APC allows you to use the [Kubernetes liveness and readiness probe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes) definitions so you can monitor the state of your Pods.

<Tip>
  Liveness probes can be useful in all cases. However, readiness probes might be most useful for the following scenarios:

  * If you have network ports open on the container.
  * If a container doesn't have open ports but has multiple processes.
  * If a process in a container might never reach a `healthy` state because it's waiting for some state to be achieved.
</Tip>

## Default probe behavior

You can use the following structure to define your probes in your `values.yaml` file. For example, you might want to adjust any default values by configuring the amount of time until a timeout.

You can add any definitions that are compatible with [Kubernetes probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes). However, because Kubernetes doesn't allow having more than one handler of probes, you must be sure that you don't define probes that use both `exec` and `httpGet`. For consistency, the examples shown in the [Default Astronomer Helm probe configurations](#default-astronomer-helm-probe-configurations) use `httpGet`, but you can use `exec` when appropriate.

### Liveness probe templates

```yaml wrap theme={null}

livenessProbe:
  enabled: true
  httpGet:
    path: /index.html
    port: 443
```

### Readiness probe templates

```yaml wrap theme={null}

readinessProbe:
  enabled: true
  httpGet:
    path: /index.html
    port: 444
```

## Retrieve existing probe definitions

You can retrieve the default probe definitions from the Kubernetes manifest. The following example shows how to retrieve the definitions for the APC API.

```bash wrap theme={null}
kubectl -n "${NAMESPACE}" get deployment -l component=houston -o yaml
```

This command produces a large amount of YAML output describing your APC API configuration. Within this output is a section describing the `livenessProbe`, which looks like this:

```yaml wrap theme={null}
livenessProbe:
  failureThreshold: 10
  httpGet:
    path: /v1/healthz
    port: 8871
    scheme: HTTP
  initialDelaySeconds: 30
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1
```

You can copy and paste this output into your `values.yaml` file for your APC API configuration, then adjust the values you want to customize. Then [apply a platform config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

## Reference Helm values within your probes

The liveness and readiness probes specified in Helm values are passed through the Helm template function, which allows you to reference other Helm values within the probes. Specifically, the `livenessProbe` and `readinessProbe` values are rendered to YAML, then passed through the Helm template function, which renders any Helm template syntax into the produced YAML.

For example, instead of hardcoding values for your probes to match values defined by other configurations in your `values.yaml` file, you can use the configuration variable itself.

The following example, using the `alertmanager` YAML configuration, shows how the path and ports are defined by `Values.ports.http` and `Values.prefixURL` elsewhere in the `values.yaml` file.

```yaml wrap theme={null}
readinessProbe:
  httpGet:
    path: {{ .Values.prefixURL }}/#/status
    port: {{ .Values.ports.http }}
  initialDelaySeconds: 30
  timeoutSeconds: 30
```

## Default Astronomer Helm probe configurations

The following components have their default probe configuration defined in the [Astronomer Helm chart](https://github.com/astronomer/astronomer).

If a component doesn't have probes defined by default, you can see which components support custom probe configurations below.

### Airflow operator

The following can also be configured to include liveness and readiness probes:

```yaml wrap theme={null}
airflow-operator:
  livenessProbe: {}
  readinessProbe: {}
```

### Alert manager

The following can also be configured to include liveness and readiness probes:

```yaml wrap theme={null}
alertmanager:
  livenessProbe: {}
  readinessProbe: {}
```

### Astronomer

```yaml expandable wrap theme={null}
astronomer:
  astroUI:
    livenessProbe:
      httpGet:
        path: /
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 10
  commander:
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8880
        scheme: HTTP
      initialDelaySeconds: 10
      periodSeconds: 10
      failureThreshold: 5
      successThreshold: 1
      timeoutSeconds: 5
    readinessProbe:
      httpGet:
        path: /healthz
        port: 8880
      initialDelaySeconds: 10
      periodSeconds: 10
  houston:
    livenessProbe:
      httpGet:
        path: /v1/healthz
        port: 8871
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 10
    readinessProbe:
      httpGet:
        path: /v1/healthz
        port: 8871
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 10
  registry:
    livenessProbe:
      httpGet:
        path: /
        port: 5000
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 5
    readinessProbe:
      httpGet:
        path: /
        port: 5000
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 5
```

The following can also be configured to include liveness and readiness probes:

```yaml expandable wrap theme={null}
astronomer:
  configSyncer:
    livenessProbe: {}
    readinessProbe: {}
  houston:
    bootstrapper:
      livenessProbe: {}
      readinessProbe: {}
    cleanupAirflowDb:
      livenessProbe: {}
      readinessProbe: {}
    cleanupClusterAudits:
      livenessProbe: {}
      readinessProbe: {}
    cleanupDeployRevisions:
      livenessProbe: {}
      readinessProbe: {}
    cleanupDeployments:
      livenessProbe: {}
      readinessProbe: {}
    dbMigration:
      livenessProbe: {}
      readinessProbe: {}
    syncDataplaneClusters:
      livenessProbe: {}
      readinessProbe: {}
    taskUsageMetrics:
      livenessProbe: {}
      readinessProbe: {}
    updateCheck:
      livenessProbe: {}
      readinessProbe: {}
    updateResourceStrategy:
      livenessProbe: {}
      readinessProbe: {}
    updateRuntimeCheck:
      livenessProbe: {}
      readinessProbe: {}
    upgradeDeployments:
      livenessProbe: {}
      readinessProbe: {}
    waitForDB:
      livenessProbe: {}
      readinessProbe: {}
    worker:
      livenessProbe: {}
      readinessProbe: {}
```

### Elasticsearch

```yaml expandable wrap theme={null}
elasticsearch:
  client:
    livenessProbe:
      httpGet:
        path: /_cluster/health?local=true
        port: 9200
      initialDelaySeconds: 90
    readinessProbe:
      httpGet:
        path: /_cluster/health?local=true
        port: 9200
      initialDelaySeconds: 5
  exporter:
    livenessProbe:
      httpGet:
        path: /healthz
        port: http
      initialDelaySeconds: 30
      timeoutSeconds: 10
    readinessProbe:
      httpGet:
        path: /healthz
        port: http
      initialDelaySeconds: 10
      timeoutSeconds: 10
  master:
    livenessProbe:
      tcpSocket:
        port: 9300
    readinessProbe:
      httpGet:
        path: /_cluster/health?local=true
        port: 9200
      initialDelaySeconds: 5
```

The following components don't have probes configured by default:

```yaml expandable wrap theme={null}
elasticsearch:
  curator:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  data:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  nginx:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  sysctlInitContainer:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
```

### External-es-proxy

The following components don't have probes configured by default:

```yaml wrap theme={null}
external-es-proxy:
  awsproxy:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  livenessProbe:
    exec:
      command:
      - /bin/true
  readinessProbe:
    exec:
      command:
      - /bin/true
```

### Vector

The following components don't have probes configured by default:

```yaml wrap theme={null}
vector:
  vector:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
```

### Global

The following components don't have probes configured by default:

```yaml wrap theme={null}
global:
  authSidecar:
    livenessProbe: {}
    readinessProbe: {}
```

### Grafana

```yaml wrap theme={null}
grafana:
  livenessProbe:
    httpGet:
      path: /api/health
      port: 3000
    initialDelaySeconds: 10
    periodSeconds: 10
  readinessProbe:
    httpGet:
      path: /api/health
      port: 3000
    initialDelaySeconds: 10
    periodSeconds: 10
```

The following components don't have probes configured by default:

```yaml wrap theme={null}
grafana:
  bootstrapper:
    livenessProbe: {}
    readinessProbe: {}
  waitForDB:
    livenessProbe: {}
    readinessProbe: {}
```

### Kube-state

The following components don't have probes configured by default:

```yaml wrap theme={null}
kube-state:
  livenessProbe: {}
  readinessProbe: {}
```

### NATS

```yaml wrap theme={null}
nats:
  nats:
    livenessProbe:
      httpGet:
        path: /
        port: 8222
      initialDelaySeconds: 10
      timeoutSeconds: 5
    readinessProbe:
      httpGet:
        path: /
        port: 8222
      initialDelaySeconds: 10
      timeoutSeconds: 5
```

The following components don't have probes configured by default:

```yaml wrap theme={null}
nats:
  exporter:
    enabled: true
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  reloader:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
```

### nginx

The following components don't have probes configured by default:

```yaml wrap theme={null}
nginx:
  defaultBackend:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  livenessProbe:
    exec:
      command:
      - /bin/true
  readinessProbe:
    exec:
      command:
      - /bin/true
```

### PgBouncer

```yaml wrap theme={null}
pgbouncer:
  livenessProbe:
    tcpSocket:
      port: 5432
  readinessProbe:
    tcpSocket:
      port: 5432
```

### PostgreSQL

```yaml wrap theme={null}
postgresql:
  livenessProbe:
    exec:
      command:
        - sh
        - -c
        - exec pg_isready -U "postgres" -h 127.0.0.1 -p 5432
    initialDelaySeconds: 30
    periodSeconds: 10
    timeoutSeconds: 5
    successThreshold: 1
    failureThreshold: 6
  readinessProbe:
    exec:
      command:
        - sh
        - -c
        - -e
        - 'pg_isready -U "postgres" -h 127.0.0.1 -p 5432\n'
    initialDelaySeconds: 5
    periodSeconds: 10
    timeoutSeconds: 5
    successThreshold: 1
    failureThreshold: 6
```

The following components don't have probes configured by default:

```yaml wrap theme={null}
postgresql:
  metrics:
    livenessProbe: {}
    readinessProbe: {}
```

### Prometheus

```yaml wrap theme={null}
prometheus:
  livenessProbe:
    httpGet:
      path: /-/healthy
      port: 9090
    initialDelaySeconds: 10
    periodSeconds: 5
    failureThreshold: 3
    timeoutSeconds: 1
  readinessProbe:
    httpGet:
      path: /-/ready
      port: 9090
    initialDelaySeconds: 10
    periodSeconds: 5
    failureThreshold: 3
    timeoutSeconds: 1
prometheus-postgres-exporter:
  livenessProbe:
    tcpSocket:
      port: 9187
    initialDelaySeconds: 5
    periodSeconds: 10
  readinessProbe:
    tcpSocket:
      port: 9187
    initialDelaySeconds: 5
    periodSeconds: 10
```

The following components don't have probes configured by default:

```yaml wrap theme={null}
prometheus:
  configMapReloader:
    livenessProbe: {}
    readinessProbe: {}
  federation:
    livenessProbe: {}
    readinessProbe: {}
  filesdReloader:
    livenessProbe: {}
    readinessProbe: {}
```

### STAN

```yaml wrap theme={null}
    livenessProbe:
      httpGet:
        path: /streaming/serverz
        port: monitor
      initialDelaySeconds: 10
      timeoutSeconds: 5
    readinessProbe:
      httpGet:
        path: /streaming/serverz
        port: monitor
      initialDelaySeconds: 10
      timeoutSeconds: 5
```

The following components don't have probes configured by default:

```yaml wrap theme={null}
stan:
  exporter:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
  stan:
    nats:
      livenessProbe:
        exec:
          command:
          - /bin/true
      readinessProbe:
        exec:
          command:
          - /bin/true
  waitForNatsServer:
    livenessProbe:
      exec:
        command:
        - /bin/true
    readinessProbe:
      exec:
        command:
        - /bin/true
```

## Default Airflow chart probe configurations

You can also define liveness and readiness probes using the [Astronomer Airflow chart](https://github.com/astronomer/airflow-chart).

### Airflow

This includes:

* `dagProcessor`
* `flower`
* `pgbouncer`
* `postgresql`
* `scheduler`
* `triggerer`
* `webserver`
* `workers`

```yaml expandable wrap theme={null}
airflow:
  dagProcessor:
    livenessProbe:
      command: null
      failureThreshold: 5
      initialDelaySeconds: 10
      periodSeconds: 60
      timeoutSeconds: 20
    readinessProbe:
      initialDelaySeconds: 10
      timeoutSeconds: 20
      failureThreshold: 5
      periodSeconds: 60
    logGroomerSidecar:
      enabled: true
      livenessProbe:
        initialDelaySeconds: 60
        timeoutSeconds: 20
        failureThreshold: 5
        periodSeconds: 60
      readinessProbe:
        initialDelaySeconds: 60
        timeoutSeconds: 20
        failureThreshold: 5
        periodSeconds: 60
  flower:
    livenessProbe:
      failureThreshold: 10
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 5
    readinessProbe:
      failureThreshold: 10
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 5
  pgbouncer:
    metricsExporterSidecar:
      livenessProbe:
        initialDelaySeconds: 10
        periodSeconds: 10
        timeoutSeconds: 1
      readinessProbe:
        initialDelaySeconds: 10
        periodSeconds: 10
        timeoutSeconds: 1
  postgresql:
    metrics:
      customLivenessProbe: {}
      customReadinessProbe: {}
      livenessProbe:
        enabled: true
        failureThreshold: 6
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      readinessProbe:
        enabled: true
        failureThreshold: 6
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
    primary:
      customLivenessProbe: {}
      customReadinessProbe: {}
      livenessProbe:
        enabled: true
        failureThreshold: 6
        initialDelaySeconds: 30
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      readinessProbe:
        enabled: true
        failureThreshold: 6
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
    readReplicas:
      customLivenessProbe: {}
      customReadinessProbe: {}
      livenessProbe:
        enabled: true
        failureThreshold: 6
        initialDelaySeconds: 30
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      readinessProbe:
        enabled: true
        failureThreshold: 6
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
  scheduler:
    livenessProbe:
      command: null
      failureThreshold: 5
      initialDelaySeconds: 10
      periodSeconds: 60
      timeoutSeconds: 30
  triggerer:
    livenessProbe:
      command: null
      failureThreshold: 5
      initialDelaySeconds: 10
      periodSeconds: 60
      timeoutSeconds: 20
  webserver:
    livenessProbe:
      failureThreshold: 5
      initialDelaySeconds: 15
      periodSeconds: 10
      scheme: HTTP
      timeoutSeconds: 5
    readinessProbe:
      failureThreshold: 5
      initialDelaySeconds: 15
      periodSeconds: 10
      scheme: HTTP
      timeoutSeconds: 5
  workers:
    livenessProbe:
      command: null
      enabled: true
      failureThreshold: 5
      initialDelaySeconds: 10
      periodSeconds: 60
      timeoutSeconds: 20
```

### Auth sidecar

The following can also be configured to include liveness and readiness probes:

```yaml wrap theme={null}
authSidecar:
  livenessProbe: {}
  readinessProbe: {}
```

### Dag deploy server

The following can also be configured to include liveness and readiness probes:

```yaml wrap theme={null}
dagDeploy:
  livenessProbe: {}
  readinessProbe: {}
```

### Logging sidecar

The following can also be configured to include liveness and readiness probes:

```yaml wrap theme={null}
loggingSidecar:
  livenessProbe: {}
  readinessProbe: {}
```


# Deployment metrics
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/deployment-metrics

Enable node exporter and cAdvisor to view CPU, memory, and network metrics for your Deployments.

In Astro Private Cloud versions before 1.0, Deployment-level metrics such as CPU usage, memory usage, and network I/O were available by default through built-in node exporter and cAdvisor integrations. These were [removed in APC 1.0](/docs/astro-private-cloud/v-2-x/breaking-changes-removals#prometheus-blackbox-exporter-removed-0-37-x-→-2-0-only) to reduce Prometheus data volume and improve performance.

Starting with APC 2.0, you can re-enable these metrics with two Helm flags. After you enable them, four additional panels appear in the **Metrics** tab for each Deployment in the Astro Private Cloud UI:

* **CPU Usage**: Percentage of CPU consumed by the Deployment's containers.
* **Memory Usage**: Percentage of memory consumed by the Deployment's containers.
* **Network Rx**: Inbound network traffic received by the Deployment's Pods.
* **Network Tx**: Outbound network traffic transmitted by the Deployment's Pods.

<Frame>
  <img alt="Deployment metrics panels showing CPU Usage, Memory Usage, Network Rx, and Network Tx" />
</Frame>

Both features are disabled by default.

## Prerequisites

* Astro Private Cloud 2.0 or later with Helm chart version 2.0.0 or later.
* System Admin access to update the Astro Private Cloud Helm configuration.

### Enable Deployment metrics

<Steps>
  <Step title="Update your Helm configuration">
    Add the following values to your `values.yaml` file:

    ```yaml focus={2-3,5-6} wrap theme={null}
    global:
      nodeExporter:
        enabled: true
      cadvisor:
        enabled: true
    ```
  </Step>

  <Step title="Apply the configuration">
    Run a Helm upgrade to apply the changes:

    ```bash wrap theme={null}
    helm upgrade -f values.yaml -n <your-namespace> astronomer astronomer/astronomer
    ```
  </Step>

  <Step title="Verify the metrics">
    Open the Astro Private Cloud UI, navigate to a Deployment, and click the **Metrics** tab. The CPU Usage, Memory Usage, Network Rx, and Network Tx panels appear alongside the existing Database Connections and Waiting Clients panels.
  </Step>
</Steps>


# Configure task log collection and exporting to Elasticsearch
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/export-task-logs

Configure how Astronomer exports task logs to your Elasticsearch instance.

Apache Airflow task logs are stored in a logging backend to ensure you can access them after your Pods terminate. By default, Astro Private Cloud uses [Vector](https://vector.dev/) to collect task logs and export them to an Elasticsearch instance.

You can configure how Astro Private Cloud collects Deployment task logs and exports them to Elasticsearch. The following are the supported methods for exporting task logs to Elasticsearch:

* Using a DaemonSet Pod on each Kubernetes node in your cluster.
* Using container sidecars for Deployment components.

## Export task logs using a Vector DaemonSet

<Warning>
  Exporting task logs using a Vector DaemonSet isn't supported for Airflow 3.
</Warning>

By default, Astro Private Cloud uses a Vector DaemonSet to aggregate task logs. This is the workflow for the default implementation:

* Deployments write task logs to `stdout`.
* Kubernetes takes the output from `stdout` and writes it to the Deployment’s node.
* A Vector Pod reads logs from the node and forwards them to Elasticsearch.

Astronomer recommends using Vector DaemonSet for organizations that:

* Run longer tasks using Celery executor.
* Run Astro Private Cloud in a dedicated cluster.
* Run privileged containers in a cluster with a ClusterRole.

This approach isn't suited for organizations that don't allow logging container to run in privileged mode and run many small tasks using the Kubernetes executor. Because task logs exist only for the lifetime of the Pod, your Pods running small tasks might complete before Vector can collect their task logs.

## Export logs using container sidecars

You can use a logging sidecar container to collect and export logs. In this implementation:

* Each container running an Airflow component for a Deployment receives its own [Vector](https://vector.dev/) sidecar.
* Task logs are written to a shared directory.
* The Vector sidecar reads logs from the shared directory and writes them to Elasticsearch.

This implementation is recommended for organizations that:

* Run Astro Private Cloud in a multi-tenant cluster, where security is a concern.
* Use the Kubernetes executor to run many short-lived tasks, which requires improved reliability.

### Configure logging sidecars

1. Retrieve your `values.yaml` file. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

2. Add the following entry to your `values.yaml` file:

   ```yaml wrap theme={null}
   global:
     daemonsetLogging:
       enabled: false
     logging:
       loggingSidecar:
         enabled: true
         name: sidecar-log-consumer
   ```

   <Tip>
     If you're migrating from Fluentd, you must also set the following configuration so that Astro Private Cloud can retain logs:

     ```yaml wrap theme={null}
     global:
       logging:
         indexNamePrefix: <your-index-prefix>
     ```
   </Tip>

3. Push the configuration change. See [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

#### Customize Vector logging sidecars

You can customize the default Astronomer Vector logging sidecar to have different transformations and sinks based on your team's requirements. This is useful if you want to annotate, customize, or filter your logs before sending them to your logging platform.

1. In the **Astro UI**, go to your **Clusters** page and select your cluster.

2. In the cluster details, click **Edit** in the **Deployment Configuration** section and add the following override in the **Configuration Override** field:

   ```yaml wrap theme={null}
   global:
     logging:
       loggingSidecar:
         enabled: true
         name: sidecar-log-consumer
         customConfig: true
   ```

   For more information on using the **Configuration Override** in the UI, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).

3. Save and apply your changes in the UI.

<Note>
  Pushing this change updates the configuration for the cluster. Individual Deployments will receive the new sidecar logging configuration once they are redeployed.
</Note>

4. Create a custom [vector configuration `yaml` file](https://vector.dev/docs/reference/configuration/) to change how and where sidecars forward your logs. The following examples are template configurations for each commonly used external logging service. For the complete default logging sidecar configmap, see the [Astronomer GitHub](https://github.com/astronomer/airflow-chart/blob/master/templates/logging-sidecar-configmap.yaml).

<Tabs>
  <Tab title="Elasticsearch">
    ```yaml expandable wrap theme={null}
    log_schema:
      timestamp_key : "@timestamp"
    data_dir: "${SIDECAR_LOGS}"
    sources:
      airflow_log_files:
        type: file
        include:
          - "${SIDECAR_LOGS}/*.log"
        read_from: beginning
    transforms:
      transform_airflow_logs:
        type: remap
        inputs:
          - airflow_log_files
        source: |
          .component = "${COMPONENT:--}"
          .workspace = "${WORKSPACE:--}"
          .release = "${RELEASE:--}"
          .date_nano = parse_timestamp!(.@timestamp, format: "%Y-%m-%dT%H:%M:%S.%f%Z")

      filter_common_logs:
        type: filter
        inputs:
          - transform_airflow_logs
        condition:
          type: "vrl"
          source: '!includes(["worker","scheduler"], .component)'

      filter_scheduler_logs:
        type: filter
        inputs:
          - transform_airflow_logs
        condition:
          type: "vrl"
          source: 'includes(["scheduler"], .component)'

      filter_worker_logs:
        type: filter
        inputs:
          - transform_airflow_logs
        condition:
          type: "vrl"
          source: 'includes(["worker"], .component)'

      filter_gitsyncrelay_logs:
        type: filter
        inputs:
          - transform_airflow_logs
        condition:
          type: "vrl"
          source: 'includes(["git-sync-relay"], .component)'

      transform_task_log:
        type: remap
        inputs:
          - filter_worker_logs
          - filter_scheduler_logs
        source: |-
          . = parse_json(.message) ?? .
          .@timestamp = parse_timestamp(.timestamp, "%Y-%m-%dT%H:%M:%S%Z") ?? now()
          .check_log_id = exists(.log_id)
          if .check_log_id != true {
          .log_id = join!([to_string!(.dag_id), to_string!(.task_id), to_string!(.execution_date), to_string!(.try_number)], "_")
          }
          .offset = to_int(now()) * 1000000000 + to_unix_timestamp(now()) * 1000000

      final_task_log:
        type: remap
        inputs:
          - transform_task_log
        source: |
          .component = "${COMPONENT:--}"
          .workspace = "${WORKSPACE:--}"
          .release = "${RELEASE:--}"
          .date_nano = parse_timestamp!(.@timestamp, format: "%Y-%m-%dT%H:%M:%S.%f%Z")

      transform_remove_fields:
        type: remap
        inputs:
          - final_task_log
          - filter_common_logs
          - filter_gitsyncrelay_logs
        source: |
          del(.host)
          del(.file)

    ## Configuration for ElasticSearch sink

    sinks:
      out:
        type: elasticsearch

    ## Specify the transforms you want to run before your logs are exported

        inputs:
          - transform_remove_fields
        mode: bulk
        compression: none
        endpoint: "http://example-host:<example-port>"
        auth:
          strategy: "basic"
          user: "example-user"
          password : "example-pass"
        bulk:
          index: "vector.${RELEASE:--}.%Y.%m.%d"
          action: create

    ```
  </Tab>

  <Tab title="Honeycomb">
    ```yaml expandable wrap theme={null}
    log_schema:
      timestamp_key : "@timestamp"
    data_dir: "${SIDECAR_LOGS}"
    sources:
      airflow_log_files:
        type: file
        include:
          - "${SIDECAR_LOGS}/*.log"
        read_from: beginning
    transforms:
      transform_syslog:
        type: add_fields
        inputs:
          - generate_syslog
        fields:
          component: "${COMPONENT:--}"
          workspace: "${WORKSPACE:--}"
          release: "${RELEASE:--}"
      transform_task_log:
        type: remap
        inputs:
          - transform_syslog
        source: |-
          # Parse Syslog input. The "!" means that the script should abort on error.
          . = parse_json!(.message)
          .@timestamp = parse_timestamp(.timestamp, "%Y-%m-%dT%H:%M:%S%Z") ?? now()
          .check_log_id = exists(.log_id)
          if .check_log_id != true {
          .log_id = join!([.dag_id, .task_id, .execution_date, .try_number], "_")
          }
          .offset = to_int(now()) * 1000000000 + to_unix_timestamp(now()) * 1000000
    # Configuration for Datadog sinks
    sinks:
      my_sink_id:
        type: datadog_logs
        # Specify the transforms you want to run before your logs are exported.
        inputs:
          - transform_task_log
        site: us1.datadoghq.com
        default_api_key: <your-api-key>
        encoding:
          codec: json
    ```
  </Tab>

  <Tab title="Datadog">
    ```yaml expandable wrap theme={null}
    log_schema:
      timestamp_key : "@timestamp"
    data_dir: "${SIDECAR_LOGS}"
    sources:
      airflow_log_files:
        type: file
        include:
          - "${SIDECAR_LOGS}/*.log"
        read_from: beginning
    transforms:
      transform_syslog:
        type: add_fields
        inputs:
          - generate_syslog
        fields:
          component: "${COMPONENT:--}"
          workspace: "${WORKSPACE:--}"
          release: "${RELEASE:--}"

    ## Configuration for Honeycomb sinks

    sinks:
      my_sink_id:
        type: honeycomb

    ## Specify the transforms you want to run before your logs are exported

        inputs:
          - transform_syslog
        api_key: <your-api-key>
        dataset: my-honeycomb-dataset

    ```
  </Tab>
</Tabs>

5. Run the following command to add the configuration file to your cluster as a Kubernetes secret:

   ```bash wrap theme={null}
   kubectl create secret generic sidecar-config --from-file=vector-values.yaml=vector-values.yaml
   ```

6. Run the following command to annotate the secret so that it's automatically applied to all new Deployments:

   ```bash wrap theme={null}
   kubectl annotate secret secret-name astronomer.io/commander-sync="platform-release=astronomer"
   ```

7. Run the following command to sync existing Deployments with the new configuration:

   ```bash wrap theme={null}
   kubectl create job --from=cronjob/astronomer-config-syncer sync-secrets -n astronomer
   ```

## Use an external Elasticsearch instance for Airflow task log management

Add Airflow task logs from your Astronomer Deployment to an existing Elasticsearch instance on [Elastic Cloud](https://www.elastic.co/cloud/) to centralize log management and analysis. Centralized log management allows you to quickly identify, troubleshoot, and resolve task failure issues. Although these examples use Elastic Cloud, you can also use AWS Managed OpenSearch Service or any other elastic service (managed or hosted). With an external Elasticsearch instance configured for Astro Private Cloud, you can see the logs in your Elasticsearch instance and browse the logs from the APC UI.

<Note>If you use an existing Elasticsearch instance, make sure that the index template is configured to enable auto creation of new indices.</Note>

### Create an Elastic Deployment and endpoint

1. In your browser, go to `https://cloud.elastic.co/` and create a new Elastic Cloud deployment. See [Create a deployment](https://www.elastic.co/guide/en/cloud/current/ec-create-deployment.html#ec-create-deployment).

2. Copy and save your Elastic Cloud deployment credentials when the **Save the deployment credentials** screen appears.

3. On the Elastic dashboard, click the **Gear** icon for your Deployment.

4. Click **Copy endpoint** next to **Elasticsearch**.

5. (Optional) Test the Elastic Cloud deployment endpoint:
   * Open a new browser window, paste the endpoint you copied in step 4 in the **Address** bar, and then press **Enter**.
   * Enter the username and password you copied in step 2 and click **Sign in**. Output similar to the following appears:
   ```text wrap theme={null}
       name	"instance-0000000000"
       cluster_name	"<cluster-name>"
       cluster_uuid	"<cluster-uuid>"
       version
       number	"8.3.2"
       build_type	"docker"
       build_hash	"8b0b1f23fbebecc3c88e4464319dea8989f374fd"
       build_date	"2022-07-06T15:15:15.901688194Z"
       build_snapshot	false
       lucene_version	"9.2.0"
       minimum_wire_compatibility_version	"7.17.0"
       minimum_index_compatibility_version	"7.0.0"
       tagline	"You Know, for Search"
   ```

### Save your Elastic Cloud deployment credentials

After you've created an Elastic deployment and endpoint, you have two options to store your Elastic deployment credentials. You can store the credentials in your Astro Private Cloud Helm values, or for greater security, as a secret in your Astro Private Cloud Kubernetes cluster. For additional information about adding an Astro Private Cloud configuration change, see [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

<Note>
  Updating the Elasticsearch host in your data plane `values.yaml` doesn't automatically update the cluster configuration database. After you upgrade your data plane Helm release, manually update the Elasticsearch proxy host through the UI:

  1. In the Astro UI, go to **Clusters** and select your cluster.

  2. Click **Edit** to unlock **Configuration Override**.

  3. Add the following to the **Configuration Override**:

     ```json wrap theme={null}
     {
       "helm": {
         "airflow": {
           "elasticsearch": {
             "enabled": true,
             "connection": {
               "host": "<your-elasticsearch-host>",
               "port": 9200
             }
           }
         }
       }
     }
     ```

  4. Click **Update cluster** to apply your changes.

  5. Update each existing Deployment to apply the new Elasticsearch configuration.

  For more information, see [Override base configuration](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster#override-base-configuration).
</Note>

<Tabs>
  <Tab title="values.yaml">
    1. Run the following command to base64 encode your Elastic Cloud deployment credentials:

    ```bash wrap theme={null}
       echo -n "<username>:<password>" | base64
    ```

    2. Add the following entry to your `values.yaml` file:

       ```yaml wrap theme={null}
       global:
       daemonsetLogging:
         enabled: true
       customLogging:
         enabled: true
         scheme: https
         # host endpoint copied from elasticsearch console with https
         # and port number removed.
         host: "<host-URL>"
         port: "9243"
         # encoded credentials from above step 1
         secret: "<encoded credentials>"
       ```

    3. Add the following entry to your `values.yaml` file to disable internal logging:

       ```yaml wrap theme={null}
       tags:
       logging: false
       ```

    4. Run the following command to upgrade the Astro Private Cloud release version in the `values.yaml` file:

       ```bash wrap theme={null}
       helm upgrade -f values.yaml --version=2.0.0 --namespace=<your-platform-namespace> <your-platform-release-name> astronomer/astronomer
       ```
  </Tab>

  <Tab title="Kubernetes secret">
    1. Run the following command to create a secret for your Elastic Cloud Deployment credentials in the Kubernetes cluster:

    ```bash wrap theme={null}
    kubectl create secret generic elasticcreds --from-literal elastic=<username>:<password> --namespace=<your-platform-namespace>
    ```

    2. Add the following entry to your `values.yaml` file:

       ```yaml wrap theme={null}
       global:
       daemonsetLogging:
         enabled: true
       customLogging:
         enabled: true
         scheme: https
         # host endpoint copied from elasticsearch console with https
         # and port number removed.
         host: "<host-URL>"
         port: "9243"
         # kubernetes secret containing credentials
         secretName: elasticcreds
       ```

    3. Add the following entry to your `values.yaml` file to disable internal logging:

       ```yaml wrap theme={null}
       tags:
          logging: false
       ```

    4. Run the following command to upgrade the Astro Private Cloud release version in the `values.yaml` file:

       ```bash wrap theme={null}
       helm upgrade -f values.yaml --version=2.0.0 --namespace=<your-platform-namespace> <your-platform-release-name> astronomer/astronomer
       ```
  </Tab>
</Tabs>

### View Airflow task logs in Elastic

1. On the Elastic dashboard in the **Elasticsearch Service** area, click the Deployment name.
2. Click **Menu** > **Discover**. The **Create index pattern** screen appears.
3. Enter `vector.*` if you use Vector sidecar logging. In the **Name** field, enter `@timestamp` in the **Timestamp field**, and then click **Create index pattern**.
4. Click **Menu** > **Dashboard** to view all of the Airflow task logs for your Deployment on Astronomer.


# Git-sync relay metrics
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/git-sync-relay-metrics

Enable and monitor git-sync relay metrics for Deployments that use git-sync to deploy Dags.

For Deployments that use git-sync to deploy Dags, the git-sync relay emits operational metrics through StatsD. These metrics provide visibility into sync operations, repository size, and sync performance for each Deployment.

Git-sync relay metrics are disabled by default and require configuration at both the platform and cluster levels.

<Warning>
  In APC 2.0, the strict schema validator doesn't recognize git-sync relay metrics. You must disable the strict schema check to use this feature. See the following configuration steps.
</Warning>

## Prerequisites

* Astro Private Cloud 2.0 or later with Helm chart version 2.0.0 or later.
* System Admin access to update the Astro Private Cloud Helm configuration.
* One or more Deployments configured to use git-sync for Dag deployment.

### Enable git-sync relay metrics

<Steps>
  <Step title="Disable the strict schema check">
    Because the strict schema validator in APC 2.0 doesn't recognize git-sync relay metrics fields, you must disable it. Add the following to your platform-level `values.yaml`:

    ```yaml focus={2-3} wrap theme={null}
    astronomer:
      strictSchemaCheck:
        enabled: false
    ```
  </Step>

  <Step title="Enable metrics at the platform level">
    In the same platform-level `values.yaml`, enable git-sync relay metrics globally:

    ```yaml focus={2-3} wrap theme={null}
    global:
      gitSyncRelay:
        metrics:
          enabled: true
    ```
  </Step>

  <Step title="Enable metrics in the cluster configuration">
    In your cluster configuration, enable git-sync relay metrics:

    ```yaml focus={2-3} wrap theme={null}
    helm:
      gitSyncRelay:
        metrics:
          enabled: true
    ```
  </Step>

  <Step title="Apply the configuration">
    Run a Helm upgrade to apply the changes:

    ```bash wrap theme={null}
    helm upgrade -f values.yaml -n <your-namespace> <Release-Name> astronomer/astronomer
    ```
  </Step>

  <Step title="Verify the metrics">
    After a successful Helm upgrade, git-sync relay metrics are available through the Deployment's StatsD exporter. Run the following command to confirm metrics are being emitted:

    ```bash wrap theme={null}
    curl -s http://localhost:9000/metrics | grep git_sync_relay
    ```
  </Step>
</Steps>

## Available metrics

The following metrics are available with the `git_sync_relay_` prefix. All metrics are emitted per Deployment.

### Sync operation metrics

| Metric                                                   | Type    | Description                                        |
| -------------------------------------------------------- | ------- | -------------------------------------------------- |
| `git_sync_relay_git_sync_count`                          | Counter | Total number of git sync operations initiated      |
| `git_sync_relay_git_sync_success_count`                  | Counter | Total number of successful git sync operations     |
| `git_sync_relay_git_sync_duration_seconds`               | Gauge   | Duration of the last git sync operation in seconds |
| `git_sync_relay_git_sync_interval_seconds`               | Gauge   | Git sync polling interval in seconds               |
| `git_sync_relay_last_sync_timestamp`                     | Gauge   | Timestamp of the last successful sync              |
| `git_sync_relay_git_sync_last_successful_sync_timestamp` | Gauge   | Timestamp of the last successful sync operation    |

### Git operation metrics

| Metric                                           | Type    | Description                                                                      |
| ------------------------------------------------ | ------- | -------------------------------------------------------------------------------- |
| `git_sync_relay_git_operations_total`            | Counter | Total git operations (clone, fetch, clean, reset, sync)                          |
| `git_sync_relay_git_operations_duration_seconds` | Summary | Duration of individual git operations with quantile distribution (p50, p90, p99) |
| `git_sync_relay_git_sync_clone_duration`         | Gauge   | Duration of git clone operation in seconds                                       |

### File and repository metrics

| Metric                                       | Type    | Description                                           |
| -------------------------------------------- | ------- | ----------------------------------------------------- |
| `git_sync_relay_files_updated_total`         | Counter | Total files updated during sync operations            |
| `git_sync_relay_git_sync_file_changed_count` | Gauge   | Number of files changed in the last sync              |
| `git_sync_relay_repo_size_bytes`             | Gauge   | Size of the git repository in bytes                   |
| `git_sync_relay_git_dir_size_bytes`          | Gauge   | Size of the `.git` folder in bytes                    |
| `git_sync_relay_repo_total_size_bytes`       | Gauge   | Total repository size including working tree in bytes |
| `git_sync_relay_working_tree_size_bytes`     | Gauge   | Size of the working tree in bytes                     |
| `git_sync_relay_git_sync_repo_size`          | Gauge   | Git repository size in bytes                          |

### Initialization metrics

| Metric                                                  | Type  | Description                                    |
| ------------------------------------------------------- | ----- | ---------------------------------------------- |
| `git_sync_relay_git_sync_init_duration_seconds`         | Gauge | Application initialization duration in seconds |
| `git_sync_relay_git_sync_init_attempt`                  | Gauge | Number of initialization attempts              |
| `git_sync_relay_git_sync_light_init_duration_seconds`   | Gauge | Light initialization duration in seconds       |
| `git_sync_relay_git_sync_deferred_init_start_timestamp` | Gauge | Deferred initialization start timestamp        |

### State metrics

| Metric                                     | Type  | Description                                                                                                                                                                                                |
| ------------------------------------------ | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `git_sync_relay_git_sync_ready_state`      | Gauge | git-sync relay readiness state: `0` = initializing, `1` = ready, `2` = failed, `3` = retrying. Starting in APC 2.1.0, the Deployment's status indicator in the Astro Private Cloud UI reflects this value. |
| `git_sync_relay_git_sync_shallow_state`    | Gauge | Whether the repository is in a shallow clone state                                                                                                                                                         |
| `git_sync_relay_git_sync_configured_depth` | Gauge | Configured git clone depth                                                                                                                                                                                 |


# Helm configuration reference
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/helm-config-reference

Reference for APC platform values.yaml: chart settings, subcharts, and astronomer.houston.config (including deployments defaults and config governance).

This reference is for the platform `values.yaml` you apply with Helm: chart-level settings (`global`, `tags`, `nginx`, `prometheus`, and other subcharts) and the APC API application block `astronomer.houston.config`, including `deployments.*` defaults used with [Config governance](/docs/astro-private-cloud/v-2-x/config-governance) (platform defaults in this file, then optional cluster, workspace, and deployment overrides). Tabular sections use the same path prefix you use in your install values.

## Values file structure

The APC Helm chart uses a hierarchical structure:

```yaml wrap theme={null}
global:           # Platform-wide settings
  baseDomain: ""  # Required: your base domain
  plane:
    mode: ""      # unified, control, or data
  # ... more global settings

tags:             # Enable/disable component groups

  monitoring: true
  logging: true

astronomer:       # Platform component settings
  houston: {}
  commander: {}
  registry: {}
  astroUI: {}

nginx: {}         # Ingress configuration
prometheus: {}    # Metrics collection
elasticsearch: {} # Log storage
grafana: {}       # Dashboards
# ... more component sections
```

## Required configuration

Configure the following required values:

```yaml wrap theme={null}
global:
  baseDomain: "example.com"  # Your platform domain
  tlsSecret: "astronomer-tls"  # TLS certificate secret name
```

## Helm chart parameter tables

Selected defaults from the APC umbrella chart (`values.yaml`). Parameter paths are relative to the root of your install values file.

### Tags

| Parameter         | Type    | Description                                                    | Default | Allowed values  |
| ----------------- | ------- | -------------------------------------------------------------- | ------- | --------------- |
| `tags.platform`   | boolean | Enable core platform chart groups (ingress, Astronomer stack). | `true`  | `true`, `false` |
| `tags.monitoring` | boolean | Enable Prometheus, kube-state-metrics, Grafana stack.          | `true`  | `true`, `false` |
| `tags.logging`    | boolean | Enable Elasticsearch and logging collectors.                   | `true`  | `true`, `false` |

### Global settings

| Parameter                          | Type    | Description                                                                   | Default          | Allowed values               |
| ---------------------------------- | ------- | ----------------------------------------------------------------------------- | ---------------- | ---------------------------- |
| `global.baseDomain`                | string  | DNS zone for platform hosts (`app`, `houston`, ingress).                      | `~`              | Domain name                  |
| `global.tlsSecret`                 | string  | TLS Secret used by ingress.                                                   | `astronomer-tls` | Kubernetes Secret name       |
| `global.privateCaCerts`            | array   | Extra CA Secrets for private PKI.                                             | `[]`             | Secret names                 |
| `global.plane.mode`                | string  | Control plane / data plane / unified topology.                                | `"unified"`      | `unified`, `control`, `data` |
| `global.plane.domainPrefix`        | string  | Prefix for split-plane routing when used.                                     | `""`             | String                       |
| `global.networkPolicy.enabled`     | boolean | Install NetworkPolicies for platform namespaces.                              | `true`           | `true`, `false`              |
| `global.defaultDenyNetworkPolicy`  | boolean | Default-deny ingress NetworkPolicy baseline.                                  | `true`           | `true`, `false`              |
| `global.networkNSLabels.enabled`   | boolean | Enable namespace labels for network policies.                                 | `false`          | `true`, `false`              |
| `global.rbac.enabled`              | boolean | Manage Kubernetes RBAC objects for the platform.                              | `true`           | `true`, `false`              |
| `global.clusterRoles`              | boolean | Use ClusterRole bindings where required.                                      | `true`           | `true`, `false`              |
| `global.nats.enabled`              | boolean | Deploy NATS for the APC API messaging.                                        | `true`           | `true`, `false`              |
| `global.nats.replicas`             | integer | NATS replicas.                                                                | `3`              | Positive integer             |
| `global.airflowOperator.enabled`   | boolean | Enable Airflow Kubernetes operator integration path.                          | `false`          | `true`, `false`              |
| `global.dataPlaneFailover.enabled` | boolean | Master switch for data-plane failover features when components are installed. | `false`          | `true`, `false`              |

### Astronomer platform images

| Parameter                                | Type   | Description                               | Default                             | Allowed values |
| ---------------------------------------- | ------ | ----------------------------------------- | ----------------------------------- | -------------- |
| `astronomer.images.commander.repository` | string | Deployment orchestrator image repository. | `quay.io/astronomer/ap-commander`   | OCI repository |
| `astronomer.images.commander.tag`        | string | Deployment orchestrator image tag.        | `2.0.14`                            | Tag string     |
| `astronomer.images.houston.repository`   | string | APC API image repository.                 | `quay.io/astronomer/ap-houston-api` | OCI repository |
| `astronomer.images.houston.tag`          | string | APC API image tag.                        | `2.0.18`                            | Tag string     |
| `astronomer.images.astroUI.repository`   | string | Astro UI image repository.                | `quay.io/astronomer/ap-astro-ui`    | OCI repository |
| `astronomer.images.astroUI.tag`          | string | Astro UI tag.                             | `2.0.7`                             | Tag string     |
| `astronomer.images.registry.repository`  | string | Internal registry image repository.       | `quay.io/astronomer/ap-registry`    | OCI repository |
| `astronomer.images.registry.tag`         | string | Internal registry tag.                    | `3.0.0-9`                           | Tag string     |

### Astronomer workload defaults

Resource defaults for core platform Deployments often ship in the umbrella chart—override `requests` / `limits` per component:

| Parameter                                      | Type    | Description                                                                                                                    | Default                                           | Allowed values       |
| ---------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------- | -------------------- |
| `astronomer.astroUI.resources`                 | object  | Astro UI CPU/memory requests and limits.                                                                                       | requests `100m`/`256Mi`, limits `500m`/`1024Mi`   | Kubernetes resources |
| `astronomer.houston.resources`                 | object  | APC API CPU/memory.                                                                                                            | requests `500m`/`1024Mi`, limits `1000m`/`2048Mi` | Kubernetes resources |
| `astronomer.houston.strictSchemaCheck.enabled` | boolean | Chart-level toggle aligned with the APC API; keep consistent with `astronomer.houston.config.strictSchemaCheck.enabled` below. | `true`                                            | `true`, `false`      |
| `astronomer.commander.resources`               | object  | Deployment orchestrator CPU/memory.                                                                                            | requests `250m`/`1Gi`, limits `500m`/`2Gi`        | Kubernetes resources |
| `astronomer.registry.persistence.enabled`      | boolean | Persist registry storage.                                                                                                      | `true`                                            | `true`, `false`      |
| `astronomer.registry.persistence.size`         | string  | Registry PVC size.                                                                                                             | `"100Gi"`                                         | Quantity string      |
| `astronomer.install.resources`                 | object  | Helm install job resources.                                                                                                    | requests `100m`/`256Mi`, limits `500m`/`1024Mi`   | Kubernetes resources |

For nginx, Prometheus, Elasticsearch, Grafana, NATS workload tables, follow the same pattern in your `values.yaml`; defaults ship alongside those keys in the umbrella chart.

## APC API configuration

Set these keys under `astronomer.houston.config` in your platform `values.yaml`. Published defaults mirror [the APC API `config/default.yaml`](https://github.com/astronomer/houston-api/blob/main/config/default.yaml). The subtree **`deployments.*`** participates in layered overrides (platform → cluster → workspace → deployment); see [Config governance](/docs/astro-private-cloud/v-2-x/config-governance). Override APIs may use the string `DELETE_KEY` at mergeable leaves where the schema allows (see [Config governance](/docs/astro-private-cloud/v-2-x/config-governance)).

### Outside the `deployments` subtree

These keys configure the APC API, workers, auth, UI metadata, and integrations. They are *not* subject to the four deployment override tiers unless documented elsewhere.

| Parameter                                                          | Type    | Description                                                                                                                           | Default                               | Allowed values                                                               |
| ------------------------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | ---------------------------------------------------------------------------- |
| `astronomer.houston.config.strictSchemaCheck.enabled`              | boolean | When `true`, unknown keys or invalid types under `deployments` overrides are rejected (runtime enforcement aligns with this setting). | `true`                                | `true`, `false`                                                              |
| `astronomer.houston.config.webserver.port`                         | integer | HTTP listen port for the APC API inside the pod.                                                                                      | `8871`                                | Positive integer                                                             |
| `astronomer.houston.config.webserver.endpoint`                     | string  | REST API base path.                                                                                                                   | `"/v1"`                               | String                                                                       |
| `astronomer.houston.config.webserver.graphqlPlayground.enabled`    | boolean | Expose GraphQL playground endpoint.                                                                                                   | `true`                                | `true`, `false`                                                              |
| `astronomer.houston.config.logging.level`                          | string  | APC API log level.                                                                                                                    | `"info"`                              | Typical log levels (`error`, `warn`, `info`, `debug`, …)                     |
| `astronomer.houston.config.platformReleasesFile`                   | string  | Platform releases manifest filename.                                                                                                  | `'astronomer_platform_releases.json'` | Filename string                                                              |
| `astronomer.houston.config.publicSignups.enabled`                  | boolean | Allow public user registration.                                                                                                       | `false`                               | `true`, `false`                                                              |
| `astronomer.houston.config.emailConfirmation.enabled`              | boolean | Require email confirmation for new accounts.                                                                                          | `true`                                | `true`, `false`                                                              |
| `astronomer.houston.config.email.enabled`                          | boolean | Enable outbound email.                                                                                                                | `false`                               | `true`, `false`                                                              |
| `astronomer.houston.config.email.reply`                            | string  | Default From / reply address.                                                                                                         | `"noreply@astronomer.io"`             | Email string                                                                 |
| `astronomer.houston.config.email.smtpUrl`                          | string  | SMTP connection URL.                                                                                                                  | `~` (null)                            | SMTP URL or null                                                             |
| `astronomer.houston.config.prometheus.enabled`                     | boolean | Query Prometheus for metrics in the APC API features that support it.                                                                 | `false`                               | `true`, `false`                                                              |
| `astronomer.houston.config.prometheus.host`                        | string  | Prometheus hostname.                                                                                                                  | `localhost`                           | Hostname                                                                     |
| `astronomer.houston.config.prometheus.port`                        | integer | Prometheus port.                                                                                                                      | `9090`                                | Port number                                                                  |
| `astronomer.houston.config.plane.mode`                             | string  | Logical plane mode for the APC API runtime behavior.                                                                                  | `unified`                             | `unified`, `control`, `data` (also set chart-wide using `global.plane.mode`) |
| `astronomer.houston.config.auth.openidConnect.flow`                | string  | OIDC OAuth flow.                                                                                                                      | `"implicit"`                          | `"code"`, `"implicit"`                                                       |
| `astronomer.houston.config.jwt.authDuration`                       | integer | Session length bound (ms), coordinated with IdP token lifetimes.                                                                      | `86400000`                            | Positive integer                                                             |
| `astronomer.houston.config.airgapped.enabled`                      | boolean | Air-gapped installation behaviors.                                                                                                    | `false`                               | `true`, `false`                                                              |
| `astronomer.houston.config.updateAirflowCheck.enabled`             | boolean | Enable checks related to Airflow version updates.                                                                                     | `true`                                | `true`, `false`                                                              |
| `astronomer.houston.config.updateRuntimeCheck.enabled`             | boolean | Enable Astro Runtime update checks.                                                                                                   | `true`                                | `true`, `false`                                                              |
| `astronomer.houston.config.sslVerification.enabled`                | boolean | Verify TLS for outbound connections (replaces legacy inverted `disableSSLVerify`).                                                    | `true`                                | `true`, `false`                                                              |
| `astronomer.houston.config.autoCompleteForSensitiveFields.enabled` | boolean | Allow autocomplete on sensitive fields in UI.                                                                                         | `true`                                | `true`, `false`                                                              |
| `astronomer.houston.config.logUsername.enabled`                    | boolean | Include usernames in logs when enabled.                                                                                               | `false`                               | `true`, `false`                                                              |
| `astronomer.houston.config.maxDockerJwtExtraDeployments`           | integer | Upper bound for JWT-scoped docker operations across extra deployments.                                                                | `50`                                  | Non-negative integer                                                         |

For keys not listed here (full auth providers, `workers.*`, `prisma`, `nats`, Helm runtime templating `helm.*`), refer to `config/default.yaml` and [Configuration Flag Migration (2.x)](https://github.com/astronomer/houston-api/blob/main/docs/config-flag-migration-2.x.md) in the APC API repository.

***

## Deployment defaults for the APC API

The following tables list defaults under `astronomer.houston.config.deployments.*`. Keys in this subtree participate in the platform → cluster → workspace → deployment override chain described in [Config governance](/docs/astro-private-cloud/v-2-x/config-governance).

### Operational and platform defaults

| Parameter                                                                    | Type    | Description                                                                                                                    | Default         | Allowed values                                                                    |
| ---------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------ | --------------- | --------------------------------------------------------------------------------- |
| `astronomer.houston.config.deployments.chart.version`                        | string  | Default Airflow chart version for new deployments; also used for upgrade coordination.                                         | `0.0.0`         | Chart version string                                                              |
| `astronomer.houston.config.deployments.releaseVerification`                  | string  | Which image classes may be deployed (`STABLE` only official runtime releases; `EDGE` adds edge images; `DEV` adds dev builds). | `STABLE`        | `STABLE`, `EDGE`, `DEV`; override layers may use `DELETE_KEY` where schema allows |
| `astronomer.houston.config.deployments.subdomain`                            | string  | Subdomain segment used for deployment hostnames.                                                                               | `'deployments'` | DNS label–safe string                                                             |
| `astronomer.houston.config.deployments.performanceOptimization.enabled`      | boolean | Feature gate for performance optimization mode.                                                                                | `false`         | `true`, `false`                                                                   |
| `astronomer.houston.config.deployments.upsertDeployment.extraIniAllowed`     | boolean | Allow `pgbouncer` extra INI fields via upsert API.                                                                             | `true`          | `true`, `false`                                                                   |
| `astronomer.houston.config.deployments.upsertDeployment.allowFromUi.enabled` | boolean | Allow creating/updating deployments from the UI.                                                                               | `true`          | `true`, `false`                                                                   |
| `astronomer.houston.config.deployments.logHelmValues.enabled`                | boolean | Log generated Helm values (debug; noisy).                                                                                      | `false`         | `true`, `false`                                                                   |
| `astronomer.houston.config.deployments.tagPrefix`                            | string  | Prefix for deployment tags on images.                                                                                          | `"deploy"`      | String                                                                            |
| `astronomer.houston.config.deployments.fluentdIndexPrefix`                   | string  | Fluentd index prefix.                                                                                                          | `"fluentd"`     | String                                                                            |

### Runtime management

| Parameter                                                                                      | Type    | Description                                                     | Default                         | Allowed values         |
| ---------------------------------------------------------------------------------------------- | ------- | --------------------------------------------------------------- | ------------------------------- | ---------------------- |
| `astronomer.houston.config.deployments.runtimeManagement.airflowV3.enabled`                    | boolean | Enable Airflow 3–related runtime behaviors where supported.     | `false`                         | `true`, `false`        |
| `astronomer.houston.config.deployments.runtimeManagement.airflowV3.minimumAstroRuntimeVersion` | string  | Minimum Astro Runtime version when Airflow 3 is in play.        | `"3.1-2"`                       | Runtime version string |
| `astronomer.houston.config.deployments.runtimeManagement.customImageSha.enabled`               | boolean | Allow deployment create/update using image SHA selection flows. | `false`                         | `true`, `false`        |
| `astronomer.houston.config.deployments.runtimeManagement.listAllRuntimeVersions.enabled`       | boolean | Expose full runtime version list where applicable.              | `false`                         | `true`, `false`        |
| `astronomer.houston.config.deployments.runtimeManagement.runtimeEnvOverrideSemverCheck`        | string  | SemVer constraint for runtime environment overrides.            | See `default.yaml`              | SemVer range string    |
| `astronomer.houston.config.deployments.runtimeManagement.astroRuntimeReleasesFile`             | string  | Astro Runtime releases manifest file.                           | `'astro_runtime_releases.json'` | Filename               |
| `astronomer.houston.config.deployments.runtimeManagement.airflowMinimumAstroRuntimeVersion`    | string  | Minimum Astro Runtime for Airflow 2 paths.                      | `2.2.5`                         | Version string         |

### Logging

Vector sidecar and optional Elasticsearch client settings for deployment workloads.

| Parameter                                                                   | Type    | Description                                               | Default                                 | Allowed values  |
| --------------------------------------------------------------------------- | ------- | --------------------------------------------------------- | --------------------------------------- | --------------- |
| `astronomer.houston.config.deployments.logging.loggingSidecar.enabled`      | boolean | Enable Vector sidecar for deployment logs.                | `false`                                 | `true`, `false` |
| `astronomer.houston.config.deployments.logging.loggingSidecar.name`         | string  | Sidecar container name.                                   | `sidecar-log-consumer`                  | String          |
| `astronomer.houston.config.deployments.logging.loggingSidecar.image`        | string  | Vector image reference.                                   | `quay.io/astronomer/ap-vector:0.47.0-5` | Image reference |
| `astronomer.houston.config.deployments.logging.loggingSidecar.customConfig` | boolean | Use custom Vector config.                                 | `false`                                 | `true`, `false` |
| `astronomer.houston.config.deployments.logging.elasticsearch.enabled`       | boolean | Enable Elasticsearch logging integration for this domain. | `false`                                 | `true`, `false` |

Logging blocklists for workspace/deployment overrides are described in [Config governance](/docs/astro-private-cloud/v-2-x/config-governance#workspace-level-blocklist).

### Dag deploy

Defaults for Dag-only deployment when that mechanism is enabled under `deployMechanisms`.

| Parameter                                                                     | Type    | Description                                                   | Default                            | Allowed values  |
| ----------------------------------------------------------------------------- | ------- | ------------------------------------------------------------- | ---------------------------------- | --------------- |
| `astronomer.houston.config.deployments.dagDeploy.enabled`                     | boolean | Master enable for Dag-deploy server/client paths in defaults. | `false`                            | `true`, `false` |
| `astronomer.houston.config.deployments.dagDeploy.images.dagServer.repository` | string  | Dag server image repository.                                  | `quay.io/astronomer/ap-dag-deploy` | Repository URL  |
| `astronomer.houston.config.deployments.dagDeploy.images.dagServer.tag`        | string  | Dag server image tag.                                         | `0.7.2`                            | Tag string      |
| `astronomer.houston.config.deployments.dagDeploy.serviceAccount.create`       | boolean | Create service account for Dag deploy components.             | `true`                             | `true`, `false` |

### Deploy mechanisms

Feature gates for Dag deploy, NFS, git-sync, and git-sync relay defaults.

| Parameter                                                                                             | Type    | Description                                            | Default | Allowed values             |
| ----------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------ | ------- | -------------------------- |
| `astronomer.houston.config.deployments.deployMechanisms.configureDagDeployment.enabled`               | boolean | Allow configuring Dag deployment mechanisms.           | `false` | `true`, `false`            |
| `astronomer.houston.config.deployments.deployMechanisms.dagOnlyDeployment.enabled`                    | boolean | Dag-only deployment mechanism.                         | `false` | `true`, `false`            |
| `astronomer.houston.config.deployments.deployMechanisms.nfsMountDagDeployment.enabled`                | boolean | NFS-mounted Dag bundles.                               | `false` | `true`, `false`            |
| `astronomer.houston.config.deployments.deployMechanisms.gitSyncDagDeployment.enabled`                 | boolean | Git-sync–based Dag deployment.                         | `false` | `true`, `false`            |
| `astronomer.houston.config.deployments.deployMechanisms.gitSyncRelay.storageClassName`                | string  | Storage class for relay PVCs.                          | `~`     | Storage class name or null |
| `astronomer.houston.config.deployments.deployMechanisms.gitSyncRelay.webhookSecretKey.showForMinutes` | integer | How long relay webhook secrets are shown in responses. | `1`     | Non-negative integer       |

### Airflow components

| Parameter                                                                      | Type    | Description                   | Default | Allowed values  |
| ------------------------------------------------------------------------------ | ------- | ----------------------------- | ------- | --------------- |
| `astronomer.houston.config.deployments.airflowComponents.triggerer.enabled`    | boolean | Deploy Airflow triggerer.     | `true`  | `true`, `false` |
| `astronomer.houston.config.deployments.airflowComponents.dagProcessor.enabled` | boolean | Deploy Airflow Dag processor. | `true`  | `true`, `false` |

### Auth sidecar

Auth sidecar for routing/auth integration (often cluster-scoped).

| Parameter                                                      | Type    | Description                                    | Default                              | Allowed values   |
| -------------------------------------------------------------- | ------- | ---------------------------------------------- | ------------------------------------ | ---------------- |
| `astronomer.houston.config.deployments.authSideCar.enabled`    | boolean | Deploy auth sidecar template with deployments. | `false`                              | `true`, `false`  |
| `astronomer.houston.config.deployments.authSideCar.repository` | string  | Container image repository.                    | `quay.io/astronomer/ap-auth-sidecar` | Image repository |
| `astronomer.houston.config.deployments.authSideCar.tag`        | string  | Image tag.                                     | `1.29.2`                             | Tag string       |
| `astronomer.houston.config.deployments.authSideCar.port`       | integer | Listen port.                                   | `8084`                               | Port             |

### Resource management

Cluster sizing, executor availability, Astro Units, and capacity ceilings.

| Parameter                                                                            | Type    | Description                                                          | Default                    | Allowed values                               |
| ------------------------------------------------------------------------------------ | ------- | -------------------------------------------------------------------- | -------------------------- | -------------------------------------------- |
| `astronomer.houston.config.deployments.resourceManagement.resourceQuotas.enabled`    | boolean | Manage Kubernetes ResourceQuota / LimitRange objects for namespaces. | `true`                     | `true`, `false`                              |
| `astronomer.houston.config.deployments.resourceManagement.components`                | array   | Component resource profiles (scheduler, workers, …).                 | `[]`                       | Array of component specs; see `default.yaml` |
| `astronomer.houston.config.deployments.resourceManagement.executors`                 | array   | Supported executors and required components per executor.            | Local, Celery, K8s entries | Fixed structure in `default.yaml`            |
| `astronomer.houston.config.deployments.resourceManagement.astroUnit.cpu`             | integer | Millicores per Astro Unit.                                           | `100`                      | Positive integer                             |
| `astronomer.houston.config.deployments.resourceManagement.astroUnit.memory`          | integer | MiB per Astro Unit.                                                  | `384`                      | Positive integer                             |
| `astronomer.houston.config.deployments.resourceManagement.maxExtraCapacity.cpu`      | integer | Max extra CPU (millicores) schedulable above base.                   | `40000`                    | Integer                                      |
| `astronomer.houston.config.deployments.resourceManagement.maxExtraCapacity.memory`   | integer | Max extra memory (MiB).                                              | `153600`                   | Integer                                      |
| `astronomer.houston.config.deployments.resourceManagement.maxPodCapacity.cpu`        | integer | Max CPU per pod (millicores).                                        | `3500`                     | Integer                                      |
| `astronomer.houston.config.deployments.resourceManagement.maxPodCapacity.memory`     | integer | Max memory per pod (MiB).                                            | `13440`                    | Integer                                      |
| `astronomer.houston.config.deployments.resourceManagement.overProvisioningFactorMem` | number  | Memory over-provisioning factor.                                     | `1`                        | Number ≥ 1                                   |
| `astronomer.houston.config.deployments.resourceManagement.overProvisioningFactorCPU` | number  | CPU over-provisioning factor.                                        | `1`                        | Number ≥ 1                                   |

### Namespace management

| Parameter                                                                                  | Type    | Description                                         | Default | Allowed values            |
| ------------------------------------------------------------------------------------------ | ------- | --------------------------------------------------- | ------- | ------------------------- |
| `astronomer.houston.config.deployments.namespaceManagement.manualReleaseNames.enabled`     | boolean | Allow operators to set Helm release names manually. | `false` | `true`, `false`           |
| `astronomer.houston.config.deployments.namespaceManagement.manualNamespaceNames.enabled`   | boolean | Manual Kubernetes namespace names for deployments.  | `false` | `true`, `false`           |
| `astronomer.houston.config.deployments.namespaceManagement.namespaceFreeFormEntry.enabled` | boolean | Free-form namespace entry flows.                    | `false` | `true`, `false`           |
| `astronomer.houston.config.deployments.namespaceManagement.namespaceLabels`                | object  | Labels applied to deployment namespaces.            | `{}`    | String map                |
| `astronomer.houston.config.deployments.namespaceManagement.preCreatedNamespaces`           | array   | Pre-created namespace names for routing.            | `[]`    | Array of `{name}` objects |

`namespaceManagement` is blocklisted for deployment-level overrides (immutable after create); see [Config governance](/docs/astro-private-cloud/v-2-x/config-governance#deployment-level-blocklist).

### Deployment lifecycle

| Parameter                                                                                                      | Type    | Description                                                                                                        | Default | Allowed values   |
| -------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------ | ------- | ---------------- |
| `astronomer.houston.config.deployments.deploymentLifecycle.deployRollback.enabled`                             | boolean | Allow rollback of code deploys / revisions.                                                                        | `false` | `true`, `false`  |
| `astronomer.houston.config.deployments.deploymentLifecycle.deployRollback.deployRevisionReportNumberOfDays`    | integer | Window for deploy revision reporting (days).                                                                       | `90`    | Positive integer |
| `astronomer.houston.config.deployments.deploymentLifecycle.deployRollback.dagTarballVersionValidation.enabled` | boolean | Validate Dag tarball versions on rollback paths.                                                                   | `true`  | `true`, `false`  |
| `astronomer.houston.config.deployments.deploymentLifecycle.hardDeleteDeployment.enabled`                       | boolean | Deprecated — hard delete is unconditionally enabled by default in APC 2.0. This flag no longer gates any behavior. | N/A     | N/A              |
| `astronomer.houston.config.deployments.deploymentLifecycle.cleanupAirflowDb.enabled`                           | boolean | Enable cleanup jobs for Airflow metadata DB.                                                                       | `false` | `true`, `false`  |

### Database management

| Parameter                                                                                       | Type    | Description                                                         | Default | Allowed values        |
| ----------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------- | ------- | --------------------- |
| `astronomer.houston.config.deployments.databaseManagement.database.enabled`                     | boolean | Provision per-deployment Airflow databases when enabled.            | `true`  | `true`, `false`       |
| `astronomer.houston.config.deployments.databaseManagement.database.retainOnDelete`              | boolean | Keep DB objects after deployment deletion.                          | `false` | `true`, `false`       |
| `astronomer.houston.config.deployments.databaseManagement.database.allowRootAccess`             | boolean | Allow root-level DB access patterns documented in the APC API.      | `false` | `true`, `false`       |
| `astronomer.houston.config.deployments.databaseManagement.manualConnectionStrings.enabled`      | boolean | Allow manual connection string configuration.                       | `false` | `true`, `false`       |
| `astronomer.houston.config.deployments.databaseManagement.pgBouncerResourceCalculationStrategy` | string  | Strategy key for PgBouncer sizing (`null` uses built-in algorithm). | `~`     | Strategy name or null |

### Image registry

| Parameter                                                                                                              | Type    | Description                                                                                                        | Default | Allowed values         |
| ---------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------ | ------- | ---------------------- |
| `astronomer.houston.config.deployments.deploymentImagesRegistry.exposeDockerWebhookEndpoint.enabled`                   | boolean | Expose webhook endpoint for image push flows.                                                                      | `true`  | `true`, `false`        |
| `astronomer.houston.config.deployments.deploymentImagesRegistry.updateDeploymentImageEndpoint.enabled`                 | boolean | Allow API to update deployment images directly.                                                                    | `false` | `true`, `false`        |
| `astronomer.houston.config.deployments.deploymentImagesRegistry.updateDeploymentImageEndpointDockerValidation.enabled` | boolean | Validate docker payloads on image update endpoint.                                                                 | `false` | `true`, `false`        |
| `astronomer.houston.config.deployments.deploymentImagesRegistry.serviceAccountAnnotationKey`                           | string  | Cloud IAM annotation key on workload identity (`eks.amazonaws.com/role-arn`, `iam.gke.io/gcp-service-account`, …). | `~`     | Annotation key or null |

### Metrics reporting

| Parameter                                                                                    | Type    | Description                                            | Default | Allowed values   |
| -------------------------------------------------------------------------------------------- | ------- | ------------------------------------------------------ | ------- | ---------------- |
| `astronomer.houston.config.deployments.metricsReporting.grafana.enabled`                     | boolean | Surface Grafana links / metrics UI integration points. | `true`  | `true`, `false`  |
| `astronomer.houston.config.deployments.metricsReporting.taskUsageMetrics.enabled`            | boolean | Enable task usage metrics reporting features.          | `false` | `true`, `false`  |
| `astronomer.houston.config.deployments.metricsReporting.taskUsageMetrics.reportNumberOfDays` | integer | Reporting horizon for task usage (days).               | `90`    | Positive integer |

Pagination `maxTake` limits under `metricsReporting.pagination.*` follow defaults in `config/default.yaml` (typically `101` per collection).

### Orchestration mode

Deployment orchestration mode (Helm vs operator).

| Parameter                                                     | Type    | Description                                                   | Default | Allowed values  |
| ------------------------------------------------------------- | ------- | ------------------------------------------------------------- | ------- | --------------- |
| `astronomer.houston.config.deployments.mode.helm.enabled`     | boolean | Use Helm-based deployment reconciler.                         | `true`  | `true`, `false` |
| `astronomer.houston.config.deployments.mode.operator.enabled` | boolean | Use Kubernetes operator–based deployment path when supported. | `false` | `true`, `false` |

Operator probes and detailed probe specs are defined under `deployments.mode.operator` in `default.yaml`.

### Deployments Helm defaults

The `deployments.helm` subtree supplies defaults merged into Airflow Helm chart values for each deployment (scheduler resources, PgBouncer sidecars, ingress, runtime images, Elasticsearch hooks, etc.). It is large and nested; defaults are authoritative in [`config/default.yaml`](https://github.com/astronomer/houston-api/blob/main/config/default.yaml) under `deployments.helm`. For merge and validation rules at override layers, rely on `deployments-config-override.schema.json` and [Config governance](/docs/astro-private-cloud/v-2-x/config-governance).

### Mock webhook (test only)

Test-only mock webhook settings for development scenarios.

| Parameter                                                          | Type    | Description                                  | Default | Allowed values  |
| ------------------------------------------------------------------ | ------- | -------------------------------------------- | ------- | --------------- |
| `astronomer.houston.config.deployments.mockWebhook.enabled`        | boolean | Enable mock webhook server wiring.           | `false` | `true`, `false` |
| `astronomer.houston.config.deployments.mockWebhook.krb.enabled`    | boolean | Kerberos realm defaults for mock.            | `true`  | `true`, `false` |
| `astronomer.houston.config.deployments.mockWebhook.shouldCreateDb` | boolean | Auto-create backing DB objects in mock mode. | `true`  | `true`, `false` |

***

## Global configuration

### Base domain and TLS

```yaml wrap theme={null}
global:
  # Required: Base domain for all platform endpoints
  # Results in: app.example.com, houston.example.com, etc.
  baseDomain: "example.com"

  # Name of Kubernetes secret containing TLS certificate
  tlsSecret: "astronomer-tls"

  # List of secrets containing private CA certificates
  privateCaCerts: []
```

### Plane mode (control, data, unified)

Astro Private Cloud 2.0 supports split control plane and data plane deployments:

```yaml wrap theme={null}
global:
  plane:
    # Options: unified (default), control, data
    mode: "unified"

    # Domain prefix for this plane (used in split deployments)
    domainPrefix: ""
```

| Mode      | Description                                                |
| --------- | ---------------------------------------------------------- |
| `unified` | Control and data plane in same cluster (default, like 0.x) |
| `control` | Control plane only - manages Deployments                   |
| `data`    | Data plane only - runs Airflow workloads                   |

### Network policies

```yaml wrap theme={null}
global:
  # Enable platform-level network policies
  networkPolicy:
    enabled: true

  # Apply default deny ingress policy
  defaultDenyNetworkPolicy: true

  # Enable namespace labels for network policies
  networkNSLabels:
    enabled: false
```

### RBAC and cluster roles

```yaml wrap theme={null}
global:
  # Enable Kubernetes RBAC
  rbac:
    enabled: true

  # Use cluster-wide roles (required for some features)
  clusterRoles: true

  # Management of cluster-scoped resources (RBAC objects, etc.)
  manageClusterScopedResources:
    enabled: true
```

### Node selection

Separate platform Pods from Airflow Pods:

```yaml wrap theme={null}
global:
  platformNodePool:
    nodeSelector:
      node-role: platform
    affinity: {}
    tolerations:
      - key: "platform"
        operator: "Equal"
        value: "true"
        effect: "NoSchedule"
```

### Private registry

Use a private container registry:

```yaml wrap theme={null}
global:
  privateRegistry:
    enabled: true
    repository: "registry.example.com/astronomer"
    secretName: "registry-credentials"
```

### Namespace pools

Pre-provision namespaces for Airflow Deployments:

```yaml wrap theme={null}
global:
  namespaceManagement:
    namespacePools:
      enabled: true
      createRbac: true
      namespaces:
        create: false  # Set true to auto-create
        names:
          - airflow-prod
          - airflow-staging
          - airflow-dev
```

### Storage class

Specify a storage class for all persistent volumes:

```yaml wrap theme={null}
global:
  storageClass: "gp3"
```

### OpenShift support

```yaml wrap theme={null}
global:
  openshift:
    enabled: true
  scc:
    enabled: true  # Security context constraints
```

## Astronomer platform components

### APC API

The APC API is the core internal API that powers the platform:

```yaml expandable wrap theme={null}
astronomer:
  houston:
    replicas: 2

    resources:
      requests:
        cpu: "500m"
        memory: "1024Mi"
      limits:
        cpu: "1000m"
        memory: "2048Mi"

    # Database connection
    backendSecretName: "houston-backend-secret"
    # Or specify directly:
    backendConnection:
      user: houston
      pass: "<YOUR_DATABASE_PASSWORD>"
      host: postgres.example.com
      port: 5432
      db: houston

    # Airflow database connection template
    airflowBackendSecretName: "airflow-backend-secret"

    # APC API configuration (see the following APC API configuration section)
    config: {}

    # Environment variables common to all houston containers
    env:
      - name: LOG_LEVEL
        value: "info"

    # Worker pods for async processing
    worker:
      enabled: true
      replicas: 2

    # Upgrade all airflow helm deployments when upgrading APC helm deployment
    upgradeDeployments:
      enabled: true

    # Cleanup soft-deleted deployments
    cleanupDeployments:
      enabled: true
      schedule: "0 0 * * *"
      olderThan: 14

    # Cleanup Airflow database metadata
    cleanupAirflowDb:
      enabled: false
      schedule: "23 5 * * *"
      olderThan: 365
```

### APC API configuration examples (`houston.config`)

The APC API accepts extensive configuration via `houston.config`:

#### Authentication

```yaml expandable wrap theme={null}
astronomer:
  houston:
    config:
      auth:
        # Local username/password auth
        local:
          enabled: true

        # OpenID Connect
        openidConnect:
          # Auth flow: "code" (recommended) or "implicit"
          flow: "code"

          # Microsoft/Azure AD
          microsoft:
            enabled: true
            clientId: "<YOUR_CLIENT_ID>"
            clientSecret: "<YOUR_CLIENT_SECRET>"
            discoveryUrl: "https://login.microsoftonline.com/<TENANT_ID>/v2.0/.well-known/openid-configuration"

          # Google OAuth
          google:
            enabled: false
            clientId: ""
            clientSecret: ""

          # Okta
          okta:
            enabled: false
            clientId: ""
            clientSecret: ""
            discoveryUrl: ""

          # Import groups from IDP (nested `.enabled` in APC 2.0)
          idpGroupsImport:
            enabled: true
          idpGroupsRefresh:
            enabled: false

        # GitHub via Auth0
        github:
          enabled: false
```

#### Deployment defaults (`deployments`)

APC 2.0 groups deployment-related settings under domains (for example `deployMechanisms`, `deploymentLifecycle`, `runtimeManagement`) using nested objects and `feature.enabled` toggles—not flat keys such as `dagOnlyDeployment` or `hardDeleteDeployment`.

For tables of defaults, allowed values, and override semantics, see [APC API configuration](#apc-api-configuration) earlier on this page. For how cluster, workspace, and deployment layers merge, see [Config governance](/docs/astro-private-cloud/v-2-x/config-governance).

Example shape (abbreviated):

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        deployMechanisms:
          dagOnlyDeployment:
            enabled: true
          gitSyncDagDeployment:
            enabled: true
        deploymentLifecycle:
          hardDeleteDeployment:
            enabled: false
          deployRollback:
            enabled: true
        resourceManagement:
          components: []
```

Airflow Helm value defaults merged per deployment live under `deployments.helm` in the APC API’s `config/default.yaml`; see [Deployments Helm defaults](#deployments-helm-defaults) earlier on this page.

#### Email configuration

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      email:
        enabled: true
        smtpUrl: "smtp://smtp.example.com:587"
        reply: "noreply@example.com"

      # Root-level toggles (APC 2.0 nested `.enabled` pattern)
      emailConfirmation:
        enabled: true

      publicSignups:
        enabled: false
```

#### Prometheus integration

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      prometheus:
        enabled: true
        host: "http://astronomer-prometheus:9090"
```

### Deployment orchestrator

The deployment orchestrator manages Kubernetes resources for Deployments:

```yaml wrap theme={null}
astronomer:
  commander:
    replicas: 2

    resources:
      requests:
        cpu: "250m"
        memory: "1Gi"
      limits:
        cpu: "500m"
        memory: "2Gi"

    # Air-gapped mode (no external registry access)
    airGapped:
      enabled: false

    # Helm upgrade timeout (seconds)
    upgradeTimeout: 600

    # Environment variables
    env: []
```

### Registry

Container registry for deployment images:

```yaml expandable wrap theme={null}
astronomer:
  registry:
    replicas: 1

    resources:
      requests:
        cpu: "250m"
        memory: "512Mi"
      limits:
        cpu: "500m"
        memory: "1024Mi"

    # Persistent storage
    persistence:
      enabled: true
      size: "100Gi"
      storageClassName: ~

    # Use external storage backends
    # AWS S3
    s3:
      enabled: false
      accesskey: ""
      secretkey: ""
      region: "us-east-1"
      bucket: "astronomer-registry"

    # Google Cloud Storage
    gcs:
      enabled: false
      bucket: ""
      useKeyfile: true
      keyfile: /var/gcs-keyfile/astronomer-gcs-keyfile

    # Azure Blob Storage
    azure:
      enabled: false
      accountname: ""
      accountkey: ""
      container: ""
```

### Astro UI

```yaml wrap theme={null}
astronomer:
  astroUI:
    replicas: 2

    resources:
      requests:
        cpu: "100m"
        memory: "256Mi"
      limits:
        cpu: "500m"
        memory: "1024Mi"

    env: []
```

## NGINX ingress

```yaml expandable wrap theme={null}
nginx:
  replicas: 2

  resources:
    requests:
      cpu: "500m"
      memory: "1024Mi"
    limits:
      cpu: "1"
      memory: "2048Mi"

  # Service type: LoadBalancer, ClusterIP, or NodePort
  serviceType: "LoadBalancer"

  # Specific load balancer IP (optional)
  loadBalancerIP: ~

  # Restrict source IPs
  loadBalancerSourceRanges:
    - "10.0.0.0/8"

  # Private/internal load balancer
  privateLoadBalancer: false

  # NodePort configuration (when serviceType: NodePort)
  httpNodePort: ~
  httpsNodePort: ~

  # Ingress annotations
  ingressAnnotations:
    # AWS
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    # GCP
    cloud.google.com/load-balancer-type: "Internal"
    # Azure
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

  # Proxy settings
  proxyConnectTimeout: 15
  proxyReadTimeout: 600
  proxySendTimeout: 600
  proxyBodySize: "1024m"

  # Default backend
  defaultBackend:
    enabled: true
    resources:
      requests:
        cpu: "100m"
        memory: "50Mi"
```

## Prometheus (monitoring)

```yaml wrap theme={null}
prometheus:
  # Data retention period
  retention: 15d

  # Persistent storage
  persistence:
    enabled: true
    size: "150Gi"

  resources:
    requests:
      cpu: "1000m"
      memory: "4Gi"
    limits:
      cpu: "2000m"
      memory: "8Gi"
```

### Node exporter tolerations

<Note>
  **Astro Private Cloud 2.1**

  This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
</Note>

`prometheus-node-exporter` runs as a DaemonSet. APC applies the following default toleration so node exporter schedules onto every node, including tainted control-plane nodes, which it needs in order to collect complete host metrics:

```yaml wrap theme={null}
prometheus-node-exporter:
  tolerations:
    - effect: NoSchedule
      operator: Exists
```

If your cluster policy disallows a broad toleration like this, override `prometheus-node-exporter.tolerations` with a narrower list, or set it to an empty list to remove the toleration entirely:

```yaml wrap theme={null}
prometheus-node-exporter:
  tolerations: []
```

<Note>
  `prometheus-node-exporter.tolerations` is different from `global.platformNodePool.tolerations` under [Node selection](#node-selection). The `platformNodePool` value adds a toleration so platform Pods can schedule onto nodes you've tainted for dedicated platform use. `prometheus-node-exporter.tolerations` is the node exporter DaemonSet's own toleration list, which controls whether node exporter itself can schedule onto tainted nodes.
</Note>

## Grafana

```yaml wrap theme={null}
grafana:
  resources:
    requests:
      cpu: "250m"
      memory: "512Mi"
    limits:
      cpu: "500m"
      memory: "1024Mi"

  # Custom dashboards
  dashboards:
    default:
      custom-dashboard:
        file: dashboards/custom.json

  # Extra environment variables (for example, SMTP for alerts)
  extraEnvVars:
    - name: GF_SMTP_ENABLED
      value: "true"
    - name: GF_SMTP_HOST
      value: "smtp.example.com:587"
```

## Elasticsearch (logging)

```yaml expandable wrap theme={null}
elasticsearch:
  # Enable persistence
  common:
    persistence:
      enabled: true

  # Client nodes
  client:
    replicas: 2
    heapMemory: "2g"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "4Gi"

  # Data nodes
  data:
    replicas: 3
    heapMemory: "2g"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "4Gi"
    persistence:
      size: "100Gi"

  # Master nodes
  master:
    replicas: 3
    heapMemory: "2g"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "4Gi"
    persistence:
      size: "20Gi"
```

## Vector (log collection)

```yaml wrap theme={null}
vector:
  vector:
    resources:
      requests:
        cpu: "250m"
        memory: "512Mi"
      limits:
        cpu: "1000m"
        memory: "1024Mi"
```

## External logging

Forward logs to external Elasticsearch:

```yaml wrap theme={null}
global:
  customLogging:
    enabled: true
    scheme: https
    host: "elasticsearch.example.com"
    port: "9200"
    secret: "es-credentials"
```

## NATS (messaging)

```yaml wrap theme={null}
global:
  nats:
    enabled: true
    replicas: 3
    jetStream:
      enabled: true
      tls: false

nats:
  nats:
    resources:
      requests:
        cpu: "75m"
        memory: "30Mi"
      limits:
        cpu: "250m"
        memory: "100Mi"
```

## Database configuration

### External PostgreSQL (recommended)

```yaml wrap theme={null}
global:
  # Disable in-cluster PostgreSQL
  postgresql:
    enabled: false

astronomer:
  houston:
    backendSecretName: "houston-db-secret"
    # Secret should contain: connection=postgres://user:pass@host:5432/houston
    airflowBackendSecretName: "airflow-db-secret"
```

### Database SSL

```yaml wrap theme={null}
global:
  ssl:
    enabled: true
    mode: "require"  # disable, allow, prefer, require, verify-ca, verify-full
    grafana:
      sslmode: "require"
```

### PgBouncer (connection pooling)

```yaml wrap theme={null}
global:
  pgbouncer:
    enabled: true
    gssSupport: true
    secretName: "astronomer-pgbouncer-config"
    servicePort: "6543"
```

## Auth sidecar (OpenShift)

`global.authSidecar` configures the chart-level platform auth proxy used for OpenShift and similar environments. This is separate from `astronomer.houston.config.deployments.authSideCar`, which controls the per-Deployment auth sidecar injected by the APC API (default tag `1.29.2`).

```yaml wrap theme={null}
global:
  authSidecar:
    enabled: true
    repository: quay.io/astronomer/ap-auth-sidecar
    tag: 1.29.8
    port: 8084
    resources:
      requests:
        cpu: "500m"
        memory: "512Mi"
      limits:
        cpu: "1000m"
        memory: "1024Mi"
```

## Logging sidecar

Add Vector sidecar to Airflow Pods:

```yaml wrap theme={null}
global:
  logging:
    loggingSidecar:
      enabled: true
      name: sidecar-log-consumer
      repository: quay.io/astronomer/ap-vector
      tag: 0.54.0
      resources:
        requests:
          cpu: "100m"
          memory: "386Mi"
```

## Dag-only deployments

```yaml wrap theme={null}
global:
  deployMechanisms:
    dagOnlyDeployment:
      enabled: true
      repository: quay.io/astronomer/ap-dag-deploy
      tag: 0.9.4
      resources: {}
      persistence: {}
```

## Airflow operator

Enable Kubernetes operator-based deployments:

```yaml wrap theme={null}
global:
  airflowOperator:
    enabled: false
```

## Extra objects

Add custom Kubernetes resources:

```yaml wrap theme={null}
astronomer:
  extraObjects:
    # Custom LimitRange
    - apiVersion: v1
      kind: LimitRange
      metadata:
        name: default-limits
        namespace: astronomer
      spec:
        limits:
          - default:
              cpu: "1"
              memory: "1Gi"
            defaultRequest:
              cpu: "100m"
              memory: "128Mi"
            type: Container

    # Custom NetworkPolicy
    - apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: custom-policy
      spec:
        podSelector: {}
        policyTypes:
          - Ingress
```

## Complete example

Here's an example configuration. The `astronomer.houston.config.deployments.*` values shown below only seed the platform default the first time a cluster is registered — to change these settings for a cluster that's already running, use the **Configuration Override** section of the Astro UI or the `updateCluster` API instead (see [Override data plane cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster)).

```yaml expandable wrap theme={null}
global:
  baseDomain: "airflow.example.com"
  tlsSecret: "astronomer-tls"

  plane:
    mode: "unified"

  rbac:
    enabled: true
  clusterRoles: true

  networkPolicy:
    enabled: true

  postgresql:
    enabled: false

  privateRegistry:
    enabled: true
    repository: "registry.example.com/astronomer"
    secretName: "registry-creds"

  platformNodePool:
    nodeSelector:
      node-type: platform
    tolerations:
      - key: "dedicated"
        value: "platform"
        effect: "NoSchedule"

  ssl:
    enabled: true
    mode: "require"

tags:
  platform: true
  monitoring: true
  logging: true

astronomer:
  houston:
    replicas: 2
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "1000m"
        memory: "2Gi"
    backendSecretName: "houston-db-secret"
    config:
      auth:
        local:
          enabled: false
        openidConnect:
          microsoft:
            enabled: true
            clientId: "<YOUR_CLIENT_ID>"
            clientSecret: "<YOUR_CLIENT_SECRET>"
            discoveryUrl: "https://login.microsoftonline.com/<TENANT_ID>/v2.0/.well-known/openid-configuration"
      email:
        enabled: true
        smtpUrl: "smtp://smtp.example.com:587"
      publicSignups:
        enabled: false
      deployments:
        namespaceManagement:
          manualReleaseNames:
            enabled: true

  commander:
    replicas: 2
    resources:
      requests:
        cpu: "250m"
        memory: "1Gi"
      limits:
        cpu: "500m"
        memory: "2Gi"

  registry:
    persistence:
      enabled: true
      size: "200Gi"

nginx:
  replicas: 2
  serviceType: LoadBalancer
  privateLoadBalancer: true
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"

prometheus:
  retention: 30d
  persistence:
    enabled: true
    size: "200Gi"
  resources:
    requests:
      cpu: "1"
      memory: "4Gi"

elasticsearch:
  data:
    replicas: 3
    persistence:
      size: "200Gi"
    resources:
      requests:
        cpu: "1"
        memory: "4Gi"
```

## Validate configuration

After creating your values file, validate it:

```bash wrap theme={null}
# Dry-run to check for errors
helm template astronomer astronomer/astronomer \
  -f values.yaml \
  --namespace astronomer \
  --debug

# Check rendered templates
helm template astronomer astronomer/astronomer \
  -f values.yaml \
  --namespace astronomer > rendered.yaml
```

## Upgrade configuration

(Optional) When updating your values file, you can use the [`helm diff`](https://github.com/databus23/helm-diff) plugin, and then run the following command to see a diff of your changes:

```bash wrap theme={null}
# Compare changes
helm diff upgrade astronomer astronomer/astronomer \
  -f values.yaml \
  --namespace astronomer

# Apply changes
helm upgrade astronomer astronomer/astronomer \
  -f values.yaml \
  --namespace astronomer
```

## Related documentation

* [Configure Astro Private Cloud](/docs/astro-private-cloud/v-2-x/configure-astro-private-cloud)
* [Config governance](/docs/astro-private-cloud/v-2-x/config-governance)
* [Configure platform resources](/docs/astro-private-cloud/v-2-x/configure-platform-resources)


# Assign a team role with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-assign-team-role

Assign a team a role on a Workspace or Deployment using the APC API workspaceAddTeam and deploymentAddTeamRole mutations.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

### Assign a team to a workspace

<Note>
  If you omit `role`, the team defaults to `WORKSPACE_VIEWER`.
</Note>

```graphql wrap theme={null}
mutation {
  workspaceAddTeam(
    teamUuid: "<team-uuid>"
    workspaceUuid: "<workspace-uuid>"
    role: WORKSPACE_EDITOR
  ) {
    id
    label
  }
}
```

Assign workspace and deployment roles in a single mutation:

```graphql wrap theme={null}
mutation {
  workspaceAddTeam(
    teamUuid: "<team-uuid>"
    workspaceUuid: "<workspace-uuid>"
    role: WORKSPACE_VIEWER
    deploymentRoles: [
      { deploymentId: "<deployment-uuid-1>", role: DEPLOYMENT_ADMIN }
      { deploymentId: "<deployment-uuid-2>", role: DEPLOYMENT_EDITOR }
    ]
  ) {
    id
  }
}
```

### Assign a team to a Deployment

```graphql wrap theme={null}
mutation {
  deploymentAddTeamRole(
    teamUuid: "<team-uuid>"
    deploymentUuid: "<deployment-uuid>"
    role: DEPLOYMENT_EDITOR
  ) {
    id
    role
  }
}
```

For the full teams model, roles, and error reference, see [team management reference](/docs/astro-private-cloud/v-2-x/team-management-api).


# Delete workspace configuration with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-delete-workspace-config

Remove all workspace-level Deployment configuration overrides using the APC API deleteWorkspaceDeploymentsConfig mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

Deleting workspace configuration removes all workspace-level overrides, reverting all Deployments in the workspace to use cluster-level and platform-level defaults.

Use the `deleteWorkspaceDeploymentsConfig` mutation:

```graphql wrap theme={null}
mutation {
  deleteWorkspaceDeploymentsConfig(
    workspaceUuid: "<workspace-id>"
    reason: "Revert workspace to cluster defaults"
  ) {
    id
    config
    deletedAt
  }
}
```

For the full config governance model these overrides participate in, see [Config governance](/docs/astro-private-cloud/v-2-x/config-governance).


# Deregister a cluster with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-deregister-cluster

Remove a data plane cluster from the control plane using the APC API deregisterCluster mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

Send the request to `https://houston.<your-base-domain>/v1` with a system service account or System Admin user token in the `Authorization` header. For details, see [Authenticate to the APC API](/docs/astro-private-cloud/v-2-x/houston-api-authenticate).

```graphql wrap theme={null}
mutation {
  deregisterCluster(id: "<cluster-id>") {
    id
    name
  }
}
```

Argument reference:

| Argument | Type   | Required | Description                                                                                     |
| -------- | ------ | -------- | ----------------------------------------------------------------------------------------------- |
| `id`     | `Uuid` | Yes      | The unique identifier of the cluster to deregister. Find it with the `paginatedClusters` query. |

<Warning>
  This permanently deletes the cluster record from the control plane. Deregistration doesn't delete platform resources running in the data plane Kubernetes cluster (namespaces, services, persistent volumes). Remove those by uninstalling the data plane Helm release.
</Warning>

For the full deregistration workflow, including the UI flow and prerequisites, see [Deregister a data plane cluster](/docs/astro-private-cloud/v-2-x/deregister-data-plane).


# Manage Workspace users with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-manage-workspace-users



<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

You can use the APC API to manage the users in a Workspace. These mutations require Workspace Admin permissions on the target Workspace. Each mutation requires a `workspaceUuid`, which you can retrieve by running `astro workspace list` or using the [`workspaces`](/docs/astro-private-cloud/v-2-x/houston-api-example-queries#workspaces) query.

The `role` value must be one of `WORKSPACE_ADMIN`, `WORKSPACE_EDITOR`, or `WORKSPACE_VIEWER`.

## Add a user to a Workspace

Use the `workspaceAddUser` mutation to add an existing user to a Workspace with a specified role.

```graphql wrap theme={null}
mutation AddWorkspaceUser(
  $workspaceUuid: Uuid! = "<workspace-id>"
  $email: String! = "<user-email-address>"
  $role: Role! = WORKSPACE_VIEWER
) {
  workspaceAddUser(
    workspaceUuid: $workspaceUuid
    email: $email
    role: $role
  ) {
    id
    label
    users {
      id
      username
      roleBindings {
        role
      }
    }
  }
}
```

## Update a user's Workspace role

Use the `workspaceUpsertUserRole` mutation to change the role of a user who already belongs to a Workspace.

```graphql wrap theme={null}
mutation UpdateWorkspaceUserRole(
  $workspaceUuid: Uuid! = "<workspace-id>"
  $email: String! = "<user-email-address>"
  $role: Role! = WORKSPACE_EDITOR
) {
  workspaceUpsertUserRole(
    workspaceUuid: $workspaceUuid
    email: $email
    role: $role
  )
}
```

## Remove a user from a Workspace

Use the `workspaceRemoveUser` mutation to remove a user from a Workspace. Provide the `workspaceUuid` and the `userUuid` of the user to remove. To retrieve the `userUuid`, use the [`workspaceUsers`](/docs/astro-private-cloud/v-2-x/houston-api-example-queries#workspace-users) query or a `users` query.

```graphql wrap theme={null}
mutation RemoveWorkspaceUser(
  $workspaceUuid: Uuid! = "<workspace-id>"
  $userUuid: Uuid! = "<user-id>"
) {
  workspaceRemoveUser(
    workspaceUuid: $workspaceUuid
    userUuid: $userUuid
  ) {
    id
    label
    users {
      id
      username
    }
  }
}
```


# Manage Workspaces with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-manage-workspaces



<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

You can use the APC API to create, update, and delete Workspaces programmatically. These mutations require System Admin permissions, or Workspace Admin permissions on the target Workspace when updating or deleting it.

## Create a Workspace

Use the `createWorkspace` mutation to create a Workspace. The `label` is required, and `description` is optional. The mutation returns details about the new Workspace, including its `id`.

```graphql wrap theme={null}
mutation CreateWorkspace(
  $label: String! = "<workspace-label>"
  $description: String = "<workspace-description>"
) {
  createWorkspace(
    label: $label
    description: $description
  ) {
    id
    label
    description
    createdAt
    updatedAt
  }
}
```

## Update a Workspace

Use the `updateWorkspace` mutation to update an existing Workspace. Provide the `workspaceUuid` and a `payload` that contains the fields to update, such as `label` and `description`. To retrieve the `workspaceUuid`, run `astro workspace list` or use the [`workspaces`](/docs/astro-private-cloud/v-2-x/houston-api-example-queries#workspaces) query.

```graphql wrap theme={null}
mutation UpdateWorkspace(
  $workspaceUuid: Uuid! = "<workspace-id>"
  $payload: JSON! = { label: "<new-label>", description: "<new-description>" }
) {
  updateWorkspace(
    workspaceUuid: $workspaceUuid
    payload: $payload
  ) {
    id
    label
    description
    createdAt
    updatedAt
  }
}
```

## Delete a Workspace

Use the `deleteWorkspace` mutation to delete a Workspace. Provide the `workspaceUuid` of the Workspace to delete. The mutation returns details about the deleted Workspace to confirm the operation.

```graphql wrap theme={null}
mutation DeleteWorkspace(
  $workspaceUuid: Uuid! = "<workspace-id>"
) {
  deleteWorkspace(
    workspaceUuid: $workspaceUuid
  ) {
    id
    label
    description
  }
}
```


# Remove a team with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-remove-team

Remove a local or IdP team using the APC API removeTeam mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

### Remove by UUID

```graphql wrap theme={null}
mutation {
  removeTeam(teamUuid: "<team-uuid>") {
    id
    name
  }
}
```

### Remove by name and provider

```graphql wrap theme={null}
mutation {
  removeTeam(
    name: "Data Engineering"
    provider: "local"
  ) {
    id
    name
  }
}
```

<Note>
  You can only remove IdP teams that have no attached users.
</Note>

For the full teams model, roles, and error reference, see [team management reference](/docs/astro-private-cloud/v-2-x/team-management-api).


# Remove a team's role with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-remove-team-role

Remove a team's role from a Workspace or Deployment using the APC API workspaceRemoveTeam and deploymentRemoveTeamRole mutations.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

### Remove a team from a workspace

```graphql wrap theme={null}
mutation {
  workspaceRemoveTeam(
    teamUuid: "<team-uuid>"
    workspaceUuid: "<workspace-uuid>"
  ) {
    id
  }
}
```

### Remove a team from a Deployment

```graphql wrap theme={null}
mutation {
  deploymentRemoveTeamRole(
    teamUuid: "<team-uuid>"
    deploymentUuid: "<deployment-uuid>"
  ) {
    id
  }
}
```

For the full teams model, roles, and error reference, see [team management reference](/docs/astro-private-cloud/v-2-x/team-management-api).


# Update a cluster with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-update-cluster

Change a data plane cluster's status or configuration using the APC API updateCluster mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

A user with permission to update clusters can change a cluster's status manually. The `statusReason` argument accepts a JSON object whose shape isn't enforced by the schema, but the APC API itself writes the value the deployment orchestrator returns in its `/metadata` response when reconciling. To stay consistent, use the same shape the APC API uses or include a descriptive `message` field.

```graphql wrap theme={null}
mutation {
  updateCluster(
    id: "<cluster-id>"
    status: INACTIVE
    statusReason: { message: "Maintenance window — cluster offline for upgrades" }
  ) {
    id
    status
    statusReason
  }
}
```

For status changes, supply `id` (required), `status`, and `statusReason`. The `updateCluster` mutation also accepts `name` and `deploymentsConfigOverride` for non-status changes; see [Update data plane cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster) for those workflows.

<Note>
  The APC API blocks configuration updates (`deploymentsConfigOverride`, `name`) while the cluster status is `INACTIVE` and returns the error `This operation is not allowed as the cluster is not active.` Status itself can still be updated in any state.
</Note>

To manually restore a cluster to `ACTIVE` after confirming it's healthy:

```graphql wrap theme={null}
mutation {
  updateCluster(
    id: "<cluster-id>"
    status: ACTIVE
    statusReason: { message: "Manually verified healthy" }
  ) {
    id
    status
  }
}
```

For querying cluster status and troubleshooting unhealthy clusters, see [Manage cluster status](/docs/astro-private-cloud/v-2-x/cluster-status-management). For updating a cluster's `deploymentsConfigOverride`, see [Update data plane cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster).


# Update a team with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-update-team

Update a team's name, description, or membership using the APC API updateTeam mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

### Update team details

```graphql wrap theme={null}
mutation {
  updateTeam(
    id: "<team-uuid>"
    newName: "Platform Engineering"
    description: "Updated description"
  ) {
    team {
      id
      name
      description
    }
    message
  }
}
```

### Add users to local team

```graphql wrap theme={null}
mutation {
  updateTeam(
    id: "<team-uuid>"
    addUserIds: ["<user-uuid-3>", "<user-uuid-4>"]
  ) {
    team {
      id
      users {
        id
        username
      }
    }
  }
}
```

### Remove users from local team

```graphql wrap theme={null}
mutation {
  updateTeam(
    id: "<team-uuid>"
    removeUserIds: ["<user-uuid-1>"]
  ) {
    team {
      id
      users {
        id
        username
      }
    }
  }
}
```

### Replace all users

```graphql wrap theme={null}
mutation {
  updateTeam(
    id: "<team-uuid>"
    teamUserIds: ["<user-uuid-5>", "<user-uuid-6>"]
  ) {
    team {
      users {
        id
        username
      }
    }
  }
}
```

### Update by name (alternative)

Team names are unique per provider, not globally. You must include `provider` alongside `name` to uniquely identify a team.

```graphql wrap theme={null}
mutation {
  updateTeam(
    name: "Data Engineering"
    provider: "local"
    newName: "Data Platform"
  ) {
    team {
      id
      name
    }
  }
}
```

For the full teams model, roles, and error reference, see [team management reference](/docs/astro-private-cloud/v-2-x/team-management-api).


# Update a team's role with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-update-team-role

Change a team's role on a Workspace or Deployment using the APC API workspaceUpdateTeamRole and deploymentUpdateTeamRole mutations.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

### Update a team's workspace role

```graphql wrap theme={null}
mutation {
  workspaceUpdateTeamRole(
    teamUuid: "<team-uuid>"
    workspaceUuid: "<workspace-uuid>"
    role: WORKSPACE_ADMIN
  )
}
```

### Update a team's Deployment role

```graphql wrap theme={null}
mutation {
  deploymentUpdateTeamRole(
    teamUuid: "<team-uuid>"
    deploymentUuid: "<deployment-uuid>"
    role: DEPLOYMENT_ADMIN
  ) {
    id
    role
  }
}
```

For the full teams model, roles, and error reference, see [team management reference](/docs/astro-private-cloud/v-2-x/team-management-api).


# Update workspace configuration with the APC API
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-update-workspace-config

Add or update workspace-level Deployment configuration overrides using the APC API updateWorkspaceDeploymentsConfig mutation.

<Note>
  The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston.<your-base-domain>/v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer.
</Note>

Use the `updateWorkspaceDeploymentsConfig` mutation to add or update workspace-level overrides:

```graphql wrap theme={null}
mutation {
  updateWorkspaceDeploymentsConfig(
    workspaceUuid: "<workspace-id>"
    deploymentsConfigOverride: {
      deploymentLifecycle: {
        deployRollback: {
          enabled: true
        }
      }
    }
    reason: "Enable deploy rollback for all deployments in this workspace"
  ) {
    id
    config
  }
}
```

The `deploymentsConfigOverride` argument accepts a partial JSON object. Keys you provide are merged into the existing workspace override. Keys you omit are left unchanged. To remove a specific key from the stored override, set its value to the string `"DELETE_KEY"`:

```graphql wrap theme={null}
mutation {
  updateWorkspaceDeploymentsConfig(
    workspaceUuid: "<workspace-id>"
    deploymentsConfigOverride: {
      deploymentLifecycle: {
        deployRollback: {
          enabled: "DELETE_KEY"
        }
      }
    }
    reason: "Remove deploy rollback override, revert to cluster/platform default"
  ) {
    id
    config
  }
}
```

For the full config governance model these overrides participate in, see [Config governance](/docs/astro-private-cloud/v-2-x/config-governance).


# Kubernetes version support policy
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/kubernetes-version-support

A reference of all Kubernetes versions supported by Astro Private Cloud.

In general, Astro Private Cloud (APC) supports a given version of Kubernetes through its End of Life. This includes Kubernetes upstream and cloud-managed variants. When a version of Kubernetes reaches End of Life, support is removed in the next major or minor release of APC. For more information on Kubernetes versioning and release policies, refer to [Kubernetes Release History](https://kubernetes.io/releases/) or your cloud provider.

## Supported Kubernetes platforms

APC is tested and supported on the following Kubernetes platforms:

* Google Kubernetes Engine (GKE)
* Amazon Elastic Kubernetes Service (EKS)
* Azure Kubernetes Service (AKS)
* Red Hat OpenShift

## Supported Kubernetes versions

See the following table for all supported Kubernetes versions in each maintained version of APC.

| Astro Private Cloud | Kubernetes  1.31 | 1.32 | 1.33 | 1.34 | 1.35 |
| :-----------------: | :--------------: | :--: | :--: | :--: | :--: |
|        2.0.0        |        ✔️        |  ✔️  |  ✔️  |  ✔️  |  ✔️  |
|        2.1.0        |        ✔️        |  ✔️  |  ✔️  |  ✔️  |  ✔️  |

## General recommendations for Kubernetes upgrades

If there are no workloads running on the nodes you want to upgrade, there won't be an immediate impact on the Astro Private Cloud and Airflow components during the initial phase of upgrading your Kubernetes node pools. To minimize disruptions, however, perform a controlled rollout restart of the worker nodes. During your controlled rollout, monitor the health of the new nodes and workloads before decommissioning the old nodes.

Before beginning the upgrade process, ensure you have all necessary backups ready. After upgrading, verify that all Astro Private Cloud and Airflow components are running as expected on the new nodes. Once your Kubernetes cluster has been upgraded to a version that is compatible with your Astro Private Cloud version, you don't need to change your Astro Private Cloud configurations or settings. However, the upgrade requires restarting the kubelet on each node, which causes the Astro and Airflow components to also restart.

For more information on upgrading Kubernetes versions, follow the guidelines offered by your cloud provider.

* [Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html)
* [Azure AKS](https://docs.microsoft.com/en-us/azure/aks/upgrade-cluster)
* [Google GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-upgrades)
* [RedHat OpenShift](https://access.redhat.com/documentation/en-us/openshift_container_platform/4.11/html/updating_clusters/index)


# Logs configuration
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/logs-configuration

Configure centralized logging with Vector and Elasticsearch in Astro Private Cloud.

Astro Private Cloud (APC) provides centralized logging through Vector and Elasticsearch. Task logs, platform logs, and audit logs are collected by Vector and indexed in Elasticsearch for searchability and troubleshooting.

<Note>
  This page covers the Vector DaemonSet pipeline that ships task and platform logs. The APC API control plane audit events have a separate Vector sidecar with its own configuration and supported sinks. See [the APC API audit logging overview](/docs/astro-private-cloud/v-2-x/audit-logging-overview).
</Note>

## Architecture

```mermaid actions={true} theme={null}
flowchart LR
    A[Airflow Components] --> V[Vector] --> E[Elasticsearch]
    E --> BYO([Bring your own visualization tool])
```

Vector runs as a DaemonSet collecting logs from all Pods. Logs are shipped to Elasticsearch for storage and indexing. For log visualization, you can connect your own tools (Kibana, Grafana, OpenSearch Dashboards, etc.) to query Elasticsearch.

## Access logs

### Airflow UI

Task logs are accessible directly in the Airflow webserver UI:

<Steps>
  <Step title="Go to the Dag run" />

  <Step title="Click a task instance" />

  <Step title="Click **Log**" />
</Steps>

### Elasticsearch API

Query logs directly via Elasticsearch:

```bash wrap theme={null}
# Search for errors in the last hour
curl -X GET "https://elasticsearch.<base-domain>/_search" \
  -H "Content-Type: application/json" \
  -d '{
    "query": {
      "bool": {
        "must": [
          { "match": { "log_level": "ERROR" } },
          { "range": { "@timestamp": { "gte": "now-1h" } } }
        ]
      }
    }
  }'
```

### BYO visualization

APC doesn't include a log visualization UI. Connect your preferred tool to Elasticsearch:

* Kibana: Deploy separately and point to the Elasticsearch endpoint.
* Grafana: Use the Elasticsearch data source.
* OpenSearch Dashboards: Use Elasticsearch API compatibility.

## Vector configuration

Vector is the log collection agent in APC.

### Enable Vector

```yaml wrap theme={null}
tags:
  logging: true

global:
  daemonsetLogging:
    enabled: true
vector:
  resources:
    requests:
      cpu: "250m"
      memory: "512Mi"
    limits:
      cpu: "1000m"
      memory: "1Gi"
```

### Custom log parsing

Add custom transforms to parse Airflow log formats:

```yaml wrap theme={null}
vector:
  customConfig: |
    [transforms.parse_airflow]
    type = "remap"
    inputs = ["kubernetes_logs"]
    source = '''
    .parsed = parse_regex!(.message, r'^\[(?P<timestamp>.+)\] \{(?P<logger>.+)\} (?P<level>\w+) - (?P<message>.+)$')
    '''
```

### Logging sidecar

APC supports either DaemonSet or sidecar logging on a data plane cluster, but not both simultaneously. To use sidecar logging, you must first disable the Vector DaemonSet, then enable the sidecar:

```yaml wrap theme={null}
global:
  daemonsetLogging:
    enabled: false
  logging:
    loggingSidecar:
      enabled: true
      name: sidecar-log-consumer
      repository: quay.io/astronomer/ap-vector
      tag: 0.52.0
      resources:
        requests:
          cpu: "100m"
          memory: "386Mi"
```

## Elasticsearch configuration

### Enable Elasticsearch

```yaml expandable wrap theme={null}
tags:
  logging: true

elasticsearch:
  common:
    persistence:
      enabled: true

  client:
    replicas: 2
    heapMemory: "2g"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "4Gi"

  data:
    replicas: 3
    heapMemory: "2g"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
      limits:
        cpu: "2"
        memory: "4Gi"
    persistence:
      size: "100Gi"

  master:
    replicas: 3
    heapMemory: "2g"
    resources:
      requests:
        cpu: "1"
        memory: "2Gi"
```

### Index lifecycle management

Configure log retention:

```yaml wrap theme={null}
elasticsearch:
  indexLifecycleManagement:
    enabled: true
    policies:
      - name: airflow-logs
        phases:
          hot:
            actions:
              rollover:
                max_size: 50gb
                max_age: 7d
          delete:
            min_age: 30d
            actions:
              delete: {}
```

## External logging

### Forward to external Elasticsearch

Send logs to your own Elasticsearch cluster:

```yaml wrap theme={null}
global:
  customLogging:
    enabled: true
    scheme: https
    host: "elasticsearch.example.com"
    port: "9200"
    secret: "es-credentials"
```

### Forward to S3

Archive logs to object storage:

```yaml wrap theme={null}
vector:
  sinks:
    s3:
      type: "aws_s3"
      inputs: ["kubernetes_logs"]
      bucket: "my-logs-bucket"
      region: "us-west-2"
      compression: "gzip"
      encoding:
        codec: "json"
```

### Forward to external systems

Configure Vector to send to any destination:

```yaml wrap theme={null}
vector:
  sinks:
    # Splunk
    splunk:
      type: "splunk_hec"
      inputs: ["kubernetes_logs"]
      endpoint: "https://splunk.example.com:8088"
      token: "${SPLUNK_TOKEN}"

    # Datadog
    datadog:
      type: "datadog_logs"
      inputs: ["kubernetes_logs"]
      default_api_key: "${DATADOG_API_KEY}"

    # Generic HTTP
    http:
      type: "http"
      inputs: ["kubernetes_logs"]
      uri: "https://logs.example.com/v1/logs"
      encoding:
        codec: "json"
```

## Deployment log settings

### Task log retention

Configure log groomer to manage disk usage:

```yaml wrap theme={null}
# In deployment values
scheduler:
  logGroomerSidecar:
    enabled: true
    retentionDays: 15
    frequencyMinutes: 15

workers:
  logGroomerSidecar:
    enabled: true
    retentionDays: 15
```

### Log level configuration

```yaml wrap theme={null}
env:
  - name: AIRFLOW__LOGGING__LOGGING_LEVEL
    value: "INFO"
  - name: AIRFLOW__LOGGING__FAB_LOGGING_LEVEL
    value: "WARNING"
```

## Query logs

### Elasticsearch query examples

#### Find task failures

```json wrap theme={null}
{
  "query": {
    "bool": {
      "must": [
        { "match": { "log_level": "ERROR" } },
        { "match": { "kubernetes.labels.component": "worker" } }
      ]
    }
  }
}
```

#### Search specific Dag

```json wrap theme={null}
{
  "query": {
    "bool": {
      "must": [
        { "match": { "dag_id": "my_dag" } },
        { "match": { "task_id": "my_task" } }
      ]
    }
  }
}
```

#### Filter by time range

```json wrap theme={null}
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2026-02-01T00:00:00",
        "lt": "2026-02-02T00:00:00"
      }
    }
  }
}
```

### Common log fields

| Field                         | Description                         |
| ----------------------------- | ----------------------------------- |
| `kubernetes.namespace_name`   | Deployment namespace                |
| `kubernetes.labels.component` | Component (scheduler, worker, etc.) |
| `kubernetes.pod_name`         | Pod name                            |
| `dag_id`                      | Dag identifier                      |
| `task_id`                     | Task identifier                     |
| `log_level`                   | DEBUG, INFO, WARNING, ERROR         |
| `@timestamp`                  | Log timestamp                       |

## Troubleshooting

### Logs aren't appearing

1. Check Vector is running:

   ```bash wrap theme={null}
   kubectl get pods -n astronomer -l app=vector
   ```

2. Check Elasticsearch health:

   ```bash wrap theme={null}
   kubectl exec -n astronomer elasticsearch-0 -- \
     curl -s localhost:9200/_cluster/health
   ```

3. Verify Vector logs:

   ```bash wrap theme={null}
   kubectl logs -n astronomer -l app=vector --tail=100
   ```

### High disk usage

1. Enable index lifecycle management.
2. Reduce retention period.
3. Increase Elasticsearch storage.
4. Forward logs to external storage (S3).

### Slow queries

1. Add index patterns for common searches.
2. Increase Elasticsearch resources.
3. Reduce log verbosity.

## Security

### Access control

Elasticsearch access is restricted to platform components. For external access, configure authentication:

```yaml wrap theme={null}
elasticsearch:
  auth:
    enabled: true
    secretName: "es-credentials"
```

### Log redaction

Redact sensitive data before indexing:

```yaml wrap theme={null}
vector:
  transforms:
    redact:
      type: "remap"
      source: '''
      .message = replace(.message, r'password=\S+', "password=***")
      .message = replace(.message, r'api_key=\S+', "api_key=***")
      '''
```

## Best practices

* Set appropriate retention based on compliance requirements.
* Use log levels wisely — avoid DEBUG in production.
* Enable log groomer to prevent disk exhaustion on Airflow Pods.
* Forward logs externally for long-term retention and compliance.
* Monitor Elasticsearch health and disk usage.
* Use your preferred visualization tool — deploy Kibana, Grafana, or other tools separately.


# Forward logs to Amazon S3
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/logs-to-s3

Configure APC to forward Airflow logs to Amazon S3 using Vector.

Astro Private Cloud (APC) uses [Vector](https://vector.dev/) for log collection and forwarding. You can configure Vector to send Airflow task logs to Amazon S3 for long-term storage, compliance, or integration with other analytics tools.

<Note>
  If you previously configured S3 log forwarding using Fluentd in APC 0.37 or earlier, you must replace your `fluentd.s3` configuration with the Vector `extraSinks` configuration described in this document. Fluentd is no longer used for log collection in APC 2.0.
</Note>

## Architecture

```mermaid actions={true} theme={null}
flowchart LR
    A[Airflow Pods] --> V[Vector DaemonSet]
    V --> E[Elasticsearch\ndefault]
    V --> S[S3 Bucket\nadditional sink]
```

Vector continues forwarding logs to Elasticsearch for the Airflow UI while also sending copies to S3.

<Note>
  The logs forwarded to S3 are Airflow task logs and deployment logs, not APC platform logs from the APC API, the deployment orchestrator, or Registry.
</Note>

## Prerequisites

* An existing S3 bucket
* AWS IAM credentials with S3 write access
* APC 2.0 or later

## Configure log forwarding to S3

<Steps>
  <Step title="Configure AWS IAM">
    #### Create IAM policy

    Create an IAM policy with S3 write permissions:

    ```json wrap theme={null}
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "s3:ListBucket"
          ],
          "Resource": "arn:aws:s3:::<YOUR_LOGS_BUCKET>"
        },
        {
          "Effect": "Allow",
          "Action": [
            "s3:PutObject",
            "s3:GetObject"
          ],
          "Resource": "arn:aws:s3:::<YOUR_LOGS_BUCKET>/*"
        }
      ]
    }
    ```

    For more information on S3 permissions, see [Amazon S3 actions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-with-s3-actions.html).

    #### Provide credentials to Vector

    <Tabs>
      <Tab title="IRSA (Recommended)">
        For EKS clusters, use IAM Roles for Service Accounts (IRSA) to securely provide AWS credentials:

        1. Create an IAM role with the S3 policy attached
        2. Configure the trust relationship for the Vector service account:

        ```json wrap theme={null}
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                "StringEquals": {
                  "oidc.eks.<AWS_REGION>.amazonaws.com/id/<OIDC_ID>:sub": "system:serviceaccount:astronomer:astronomer-vector"
                }
              }
            }
          ]
        }
        ```

        3. Annotate the Vector service account in your `values.yaml`:

        ```yaml wrap theme={null}
        vector:
          serviceAccount:
            annotations:
              eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<VECTOR_S3_ROLE>
        ```
      </Tab>

      <Tab title="EC2 instance profile">
        For self-managed Kubernetes on EC2, attach the IAM policy to the EC2 instance profile used by your worker nodes.
      </Tab>

      <Tab title="Static credentials">
        For non-AWS environments or testing, use static credentials:

        ```yaml wrap theme={null}
        vector:
          extraEnv:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: aws-credentials
                  key: access-key-id
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: aws-credentials
                  key: secret-access-key
            - name: AWS_REGION
              value: "us-east-1"
        ```

        Create the secret:

        ```bash wrap theme={null}
        kubectl create secret generic aws-credentials \
          --namespace astronomer \
          --from-literal=access-key-id=AKIAIOSFODNN7EXAMPLE \
          --from-literal=secret-access-key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
        ```

        <Warning>
          Static credentials are less secure than IRSA or instance profiles. Use only for testing or non-AWS environments.
        </Warning>
      </Tab>
    </Tabs>
  </Step>

  <Step title="Configure Vector S3 sink">
    Add the S3 sink to your `values.yaml`:

    ```yaml wrap theme={null}
    vector:
      extraSinks:
        s3_logs:
          type: aws_s3
          inputs:
            - transform_remove_fields
          bucket: "<YOUR_LOGS_BUCKET>"
          region: "<AWS_REGION>"
          key_prefix: "airflow-logs/{{ "{{ namespace }}" }}/{{ "{{ release }}" }}/%Y/%m/%d/"
          compression: gzip
          encoding:
            codec: json
          batch:
            max_bytes: 10485760
            timeout_secs: 300
          request:
            retry_attempts: 5
    ```

    #### Configuration options

    For a full list of available options, see the [Vector `aws_s3` sink configuration reference](https://vector.dev/docs/reference/configuration/sinks/aws_s3).

    | Option               | Description                          | Example                  |
    | -------------------- | ------------------------------------ | ------------------------ |
    | `bucket`             | S3 bucket name                       | `<YOUR_LOGS_BUCKET>`     |
    | `region`             | AWS region                           | `us-east-1`              |
    | `key_prefix`         | S3 object key prefix with templating | `logs/%Y/%m/%d/`         |
    | `compression`        | Compression algorithm                | `gzip`, `zstd`, `none`   |
    | `encoding.codec`     | Output format                        | `json`, `text`, `ndjson` |
    | `batch.max_bytes`    | Max batch size before flush          | `10485760` (10 MB)       |
    | `batch.timeout_secs` | Max time before flush                | `300` (5 minutes)        |

    #### Key prefix templating

    Use template variables in `key_prefix`:

    | Variable                  | Description             |
    | ------------------------- | ----------------------- |
    | `{{ "{{ namespace }}" }}` | Kubernetes namespace    |
    | `{{ "{{ release }}" }}`   | Deployment release name |
    | `%Y`, `%m`, `%d`          | Date components         |
    | `%H`, `%M`, `%S`          | Time components         |

    Example: `airflow-logs/{{ "{{ namespace }}" }}/%Y/%m/%d/%H/`
  </Step>

  <Step title="Apply configuration">
    Push the configuration to your APC installation. For detailed instructions, see [Apply a config change](/docs/astro-private-cloud/v-2-x/apply-platform-config).

    ```bash wrap theme={null}
    helm upgrade astronomer astronomer/astronomer \
      -f values.yaml \
      --namespace astronomer
    ```

    Verify Vector pods restart with the new configuration:

    ```bash wrap theme={null}
    kubectl rollout status daemonset/astronomer-vector -n astronomer
    ```
  </Step>

  <Step title="Verify log delivery">
    #### Check Vector logs

    ```bash wrap theme={null}
    kubectl logs -n astronomer -l app=vector --tail=100 | grep -i s3
    ```

    #### List S3 objects

    ```bash wrap theme={null}
    aws s3 ls s3://<YOUR_LOGS_BUCKET>/airflow-logs/ --recursive | head -20
    ```

    #### Read a log file

    ```bash wrap theme={null}
    aws s3 cp s3://<YOUR_LOGS_BUCKET>/airflow-logs/path/to/file.json.gz - | gunzip | head -5
    ```
  </Step>
</Steps>

## Advanced configuration

### Filter logs by severity

Only forward ERROR and WARNING logs to S3 using a [Vector Remap Language (VRL)](https://vector.dev/docs/reference/vrl) filter condition:

```yaml wrap theme={null}
vector:
  extraTransforms:
    filter_errors:
      type: filter
      inputs:
        - transform_remove_fields
      condition:
        type: vrl
        source: '.level == "ERROR" || .level == "WARNING"'

  extraSinks:
    s3_errors:
      type: aws_s3
      inputs:
        - filter_errors
      bucket: "<YOUR_LOGS_BUCKET>"
      # ... rest of config
```

### Partition by deployment

Organize logs by deployment namespace:

```yaml wrap theme={null}
vector:
  extraSinks:
    s3_logs:
      type: aws_s3
      inputs:
        - transform_remove_fields
      bucket: "<YOUR_LOGS_BUCKET>"
      key_prefix: "deployments/{{ "{{ namespace }}" }}/{{ "{{ pod }}" }}/%Y/%m/%d/"
      # ... rest of config
```

### Multiple destinations

Forward to both S3 and another system:

```yaml wrap theme={null}
vector:
  extraSinks:
    s3_archive:
      type: aws_s3
      inputs:
        - transform_remove_fields
      bucket: "<ARCHIVE_BUCKET>"
      # ... config

    splunk_realtime:
      type: splunk_hec
      inputs:
        - transform_remove_fields
      endpoint: "<SPLUNK_ENDPOINT>"
      token: "${SPLUNK_TOKEN}"
```

## S3 lifecycle policies

Configure S3 lifecycle rules to manage log retention:

```json wrap theme={null}
{
  "Rules": [
    {
      "ID": "ArchiveOldLogs",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "airflow-logs/"
      },
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ],
      "Expiration": {
        "Days": 365
      }
    }
  ]
}
```

Apply via AWS CLI:

```bash wrap theme={null}
aws s3api put-bucket-lifecycle-configuration \
  --bucket <YOUR_LOGS_BUCKET> \
  --lifecycle-configuration file://lifecycle.json
```

## Troubleshooting

### Logs not appearing in S3

1. Check Vector pod logs:

   ```bash wrap theme={null}
   kubectl logs -n astronomer -l app=vector | grep -i error
   ```

2. Verify AWS credentials:

   ```bash wrap theme={null}
   kubectl exec -n astronomer -it ds/astronomer-vector -c vector -- \
     sh -c 'echo $AWS_ACCESS_KEY_ID'
   ```

3. Inspect the logs for credential errors or permission issues.

   Look for lines containing `CredentialsNotLoaded` (no credentials found) or `Invalid credentials` (credentials rejected by AWS). For example:

   ```text wrap theme={null}
   2026-04-16T18:27:48.827213Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Invalid credentials component_kind="sink" component_type="aws_s3" component_id=s3_logs
   ```

   To see which credentials Vector loaded, look for lines matching `aws_config::profile::credentials`:

   ```text wrap theme={null}
   2026-04-16T18:27:48.247566Z  INFO aws_config::profile::credentials: constructed abstract provider from config file chain=ProfileChain { base: AccessKey(Credentials { provider_name: "ProfileFile", access_key_id: "AKIA5WLLPVSPD7JDVSXF", secret_access_key: "** redacted **", expires_after: "never" }), chain: [] }
   ```

   These lines show the access key ID in use, which can help confirm whether the correct credentials are being picked up.

### Permission denied errors

Verify your IAM policy includes both `s3:PutObject` and `s3:ListBucket` permissions. The bucket resource ARN shouldn't include `/*` for ListBucket.

### High latency

Adjust batch settings for faster delivery:

```yaml wrap theme={null}
vector:
  extraSinks:
    s3_logs:
      batch:
        max_bytes: 5242880    # 5MB
        timeout_secs: 60      # 1 minute
```

## Related documentation

* [Logs configuration](/docs/astro-private-cloud/v-2-x/logs-configuration)
* [Vector Documentation](https://vector.dev/docs/)
* [AWS S3 Sink Reference](https://vector.dev/docs/reference/configuration/sinks/aws_s3/)


# Platform and deployment alerts
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/platform-alerts

Configure alerting for APC platform health and Airflow Deployments.

APC includes two built-in alerting systems for monitoring health:

* **Deployment-level alerts**: Notify you when an Airflow Deployment is unhealthy or components are underperforming.
* **Platform-level alerts**: Notify you when APC platform components are unhealthy (Elasticsearch, APC API, Registry, the deployment orchestrator).

Alerts fire based on metrics collected by Prometheus. When alert conditions are met, [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) sends notifications to your configured channels.

The APC monitoring stack enables Alertmanager by default (`tags.monitoring: true`). To disable it individually, set `global.alertmanager.enabled: false` in your `values.yaml`. See [Apply platform configuration](/docs/astro-private-cloud/v-2-x/apply-platform-config) for details.

## Alert architecture

```mermaid actions={true} theme={null}
flowchart LR
    A["Prometheus (metrics)"]
    B["Alert Rules (PromQL)"]
    C["Alertmanager"]
    D["Notification Channels"]

    A --> B --> C --> D
```

## Anatomy of an alert

Alerts are defined in YAML using [PromQL queries](https://prometheus.io/docs/prometheus/latest/querying/basics/):

```yaml wrap theme={null}
- alert: ManyUnhealthySchedulers
  expr: count(rate(airflow_scheduler_heartbeat{}[1m]) <= 0) > 5
  for: 5m
  labels:
    tier: platform
    severity: critical
  annotations:
    summary: "{{ $value }} airflow schedulers are not heartbeating"
    description: "More than 5 Airflow schedulers have not emitted a heartbeat for over 5 minutes."
```

| Field                     | Description                                                   |
| ------------------------- | ------------------------------------------------------------- |
| `expr`                    | PromQL expression that determines when to fire                |
| `for`                     | Duration the condition must be true (for example, `5m`, `1h`) |
| `labels.tier`             | Alert level: `airflow` (Deployment) or `platform`             |
| `labels.severity`         | Severity: `info`, `warning`, `high`, `critical`               |
| `annotations.summary`     | Alert message text                                            |
| `annotations.description` | Human-readable description                                    |

## Subscribe to alerts

### Configure alert receivers

Alertmanager uses [receivers](https://prometheus.io/docs/alerting/latest/configuration/#receiver) to integrate with notification platforms. Define receivers in your `values.yaml`:

#### Email alerts

```yaml wrap theme={null}
alertmanager:
  receivers:
    platform:
      email_configs:
        - smarthost: smtp.example.com:587
          from: alerts@example.com
          to: ops-team@example.com
          auth_username: alerts@example.com
          auth_password: ${SMTP_PASSWORD}
          send_resolved: true
```

#### Slack alerts

```yaml wrap theme={null}
alertmanager:
  receivers:
    platformCritical:
      slack_configs:
        - api_url: https://hooks.slack.com/services/xxx/yyy/zzz
          channel: '#platform-alerts'
          title: '{{ .CommonAnnotations.summary }}'
          text: |-
            {{ range .Alerts }}{{ .Annotations.description }}
            {{ end }}
```

#### PagerDuty alerts

```yaml wrap theme={null}
alertmanager:
  receivers:
    platformCritical:
      pagerduty_configs:
        - service_key: ${PAGERDUTY_SERVICE_KEY}
          severity: '{{ .CommonLabels.severity }}'
          description: '{{ .CommonAnnotations.summary }}'
```

#### OpsGenie alerts

```yaml wrap theme={null}
alertmanager:
  receivers:
    platformCritical:
      opsgenie_configs:
        - api_key: ${OPSGENIE_API_KEY}
          message: '{{ .CommonAnnotations.summary }}'
          priority: '{{ if eq .CommonLabels.severity "critical" }}P1{{ else }}P3{{ end }}'
```

### Default receiver groups

APC includes default receiver groups based on tier and severity:

| Receiver           | Tier     | Severity |
| ------------------ | -------- | -------- |
| `platform`         | platform | all      |
| `platformCritical` | platform | critical |
| `airflow`          | airflow  | all      |

### Custom routes

If you define a `platform`, `platformCritical`, or `airflow` receiver, you don't need a `customRoute` to route to it — alerts are automatically routed based on the `tier` label. Use `customRoutes` only for non-default routing (for example, high-severity Deployment alerts):

```yaml wrap theme={null}
alertmanager:
  customRoutes:
    - receiver: deployment-high-receiver
      match_re:
        tier: airflow
        severity: high
    - receiver: deployment-warning-receiver
      match_re:
        tier: airflow
        severity: warning
```

### Custom receivers

Use `alertmanager.customReceiver` to define receivers for notification services not covered by the built-in receiver keys. Custom receivers work alongside `customRoutes` to route alerts to those services:

```yaml wrap theme={null}
alertmanager:
  customReceiver:
    - name: sns-receiver
      sns_configs:
        - api_url: <SNS_ENDPOINT>
          topic_arn: <SNS_TOPIC_ARN>
          subject: '[Alert: {{ .GroupLabels.alertname }}]'
          sigv4:
            region: <AWS_REGION>
            role_arn: <SNS_ROLE_ARN>
  customRoutes:
    - receiver: sns-receiver
      match_re:
        tier: platform
        severity: critical
```

### Apply configuration

Push receiver configuration to your installation:

```bash wrap theme={null}
helm upgrade astronomer astronomer/astronomer \
  -f values.yaml \
  --namespace astronomer
```

## Create custom alerts

Add custom alerts using the Prometheus Helm chart:

### Platform alert example

Alert when multiple schedulers are unhealthy:

```yaml wrap theme={null}
prometheus:
  additionalAlerts:
    platform: |
      - alert: MultipleSchedulersUnhealthy
        expr: count(rate(airflow_scheduler_heartbeat{}[1m]) <= 0) > 2
        for: 5m
        labels:
          tier: platform
          severity: critical
        annotations:
          summary: "{{ $value }} schedulers are not heartbeating"
          description: "More than 2 Airflow schedulers are unhealthy for over 5 minutes."
```

### Deployment alert example

Alert on high task failure rate:

```yaml wrap theme={null}
prometheus:
  additionalAlerts:
    airflow: |
      - alert: HighTaskFailureRate
        expr: |
          (
            sum(increase(airflow_ti_failures{}[1h])) by (deployment)
            /
            sum(increase(airflow_ti_successes{}[1h]) + increase(airflow_ti_failures{}[1h])) by (deployment)
          ) > 0.1
        for: 15m
        labels:
          tier: airflow
          severity: warning
        annotations:
          summary: "High task failure rate in {{ $labels.deployment }}"
          description: "Task failure rate exceeds 10% for the past 15 minutes."
```

## Built-in deployment alerts

For a complete list of built-in alerts, see the [Prometheus alerts configmap](https://github.com/astronomer/astronomer/blob/master/charts/prometheus/templates/prometheus-alerts-configmap.yaml).

| Alert                           | Description                                            | Action                                   |
| ------------------------------- | ------------------------------------------------------ | ---------------------------------------- |
| `AirflowDeploymentUnhealthy`    | Deployment is unhealthy or unavailable for 15+ minutes | Check pod status, review logs            |
| `AirflowPodQuota`               | Using more than 95% pod quota for 10+ minutes          | Increase Extra Capacity or optimize Dags |
| `AirflowSchedulerUnhealthy`     | Scheduler not heartbeating for 6+ minutes              | Check scheduler logs, restart if needed  |
| `AirflowTasksPendingIncreasing` | Tasks pending faster than clearing for 30+ minutes     | Increase concurrency or worker resources |

## Built-in platform alerts

| Alert                              | Description                                                                                                                                      | Action                                                           |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------- |
| `CriticalComponentPodCrashLooping` | A core platform component pod (the APC API, the deployment orchestrator, Grafana, Prometheus, Registry) is repeatedly restarting for 15+ minutes | Check pod logs in the APC namespace, investigate the crash cause |
| `CriticalComponentPodNotReady`     | A pod in the APC platform namespace has been in a non-ready state for 15+ minutes                                                                | Check pod events and logs in the APC namespace                   |
| `TargetDown`                       | More than 10% of Prometheus scrape targets for a job are unreachable for 10+ minutes                                                             | Check the failing service's pods and endpoints                   |
| `ElasticSeachUnassignedShards`     | Elasticsearch cluster has unassigned shards for 10+ minutes                                                                                      | Check Elasticsearch cluster health and logs                      |
| `ElasticDiskHighWatermarkReached`  | Elasticsearch node disk usage exceeds 90% for 5+ minutes                                                                                         | Increase Elasticsearch storage or clean up old indices           |
| `ElasticDiskFloodWatermarkReached` | Elasticsearch node disk usage exceeds 95% for 5+ minutes — Elasticsearch enforces a read-only index block at this threshold                      | Immediately increase storage or delete old indices               |
| `IngessCertificateExpiration`      | A TLS certificate for a platform hostname expires in less than one week                                                                          | Renew the TLS certificate                                        |

<Note>
  The `ElasticSeachUnassignedShards` and `IngessCertificateExpiration` alert names contain typos in their current implementation. Use the exact names shown when creating silences or custom routes.
</Note>

## View active alerts

### Alertmanager UI

Access Alertmanager to view active alerts:

```text wrap theme={null}
https://alertmanager.<base-domain>
```

### Prometheus UI

Query alerts in Prometheus:

```text wrap theme={null}
https://prometheus.<base-domain>/alerts
```

### CLI

```bash wrap theme={null}
# View firing alerts
kubectl exec -n astronomer prometheus-0 -- \
  wget -qO- localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing")'
```

## Silence alerts

Temporarily silence alerts during maintenance:

### Via Alertmanager UI

1. Go to `https://alertmanager.<base-domain>`
2. Click **Silences** > **New Silence**
3. Add matchers (for example, `alertname=AirflowSchedulerUnhealthy`)
4. Set duration and comment
5. Click **Create**

### Via API

```bash wrap theme={null}
curl -X POST https://alertmanager.<base-domain>/api/v2/silences \
  -H "Content-Type: application/json" \
  -d '{
    "matchers": [{"name": "alertname", "value": "AirflowSchedulerUnhealthy", "isRegex": false}],
    "startsAt": "2026-02-05T00:00:00Z",
    "endsAt": "2026-02-05T06:00:00Z",
    "createdBy": "admin",
    "comment": "Maintenance window"
  }'
```

## Best practices

1. **Start with built-in alerts** before creating custom ones
2. **Set appropriate thresholds** - avoid alert fatigue
3. **Use severity levels** - reserve `critical` for pages
4. **Include runbook links** in alert descriptions
5. **Test alerts** in non-production environments first
6. **Document escalation paths** for each severity level

## Related documentation

* [Apply platform configuration](/docs/astro-private-cloud/v-2-x/apply-platform-config)
* [Prometheus Alertmanager documentation](https://prometheus.io/docs/alerting/latest/configuration/)


# Astro Private Cloud release and lifecycle policy
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/release-lifecycle-policy

Astronomer's release and lifecycle policy for Astro Private Cloud.

Astronomer supports a variety of policies that drives the naming, release cadence, and maintenance commitments associated with all published software.

This document offers guidelines on the version lifecycle of Astro Private Cloud (APC). It includes a description of:

* How APC is versioned.
* Which versions of APC are currently available.
* Support periods and the maintenance schedule for all versions.

For information on the latest APC releases, see [Release notes](/docs/astro-private-cloud/v-2-x/release-notes). For information on compatibility between all versioned software, see [APC Version compatibility reference](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).

<Info>These policies apply only to the APC platform. For release and lifecycle policies related to Astro Runtime, see [Runtime release and lifecycle Policy](/docs/runtime/runtime-version-lifecycle-policy).</Info>

## Support periods

Every APC 1.0 and later release has two consecutive support periods:

* **Standard support**: Starts on the release date and runs for two years.
* **Long-term support (LTS)**: Extends support for the same version to three years from the release date.

Standard support and long-term support are two periods in the lifecycle of one release, not two versions of APC that you choose between. Astronomer publishes one build for each APC version. For the dates that apply to each version, see [Astro Private Cloud lifecycle schedule](#astro-private-cloud-lifecycle-schedule).

Long-term support includes bug fixes, security patches, and maintenance for the features in that version. Astronomer doesn't backport new features to an earlier version, so a version stays feature-complete after its release.

To get the newest APC and Airflow features, upgrade to each new version as soon as it becomes available.

<Note>Support periods apply to APC image versions. Long-term support doesn't apply to the Astro CLI, Airflow, or Astro Runtime versions.</Note>

Astronomer Software 0.x used two release channels, Stable and LTS. That model applies only to the versions in [Astronomer Software 0.3x lifecycle schedule](#astronomer-software-03x-lifecycle-schedule).

## Astro Private Cloud versioning

Astro Private Cloud follows [Semantic Versioning](https://semver.org/) for all published APC. This means that Astronomer use Major, Minor, and Patch releases across the product in the format of `major.minor.patch`.

* **Major** versions: Significant feature additions, including backward-incompatible changes to an API or Dag specification.
* **Minor** versions: Functional changes, including backward-compatible changes to an API or Dag specification.
* **Patch** versions: Bug and security fixes that resolve incorrect behavior.

It's safe to upgrade to minor and patch versions within a major version. Astronomer provides upgrade guidance with each release.

## Version release cadence

Astro Private Cloud major, minor, and patch versions are released approximately on the following cadence:

| Release Type | Approximate Frequency of Releases |
| ------------ | --------------------------------- |
| Major        | Half-yearly                       |
| Minor        | Quarterly                         |
| Patch        | Monthly                           |

For each `major.minor` pair, only the latest patch is supported at any given time.

## Backport policy for bug and security fixes

When a new minor version is released within a major release series (for example, 1.1 within the 1.x series), the previous minor version (for example, 1.0) no longer receives new feature or bug fix patches. Astronomer recommends upgrading to the latest minor version to receive continued bug fixes and new features.

Astronomer backports a fix for a major stability bug to the latest minor version of each supported major version. If you run an earlier minor version, upgrade to get the fix. Major issues in this category can cause significant delays in task scheduling and potential data loss.

If a major security issue is identified, Astronomer will release fixes as new patch versions for *all* supported `major.minor` versions within their support window, whether the version is in its standard or long-term support period, in accordance with Astronomer's CVE policy. Major issues in this category are classified by a combination of impact and exploitability.

## Astro Private Cloud lifecycle schedule

The following tables contain the exact lifecycle for each published version of APC. These timelines follow the standard and long-term support periods.

| APC Version | Release Date     | End of Standard Support | End of Long-Term Support |
| ----------- | ---------------- | ----------------------- | ------------------------ |
| 1.0         | October 14, 2025 | October 14, 2027        | October 14, 2028         |
| 1.1         | February 2, 2026 | February 2, 2028        | February 2, 2029         |
| 1.2         | July 15, 2026    | July 15, 2028           | July 15, 2029            |
| 2.0         | May 1, 2026      | May 1, 2028             | May 1, 2029              |
| 2.1         | August 10, 2026  | August 10, 2028         | August 10, 2029          |

## Astronomer Software 0.3x lifecycle schedule

The following tables contain the lifecycle for each published version of Astronomer Software 0.3x. For the full Astronomer Software 0.3x release and lifecycle policy, see the [Software 0.37 documentation](/docs/astro-private-cloud/v-0-37/release-lifecycle-policy).

### Supported

| Software Version | Release Date      | End of Maintenance Date |
| ---------------- | ----------------- | ----------------------- |
| 0.36 (LTS)       | November 13, 2024 | April 2026              |
| 0.37 (LTS)       | February 28, 2025 | August 2026             |


# Astro Private Cloud release notes
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/release-notes

Astro Private Cloud release notes.

This document contains release notes for each version of Astro Private Cloud (APC).

Version 2.0 is the latest long-term support (LTS) version of Astro Private Cloud. To upgrade to version 2.0, see [Upgrade Astro Private Cloud](/docs/astro-private-cloud/v-2-x/breaking-changes-removals). For more information about APC support periods, see [Release and lifecycle policies](/docs/astro-private-cloud/v-2-x/release-lifecycle-policy). To read release notes specifically for the Astro CLI, see [Astro CLI release notes](/docs/cli/v1.43/release-notes).

<Info>
  Because Astronomer has separate [maintenance life cycles](/docs/astro-private-cloud/v-2-x/release-lifecycle-policy) for each minor version of APC, the same change can be introduced multiple times across minor versions, resulting in multiple identical release notes. When a new minor version releases, such as version 1.1.0, all changes from previously released versions are included in the new minor version.

  If you're upgrading to receive a specific change, ensure the release note for the change appears either:

  * Within your target minor version.
  * In a patch version that was released before the first release of your target minor version.
</Info>

<Update label="2.1.0" description="August 10, 2026">
  Astro Private Cloud 2.1.0 adds control plane reliability, custom roles and permission audit, Airflow Operator mode, and per-deployment migration, and significantly expands data plane failover.

  Before you upgrade, review [Breaking changes](#breaking-changes) for the changes to default behavior in this release.

  ### New features

  * **Control plane reliability**: You can now run two or more control planes that share one database and serve one customer-facing domain. Within a region they serve traffic together, and an unhealthy control plane drops out of rotation automatically. Across regions, you can stand up control planes in a standby region and fail the platform over to it during an outage or a planned migration. Two new admin pages, **Regions** and **Control Planes**, manage this system. See [Control plane reliability](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery), [Configure control plane reliability](/docs/astro-private-cloud/v-2-x/configure-control-plane-disaster-recovery), [Manage a control plane reliability group](/docs/astro-private-cloud/v-2-x/manage-control-plane-disaster-recovery), and the [Control plane reliability reference](/docs/astro-private-cloud/v-2-x/control-plane-disaster-recovery-reference).
  * **Custom roles and permission audit**: You can now define custom roles from a catalog of granular permissions that spans both platform and Airflow permissions, scope them to the system, a Workspace, or a Deployment, and assign them to users, teams, and service accounts. Custom roles are additive and don't change the built-in Viewer, Editor, and Admin roles. They're disabled by default until a System Admin enables them for the platform. A new **Permission Audit** page shows who has access to a given system, Workspace, or Deployment, and what any single user, team, or service account can do. See [Custom roles](/docs/astro-private-cloud/v-2-x/custom-roles), [Permission audit](/docs/astro-private-cloud/v-2-x/permission-audit), and the [Role and permission reference](/docs/astro-private-cloud/v-2-x/role-permission-reference).
  * **Airflow Operator mode and Deployment adoption (Preview)**: You can now run a Deployment as an Airflow custom resource managed by the Airflow Kubernetes Operator instead of as a Helm release. If you already run Airflow with the operator, you can register that cluster as an APC data plane and adopt its existing Deployments without recreating them or interrupting Airflow, bringing their environment variables, users, registry, logs, and metrics under APC. Unadopt a Deployment to hand control back. Adopt and unadopt from the APC UI or with `astro deployment adopt` and `astro deployment unadopt`. Operator support is disabled by default and enabled per data plane. See [Airflow Operator mode](/docs/astro-private-cloud/v-2-x/airflow-operator-mode), [Adopt Astro Runtime Operator managed Deployments](/docs/astro-private-cloud/v-2-x/adopt-operator-deployments), and [Move the operator under APC](/docs/astro-private-cloud/v-2-x/transition-operator-to-apc).
  * **Per-deployment migration**: You can now migrate individual Deployments to another data plane cluster, rather than failing over an entire data plane. Use it to shift a busy or unhealthy Deployment onto a neighboring cluster, or to let one team test failover on their own Deployment. Requires data plane failover to be enabled on your clusters. See [Configure per-deployment migration](/docs/astro-private-cloud/v-2-x/configure-per-deployment-migration), [Manage and observe migration](/docs/astro-private-cloud/v-2-x/manage-per-deployment-migration), and the [Per-deployment migration reference](/docs/astro-private-cloud/v-2-x/per-deployment-migration-reference).
  * **LDAP authentication**: You can now authenticate users directly against an LDAP or LDAPS directory, such as Active Directory or OpenLDAP, with no OIDC bridge in between. You can also import LDAP groups as teams and map LDAP groups to system roles. See [Configure LDAP authentication](/docs/astro-private-cloud/v-2-x/configure-ldap-authentication).
  * **Failover upgrade for existing Deployments**: You can now retrofit Deployments created before data plane failover was enabled, so they get the inactive database connection failover requires. Upgrade Deployments one at a time or in bulk, and use the indicator on the Deployments list to see which Deployments are ready for failover and which still need the upgrade. See [Run a failover upgrade](/docs/astro-private-cloud/v-2-x/run-failover-upgrade).
  * **HashiCorp Vault for data plane failover**: You can now use Vault as the `ClusterSecretStore` backend that the External Secrets Operator uses to replicate Deployment secrets between failover clusters, using Vault's Kubernetes auth method. See [Configure Vault for data plane failover](/docs/astro-private-cloud/v-2-x/configure-vault-data-plane-failover).
  * **git-sync over HTTPS**: You can now authenticate git-sync to a private repository over HTTPS with a personal access token, in addition to SSH. Credentials are validated when you save the configuration rather than at first sync. `git-sync-relay` now also trusts the bundles named in `global.privateCaCerts`, so clones against a Git host behind a private certificate authority succeed, and a Deployment's status indicator reflects git-sync health, so a failing sync is visible without opening the **Logs** tab. See [Deploy Dags with git-sync](/docs/astro-private-cloud/v-2-x/deploy-git-sync).

  ### Breaking changes

  * Soft delete is removed. Deleting a Deployment now removes it and its Airflow database immediately, the **Hard delete** checkbox is gone from the APC UI, and the `hardDeleteDeployment` flag is deprecated in the Astro CLI. The `hardDeleteDeployment` configuration key no longer gates anything and is retained only so existing configuration overrides continue to validate. Confirm that your delete workflows and any automation that relied on recovering a soft-deleted Deployment account for the new behavior before you upgrade.
  * The default `USER` role no longer includes the `system.workspace.create` permission, so authenticated users can no longer create a Workspace — and therefore Deployments in it — without an explicit grant. Every authenticated user previously held this permission, which let them create a Workspace, inherit Workspace Admin on it, and create Deployments there. If your teams create their own Workspaces, grant `system.workspace.create` through a custom role or a system role assignment before you upgrade. See [Manage platform users](/docs/astro-private-cloud/v-2-x/manage-platform-users) and [User permissions](/docs/astro-private-cloud/v-2-x/manage-permissions).
  * Default container security contexts now comply with the Kubernetes Pod Security Standards Restricted profile (PSS Restricted enforcement itself isn't enabled by default). The platform chart and the APC API's injected container contexts now set `allowPrivilegeEscalation: false`, `capabilities.drop: [ALL]`, `runAsNonRoot`, and `seccompProfile.type: RuntimeDefault`, and force `readOnlyRootFilesystem` regardless of overrides. If you set a component's `securityContext`, provide the entire block — other than `readOnlyRootFilesystem`, values aren't deep-merged with the defaults. See [Configure security contexts](/docs/astro-private-cloud/v-2-x/configure-securitycontext).
  * `global.acme` is removed. Remove this deprecated flag from your `values.yaml` before you upgrade, because the strict schema validator rejects unknown keys.
  * Cluster Admin permissions are folded into System Admin. `SYSTEM_ADMIN` now inherits the `CLUSTER_ADMIN` permission set, and both roles gain the new cordon, region, control plane registry, adoption, and custom role permissions. Review any automation that asserts an exact permission set.
  * The Airflow Operator sub-chart condition changed. The sub-chart is now gated on `airflow-operator.enabled` first and `global.airflowOperator.enabled` second. If your cluster already runs a standalone Airflow Kubernetes Operator, set `airflow-operator.enabled: false` to avoid installing a second one, and keep `global.airflowOperator.enabled: true`.

  ### Security enhancements

  * Patched the Critical and High severity CVEs identified against Astro Private Cloud 2.0.1. See the [Resolved CVE list](#resolved-cve-list).
  * Environment variables marked `isSecret` are now masked in Pod stdout logs.
  * Platform Kubernetes service accounts now set `automountServiceAccountToken` explicitly, so a Pod that doesn't need the API server no longer receives a mounted token.
  * Removed the default Workspace-create grant from the `USER` role. See [Breaking Changes](#breaking-changes).

  ### Additional improvements

  * **Postgres 18 support**: APC is now compatible with Postgres 18 and later.
  * **Migrate from unified mode to split mode**: Installations running the control plane and data plane in one cluster can now migrate to split control plane and data plane mode without Airflow downtime or data loss.
  * **Cluster maintenance mode**: You can now put a whole cluster into maintenance mode while you work on it, which blocks Deployment changes on that cluster while leaving Airflow and read access untouched. In 2.1, the cluster-level control is available through the APC API only. You can cordon individual Deployments from the Deployment **Actions** menu in the APC UI. See [Cordon and uncordon a Deployment](/docs/astro-private-cloud/v-2-x/cordon-deployment).
  * **Kubernetes security policy compatibility**: The defaults that APC ships now conform to Pod Security Standards Restricted. See [Configure security contexts](/docs/astro-private-cloud/v-2-x/configure-securitycontext).
  * **Private CA trust for git-sync HTTPS**: A new cluster-scoped setting, `astronomer.houston.config.deployments.privateCaCertSecretNames`, names the Secrets holding the private certificate authority bundles that git-sync HTTPS clones should trust. See [Deploy Dags with git-sync](/docs/astro-private-cloud/v-2-x/deploy-git-sync).
  * **More component resources configurable per Deployment**: PgBouncer and StatsD CPU and memory can now be set from the Deployment settings page, and the API server now honors over-provisioning factors. On Deployments using the Celery executor, Redis can be set the same way. Component defaults, minimums, and limits are also listed in the APC API configuration, so you can see the current values before you override them. See [Configure component size limits](/docs/astro-private-cloud/v-2-x/configure-component-size-limits).
  * **ClusterIP ingress**: For clusters whose policy forbids `LoadBalancer` and `NodePort` Services, you can now set `nginx.serviceType: ClusterIP` and front the Service with your own load balancer. The default remains `LoadBalancer`. See [Network configuration](/docs/astro-private-cloud/v-2-x/network-configuration).
  * **Node exporter tolerations documented**: The toleration that lets `prometheus-node-exporter` run on every node is now listed in the Helm configuration reference, so you can find and override it if your cluster policy restricts tolerations. The toleration and its default are unchanged. See [Helm configuration reference](/docs/astro-private-cloud/v-2-x/helm-config-reference).
  * **Dag server backup annotations**: The Dag server now accepts `annotations` and `podAnnotations`, so its persistent volume can be included in Velero backup policies.
  * **APC UI performance and polish**: The initial page load is significantly faster — the Monaco editor is slimmed down, heavy routes are lazy-loaded, and responses are gzipped. Deployment actions are consolidated into a single **Actions** menu in the Deployment header. The configuration editors gain an **Expand All** control so search reaches collapsed YAML, a confirmation step before **Update Cluster**, and guidance in place of a blank Workspace override editor. The Airflow environment variable list in the Deployment UI now reflects Airflow 3. References to "Houston" are replaced with "APC API", and the **System Admin** tab is now **System**.
  * **Duplicate Workspace label pre-upgrade check**: The upgrade now checks for duplicate Workspace labels before it runs, instead of failing partway through the database migration.
  * **`SSHRemoteJobOperator`**: A new deferrable operator in the SSH provider runs a command on a remote host as a detached job and streams its log back into the Airflow task log. The job keeps running if the SSH connection drops or the worker restarts, and it doesn't hold a worker slot while it waits. Use it instead of `SSHOperator` for long-running remote jobs.

  ### Bug fixes

  * Fixed an issue where the Deployment detail page in the APC UI rendered blank, with no loading indicator, while the Deployment query was still pending.
  * Fixed an issue where Deployment historical metrics rendered blank in the APC UI.
  * Fixed an issue where PgBouncer extra configuration was appended to the defaults instead of overriding them.
  * Fixed an issue where `apiServer` resource requests were silently overwritten with the limit values through `upsertDeployment` on Airflow 3 Deployments.
  * Fixed an issue where the `NO_PROXY` matcher did not normalize trailing-dot fully-qualified domain names, so requests to matching internal hosts were still routed through the corporate proxy.
  * Fixed an issue where the Pilot component used a hardcoded Helm release name, blocking control plane and data plane failover on installations that use a different release name.
  * Fixed an issue where failover validation timed out on Deployments with the auth sidecar enabled.
  * Fixed an issue where the `upgrade-deployments` script published a near-empty NATS payload, dropping `globalDeploymentsConfig` from the Deployments it updated.
  * Fixed an issue where the **System Admin** sidebar tab was visible to every system role, including `SYSTEM_VIEWER` and `SYSTEM_EDITOR`.
  * Fixed an issue where the cluster configuration page's **Learn more** link pointed to a broken URL, because the APC API had no `configGovernance` documentation URL in its defaults.
  * Fixed an issue where the **Cluster Deployments Configuration** section heading in the APC UI did not describe the settings it contained.
  * Fixed an issue where the failure messages for `team_system_role_binding.create` and `team_system_role_binding.delete` were grammatically incorrect.
  * Fixed an issue where PgBouncer resource customizations set through platform Helm values were not applied to Deployments.
  * Fixed an issue where hard-deleting a Deployment left its Airflow metadata database, database users, and namespace pool slot orphaned, even though the operation reported success.
  * Fixed an issue where secondary database users were not cleaned up when a Deployment was deleted.
  * Fixed an issue where a Deployment reported a status other than `HEALTHY` while all of its Pods were `Running`, because the Pod phase check read only the first result of a multi-Pod query.
  * Fixed an issue where the Deployment page returned a 404 error when a gRPC call to the data plane failed, instead of loading the page and surfacing the error.
  * Fixed an issue where the APC API worker was not restarted after a platform upgrade.
  * Fixed an issue where an upgrade from Astronomer Software 0.37 on Azure Red Hat OpenShift failed when a Deployment switched deployment types.
  * Fixed an issue where permission denials from the APC API surfaced as `INTERNAL_SERVER_ERROR` rather than `FORBIDDEN`, and where a global error handler redirected the entire APC UI, presenting a per-resource permission error as a logout.
  * Fixed an issue where a deploy rollback failed with an error pulling the Docker manifest for the target image.
  * Fixed an issue where Elasticsearch and registry naming overrides were not applied.
  * Fixed an issue where OpenShift installations could not set `podSecurityContext` independently of the OpenShift flag.
  * Fixed an issue where the data plane metadata ConfigMap was mangled into a quoted string by trailing whitespace, and where the metadata endpoint inferred the protocol instead of returning the full URL.
  * Fixed an issue where a cleanup job failed for Airflow 3 Deployments.
  * Fixed an issue where enabling external Elastic search via `global.customLogging.enabled=true` created a duplicate Ingress for the same host by also rendering the in-cluster Elasticsearch ingress.
  * Fixed an issue where Prometheus proxy targets were incorrectly generated for unified-mode clusters after the default data plane cluster name changed to `default`.
  * Fixed an issue where DR incorrectly deleted namespaces for namespace pools.
  * Fixed an issue where databases were not cleaned up properly when using `manualReleaseName`.
  * Fixed an issue where the Helm chart included an incorrect Grafana config map mapping.
  * Fixed an issue where the data plane failover network policy referred to the wrong port when the auth sidecar was enabled.
  * Fixed an issue where the data plane failover network policy used a hardcoded release name instead of the installation's release name.
  * Fixed an issue where StatsD did not reload after a change to its mapping configuration.

  ### Known issues

  * The chart version shown in the APC UI is stale after a `helm rollback` under control plane reliability. The **Control Planes** page keeps showing the pre-rollback version, and that stale value also feeds the version-eligibility gate. As a workaround, revert with `helm upgrade` to the target version rather than `helm rollback`. The reported version corrects itself within one heartbeat interval.
  * Per-deployment migration has no rollback. A migration removes the Deployment from the source cluster before creating it on the destination, so a destination failure leaves it installed on neither. No data is lost, but the Deployment is down until you re-run the migration. As a workaround, verify destination capacity and image availability first, and migrate a single Deployment before you move a batch.
  * Adopting a Deployment rewrites a digest-pinned image to a tag reference. If an Airflow custom resource references its image by digest, adoption rewrites it to a tag built from the Deployment's Astro Runtime version on the first apply. As a workaround, re-tag the image and reference it by tag before you adopt.
  * Some changes to an adopted Deployment are ignored. Worker count, worker resources, and KEDA autoscaling changes made through APC don't take effect, because the operator still owns those fields. Change them on the Airflow custom resource instead.
  * Data plane failover can stall if Commander is unavailable for several minutes during a failover. This is carried forward from 2.0. As a workaround, restart the Pilot Deployment in the affected cluster and the failover resumes.
  * Switching a Deployment to Dag-only deployment mode fails on Azure Red Hat OpenShift, because the security context constraint rejects the configured `runAsUser`.
  * Changes to roles made on the Workspace or Deployments (for Users, Teams, or ServiceAccounts) details pages are applied successfully, but the UI does not update immediately. Refresh the page to see the latest role assignment.
  * When a large number of custom roles are configured, some roles might not be visible by default on the Roles and Permissions page. As a workaround, use the **Scope** dropdown to filter by the appropriate scope. For example, select **Scope: Workspace** to view all custom Workspace roles.
  * Newly assigned roles might not appear in the **Show all** view on the Roles and Permissions page until you refresh the page or toggle the **Hide others** button. The assignment is saved; only the display is stale.

  ### Resolved CVE list

  * [CVE-2023-26144](https://nvd.nist.gov/vuln/detail/CVE-2023-26144)
  * [CVE-2025-22872](https://nvd.nist.gov/vuln/detail/CVE-2025-22872)
  * [CVE-2025-47911](https://nvd.nist.gov/vuln/detail/CVE-2025-47911)
  * [CVE-2025-58190](https://nvd.nist.gov/vuln/detail/CVE-2025-58190)
  * [CVE-2026-0864](https://nvd.nist.gov/vuln/detail/CVE-2026-0864)
  * [CVE-2026-4360](https://nvd.nist.gov/vuln/detail/CVE-2026-4360)
  * [CVE-2026-7246](https://nvd.nist.gov/vuln/detail/CVE-2026-7246)
  * [CVE-2026-11940](https://nvd.nist.gov/vuln/detail/CVE-2026-11940)
  * [CVE-2026-11972](https://nvd.nist.gov/vuln/detail/CVE-2026-11972)
  * [CVE-2026-12590](https://nvd.nist.gov/vuln/detail/CVE-2026-12590)
  * [CVE-2026-13149](https://nvd.nist.gov/vuln/detail/CVE-2026-13149)
  * [CVE-2026-13221](https://nvd.nist.gov/vuln/detail/CVE-2026-13221)
  * [CVE-2026-13676](https://nvd.nist.gov/vuln/detail/CVE-2026-13676)
  * [CVE-2026-13697](https://nvd.nist.gov/vuln/detail/CVE-2026-13697)
  * [CVE-2026-14257](https://nvd.nist.gov/vuln/detail/CVE-2026-14257)
  * [CVE-2026-14643](https://nvd.nist.gov/vuln/detail/CVE-2026-14643)
  * [CVE-2026-15157](https://nvd.nist.gov/vuln/detail/CVE-2026-15157)
  * [CVE-2026-15308](https://nvd.nist.gov/vuln/detail/CVE-2026-15308)
  * [CVE-2026-16221](https://nvd.nist.gov/vuln/detail/CVE-2026-16221)
  * [CVE-2026-16728](https://nvd.nist.gov/vuln/detail/CVE-2026-16728)
  * [CVE-2026-16729](https://nvd.nist.gov/vuln/detail/CVE-2026-16729)
  * [CVE-2026-18446](https://nvd.nist.gov/vuln/detail/CVE-2026-18446)
  * [CVE-2026-27141](https://nvd.nist.gov/vuln/detail/CVE-2026-27141)
  * [CVE-2026-34180](https://nvd.nist.gov/vuln/detail/CVE-2026-34180)
  * [CVE-2026-39822](https://nvd.nist.gov/vuln/detail/CVE-2026-39822)
  * [CVE-2026-42151](https://nvd.nist.gov/vuln/detail/CVE-2026-42151)
  * [CVE-2026-42505](https://nvd.nist.gov/vuln/detail/CVE-2026-42505)
  * [CVE-2026-54272](https://nvd.nist.gov/vuln/detail/CVE-2026-54272)
  * [CVE-2026-55170](https://nvd.nist.gov/vuln/detail/CVE-2026-55170)
  * [CVE-2026-55689](https://nvd.nist.gov/vuln/detail/CVE-2026-55689)
  * [CVE-2026-55831](https://nvd.nist.gov/vuln/detail/CVE-2026-55831)
  * [CVE-2026-55833](https://nvd.nist.gov/vuln/detail/CVE-2026-55833)
  * [CVE-2026-56745](https://nvd.nist.gov/vuln/detail/CVE-2026-56745)
  * [CVE-2026-56746](https://nvd.nist.gov/vuln/detail/CVE-2026-56746)
  * [CVE-2026-56819](https://nvd.nist.gov/vuln/detail/CVE-2026-56819)
  * [CVE-2026-57432](https://nvd.nist.gov/vuln/detail/CVE-2026-57432)
  * [CVE-2026-59869](https://nvd.nist.gov/vuln/detail/CVE-2026-59869)
  * [CVE-2026-59871](https://nvd.nist.gov/vuln/detail/CVE-2026-59871)
  * [CVE-2026-59873](https://nvd.nist.gov/vuln/detail/CVE-2026-59873)
  * [CVE-2026-59874](https://nvd.nist.gov/vuln/detail/CVE-2026-59874)
  * [CVE-2026-59875](https://nvd.nist.gov/vuln/detail/CVE-2026-59875)
  * [CVE-2026-59876](https://nvd.nist.gov/vuln/detail/CVE-2026-59876)
  * [CVE-2026-59877](https://nvd.nist.gov/vuln/detail/CVE-2026-59877)
  * [CVE-2026-59881](https://nvd.nist.gov/vuln/detail/CVE-2026-59881)
  * [CVE-2026-59884](https://nvd.nist.gov/vuln/detail/CVE-2026-59884)
  * [CVE-2026-59885](https://nvd.nist.gov/vuln/detail/CVE-2026-59885)
  * [CVE-2026-59886](https://nvd.nist.gov/vuln/detail/CVE-2026-59886)
  * [CVE-2026-59887](https://nvd.nist.gov/vuln/detail/CVE-2026-59887)
  * [CVE-2026-59890](https://nvd.nist.gov/vuln/detail/CVE-2026-59890)
  * [CVE-2026-59898](https://nvd.nist.gov/vuln/detail/CVE-2026-59898)
  * [CVE-2026-59899](https://nvd.nist.gov/vuln/detail/CVE-2026-59899)
  * [CVE-2026-59900](https://nvd.nist.gov/vuln/detail/CVE-2026-59900)
  * [CVE-2026-59901](https://nvd.nist.gov/vuln/detail/CVE-2026-59901)
  * [CVE-2026-59921](https://nvd.nist.gov/vuln/detail/CVE-2026-59921)
  * [CVE-2026-67213](https://nvd.nist.gov/vuln/detail/CVE-2026-67213)
  * [CVE-2026-67214](https://nvd.nist.gov/vuln/detail/CVE-2026-67214)
  * [CVE-2026-69152](https://nvd.nist.gov/vuln/detail/CVE-2026-69152)
  * [CVE-2026-69192](https://nvd.nist.gov/vuln/detail/CVE-2026-69192)
  * [CVE-2026-69198](https://nvd.nist.gov/vuln/detail/CVE-2026-69198)
  * [CVE-2026-69243](https://nvd.nist.gov/vuln/detail/CVE-2026-69243)
  * [CVE-2026-69244](https://nvd.nist.gov/vuln/detail/CVE-2026-69244)
  * [CVE-2026-69247](https://nvd.nist.gov/vuln/detail/CVE-2026-69247)
  * [CVE-2026-69249](https://nvd.nist.gov/vuln/detail/CVE-2026-69249)

  ### Product lifecycle, support, and compatibility matrix

  * For Astro Private Cloud compatibility with Kubernetes, Postgres, and Astro Runtime versions, see [Version compatibility](/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
  * For the Astro Private Cloud support lifecycle, see [Astro Private Cloud release and lifecycle policy](/docs/astro-private-cloud/v-2-x/release-lifecycle-policy).
</Update>

<Update label="2.0.1" description="July 7, 2026">
  Astro Private Cloud 2.0.1 is a security patch release that also resolves a number of platform, logging, and deployment bugs.

  ### Security enhancements

  * Patched the Critical and High severity CVEs identified against Astro Private Cloud 2.0.0. See the [Resolved CVE list](#resolved-cve-list-2).

  ### Bug fixes

  * Fixed an issue where the `dagOnlyDeployment` Dag-server Pod crashed with `Permission denied` while writing to its PVC because `fsGroup` was not set on the Pod `securityContext`. The Pod `securityContext` now includes `fsGroup: 50000` so the mounted volume is group-writable.
  * Fixed an issue where Deployment config override updates were rejected when `overProvisioningFactorCPU` or `overProvisioningFactorMem` were set to fractional values such as `0.5`, because the generated schema typed these fields as integers.
  * Fixed an issue where the Vector DaemonSet entered `CrashLoopBackOff` on install because the Vector configuration retained a `graphql` field that was removed in Vector v0.55.0.
  * Fixed an issue where logs from tasks in a task group (with `prefix_group_id=True`, the default) did not appear in the Airflow UI or ship to Elasticsearch when using the Kubernetes executor with Vector sidecar or daemonset logging, because the Vector log-path pattern did not account for dots in `dag_id` and `task_id` values.
  * Fixed an issue where `AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER` was silently overridden with an empty value on all Astro Runtime 3.x Deployments, ignoring values set through the Deployment environment variables, the Dockerfile, or mounted secrets.
  * Fixed an issue where Airflow workers scheduled into a different namespace than their Deployment could not reach the execution API, because the execution API URL used a relative service name instead of a fully-qualified domain name.
  * Fixed an issue where Celery worker liveness and readiness probes failed on container images without the `hostname` binary, such as UBI-based Astro Runtime images, causing Kubernetes to restart healthy workers. The probe now resolves the hostname through Python.
  * Fixed an issue where the APC API routed requests to internal hosts through the corporate proxy despite matching `NO_PROXY` entries, because the matcher did not handle bare hostnames, `*.host` wildcards, single-label hostnames, or trailing-dot fully-qualified domain names.
  * Fixed an issue where `git-sync-relay` Pods failed to schedule on OpenShift because hardcoded `fsGroup` and `runAsUser` values fell outside the namespace-allocated UID and GID range enforced by the `restricted-v2` security context constraint.
  * Fixed an issue where the APC API database migration that adds a unique constraint on Workspace labels failed with error `P3009` when duplicate Workspace labels existed, blocking all later migrations and preventing the APC API from starting. The migration now resolves duplicate labels before it creates the constraint.
  * Fixed an issue where a failed APC API database migration surfaced raw Prisma error messages that pointed operators to tooling not available in APC. Migration errors now include APC-specific guidance and links to Astronomer documentation and support.
  * Fixed an issue where the platform-labeller pre-install Job did not include `imagePullSecrets`, causing `ErrImagePull` on installations that use a private registry.
  * Fixed an issue where setting `astroUI.volumeMounts` produced invalid YAML in the Astro UI deployment template, causing the Helm release to fail to render.
  * Fixed an issue where the Astro UI built log queries using an incorrect internal Elasticsearch proxy service name, causing log retrieval to fail with a DNS resolution error.
  * Fixed an issue where the data plane metadata endpoint returned an empty `registry.version` instead of the registry image tag.
  * Fixed an issue where the data plane internal API that the APC API interacts with silently ignored an unparseable `COMMANDER_DATAPLANE_DATABASE_URL` and returned an empty database type, leaving operators with no signal about the cause. It now logs a warning, with database credentials redacted, when the URL can't be parsed.
  * Fixed an issue where failover-related error messages in the APC API were logged as `[object Object]` instead of readable text.
  * Fixed an issue where the private CA trust script for containerd 2.x removed the GKE Docker Hub pull-through mirror. The script now preserves the mirror configuration.
  * Fixed an issue where the git-sync configuration form rejected valid SSH repository URLs that use a non-standard username, such as a service account, instead of `git`.
  * Fixed an issue where NFS-backed DAG deployments created a persistent volume claim whose name did not match the claim referenced by the dag-processor, triggerer, and API server components, so those Pods failed to mount the DAGs volume.
  * Fixed an issue where a custom `COMMANDER_HOUSTON_JWKS_ENDPOINT` override for the data plane internal API was reset to its default on every Helm upgrade, causing connection timeouts until it was manually patched.
  * Fixed an issue where registering a data plane failed with a misleading metadata error when the base domain URL included a trailing slash, because the resulting request path contained a double slash.
  * Fixed an issue where an executor disabled by a cluster-level configuration override, such as the local executor, still appeared as an option when creating a Deployment, because the Deployment configuration query did not apply cluster-level overrides.
  * Fixed an issue in the cluster Deployment configuration override where the resource slider could permit values that the save action then rejected, and where wrapping the override in an extra `deployments` key silently produced an inert configuration. The override now rejects double-wrapped input and returns a clearer out-of-range error.
  * Fixed an issue where the Deployment resource configuration UI did not respect the cluster's configured default and maximum values for component resources, including scheduler CPU and memory, because the UI did not pass the cluster context to the Deployment configuration query.
  * Fixed an issue where the Vector logging pipeline could silently drop logs from Deployments in namespaces that did not match standard Kubernetes naming, such as those created through manual namespace pools, because a namespace regex filter returned no match instead of checking for a valid namespace.
  * Fixed an issue where a fresh install could hang because the Prometheus `filesd-reloader` crash-looped when it queried the APC API database before the database migration had created the required tables. The reloader now retries until the tables are available.
  * Fixed an issue where large Airflow 3 environments running many Deployments produced repetitive Dag processor manager logging. Operators can tune how often Dag bundles are checked for changes through `bundle_refresh_check_interval`.
  * Fixed an issue where Airflow 2 Dag processor manager logs were noisily sent to stdout in containerized Deployments. Operators can control this behavior through `dag_processor_manager_log_stdout`.

  ### Resolved CVE list

  * [CVE-2026-5079](https://nvd.nist.gov/vuln/detail/CVE-2026-5079)
  * [CVE-2026-27140](https://nvd.nist.gov/vuln/detail/CVE-2026-27140)
  * [CVE-2026-27143](https://nvd.nist.gov/vuln/detail/CVE-2026-27143)
  * [CVE-2026-27144](https://nvd.nist.gov/vuln/detail/CVE-2026-27144)
  * [CVE-2026-29181](https://nvd.nist.gov/vuln/detail/CVE-2026-29181)
  * [CVE-2026-32280](https://nvd.nist.gov/vuln/detail/CVE-2026-32280)
  * [CVE-2026-32281](https://nvd.nist.gov/vuln/detail/CVE-2026-32281)
  * [CVE-2026-32283](https://nvd.nist.gov/vuln/detail/CVE-2026-32283)
  * [CVE-2026-32952](https://nvd.nist.gov/vuln/detail/CVE-2026-32952)
  * [CVE-2026-33810](https://nvd.nist.gov/vuln/detail/CVE-2026-33810)
  * [CVE-2026-33814](https://nvd.nist.gov/vuln/detail/CVE-2026-33814)
  * [CVE-2026-33816](https://nvd.nist.gov/vuln/detail/CVE-2026-33816)
  * [CVE-2026-34986](https://nvd.nist.gov/vuln/detail/CVE-2026-34986)
  * [CVE-2026-39821](https://nvd.nist.gov/vuln/detail/CVE-2026-39821)
  * [CVE-2026-39823](https://nvd.nist.gov/vuln/detail/CVE-2026-39823)
  * [CVE-2026-39827](https://nvd.nist.gov/vuln/detail/CVE-2026-39827)
  * [CVE-2026-39829](https://nvd.nist.gov/vuln/detail/CVE-2026-39829)
  * [CVE-2026-39835](https://nvd.nist.gov/vuln/detail/CVE-2026-39835)
  * [CVE-2026-39836](https://nvd.nist.gov/vuln/detail/CVE-2026-39836)
  * [CVE-2026-41506](https://nvd.nist.gov/vuln/detail/CVE-2026-41506)
  * [CVE-2026-42501](https://nvd.nist.gov/vuln/detail/CVE-2026-42501)
  * [CVE-2026-44432](https://nvd.nist.gov/vuln/detail/CVE-2026-44432)
  * [CVE-2026-45022](https://nvd.nist.gov/vuln/detail/CVE-2026-45022)
  * [CVE-2026-45570](https://nvd.nist.gov/vuln/detail/CVE-2026-45570)
  * [CVE-2026-46597](https://nvd.nist.gov/vuln/detail/CVE-2026-46597)
</Update>

<Update label="2.0.0" description="May 1, 2026">
  Astro Private Cloud 2.0 introduces three key new capabilities: **Data Plane Disaster Recovery** for business continuity across clusters, **Config Governance** for fine-grained control of APC and Airflow configuration, and **Control Plane Audit Logging** for compliance-grade activity tracking.

  ### New features

  * **Data Plane Disaster Recovery**: Astro Private Cloud now offers cross-cluster and cross-region data plane disaster recovery. You can now trigger a failover of all Airflow Deployments in a data plane from a source cluster to a destination cluster. A failover is triggered via a single action through the Astro Private Cloud API or GUI. Two modes are supported: **Controlled**, which drains in-flight tasks before promoting the destination (for planned maintenance or migrations), and **Forced**, which promotes the destination immediately when the source is unreachable (for outages).

    Data Plane DR helps to enable business continuity and meet regulatory requirements (DORA, OCC, MAS, APRA) that mandate a recovery path for production workloads, and replaces lengthy manual rebuilds with an automated platform operation. See [Data plane failover](https://www.astronomer.io/docs/astro-private-cloud/v-2-x/data-plane-failover).

  * **Config Governance**: You can now turn APC and Airflow features on or off at each level of the APC data model: platform, data plane, Workspace, and Airflow Deployment. Platform administrators set defaults at the platform level, and have the option to allow Workspace and Deployment owners to override settings without impacting other Workspaces or Deployments.

    Config Governance addresses the problem of different teams, use cases, and workloads requiring different feature sets, configurations, and guardrails. With Config Governance, a single Astro Private Cloud installation can manage use cases from a locked-down regulated workload to a permissive experimental deployment, and let Deployment owners fine-tune within the bounds their administrator has set.

  * **Control Plane Audit Logging**: The Control Plane now emits structured audit events for authentication, authorization, and Airflow Deployment lifecycle actions, including login and logout, permission and role changes, service account creation and revocation, and Deployment create, update, and delete operations. Events are written to the platform log stream alongside existing Control Plane logs, so they can be exported to your existing SIEM or log aggregation tool.

    Audit logging is a baseline requirement for SOC 2, ISO 27001, PCI-DSS, and the internal controls that banking and insurance customers use. Control Plane audit logs, in addition to the already-available Airflow Deployment audit logs, give compliance and security teams a single, structured source of truth for who did what, when.

  ### Additional improvements

  * **Cluster-level metric collection**: Node exporter and cAdvisor can now be enabled through built-in Helm flags (`global.nodeExporter.enabled`, `global.cadvisor.enabled`), both disabled by default. When enabled, four additional panels appear on the Metrics tab for each Deployment in the APC UI: CPU Usage, Memory Usage, Network Rx, and Network Tx. Cluster-level metric collection requires ClusterRole. The feature is disabled by default so the default Astro Private Cloud configuration does not require ClusterRole. See [Deployment metrics](https://www.astronomer.io/docs/astro-private-cloud/v-2-x/deployment-metrics).

  * **Git-sync relay operational metrics**: Deployments that use the git-sync relay for DAG deployment can now emit StatsD metrics for sync operations, repository size, and sync performance, giving visibility into git-sync throughput and latency per Deployment. This feature is disabled by default. See [Git-sync relay metrics](https://www.astronomer.io/docs/astro-private-cloud/v-2-x/git-sync-relay-metrics).

  * **Data Plane metadata includes custom logging state**: The `dataPlane` metadata response now includes the state of `customLogging.enabled`, making it visible whether custom logging is active for a given Data Plane without requiring a separate configuration lookup.

  * **Strict schema validation for Helm configuration**: APC 2.0 introduces a strict schema validator that rejects unknown or misspelled keys in `values.yaml` before they reach the cluster, replacing the previous silent-ignore behavior. This catches configuration mistakes at install or upgrade time instead of surfacing them as runtime failures. The validator can be temporarily disabled per-Deployment where a newer feature hasn't yet been added to the schema (for example, git-sync relay metrics in the initial 2.0 release).

  * **Direct upgrade path from Astronomer Software 0.37 to APC 2.0**: Customers still running Astronomer Software 0.37 can now upgrade directly to APC 2.0 without the intermediate hop through APC 1.0. See [Upgrade 0.37 to 2.0](https://www.astronomer.io/docs/astro-private-cloud/v-2-x/upgrade-037-to-2).

  * **Configurable Prometheus scrape interval and timeout for federated data planes**: The scrape\_interval for the federated-dataplanes job is now configurable, and a new scrape\_timeout setting has been added. This reduces Prometheus load in Control Plane/Data Plane deployments running a large number of Deployments, which previously hit a fixed 5s scrape interval.

  * **All images available in Azure Container Registry, as well as Quay and Docker Hub**: The APC image set is now mirrored to Azure Container Registry (ACR) alongside Quay and Docker Hub, simplifying installs in Azure-restricted environments.

  * **Containerd 2.0 is now supported**: The DaemonSet that manages self-signed cert trust for containerd has been updated for GKE 1.33+ (containerd 2.0), and the private CA install instructions are now functional on the latest GKE versions.

  ### Bug fixes

  * Fixed an issue where the scheduler and triggerer were not scaled down during an Airflow version downgrade, allowing core components to continue running while the database migration was triggered.
  * Fixed an issue where navigating directly to a Deployment URL appended the entire URL to the end of the path, resulting in a broken redirect. The Airflow UI is now accessible when using Deployment URLs directly.
  * Fixed an issue where the `paginatedDeployRevisions` GraphQL query returned an `INTERNAL_SERVER_ERROR` on the Deploy History page, and the release name appeared as `null`.
  * Fixed an issue where git-sync Deployments failed to become healthy because the git-sync container used deprecated environment variable names.
  * Fixed an issue where the Grafana plugin failed to initialize because the root filesystem was read-only and dashboard YAML files were missing from the image.
  * Fixed an issue where Workspace Users could create new workspaces through the UI despite the UI displaying a permission denied error.
  * Fixed an issue where the UI version tooltip displayed an incorrect version after a Helm upgrade due to a version mismatch between the root and sub-chart `Chart.yaml` files.
  * Fixed an issue where each Airflow task log line appeared twice in the UI when using KubernetesExecutor with Elasticsearch logging via Vector, caused by both log pipelines consuming the same source and writing to the same sink.
  * Fixed an issue where the Elasticsearch host configuration was not automatically generated from data plane metadata, causing the Airflow chart-level property to incorrectly override the Astronomer-owned logging configuration.
  * Fixed an issue where documentation links in the APC UI pointed to a broken URL.
  * Fixed an issue where hard-deleting a Deployment did not clean up the associated Airflow database and roles in Postgres, blocking the recreation of a Deployment with the same release name.
  * Fixed an issue where errored KubernetesExecutor worker Pods were not automatically cleaned up, causing Pods to accumulate in the namespace. Errored worker Pods are now cleaned up by default, with the option to override this behavior through Deployment-level variables.
  * Fixed an issue where `dag_processor_manager` and DAG parsing logs from the scheduler were not exported to Elasticsearch, preventing operators from filtering these logs in Kibana for root cause analysis.
  * Fixed an issue where the dag server did not support mounting private CA certificates, causing deploy revision updates to fail with certificate errors in environments with private CAs enabled.
  * Fixed an issue where git-sync sidecars in DaemonSet logging Deployments emitted JSON logs with numeric `level` fields, causing Elasticsearch `document_parsing_exception` errors and breaking task log retrieval in the Airflow UI.
  * Fixed an issue where the APC API `update-runtime-check` job did not honor HTTP/HTTPS proxy environment variables, preventing the job from reaching the runtime updates endpoint in proxy-configured environments.
  * Fixed an issue where expected error messages in the APC API were logged at the `ERROR` level, creating confusion during incident triage.
  * Fixed an issue where the Airflow UI redirected to an incorrect OAuth URL after SSO authentication when accessed directly via the Deployment URL, appending the Airflow URL to the OAuth callback path.
  * Fixed an issue where Prometheus recording rules for container CPU and memory metrics silently failed due to `kube_pod_info` emitting duplicate `node` labels during Pod scheduling transitions, causing data gaps in Grafana dashboards.
  * Fixed an issue where the deployment configuration UI accepted Scheduler and Worker CPU values that were not multiples of 100 mCPU, which are invalid for Kubernetes-backed deployments. The UI now displays a banner prompting users to enter CPU resources in multiples of 100 mCPU.

  ### Breaking Changes

  * **Config Governance schema change**: The schema used to enable and disable features in `values.yaml` has changed to support the new Platform → Data Plane → Workspace → Deployment hierarchy. In-place upgrades from APC 1.x and 0.37 are supported through migration scripts that translate existing configurations to the new schema. Review the migration guide before upgrading.
  * **`astroRuntimeEnabled` flag removed**: Astro Runtime is now the only supported runtime for Airflow deployments (Astronomer Certified has been deprecated). The `astroRuntimeEnabled` flag has been removed from houston-api, the APC UI, and the APC Helm charts. Any deployment configs, automation, or GitOps workflows that explicitly set `astroRuntimeEnabled` must remove that field before upgrading.
  * **Default cluster name simplified**: Clusters auto-created during a 1.x install or upgrade are now named `default` instead of `default-populated-by-db-bootstrapper`. Existing clusters aren't renamed, but any scripts, IaC, or external tooling that reference the previous name should be updated prior to upgrading.

  ### Known Issues

  * **Data plane failover may stall if Commander is unavailable for an extended period during a failover**: If the Commander component in the Data Plane Kubernetes cluster becomes unavailable for approximately 4–5 minutes during an in-progress failover, the failover can enter a stalled state even after Commander recovers. In this state, some Deployments may not progress, either failing to clean up on the source cluster or failing to come up on the destination cluster. This issue is only observed when Commander itself is unavailable; it does not occur when the entire Data Plane Kubernetes cluster is down. **Workaround**: Restart the Pilot Deployment in the cluster where Commander was unavailable (either the source or destination cluster, depending on which side experienced the Commander outage). The failover will then resume and complete, marking itself as either successful or failed.

  ### Resolved CVE list

  * [CVE-2024-45337](https://nvd.nist.gov/vuln/detail/CVE-2024-45337)
  * [CVE-2025-15467](https://nvd.nist.gov/vuln/detail/CVE-2025-15467)
  * [CVE-2025-62718](https://nvd.nist.gov/vuln/detail/CVE-2025-62718)
  * [CVE-2025-68121](https://nvd.nist.gov/vuln/detail/CVE-2025-68121)
  * [CVE-2026-4519](https://nvd.nist.gov/vuln/detail/CVE-2026-4519)
  * [CVE-2026-24051](https://nvd.nist.gov/vuln/detail/CVE-2026-24051)
  * [CVE-2026-27140](https://nvd.nist.gov/vuln/detail/CVE-2026-27140)
  * [CVE-2026-27144](https://nvd.nist.gov/vuln/detail/CVE-2026-27144)
  * [CVE-2026-28387](https://nvd.nist.gov/vuln/detail/CVE-2026-28387)
  * [CVE-2026-28388](https://nvd.nist.gov/vuln/detail/CVE-2026-28388)
  * [CVE-2026-28389](https://nvd.nist.gov/vuln/detail/CVE-2026-28389)
  * [CVE-2026-28390](https://nvd.nist.gov/vuln/detail/CVE-2026-28390)
  * [CVE-2026-29181](https://nvd.nist.gov/vuln/detail/CVE-2026-29181)
  * [CVE-2026-29785](https://nvd.nist.gov/vuln/detail/CVE-2026-29785)
  * [CVE-2026-32280](https://nvd.nist.gov/vuln/detail/CVE-2026-32280)
  * [CVE-2026-32281](https://nvd.nist.gov/vuln/detail/CVE-2026-32281)
  * [CVE-2026-32282](https://nvd.nist.gov/vuln/detail/CVE-2026-32282)
  * [CVE-2026-32283](https://nvd.nist.gov/vuln/detail/CVE-2026-32283)
  * [CVE-2026-32829](https://nvd.nist.gov/vuln/detail/CVE-2026-32829)
  * [CVE-2026-32887](https://nvd.nist.gov/vuln/detail/CVE-2026-32887)
  * [CVE-2026-33186](https://nvd.nist.gov/vuln/detail/CVE-2026-33186)
  * [CVE-2026-33540](https://nvd.nist.gov/vuln/detail/CVE-2026-33540)
  * [CVE-2026-33810](https://nvd.nist.gov/vuln/detail/CVE-2026-33810)
  * [CVE-2026-34986](https://nvd.nist.gov/vuln/detail/CVE-2026-34986)
  * [CVE-2026-35172](https://nvd.nist.gov/vuln/detail/CVE-2026-35172)
  * [CVE-2026-39883](https://nvd.nist.gov/vuln/detail/CVE-2026-39883)
  * [CVE-2026-40175](https://nvd.nist.gov/vuln/detail/CVE-2026-40175)
  * [CVE-2026-41676](https://nvd.nist.gov/vuln/detail/CVE-2026-41676)
  * [CVE-2026-41678](https://nvd.nist.gov/vuln/detail/CVE-2026-41678)
  * [CVE-2026-41681](https://nvd.nist.gov/vuln/detail/CVE-2026-41681)

  ### Product lifecycle, support, and compatibility matrix

  * For Astro Private Cloud compatibility with Kubernetes, Postgres, and Astro Runtime versions, see [Version compatibility](https://www.astronomer.io/docs/astro-private-cloud/v-2-x/version-compatibility-reference).
  * For the Astro Private Cloud support lifecycle, see [Astro Private Cloud release and lifecycle policy](https://www.astronomer.io/docs/astro-private-cloud/v-2-x/release-lifecycle-policy).
</Update>


# Astro Private Cloud user role and permission reference
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/role-permission-reference

A list of all default permissions for each role on Astro Private Cloud.

This page lists the default permissions for each user role on Astro Private Cloud. To modify these default permissions, see [Customize role permissions](/docs/astro-private-cloud/v-2-x/manage-platform-users#customize-role-permissions).

## Config governance permissions

Astro Private Cloud 2.0 adds permissions for config governance on the `deployments` object, which covers cluster, Workspace, and Deployment [overrides](/docs/astro-private-cloud/v-2-x/config-governance). The APC API checks these permission strings, not the UI label, when you call the GraphQL API.

Workspace-scoped users need both a Workspace role and the matching Deployment role to act in a specific Deployment. The `system.*` permissions apply when a user operates across Workspaces or Deployments from a system role binding, such as a System Admin.

| Permission                                    | APC API operation it gates                                                                  | Default roles                                                                                                                     |
| --------------------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `workspace.deployments.config.get`            | `workspaceDeploymentsConfig` query: read a Workspace's stored `deployments` config override | Workspace Viewer and above. Use `system.workspace.deployments.config.get` for the same read from a system-scoped role             |
| `workspace.deployments.config.update`         | `updateWorkspaceDeploymentsConfig` mutation                                                 | Workspace Editor and above. Use `system.workspace.deployments.config.update` (System Editor and above) for a system-scoped update |
| `workspace.deployments.config.delete`         | `deleteWorkspaceDeploymentsConfig` mutation                                                 | Workspace Admin. Use `system.workspace.deployments.config.delete` (System Admin) for a system-scoped delete                       |
| `deployment.deployments.config.update`        | `updateDeploymentConfig` for that Deployment                                                | Deployment Editor and above. Use `system.deployment.deployments.config.update` (System Editor and above) for any Deployment       |
| `deployment.deployments.config.delete`        | `deleteDeploymentConfig` for that Deployment                                                | Deployment Admin. Use `system.deployment.deployments.config.delete` (System Admin) for any Deployment                             |
| `system.workspace.deployments.config.get`     | Read a Workspace `deployments` config from a system context                                 | System Viewer and above                                                                                                           |
| `system.workspace.deployments.config.update`  | Update a Workspace `deployments` config from a system context                               | System Editor and above                                                                                                           |
| `system.workspace.deployments.config.delete`  | Delete a Workspace `deployments` config from a system context                               | System Admin                                                                                                                      |
| `system.deployment.deployments.config.update` | `updateDeploymentConfig` on any Deployment from a system context                            | System Editor and above                                                                                                           |
| `system.deployment.deployments.config.delete` | `deleteDeploymentConfig` on any Deployment from a system context                            | System Admin                                                                                                                      |

Cluster-level data plane overrides use the `updateCluster` mutation and the `cluster.config.*` and `system.clusters.update` permissions, not the `workspace.*` or `deployment.deployments.*` permissions in the preceding table. See [Override data plane cluster configurations](/docs/astro-private-cloud/v-2-x/override-data-plane-cluster).

## Default role permissions tables

The following tables compare the actions each user role permits.

### Default Deployment user permissions

| Permission                                                                                                                                                                         | **Deployment Viewer** | **Deployment Editor** | **Deployment Admin** |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | --------------------- | -------------------- |
| View the Airflow UI                                                                                                                                                                | ✔️                    | ✔️                    | ✔️                   |
| View the Deployment's settings                                                                                                                                                     | ✔️                    | ✔️                    | ✔️                   |
| View the Deployment's logs                                                                                                                                                         | ✔️                    | ✔️                    | ✔️                   |
| Access the Deployment's running Docker image                                                                                                                                       | ✔️                    | ✔️                    | ✔️                   |
| View the Deployment's **Metrics** tab in the Astro Private Cloud UI                                                                                                                | ✔️                    | ✔️                    | ✔️                   |
| View any [service account](/docs/astro-private-cloud/v-2-x/ci-cd#service-account-authentication) for the Deployment                                                                     | ✔️                    | ✔️                    | ✔️                   |
| View the Deployment's [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables)                                                                                    | ✔️                    | ✔️                    | ✔️                   |
| View the list of users with access to the Deployment                                                                                                                               | ✔️                    | ✔️                    | ✔️                   |
| View all Teams belonging to the Deployment                                                                                                                                         | ✔️                    | ✔️                    | ✔️                   |
| View task usage information for the Deployment                                                                                                                                     | ✔️                    | ✔️                    | ✔️                   |
| View Deployment Admin users                                                                                                                                                        |                       | ✔️                    | ✔️                   |
| Modify the Deployment's settings                                                                                                                                                   |                       | ✔️                    | ✔️                   |
| Upgrade the Deployment's Astro Runtime version                                                                                                                                     |                       | ✔️                    | ✔️                   |
| Airflow [user permissions](https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html#user) for the Deployment, including modifying task runs and Dag runs |                       | ✔️                    | ✔️                   |
| Push code as an image or full project deploy to the Deployment using the Astro CLI                                                                                                 |                       | ✔️                    | ✔️                   |
| Push code as a Dag-only deploy to the Deployment using the Astro CLI                                                                                                               |                       | ✔️                    | ✔️                   |
| Create, update, and delete a Deployment-level service account                                                                                                                      |                       | ✔️                    | ✔️                   |
| Update the Deployment's [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables)                                                                                  |                       | ✔️                    | ✔️                   |
| Airflow [admin permissions](https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html#admin) for the Deployment                                           |                       |                       | ✔️                   |
| Delete the Deployment                                                                                                                                                              |                       |                       | ✔️                   |
| Update Deployment-level permissions for users within the Deployment                                                                                                                |                       |                       | ✔️                   |
| Update Deployment-level permissions for Teams within the Deployment                                                                                                                |                       |                       | ✔️                   |
| Upgrade the Deployment to an unsupported version of Astro Runtime                                                                                                                  |                       |                       | ✔️                   |

### Default Workspace user permissions

| Permission                                                                                                                | **Workspace Viewer** | **Workspace Editor** | **Workspace Admin** |
| ------------------------------------------------------------------------------------------------------------------------- | -------------------- | -------------------- | ------------------- |
| View the Workspace                                                                                                        | ✔️                   | ✔️                   | ✔️                  |
| View all settings and configuration pages of any Deployment                                                               | ✔️                   | ✔️                   | ✔️                  |
| View any Deployment or Workspace-level [service account](/docs/astro-private-cloud/v-2-x/ci-cd#service-account-authentication) | ✔️                   | ✔️                   | ✔️                  |
| View information for all users with access to the Workspace                                                               | ✔️                   | ✔️                   | ✔️                  |
| View Teams belonging to the Workspace                                                                                     | ✔️                   | ✔️                   | ✔️                  |
| View any [service account](/docs/astro-private-cloud/v-2-x/ci-cd#service-account-authentication) for the Deployment            | ✔️                   | ✔️                   | ✔️                  |
| View task usage in the Workspace                                                                                          | ✔️                   | ✔️                   | ✔️                  |
| View Workspace Admin users                                                                                                |                      | ✔️                   | ✔️                  |
| Modify the Workspace, including Workspace Name, Description, and user access                                              |                      | ✔️                   | ✔️                  |
| Create a Deployment in the Workspace                                                                                      |                      | ✔️                   | ✔️                  |
| Update any Deployment in the Workspace                                                                                    |                      | ✔️                   | ✔️                  |
| Upgrade any Deployment in the Workspace                                                                                   |                      | ✔️                   | ✔️                  |
| Create, modify, and delete Workspace-level service accounts                                                               |                      | ✔️                   | ✔️                  |
| View pending user invites for the Workspace                                                                               |                      |                      | ✔️                  |
| Delete the Workspace                                                                                                      |                      |                      | ✔️                  |
| Update [IAM](/docs/astro-private-cloud/v-2-x/integrate-iam) for the Workspace                                                  |                      |                      | ✔️                  |
| View all users in Teams belonging to the Workspace                                                                        |                      |                      | ✔️                  |
| View all users in the Workspace                                                                                           |                      |                      | ✔️                  |
| Upgrade any Deployment in the Workspace to an unsupported version of Astro Runtime                                        |                      |                      | ✔️                  |

### Default System user permissions

| Permission                                                                                                                                        | **System Viewer** | **System Editor** | **System Admin** |
| ------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ----------------- | ---------------- |
| Deployment Admin permissions for all Deployments                                                                                                  | ✔️                | ✔️                | ✔️               |
| View [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables) for any Deployment                                                 | ✔️                | ✔️                | ✔️               |
| View any setting for any Deployment in the Astro Private Cloud UI                                                                                 | ✔️                | ✔️                | ✔️               |
| View all pending user invites in the **System Admin** tab of the Astro Private Cloud UI                                                           | ✔️                | ✔️                | ✔️               |
| View information for any pending user invite                                                                                                      | ✔️                | ✔️                | ✔️               |
| Access to [Grafana](/docs/astro-private-cloud/v-2-x/configure-metrics#grafana-dashboards) for system-level monitoring                                  | ✔️                | ✔️                | ✔️               |
| View [service accounts](/docs/astro-private-cloud/v-2-x/ci-cd#service-account-authentication) for any Deployment or Workspace                          | ✔️                | ✔️                | ✔️               |
| View the newest platform release version number                                                                                                   | ✔️                | ✔️                | ✔️               |
| View information for any user on the platform, including their email address, the list of Workspaces that user has access to, and their user role | ✔️                | ✔️                | ✔️               |
| View System Admin users                                                                                                                           | ✔️                | ✔️                | ✔️               |
| Modify [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables) for any Deployment                                               |                   | ✔️                | ✔️               |
| Modify [IAM](/docs/astro-private-cloud/v-2-x/integrate-iam) roles for any Deployment                                                                   |                   | ✔️                | ✔️               |
| Modify [service accounts](/docs/astro-private-cloud/v-2-x/ci-cd#service-account-authentication) for any Workspace or Deployment                        |                   | ✔️                | ✔️               |
| Airflow [user permissions](https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html#user) for all Deployments           |                   | ✔️                | ✔️               |
| Modify base layer Docker images for Deployments                                                                                                   |                   | ✔️                | ✔️               |
| Clean Deployment task metadata                                                                                                                    |                   |                   | ✔️               |
| Create, update, or delete a Deployment on any Workspace                                                                                           |                   |                   | ✔️               |
| Deploy code to any Deployment                                                                                                                     |                   |                   | ✔️               |
| View logs for any Deployment                                                                                                                      |                   |                   | ✔️               |
| View metrics for any Deployment                                                                                                                   |                   |                   | ✔️               |
| View pending user invites in all Workspaces                                                                                                       |                   |                   | ✔️               |
| Create, update, or delete a service account at any level                                                                                          |                   |                   | ✔️               |
| Create, update, or delete any Team                                                                                                                |                   |                   | ✔️               |
| Invite, update, or delete any user                                                                                                                |                   |                   | ✔️               |
| Bypass email verification for any user                                                                                                            |                   |                   | ✔️               |
| Create, update, or delete a Workspace                                                                                                             |                   |                   | ✔️               |
| Airflow [admin permissions](https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html#admin) for all Deployments         |                   |                   | ✔️               |
| Create a Deployment in any Workspace                                                                                                              |                   |                   | ✔️               |
| Update a Deployment in any Workspace                                                                                                              |                   |                   | ✔️               |
| Create a Deployment with an unsupported version of Astro Runtime                                                                                  |                   |                   | ✔️               |
| Register a new cluster                                                                                                                            |                   |                   | ✔️               |
| Deregister (remove) an existing cluster                                                                                                           |                   |                   | ✔️               |
| Update cluster configuration or metadata                                                                                                          |                   |                   | ✔️               |
| View details and status of any registered cluster                                                                                                 |                   |                   | ✔️               |

## Default role permissions lists

The following sections list the permission values that each role has by default as defined in the Astronomer Helm chart. You can update these permissions in your `values.yaml` file if you want to change the permissions that each role has. See [Customize role permissions](/docs/astro-private-cloud/v-2-x/manage-platform-users#customize-role-permissions).

These lists are also published in YAML form in the [Astronomer documentation repository](https://github.com/astronomer/astronomer-docs-resources/blob/main/software/software_configs/).

### System Viewer

The System Viewer has the following permissions by default:

* `system.airflow.get`: View the Airflow UI for any Deployment
* `system.deployment.variables.get`: View [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables) for any Deployment
* `system.deployments.get`: View any setting for any Deployment in the Astro Private Cloud UI
* `system.deployRevisions.get`: Use `paginatedDeployRevisions` API to view deploy revisions
* `system.invites.get`: View all pending user invites in the **System Admin** tab of the Astro Private Cloud UI
* `system.invite.get`: View information for any pending user invite
* `system.monitoring.get`: Access to [Grafana](/docs/astro-private-cloud/v-2-x/configure-metrics#grafana-dashboards) for system-level monitoring
* `system.serviceAccounts.get`: View [service accounts](/docs/astro-private-cloud/v-2-x/ci-cd#service-account-authentication) for any Deployment or Workspace
* `system.updates.get`: View the newest platform release version number
* `system.users.get`: View information for any user on the platform, including their email address, the list of Workspaces that user has access to, and their user role
* `system.workspace.get`: View information for any Workspace
* `system.workspace.deployments.config.get`: Read a Workspace's `deployments` config override when using a system-scoped role binding (same `workspaceDeploymentsConfig` query with system permissions)

### System Editor

The System Editor has the same default permissions as the System Viewer, plus:

* `system.adminCount.get`: View System Admin users
* `system.deployment.variables.update`: Modify [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables) for any Deployment
* `system.iam.update`: Modify [IAM](/docs/astro-private-cloud/v-2-x/integrate-iam) roles for any Deployment
* `system.serviceAccounts.update`: Modify [service accounts](/docs/astro-private-cloud/v-2-x/ci-cd#service-account-authentication) for any Workspace or Deployment
* `deployment.airflow.user`: Airflow [user permissions](https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html#user) for all Deployments
* `system.registryBaseImages.push`: Modify base layer Docker images for Deployments
* `system.workspace.deployments.config.update`: Update a Workspace's `deployments` config override from a system context (`updateWorkspaceDeploymentsConfig` with a system role)
* `system.deployment.deployments.config.update`: Update any Deployment's `deployments` config override from a system context (`updateDeploymentConfig`)

### System Admin

The System Admin has the same default permissions as the System Viewer and System Editor, plus:

* `system.clusters.register`: Register a new data plane cluster
* `system.clusters.deregister`: Deregister (remove) an existing data plane cluster
* `system.clusters.update`: Update data plane cluster configuration or metadata
* `system.clusters.get`: View details and status of any registered data plane cluster
* `system.cleanupAirflowDb.delete`: Clean Deployment task metadata
* `system.deployments.create`: Create a Deployment on any Workspace
* `system.deployments.update`: Modify any Deployment
* `system.deployments.upsert`: Use `upsertDeployment` API
* `system.deployments.delete`: Delete any Deployment
* `system.deployments.images.push`: Deploy code to any Deployment
* `system.deployments.logs`: View logs for any Deployment
* `system.deployments.metrics`: View metrics for any Deployment
* `system.invites.get`: View pending user invites in all Workspaces
* `system.serviceAccounts.create`: Create a service account at any level
* `system.serviceAccounts.delete`: Delete any service account
* `system.serviceAccounts.update`: Modify any service account
* `system.teams.remove`: Delete any Team
* `system.user.invite`: Invite a user
* `system.user.delete`: Delete a user
* `system.user.forceDelete`: Delete a user that is a part of an IdP team
* `system.user.verifyEmail`: Bypass email verification for any user
* `system.workspace.delete`: Delete any Workspace
* `system.workspace.update`: Modify the name or description of any Workspace
* `system.workspace.deployments.config.delete`: Reset a Workspace's `deployments` config override from a system context (`deleteWorkspaceDeploymentsConfig`)
* `system.deployment.deployments.config.delete`: Reset any Deployment's `deployments` config override from a system context (`deleteDeploymentConfig`)
* `system.airflow.admin`: Airflow admin permissions on any Deployment, including permission to configure:

  * Pools
  * Configuration
  * Users
  * Connections
  * Variables
  * XComs

### Workspace Viewer

The Workspace Viewer has the following default permissions for a given Workspace:

* `workspace.config.get`: View the Workspace
* `system.deployments.get`: View all settings and configuration pages of any Deployment
* `workspace.serviceAccounts.get`: View any Deployment or Workspace-level [service account](/docs/astro-private-cloud/v-2-x/ci-cd#service-account-authentication)
* `workspace.users.get`: View information for all users with access to the Workspace
* `workspace.teams.get`: View Teams belonging to the Workspace
* `workspace.taskUsage.get`: View task usage in the Workspace
* `workspace.deployments.config.get`: Read the Workspace [config governance](/docs/astro-private-cloud/v-2-x/config-governance) override (`workspaceDeploymentsConfig` in the APC API)

### Workspace Editor

For a given Workspace, the Workspace Editor has the same default permissions as the Workspace Viewer, plus:

* `workspace.adminCount.get`: View Workspace Admin users
* `workspace.config.update`: Modify the Workspace, including Workspace Name, Description, and user access
* `workspace.deployments.create`: Create a Deployment in the Workspace
* `workspace.deployments.upsert`: Use Create Deployment path within the `upsertDeployment` API
* `workspace.serviceAccounts.create`: Create a Workspace-level service account
* `workspace.serviceAccounts.update`: Modify a Workspace-level service account
* `workspace.serviceAccounts.delete`: Delete a Workspace-level service account
* `workspace.deployments.config.update`: Create or update the Workspace's `deployments` config override (`updateWorkspaceDeploymentsConfig` in the APC API)

### Workspace Admin

For a given Workspace, the Workspace Admin has the same default permissions as the Workspace Viewer and Workspace Editor, plus:

* `workspace.invites.get`: View pending user invites for the Workspace
* `workspace.config.delete`: Delete the Workspace
* `workspace.iam.update`: Update [IAM](/docs/astro-private-cloud/v-2-x/integrate-iam) for the Workspace
* `workspace.teams.getAll`: View all users in Teams belonging to the Workspace
* `workspace.users.getAll`: View all users in the Workspace
* `workspace.deployments.config.delete`: Reset the Workspace's `deployments` config override (`deleteWorkspaceDeploymentsConfig` in the APC API)

In addition, Workspace Admins have Deployment Admin permissions for all Deployments within the Workspace.

### Deployment Viewer

For a given Deployment, a Deployment Viewer has the following permissions:

* `deployment.airflow.get`: View the Airflow UI
* `deployment.config.get`: View the Deployment's settings
* `deployment.deployRevisions.get`: Use the `paginatedDeployRevisions` API to view deploy revisions
* `deployment.logs.get`: View the Deployment's logs
* `deployment.images.pull`: Access the Deployment's running Docker image
* `deployment.metrics.get`: View the Deployment's **Metrics** tab in the Astro Private Cloud UI
* `deployment.serviceAccounts.get`: View any [service account](/docs/astro-private-cloud/v-2-x/ci-cd#service-account-authentication) for the Deployment
* `deployment.status.get`: View the Deployment's [status](/docs/astro-private-cloud/v-2-x/configure-metrics#key-metrics)
* `deployment.variables.get`: View the Deployment's [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables)
* `deployment.users.get`: View the list of users with access to the Deployment
* `deployment.teams.get`: View all Teams belonging to the Deployment
* `deployment.taskUsage.get`: View task usage information for the Deployment

### Deployment Editor

For a given Deployment, the Deployment Editor has the same default permissions as the Deployment Viewer, plus:

* `deployment.adminCount.get`: View Deployment Admin users
* `deployment.airflow.user`: Airflow [user permissions](https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html#user) for all Deployments, including modifying task runs and Dag runs
* `deployment.config.update`: Modify the Deployment's settings
* `deployment.config.upsert`: Use `upsertDeployment` API
* `deployment.dags.push`: Push Dag-only code deploys to the Deployment using the Astro CLI
* `deployment.images.push`: Push code to the Deployment using the Astro CLI
* `deployment.serviceAccounts.create`: Create a Deployment-level service account
* `deployment.serviceAccounts.update`: Modify a Deployment-level service account
* `deployment.serviceAccounts.delete`: Delete a Deployment-level service account
* `deployment.variables.update`: Update the Deployment's [environment variables](/docs/astro-private-cloud/v-2-x/environment-variables)
* `deployment.deployments.config.update`: Create or update the Deployment's `deployments` config override (`updateDeploymentConfig` in the APC API)

### Deployment Admin

For a given Deployment, the Deployment Admin has the same default permissions as the Deployment Viewer and the Deployment Editor, plus:

* `deployment.config.delete`: Delete the Deployment
* `deployment.userRoles.update`: Update Deployment-level permissions for users within the Deployment
* `deployment.teamRoles.update`: Update Deployment-level permissions for Teams within the Deployment
* `deployment.deployments.config.delete`: Reset the Deployment's `deployments` config override (`deleteDeploymentConfig` in the APC API)
* `deployment.airflow.admin`: Airflow [admin permissions](https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html#admin), including permission to configure:

  * Pools
  * Configuration
  * Users
  * Connections
  * Variables
  * XComs

## Custom role permission catalog

<Note>
  **Astro Private Cloud 2.1**

  This feature was introduced in Astro Private Cloud 2.1. To access this feature, upgrade your Astro Private Cloud installation to 2.1 or later.
</Note>

This is the full list of permissions you can grant through a [custom role](/docs/astro-private-cloud/v-2-x/custom-roles). Permissions use the same categories as the role builder. Some permissions depend on others. For example, an `update` or `delete` permission depends on the matching `get` permission. Each entry notes its dependencies, and the role builder applies them automatically when you select a permission.

Creating, updating, or deleting a custom role always requires a System-scope role-management permission (`system.roles.<action>` or `system.iam.update`), regardless of what scope the role itself targets. A Workspace or Deployment Admin can still be *assigned* a role at their own scope, but authoring a role definition is a System Admin action.

<Note>
  This catalog covers permissions available to custom roles. For the default permissions of the built-in System, Workspace, and Deployment roles, see the preceding sections.
</Note>

### System administration

* `system.adminCount.get`: See how many System Admins exist on the platform. Check this before removing an admin so the platform isn't left without one.
* `system.airflow.admin`: Full admin-level access to Airflow, including Dags, connections, variables, and pools, for every Deployment on the platform. Also lets you grant that same access when building another user's custom role.
* `system.airflow.get`: Read-only access to Airflow, including Dags, runs, and logs, for every Deployment on the platform. This grants you Airflow viewer access directly.
* `system.airflow.user`: Edit-level access to Airflow for every Deployment on the platform, covering actions such as triggering and editing Dags, runs, variables, and connections, but not admin-only actions. Also lets you grant that same level when building another user's custom role.
* `system.airflow.viewer`: Doesn't grant you Airflow access. It lets you grant other users read-only Airflow access, including Dags, logs, and connections, across every Deployment when building their custom role.
* `system.cleanupAirflowDb.delete`: Start a cleanup job that purges old Airflow history, such as past task runs and logs, from one or more Deployment metadata databases so they don't grow too large.
* `system.cleanupDeployRevisions.delete`: Delete old deploy/release history entries (past deploys) older than a chosen number of days, for a single Deployment or across the whole platform.
* `system.clusters.cordon`: Put a cluster into maintenance mode (cordon) and lift it again (uncordon). While cordoned, the cluster rejects all Deployment create, update, delete, and image operations. Requires `system.clusters.get`.
* `system.clusters.deregister`: Remove a cluster from the platform's list of managed clusters. Requires `system.clusters.get`.
* `system.clusters.get`: View the list of clusters on the platform, cluster details, available disaster-recovery failover targets, and regions.
* `system.clusters.register`: Register a new cluster with the platform. Requires `system.clusters.get`.
* `system.clusters.update`: Edit cluster settings, manage disaster-recovery regions, and trigger or upgrade Deployments as part of a failover. Requires `system.clusters.get`.
* `system.deployRevisions.get`: View the deploy/release history (a record of past deploys, useful for rollbacks) for Deployments across the platform.
* `system.deployment.deployments.config.delete`: Remove one specific Deployment's advanced configuration overrides (like custom resource limits or executor settings), restoring it to the Workspace/platform defaults, for any Deployment on the platform.
* `system.deployment.deployments.config.update`: Set advanced configuration overrides (like resource limits, executors, or scaling behavior) for one specific Deployment, for any Deployment on the platform. Separate from that Deployment's everyday settings such as resources, alerts, or scaling shortcuts.
* `system.deployment.variables.get`: View a Deployment's environment variables (used to configure Airflow tasks), for any Deployment on the platform.
* `system.deployment.variables.update`: Add, edit, or remove a Deployment's environment variables, for any Deployment on the platform. Requires `system.deployment.variables.get`.
* `system.deployments.adopt`: See Airflow Deployments that exist in a cluster but aren't yet managed by Astro Private Cloud, and bring them under its management. Requires `system.deployments.get`.
* `system.deployments.cordon`: Temporarily lock a Deployment to block changes to it, and unlock it again, for any Deployment on the platform. Useful during maintenance or an incident. Requires `system.deployments.get`.
* `system.deployments.create`: Has no effect in the Astro Private Cloud UI. Creating a Deployment is controlled by `system.deployments.upsert`. Requires `system.deployments.get`.
* `system.deployments.dags.push`: Roll a Deployment back to a previous Dag-only deploy, for any Deployment on the platform. Requires `system.deployments.get`.
* `system.deployments.delete`: Permanently delete a Deployment, for any Deployment on the platform. Requires `system.deployments.get`.
* `system.deployments.get`: View Deployments across the entire platform, Deployment details, lists, and access URLs, regardless of which Workspace they belong to.
* `system.deployments.images.push`: Deploy a new or existing image to a Deployment (for example, from an external CI/CD pipeline) and roll back to a previous image deploy, for any Deployment on the platform. Requires `system.deployments.get`.
* `system.deployments.logs`: View live log streams for any Deployment on the platform. Requires `system.deployments.get`.
* `system.deployments.metrics`: View live metrics (like task and worker resource usage) for any Deployment on the platform. Requires `system.deployments.get`.
* `system.deployments.status`: View the live status of any Deployment on the platform (for example, whether it's healthy or still deploying). Requires `system.deployments.get`.
* `system.deployments.unadopt`: Release a Deployment from Astro Private Cloud management without deleting the underlying Airflow instance, for any Deployment on the platform. Requires `system.deployments.get`.
* `system.deployments.update`: Change a Deployment's everyday settings, resource sizing, scaling behavior, alert configuration, and who's assigned to it, for any Deployment on the platform. Doesn't cover advanced configuration overrides, which use `system.deployment.deployments.config.update`. Requires `system.deployments.get`.
* `system.deployments.upsert`: Create a brand-new Deployment, or save changes made in a Deployment's setup/settings form (including checking that git-sync credentials work), for any Deployment on the platform. Requires `system.deployments.get`.
* `system.iam.update`: Manage user and Team access platform-wide: assign or remove system-level roles, add or remove Workspace members and Teams, bulk-import Airflow users into a Deployment, and manage custom role assignments for anyone. Requires `system.users.get`.
* `system.invite.get`: Has no effect in Astro Private Cloud. Viewing pending platform invitations is controlled by `system.invites.get`.
* `system.invites.get`: View the list of pending invitations to join the platform.
* `system.monitoring.get`: Access the platform's monitoring dashboards (such as Grafana, Prometheus, Alertmanager, and Elasticsearch).
* `system.registryBaseImages.push`: Publish updates to the platform's shared base Airflow images that every Deployment is built from.
* `system.roles.create`: Create new custom roles of any scope (system, Workspace, Deployment, or cluster) in the role builder. Requires `system.roles.get`.
* `system.roles.delete`: Delete existing custom roles of any scope. Requires `system.roles.get`.
* `system.roles.get`: View existing custom roles of any scope and see who they're assigned to.
* `system.roles.update`: Edit existing custom roles of any scope, including which permissions they grant. Requires `system.roles.get`.
* `system.serviceAccounts.create`: Create service accounts (API keys used by automation instead of a person) anywhere on the platform, system-wide, or for any Workspace or Deployment. Requires `system.serviceAccounts.get`.
* `system.serviceAccounts.delete`: Delete service accounts anywhere on the platform, system-wide, or for any Workspace or Deployment. Requires `system.serviceAccounts.get`.
* `system.serviceAccounts.get`: View service accounts anywhere on the platform, system-wide, or for any Workspace or Deployment.
* `system.serviceAccounts.update`: Edit service accounts anywhere on the platform (for example, their permissions), system-wide, or for any Workspace or Deployment. Requires `system.serviceAccounts.get`.
* `system.taskUsage.get`: View task usage metrics, the task-run counts used for consumption-based billing, for the whole platform, any Workspace, or any Deployment.
* `system.teams.create`: Create new Teams on the platform. Requires `system.teams.get`.
* `system.teams.get`: View Teams and their members, for any Team on the platform (including within a specific Workspace or Deployment).
* `system.teams.remove`: Delete a Team from the platform. Requires `system.teams.get`.
* `system.teams.update`: Edit a Team's details, for any Team on the platform. Requires `system.teams.get`.
* `system.updates.get`: See whether a newer version of the platform is available to upgrade to.
* `system.user.delete`: Remove a user from the platform. Requires `system.users.get`.
* `system.user.forceDelete`: Remove a user from the platform even if they belong to a Team synced from your company's identity provider, normally blocked to protect those accounts from accidental removal. Requires `system.users.get`.
* `system.user.invite`: Invite a new user to join the platform. Requires `system.users.get`.
* `system.user.verifyEmail`: Manually mark a user's email address as verified. Requires `system.users.get`.
* `system.users.get`: View the list of users on the platform, their role assignments, and their Workspace/Deployment membership.
* `system.workspace.create`: Create a new Workspace. Every authenticated user already holds this permission through the built-in user role, so granting it explicitly in a custom role has no additional effect.
* `system.workspace.delete`: Delete a Workspace, for any Workspace on the platform. Requires `system.workspace.get`.
* `system.workspace.deployments.config.delete`: Remove the default configuration overrides a Workspace applies to its Deployments (like resource limits or executor settings), for any Workspace on the platform. Requires `system.workspace.deployments.config.get`.
* `system.workspace.deployments.config.get`: View the default configuration overrides a Workspace applies to its Deployments, for any Workspace on the platform.
* `system.workspace.deployments.config.update`: Set default configuration overrides (like resource limits or executor settings) that apply to every Deployment within a Workspace, for any Workspace on the platform. Requires `system.workspace.deployments.config.get`.
* `system.workspace.get`: View any Workspace on the platform, and see the full list of every Workspace, regardless of your membership in it.
* `system.workspace.update`: Edit a Workspace's general settings (like its name or description), for any Workspace on the platform. Requires `system.workspace.get`.

### Cluster management

* `cluster.config.create`: Register a new data plane cluster with the platform so Deployments can run on it. Requires `cluster.config.get`.
* `cluster.config.delete`: Deregister a cluster from the platform, removing it from the list of clusters that Deployments can run on. Requires `cluster.config.get`.
* `cluster.config.cordon`: Put a cluster into maintenance mode (cordon) and lift it again (uncordon). While cordoned, the cluster rejects all Deployment create, update, delete, and image operations. Requires `cluster.config.get`.
* `cluster.config.get`: View a cluster's details and status, the cloud regions available to it, and which other clusters it could fail over to.
* `cluster.config.update`: Edit a cluster's settings (name, connection endpoints, assigned region, Deployment overrides), and trigger or manage failover to another cluster. Requires `cluster.config.get`.
* `cluster.roles.get`: View the custom roles and role assignments that control cluster-level administration.
* `cluster.roles.update`: Edit a cluster-administration role template, and assign or change who holds cluster-admin access. Requires `cluster.roles.get`.

### Workspace management

* `workspace.adminCount.get`: See how many Workspace Admins exist for this Workspace, used to warn if an action would leave it without one.
* `workspace.config.delete`: Permanently delete this Workspace and everything in it. Requires `workspace.config.get`.
* `workspace.config.get`: View a Workspace's basic details, such as its name and description.
* `workspace.config.update`: Change a Workspace's name or description. Requires `workspace.config.get`.
* `workspace.deployments.config.delete`: Remove any custom configuration overrides for the Workspace, reverting all its Deployments back to the platform's standard defaults. Requires `workspace.deployments.config.get`.
* `workspace.deployments.config.get`: View the Workspace-wide configuration overrides that apply to every Deployment in this Workspace. This is different from viewing the Deployments themselves.
* `workspace.deployments.config.update`: Change the Workspace-wide configuration overrides (such as default resource limits) that apply to every Deployment in this Workspace. Requires `workspace.deployments.config.get`.
* `workspace.deployments.adopt`: Bring an Airflow environment that's already running on a cluster, but not yet managed here, under this Workspace so it can be managed as a Deployment. Requires `workspace.deployments.get`.
* `workspace.deployments.cordon`: Pause platform updates to a Deployment, or resume them. While paused, its Airflow environment keeps running, but the platform stops applying changes to it. Requires `workspace.deployments.get`.
* `workspace.deployments.create`: Create new Deployments within this Workspace. Requires `workspace.deployments.get`.
* `workspace.deployments.get`: View the list of Deployments in this Workspace and open an individual Deployment's details. This is different from viewing the Workspace's Deployment configuration overrides.
* `workspace.deployments.unadopt`: Release a previously adopted Deployment from this Workspace's management, without affecting the underlying Airflow environment, which keeps running untouched. Requires `workspace.deployments.get`.
* `workspace.deployments.upsert`: Create a new Deployment within this Workspace. Requires `workspace.deployments.get`.
* `workspace.iam.update`: Add or remove users and Teams from this Workspace, and change which role each one holds. This controls who has access and which role they're assigned, not the role definitions themselves. Use the `workspace.roles.*` permissions to create or edit roles. Requires `workspace.users.get`.
* `workspace.invites.get`: View pending invitations to join this Workspace.
* `workspace.roles.get`: View the custom roles belonging to this Workspace, and see who currently holds each one. Creating, updating, or deleting a role always requires a System-scope role-management permission, even for a Workspace-scoped role — see the note at the top of this section.
* `workspace.serviceAccounts.create`: Create a new API service account (a non-human credential) for programmatic access to this Workspace. Requires `workspace.serviceAccounts.get`.
* `workspace.serviceAccounts.delete`: Delete an API service account from this Workspace, revoking its programmatic access. Requires `workspace.serviceAccounts.get`.
* `workspace.serviceAccounts.get`: View the API service accounts that exist in this Workspace.
* `workspace.serviceAccounts.update`: Edit an existing API service account in this Workspace, such as its name or assigned role. Requires `workspace.serviceAccounts.get`.
* `workspace.taskUsage.get`: View task-execution usage statistics for the Deployments in this Workspace.
* `workspace.teams.get`: View the Teams that already belong to this Workspace.
* `workspace.teams.getAll`: Search and browse every Team across the entire platform, for example, to find one to add to this Workspace, not just the Teams already in it. Requires `workspace.teams.get`.
* `workspace.users.get`: View the people who are already members of this Workspace.
* `workspace.users.getAll`: Search and browse every user across the entire platform, for example, to find someone to add through identity provider group import, not just the members already in this Workspace. Requires `workspace.users.get`.

### Deployment management

* `deployment.adminCount.get`: See how many admins currently have full control over this Deployment (used to warn before removing the last one).
* `deployment.config.delete`: Permanently delete this Deployment. Requires `deployment.config.get`.
* `deployment.config.get`: View this Deployment and open its pages at all. Most other Deployment permissions require this as a baseline.
* `deployment.config.update`: Change this Deployment's operating settings, things like autoscaling, alert email recipients, and moving it to another cluster during a failover. Requires `deployment.config.get`.
* `deployment.config.upsert`: Create a new Deployment, or edit an existing Deployment's core setup, name, executor, workers/resources, runtime version, and Dag deploy/git-sync configuration. Requires `deployment.config.get`.
* `deployment.dags.push`: Deploy new Dag code to this Deployment, used both by people deploying from the UI and by CI/CD automation (for example, the Astro CLI).
* `deployment.deployRevisions.get`: View this Deployment's deploy history, the list of past code/image deploys and when they happened. Requires `deployment.config.get`.
* `deployment.deployments.config.delete`: Remove an advanced, platform-level configuration override applied to this specific Deployment (an admin/support-level setting, separate from the Deployment's regular settings).
* `deployment.deployments.config.update`: Set an advanced, platform-level configuration override on this specific Deployment (an admin/support-level setting, separate from the Deployment's regular settings).
* `deployment.deployments.cordon`: Temporarily pause or resume scheduling on this Deployment (cordon/uncordon), blocking new activity without deleting it.
* `deployment.images.pull`: Retrieve this Deployment's built container images, used when promoting/retagging an image between Deployments or environments.
* `deployment.images.push`: Deploy a new container image to this Deployment, the action behind deploying custom Docker images, typically used by CI/CD automation as well as manual image deploys.
* `deployment.logs.get`: View this Deployment's raw component logs (scheduler, webserver, workers, triggerer) for troubleshooting.
* `deployment.metrics.get`: View this Deployment's performance graphs and charts, including CPU, memory, and task throughput.
* `deployment.roles.get`: View the custom role definitions available to assign on Deployments, including what each one grants. Creating, updating, or deleting a role always requires a System-scope role-management permission, even for a Deployment-scoped role — see the note at the top of this section.
* `deployment.serviceAccounts.create`: Create API-key based service accounts for this Deployment, used to authenticate automation and CI/CD tools. Requires `deployment.serviceAccounts.get`.
* `deployment.serviceAccounts.delete`: Delete API-key based service accounts on this Deployment. Requires `deployment.serviceAccounts.get`.
* `deployment.serviceAccounts.get`: View the API-key based service accounts configured on this Deployment.
* `deployment.serviceAccounts.update`: Edit API-key based service accounts on this Deployment (for example, rename or change their role). Requires `deployment.serviceAccounts.get`.
* `deployment.status.get`: View this Deployment's overall health/status indicator (for example, healthy, degraded).
* `deployment.taskUsage.get`: View task-execution usage data for this Deployment, used for usage and billing reporting.
* `deployment.teamRoles.update`: Assign or change which role a specific Team holds on this Deployment. This is about who has access and at what level, not about editing role definitions. Requires `deployment.teams.get`.
* `deployment.teams.get`: View which Teams have access to this Deployment.
* `deployment.userRoles.update`: Assign or change which role a specific person holds on this Deployment. This is about who has access and at what level, not about editing role definitions. Requires `deployment.users.get`.
* `deployment.users.get`: View which individual people have access to this Deployment.
* `deployment.variables.get`: View this Deployment's environment variables.
* `deployment.variables.update`: Add, edit, or remove this Deployment's environment variables. Requires `deployment.variables.get`.

### Airflow access

* `deployment.airflow.access`: Open this Deployment's Airflow UI. On its own this shows an empty shell — each page inside needs its own permission — so this is the starting point every other Airflow permission builds on. Requires `deployment.config.get`.
* `deployment.airflow.admin`: Full control inside this Deployment's Airflow UI, including viewing, editing, and running Dags, connections, and variables. This is the highest of the three Airflow access tiers, and grants everything the granular Airflow permissions below would grant individually. Requires `deployment.config.get`.
* `deployment.airflow.get`: View-only access to this Deployment's Airflow UI. Despite the name, this is the entire read-only viewer tier (Dags, runs, task instances, logs, and audit logs), not a single narrow capability. Grant this only when someone should see all of it; otherwise, grant the individual Airflow permissions below along with `deployment.airflow.access`. Requires `deployment.config.get`.
* `deployment.airflow.user`: Day-to-day working access to this Deployment's Airflow UI, can edit and trigger things like Dags and connections, but without full admin-level control. Requires `deployment.config.get`.

### Airflow configuration

* `deployment.airflow.config.read`: View Airflow's own webserver configuration page for this Deployment. Requires `deployment.airflow.access`.

### Dag operations

<Note>The permissions in this category apply on both Airflow 2 and Airflow 3. Airflow 2's security model uses `deployment.airflow.dag.read` as the parent check for Dag runs, task instances, logs, and more, so it isn't optional there — without it, permissions that depend on it have nothing to attach to.</Note>

* `deployment.airflow.dag.read`: View Dag-level details and metadata, such as whether a Dag is paused, in the Airflow UI for this Deployment. Requires `deployment.airflow.access`.
* `deployment.airflow.dag.edit`: Edit Dag-level settings, such as pausing or unpausing a Dag, in the Airflow UI for this Deployment. Requires `deployment.airflow.dag.read`.
* `deployment.airflow.dag.delete`: Delete a Dag's metadata from the Airflow UI for this Deployment. Requires `deployment.airflow.dag.read`.
* `deployment.airflow.dag_code.read`: View the Python source code of this Deployment's Dags in the Airflow UI. Requires `deployment.airflow.dag.read` and `deployment.airflow.task_instance.read`.

### Dag runs

* `deployment.airflow.dag_run.read`: View Dag runs (individual executions of a Dag) for this Deployment. Requires `deployment.airflow.dag.read`.
* `deployment.airflow.dag_run.create`: Manually trigger new Dag runs (individual executions of a Dag) for this Deployment. Requires `deployment.airflow.dag_run.read`.
* `deployment.airflow.dag_run.edit`: Edit Dag runs (individual executions of a Dag) for this Deployment, such as changing a run's state. Requires `deployment.airflow.dag_run.read`.
* `deployment.airflow.dag_run.delete`: Delete Dag runs (individual executions of a Dag) for this Deployment. Requires `deployment.airflow.dag_run.read`.
* `deployment.airflow.dag_run.clear`: Clear Dag runs for this Deployment, resetting the run's state so its tasks can be re-executed. Requires `deployment.airflow.dag_run.read`.

### Task instances

* `deployment.airflow.task_instance.read`: View task instances, a single task's execution within a specific Dag run, for this Deployment. Requires `deployment.airflow.dag.read`.
* `deployment.airflow.task_instance.create`: Create new task instances, a single task's execution within a specific Dag run, for this Deployment. Requires `deployment.airflow.task_instance.read`.
* `deployment.airflow.task_instance.edit`: Edit task instances for this Deployment, such as marking a task's execution as success, failed, or ready to retry. Requires `deployment.airflow.task_instance.read`.
* `deployment.airflow.task_instance.delete`: Delete task instances, a single task's execution within a specific Dag run, for this Deployment. Requires `deployment.airflow.task_instance.read`.

### Task logs

* `deployment.airflow.task_log.read`: View task logs, the log output produced by a task's execution, for this Deployment. Requires `deployment.airflow.dag.read`.

### Triggers

* `deployment.airflow.trigger.read`: View Triggers, the waiting half of a deferred task, listing which trigger classes are currently pending, for this Deployment. Requires `deployment.airflow.access`.

### Variables and connections

* `deployment.airflow.variable.read`: View Airflow variables (key-value configuration values Dags can read at runtime) for this Deployment. Requires `deployment.airflow.access`.
* `deployment.airflow.variable.create`: Create new Airflow variables (key-value configuration values Dags can read at runtime) for this Deployment. Requires `deployment.airflow.variable.read`.
* `deployment.airflow.variable.edit`: Edit existing Airflow variables (key-value configuration values Dags can read at runtime) for this Deployment. Requires `deployment.airflow.variable.read`.
* `deployment.airflow.variable.delete`: Delete Airflow variables (key-value configuration values Dags can read at runtime) for this Deployment. Requires `deployment.airflow.variable.read`.
* `deployment.airflow.connection.read`: View Airflow connections (stored credentials for external systems like databases, APIs, and cloud services) for this Deployment. Requires `deployment.airflow.access`.
* `deployment.airflow.connection.create`: Create new Airflow connections (stored credentials for external systems like databases, APIs, and cloud services) for this Deployment. Requires `deployment.airflow.connection.read`.
* `deployment.airflow.connection.edit`: Edit existing Airflow connections (stored credentials for external systems like databases, APIs, and cloud services) for this Deployment. Requires `deployment.airflow.connection.read`.
* `deployment.airflow.connection.delete`: Delete Airflow connections (stored credentials for external systems like databases, APIs, and cloud services) for this Deployment. Requires `deployment.airflow.connection.read`.

### Pools and XComs

* `deployment.airflow.pool.read`: View Airflow pools (named limits on how many tasks can run at the same time) for this Deployment. Requires `deployment.airflow.access`.
* `deployment.airflow.pool.create`: Create new Airflow pools (named limits on how many tasks can run at the same time) for this Deployment. Requires `deployment.airflow.pool.read`.
* `deployment.airflow.pool.edit`: Edit existing Airflow pools (named limits on how many tasks can run at the same time) for this Deployment. Requires `deployment.airflow.pool.read`.
* `deployment.airflow.pool.delete`: Delete Airflow pools (named limits on how many tasks can run at the same time) for this Deployment. Requires `deployment.airflow.pool.read`.
* `deployment.airflow.xcom.read`: View XComs (small pieces of data tasks pass to each other during execution) for this Deployment. Requires `deployment.airflow.dag.read`.
* `deployment.airflow.xcom.create`: Create new XComs (small pieces of data tasks pass to each other during execution) for this Deployment. Requires `deployment.airflow.xcom.read`.
* `deployment.airflow.xcom.delete`: Delete XComs (small pieces of data tasks pass to each other during execution) for this Deployment. Requires `deployment.airflow.xcom.read`.

### Datasets

* `deployment.airflow.dataset.read`: View datasets (data-aware scheduling assets that can trigger Dags when updated) for this Deployment. Requires `deployment.airflow.access`.
* `deployment.airflow.dataset.create`: Create dataset events (manually mark a dataset as updated to trigger downstream Dags) for this Deployment. Requires `deployment.airflow.dataset.read`.
* `deployment.airflow.dataset.delete`: Delete datasets for this Deployment. Requires `deployment.airflow.dataset.read`.

### Backfills

* `deployment.airflow.backfill.read`: View backfills (re-runs of a Dag over a past date range it already covers) for this Deployment. Requires `deployment.airflow.access`.
* `deployment.airflow.backfill.create`: Start new backfills (re-runs of a Dag over a past date range it already covers) for this Deployment. Requires `deployment.airflow.backfill.read`.
* `deployment.airflow.backfill.edit`: Edit or manage in-progress backfills (re-runs of a Dag over a past date range it already covers) for this Deployment. Requires `deployment.airflow.backfill.read`.
* `deployment.airflow.backfill.delete`: Cancel or delete backfills (re-runs of a Dag over a past date range it already covers) for this Deployment. Requires `deployment.airflow.backfill.read`.

### Human in the loop

* `deployment.airflow.hitl_detail.read`: View human-in-the-loop task details, for tasks that pause mid-run for manual approval or input, for this Deployment. Requires `deployment.airflow.access`.
* `deployment.airflow.hitl_detail.edit`: Respond to human-in-the-loop tasks, such as approving or providing input for tasks paused mid-run, for this Deployment. Requires `deployment.airflow.hitl_detail.read`.

### Audit and logs

* `deployment.airflow.audit_log.read`: View the audit log, a record of actions users take in the Airflow UI, for this Deployment. Requires `deployment.airflow.dag.read`.
* `deployment.airflow.import_error.read`: See the errors Airflow reports when a Dag file fails to load for this Deployment. The messages quote file paths and lines of the Dag's code. Requires `deployment.airflow.access`.

### Browse and monitoring

* `deployment.airflow.sla_miss.read`: View service-level agreement (SLA) misses, which are tasks that ran past their defined SLA, for this Deployment. Requires `deployment.airflow.dag.read`.
* `deployment.airflow.sla_miss.edit`: Edit SLA miss records for this Deployment. Requires `deployment.airflow.sla_miss.read`.
* `deployment.airflow.sla_miss.delete`: Delete SLA miss records for this Deployment. Requires `deployment.airflow.sla_miss.read`.
* `deployment.airflow.task_reschedule.read`: View task reschedules, which are records of sensor tasks rescheduled while waiting, for this Deployment. Requires `deployment.airflow.dag.read`.
* `deployment.airflow.cluster_activity.read`: View the Cluster Activity page, a dashboard of Dag run and task health, for this Deployment. Requires `deployment.airflow.access`.
* `deployment.airflow.dag_warning.read`: View Dag warnings, which are parsing and configuration warnings raised for Dags, for this Deployment. Requires `deployment.airflow.dag.read`.

### Providers and plugins

* `deployment.airflow.provider.read`: View the Providers page, the installed Airflow provider packages and their versions, for this Deployment. Requires `deployment.airflow.access`.
* `deployment.airflow.plugin.read`: View the Plugins page, the installed Airflow plugins, for this Deployment. Requires `deployment.airflow.access`.

## Full permission dependency reference

The role builder adds a permission's dependencies for you automatically when you select it. If you're calling the APC API directly instead of using the role builder, dependencies aren't added for you — submitting an incomplete permission list is rejected, and the error alone doesn't tell you the rest of the chain.

The tables below list every permission's complete dependency chain, so you can assemble a full `permissionKeys` list without tracing it yourself one entry at a time. To grant a permission through the API, include it along with everything in its dependency chain.

For example, to grant `deployment.airflow.dag_run.read` directly through the API, include all of `deployment.airflow.dag_run.read`, `deployment.airflow.dag.read`, `deployment.airflow.access`, and `deployment.config.get` in the same request.

### System scope

| Permission                                    | Full dependency chain                     |
| --------------------------------------------- | ----------------------------------------- |
| `system.adminCount.get`                       | None                                      |
| `system.airflow.admin`                        | None                                      |
| `system.airflow.get`                          | None                                      |
| `system.airflow.user`                         | None                                      |
| `system.airflow.viewer`                       | None                                      |
| `system.cleanupAirflowDb.delete`              | None                                      |
| `system.cleanupDeployRevisions.delete`        | None                                      |
| `system.clusters.cordon`                      | `system.clusters.get`                     |
| `system.clusters.deregister`                  | `system.clusters.get`                     |
| `system.clusters.get`                         | None                                      |
| `system.clusters.register`                    | `system.clusters.get`                     |
| `system.clusters.update`                      | `system.clusters.get`                     |
| `system.deployRevisions.get`                  | None                                      |
| `system.deployment.deployments.config.delete` | None                                      |
| `system.deployment.deployments.config.update` | None                                      |
| `system.deployment.variables.get`             | None                                      |
| `system.deployment.variables.update`          | `system.deployment.variables.get`         |
| `system.deployments.adopt`                    | `system.deployments.get`                  |
| `system.deployments.cordon`                   | `system.deployments.get`                  |
| `system.deployments.create`                   | `system.deployments.get`                  |
| `system.deployments.dags.push`                | `system.deployments.get`                  |
| `system.deployments.delete`                   | `system.deployments.get`                  |
| `system.deployments.get`                      | None                                      |
| `system.deployments.images.push`              | `system.deployments.get`                  |
| `system.deployments.logs`                     | `system.deployments.get`                  |
| `system.deployments.metrics`                  | `system.deployments.get`                  |
| `system.deployments.status`                   | `system.deployments.get`                  |
| `system.deployments.unadopt`                  | `system.deployments.get`                  |
| `system.deployments.update`                   | `system.deployments.get`                  |
| `system.deployments.upsert`                   | `system.deployments.get`                  |
| `system.iam.update`                           | `system.users.get`                        |
| `system.invite.get`                           | None                                      |
| `system.invites.get`                          | None                                      |
| `system.monitoring.get`                       | None                                      |
| `system.registryBaseImages.push`              | None                                      |
| `system.roles.create`                         | `system.roles.get`                        |
| `system.roles.delete`                         | `system.roles.get`                        |
| `system.roles.get`                            | None                                      |
| `system.roles.update`                         | `system.roles.get`                        |
| `system.serviceAccounts.create`               | `system.serviceAccounts.get`              |
| `system.serviceAccounts.delete`               | `system.serviceAccounts.get`              |
| `system.serviceAccounts.get`                  | None                                      |
| `system.serviceAccounts.update`               | `system.serviceAccounts.get`              |
| `system.taskUsage.get`                        | None                                      |
| `system.teams.create`                         | `system.teams.get`                        |
| `system.teams.get`                            | None                                      |
| `system.teams.remove`                         | `system.teams.get`                        |
| `system.teams.update`                         | `system.teams.get`                        |
| `system.updates.get`                          | None                                      |
| `system.user.delete`                          | `system.users.get`                        |
| `system.user.forceDelete`                     | `system.users.get`                        |
| `system.user.invite`                          | `system.users.get`                        |
| `system.user.verifyEmail`                     | `system.users.get`                        |
| `system.users.get`                            | None                                      |
| `system.workspace.create`                     | None                                      |
| `system.workspace.delete`                     | `system.workspace.get`                    |
| `system.workspace.deployments.config.delete`  | `system.workspace.deployments.config.get` |
| `system.workspace.deployments.config.get`     | None                                      |
| `system.workspace.deployments.config.update`  | `system.workspace.deployments.config.get` |
| `system.workspace.get`                        | None                                      |
| `system.workspace.update`                     | `system.workspace.get`                    |

### Cluster scope

| Permission              | Full dependency chain |
| ----------------------- | --------------------- |
| `cluster.config.cordon` | `cluster.config.get`  |
| `cluster.config.create` | `cluster.config.get`  |
| `cluster.config.delete` | `cluster.config.get`  |
| `cluster.config.get`    | None                  |
| `cluster.config.update` | `cluster.config.get`  |
| `cluster.roles.get`     | None                  |
| `cluster.roles.update`  | `cluster.roles.get`   |

### Workspace scope

| Permission                            | Full dependency chain                                                          |
| ------------------------------------- | ------------------------------------------------------------------------------ |
| `workspace.adminCount.get`            | `workspace.config.get`                                                         |
| `workspace.config.delete`             | `workspace.deployments.get`, `workspace.config.get`                            |
| `workspace.config.get`                | None                                                                           |
| `workspace.config.update`             | `workspace.config.get`                                                         |
| `workspace.deployments.adopt`         | `workspace.deployments.get`, `workspace.config.get`                            |
| `workspace.deployments.config.delete` | `workspace.deployments.config.get`, `workspace.config.get`                     |
| `workspace.deployments.config.get`    | `workspace.config.get`                                                         |
| `workspace.deployments.config.update` | `workspace.deployments.config.get`, `workspace.config.get`                     |
| `workspace.deployments.cordon`        | `workspace.deployments.get`, `workspace.config.get`                            |
| `workspace.deployments.create`        | `workspace.deployments.get`, `workspace.config.get`                            |
| `workspace.deployments.get`           | `workspace.config.get`                                                         |
| `workspace.deployments.unadopt`       | `workspace.deployments.get`, `workspace.config.get`                            |
| `workspace.deployments.upsert`        | `workspace.deployments.get`, `workspace.config.get`                            |
| `workspace.iam.update`                | `workspace.users.get`, `workspace.roles.get`, `workspace.config.get`           |
| `workspace.invites.get`               | `workspace.roles.get`, `workspace.config.get`                                  |
| `workspace.roles.get`                 | `workspace.config.get`                                                         |
| `workspace.serviceAccounts.create`    | `workspace.serviceAccounts.get`, `workspace.roles.get`, `workspace.config.get` |
| `workspace.serviceAccounts.delete`    | `workspace.serviceAccounts.get`, `workspace.config.get`                        |
| `workspace.serviceAccounts.get`       | `workspace.config.get`                                                         |
| `workspace.serviceAccounts.update`    | `workspace.serviceAccounts.get`, `workspace.roles.get`, `workspace.config.get` |
| `workspace.taskUsage.get`             | `workspace.config.get`                                                         |
| `workspace.teams.get`                 | `workspace.config.get`                                                         |
| `workspace.teams.getAll`              | `workspace.teams.get`, `workspace.config.get`                                  |
| `workspace.users.get`                 | `workspace.config.get`                                                         |
| `workspace.users.getAll`              | `workspace.users.get`, `workspace.config.get`                                  |

### Deployment scope

| Permission                                 | Full dependency chain                                                                                                                                       |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `deployment.adminCount.get`                | `deployment.config.get`                                                                                                                                     |
| `deployment.airflow.access`                | `deployment.config.get`                                                                                                                                     |
| `deployment.airflow.admin`                 | `deployment.config.get`                                                                                                                                     |
| `deployment.airflow.audit_log.read`        | `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`                                                                         |
| `deployment.airflow.backfill.create`       | `deployment.airflow.backfill.read`, `deployment.airflow.access`, `deployment.config.get`                                                                    |
| `deployment.airflow.backfill.delete`       | `deployment.airflow.backfill.read`, `deployment.airflow.access`, `deployment.config.get`                                                                    |
| `deployment.airflow.backfill.edit`         | `deployment.airflow.backfill.read`, `deployment.airflow.access`, `deployment.config.get`                                                                    |
| `deployment.airflow.backfill.read`         | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.cluster_activity.read` | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.config.read`           | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.connection.create`     | `deployment.airflow.connection.read`, `deployment.airflow.access`, `deployment.config.get`                                                                  |
| `deployment.airflow.connection.delete`     | `deployment.airflow.connection.read`, `deployment.airflow.access`, `deployment.config.get`                                                                  |
| `deployment.airflow.connection.edit`       | `deployment.airflow.connection.read`, `deployment.airflow.access`, `deployment.config.get`                                                                  |
| `deployment.airflow.connection.read`       | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.dag.delete`            | `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`                                                                         |
| `deployment.airflow.dag.edit`              | `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`                                                                         |
| `deployment.airflow.dag.read`              | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.dag_code.read`         | `deployment.airflow.dag.read`, `deployment.airflow.task_instance.read`, `deployment.airflow.access`, `deployment.config.get`                                |
| `deployment.airflow.dag_run.clear`         | `deployment.airflow.dag_run.read`, `deployment.airflow.dag.edit`, `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`       |
| `deployment.airflow.dag_run.create`        | `deployment.airflow.dag_run.read`, `deployment.airflow.dag.edit`, `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`       |
| `deployment.airflow.dag_run.delete`        | `deployment.airflow.dag_run.read`, `deployment.airflow.dag.edit`, `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`       |
| `deployment.airflow.dag_run.edit`          | `deployment.airflow.dag_run.read`, `deployment.airflow.dag.edit`, `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`       |
| `deployment.airflow.dag_run.read`          | `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`                                                                         |
| `deployment.airflow.dag_warning.read`      | `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`                                                                         |
| `deployment.airflow.dataset.create`        | `deployment.airflow.dataset.read`, `deployment.airflow.access`, `deployment.config.get`                                                                     |
| `deployment.airflow.dataset.delete`        | `deployment.airflow.dataset.read`, `deployment.airflow.access`, `deployment.config.get`                                                                     |
| `deployment.airflow.dataset.read`          | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.get`                   | `deployment.config.get`                                                                                                                                     |
| `deployment.airflow.hitl_detail.edit`      | `deployment.airflow.hitl_detail.read`, `deployment.airflow.access`, `deployment.config.get`                                                                 |
| `deployment.airflow.hitl_detail.read`      | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.import_error.read`     | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.plugin.read`           | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.pool.create`           | `deployment.airflow.pool.read`, `deployment.airflow.access`, `deployment.config.get`                                                                        |
| `deployment.airflow.pool.delete`           | `deployment.airflow.pool.read`, `deployment.airflow.access`, `deployment.config.get`                                                                        |
| `deployment.airflow.pool.edit`             | `deployment.airflow.pool.read`, `deployment.airflow.access`, `deployment.config.get`                                                                        |
| `deployment.airflow.pool.read`             | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.provider.read`         | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.sla_miss.delete`       | `deployment.airflow.sla_miss.read`, `deployment.airflow.dag.edit`, `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`      |
| `deployment.airflow.sla_miss.edit`         | `deployment.airflow.sla_miss.read`, `deployment.airflow.dag.edit`, `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`      |
| `deployment.airflow.sla_miss.read`         | `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`                                                                         |
| `deployment.airflow.task_instance.create`  | `deployment.airflow.task_instance.read`, `deployment.airflow.dag.edit`, `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get` |
| `deployment.airflow.task_instance.delete`  | `deployment.airflow.task_instance.read`, `deployment.airflow.dag.edit`, `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get` |
| `deployment.airflow.task_instance.edit`    | `deployment.airflow.task_instance.read`, `deployment.airflow.dag.edit`, `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get` |
| `deployment.airflow.task_instance.read`    | `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`                                                                         |
| `deployment.airflow.task_log.read`         | `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`                                                                         |
| `deployment.airflow.task_reschedule.read`  | `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`                                                                         |
| `deployment.airflow.trigger.read`          | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.user`                  | `deployment.config.get`                                                                                                                                     |
| `deployment.airflow.variable.create`       | `deployment.airflow.variable.read`, `deployment.airflow.access`, `deployment.config.get`                                                                    |
| `deployment.airflow.variable.delete`       | `deployment.airflow.variable.read`, `deployment.airflow.access`, `deployment.config.get`                                                                    |
| `deployment.airflow.variable.edit`         | `deployment.airflow.variable.read`, `deployment.airflow.access`, `deployment.config.get`                                                                    |
| `deployment.airflow.variable.read`         | `deployment.airflow.access`, `deployment.config.get`                                                                                                        |
| `deployment.airflow.xcom.create`           | `deployment.airflow.xcom.read`, `deployment.airflow.dag.edit`, `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`          |
| `deployment.airflow.xcom.delete`           | `deployment.airflow.xcom.read`, `deployment.airflow.dag.edit`, `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`          |
| `deployment.airflow.xcom.read`             | `deployment.airflow.dag.read`, `deployment.airflow.access`, `deployment.config.get`                                                                         |
| `deployment.config.delete`                 | `deployment.config.get`                                                                                                                                     |
| `deployment.config.get`                    | None                                                                                                                                                        |
| `deployment.config.update`                 | `deployment.config.get`                                                                                                                                     |
| `deployment.config.upsert`                 | `deployment.config.get`                                                                                                                                     |
| `deployment.dags.push`                     | `deployment.config.get`                                                                                                                                     |
| `deployment.deployRevisions.get`           | `deployment.config.get`                                                                                                                                     |
| `deployment.deployments.config.delete`     | `deployment.config.get`                                                                                                                                     |
| `deployment.deployments.config.update`     | `deployment.config.get`                                                                                                                                     |
| `deployment.deployments.cordon`            | `deployment.config.get`                                                                                                                                     |
| `deployment.images.pull`                   | `deployment.config.get`                                                                                                                                     |
| `deployment.images.push`                   | `deployment.config.get`                                                                                                                                     |
| `deployment.logs.get`                      | `deployment.config.get`                                                                                                                                     |
| `deployment.metrics.get`                   | `deployment.config.get`                                                                                                                                     |
| `deployment.roles.get`                     | `deployment.config.get`                                                                                                                                     |
| `deployment.serviceAccounts.create`        | `deployment.serviceAccounts.get`, `deployment.roles.get`, `deployment.config.get`                                                                           |
| `deployment.serviceAccounts.delete`        | `deployment.serviceAccounts.get`, `deployment.config.get`                                                                                                   |
| `deployment.serviceAccounts.get`           | `deployment.config.get`                                                                                                                                     |
| `deployment.serviceAccounts.update`        | `deployment.serviceAccounts.get`, `deployment.roles.get`, `deployment.config.get`                                                                           |
| `deployment.status.get`                    | `deployment.config.get`                                                                                                                                     |
| `deployment.taskUsage.get`                 | `deployment.config.get`                                                                                                                                     |
| `deployment.teamRoles.update`              | `deployment.teams.get`, `deployment.roles.get`, `deployment.config.get`                                                                                     |
| `deployment.teams.get`                     | `deployment.config.get`                                                                                                                                     |
| `deployment.userRoles.update`              | `deployment.users.get`, `deployment.roles.get`, `deployment.config.get`                                                                                     |
| `deployment.users.get`                     | `deployment.config.get`                                                                                                                                     |
| `deployment.variables.get`                 | `deployment.config.get`                                                                                                                                     |
| `deployment.variables.update`              | `deployment.variables.get`, `deployment.config.get`                                                                                                         |


# Submit a support request
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/support

Get Astro Private Cloud support when you need it.

In addition to product documentation, the following resources are available to help you resolve issues:

* [Astronomer knowledge base](https://support.astronomer.io/hc/en-us)

* [Airflow guides](https://www.astronomer.io/docs/learn/)
  If you're experiencing an issue or have a question that requires Astronomer expertise, you can use one of the following methods to contact Astronomer support:

* Submit a support request on the [Astronomer support portal](https://support.astronomer.io/hc/en-us)

## Best practices for support request submissions

The following are the best practices for submitting support requests in the Astronomer support portal:

### Always indicate priority

To help Astronomer support respond effectively to your support request, it's important that you correctly identify the severity of your issue. The following are the categories that Astronomer uses to determine the severity of your support request:

**P1**: Critical impact. A Deployment is completely unavailable, or a Dag that was previously working in production is no longer working.

P1 tickets are handled with the highest levels of urgency, if Astronomer Support responds on a P1 ticket and subsequently doesn't hear back for 2 hours, the ticket priority will be automatically changed to P2.

**P2**: High impact. Ability to use Astro Private Cloud is severely impaired but doesn't affect critical, previously working pipelines in production.

Examples:

* A newly deployed production Dag isn't working, even though it was tested in another Astro Deployment (for example, dev or test) beforehand.
* The Airflow UI is unavailable.
* You are unable to deploy code to your Deployment, but existing Dags and tasks are running as expected.
* Task logs are missing in the Airflow UI.

**P3**: Medium impact. Service is partially impaired.

Examples:

* A Dag that has been deployed for the first time, or is on a local development environment, is unexpectedly not working.
* There is a bug in the UI.
* Astro CLI usage is impaired (for example, there are incompatibility errors between installed packages).
* There is an Airflow issue that has a code-based solution.
* An error message appeared in the Astro Private Cloud logs.

**P4**: Low impact. Astro Private Cloud is fully usable but you have a question for our team.

Examples:

* There are package incompatibilities caused by a specific, complex use case.
* You have questions about best practices for an action in Airflow or on Astro Private Cloud.
* You have a feature request related to Astro Private Cloud or Airflow.

### Be as descriptive as possible

The more information you can provide about the issue you're experiencing, the quicker Astronomer support can start the troubleshooting and resolution process. When submitting a support request, include the following information:

* Have you made any recent changes to your Deployment or running Dags?
* What solutions have you already tried?
* Is this a problem in more than one Deployment?

### Include logs or code snippets

If you've already copied task logs or Airflow component logs, send them as a part of your request. The more context you can provide, the better.

### Check recommended support articles

If you draft your support ticket on the [Astronomer support portal](https://support.astronomer.io), the portal automatically recommends support articles to you based on the content in your ticket. Astronomer recommends looking through these recommendations to see if your issue has a documented solution before submitting your ticket.

You can also proactively search support articles without submitting a support ticket on the [Astronomer knowledge base](https://support.astronomer.io/hc/en-us).

## Submit a support request on the Astronomer support portal

If you're new to Astronomer, you'll need to create an account on the Astronomer support portal to submit a support request. Astronomer recommends that you use the same email address that you use to access Astro Private Cloud. If you're working with a team and want to view support tickets created by other team members, use your work email or the domain you share with other team members for your account (for example, `@astronomer.io`). If your team uses more than one email domain (for example, `@astronomer.io`), contact Astronomer and ask to have the additional domains added to your organization.

If you're an existing customer, sign in to the [Astronomer support portal](https://support.astronomer.io) and create a new support request.

## Monitor existing support requests

If you've submitted your support request on the Astronomer support portal, sign in to the [Astronomer support portal](https://support.astronomer.io) to:

* Review and comment on requests from your team.
* Monitor the status of all requests in your organization.

<Tip>To add a teammate to an existing support request, cc them when replying on the support ticket email thread.</Tip>


# Version compatibility reference for Astro Private Cloud
Source: https://astronomer.io/docs/astro-private-cloud/v-2-x/version-compatibility-reference

A reference of all adjacent tooling required to run Astro Private Cloud and corresponding version compatibility.

Astro Private Cloud (APC) ships with and requires a number of adjacent technologies that support it, including Kubernetes, Helm, and Apache Airflow itself. This guide provides a reference of all required tools and versions for running APC.

While the tables below reference the minimum compatible versions, Astronomer typically recommends running the latest versions of all tooling if and when possible.

<Tip>Find detailed information about the component image versions, including which versions were used to test a particular APC version in the [APC release index](https://updates.astronomer.io/astronomer-software/releases/index.html).</Tip>

## Astro Private Cloud compatibility reference

* See [Kubernetes version support table and policy](/docs/astro-private-cloud/v-2-x/kubernetes-version-support) for Astro Private Cloud compatibility with Kubernetes.
* See the [Astro Runtime lifecycle schedule](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-lifecycle-schedule) for supported Astro Runtime versions.
* Astronomer recommends using the latest available version of the Astro CLI for all Astro Private Cloud versions. To upgrade from an earlier version of the CLI to the latest, see [Upgrade to Astro CLI version 1.0+](/docs/cli/v1.43/upgrade-cli).

Astronomer doesn't support Postgres versions that are beyond their end-of-life date. You can check the [currently supported versions of Postgres](https://www.postgresql.org/support/versioning/). The following table shows version compatibility information for all currently supported versions of APC. Check [APC lifecycle schedule](/docs/astro-private-cloud/v-2-x/release-lifecycle-policy#astro-private-cloud-lifecycle-schedule) for more information about supported versions of APC.

| Astro Private Cloud version | Supported Postgres versions  | Supported Astro runtime versions                                                            |
| --------------------------- | ---------------------------- | ------------------------------------------------------------------------------------------- |
| 2.0                         | Supported versions up to v17 | All in-support Astro Runtime versions for Airflow 2, and Astro Runtime 3.1-2+ for Airflow 3 |

For more detail about the changes in each APC release, see the [APC Release Notes](/docs/astro-private-cloud/v-2-x/release-notes).

Only Astronomer-distributed images supporting Airflow 2.x are compatible with all versions of APC. Astro Runtime maintenance is independent of Astro Private Cloud maintenance. For more information, see [Astro Runtime maintenance and lifecycle policy](/docs/runtime/runtime-version-lifecycle-policy).

### Kubernetes version support table and policy

In general, APC supports a given version of Kubernetes through its end of life. This includes Kubernetes upstream and cloud-managed variants like GKE, AKS, and EKS. When a version of Kubernetes reaches end of life, support is removed in the next major or minor release of APC. For more information on Kubernetes versioning and release policies, see [Kubernetes Release History](https://kubernetes.io/releases/) or your cloud provider.

See the following table for all supported Kubernetes versions in each maintained version of APC.

| Astro Private Cloud | Kubernetes 1.31 | 1.32 | 1.33 | 1.34 | 1.35 |
| :-----------------: | :-------------: | :--: | :--: | :--: | :--: |
|        2.0.0        |        ✔️       |  ✔️  |  ✔️  |  ✔️  |  ✔️  |
|        2.1.0        |        ✔️       |  ✔️  |  ✔️  |  ✔️  |  ✔️  |


# astro api
Source: https://astronomer.io/docs/cli/v1.45/astro-api

Make authenticated API requests to Astronomer services.

Use `astro api` commands to make authenticated API requests to Astronomer services directly from the Astro CLI.

<CardGroup>
  <Card title="astro api airflow" href="/cli/v1.45/astro-api-airflow">
    View documentation for `astro api airflow`.
  </Card>

  <Card title="astro api cloud" href="/cli/v1.45/astro-api-cloud">
    View documentation for `astro api cloud`.
  </Card>

  <Card title="astro api registry" href="/cli/v1.45/astro-api-registry">
    View documentation for `astro api registry`.
  </Card>
</CardGroup>


# astro api airflow
Source: https://astronomer.io/docs/cli/v1.45/astro-api-airflow

Make authenticated requests to the Airflow REST API.

Make authenticated requests to the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) for a local or deployed Airflow instance. The CLI automatically detects the Airflow version and resolves the OpenAPI specification to provide endpoint discovery.

## Usage

```bash wrap theme={null}
astro api airflow [command] [flags]
```

## Commands

| Command                                | Description                                                                                |
| -------------------------------------- | ------------------------------------------------------------------------------------------ |
| `ls`                                   | List all available Airflow REST API endpoints.                                             |
| `describe <operation-id>`              | Show details about an endpoint, including parameters and a fully expanded response schema. |
| `<operation-id> [request-body-fields]` | Call an endpoint by its operation ID.                                                      |

To call an endpoint on an Astro Deployment instead of your local environment, use the `-d` flag with a Deployment ID.

## Options

| Option                     | Description                                                                   | Possible Values                         |
| -------------------------- | ----------------------------------------------------------------------------- | --------------------------------------- |
| `--generate`               | Print the equivalent `curl` command instead of executing the request.         | None                                    |
| `-d`, `--deployment-id`    | The ID of an Astro Deployment to send the request to.                         | Any valid Deployment ID                 |
| `--jq <filter>`            | Apply a `jq` filter to the response output.                                   | Any valid `jq` expression               |
| `--json`                   | Output the `ls` or `describe` result as JSON instead of a formatted table.    | None                                    |
| `--path-param <key=value>` | Override a path parameter in the request URL.                                 | A key-value pair such as `dagId=my_dag` |
| `--silent`                 | Suppress response output.                                                     | None                                    |
| `--slurp`                  | Collect all paginated results into a single array instead of streaming pages. | None                                    |
| `--template <template>`    | Format the response using a Go template.                                      | Any valid Go template string            |

## Examples

```bash wrap theme={null}
# List all available Airflow REST API endpoints for a local environment
$ astro api airflow ls

# Show details about the get_dags endpoint
$ astro api airflow describe get_dags

# Call the get_health endpoint on a local Airflow instance
$ astro api airflow get_health

# Call the get_dags endpoint on a deployed Airflow instance
$ astro api airflow get_dags -d <deployment-id>

# Call an endpoint and filter the response with jq
$ astro api airflow get_dags --jq '.dags[].dag_id'

# Print the equivalent curl command for a request
$ astro api airflow get_health --generate
```

## Related commands

* [`astro api cloud`](/docs/cli/v1.45/astro-api-cloud)


# astro api cloud
Source: https://astronomer.io/docs/cli/v1.45/astro-api-cloud

Make authenticated requests to the Astro platform API.

Make authenticated requests to the [Astro platform API](https://www.astronomer.io/docs/astro/api/v-1/overview) using the current context's bearer token.

## Usage

```bash wrap theme={null}
astro api cloud [command] [flags]
```

## Commands

| Command                                | Description                                                                                |
| -------------------------------------- | ------------------------------------------------------------------------------------------ |
| `ls`                                   | List all available Astro API endpoints.                                                    |
| `describe <operation-id>`              | Show details about an endpoint, including parameters and a fully expanded response schema. |
| `<operation-id> [request-body-fields]` | Call an endpoint by its operation ID.                                                      |

## Options

| Option                     | Description                                                                                                                                                              | Possible Values                                    |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------- |
| `--generate`               | Print the equivalent `curl` command instead of executing the request.                                                                                                    | None                                               |
| `--jq <filter>`            | Apply a `jq` filter to the response output.                                                                                                                              | Any valid `jq` expression                          |
| `--json`                   | Output the `ls` or `describe` result as JSON instead of a formatted table.                                                                                               | None                                               |
| `--path-param <key=value>` | Override a path parameter in the request URL.                                                                                                                            | A key-value pair such as `organizationId=<org-id>` |
| `--silent`                 | Suppress response output.                                                                                                                                                | None                                               |
| `--slurp`                  | Collect all paginated results into a single array instead of streaming pages.                                                                                            | None                                               |
| `--spec-url`               | An OpenAPI spec source. Supports HTTP URLs as well as local file paths (absolute, relative, `~/`, or `file://`). Local specs are read fresh on every command invocation. | A valid HTTP URL or local file path                |
| `--spec-token-env-var`     | The name of an environment variable that holds a bearer token used to fetch a remote spec. Ignored when `--spec-url` is a local file path.                               | Any valid environment variable name                |
| `--template <template>`    | Format the response using a Go template.                                                                                                                                 | Any valid Go template string                       |

## Examples

```bash wrap theme={null}
# List all available Astro API endpoints
$ astro api cloud ls

# Show details about the list-deployments endpoint
$ astro api cloud describe list-deployments

# List Deployments for an organization
$ astro api cloud list-deployments --path-param organizationId=<org-id>

# Filter the response with jq
$ astro api cloud list-deployments --path-param organizationId=<org-id> --jq '.deployments[].name'

# Print the equivalent curl command for a request
$ astro api cloud list-deployments --path-param organizationId=<org-id> --generate
```

## Related commands

* [`astro api airflow`](/docs/cli/v1.45/astro-api-airflow)


# astro api registry
Source: https://astronomer.io/docs/cli/v1.45/astro-api-registry

Make requests to the Airflow Provider Registry API.

Make HTTP requests to the [Airflow Provider Registry](https://airflow.apache.org/registry) API. No authentication is required.

The argument can be either:

* A path of a registry API endpoint, such as `/providers.json` or `/providers/amazon/modules.json`
* An operation ID from the API spec, such as `listProviders` or `getProviderModulesLatest`

The `/api` prefix is added automatically to paths. Use `astro api registry ls` to discover all available endpoints and operation IDs.

## Usage

```bash wrap theme={null}
astro api registry <endpoint | operation-id> [flags]
```

## Commands

| Command               | Description                                                                                    |
| --------------------- | ---------------------------------------------------------------------------------------------- |
| `ls`                  | List all available registry API endpoints. Supports an optional filter argument                |
| `describe <endpoint>` | Show details about an endpoint, including path parameters and a fully expanded response schema |

## Options

| Option               | Description                                                                                         | Possible Values                              |
| -------------------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------- |
| `-F`, `--field`      | Add a typed parameter in `key=value` format.                                                        | Any valid key-value pair                     |
| `--generate`         | Print the equivalent `curl` command instead of executing the request.                               | None                                         |
| `-H`, `--header`     | Add an HTTP request header in `key:value` format.                                                   | Any valid header                             |
| `-i`, `--include`    | Include HTTP response status line and headers in the output.                                        | None                                         |
| `--input`            | The file to use as the body for the HTTP request. Use `-` for stdin.                                | A valid file path                            |
| `-q`, `--jq`         | Apply a `jq` filter to the response output.                                                         | Any valid `jq` expression                    |
| `--json`             | Output the `ls` or `describe` result as JSON instead of a formatted table.                          | None                                         |
| `-X`, `--method`     | The HTTP method for the request. The default is `GET`.                                              | Any valid HTTP method                        |
| `-p`, `--path-param` | Override a path parameter in `key=value` format. For use with operation IDs.                        | A key-value pair such as `providerId=amazon` |
| `-f`, `--raw-field`  | Add a string parameter in `key=value` format.                                                       | Any valid key-value pair                     |
| `--registry-url`     | Override the registry base URL. Can also be set with the `ASTRO_REGISTRY_URL` environment variable. | A valid URL                                  |
| `--silent`           | Suppress response output.                                                                           | None                                         |
| `-t`, `--template`   | Format the response using a Go template.                                                            | Any valid Go template string                 |
| `--verbose`          | Include full HTTP request and response in the output.                                               | None                                         |

## Examples

```bash wrap theme={null}
# List all available registry API endpoints
astro api registry ls

# Filter endpoints by keyword
astro api registry ls providers

# Query by operation ID with path parameters
astro api registry getProviderModulesLatest -p providerId=amazon

# Query by path
astro api registry /providers.json

# Use jq filter on response
astro api registry /providers.json --jq '.providers[0].id'

# Use Go template for output
astro api registry listProviders \
  --template '{{range .providers}}{{.id}}{{"\n"}}{{end}}'

# Generate curl command instead of executing
astro api registry /providers.json --generate

# Show full request and response details
astro api registry /providers.json --verbose
```

## Related commands

* [`astro api airflow`](/docs/cli/v1.45/astro-api-airflow)
* [`astro api cloud`](/docs/cli/v1.45/astro-api-cloud)


# astro auth
Source: https://astronomer.io/docs/cli/v1.45/astro-auth

Commands for authenticating to Astro or Astro Private Cloud.

Use `astro auth` commands to authenticate to Astro or Astro Private Cloud from the Astro CLI.

<CardGroup>
  <Card title="astro auth login" href="/cli/v1.45/astro-auth-login">
    View documentation for `astro auth login`.
  </Card>

  <Card title="astro auth logout" href="/cli/v1.45/astro-auth-logout">
    View documentation for `astro auth logout`.
  </Card>

  <Card title="astro auth token" href="/cli/v1.45/astro-auth-token">
    View documentation for `astro auth token`.
  </Card>
</CardGroup>


# astro auth login
Source: https://astronomer.io/docs/cli/v1.45/astro-auth-login

Authenticate to Astro or Astro Private Cloud.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change product contexts.</Info>

<Tabs>
  <Tab title="Astro">
    Authenticate to Astro. After you run this command, the CLI prompts you for your login email address. Using the provided email address, the CLI assumes your organization and redirects you to a web browser where you can log in to the Astro UI. After you log in, the CLI automatically recognizes this and authenticates your account.

    If you're running the Astro CLI on a headless system or in an environment without browser access (such as a remote server, Docker container, or CI/CD pipeline), use the `--login-link` or `--token-login` flags to authenticate without needing a local browser.

    ## Usage

    ```sh wrap theme={null}
    astro auth login
    ```

    ## Options

    | Option                | Description                                                                                                                                                                                                                                                                                                                                              | Possible Values                                               |
    | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
    | `-l`, `--login-link`  | Force the CLI to print a login URL instead of automatically opening a browser. Copy the URL and open it on any device that has browser access to complete authentication. This is useful for headless environments, remote servers, or SSH sessions where a browser is not available                                                                     | None                                                          |
    | `-t`, `--token-login` | Authenticate by providing a token directly, without any browser interaction. Generate a token at `cloud.astronomer.io/token` from a device with browser access, then pass it to this flag. This is the recommended approach for fully headless systems and non-interactive environments such as CI/CD pipelines, Docker containers, or automated scripts | A valid authentication token from `cloud.astronomer.io/token` |

    ## Examples

    ```sh wrap theme={null}
    astro auth login
    # The CLI automatically opens the Astro UI in a web browser, which prompts you to log in.

    astro auth login --login-link
    # The CLI prints a login URL instead of opening a browser.
    # Use this on headless systems or SSH sessions: copy the URL to a device with a browser to authenticate.

    astro auth login --token-login <your-token>
    # Authenticate without any browser interaction.
    # Generate a token at cloud.astronomer.io/token and pass it directly.
    # Recommended for CI/CD pipelines, Docker containers, and other non-interactive environments.
    ```
  </Tab>

  <Tab title="APC">
    Authenticate to Astro Private Cloud. After you run this command, the CLI prompts you to either enter a username and password or retrieve an OAuth token from `<basedomain>/token`.

    ## Usage

    ```sh wrap theme={null}
    astro auth login <basedomain>
    ```

    ## Options

    | Option               | Description                                                                              | Possible Values |
    | -------------------- | ---------------------------------------------------------------------------------------- | --------------- |
    | `-l`, `--login-link` | Generate a login link to login on a separate device for cloud CLI login                  | None            |
    | `-o`, `--oauth`      | Skip the prompt for local authentication, proceed directly to OAuth token authentication | None            |

    ## Examples

    ```sh wrap theme={null}
    astro auth login mycompany.astronomer.io
    # The CLI prompts you for a username and password, or to leave the prompt empty for OAuth authentication

    astro auth login mycompany.astronomer.io -o
    # The CLI does not prompt you for a username and password and instead directly prompts you for an OAuth login token
    ```
  </Tab>
</Tabs>

### Related commands

* [`astro auth logout`](/docs/cli/v1.45/astro-auth-logout)
* [`astro auth token`](/docs/cli/v1.45/astro-auth-token)
* [`astro deploy`](/docs/cli/v1.45/astro-deploy)


# astro auth logout
Source: https://astronomer.io/docs/cli/v1.45/astro-auth-logout

Log out of Astro or Astro Private Cloud.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Log out of the Astro CLI. This command does not affect your current web browser login session in either Astro or Astro Private Cloud.

## Usage

```sh wrap theme={null}
astro auth logout
```

## Related commands

* [`astro auth login`](/docs/cli/v1.45/astro-auth-login)
* [`astro auth token`](/docs/cli/v1.45/astro-auth-token)


# astro auth token
Source: https://astronomer.io/docs/cli/v1.45/astro-auth-token

Print the current authentication token.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Print the current authentication token to standard output. The output contains only the raw token value with the `Bearer` prefix stripped, making it suitable for use in scripts and CI/CD pipelines. The command returns an error if you are not authenticated.

## Usage

```sh wrap theme={null}
astro auth token
```

## Options

| Option           | Description                                                                                                                                            | Possible Values         |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- |
| `-d`, `--domain` | Print the token for a specific context domain instead of the current context. Useful for Astro Private Cloud environments with multiple control planes | Any valid domain string |

## Examples

```sh wrap theme={null}
astro auth token
# Print the authentication token for the current context

astro auth token --domain cloud.astronomer.io
# Print the token for a specific context domain

# Use in scripts
export ASTRO_TOKEN=$(astro auth token)
curl -H "Authorization: Bearer $ASTRO_TOKEN" https://api.astronomer.io/...
```

## Related commands

* [`astro auth login`](/docs/cli/v1.45/astro-auth-login)
* [`astro auth logout`](/docs/cli/v1.45/astro-auth-logout)


# astro completion
Source: https://astronomer.io/docs/cli/v1.45/astro-completion

Generate completion scripts.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Generate completion scripts for Astro CLI commands. You can modify the generated scripts and add them to the appropriate directory to customize your command line autocompletion behavior.

For example, you could set automation such that typing `astro lo` autocompletes to `astro login` if you press **TAB** on your keyboard and `astro login` or `astro logout` if you press **TAB** again.

This command is helpful for users interacting with the CLI on a regular basis.

<Info>
  If you're running the CLI MacOS, install [Bash Completion](https://github.com/scop/bash-completion) before creating autocompletion scripts. To do this with Homebrew, run:

  ```sh wrap theme={null}
   brew install bash-completion
  ```
</Info>

## Usage

```sh wrap theme={null}
astro completion <shell>
```

## Options

| Option    | Description                                          | Possible Values                   |
| --------- | ---------------------------------------------------- | --------------------------------- |
| `<shell>` | The type of shell to generate completion scripts for | `bash`,`fish`, `powershell`,`zsh` |

## Example

To generate a shell completion script for zsh, for example, you can run:

```sh wrap theme={null}
$ astro completion zsh > /usr/local/share/zsh/site-functions/_astro
# Completion script saved in your local directory
```

Then, to enable autocompletion, ensure that the following lines are present in your `~/.zshrc` file:

```sh wrap theme={null}
autoload -U compinit
compinit -i
```


# astro config
Source: https://astronomer.io/docs/cli/v1.45/astro-config

Configure Astro CLI behavior.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Use `astro config` commands to configure how the Astro CLI behaves in your local environment.

<CardGroup>
  <Card title="astro config get" href="/cli/v1.45/astro-config-get">
    View documentation for `astro config get`.
  </Card>

  <Card title="astro config set" href="/cli/v1.45/astro-config-set">
    View documentation for `astro config set`.
  </Card>
</CardGroup>


# astro config get
Source: https://astronomer.io/docs/cli/v1.45/astro-config-get

View current Astro CLI configurations.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

View the current configuration of your Astro project as defined in the `.astro/config.yaml` file. The configuration in this file contains details about how your project runs in a local Airflow environment, including your Postgres username and password, your Webserver port, and your project name.

## Usage

Within your Astro project directory, run:

```sh wrap theme={null}
astro config get <option>
```

## Options

| Option           | Description                                                                                                                  | Possible Values |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------- | --------------- |
| `-g`, `--global` | View global CLI configuration settings. See [Set global configurations](/docs/cli/v1.45/configure-cli#set-global-configurations). | None            |

For a list of available configurations, see [Configure the CLI](/docs/cli/v1.45/configure-cli).

## Examples

```sh wrap theme={null}
## View the username for your project's postgres user
$ astro config get postgres.user
```

## Related commands

* [astro config set](/docs/cli/v1.45/astro-config-set)


# astro config set
Source: https://astronomer.io/docs/cli/v1.45/astro-config-set

Update Astro CLI configurations.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Update any part of the current configuration of your Astro project as defined in the `.astro/config.yaml` file. The configuration in this file contains details about how your project runs in a local Airflow environment, including your Postgres username and password, your Webserver port, and your project name.

## Usage

Within your Astro project directory, run:

```sh wrap theme={null}
astro config set <configuration> <value>
```

## Options

| Option           | Description                                                                                                                    | Possible Values |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------ | --------------- |
| `-g`, `--global` | Modify global CLI configuration settings. See [Set global configurations](/docs/cli/v1.45/configure-cli#set-global-configurations). | None            |

For a list of available configurations, see [Configure the CLI](/docs/cli/v1.45/configure-cli).

## Examples

```sh wrap theme={null}
## Set your webserver port to 8081
$ astro config set webserver.port 8081
```

## Related commands

* [astro config get](/docs/cli/v1.45/astro-config-get)


# astro context
Source: https://astronomer.io/docs/cli/v1.45/astro-context

Switch between Astronomer product types.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Use `astro context` commands to switch between Astronomer product types.

<CardGroup>
  <Card title="astro context delete" href="/cli/v1.45/astro-context-delete">
    View documentation for `astro context delete`.
  </Card>

  <Card title="astro context list" href="/cli/v1.45/astro-context-list">
    View documentation for `astro context list`.
  </Card>

  <Card title="astro context switch" href="/cli/v1.45/astro-context-switch">
    View documentation for `astro context switch`.
  </Card>
</CardGroup>


# astro context delete
Source: https://astronomer.io/docs/cli/v1.45/astro-context-delete

Delete an Astronomer product context.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Delete the locally stored information for a given Astronomer installation or base domain. After running this command, the domain for the installation that you specify will no longer appear when you run `astro context list`, and you will not be able to use `astro context switch` to switch to the installation.

If you re-authenticate to an installation that you previously deleted with this command, its information will again be available from `astro context list` and `astro context switch`.

## Usage

```sh wrap theme={null}
astro context delete <basedomain>
```

## Related commands

* [astro context list](/docs/cli/v1.45/astro-context-list)
* [astro context switch](/docs/cli/v1.45/astro-context-switch)


# astro context list
Source: https://astronomer.io/docs/cli/v1.45/astro-context-list

List available Astronomer product contexts.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

View a list of domains for all Astronomer installations that you have access to. An Astronomer installation will appear on this list if you have authenticated to it at least once using `astro login`.

If you're an Astro user, you should only see `astronomer.io` on this list.

## Usage

```sh wrap theme={null}
astro context list
```

## Output

| Output | Description                                                      | Data Type |
| ------ | ---------------------------------------------------------------- | --------- |
| `NAME` | The names of the domains that you have logged into from the CLI. | String    |

## Related commands

* [astro context switch](/docs/cli/v1.45/astro-context-switch)
* [astro context delete](/docs/cli/v1.45/astro-context-delete)


# astro context switch
Source: https://astronomer.io/docs/cli/v1.45/astro-context-switch

Switch to an Astronomer product context.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Switch to a different Astronomer installation. You can switch to a given Astronomer installation only if you have authenticated to it at least once using `astro login`. If you have not authenticated, run `astro login <base-domain>` instead.

Note that after switching to a different Astronomer installation, you might have to re-authenticate to the installation using `astro login`.

## Usage

```sh wrap theme={null}
astro context switch <basedomain>
```

## Related commands

* [astro context list](/docs/cli/v1.45/astro-context-list)
* [astro context delete](/docs/cli/v1.45/astro-context-delete)


# astro dbt
Source: https://astronomer.io/docs/cli/v1.45/astro-dbt

Manage connected dbt projects.

<Info>
  This command is only available on Astro.
</Info>

Use `astro dbt` to manage dbt projects that you connect to Astro.

<CardGroup>
  <Card title="astro dbt cleanup" href="/cli/v1.45/astro-dbt-cleanup">
    View documentation for `astro dbt cleanup`.
  </Card>

  <Card title="astro dbt delete" href="/cli/v1.45/astro-dbt-delete">
    View documentation for `astro dbt delete`.
  </Card>

  <Card title="astro dbt deploy" href="/cli/v1.45/astro-dbt-deploy">
    View documentation for `astro dbt deploy`.
  </Card>
</CardGroup>


# astro dbt cleanup
Source: https://astronomer.io/docs/cli/v1.45/astro-dbt-cleanup

Remove Cosmos Boost pre-deploy artifacts from a dbt project.

Remove Cosmos Boost artifacts from one or more paths in your local project. Use this command after you disable the [Cosmos Boost pre-deploy step](/docs/cli/v1.45/configure-cli), or to clear artifacts left by an earlier deploy.

`astro dbt cleanup` only removes `.astro/dbt_metadata.json` and `.astro/manifest.slim.json` files that the Cosmos Boost pre-deploy step generated. It doesn't remove any other files under `.astro/`.

## Usage

```bash wrap theme={null}
astro dbt cleanup <path> <options>
```

## Options

| Option   | Description                                                             | Possible Values                                           |
| -------- | ----------------------------------------------------------------------- | --------------------------------------------------------- |
| `<path>` | One or more paths to remove Cosmos Boost artifacts from. Default is `.` | Any valid filepath. Repeat to specify more than one path. |

## Examples

To remove Cosmos Boost artifacts from the current directory:

```bash wrap theme={null}
astro dbt cleanup
```

To remove Cosmos Boost artifacts from specific paths:

```bash wrap theme={null}
astro dbt cleanup dbt/project-a dbt/project-b
```

## Related commands

* [`astro dbt deploy`](/docs/cli/v1.45/astro-dbt-deploy)
* [`astro dbt delete`](/docs/cli/v1.45/astro-dbt-delete)


# astro dbt delete
Source: https://astronomer.io/docs/cli/v1.45/astro-dbt-delete

Delete a dbt project from an Astro Deployment.

<Info>
  This command is only available on Astro.
</Info>

Delete a dbt project from a Deployment on Astro. This command deletes a dbt project from the Airflow environments where you deployed it.

When you run `astro dbt delete`, you are prompted to select from a list of Deployments that you can access in your Workspace. You can bypass this prompt and specify a Deployment name or ID in the command. To retrieve a Deployment ID, open your Deployment in the Astro UI and copy the value in the **ID** section of the Deployment page. You can also run `astro deployment list` to find a Deployment ID or name.

<Info>To complete this action, [Workspace Owner](/docs/astro/user-permissions#workspace-roles) permissions are required.</Info>

## Usage

```bash wrap theme={null}
astro dbt delete <your-deployment-id> <options>
```

## Options

| Option                    | Description                                                                                                                                 | Possible Values                                                                |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| `-n`, `--deployment-name` | Name of the Deployment to delete your dbt project from                                                                                      | Any valid Deployment name                                                      |
| `--description`           | Description of the project to store on the deploy                                                                                           | String                                                                         |
| `-m`, `--mount-path`      | Path describing where the dbt project you want to delete is mounted in Airflow. Default path is `/usr/local/airflow/dbt/<dbt-project-name>` | Any valid path except `/usr/local/airflow/dags`, which is used for dag deploys |
| `-p`, `--project-path`    | Path to the dbt project that you want to delete. The default is your current directory                                                      | Any valid filepath to a dbt project                                            |
| `-w`, `--wait`            | Wait for the Deployment to become healthy before ending the delete command. Default is `False`                                              | `True` or `False`                                                              |
| `--workspace-id`          | The Workspace ID for the Deployment from which you want to delete your dbt project                                                          | Any valid Workspace ID                                                         |

## Examples

To delete a dbt project from a specific Deployment:

```bash wrap theme={null}
astro dbt delete clyxf6ivz000008jth73k37r6
```

To delete a project that has the mount path, `/usr/local/airflow/dbt/test-dbt-project` from a specific Deployment.

```bash wrap theme={null}
astro dbt delete clyxf6ivz000008jth73k37r6 --mount-path="/usr/local/airflow/dbt/test-dbt-project"
```

## Related commands

* [`astro dbt deploy`](/docs/cli/v1.45/astro-dbt-deploy)
* [`astro deployment list`](/docs/cli/v1.45/astro-deployment-list)


# astro dbt deploy
Source: https://astronomer.io/docs/cli/v1.45/astro-dbt-deploy

Deploy a dbt project to an Astro Deployment.

<Info>
  This command is only available on Astro.
</Info>

This command allows you to deploy your dbt code directly to Astro, independently of any dag deploys or full project image deploys.

This command bundles all files in your dbt project and pushes them to Astro, where they are mounted on your Airflow containers so that your dags can access them.

When you run `astro dbt deploy`, the CLI prompts you to select from a list of all Deployments that you can access across Workspaces. To bypass this prompt, you can also specify a Deployment ID in the command. To retrieve a Deployment ID, open your Deployment in the Astro UI and copy the value in the **ID** section of the Deployment page. You can also run `astro deployment list` to find a Deployment ID or name.

## Usage

```bash wrap theme={null}
astro dbt deploy <your-deployment-id> <options>
```

## Options

| Option                    | Description                                                                                                                                                   | Possible Values                                                                 |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| `-n`, `--deployment-name` | Name of the Deployment to deploy your dbt project to                                                                                                          | Any valid Deployment name                                                       |
| `--description`           | Description of the project to store on the deploy                                                                                                             | String                                                                          |
| `-m`, `--mount-path`      | Path describing where to mount the dbt project in Airflow, so that it is accessible to your dags. Default path is `/usr/local/airflow/dbt/<dbt-project-name>` | Any valid path except  `/usr/local/airflow/dags`, which is used for dag deploys |
| `-p`, `--project-path`    | Path to the dbt project that you want to deploy. The default is your current directory                                                                        | Any valid filepath to a dbt project                                             |
| `-w`, `--wait`            | Wait for the Deployment to become healthy before ending the deploy command. Default is `False`                                                                | `True` or `False`                                                               |
| `--wait-time`             | Time to wait for the Deployment to become healthy before ending the command. Can only be used with `--wait=true`.                                             | A time duration amount, such as `10s` or `1m11s`.                               |
| `--workspace-id`          | The Workspace ID for the Deployment where you want to deploy your dbt project                                                                                 | Any valid Workspace ID                                                          |

## Examples

To deploy directly to a specific Deployment:

```bash wrap theme={null}
astro dbt deploy clyxf7lfs000108jtcs4b59nm
```

to deploy a project to a specific deployment on the mount path, `example-path`.

```bash wrap theme={null}
astro dbt deploy clyxf7lfs000108jtcs4b59nm --mount-path="/usr/local/airflow/dbt/test-dbt-project"
```

## Related commands

* [`astro dbt delete`](/docs/cli/v1.45/astro-dbt-delete)
* [`astro deployment list`](/docs/cli/v1.45/astro-deployment-list)

## See also

* Webinar: [Introducing Cosmos: The Easy Way to Run dbt Models in Airflow](https://www.astronomer.io/events/webinars/introducing-cosmos-the-east-way-to-run-dbt-models-in-airflow/).
* Demo: [See how to deploy your dbt projects to Astro](https://www.astronomer.io/dbt-demo/)


# astro deploy
Source: https://astronomer.io/docs/cli/v1.45/astro-deploy

Deploy a local project to an Astro Deployment.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change between product contexts.</Info>

<Tabs>
  <Tab title="Astro">
    [Deploy code](/docs/astro/deploy-code) to a Deployment on Astro.

    This command bundles all files in your Astro project and pushes them to Astro. Before completing the process, it tests your dags in your Astro project for parse errors. If this test fails, the deploy to Astro will also fail. This is the same test which runs locally with `astro dev parse`.

    When you run `astro deploy`, the CLI prompts you to select from a list of all Deployments that you can access across Workspaces. To bypass this prompt, you can also specify a Deployment ID in the command. To retrieve a Deployment ID, open your Deployment in the Astro UI and copy the value in the **ID** section of the Deployment page. You can also run `astro deployment list` to find a Deployment ID or name.

    For teams operating at scale, this command can be automated with a [CI/CD pipeline](/docs/astro/set-up-ci-cd) by using [Deployment API tokens](/docs/astro/deployment-api-tokens) in the request. When `ASTRO_API_TOKEN` is specified as OS-level environment variables on your local machine or in a CI tool, `astro deploy <deployment-id>` can be run without requiring user authentication.

    <Tip>
      To skip the parsing process before deploys, complete one of the following setups:

      * Add `skip_parse: true` to `.astro/config.yaml` in your Astro project.
      * Add `ASTRONOMER_SKIP_PARSE=true` as an environment variable to your local environment or CI/CD pipeline.
    </Tip>

    ## Usage

    ```sh wrap theme={null}
    astro deploy <options>
    ```

    ## Options

    | Option                    | Description                                                                                                                                                                              | Possible Values                                                                                                                          |
    | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
    | `<deployment-id>`         | Specifies the Deployment to deploy to and bypasses the Deployment selection prompt                                                                                                       | Any valid Deployment ID                                                                                                                  |
    | `--build-secret`          | Run `docker build --secret` to mount a secret value to your Docker image. Repeat the flag to mount more than one secret. Replaces the deprecated `--build-secrets` flag.                 | `id=<your-secret-id>, src=<path-to-secret> .` See [Docker documentation](https://docs.docker.com/build/building/secrets/#secret-mounts). |
    | `--description`           | A description for your code deploy. Descriptions appear in the Astro UI in your Deployment's **Deploy History**                                                                          | None                                                                                                                                     |
    | `-d`, `--dags`            | Deploy only your `dags` directory. See [dag-only deploys](/docs/astro/deploy-dags)                                                                                                            | None                                                                                                                                     |
    | `--dags-path`             | Deploy Dags from this path instead of the `dags` directory in your working directory. Can be combined with `--dags` to deploy only Dags without requiring an Astro project directory.    | Any valid filepath                                                                                                                       |
    | `-e`, `--env`             | Location of the file containing environment variables for pytests. By default, this is `.env`.                                                                                           | Any valid filepath to an `.env` file                                                                                                     |
    | `-f`, `--force`           | Force deploy even if your project contains parse errors, uncommitted changes, or missing Dags. Parse tests are defined in `.astro/test_dag_integrity_default.py`.                        | None                                                                                                                                     |
    | `-i`, `--image-name`      | The name of a pre-built custom Docker image to use with your project. The image must be available from a Docker registry hosted on your local machine                                    | A valid name for a pre-built Docker image based on Astro Runtime                                                                         |
    | `--image`                 | If you have dags-only deploys enabled, use this flag to deploy only your Astro project image. When you use this option, your `dags` folder is not deployed to Astro                      | None                                                                                                                                     |
    | `-n`, `--deployment-name` | The name of the Deployment to deploy to. Use as an alternative to `<deployment-id>`.                                                                                                     | Any valid Deployment name                                                                                                                |
    | `--no-dags-base-dir`      | When deploying with `--dags`, place DAG files at the bundle root instead of in a `dags/` folder prefix. Use with Airflow 3 Deployments where the DAG bundle root is added to `sys.path`. | None                                                                                                                                     |
    | `-p`, `--prompt`          | Force the Deployment selection prompt even if a Deployment ID is specified                                                                                                               | None                                                                                                                                     |
    | `--pytest`                | Deploy code to Astro only if the pytests are passed. By default the pytests are read from the `tests` directory.                                                                         | None                                                                                                                                     |
    | `-s`, `--save`            | Save the current Deployment and working directory combination for future deploys.                                                                                                        | None                                                                                                                                     |
    | `-t`, `--test`            | The filepath to an alternative pytest file or directory.                                                                                                                                 | Valid filepath within your Astro project                                                                                                 |
    | `-w`, `--wait`            | Wait for the Deployment to become healthy before completing the command.                                                                                                                 | None                                                                                                                                     |
    | `--wait-time`             | Time to wait for the Deployment to become healthy before ending the command. Can only be used with `--wait=true`.                                                                        | A time duration amount, such as `10s` or `1m11s`.                                                                                        |
    | `--workspace-id <string>` | In the prompt to select a Deployment, only show Deployments within this Workspace.                                                                                                       | Any valid Workspace ID                                                                                                                   |

    ## Examples

    To deploy directly to a specific Deployment:

    ```bash wrap theme={null}
    astro deploy ckvvfp9tf509941drl4vela81n
    ```

    To configure the Astro CLI to use a given Deployment and directory as a default for future deploys:

    ```bash wrap theme={null}
    astro deploy ckvvfp9tf509941drl4vela81n --save
    ```

    To use a custom Docker image from your local Docker registry to build your Astro project:

    ```bash wrap theme={null}
    astro deploy --image-name your-custom-runtime-image
    ```

    To deploy only dags from your Astro project to a specific Deployment:

    ```bash wrap theme={null}
    astro deploy ckvvfp9tf509941drl4vela81n --dags
    ```
  </Tab>

  <Tab title="APC">
    [Deploy code](/docs/astro/deploy-code) to a Deployment on Astro Private Cloud.

    This command bundles all files in your Astro project and pushes them to Astro Private Cloud.

    When you run `astro deploy`, you'll be prompted to select from a list of all Deployments that you can access in all Workspaces. To bypass this prompt, you can specify a Deployment ID in the command. To retrieve a Deployment ID, go to your Deployment's information page in the Astro UI and copy the value after the last `/` in the URL. You can also run `astro deployment list` to retrieve a Deployment ID .

    ## Options

    | Option                    | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                     | Possible Values                                                                                                                          |
    | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
    | `--build-secret`          | Run `docker build --secret` to mount a secret value to your Docker image. Repeat the flag to mount more than one secret. Replaces the deprecated `--build-secrets` flag.                                                                                                                                                                                                                                                                                                        | `id=<your-secret-id>, src=<path-to-secret> .` See [Docker documentation](https://docs.docker.com/build/building/secrets/#secret-mounts). |
    | `--description <string>`  | Improve traceability by attaching a description to a code deploy. If you don't provide a description, the system automatically assigns a default description based on the deploy type.                                                                                                                                                                                                                                                                                          | Any string surrounded by quotations                                                                                                      |
    | `-d`, `--dags`            | Deploy only your `dags` directory. Works only if dag-only deploys are enabled for the Deployment.                                                                                                                                                                                                                                                                                                                                                                               | None                                                                                                                                     |
    | `<deployment-id>`         | Specifies the Deployment to deploy to and bypasses the Deployment selection prompt. Required for dag-only deploys.                                                                                                                                                                                                                                                                                                                                                              | Any valid Deployment ID                                                                                                                  |
    | `-f`,`--force`            | Force deploy even if your project contains errors or uncommitted changes                                                                                                                                                                                                                                                                                                                                                                                                        | None                                                                                                                                     |
    | `-i`, `--image-name`      | The name of a pre-built custom Docker image to use with your project, based on Astro Runtime. Building the image is skipped. By default, the image must be available on your local machine so the CLI can push it to your Deployment's registry. If used with `--remote`, the CLI instead points the Deployment directly at the image already in a remote registry and skips pushing it — in that case the image doesn't need to be local, but `--runtime-version` is required. | A valid name for a pre-built Docker image based on Astro Runtime                                                                         |
    | `--image`                 | Use this flag to deploy only your Astro project image. When you use this option, your `dags` folder is not deployed to Astro. This flag only works for dag-only, Git-sync-based and NFS-based Deployments.                                                                                                                                                                                                                                                                      | None                                                                                                                                     |
    | `--no-cache`              | Do not use any images from the container engine's cache when building your project                                                                                                                                                                                                                                                                                                                                                                                              | None                                                                                                                                     |
    | `-p`,`--prompt`           | Force the Deployment selection prompt even if a Deployment ID is specified                                                                                                                                                                                                                                                                                                                                                                                                      | None                                                                                                                                     |
    | `--remote`                | Directly point the deployment to the remote image and skip pushing the image, if `--image-name` is passed.                                                                                                                                                                                                                                                                                                                                                                      | None                                                                                                                                     |
    | `--runtime-version`       | Specify Runtime version of your image, if `--image-name` is passed.                                                                                                                                                                                                                                                                                                                                                                                                             | Valid Runtime version                                                                                                                    |
    | `-s`,`--save`             | Save the current Deployment and working directory combination for future deploys                                                                                                                                                                                                                                                                                                                                                                                                | None                                                                                                                                     |
    | `--workspace-id <string>` | In the prompt to select a Deployment, only show Deployments within this Workspace                                                                                                                                                                                                                                                                                                                                                                                               | Any valid Workspace ID                                                                                                                   |

    ## Examples

    ```sh wrap theme={null}
    # List of Deployments appears
    $ astro deploy

    # Deploy directly to a specific Deployment
    $ astro deploy ckvvfp9tf509941drl4vela81n

    # The CLI automatically selects this Deployment for your Astro project
    $ astro deploy ckvvfp9tf509941drl4vela81n --save

    # Deploy an image from my private registry without first building it/putting in on my local computer
    $ astro deploy --image-name your-custom-runtime-image --remote --runtime-version=rt_version

    ```
  </Tab>
</Tabs>

<Info>
  The following error can sometimes occur when the CLI tries to build your Astro Runtime image using Podman:

  ```bash wrap theme={null}
  WARN[0010] SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format
  ```

  You can resolve this issue by exporting the `BUILDAH_FORMAT` [environment variable](/docs/astro/environment-variables) to Podman:

  ```dockerfile wrap theme={null}
  export BUILDAH_FORMAT=docker
  ```
</Info>

<Info>
  If you use Docker Desktop, ensure that the [**Use containerd for pulling and storing images**](https://docs.docker.com/desktop/containerd/#turn-on-the-containerd-image-store-feature) setting is turned off. Otherwise, you might receive errors when you run `astro deploy` such as:

  ```text wrap theme={null}
  Push access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  # or
  Unable to find image 'barren-ionization-0185/airflow:latest' locally
  Error response from daemon: pull access denied for barren-ionization-0185/airflow, repository does not exist or may require 'docker login'
  ```
</Info>

## Related commands

* [`astro login`](/docs/cli/v1.45/astro-login)
* [`astro deployment list`](/docs/cli/v1.45/astro-deployment-list)
* [`astro dev parse`](/docs/cli/v1.45/astro-dev-parse)


# Get started with Airflow using the Astro CLI
Source: https://astronomer.io/docs/cli/v1.45/get-started-cli

Create an Airflow project and run it locally on your computer in just a few minutes.

With the Astro CLI, you can run Airflow on your local machine. Follow this quickstart to build an Airflow project from the [Learning Airflow](https://github.com/astronomer/templates/tree/main/learning-airflow) template and run it in a local Airflow environment with just a few commands. At the end of the tutorial, you'll have all of the files and components you need to develop and test Airflow dags locally.

## Step 1: Install the CLI

<Tabs>
  <Tab title="Mac">
    ```text wrap theme={null}
    brew install astro
    ```
  </Tab>

  <Tab title="Windows with winget">
    ```text wrap theme={null}
    winget install -e --id Astronomer.Astro
    ```
  </Tab>

  <Tab title="Linux">
    ```text wrap theme={null}
    curl -sSL install.astronomer.io | sudo bash -s
    ```
  </Tab>
</Tabs>

## Step 2: Create an Astro project

An *Astro project* contains the set of files necessary to run Airflow, including dedicated folders for your dag files, plugins, and dependencies. This set of files builds an image that you can run both on your local machine with Airflow and deploy to Astro.

Use [`astro dev init`](/docs/cli/v1.45/astro-dev-init) with the `--from-template` flag to create the project based off of the [Learning Airflow template](https://github.com/astronomer/templates/tree/main/learning-airflow).

```sh wrap theme={null}
astro dev init --from-template learning-airflow
```

This command generates all of the project files you need to run Airflow locally, including an example dag that you can run out of the box. Different [templates](https://github.com/astronomer/templates/tree/main) generate different example dags. See [Create an Astro project](/docs/cli/v1.45/develop-project#create-an-astro-project) for more information about the default project structure.

## Step 3: Run Airflow locally

Running your project locally allows you to test your dags before you deploy them to a production environment. While this step is not required for deploying and running your code on Astro, Astronomer recommends always using the Astro CLI to test locally before deploying.

1. To start running your project in a local Airflow environment, run the following command from the `learning-airflow` project directory:

<Tabs>
  <Tab title="Docker / Podman">
    ```sh wrap theme={null}
    astro dev start
    ```

    This command builds your project and spins up 4 containers on your machine, each for a different Airflow component:

    * **Postgres:** Airflow's metadata database
    * **Webserver:** The Airflow component responsible for rendering the Airflow UI
    * **Scheduler:** The Airflow component responsible for monitoring and triggering tasks
    * **Triggerer:** The Airflow component responsible for running Triggers and signaling tasks to resume when their conditions have been met. The triggerer is used exclusively for tasks that are run with [deferrable operators](/docs/learn/deferrable-operators)
  </Tab>

  <Tab title="Standalone (no Docker)">
    ```sh wrap theme={null}
    astro dev start --standalone
    ```

    This command runs Airflow directly on your machine in a virtual environment, without Docker or Podman. To make standalone mode the default for your project, run:

    ```sh wrap theme={null}
    astro config set dev.mode standalone
    ```

    See [astro dev start](/docs/cli/v1.45/astro-dev-start) for all available options.
  </Tab>
</Tabs>

<Note>If you don't have Docker installed, you can use standalone mode to run Airflow directly on your machine in a virtual environment instead. Run `astro dev start --standalone`, or set it as the default for your project with `astro config set dev.mode standalone`. See [astro dev start](/docs/cli/v1.45/astro-dev-start) for all available options.</Note>

2. After your project builds successfully, open the Airflow UI in your web browser at `https://localhost:8080/`.

3. Find your dags in the`dags` directory in the Airflow UI.

   In this directory, you can find an example dag, `example-astronauts`, which was generated with your Astro project. To provide a basic demonstration of an ETL pipeline, this dag shows a simple ETL pipeline example that queries the list of astronauts currently in space from the Open Notify API and prints a statement for each astronaut. The dag uses the TaskFlow API to define tasks in Python, and dynamic task mapping to dynamically print a statement for each astronaut.

<Info>The Astro CLI uses port `8080` for the Airflow webserver and port `5432` for the Airflow metadata database by default. If these ports are already in use on your local computer, an error message might appear. To resolve this error message, see [Run Airflow locally](/docs/cli/v1.45/troubleshoot-locally#ports-are-not-available-for-my-local-airflow-webserver).</Info>

## Step 4: Develop locally with the CLI

Now that you have a locally running project, you can start to develop your Astro project by adding dags, dependencies, environment variables, and more. See [Develop your project](/docs/cli/v1.45/develop-project) for more details on how to modify all aspects of your Astro project.

Most changes you make, including updates to your dag code, are applied automatically to your running environment and don't require rebuilding your project. However, you must rebuild your project and restart your environment to apply changes from any of the following files in your Astro project:

* `packages.txt`
* `Dockerfile`
* `requirements.txt`
* `airflow_settings.yaml`

To restart your local Airflow environment, run:

```sh wrap theme={null}
astro dev restart
```

This command rebuilds your project and restarts your local Airflow environment. Alternatively, you can run `astro dev stop` to stop your environment without restarting, then run `astro dev start` when you want to restart.

## Next Steps

After you have finished Getting Started with the CLI, you can configure your CLI to locally debug your Airflow environment, authenticate to cloud services to test your dags with data stored on the cloud, or you can learn more about developing dags with Astro.

* [Configure the CLI](/docs/cli/v1.45/configure-cli)
* [Authenticate to cloud services](/docs/cli/v1.45/authenticate-to-clouds)
* [Build and run a project locally](/docs/cli/v1.45/run-airflow-locally)


# Astro CLI
Source: https://astronomer.io/docs/cli/v1.45/overview

A reference for all available Astro command-line interface (CLI) commands and settings.

The Astro CLI is the command line interface for data orchestration. It's the easiest way to get started with Apache Airflow and can be used with all Astronomer products.

<Info>
  **Install the CLI**

  To install with Homebrew, run:

  ```sh wrap theme={null}
  brew install astro
  ```

  For alternative installation steps, see [Install the Astro CLI](/docs/cli/v1.45/install-cli).
</Info>

The Astro CLI is open source and built for data practitioners everywhere. The binary is maintained in the public [Astro CLI GitHub repository](https://github.com/astronomer/astro-cli), where pull requests and GitHub issues are welcome.

## Astro CLI features

With the Astro CLI, you can:

* Run Airflow on your local machine in minutes.
* Parse, debug, and test dags in a dedicated testing environment.
* Manage your Astro resources, including Workspaces and Deployments.

## Get started

<CardGroup>
  <Card title="Astro CLI Quickstart" icon="rocket-launch" href="/cli/v1.45/get-started-cli">
    Start a new Airflow project with just a few commands.
  </Card>

  <Card title="Install the CLI" icon="cloud-arrow-down" href="/cli/v1.45/install-cli">
    Instructions for installing, upgrading, and uninstalling the Astro command-line interface (CLI).
  </Card>

  <Card title="Release notes" icon="notes" href="/cli/v1.45/release-notes">
    Review the latest changes to the Astro CLI.
  </Card>

  <Card title="Command reference" icon="book-open-cover" href="/cli/v1.45/reference">
    Learn about all available Astro CLI commands.
  </Card>
</CardGroup>


# Astro CLI command reference
Source: https://astronomer.io/docs/cli/v1.45/reference

A reference for all available Astro command-line interface (CLI) commands and settings.

This document contains information about all commands and settings available in the Astro CLI, including examples and flags. To get started with the Astro CLI, see [Get Started](/docs/cli/v1.45/install-cli).

All reference documentation is based on the latest available version of the Astro CLI. To see the differences across various CLI versions, see the [Astro CLI Release Notes](/docs/cli/v1.45/release-notes).

The Astronomer product you're using determines the behavior and format of commands. The documentation identifies commands that are specific to Astro or Astro Private Cloud, or when specific behavior is product dependent.

## Core Commands

* [`astro login`](/docs/cli/v1.45/astro-login)
* [`astro dev init`](/docs/cli/v1.45/astro-dev-init)
* [`astro dev start`](/docs/cli/v1.45/astro-dev-start)
* [`astro dev restart`](/docs/cli/v1.45/astro-dev-restart)
* [`astro deploy`](/docs/cli/v1.45/astro-deploy)
* [`astro remote`](/docs/cli/v1.45/astro-remote)

## Global Options

### Global flags

The Astro CLI has the following global flags that can be used with any command:

* `-h`, `--help`: Output more information about a given command to the CLI.
* `--verbosity <string>`: Specify the log level to expose for each CLI command. Possible values are `debug`, `info`, `warn`, `error`, `fatal`, and `panic`.

### Global configuration options

* `g`, `--global`: You set global configurations for the Astro CLI by running `astro config --global`. See [Configure the CLI](/docs/cli/v1.45/configure-cli) for more information.


# Astro Private Cloud: Upgrade Runtime
Source: https://astronomer.io/docs/runtime/manage-airflow-versions

Adjust and upgrade Airflow versions on Astro Private Cloud.

## Overview

Regularly upgrading the [Astro Runtime](/docs/runtime/overview) version of your deployments on Astro Private Cloud ensures that you continue to be supported and that your Airflow deployments have the latest features and functionality.

To upgrade your Airflow Deployment to a later version of Airflow:

* Select a new Airflow version with the APC UI or CLI to start the upgrade.
* Change the `FROM` statement in your project's `Dockerfile` to reference an Astro Runtime image that corresponds to your current Airflow version.
* Deploy your upgrade to Astronomer.

## Available Astronomer image versions

A cron job automatically pulls new Astronomer image versions from the Astronomer [update service](http://updates.astronomer.io/) and adds them to the APC UI and CLI within 24 hours of their publication. You don't have to upgrade Astronomer to upgrade Airflow.

If you don't want to wait for new Astronomer image versions, you can manually trigger the cron job with the following Kubernetes command:

```bash wrap theme={null}
kubectl create job --namespace astronomer --from=cronjob/astronomer-houston-update-airflow-check airflow-update-check-first-run
```

If you get a message indicating that a job already exists, delete the job and rerun the command.

## Upgrade considerations

Consider the following when you upgrade Astro Runtime:

* Astronomer supports rollbacks within Astro Runtime 3.x (3.1.0 and later) and from Astro Runtime 3.x to 2.x (2.3.0 and later) on Astro Private Cloud 1.1 and later. Astro Runtime 3.0.x versions are not supported for automated rollbacks. See [Roll back a deploy](/docs/astro-private-cloud/v-1-x/deploy-rollbacks).
* All versions of the Astro CLI support all versions of Astro Runtime. There are no dependencies between the two products.
* Upgrading to certain versions of Runtime might result in extended upgrade times or otherwise disruptive changes to your environment. To learn more, see [Version-specific upgrade considerations](/docs/runtime/version-upgrade-considerations).

To stay up to date on the latest versions of Astro Runtime, see [Astro Runtime release notes](/docs/runtime/runtime-release-notes). For more information on Astro Runtime versioning and support, see [Astro Runtime versioning and lifecycle policy](/docs/runtime/runtime-version-lifecycle-policy). For a full collection of Astro Runtime Docker images, go to the [Astro Runtime repository on Quay.io](https://quay.io/repository/astronomer/astro-runtime?tab=tags).

## Step 1: Review upgrade considerations

Astro Runtime upgrades can include breaking changes, especially when you're upgrading to a new major version. Check the [upgrade considerations](/docs/runtime/version-upgrade-considerations) for your upgrade version to anticipate any breaking changes or upgrade-specific instructions before you proceed.

## Step 2: (Optional) Start the upgrade process

Now, by default, Astro Private Cloud uses the Runtime version that you configure in your Dockerfile in **Step 5: Update your Astro Project**. Prior to version 0.36, by default, you must first specify the Airflow and Runtime version you wanted to upgrade to, before starting your upgrade process.

If you want to use the prior workflow, you must change the feature flag, `disableDesiredRuntimeVersion`, in your Houston API `values.yaml` configuration to `false`:

```yaml wrap theme={null}
astronomer:
    houston:
        config:
            deployments:
                disableDesiredRuntimeVersion: false

```

After you apply the configuration, you can start the upgrade process. Starting the upgrade process doesn't interrupt or otherwise impact your Airflow Deployment. It only signals to Astro Private Cloud that you intend to upgrade at a later time.

### With the APC UI

1. Go to **Deployment** > **Settings** > **Basics** > **Runtime Version**.
2. Select an [Astro Runtime](/docs/runtime/overview) version. Each version shows the Airflow version it's based on.
3. Click **Upgrade**.

### With the Astro CLI

1. Run `astro login <base-domain>` to confirm you're authenticated.

2. Run the following command to list your current Deployments.

   ```bash wrap theme={null}
   astro deployment list
   ```

   Copy the ID of the Deployment you want to upgrade.

3. Run the following command to list the available Airflow versions:

   ```bash wrap theme={null}
   astro deployment airflow upgrade --deployment-id=<deployment-id>
   ```

4. Enter the Airflow version you want to upgrade to and press `Enter`.

<a />

## Step 3: (Optional) Pin provider package versions

Major Astro Runtime upgrades can include significant upgrades to built-in provider packages. These package upgrades can sometimes include breaking changes for your dags. See the [Apache Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers/packages-ref.html) for a list of all available provider packages and their release notes.

For the most stable upgrade path, Astronomer recommends pinning all provider package versions from your current Runtime version before upgrading.

1. Run the following command to check the version of all provider packages installed in your Astro Runtime version:

   ```bash wrap theme={null}
   docker run --rm quay.io/astronomer/astro-runtime:<current-runtime-version> pip freeze | grep apache-airflow-providers
   ```

2. After reviewing this list, pin the version for each provider package in your Astro project `requirements.txt` file. For example, Runtime 7.4.1 uses version 4.0.0 of `apache-airflow-providers-databricks`. To pin this version of the Databricks provider package when you upgrade to a later version of Runtime, you add the following line to your `requirements.txt` file:

   ```text wrap theme={null}
   apache-airflow-providers-databricks==4.0.0
   ```

## Step 4: (Optional) Run upgrade tests with the Astro CLI

You can use the Astro CLI to anticipate and address problems before upgrading to a newer version of Astro Runtime. Before you upgrade, run the following command to run tests against the version of Astro Runtime you're upgrading to:

```bash wrap theme={null}
astro dev upgrade-test --runtime-version <upgraded-runtime-version>
```

The Astro CLI then generates test results in your Astro project that identify dependency conflicts and import errors that you would experience using the new Astro Runtime version. Review these results and make the recommended changes to reduce the risk of your project generating errors after you upgrade. For more information about using this command and the test results, see [Test before upgrade your Astro project](/docs/cli/v1.43/test-your-astro-project-locally#test-before-an-astro-runtime-upgrade).

## Step 5: Update your Astro project

1. In your Astro project, open your `Dockerfile`.
2. Change the Docker image in the `FROM` statement of your `Dockerfile` to a new version of Astro Runtime. For example, to upgrade to the latest version of Runtime, you would change the `FROM` statement in your Dockerfile to:

   ```dockerfile wrap theme={null}
   FROM quay.io/astronomer/astro-runtime:13.2.0
   ```

   For a list of supported Astro Runtime versions, see [Astro Runtime maintenance and lifecycle policy](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-lifecycle-schedule).

<Warning>Rollbacks are only supported on Astro Private Cloud 1.1 and later, for Astro Runtime 3.1.0+ and 2.3.0+. Astro Runtime 3.0.x versions require manual intervention. See [Roll back a deploy](/docs/astro-private-cloud/v-1-x/deploy-rollbacks).</Warning>

1. Save the changes to your `Dockerfile`.

## Step 6: Test Astro Runtime locally

Astronomer recommends testing new versions of Astro Runtime locally to ensure that Airflow starts as expected before upgrading your Deployment on Astro.

1. Open your project directory in your terminal and run `astro dev restart`. This restarts the Docker containers for the Airflow webserver, scheduler, triggerer, and Postgres metadata database.

2. Access the Airflow UI of your local environment by navigating to `http://localhost:8080` in your browser.

3. Confirm that your local upgrade was successful by scrolling to the bottom of any page. You should see your new Astro Runtime version in the footer as well as the version of Airflow it is based on.

   <Frame>
     <img alt="Runtime Version banner - Local" />
   </Frame>

4. (Optional) Run DAGs locally to ensure that all of your code works as expected. If you encounter errors after your upgrade, it's possible that your new Astro Runtime version includes a breaking provider package change. If you experience one of these breaking changes, follow the steps in [Upgrade or pin provider package versions](#step-3-optional-pin-provider-package-versions) to check your provider package versions and, if required, pin the provider package version from your previous Runtime version in your `requirements.txt` file.

## Step 7: Deploy to Astronomer

1. Run the following command to push your upgraded Astro project to your Deployment:

   ```bash wrap theme={null}
   astro deploy
   ```

2. In the APC UI, open your Deployment and click **Open Airflow**.

3. In the Airflow UI, scroll to the bottom of any page. You should see your new Runtime version in the footer.

## Cancel Airflow upgrade

As a System Admin, you can cancel an Airflow Deployment upgrade at any time if you haven't yet changed the Astronomer Runtime image in your `Dockerfile` and deployed it.

In the APC UI, select **Cancel** next to **Airflow Version**.

Using the Astro CLI, run:

```bash wrap theme={null}
astro deployment airflow upgrade --cancel --deployment-id=<deployment-id>
```

For example, if you cancel an upgrade from Airflow 2.1.0 to Airflow 2.2.0 in the CLI, the following message appears:

```bash wrap theme={null}
Airflow upgrade process has been successfully canceled. Your Deployment was not interrupted and you are still running Airflow 2.1.0.
```

Canceling the Airflow upgrade process does not interrupt or otherwise impact your Airflow Deployment or code that's running.

<Info>
  If you can't cancel your upgrade and receive an error message about using an unsupported Airflow version, set the following value in your `values.yaml` file and [apply the change](/docs/astro-private-cloud/v-2-x/apply-platform-config) to successfully cancel your upgrade. This configuration allows you to roll back to your current version of Airflow, even if it's not supported.

  ```yaml wrap theme={null}
  astronomer:
    houston:
      config:
        deployments:
          enableSystemAdminCanCreateDeprecatedAirflows: true
  ```
</Info>


# Astronomer Runtime overview
Source: https://astronomer.io/docs/runtime/overview

Overview of Astronomer Runtime - the distribution of Apache Airflow that powers Astro.

Astro Runtime is Astronomer's managed distribution of Airflow. It provides a prebuilt, secure environment with Airflow, core providers, and dependencies bundled into standardized images, so you can focus on building and running pipelines rather than maintaining infrastructure.

Astronomer Runtime includes exclusive features, timely support for new Apache Airflow versions, and enhanced functionality designed specifically for production data orchestration environments.

## Astronomer Runtime features

With Astronomer Runtime, you get:

* **Production-ready distribution**: A carefully curated and tested distribution of Apache Airflow with enhanced stability and performance.
* **Exclusive features**: Smart task concurrency defaults, high availability configurations, and custom UI enhancements.
* **Data lineage**: Built-in OpenLineage integration for comprehensive data lineage tracking across your workflows.

## Get started

<CardGroup>
  <Card title="Runtime architecture" icon="building" href="/runtime/runtime-image-architecture">
    Learn about Astronomer Runtime architecture, versioning, and Docker image types.
  </Card>

  <Card title="Provider reference" icon="puzzle-piece" href="/runtime/runtime-provider-reference">
    View the provider packages and versions included in each Runtime release.
  </Card>

  <Card title="Release notes" icon="notes" href="/runtime/runtime-release-notes">
    Review the latest changes and features in Astronomer Runtime releases.
  </Card>

  <Card title="Lifecycle policy" icon="calendar-clock" href="/runtime/runtime-version-lifecycle-policy">
    Understand Runtime maintenance windows, support policies, and version lifecycle.
  </Card>
</CardGroup>


# Astro Runtime architecture
Source: https://astronomer.io/docs/runtime/runtime-image-architecture

Reference documentation for Astro Runtime, a differentiated distribution of Apache Airflow.

Astro Runtime is a production ready, data orchestration tool based on Apache Airflow that is distributed as a container image and is required by all Astronomer products. It is intended to provide organizations with improved functionality, reliability, efficiency, and performance.

Deploying Astro Runtime is a requirement if your organization is using Astro. Astro Runtime includes the following features:

* Timely support for new patch, minor, and major versions of Apache Airflow. This includes bug fixes that have not been released by the open source project but are backported to Astro Runtime and available to users earlier.
* Exclusive features to enrich the task execution experience, including smart task concurrency defaults and high availability configurations.
* Built-in lineage capabilities with [OpenLineage Airflow provider package](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) enabling data lineage features on Astro. [OpenLineage](https://openlineage.io/) standardizes the definition of data lineage, the metadata that forms lineage metadata, and how data lineage metadata is collected from external systems. See [OpenLineage and Airflow](/docs/learn/airflow-openlineage).
* A custom Airflow UI that includes links to Astronomer resources and exposes the currently running the image tag in the footer of all UI pages.
* A custom logging module that ensures Airflow task logs are reliably available to the Astro data plane. (*Astro only*).
* A custom security manager that enforces user roles and permissions as defined by Astro. (*Astro only*).

For more information about the features that are available in Astro Runtime releases, see the [Astro Runtime release notes](/docs/runtime/runtime-release-notes).

## Runtime versioning

Astro Runtime versions are released regularly and follow [semantic versioning](https://semver.org). For Airflow 3 versions of Astro Runtime, Astronomer uses the syntax `major.minor-patch`, including a hyphen between minor and patch, to avoid confusion with the corresponding Airflow `major.minor` version because Astro Runtime patch releases can include early-access fixes. For Airflow 2, the format is `major.minor.patch`.

* **Major** versions are released for significant feature additions. This includes new major or minor versions of Apache Airflow, as well as API or dag specification changes that are not backward compatible.
* **Minor** versions are released for functional changes. This includes API or dag specification changes that are backward compatible, which might include new minor versions of `astronomer-providers`.
* **Patch** versions are released for bug and security fixes that resolve unwanted behavior. This includes new patch versions of Apache Airflow and `astronomer-providers`.

Every version of Astro Runtime correlates to an Apache Airflow version. All Deployments must run only one version of Astro Runtime, but you can run different versions of Astro Runtime on different Deployments within a given cluster or Workspace.

For a list of supported Astro Runtime versions and more information on the Astro Runtime maintenance policy, see [Astro Runtime versioning and lifecycle policy](/docs/runtime/runtime-version-lifecycle-policy).

### Astro Runtime and Apache Airflow parity

This table lists Astro Runtime releases and their associated Apache Airflow versions.

| Astro Runtime | Apache Airflow version |
| ------------- | ---------------------- |
| 6             | 2.4                    |
| 7             | 2.5                    |
| 8             | 2.6                    |
| 9             | 2.7                    |
| 10            | 2.8                    |
| 11            | 2.9                    |
| 12            | 2.10                   |
| 13            | 2.11                   |
| 3.0           | 3.0                    |
| 3.1           | 3.1                    |
| 3.2           | 3.2                    |

For version compatibility information, see the [Runtime release notes](/docs/runtime/runtime-release-notes).

## Provider packages

The Astro Runtime 3.0 and more recent versions includes the following pre-installed open source provider packages. Providers marked with an asterisk (\*) are also installed by default on open source Apache Airflow. See [Runtime Provider Reference](/docs/runtime/runtime-provider-reference) for a complete list.

* Celery [`apache-airflow-providers-celery`](https://pypi.org/project/apache-airflow-providers-celery/)
* Common Compat [`apache-airflow-providers-common-compat`](https://pypi.org/project/apache-airflow-providers-common-compat/)\*
* Common IO [`apache-airflow-providers-common-io`](https://pypi.org/project/apache-airflow-providers-common-io/)\*
* Common SQL [`apache-airflow-providers-common-sql`](https://pypi.org/project/apache-airflow-providers-common-sql/)\*
* Elasticsearch [`apache-airflow-providers-elasticsearch`](https://pypi.org/project/apache-airflow-providers-elasticsearch/)
* MySQL [`apache-airflow-providers-mysql`](https://pypi.org/project/apache-airflow-providers-mysql/)
* OpenLineage [`apache-airflow-providers-openlineage`](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html)
* PostgreSQL (Postgres) [`apache-airflow-providers-postgres`](https://pypi.org/project/apache-airflow-providers-postgres/)
* SMTP [`apache-airflow-providers-smtp`](https://pypi.org/project/apache-airflow-providers-smtp/)\*
* Standard [`apache-airflow-providers-standard`](https://pypi.org/project/apache-airflow-providers-standard/)\*

### Provider package versioning

If an Astro Runtime release includes changes to an installed version of a provider package that is maintained by Astronomer (e.g., `astronomer-providers`), the version change is documented in the [Astro Runtime release notes](/docs/runtime/runtime-release-notes).

To determine the version of any provider package installed in your current Astro Runtime image, run:

```text wrap theme={null}
docker run --rm <runtime-image> pip freeze | grep <provider>
```

For example, to find the version of the current package for Astronomer providers, run the following command:

```text wrap theme={null}
astro dev bash pip freeze | grep astronomer-providers
```

## Python versioning

| Astro Runtime | Apache Airflow version | Default Python version | Supported Python versions |
| ------------- | ---------------------- | ---------------------- | ------------------------- |
| 6             | 2.4                    | 3.9                    | 3.9                       |
| 7             | 2.5                    | 3.9                    | 3.9                       |
| 8             | 2.6                    | 3.10                   | 3.10                      |
| 9             | 2.7                    | 3.11                   | 3.9 - 3.11                |
| 10            | 2.8                    | 3.11                   | 3.9 - 3.11                |
| 11            | 2.9                    | 3.11                   | 3.9 - 3.11                |
| 12            | 2.10                   | 3.12                   | 3.10 - 3.12               |
| 13            | 2.11                   | 3.12                   | 3.10 - 3.12               |
| 3.0           | 3.0                    | 3.12                   | 3.11 - 3.12               |
| 3.1           | 3.1                    | 3.12                   | 3.11 - 3.12               |
| 3.2           | 3.2                    | 3.13                   | 3.12 - 3.14               |
| 3.3           | 3.3                    | 3.14                   | 3.12 - 3.14               |

Starting with Astro Runtime 9, if you require a different version of Python than what's included in the base distribution, you can use a Python distribution of Astro Runtime. See [Image types](#image-types).

If you're running Astro Runtime 6.0 (based on Airflow 2.4) to Runtime 8, Astronomer recommends that you use the `ExternalPythonOperator` to run different Python versions in Airflow. See [`ExternalPythonOperator`](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#externalpythonoperator).

If you're currently using the `KubernetesPodOperator` or the `PythonVirtualenvOperator` in your dags, you can continue to use them to create virtual or isolated environments that can run tasks with different versions of Python.

### Python version considerations

Starting with Python version 3.12, Python has removed the module `imp`. This can cause errors for your dags if you Astro Runtime version 12 or higher, which supports Python Version 3.12 or higher.

See [How do I migrate from `imp`](https://discuss.python.org/t/how-do-i-migrate-from-imp/27885) guidance from Python for remediation steps.

## Postgres version compatibility

The following table shows which versions Postgres are compatible with each version of Astro Runtime. Note that Postgres versioning is handled automatically on Astro.

| Astro Runtime | Apache Airflow version | Postgres versions |
| ------------- | ---------------------- | ----------------- |
| 6             | 2.4                    | 10-13             |
| 7             | 2.5                    | 11-15             |
| 8             | 2.6                    | 11-15             |
| 9             | 2.7                    | 11-15             |
| 10            | 2.8                    | 12-16             |
| 11            | 2.9                    | 12-16             |
| 12            | 2.10                   | 12-16             |
| 13            | 2.11                   | 12-16             |
| 3.0           | 3.0                    | 13-17             |
| 3.1           | 3.1                    | 13-17             |
| 3.2           | 3.2                    | 14-18             |

## Executors

In Airflow, the executor is responsible for determining how and where a task is completed.

In all local environments created with the Astro CLI, Astro Runtime runs the [Local executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/local.html). On Astro and Astronomer Software, you can use the Astro executor, Celery executor, or the Kubernetes executor.

## Image tag conventions

Astro Runtime image tags encode the Runtime version, optional OS variant, optional Python version, and optional base designation into a single string. The general format differs between Airflow 3.x and 2.x.

**Airflow 3.x:**

```text wrap theme={null}
astrocrpublic.azurecr.io/runtime:<major.minor>-<patch>[-<os-variant>][-python-<python-version>][-base]
```

**Airflow 2.x:**

```text wrap theme={null}
astrocrpublic.azurecr.io/astronomer/astro-runtime:<major.minor.patch>[-<os-variant>][-python-<python-version>][-slim][-base]
```

All segments after the version are optional. When multiple segments are present, they must appear in the order shown above.

| Segment        | Description                                                                                                                                                      | Example                          |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- |
| Version        | Required. The Astro Runtime version.                                                                                                                             | `3.1-14` (3.x) or `13.2.0` (2.x) |
| OS variant     | Optional. The operating system variant. Omit for the default (Debian). Currently the only option is `ubi9` (Up to Runtime 3.1) or `-ubi` (Runtime 3.2 and later) | `-ubi9`/`-ubi`                   |
| Python version | Optional. Overrides the default Python version for that Runtime release.                                                                                         | `-python-3.11`                   |
| Slim           | Optional. Airflow 2.x only. Selects the slim image with fewer pre-installed providers. All 3.x images are slim by default.                                       | `-slim`                          |
| Base           | Optional. Selects the base image without ONBUILD commands.                                                                                                       | `-base`                          |

### Floating tags

For Airflow 3.x, you can use a tag without a patch version to always pull the most recent patch release within that minor version. For example, `3.1` resolves to the latest patch of Runtime 3.1 with the default Python version, `3.1-base` resolves to the latest patch base image, and `3.1-ubi9` resolves to the latest patch RHEL UBI 9 image.

For production deployments, Astronomer recommends pinning a specific patch version for reproducibility.

### Tag examples

| Tag                                 | Description                                                        |
| ----------------------------------- | ------------------------------------------------------------------ |
| `3.1-14`                            | Runtime 3.1 patch 14, Debian, default Python, with ONBUILD         |
| `3.1`                               | Latest patch of Runtime 3.1, Debian, default Python, with ONBUILD  |
| `3.2-1-ubi`                         | Runtime 3.2 patch 1, RHEL UBI 10, default Python, with ONBUILD     |
| `3.1-14-python-3.11`                | Runtime 3.1 patch 14, Debian, Python 3.11, with ONBUILD            |
| `3.1-14-ubi9-python-3.11-base`      | Runtime 3.1 patch 14, RHEL UBI 9, Python 3.11, base image          |
| `13.2.0`                            | Runtime 13 (Airflow 2.x), Debian, default Python, with ONBUILD     |
| `13.2.0-ubi9-python-3.11-slim-base` | Runtime 13 (Airflow 2.x), RHEL UBI 9, Python 3.11, slim base image |

## Image types

Astro Runtime is distributed by default as a Debian-based container image. An Astro Runtime image must be specified in the `Dockerfile` of your Astro project.

### Container registry URLs

Starting with Airflow 3.x, Runtime images are hosted at a new registry:

```text wrap theme={null}
astrocrpublic.azurecr.io/runtime:3.1-5
```

Airflow 2.x images are still available under the original registry:

```text wrap theme={null}
quay.io/astronomer/astro-runtime:13.2.0
```

but can also be pulled from the new domain as an alternative:

```text wrap theme={null}
astrocrpublic.azurecr.io/astronomer/astro-runtime:13.2.0
```

You can modify this image tag in your Astro project Dockerfile to use different versions of Astro Runtime. The following sections explain each image type and how to specify them.

### Base images

The base Astro Runtime Docker image has the following format:

**Airflow 3.x:**

```text wrap theme={null}
astrocrpublic.azurecr.io/runtime:<version>-base
```

**Airflow 2.x:**

```text wrap theme={null}
astrocrpublic.azurecr.io/astronomer/astro-runtime:<version>-base
```

A `base` Astro Runtime image is recommended for complex use cases that require additional customization, such as [installing Python packages from private sources](/docs/cli/v1.43/private-python-packages).

For all other cases, Astronomer recommends using non-`base` images, which incorporate ONBUILD commands that copy and scaffold your Astro project directory so you can more easily pass those files to the images running each core Airflow component.

### Python version images

Starting with Astro Runtime 9, Astronomer maintains different Astro Runtime images for each [supported Python version](#python-versioning).
Python version images have the following format:

**Airflow 3.x:**

```text wrap theme={null}
astrocrpublic.azurecr.io/runtime:<runtime-version>-python-<python-version>
```

**Airflow 2.x:**

```text wrap theme={null}
astrocrpublic.azurecr.io/astronomer/astro-runtime:<runtime-version>-python-<python-version>
```

### Operating system variant images

Starting with Astro Runtime 11, Astronomer offers images built on RHEL UBI as an alternative to the default Debian-based images. RHEL-based images are useful for organizations with compliance or security requirements that mandate a Red Hat-certified base.

To select a RHEL UBI image, add `-ubi9` (Airflow 3.1 and earlier) or `-ubi` (Airflow 3.2 and later) after the runtime version in your image tag. If you omit the OS variant segment, you get the Debian-based image (the default).

**Airflow 3.2 and later:**

```text wrap theme={null}
astrocrpublic.azurecr.io/runtime:<version>-ubi
```

**Airflow 3.0 and 3.1:**

```text wrap theme={null}
astrocrpublic.azurecr.io/runtime:<version>-ubi9
```

**Airflow 2.x:**

```text wrap theme={null}
astrocrpublic.azurecr.io/astronomer/astro-runtime:<version>-ubi9
```

You can combine the OS variant with other image type segments. For example, to use a RHEL UBI 9 base image with Python 3.11:

**Airflow 3.2 and later:**

```text wrap theme={null}
astrocrpublic.azurecr.io/runtime:3.2-1-ubi-python-3.13-base
```

**Airflow 3.0 and 3.1:**

```text wrap theme={null}
astrocrpublic.azurecr.io/runtime:3.1-14-ubi9-python-3.11-base
```

**Airflow 2.x:**

```text wrap theme={null}
astrocrpublic.azurecr.io/astronomer/astro-runtime:13.2.0-ubi9-python-3.11-base
```

UBI is currently the only alternative OS variant. Additional variants may be offered in the future.

For availability details by Runtime version, see [Operating system support](#operating-system-support).

### Slim images

Starting with Astro Runtime 11, Astronomer maintains a slim Astro Runtime image. Slim Astro Runtime images only include the dependencies required for the basic functionality of Astro. Providers marked with an asterisk (\*) are also installed by default on open source Apache Airflow. The providers installed in the slim image are:

* Common IO [`apache-airflow-providers-common-io`](https://pypi.org/project/apache-airflow-providers-common-io/)\*
* Common SQL [`apache-airflow-providers-common-sql`](https://pypi.org/project/apache-airflow-providers-common-sql/)\*
* FAB [`apache-airflow-providers-fab`](https://pypi.org/project/apache-airflow-providers-fab/)\*
* FTP [`apache-airflow-providers-ftp`](https://pypi.org/project/apache-airflow-providers-ftp/)\*
* HTTP [`apache-airflow-providers-http`](https://pypi.org/project/apache-airflow-providers-http/)\*
* IMAP [`apache-airflow-providers-imap`](https://pypi.org/project/apache-airflow-providers-imap/)\*
* SMTP [`apache-airflow-providers-smtp`](https://pypi.org/project/apache-airflow-providers-smtp/)\*
* SQLite [`apache-airflow-providers-sqlite`](https://pypi.org/project/apache-airflow-providers-sqlite/)\*
* Celery [`apache-airflow-providers-celery`](https://pypi.org/project/apache-airflow-providers-celery/)
* Elasticsearch [`apache-airflow-providers-elasticsearch`](https://pypi.org/project/apache-airflow-providers-elasticsearch/)
* MySQL [`apache-airflow-providers-mysql`](https://pypi.org/project/apache-airflow-providers-mysql/)
* PostgreSQL (Postgres) [`apache-airflow-providers-postgres`](https://pypi.org/project/apache-airflow-providers-postgres/)
* `astronomer-kubernetes-executor`

<Note>
  **Provider packages in Astro Runtime 3.0 and higher**

  Astro Runtime for 3.0 and higher includes only the provider packages required to run on Astro. All Airflow 3.x images are considered "slim" and do not contain extra providers. If you need additional providers, you must explicitly add them to your `requirements.txt` file.
</Note>

Use the slim Astro Runtime image if you want faster local builds and deploys, smaller footprint for security vulnerabilities and dependency conflicts, or you don't require the packages included in the default Astro Runtime distribution.

The slim Astro Runtime image has the following format:

**Airflow 2.x:**

```text wrap theme={null}
astrocrpublic.azurecr.io/astronomer/astro-runtime:<version>-slim
```

### Combine image types

Image types are additive, meaning that you can combine multiple type segments in your image tag to specify an image with multiple variations. Segments must appear in the order defined in [Image tag conventions](#image-tag-conventions).

**Airflow 2.x:**

```text wrap theme={null}
astrocrpublic.azurecr.io/astronomer/astro-runtime:<version>[-<os-variant>][-python-<python-version>][-slim][-base]
```

You can add or remove any optional segments as needed. For example, to use the RHEL UBI 9 base image with Python 3.11 on Airflow 3.x:

```text wrap theme={null}
astrocrpublic.azurecr.io/runtime:3.1-14-ubi9-python-3.11
```

For Airflow 2.x, to use the slim base image with Python 3.11 on Debian:

```text wrap theme={null}
astrocrpublic.azurecr.io/astronomer/astro-runtime:13.0.0-python-3.11-slim-base
```

The `-slim` segment applies only to Airflow 2.x. All Airflow 3.x images include only the providers required to run on Astro by default.

## Operating system support

The following table lists the operating systems and architectures supported by each Astro Runtime version.

| Astro Runtime | Apache Airflow version | Operating System (OS)                | Architecture    |
| ------------- | ---------------------- | ------------------------------------ | --------------- |
| 6             | 2.4                    | Debian 11.5 (bullseye)               | AMD64 and ARM64 |
| 7             | 2.5                    | Debian 11.5 (bullseye)               | AMD64 and ARM64 |
| 8             | 2.6                    | Debian 11.7 (bullseye)               | AMD64 and ARM64 |
| 9             | 2.7                    | Debian 11.7 (bullseye)               | AMD64 and ARM64 |
| 10            | 2.8                    | Debian 11.8 (bullseye)               | AMD64 and ARM64 |
| 11            | 2.9                    | Debian 11.8 (bullseye), RHEL UBI 9¹² | AMD64 and ARM64 |
| 12            | 2.10                   | Debian 12 (bookworm), RHEL UBI 9¹³   | AMD64 and ARM64 |
| 13            | 2.11                   | Debian 12 (bookworm), RHEL UBI 9     | AMD64 and ARM64 |
| 3.0           | 3.0                    | Debian 12 (bookworm), RHEL UBI 9     | AMD64 and ARM64 |
| 3.1           | 3.1                    | Debian 12 (bookworm), RHEL UBI 9     | AMD64 and ARM64 |
| 3.2           | 3.2                    | Debian 13 (trixie), RHEL UBI 10      | AMD64 and ARM64 |

¹ RHEL UBI 9 images are officially supported starting with Astro Runtime 12.5.0 in the 12.x series, and available experimentally starting with Astro Runtime 11.13.0 in the 11.x series. \
² Astro Runtime 11.13.0 and later support RHEL UBI 9. \
³ Astro Runtime 12.3.0 and later support RHEL UBI 9.

Astro Runtime 6.0.4 and later images are multi-arch and support AMD64 and ARM64 processor architectures for local development. Docker automatically uses the correct processor architecture based on the computer you are using.

## Related documentation

* [Astro Runtime release notes](/docs/runtime/runtime-release-notes)
* [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime)
* [Upgrade Astronomer Software Runtime](/docs/runtime/manage-airflow-versions)
* [Astro Runtime versioning and lifecycle policy](/docs/runtime/runtime-version-lifecycle-policy)


# Astro Runtime provider package reference
Source: https://astronomer.io/docs/runtime/runtime-provider-reference

View the provider packages and versions included in each release of Astro Runtime

This page is a reference of the provider packages included in each release of Astro Runtime.

<Tip>
  To find the version of a provider package installed in an Astro Runtime image, run:

  ```sh wrap theme={null}
  docker run --rm <runtime-image> pip freeze | grep <provider>
  ```

  For example, to find the version of Celery, run the following command:

  ```sh wrap theme={null}
  docker run --rm quay.io/astronomer/astro-runtime:12.0.0 pip freeze | grep apache-airflow-providers-celery
  ```
</Tip>

## Astro Runtime 3.3-5

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.23.1  |
| apache-airflow-providers-common-compat | 1.18.0  |
| apache-airflow-providers-common-io     | 1.8.0   |
| apache-airflow-providers-common-sql    | 2.1.0   |
| apache-airflow-providers-elasticsearch | 6.9.0   |
| apache-airflow-providers-openlineage   | 2.20.0  |
| apache-airflow-providers-smtp          | 3.0.3   |
| apache-airflow-providers-standard      | 1.17.0  |
| astronomer-providers-logging           | 1.6.8   |

## Astro Runtime 3.3-4

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.23.1  |
| apache-airflow-providers-common-compat | 1.18.0  |
| apache-airflow-providers-common-io     | 1.8.0   |
| apache-airflow-providers-common-sql    | 2.1.0   |
| apache-airflow-providers-elasticsearch | 6.9.0   |
| apache-airflow-providers-openlineage   | 2.20.0  |
| apache-airflow-providers-smtp          | 3.0.3   |
| apache-airflow-providers-standard      | 1.17.0  |
| astronomer-providers-logging           | 1.6.8   |

## Astro Runtime 3.3-3

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.23.1  |
| apache-airflow-providers-common-compat | 1.18.0  |
| apache-airflow-providers-common-io     | 1.8.0   |
| apache-airflow-providers-common-sql    | 2.1.0   |
| apache-airflow-providers-elasticsearch | 6.9.0   |
| apache-airflow-providers-openlineage   | 2.20.0  |
| apache-airflow-providers-smtp          | 3.0.3   |
| apache-airflow-providers-standard      | 1.17.0  |
| astronomer-providers-logging           | 1.6.8   |

## Astro Runtime 3.3-2

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.21.0  |
| apache-airflow-providers-common-compat | 1.15.0  |
| apache-airflow-providers-common-io     | 1.8.0   |
| apache-airflow-providers-common-sql    | 2.0.1   |
| apache-airflow-providers-elasticsearch | 6.7.0   |
| apache-airflow-providers-openlineage   | 2.18.1  |
| apache-airflow-providers-smtp          | 3.0.1   |
| apache-airflow-providers-standard      | 1.15.0  |
| astronomer-providers-logging           | 1.6.7   |

## Astro Runtime 3.3-1

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.21.0  |
| apache-airflow-providers-common-compat | 1.15.0  |
| apache-airflow-providers-common-io     | 1.8.0   |
| apache-airflow-providers-common-sql    | 2.0.1   |
| apache-airflow-providers-elasticsearch | 6.7.0   |
| apache-airflow-providers-openlineage   | 2.18.1  |
| apache-airflow-providers-smtp          | 3.0.1   |
| apache-airflow-providers-standard      | 1.15.0  |
| astronomer-providers-logging           | 1.6.7   |

## Astro Runtime 3.2-6

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.22.0  |
| apache-airflow-providers-common-compat | 1.16.0  |
| apache-airflow-providers-common-io     | 1.8.0   |
| apache-airflow-providers-common-sql    | 1.36.0  |
| apache-airflow-providers-elasticsearch | 6.8.0   |
| apache-airflow-providers-openlineage   | 2.19.0  |
| apache-airflow-providers-smtp          | 3.0.2   |
| apache-airflow-providers-standard      | 1.16.0  |
| astronomer-providers-logging           | 1.6.7   |

## Astro Runtime 3.2-5

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.20.0  |
| apache-airflow-providers-common-compat | 1.15.0  |
| apache-airflow-providers-common-io     | 1.7.2   |
| apache-airflow-providers-common-sql    | 1.36.0  |
| apache-airflow-providers-elasticsearch | 6.5.4   |
| apache-airflow-providers-openlineage   | 2.17.0  |
| apache-airflow-providers-smtp          | 3.0.1   |
| apache-airflow-providers-standard      | 1.13.1  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.2-4

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.18.0  |
| apache-airflow-providers-common-compat | 1.14.3  |
| apache-airflow-providers-common-io     | 1.7.2   |
| apache-airflow-providers-common-sql    | 1.34.0  |
| apache-airflow-providers-elasticsearch | 6.5.2   |
| apache-airflow-providers-openlineage   | 2.14.0  |
| apache-airflow-providers-smtp          | 2.4.5   |
| apache-airflow-providers-standard      | 1.12.3  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.2-3

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.18.0  |
| apache-airflow-providers-common-compat | 1.14.3  |
| apache-airflow-providers-common-io     | 1.7.2   |
| apache-airflow-providers-common-sql    | 1.34.0  |
| apache-airflow-providers-elasticsearch | 6.5.2   |
| apache-airflow-providers-openlineage   | 2.14.0  |
| apache-airflow-providers-smtp          | 2.4.5   |
| apache-airflow-providers-standard      | 1.12.3  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.2-2

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.18.0  |
| apache-airflow-providers-common-compat | 1.14.3  |
| apache-airflow-providers-common-io     | 1.7.2   |
| apache-airflow-providers-common-sql    | 1.34.0  |
| apache-airflow-providers-elasticsearch | 6.5.2   |
| apache-airflow-providers-openlineage   | 2.14.0  |
| apache-airflow-providers-smtp          | 2.4.5   |
| apache-airflow-providers-standard      | 1.12.3  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.2-1

<Warning>
  **Restricted release**

  Astro Runtime 3.2-1 was restricted from use on April 16, 2026, after its initial release because of an issue where environment manager connections are not found. See [Restricted Runtime versions](/docs/runtime/runtime-version-lifecycle-policy#restricted-runtime-versions).
</Warning>

## Astro Runtime 3.1-19

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.23.0  |
| apache-airflow-providers-common-compat | 1.17.0  |
| apache-airflow-providers-common-io     | 1.8.0   |
| apache-airflow-providers-common-sql    | 1.36.0  |
| apache-airflow-providers-elasticsearch | 6.8.1   |
| apache-airflow-providers-openlineage   | 2.19.0  |
| apache-airflow-providers-smtp          | 3.0.2   |
| apache-airflow-providers-standard      | 1.16.0  |
| astronomer-providers-logging           | 1.6.8   |

## Astro Runtime 3.1-18

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.22.0  |
| apache-airflow-providers-common-compat | 1.16.0  |
| apache-airflow-providers-common-io     | 1.8.0   |
| apache-airflow-providers-common-sql    | 1.36.0  |
| apache-airflow-providers-elasticsearch | 6.8.0   |
| apache-airflow-providers-openlineage   | 2.19.0  |
| apache-airflow-providers-smtp          | 3.0.2   |
| apache-airflow-providers-standard      | 1.16.0  |
| astronomer-providers-logging           | 1.6.8   |

## Astro Runtime 3.1-17

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.20.0  |
| apache-airflow-providers-common-compat | 1.15.0  |
| apache-airflow-providers-common-io     | 1.7.2   |
| apache-airflow-providers-common-sql    | 1.36.0  |
| apache-airflow-providers-elasticsearch | 6.5.4   |
| apache-airflow-providers-openlineage   | 2.17.0  |
| apache-airflow-providers-smtp          | 3.0.1   |
| apache-airflow-providers-standard      | 1.13.1  |
| astronomer-providers-logging           | 1.6.6   |

## Astro Runtime 3.1-16

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.20.0  |
| apache-airflow-providers-common-compat | 1.15.0  |
| apache-airflow-providers-common-io     | 1.7.2   |
| apache-airflow-providers-common-sql    | 1.36.0  |
| apache-airflow-providers-elasticsearch | 6.5.4   |
| apache-airflow-providers-openlineage   | 2.17.0  |
| apache-airflow-providers-smtp          | 3.0.1   |
| apache-airflow-providers-standard      | 1.13.1  |
| astronomer-providers-logging           | 1.6.5   |

## Astro Runtime 3.1-15

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.19.0  |
| apache-airflow-providers-common-compat | 1.14.3  |
| apache-airflow-providers-common-io     | 1.7.2   |
| apache-airflow-providers-common-sql    | 1.36.0  |
| apache-airflow-providers-elasticsearch | 6.5.3   |
| apache-airflow-providers-openlineage   | 2.16.0  |
| apache-airflow-providers-smtp          | 2.4.5   |
| apache-airflow-providers-standard      | 1.13.0  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-14

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.17.1  |
| apache-airflow-providers-common-compat | 1.14.1  |
| apache-airflow-providers-common-io     | 1.7.1   |
| apache-airflow-providers-common-sql    | 1.33.0  |
| apache-airflow-providers-elasticsearch | 6.5.0   |
| apache-airflow-providers-openlineage   | 2.12.0  |
| apache-airflow-providers-smtp          | 2.4.3   |
| apache-airflow-providers-standard      | 1.12.1  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-13

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.16.0  |
| apache-airflow-providers-common-compat | 1.13.1  |
| apache-airflow-providers-common-io     | 1.7.1   |
| apache-airflow-providers-common-sql    | 1.31.0  |
| apache-airflow-providers-elasticsearch | 6.4.4   |
| apache-airflow-providers-openlineage   | 2.10.2  |
| apache-airflow-providers-smtp          | 2.4.2   |
| apache-airflow-providers-standard      | 1.11.1  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-12

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.15.2  |
| apache-airflow-providers-common-compat | 1.13.0  |
| apache-airflow-providers-common-io     | 1.7.1   |
| apache-airflow-providers-common-sql    | 1.30.4  |
| apache-airflow-providers-elasticsearch | 6.4.4   |
| apache-airflow-providers-openlineage   | 2.10.1  |
| apache-airflow-providers-smtp          | 2.4.2   |
| apache-airflow-providers-standard      | 1.11.0  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-11

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.15.1  |
| apache-airflow-providers-common-compat | 1.12.0  |
| apache-airflow-providers-common-io     | 1.7.1   |
| apache-airflow-providers-common-sql    | 1.30.3  |
| apache-airflow-providers-elasticsearch | 6.4.3   |
| apache-airflow-providers-openlineage   | 2.10.0  |
| apache-airflow-providers-smtp          | 2.4.2   |
| apache-airflow-providers-standard      | 1.10.3  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-10

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.15.0  |
| apache-airflow-providers-common-compat | 1.11.0  |
| apache-airflow-providers-common-io     | 1.7.0   |
| apache-airflow-providers-common-sql    | 1.30.2  |
| apache-airflow-providers-elasticsearch | 6.4.2   |
| apache-airflow-providers-openlineage   | 2.9.2   |
| apache-airflow-providers-smtp          | 2.4.1   |
| apache-airflow-providers-standard      | 1.10.2  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-9

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.14.0  |
| apache-airflow-providers-common-compat | 1.10.0  |
| apache-airflow-providers-common-io     | 1.7.0   |
| apache-airflow-providers-common-sql    | 1.30.0  |
| apache-airflow-providers-elasticsearch | 6.4.0   |
| apache-airflow-providers-openlineage   | 2.9.0   |
| apache-airflow-providers-smtp          | 2.4.0   |
| apache-airflow-providers-standard      | 1.10.0  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-8

<Warning>
  **Restricted release**

  Astro Runtime 3.1-8 was restricted from use on December 16, 2025, after its initial release because the Airflow version it is based on, Apache Airflow 3.1.4, was yanked from the OSS Airflow project.
</Warning>

## Astro Runtime 3.1-7

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.14.0  |
| apache-airflow-providers-common-compat | 1.10.0  |
| apache-airflow-providers-common-io     | 1.7.0   |
| apache-airflow-providers-common-sql    | 1.30.0  |
| apache-airflow-providers-elasticsearch | 6.4.0   |
| apache-airflow-providers-openlineage   | 2.9.0   |
| apache-airflow-providers-smtp          | 2.4.0   |
| apache-airflow-providers-standard      | 1.10.0  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-6

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.14.0  |
| apache-airflow-providers-common-compat | 1.10.0  |
| apache-airflow-providers-common-io     | 1.7.0   |
| apache-airflow-providers-common-sql    | 1.30.0  |
| apache-airflow-providers-elasticsearch | 6.4.0   |
| apache-airflow-providers-openlineage   | 2.9.0   |
| apache-airflow-providers-smtp          | 2.4.0   |
| apache-airflow-providers-standard      | 1.10.0  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-5

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.13.0  |
| apache-airflow-providers-common-compat | 1.8.0   |
| apache-airflow-providers-common-io     | 1.6.4   |
| apache-airflow-providers-common-sql    | 1.28.2  |
| apache-airflow-providers-elasticsearch | 6.3.4   |
| apache-airflow-providers-openlineage   | 2.7.3   |
| apache-airflow-providers-smtp          | 2.3.1   |
| apache-airflow-providers-standard      | 1.9.1   |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-4

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.13.0  |
| apache-airflow-providers-common-compat | 1.8.0   |
| apache-airflow-providers-common-io     | 1.6.4   |
| apache-airflow-providers-common-sql    | 1.28.2  |
| apache-airflow-providers-elasticsearch | 6.3.4   |
| apache-airflow-providers-openlineage   | 2.7.3   |
| apache-airflow-providers-smtp          | 2.3.1   |
| apache-airflow-providers-standard      | 1.9.1   |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-3

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.13.0  |
| apache-airflow-providers-common-compat | 1.8.0   |
| apache-airflow-providers-common-io     | 1.6.4   |
| apache-airflow-providers-common-sql    | 1.28.2  |
| apache-airflow-providers-elasticsearch | 6.3.4   |
| apache-airflow-providers-openlineage   | 2.7.3   |
| apache-airflow-providers-smtp          | 2.3.1   |
| apache-airflow-providers-standard      | 1.9.1   |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-2

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.12.4  |
| apache-airflow-providers-common-compat | 1.7.4   |
| apache-airflow-providers-common-io     | 1.6.3   |
| apache-airflow-providers-common-sql    | 1.28.1  |
| apache-airflow-providers-elasticsearch | 6.3.3   |
| apache-airflow-providers-openlineage   | 2.7.2   |
| apache-airflow-providers-smtp          | 2.3.1   |
| apache-airflow-providers-standard      | 1.9.0   |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.1-1

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.12.3  |
| apache-airflow-providers-common-compat | 1.7.4   |
| apache-airflow-providers-common-io     | 1.6.3   |
| apache-airflow-providers-common-sql    | 1.28.1  |
| apache-airflow-providers-elasticsearch | 6.3.3   |
| apache-airflow-providers-openlineage   | 2.7.1   |
| apache-airflow-providers-smtp          | 2.2.1   |
| apache-airflow-providers-standard      | 1.8.0   |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.0-16

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.20.0  |
| apache-airflow-providers-common-compat | 1.15.0  |
| apache-airflow-providers-common-io     | 1.7.3   |
| apache-airflow-providers-common-sql    | 1.36.0  |
| apache-airflow-providers-elasticsearch | 6.6.0   |
| apache-airflow-providers-mysql         | 6.6.1   |
| apache-airflow-providers-openlineage   | 2.18.0  |
| apache-airflow-providers-postgres      | 6.7.1   |
| apache-airflow-providers-smtp          | 3.0.1   |
| apache-airflow-providers-standard      | 1.14.0  |
| astronomer-providers-logging           | 1.6.6   |

## Astro Runtime 3.0-15

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.15.1  |
| apache-airflow-providers-common-compat | 1.12.0  |
| apache-airflow-providers-common-io     | 1.7.1   |
| apache-airflow-providers-common-sql    | 1.30.3  |
| apache-airflow-providers-elasticsearch | 6.4.3   |
| apache-airflow-providers-mysql         | 6.4.1   |
| apache-airflow-providers-openlineage   | 2.7.3   |
| apache-airflow-providers-postgres      | 6.5.2   |
| apache-airflow-providers-smtp          | 2.4.2   |
| apache-airflow-providers-standard      | 1.10.3  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.0-14

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.15.1  |
| apache-airflow-providers-common-compat | 1.12.0  |
| apache-airflow-providers-common-io     | 1.7.1   |
| apache-airflow-providers-common-sql    | 1.30.3  |
| apache-airflow-providers-elasticsearch | 6.4.3   |
| apache-airflow-providers-mysql         | 6.4.1   |
| apache-airflow-providers-openlineage   | 2.7.3   |
| apache-airflow-providers-postgres      | 6.5.2   |
| apache-airflow-providers-smtp          | 2.4.2   |
| apache-airflow-providers-standard      | 1.10.3  |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.0-13

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.13.0  |
| apache-airflow-providers-common-compat | 1.8.0   |
| apache-airflow-providers-common-io     | 1.6.4   |
| apache-airflow-providers-common-sql    | 1.28.2  |
| apache-airflow-providers-elasticsearch | 6.3.4   |
| apache-airflow-providers-mysql         | 6.3.4   |
| apache-airflow-providers-openlineage   | 2.7.3   |
| apache-airflow-providers-postgres      | 6.4.0   |
| apache-airflow-providers-smtp          | 2.3.1   |
| apache-airflow-providers-standard      | 1.9.1   |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.0-12

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.12.3  |
| apache-airflow-providers-common-compat | 1.7.4   |
| apache-airflow-providers-common-io     | 1.6.3   |
| apache-airflow-providers-common-sql    | 1.28.1  |
| apache-airflow-providers-elasticsearch | 6.3.3   |
| apache-airflow-providers-mysql         | 6.3.4   |
| apache-airflow-providers-openlineage   | 2.7.1   |
| apache-airflow-providers-postgres      | 6.3.0   |
| apache-airflow-providers-smtp          | 2.2.1   |
| apache-airflow-providers-standard      | 1.8.0   |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.0-11

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.12.2  |
| apache-airflow-providers-common-compat | 1.7.3   |
| apache-airflow-providers-common-io     | 1.6.2   |
| apache-airflow-providers-common-sql    | 1.27.5  |
| apache-airflow-providers-elasticsearch | 6.3.2   |
| apache-airflow-providers-mysql         | 6.3.3   |
| apache-airflow-providers-openlineage   | 2.6.1   |
| apache-airflow-providers-postgres      | 6.2.3   |
| apache-airflow-providers-smtp          | 2.2.0   |
| apache-airflow-providers-standard      | 1.6.0   |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.0-10

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.12.2  |
| apache-airflow-providers-common-compat | 1.7.3   |
| apache-airflow-providers-common-io     | 1.6.2   |
| apache-airflow-providers-common-sql    | 1.27.5  |
| apache-airflow-providers-elasticsearch | 6.3.2   |
| apache-airflow-providers-mysql         | 6.3.3   |
| apache-airflow-providers-openlineage   | 2.6.1   |
| apache-airflow-providers-postgres      | 6.2.3   |
| apache-airflow-providers-smtp          | 2.2.0   |
| apache-airflow-providers-standard      | 1.6.0   |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.0-8

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.12.2  |
| apache-airflow-providers-common-compat | 1.7.3   |
| apache-airflow-providers-common-io     | 1.6.2   |
| apache-airflow-providers-common-sql    | 1.27.5  |
| apache-airflow-providers-elasticsearch | 6.3.2   |
| apache-airflow-providers-mysql         | 6.3.3   |
| apache-airflow-providers-openlineage   | 2.6.1   |
| apache-airflow-providers-postgres      | 6.2.3   |
| apache-airflow-providers-smtp          | 2.2.0   |
| apache-airflow-providers-standard      | 1.6.0   |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.0-7

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.12.2  |
| apache-airflow-providers-common-compat | 1.7.3   |
| apache-airflow-providers-common-io     | 1.6.2   |
| apache-airflow-providers-common-sql    | 1.27.4  |
| apache-airflow-providers-elasticsearch | 6.3.2   |
| apache-airflow-providers-mysql         | 6.3.3   |
| apache-airflow-providers-openlineage   | 2.5.0   |
| apache-airflow-providers-postgres      | 6.2.2   |
| apache-airflow-providers-smtp          | 2.1.2   |
| apache-airflow-providers-standard      | 1.5.0   |
| astronomer-providers-logging           | 1.6.4   |

## Astro Runtime 3.0-6

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.12.1  |
| apache-airflow-providers-common-compat | 1.7.2   |
| apache-airflow-providers-common-io     | 1.6.1   |
| apache-airflow-providers-common-sql    | 1.27.2  |
| apache-airflow-providers-elasticsearch | 6.3.1   |
| apache-airflow-providers-mysql         | 6.3.2   |
| apache-airflow-providers-openlineage   | 2.5.0   |
| apache-airflow-providers-postgres      | 6.2.1   |
| apache-airflow-providers-smtp          | 2.1.1   |
| apache-airflow-providers-standard      | 1.4.1   |
| astronomer-providers-logging           | 1.6.3   |

## Astro Runtime 3.0-5

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.12.1  |
| apache-airflow-providers-common-compat | 1.7.2   |
| apache-airflow-providers-common-io     | 1.6.1   |
| apache-airflow-providers-common-sql    | 1.27.2  |
| apache-airflow-providers-elasticsearch | 6.3.1   |
| apache-airflow-providers-mysql         | 6.3.2   |
| apache-airflow-providers-openlineage   | 2.5.0   |
| apache-airflow-providers-postgres      | 6.2.1   |
| apache-airflow-providers-smtp          | 2.1.1   |
| apache-airflow-providers-standard      | 1.4.0   |
| astronomer-providers-logging           | 1.6.3   |

## Astro Runtime 3.0-4

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.11.0  |
| apache-airflow-providers-common-compat | 1.7.0   |
| apache-airflow-providers-common-io     | 1.6.0   |
| apache-airflow-providers-common-sql    | 1.27.1  |
| apache-airflow-providers-elasticsearch | 6.3.0   |
| apache-airflow-providers-mysql         | 6.3.0   |
| apache-airflow-providers-openlineage   | 2.3.0   |
| apache-airflow-providers-postgres      | 6.2.0   |
| apache-airflow-providers-smtp          | 2.1.0   |
| apache-airflow-providers-standard      | 1.2.0   |
| astronomer-providers-logging           | 1.6.2   |

## Astro Runtime 3.0-3

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.11.0  |
| apache-airflow-providers-common-compat | 1.7.0   |
| apache-airflow-providers-common-io     | 1.6.0   |
| apache-airflow-providers-common-sql    | 1.27.1  |
| apache-airflow-providers-elasticsearch | 6.3.0   |
| apache-airflow-providers-mysql         | 6.3.0   |
| apache-airflow-providers-openlineage   | 2.3.0   |
| apache-airflow-providers-postgres      | 6.2.0   |
| apache-airflow-providers-smtp          | 2.1.0   |
| apache-airflow-providers-standard      | 1.2.0   |
| astronomer-providers-logging           | 1.6.2   |

## Astro Runtime 3.0-2

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.10.6  |
| apache-airflow-providers-common-compat | 1.6.1   |
| apache-airflow-providers-common-io     | 1.5.4   |
| apache-airflow-providers-common-sql    | 1.26.0  |
| apache-airflow-providers-elasticsearch | 6.2.2   |
| apache-airflow-providers-mysql         | 6.2.2   |
| apache-airflow-providers-openlineage   | 2.2.0   |
| apache-airflow-providers-postgres      | 6.1.3   |
| apache-airflow-providers-smtp          | 2.0.3   |
| apache-airflow-providers-standard      | 1.1.0   |
| astronomer-providers-logging           | 1.6.1   |

## Astro Runtime 3.0-1

| Package Name                           | Version |
| :------------------------------------- | :------ |
| apache-airflow-providers-celery        | 3.10.6  |
| apache-airflow-providers-common-compat | 1.6.0   |
| apache-airflow-providers-common-io     | 1.5.3   |
| apache-airflow-providers-common-sql    | 1.25.0  |
| apache-airflow-providers-elasticsearch | 6.2.1   |
| apache-airflow-providers-mysql         | 6.2.1   |
| apache-airflow-providers-openlineage   | 2.2.0   |
| apache-airflow-providers-postgres      | 6.1.2   |
| apache-airflow-providers-smtp          | 2.0.2   |
| apache-airflow-providers-standard      | 1.0.0   |
| astronomer-providers-logging           | 1.6.0   |

## Astro Runtime 13.9.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 9.33.0  |
    | apache-airflow-providers-celery          | 3.23.0  |
    | apache-airflow-providers-cncf-kubernetes | 10.20.0 |
    | apache-airflow-providers-common-compat   | 1.17.0  |
    | apache-airflow-providers-common-io       | 1.8.0   |
    | apache-airflow-providers-common-sql      | 1.36.0  |
    | apache-airflow-providers-datadog         | 3.10.5  |
    | apache-airflow-providers-elasticsearch   | 6.8.1   |
    | apache-airflow-providers-fab             | 1.5.4   |
    | apache-airflow-providers-ftp             | 3.15.2  |
    | apache-airflow-providers-google          | 15.1.0  |
    | apache-airflow-providers-http            | 6.0.5   |
    | apache-airflow-providers-imap            | 3.12.0  |
    | apache-airflow-providers-microsoft-azure | 12.10.3 |
    | apache-airflow-providers-mysql           | 6.6.1   |
    | apache-airflow-providers-openlineage     | 2.19.0  |
    | apache-airflow-providers-postgres        | 6.8.0   |
    | apache-airflow-providers-redis           | 4.5.0   |
    | apache-airflow-providers-smtp            | 3.0.2   |
    | apache-airflow-providers-sqlite          | 4.3.3   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.8   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.23.0  |
    | apache-airflow-providers-common-compat | 1.17.0  |
    | apache-airflow-providers-common-io     | 1.8.0   |
    | apache-airflow-providers-common-sql    | 1.36.0  |
    | apache-airflow-providers-elasticsearch | 6.8.1   |
    | apache-airflow-providers-fab           | 1.5.4   |
    | apache-airflow-providers-ftp           | 3.15.2  |
    | apache-airflow-providers-http          | 6.0.5   |
    | apache-airflow-providers-imap          | 3.12.0  |
    | apache-airflow-providers-mysql         | 6.6.1   |
    | apache-airflow-providers-openlineage   | 2.19.0  |
    | apache-airflow-providers-postgres      | 6.8.0   |
    | apache-airflow-providers-smtp          | 3.0.2   |
    | apache-airflow-providers-sqlite        | 4.3.3   |
    | astronomer-providers-logging           | 1.6.8   |
  </Tab>
</Tabs>

## Astro Runtime 13.8.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 9.30.0  |
    | apache-airflow-providers-celery          | 3.20.0  |
    | apache-airflow-providers-cncf-kubernetes | 10.18.0 |
    | apache-airflow-providers-common-compat   | 1.15.0  |
    | apache-airflow-providers-common-io       | 1.7.3   |
    | apache-airflow-providers-common-sql      | 1.36.0  |
    | apache-airflow-providers-datadog         | 3.10.5  |
    | apache-airflow-providers-elasticsearch   | 6.6.0   |
    | apache-airflow-providers-fab             | 1.5.4   |
    | apache-airflow-providers-ftp             | 3.15.0  |
    | apache-airflow-providers-google          | 15.1.0  |
    | apache-airflow-providers-http            | 6.0.3   |
    | apache-airflow-providers-imap            | 3.11.3  |
    | apache-airflow-providers-microsoft-azure | 12.10.3 |
    | apache-airflow-providers-mysql           | 6.6.1   |
    | apache-airflow-providers-openlineage     | 2.18.0  |
    | apache-airflow-providers-postgres        | 6.7.1   |
    | apache-airflow-providers-redis           | 4.4.5   |
    | apache-airflow-providers-smtp            | 3.0.1   |
    | apache-airflow-providers-sqlite          | 4.3.3   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.6   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.20.0  |
    | apache-airflow-providers-common-compat | 1.15.0  |
    | apache-airflow-providers-common-io     | 1.7.3   |
    | apache-airflow-providers-common-sql    | 1.36.0  |
    | apache-airflow-providers-elasticsearch | 6.6.0   |
    | apache-airflow-providers-fab           | 1.5.4   |
    | apache-airflow-providers-ftp           | 3.15.0  |
    | apache-airflow-providers-http          | 6.0.3   |
    | apache-airflow-providers-imap          | 3.11.3  |
    | apache-airflow-providers-mysql         | 6.6.1   |
    | apache-airflow-providers-openlineage   | 2.18.0  |
    | apache-airflow-providers-postgres      | 6.7.1   |
    | apache-airflow-providers-smtp          | 3.0.1   |
    | apache-airflow-providers-sqlite        | 4.3.3   |
    | astronomer-providers-logging           | 1.6.6   |
  </Tab>
</Tabs>

## Astro Runtime 13.7.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 9.28.0  |
    | apache-airflow-providers-celery          | 3.19.0  |
    | apache-airflow-providers-cncf-kubernetes | 10.17.0 |
    | apache-airflow-providers-common-compat   | 1.14.3  |
    | apache-airflow-providers-common-io       | 1.7.2   |
    | apache-airflow-providers-common-sql      | 1.36.0  |
    | apache-airflow-providers-datadog         | 3.10.4  |
    | apache-airflow-providers-elasticsearch   | 6.5.3   |
    | apache-airflow-providers-fab             | 1.5.4   |
    | apache-airflow-providers-ftp             | 3.14.3  |
    | apache-airflow-providers-google          | 15.1.0  |
    | apache-airflow-providers-http            | 6.0.2   |
    | apache-airflow-providers-imap            | 3.11.2  |
    | apache-airflow-providers-microsoft-azure | 12.10.3 |
    | apache-airflow-providers-mysql           | 6.5.3   |
    | apache-airflow-providers-openlineage     | 2.16.0  |
    | apache-airflow-providers-postgres        | 6.6.3   |
    | apache-airflow-providers-redis           | 4.4.4   |
    | apache-airflow-providers-smtp            | 2.4.5   |
    | apache-airflow-providers-sqlite          | 4.3.2   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.19.0  |
    | apache-airflow-providers-common-compat | 1.14.3  |
    | apache-airflow-providers-common-io     | 1.7.2   |
    | apache-airflow-providers-common-sql    | 1.36.0  |
    | apache-airflow-providers-elasticsearch | 6.5.3   |
    | apache-airflow-providers-fab           | 1.5.4   |
    | apache-airflow-providers-ftp           | 3.14.3  |
    | apache-airflow-providers-http          | 6.0.2   |
    | apache-airflow-providers-imap          | 3.11.2  |
    | apache-airflow-providers-mysql         | 6.5.3   |
    | apache-airflow-providers-openlineage   | 2.16.0  |
    | apache-airflow-providers-postgres      | 6.6.3   |
    | apache-airflow-providers-smtp          | 2.4.5   |
    | apache-airflow-providers-sqlite        | 4.3.2   |
    | astronomer-providers-logging           | 1.6.2   |
  </Tab>
</Tabs>

## Astro Runtime 13.6.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 9.23.0  |
    | apache-airflow-providers-celery          | 3.15.2  |
    | apache-airflow-providers-cncf-kubernetes | 10.14.0 |
    | apache-airflow-providers-common-compat   | 1.14.1  |
    | apache-airflow-providers-common-io       | 1.7.1   |
    | apache-airflow-providers-common-sql      | 1.33.0  |
    | apache-airflow-providers-datadog         | 3.10.2  |
    | apache-airflow-providers-elasticsearch   | 6.5.0   |
    | apache-airflow-providers-fab             | 1.5.4   |
    | apache-airflow-providers-ftp             | 3.14.1  |
    | apache-airflow-providers-google          | 15.1.0  |
    | apache-airflow-providers-http            | 6.0.0   |
    | apache-airflow-providers-imap            | 3.11.0  |
    | apache-airflow-providers-microsoft-azure | 12.10.3 |
    | apache-airflow-providers-mysql           | 6.5.0   |
    | apache-airflow-providers-openlineage     | 2.12.0  |
    | apache-airflow-providers-postgres        | 6.6.1   |
    | apache-airflow-providers-redis           | 4.4.2   |
    | apache-airflow-providers-smtp            | 2.4.3   |
    | apache-airflow-providers-sqlite          | 4.3.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.15.2  |
    | apache-airflow-providers-common-compat | 1.14.1  |
    | apache-airflow-providers-common-io     | 1.7.1   |
    | apache-airflow-providers-common-sql    | 1.33.0  |
    | apache-airflow-providers-elasticsearch | 6.5.0   |
    | apache-airflow-providers-fab           | 1.5.4   |
    | apache-airflow-providers-ftp           | 3.14.1  |
    | apache-airflow-providers-http          | 6.0.0   |
    | apache-airflow-providers-imap          | 3.11.0  |
    | apache-airflow-providers-mysql         | 6.5.0   |
    | apache-airflow-providers-openlineage   | 2.12.0  |
    | apache-airflow-providers-postgres      | 6.6.1   |
    | apache-airflow-providers-smtp          | 2.4.3   |
    | apache-airflow-providers-sqlite        | 4.3.0   |
    | astronomer-providers-logging           | 1.6.2   |
  </Tab>
</Tabs>

## Astro Runtime 13.5.1

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 9.21.0  |
    | apache-airflow-providers-celery          | 3.15.2  |
    | apache-airflow-providers-cncf-kubernetes | 10.12.4 |
    | apache-airflow-providers-common-compat   | 1.13.1  |
    | apache-airflow-providers-common-io       | 1.7.1   |
    | apache-airflow-providers-common-sql      | 1.31.0  |
    | apache-airflow-providers-datadog         | 3.10.2  |
    | apache-airflow-providers-elasticsearch   | 6.4.4   |
    | apache-airflow-providers-fab             | 1.5.4   |
    | apache-airflow-providers-ftp             | 3.14.1  |
    | apache-airflow-providers-google          | 15.1.0  |
    | apache-airflow-providers-http            | 5.6.4   |
    | apache-airflow-providers-imap            | 3.11.0  |
    | apache-airflow-providers-microsoft-azure | 12.10.3 |
    | apache-airflow-providers-mysql           | 6.4.3   |
    | apache-airflow-providers-openlineage     | 2.10.2  |
    | apache-airflow-providers-postgres        | 6.5.4   |
    | apache-airflow-providers-redis           | 4.4.2   |
    | apache-airflow-providers-smtp            | 2.4.2   |
    | apache-airflow-providers-sqlite          | 4.2.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.15.2  |
    | apache-airflow-providers-common-compat | 1.13.1  |
    | apache-airflow-providers-common-io     | 1.7.1   |
    | apache-airflow-providers-common-sql    | 1.31.0  |
    | apache-airflow-providers-elasticsearch | 6.4.4   |
    | apache-airflow-providers-fab           | 1.5.4   |
    | apache-airflow-providers-ftp           | 3.14.1  |
    | apache-airflow-providers-http          | 5.6.4   |
    | apache-airflow-providers-imap          | 3.11.0  |
    | apache-airflow-providers-mysql         | 6.4.3   |
    | apache-airflow-providers-openlineage   | 2.10.2  |
    | apache-airflow-providers-postgres      | 6.5.4   |
    | apache-airflow-providers-smtp          | 2.4.2   |
    | apache-airflow-providers-sqlite        | 4.2.1   |
    | astronomer-providers-logging           | 1.6.2   |
  </Tab>
</Tabs>

## Astro Runtime 13.5.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 9.21.0  |
    | apache-airflow-providers-celery          | 3.15.2  |
    | apache-airflow-providers-cncf-kubernetes | 10.12.4 |
    | apache-airflow-providers-common-compat   | 1.13.1  |
    | apache-airflow-providers-common-io       | 1.7.1   |
    | apache-airflow-providers-common-sql      | 1.31.0  |
    | apache-airflow-providers-datadog         | 3.10.2  |
    | apache-airflow-providers-elasticsearch   | 6.4.4   |
    | apache-airflow-providers-fab             | 1.5.4   |
    | apache-airflow-providers-ftp             | 3.14.1  |
    | apache-airflow-providers-google          | 15.1.0  |
    | apache-airflow-providers-http            | 5.6.4   |
    | apache-airflow-providers-imap            | 3.11.0  |
    | apache-airflow-providers-microsoft-azure | 12.10.3 |
    | apache-airflow-providers-mysql           | 6.4.3   |
    | apache-airflow-providers-openlineage     | 2.10.2  |
    | apache-airflow-providers-postgres        | 6.5.4   |
    | apache-airflow-providers-redis           | 4.4.2   |
    | apache-airflow-providers-smtp            | 2.4.2   |
    | apache-airflow-providers-sqlite          | 4.2.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.15.2  |
    | apache-airflow-providers-common-compat | 1.13.1  |
    | apache-airflow-providers-common-io     | 1.7.1   |
    | apache-airflow-providers-common-sql    | 1.31.0  |
    | apache-airflow-providers-elasticsearch | 6.4.4   |
    | apache-airflow-providers-fab           | 1.5.4   |
    | apache-airflow-providers-ftp           | 3.14.1  |
    | apache-airflow-providers-http          | 5.6.4   |
    | apache-airflow-providers-imap          | 3.11.0  |
    | apache-airflow-providers-mysql         | 6.4.3   |
    | apache-airflow-providers-openlineage   | 2.10.2  |
    | apache-airflow-providers-postgres      | 6.5.4   |
    | apache-airflow-providers-smtp          | 2.4.2   |
    | apache-airflow-providers-sqlite        | 4.2.1   |
    | astronomer-providers-logging           | 1.6.2   |
  </Tab>
</Tabs>

## Astro Runtime 13.4.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 9.19.0  |
    | apache-airflow-providers-celery          | 3.15.0  |
    | apache-airflow-providers-cncf-kubernetes | 10.12.0 |
    | apache-airflow-providers-common-compat   | 1.11.0  |
    | apache-airflow-providers-common-io       | 1.7.0   |
    | apache-airflow-providers-common-sql      | 1.30.2  |
    | apache-airflow-providers-datadog         | 3.10.1  |
    | apache-airflow-providers-elasticsearch   | 6.4.2   |
    | apache-airflow-providers-fab             | 1.5.3   |
    | apache-airflow-providers-ftp             | 3.14.0  |
    | apache-airflow-providers-google          | 15.1.0  |
    | apache-airflow-providers-http            | 5.6.2   |
    | apache-airflow-providers-imap            | 3.10.2  |
    | apache-airflow-providers-microsoft-azure | 12.10.1 |
    | apache-airflow-providers-mysql           | 6.4.0   |
    | apache-airflow-providers-openlineage     | 2.9.2   |
    | apache-airflow-providers-postgres        | 6.5.1   |
    | apache-airflow-providers-redis           | 4.4.1   |
    | apache-airflow-providers-smtp            | 2.4.1   |
    | apache-airflow-providers-sqlite          | 4.2.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.15.0  |
    | apache-airflow-providers-common-compat | 1.11.0  |
    | apache-airflow-providers-common-io     | 1.7.0   |
    | apache-airflow-providers-common-sql    | 1.30.2  |
    | apache-airflow-providers-elasticsearch | 6.4.2   |
    | apache-airflow-providers-fab           | 1.5.3   |
    | apache-airflow-providers-ftp           | 3.14.0  |
    | apache-airflow-providers-http          | 5.6.2   |
    | apache-airflow-providers-imap          | 3.10.2  |
    | apache-airflow-providers-mysql         | 6.4.0   |
    | apache-airflow-providers-openlineage   | 2.9.2   |
    | apache-airflow-providers-postgres      | 6.5.1   |
    | apache-airflow-providers-smtp          | 2.4.1   |
    | apache-airflow-providers-sqlite        | 4.2.0   |
    | astronomer-providers-logging           | 1.6.2   |
  </Tab>
</Tabs>

## Astro Runtime 13.3.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 9.18.0  |
    | apache-airflow-providers-celery          | 3.14.0  |
    | apache-airflow-providers-cncf-kubernetes | 10.9.0  |
    | apache-airflow-providers-common-compat   | 1.10.0  |
    | apache-airflow-providers-common-io       | 1.7.0   |
    | apache-airflow-providers-common-sql      | 1.30.0  |
    | apache-airflow-providers-datadog         | 3.10.0  |
    | apache-airflow-providers-elasticsearch   | 6.4.0   |
    | apache-airflow-providers-fab             | 1.5.3   |
    | apache-airflow-providers-ftp             | 3.14.0  |
    | apache-airflow-providers-google          | 15.1.0  |
    | apache-airflow-providers-http            | 5.6.0   |
    | apache-airflow-providers-imap            | 3.10.0  |
    | apache-airflow-providers-microsoft-azure | 12.9.0  |
    | apache-airflow-providers-mysql           | 6.4.0   |
    | apache-airflow-providers-openlineage     | 2.9.0   |
    | apache-airflow-providers-postgres        | 6.5.0   |
    | apache-airflow-providers-redis           | 4.4.0   |
    | apache-airflow-providers-smtp            | 2.4.0   |
    | apache-airflow-providers-sqlite          | 4.2.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.14.0  |
    | apache-airflow-providers-common-compat | 1.10.0  |
    | apache-airflow-providers-common-io     | 1.7.0   |
    | apache-airflow-providers-common-sql    | 1.30.0  |
    | apache-airflow-providers-elasticsearch | 6.4.0   |
    | apache-airflow-providers-fab           | 1.5.3   |
    | apache-airflow-providers-ftp           | 3.14.0  |
    | apache-airflow-providers-http          | 5.6.0   |
    | apache-airflow-providers-imap          | 3.10.0  |
    | apache-airflow-providers-mysql         | 6.4.0   |
    | apache-airflow-providers-openlineage   | 2.9.0   |
    | apache-airflow-providers-postgres      | 6.5.0   |
    | apache-airflow-providers-smtp          | 2.4.0   |
    | apache-airflow-providers-sqlite        | 4.2.0   |
    | astronomer-providers-logging           | 1.6.2   |
  </Tab>
</Tabs>

## Astro Runtime 13.2.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 9.12.0  |
    | apache-airflow-providers-celery          | 3.12.2  |
    | apache-airflow-providers-cncf-kubernetes | 10.4.3  |
    | apache-airflow-providers-common-compat   | 1.7.3   |
    | apache-airflow-providers-common-io       | 1.6.2   |
    | apache-airflow-providers-common-sql      | 1.27.5  |
    | apache-airflow-providers-datadog         | 3.9.2   |
    | apache-airflow-providers-elasticsearch   | 6.3.2   |
    | apache-airflow-providers-fab             | 1.5.3   |
    | apache-airflow-providers-ftp             | 3.13.2  |
    | apache-airflow-providers-google          | 15.1.0  |
    | apache-airflow-providers-http            | 5.3.3   |
    | apache-airflow-providers-imap            | 3.9.2   |
    | apache-airflow-providers-microsoft-azure | 12.6.1  |
    | apache-airflow-providers-mysql           | 6.3.3   |
    | apache-airflow-providers-openlineage     | 2.6.1   |
    | apache-airflow-providers-postgres        | 6.2.3   |
    | apache-airflow-providers-redis           | 4.2.0   |
    | apache-airflow-providers-smtp            | 2.2.0   |
    | apache-airflow-providers-sqlite          | 4.1.2   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.12.2  |
    | apache-airflow-providers-common-compat | 1.7.3   |
    | apache-airflow-providers-common-io     | 1.6.2   |
    | apache-airflow-providers-common-sql    | 1.27.5  |
    | apache-airflow-providers-elasticsearch | 6.3.2   |
    | apache-airflow-providers-fab           | 1.5.3   |
    | apache-airflow-providers-ftp           | 3.13.2  |
    | apache-airflow-providers-http          | 5.3.3   |
    | apache-airflow-providers-imap          | 3.9.2   |
    | apache-airflow-providers-mysql         | 6.3.3   |
    | apache-airflow-providers-openlineage   | 2.6.1   |
    | apache-airflow-providers-postgres      | 6.2.3   |
    | apache-airflow-providers-smtp          | 2.2.0   |
    | apache-airflow-providers-sqlite        | 4.1.2   |
    | astronomer-providers-logging           | 1.6.2   |
  </Tab>
</Tabs>

## Astro Runtime 13.1.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 9.9.0   |
    | apache-airflow-providers-celery          | 3.12.0  |
    | apache-airflow-providers-cncf-kubernetes | 10.4.3  |
    | apache-airflow-providers-common-compat   | 1.7.1   |
    | apache-airflow-providers-common-io       | 1.6.0   |
    | apache-airflow-providers-common-sql      | 1.27.2  |
    | apache-airflow-providers-datadog         | 3.9.0   |
    | apache-airflow-providers-elasticsearch   | 6.3.0   |
    | apache-airflow-providers-fab             | 1.5.3   |
    | apache-airflow-providers-ftp             | 3.13.0  |
    | apache-airflow-providers-google          | 15.1.0  |
    | apache-airflow-providers-http            | 5.3.1   |
    | apache-airflow-providers-imap            | 3.9.0   |
    | apache-airflow-providers-microsoft-azure | 12.4.1  |
    | apache-airflow-providers-mysql           | 6.3.1   |
    | apache-airflow-providers-openlineage     | 2.4.0   |
    | apache-airflow-providers-postgres        | 6.2.0   |
    | apache-airflow-providers-redis           | 4.1.0   |
    | apache-airflow-providers-smtp            | 2.1.0   |
    | apache-airflow-providers-sqlite          | 4.1.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.12.0  |
    | apache-airflow-providers-common-compat | 1.7.1   |
    | apache-airflow-providers-common-io     | 1.6.0   |
    | apache-airflow-providers-common-sql    | 1.27.2  |
    | apache-airflow-providers-elasticsearch | 6.3.0   |
    | apache-airflow-providers-fab           | 1.5.3   |
    | apache-airflow-providers-ftp           | 3.13.0  |
    | apache-airflow-providers-http          | 5.3.1   |
    | apache-airflow-providers-imap          | 3.9.0   |
    | apache-airflow-providers-mysql         | 6.3.1   |
    | apache-airflow-providers-openlineage   | 2.4.0   |
    | apache-airflow-providers-postgres      | 6.2.0   |
    | apache-airflow-providers-smtp          | 2.1.0   |
    | apache-airflow-providers-sqlite        | 4.1.0   |
    | astronomer-providers-logging           | 1.6.2   |
  </Tab>
</Tabs>

## Astro Runtime 13.0.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 9.7.0   |
    | apache-airflow-providers-celery          | 3.10.6  |
    | apache-airflow-providers-cncf-kubernetes | 10.4.3  |
    | apache-airflow-providers-common-compat   | 1.6.1   |
    | apache-airflow-providers-common-io       | 1.5.4   |
    | apache-airflow-providers-common-sql      | 1.27.0  |
    | apache-airflow-providers-datadog         | 3.8.3   |
    | apache-airflow-providers-elasticsearch   | 6.2.2   |
    | apache-airflow-providers-fab             | 1.5.3   |
    | apache-airflow-providers-ftp             | 3.12.3  |
    | apache-airflow-providers-google          | 15.1.0  |
    | apache-airflow-providers-http            | 5.2.2   |
    | apache-airflow-providers-imap            | 3.8.3   |
    | apache-airflow-providers-microsoft-azure | 12.3.1  |
    | apache-airflow-providers-mysql           | 6.2.2   |
    | apache-airflow-providers-openlineage     | 2.2.0   |
    | apache-airflow-providers-postgres        | 6.1.3   |
    | apache-airflow-providers-redis           | 4.0.2   |
    | apache-airflow-providers-smtp            | 2.0.3   |
    | apache-airflow-providers-sqlite          | 4.0.2   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.10.6  |
    | apache-airflow-providers-common-compat | 1.6.1   |
    | apache-airflow-providers-common-io     | 1.5.4   |
    | apache-airflow-providers-common-sql    | 1.27.0  |
    | apache-airflow-providers-elasticsearch | 6.2.2   |
    | apache-airflow-providers-fab           | 1.5.3   |
    | apache-airflow-providers-ftp           | 3.12.3  |
    | apache-airflow-providers-http          | 5.2.2   |
    | apache-airflow-providers-imap          | 3.8.3   |
    | apache-airflow-providers-mysql         | 6.2.2   |
    | apache-airflow-providers-openlineage   | 2.2.0   |
    | apache-airflow-providers-postgres      | 6.1.3   |
    | apache-airflow-providers-smtp          | 2.0.3   |
    | apache-airflow-providers-sqlite        | 4.0.2   |
    | astronomer-providers-logging           | 1.6.2   |
  </Tab>
</Tabs>

## Astro Runtime 12.12.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.29.0  |
    | apache-airflow-providers-celery          | 3.13.1  |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.9.0   |
    | apache-airflow-providers-common-io       | 1.6.5   |
    | apache-airflow-providers-common-sql      | 1.29.0  |
    | apache-airflow-providers-datadog         | 3.9.3   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.5.3   |
    | apache-airflow-providers-ftp             | 3.13.3  |
    | apache-airflow-providers-google          | 10.26.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.9.4   |
    | apache-airflow-providers-microsoft-azure | 10.5.1  |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 2.8.0   |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.9.0   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.13.1  |
    | apache-airflow-providers-common-compat | 1.9.0   |
    | apache-airflow-providers-common-io     | 1.6.5   |
    | apache-airflow-providers-common-sql    | 1.29.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.5.3   |
    | apache-airflow-providers-ftp           | 3.13.3  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.9.4   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-openlineage   | 2.8.0   |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.9.0   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.6.2   |
  </Tab>
</Tabs>

## Astro Runtime 12.11.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.29.0  |
    | apache-airflow-providers-celery          | 3.13.1  |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.9.0   |
    | apache-airflow-providers-common-io       | 1.6.5   |
    | apache-airflow-providers-common-sql      | 1.29.0  |
    | apache-airflow-providers-datadog         | 3.9.3   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.5.3   |
    | apache-airflow-providers-ftp             | 3.13.3  |
    | apache-airflow-providers-google          | 10.26.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.9.4   |
    | apache-airflow-providers-microsoft-azure | 10.5.1  |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 2.8.0   |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.9.0   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.0   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.13.1  |
    | apache-airflow-providers-common-compat | 1.9.0   |
    | apache-airflow-providers-common-io     | 1.6.5   |
    | apache-airflow-providers-common-sql    | 1.29.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.5.3   |
    | apache-airflow-providers-ftp           | 3.13.3  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.9.4   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-openlineage   | 2.8.0   |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.9.0   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.6.0   |
  </Tab>
</Tabs>

## Astro Runtime 12.10.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.29.0  |
    | apache-airflow-providers-celery          | 3.12.0  |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.7.1   |
    | apache-airflow-providers-common-io       | 1.6.0   |
    | apache-airflow-providers-common-sql      | 1.27.2  |
    | apache-airflow-providers-datadog         | 3.9.0   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.5.3   |
    | apache-airflow-providers-ftp             | 3.13.0  |
    | apache-airflow-providers-google          | 10.26.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.9.0   |
    | apache-airflow-providers-microsoft-azure | 10.5.1  |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 2.4.0   |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.9.0   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.0   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.12.0  |
    | apache-airflow-providers-common-compat | 1.7.1   |
    | apache-airflow-providers-common-io     | 1.6.0   |
    | apache-airflow-providers-common-sql    | 1.27.2  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.5.3   |
    | apache-airflow-providers-ftp           | 3.13.0  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.9.0   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-openlineage   | 2.4.0   |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.9.0   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.6.0   |
  </Tab>
</Tabs>

## Astro Runtime 12.9.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.29.0  |
    | apache-airflow-providers-celery          | 3.10.6  |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.6.1   |
    | apache-airflow-providers-common-io       | 1.5.4   |
    | apache-airflow-providers-common-sql      | 1.26.0  |
    | apache-airflow-providers-datadog         | 3.8.3   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.5.2   |
    | apache-airflow-providers-ftp             | 3.12.3  |
    | apache-airflow-providers-google          | 10.26.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.8.3   |
    | apache-airflow-providers-microsoft-azure | 10.5.1  |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 1.14.0  |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.9.0   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.6.0   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.10.6  |
    | apache-airflow-providers-common-compat | 1.6.1   |
    | apache-airflow-providers-common-io     | 1.5.4   |
    | apache-airflow-providers-common-sql    | 1.26.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.5.2   |
    | apache-airflow-providers-ftp           | 3.12.3  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.8.3   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-openlineage   | 1.14.0  |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.9.0   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.6.0   |
  </Tab>
</Tabs>

## Astro Runtime 12.8.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.29.0  |
    | apache-airflow-providers-celery          | 3.10.5  |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.6.0   |
    | apache-airflow-providers-common-io       | 1.5.2   |
    | apache-airflow-providers-common-sql      | 1.25.0  |
    | apache-airflow-providers-datadog         | 3.8.3   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.5.2   |
    | apache-airflow-providers-ftp             | 3.12.3  |
    | apache-airflow-providers-google          | 10.26.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.8.3   |
    | apache-airflow-providers-microsoft-azure | 10.5.1  |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 1.14.0  |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.9.0   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.3   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.10.5  |
    | apache-airflow-providers-common-compat | 1.6.0   |
    | apache-airflow-providers-common-io     | 1.5.2   |
    | apache-airflow-providers-common-sql    | 1.25.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.5.2   |
    | apache-airflow-providers-ftp           | 3.12.3  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.8.3   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-openlineage   | 1.14.0  |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.9.0   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.5.3   |
  </Tab>
</Tabs>

## Astro Runtime 12.7.1

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.29.0  |
    | apache-airflow-providers-celery          | 3.10.0  |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.3.0   |
    | apache-airflow-providers-common-io       | 1.5.0   |
    | apache-airflow-providers-common-sql      | 1.21.0  |
    | apache-airflow-providers-datadog         | 3.8.0   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.5.2   |
    | apache-airflow-providers-ftp             | 3.12.0  |
    | apache-airflow-providers-google          | 10.26.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.8.0   |
    | apache-airflow-providers-microsoft-azure | 10.5.1  |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 1.14.0  |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.9.0   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.3   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.10.0  |
    | apache-airflow-providers-common-compat | 1.3.0   |
    | apache-airflow-providers-common-io     | 1.5.0   |
    | apache-airflow-providers-common-sql    | 1.21.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.5.2   |
    | apache-airflow-providers-ftp           | 3.12.0  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.8.0   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-openlineage   | 1.14.0  |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.9.0   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.5.3   |
  </Tab>
</Tabs>

## Astro Runtime 12.7.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.29.0  |
    | apache-airflow-providers-celery          | 3.10.0  |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.3.0   |
    | apache-airflow-providers-common-io       | 1.5.0   |
    | apache-airflow-providers-common-sql      | 1.21.0  |
    | apache-airflow-providers-datadog         | 3.8.0   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.5.2   |
    | apache-airflow-providers-ftp             | 3.12.0  |
    | apache-airflow-providers-google          | 10.26.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.8.0   |
    | apache-airflow-providers-microsoft-azure | 10.5.1  |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 1.14.0  |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.9.0   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.3   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.10.0  |
    | apache-airflow-providers-common-compat | 1.3.0   |
    | apache-airflow-providers-common-io     | 1.5.0   |
    | apache-airflow-providers-common-sql    | 1.21.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.5.2   |
    | apache-airflow-providers-ftp           | 3.12.0  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.8.0   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-openlineage   | 1.14.0  |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.9.0   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.5.3   |
  </Tab>
</Tabs>

## Astro Runtime 12.6.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.29.0  |
    | apache-airflow-providers-celery          | 3.8.5   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.2.2   |
    | apache-airflow-providers-common-io       | 1.4.2   |
    | apache-airflow-providers-common-sql      | 1.20.0  |
    | apache-airflow-providers-datadog         | 3.7.1   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.5.1   |
    | apache-airflow-providers-ftp             | 3.11.1  |
    | apache-airflow-providers-google          | 10.26.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 10.5.1  |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 1.14.0  |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.1   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.3   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.5   |
    | apache-airflow-providers-common-compat | 1.2.2   |
    | apache-airflow-providers-common-io     | 1.4.2   |
    | apache-airflow-providers-common-sql    | 1.20.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.5.1   |
    | apache-airflow-providers-ftp           | 3.11.1  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-openlineage   | 1.14.0  |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.8.1   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.5.3   |
  </Tab>
</Tabs>

## Astro Runtime 12.5.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.29.0  |
    | apache-airflow-providers-celery          | 3.8.5   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.2.2   |
    | apache-airflow-providers-common-io       | 1.4.2   |
    | apache-airflow-providers-common-sql      | 1.20.0  |
    | apache-airflow-providers-datadog         | 3.7.1   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.5.1   |
    | apache-airflow-providers-ftp             | 3.11.1  |
    | apache-airflow-providers-google          | 10.26.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 10.5.1  |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 1.14.0  |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.1   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.3   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.5   |
    | apache-airflow-providers-common-compat | 1.2.2   |
    | apache-airflow-providers-common-io     | 1.4.2   |
    | apache-airflow-providers-common-sql    | 1.20.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.5.1   |
    | apache-airflow-providers-ftp           | 3.11.1  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-openlineage   | 1.14.0  |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.8.1   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.5.3   |
  </Tab>
</Tabs>

## Astro Runtime 12.4.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.29.0  |
    | apache-airflow-providers-celery          | 3.8.3   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.2.1   |
    | apache-airflow-providers-common-io       | 1.4.2   |
    | apache-airflow-providers-common-sql      | 1.19.0  |
    | apache-airflow-providers-datadog         | 3.7.1   |
    | apache-airflow-providers-elasticsearch   | 5.5.2   |
    | apache-airflow-providers-fab             | 1.5.0   |
    | apache-airflow-providers-ftp             | 3.11.1  |
    | apache-airflow-providers-google          | 10.25.0 |
    | apache-airflow-providers-http            | 4.13.2  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 10.5.1  |
    | apache-airflow-providers-mysql           | 5.7.3   |
    | apache-airflow-providers-openlineage     | 1.13.0  |
    | apache-airflow-providers-postgres        | 5.13.1  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.0   |
    | apache-airflow-providers-sqlite          | 3.9.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.3   |
    | apache-airflow-providers-common-compat | 1.2.1   |
    | apache-airflow-providers-common-io     | 1.4.2   |
    | apache-airflow-providers-common-sql    | 1.19.0  |
    | apache-airflow-providers-elasticsearch | 5.5.2   |
    | apache-airflow-providers-fab           | 1.5.0   |
    | apache-airflow-providers-ftp           | 3.11.1  |
    | apache-airflow-providers-http          | 4.13.2  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.3   |
    | apache-airflow-providers-openlineage   | 1.13.0  |
    | apache-airflow-providers-postgres      | 5.13.1  |
    | apache-airflow-providers-smtp          | 1.8.0   |
    | apache-airflow-providers-sqlite        | 3.9.0   |
    | astronomer-providers-logging           | 1.5.2   |
  </Tab>
</Tabs>

## Astro Runtime 12.3.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.3   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.1   |
    | apache-airflow-providers-common-compat   | 1.2.1   |
    | apache-airflow-providers-common-io       | 1.4.2   |
    | apache-airflow-providers-common-sql      | 1.19.0  |
    | apache-airflow-providers-datadog         | 3.7.1   |
    | apache-airflow-providers-elasticsearch   | 5.5.2   |
    | apache-airflow-providers-fab             | 1.5.0   |
    | apache-airflow-providers-ftp             | 3.11.1  |
    | apache-airflow-providers-google          | 10.25.0 |
    | apache-airflow-providers-http            | 4.13.2  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 10.5.1  |
    | apache-airflow-providers-mysql           | 5.7.3   |
    | apache-airflow-providers-openlineage     | 1.13.0  |
    | apache-airflow-providers-postgres        | 5.13.1  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.0   |
    | apache-airflow-providers-sqlite          | 3.9.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.3   |
    | apache-airflow-providers-common-compat | 1.2.1   |
    | apache-airflow-providers-common-io     | 1.4.2   |
    | apache-airflow-providers-common-sql    | 1.19.0  |
    | apache-airflow-providers-elasticsearch | 5.5.2   |
    | apache-airflow-providers-fab           | 1.5.0   |
    | apache-airflow-providers-ftp           | 3.11.1  |
    | apache-airflow-providers-http          | 4.13.2  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.3   |
    | apache-airflow-providers-openlineage   | 1.13.0  |
    | apache-airflow-providers-postgres      | 5.13.1  |
    | apache-airflow-providers-smtp          | 1.8.0   |
    | apache-airflow-providers-sqlite        | 3.9.0   |
    | astronomer-providers-logging           | 1.5.2   |
  </Tab>
</Tabs>

## Astro Runtime 12.2.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.3   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.1   |
    | apache-airflow-providers-common-compat   | 1.2.1   |
    | apache-airflow-providers-common-io       | 1.4.2   |
    | apache-airflow-providers-common-sql      | 1.18.0  |
    | apache-airflow-providers-datadog         | 3.7.1   |
    | apache-airflow-providers-elasticsearch   | 5.5.2   |
    | apache-airflow-providers-fab             | 1.3.0   |
    | apache-airflow-providers-ftp             | 3.11.1  |
    | apache-airflow-providers-google          | 10.24.0 |
    | apache-airflow-providers-http            | 4.13.1  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 10.5.1  |
    | apache-airflow-providers-mysql           | 5.7.2   |
    | apache-airflow-providers-openlineage     | 1.12.2  |
    | apache-airflow-providers-postgres        | 5.13.1  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.0   |
    | apache-airflow-providers-sqlite          | 3.9.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.3   |
    | apache-airflow-providers-common-compat | 1.2.1   |
    | apache-airflow-providers-common-io     | 1.4.2   |
    | apache-airflow-providers-common-sql    | 1.18.0  |
    | apache-airflow-providers-elasticsearch | 5.5.2   |
    | apache-airflow-providers-fab           | 1.3.0   |
    | apache-airflow-providers-ftp           | 3.11.1  |
    | apache-airflow-providers-http          | 4.13.1  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.2   |
    | apache-airflow-providers-openlineage   | 1.12.2  |
    | apache-airflow-providers-postgres      | 5.13.1  |
    | apache-airflow-providers-smtp          | 1.8.0   |
    | apache-airflow-providers-sqlite        | 3.9.0   |
    | astronomer-providers-logging           | 1.5.2   |
  </Tab>
</Tabs>

## Astro Runtime 12.1.1

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.1   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.1   |
    | apache-airflow-providers-common-compat   | 1.2.0   |
    | apache-airflow-providers-common-io       | 1.4.0   |
    | apache-airflow-providers-common-sql      | 1.16.0  |
    | apache-airflow-providers-datadog         | 3.7.0   |
    | apache-airflow-providers-elasticsearch   | 5.5.0   |
    | apache-airflow-providers-fab             | 1.3.0   |
    | apache-airflow-providers-ftp             | 3.11.0  |
    | apache-airflow-providers-google          | 10.22.0 |
    | apache-airflow-providers-http            | 4.13.0  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 10.4.0  |
    | apache-airflow-providers-mysql           | 5.7.0   |
    | apache-airflow-providers-openlineage     | 1.11.0  |
    | apache-airflow-providers-postgres        | 5.12.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.0   |
    | apache-airflow-providers-sqlite          | 3.9.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.1   |
    | apache-airflow-providers-common-compat | 1.2.0   |
    | apache-airflow-providers-common-io     | 1.4.0   |
    | apache-airflow-providers-common-sql    | 1.16.0  |
    | apache-airflow-providers-elasticsearch | 5.5.0   |
    | apache-airflow-providers-fab           | 1.3.0   |
    | apache-airflow-providers-ftp           | 3.11.0  |
    | apache-airflow-providers-http          | 4.13.0  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.0   |
    | apache-airflow-providers-openlineage   | 1.11.0  |
    | apache-airflow-providers-postgres      | 5.12.0  |
    | apache-airflow-providers-smtp          | 1.8.0   |
    | apache-airflow-providers-sqlite        | 3.9.0   |
    | astronomer-providers-logging           | 1.5.2   |
  </Tab>
</Tabs>

## Astro Runtime 12.1.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.1   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.1   |
    | apache-airflow-providers-common-compat   | 1.2.0   |
    | apache-airflow-providers-common-io       | 1.4.0   |
    | apache-airflow-providers-common-sql      | 1.16.0  |
    | apache-airflow-providers-datadog         | 3.7.0   |
    | apache-airflow-providers-elasticsearch   | 5.5.0   |
    | apache-airflow-providers-fab             | 1.3.0   |
    | apache-airflow-providers-ftp             | 3.11.0  |
    | apache-airflow-providers-google          | 10.22.0 |
    | apache-airflow-providers-http            | 4.13.0  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 10.4.0  |
    | apache-airflow-providers-mysql           | 5.7.0   |
    | apache-airflow-providers-openlineage     | 1.11.0  |
    | apache-airflow-providers-postgres        | 5.12.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.0   |
    | apache-airflow-providers-sqlite          | 3.9.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.1   |
    | apache-airflow-providers-common-compat | 1.2.0   |
    | apache-airflow-providers-common-io     | 1.4.0   |
    | apache-airflow-providers-common-sql    | 1.16.0  |
    | apache-airflow-providers-elasticsearch | 5.5.0   |
    | apache-airflow-providers-fab           | 1.3.0   |
    | apache-airflow-providers-ftp           | 3.11.0  |
    | apache-airflow-providers-http          | 4.13.0  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.0   |
    | apache-airflow-providers-openlineage   | 1.11.0  |
    | apache-airflow-providers-postgres      | 5.12.0  |
    | apache-airflow-providers-smtp          | 1.8.0   |
    | apache-airflow-providers-sqlite        | 3.9.0   |
    | astronomer-providers-logging           | 1.5.2   |
  </Tab>
</Tabs>

## Astro Runtime 12.0.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.27.0  |
    | apache-airflow-providers-celery          | 3.7.3   |
    | apache-airflow-providers-cncf-kubernetes | 8.3.4   |
    | apache-airflow-providers-common-compat   | 1.1.0   |
    | apache-airflow-providers-common-io       | 1.4.0   |
    | apache-airflow-providers-common-sql      | 1.15.0  |
    | apache-airflow-providers-datadog         | 3.6.1   |
    | apache-airflow-providers-elasticsearch   | 5.4.2   |
    | apache-airflow-providers-fab             | 1.2.2   |
    | apache-airflow-providers-ftp             | 3.10.1  |
    | apache-airflow-providers-google          | 10.21.1 |
    | apache-airflow-providers-http            | 4.12.0  |
    | apache-airflow-providers-imap            | 3.6.1   |
    | apache-airflow-providers-microsoft-azure | 10.3.0  |
    | apache-airflow-providers-mysql           | 5.6.3   |
    | apache-airflow-providers-openlineage     | 1.8.0   |
    | apache-airflow-providers-postgres        | 5.11.3  |
    | apache-airflow-providers-redis           | 3.7.1   |
    | apache-airflow-providers-smtp            | 1.7.1   |
    | apache-airflow-providers-sqlite          | 3.8.2   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.2  |
    | astronomer-providers-logging             | 1.5.1   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.7.3   |
    | apache-airflow-providers-common-compat | 1.1.0   |
    | apache-airflow-providers-common-io     | 1.4.0   |
    | apache-airflow-providers-common-sql    | 1.15.0  |
    | apache-airflow-providers-elasticsearch | 5.4.2   |
    | apache-airflow-providers-fab           | 1.2.2   |
    | apache-airflow-providers-ftp           | 3.10.1  |
    | apache-airflow-providers-http          | 4.12.0  |
    | apache-airflow-providers-imap          | 3.6.1   |
    | apache-airflow-providers-mysql         | 5.6.3   |
    | apache-airflow-providers-openlineage   | 1.8.0   |
    | apache-airflow-providers-postgres      | 5.11.3  |
    | apache-airflow-providers-smtp          | 1.7.1   |
    | apache-airflow-providers-sqlite        | 3.8.2   |
    | astronomer-providers-logging           | 1.5.1   |
  </Tab>
</Tabs>

## Astro Runtime 11.20.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.29.0  |
    | apache-airflow-providers-celery          | 3.10.6  |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.6.1   |
    | apache-airflow-providers-common-io       | 1.5.4   |
    | apache-airflow-providers-common-sql      | 1.26.0  |
    | apache-airflow-providers-datadog         | 3.8.3   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.5.3   |
    | apache-airflow-providers-ftp             | 3.12.3  |
    | apache-airflow-providers-google          | 10.26.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.8.3   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 2.2.0   |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.9.0   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.3   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.10.6  |
    | apache-airflow-providers-common-compat | 1.6.1   |
    | apache-airflow-providers-common-io     | 1.5.4   |
    | apache-airflow-providers-common-sql    | 1.26.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.5.3   |
    | apache-airflow-providers-ftp           | 3.12.3  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.8.3   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-openlineage   | 2.2.0   |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.9.0   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.5.3   |
  </Tab>
</Tabs>

## Astro Runtime 11.19.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.29.0  |
    | apache-airflow-providers-celery          | 3.10.6  |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.6.1   |
    | apache-airflow-providers-common-io       | 1.5.4   |
    | apache-airflow-providers-common-sql      | 1.26.0  |
    | apache-airflow-providers-datadog         | 3.8.3   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.5.3   |
    | apache-airflow-providers-ftp             | 3.12.3  |
    | apache-airflow-providers-google          | 10.26.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.8.3   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 2.2.0   |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.9.0   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.3   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.10.6  |
    | apache-airflow-providers-common-compat | 1.6.1   |
    | apache-airflow-providers-common-io     | 1.5.4   |
    | apache-airflow-providers-common-sql    | 1.26.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.5.3   |
    | apache-airflow-providers-ftp           | 3.12.3  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.8.3   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-openlineage   | 2.2.0   |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.9.0   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.5.3   |
  </Tab>
</Tabs>

## Astro Runtime 11.18.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.10.6  |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.2.2   |
    | apache-airflow-providers-common-io       | 1.5.4   |
    | apache-airflow-providers-common-sql      | 1.26.0  |
    | apache-airflow-providers-datadog         | 3.8.3   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.4.0   |
    | apache-airflow-providers-ftp             | 3.12.3  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.8.3   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 1.14.0  |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.9.0   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.3   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.10.6  |
    | apache-airflow-providers-common-io     | 1.5.4   |
    | apache-airflow-providers-common-sql    | 1.26.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.4.0   |
    | apache-airflow-providers-ftp           | 3.12.3  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.8.3   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.9.0   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.5.3   |
  </Tab>
</Tabs>

## Astro Runtime 11.17.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.5   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.2.2   |
    | apache-airflow-providers-common-io       | 1.5.0   |
    | apache-airflow-providers-common-sql      | 1.22.0  |
    | apache-airflow-providers-datadog         | 3.8.1   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.4.0   |
    | apache-airflow-providers-ftp             | 3.12.1  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.8.1   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 1.14.0  |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.9.0   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.3   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.5   |
    | apache-airflow-providers-common-io     | 1.5.0   |
    | apache-airflow-providers-common-sql    | 1.22.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.4.0   |
    | apache-airflow-providers-ftp           | 3.12.1  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.8.1   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.9.0   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.5.3   |
  </Tab>
</Tabs>

## Astro Runtime 11.16.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.5   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.2.2   |
    | apache-airflow-providers-common-io       | 1.4.2   |
    | apache-airflow-providers-common-sql      | 1.21.0  |
    | apache-airflow-providers-datadog         | 3.8.0   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.4.0   |
    | apache-airflow-providers-ftp             | 3.12.0  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.8.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 1.14.0  |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.9.0   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.3   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.5   |
    | apache-airflow-providers-common-io     | 1.4.2   |
    | apache-airflow-providers-common-sql    | 1.21.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.4.0   |
    | apache-airflow-providers-ftp           | 3.12.0  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.8.0   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.9.0   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.5.3   |
  </Tab>
</Tabs>

## Astro Runtime 11.15.1

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.5   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.2.2   |
    | apache-airflow-providers-common-io       | 1.4.2   |
    | apache-airflow-providers-common-sql      | 1.20.0  |
    | apache-airflow-providers-datadog         | 3.7.1   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.4.0   |
    | apache-airflow-providers-ftp             | 3.11.1  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 1.14.0  |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.1   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.3   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.5   |
    | apache-airflow-providers-common-io     | 1.4.2   |
    | apache-airflow-providers-common-sql    | 1.20.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.4.0   |
    | apache-airflow-providers-ftp           | 3.11.1  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.8.1   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.5.3   |
  </Tab>
</Tabs>

## Astro Runtime 11.15.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.5   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.2.2   |
    | apache-airflow-providers-common-io       | 1.4.2   |
    | apache-airflow-providers-common-sql      | 1.20.0  |
    | apache-airflow-providers-datadog         | 3.7.1   |
    | apache-airflow-providers-elasticsearch   | 5.5.3   |
    | apache-airflow-providers-fab             | 1.4.0   |
    | apache-airflow-providers-ftp             | 3.11.1  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.13.3  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.4   |
    | apache-airflow-providers-openlineage     | 1.14.0  |
    | apache-airflow-providers-postgres        | 5.14.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.1   |
    | apache-airflow-providers-sqlite          | 3.9.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.3   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.5   |
    | apache-airflow-providers-common-io     | 1.4.2   |
    | apache-airflow-providers-common-sql    | 1.20.0  |
    | apache-airflow-providers-elasticsearch | 5.5.3   |
    | apache-airflow-providers-fab           | 1.4.0   |
    | apache-airflow-providers-ftp           | 3.11.1  |
    | apache-airflow-providers-http          | 4.13.3  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.4   |
    | apache-airflow-providers-postgres      | 5.14.0  |
    | apache-airflow-providers-smtp          | 1.8.1   |
    | apache-airflow-providers-sqlite        | 3.9.1   |
    | astronomer-providers-logging           | 1.5.3   |
  </Tab>
</Tabs>

## Astro Runtime 11.14.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.3   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.2.1   |
    | apache-airflow-providers-common-io       | 1.4.2   |
    | apache-airflow-providers-common-sql      | 1.19.0  |
    | apache-airflow-providers-datadog         | 3.7.1   |
    | apache-airflow-providers-elasticsearch   | 5.5.2   |
    | apache-airflow-providers-fab             | 1.4.0   |
    | apache-airflow-providers-ftp             | 3.11.1  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.13.2  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.3   |
    | apache-airflow-providers-openlineage     | 1.13.0  |
    | apache-airflow-providers-postgres        | 5.13.1  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.0   |
    | apache-airflow-providers-sqlite          | 3.9.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.3   |
    | apache-airflow-providers-common-io     | 1.4.2   |
    | apache-airflow-providers-common-sql    | 1.19.0  |
    | apache-airflow-providers-elasticsearch | 5.5.2   |
    | apache-airflow-providers-fab           | 1.4.0   |
    | apache-airflow-providers-ftp           | 3.11.1  |
    | apache-airflow-providers-http          | 4.13.2  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.3   |
    | apache-airflow-providers-postgres      | 5.13.1  |
    | apache-airflow-providers-smtp          | 1.8.0   |
    | apache-airflow-providers-sqlite        | 3.9.0   |
    | astronomer-providers-logging           | 1.5.2   |
  </Tab>
</Tabs>

## Astro Runtime 11.13.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.3   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.2.1   |
    | apache-airflow-providers-common-io       | 1.4.2   |
    | apache-airflow-providers-common-sql      | 1.18.0  |
    | apache-airflow-providers-datadog         | 3.7.1   |
    | apache-airflow-providers-elasticsearch   | 5.5.2   |
    | apache-airflow-providers-fab             | 1.4.0   |
    | apache-airflow-providers-ftp             | 3.11.1  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.13.1  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.2   |
    | apache-airflow-providers-openlineage     | 1.12.2  |
    | apache-airflow-providers-postgres        | 5.13.1  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.0   |
    | apache-airflow-providers-sqlite          | 3.9.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.3   |
    | apache-airflow-providers-common-io     | 1.4.2   |
    | apache-airflow-providers-common-sql    | 1.18.0  |
    | apache-airflow-providers-elasticsearch | 5.5.2   |
    | apache-airflow-providers-fab           | 1.4.0   |
    | apache-airflow-providers-ftp           | 3.11.1  |
    | apache-airflow-providers-http          | 4.13.1  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.2   |
    | apache-airflow-providers-postgres      | 5.13.1  |
    | apache-airflow-providers-smtp          | 1.8.0   |
    | apache-airflow-providers-sqlite        | 3.9.0   |
    | astronomer-providers-logging           | 1.5.2   |
  </Tab>
</Tabs>

## Astro Runtime 11.12.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.3   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.2.1   |
    | apache-airflow-providers-common-io       | 1.4.1   |
    | apache-airflow-providers-common-sql      | 1.17.1  |
    | apache-airflow-providers-datadog         | 3.7.1   |
    | apache-airflow-providers-elasticsearch   | 5.5.2   |
    | apache-airflow-providers-fab             | 1.4.0   |
    | apache-airflow-providers-ftp             | 3.11.1  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.13.1  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.2   |
    | apache-airflow-providers-openlineage     | 1.12.2  |
    | apache-airflow-providers-postgres        | 5.13.1  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.0   |
    | apache-airflow-providers-sqlite          | 3.9.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.3   |
    | apache-airflow-providers-common-io     | 1.4.1   |
    | apache-airflow-providers-common-sql    | 1.17.1  |
    | apache-airflow-providers-elasticsearch | 5.5.2   |
    | apache-airflow-providers-fab           | 1.4.0   |
    | apache-airflow-providers-ftp           | 3.11.1  |
    | apache-airflow-providers-http          | 4.13.1  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.2   |
    | apache-airflow-providers-postgres      | 5.13.1  |
    | apache-airflow-providers-smtp          | 1.8.0   |
    | apache-airflow-providers-sqlite        | 3.9.0   |
    | astronomer-providers-logging           | 1.5.2   |
  </Tab>
</Tabs>

## Astro Runtime 11.11.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.2   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.2   |
    | apache-airflow-providers-common-compat   | 1.2.0   |
    | apache-airflow-providers-common-io       | 1.4.1   |
    | apache-airflow-providers-common-sql      | 1.17.1  |
    | apache-airflow-providers-datadog         | 3.7.1   |
    | apache-airflow-providers-elasticsearch   | 5.5.1   |
    | apache-airflow-providers-fab             | 1.4.0   |
    | apache-airflow-providers-ftp             | 3.11.1  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.13.1  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.1   |
    | apache-airflow-providers-openlineage     | 1.12.1  |
    | apache-airflow-providers-postgres        | 5.13.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.0   |
    | apache-airflow-providers-sqlite          | 3.9.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.2   |
    | apache-airflow-providers-common-io     | 1.4.1   |
    | apache-airflow-providers-common-sql    | 1.17.1  |
    | apache-airflow-providers-elasticsearch | 5.5.1   |
    | apache-airflow-providers-fab           | 1.4.0   |
    | apache-airflow-providers-ftp           | 3.11.1  |
    | apache-airflow-providers-http          | 4.13.1  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.1   |
    | apache-airflow-providers-postgres      | 5.13.0  |
    | apache-airflow-providers-smtp          | 1.8.0   |
    | apache-airflow-providers-sqlite        | 3.9.0   |
    | astronomer-providers-logging           | 1.5.2   |
  </Tab>
</Tabs>

## Astro Runtime 11.10.1

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.1   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.1   |
    | apache-airflow-providers-common-compat   | 1.2.0   |
    | apache-airflow-providers-common-io       | 1.4.0   |
    | apache-airflow-providers-common-sql      | 1.16.0  |
    | apache-airflow-providers-datadog         | 3.7.0   |
    | apache-airflow-providers-elasticsearch   | 5.5.0   |
    | apache-airflow-providers-fab             | 1.3.0   |
    | apache-airflow-providers-ftp             | 3.11.0  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.13.0  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.0   |
    | apache-airflow-providers-openlineage     | 1.11.0  |
    | apache-airflow-providers-postgres        | 5.12.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.0   |
    | apache-airflow-providers-sqlite          | 3.9.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.1   |
    | apache-airflow-providers-common-io     | 1.4.0   |
    | apache-airflow-providers-common-sql    | 1.16.0  |
    | apache-airflow-providers-elasticsearch | 5.5.0   |
    | apache-airflow-providers-fab           | 1.3.0   |
    | apache-airflow-providers-ftp           | 3.11.0  |
    | apache-airflow-providers-http          | 4.13.0  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.0   |
    | apache-airflow-providers-postgres      | 5.12.0  |
    | apache-airflow-providers-smtp          | 1.8.0   |
    | apache-airflow-providers-sqlite        | 3.9.0   |
    | astronomer-providers-logging           | 1.5.2   |
  </Tab>
</Tabs>

## Astro Runtime 11.10.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.28.0  |
    | apache-airflow-providers-celery          | 3.8.1   |
    | apache-airflow-providers-cncf-kubernetes | 8.4.1   |
    | apache-airflow-providers-common-compat   | 1.2.0   |
    | apache-airflow-providers-common-io       | 1.4.0   |
    | apache-airflow-providers-common-sql      | 1.16.0  |
    | apache-airflow-providers-datadog         | 3.7.0   |
    | apache-airflow-providers-elasticsearch   | 5.5.0   |
    | apache-airflow-providers-fab             | 1.3.0   |
    | apache-airflow-providers-ftp             | 3.11.0  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.13.0  |
    | apache-airflow-providers-imap            | 3.7.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.7.0   |
    | apache-airflow-providers-openlineage     | 1.11.0  |
    | apache-airflow-providers-postgres        | 5.12.0  |
    | apache-airflow-providers-redis           | 3.8.0   |
    | apache-airflow-providers-smtp            | 1.8.0   |
    | apache-airflow-providers-sqlite          | 3.9.0   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.4  |
    | astronomer-providers-logging             | 1.5.2   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.8.1   |
    | apache-airflow-providers-common-io     | 1.4.0   |
    | apache-airflow-providers-common-sql    | 1.16.0  |
    | apache-airflow-providers-elasticsearch | 5.5.0   |
    | apache-airflow-providers-fab           | 1.3.0   |
    | apache-airflow-providers-ftp           | 3.11.0  |
    | apache-airflow-providers-http          | 4.13.0  |
    | apache-airflow-providers-imap          | 3.7.0   |
    | apache-airflow-providers-mysql         | 5.7.0   |
    | apache-airflow-providers-postgres      | 5.12.0  |
    | apache-airflow-providers-smtp          | 1.8.0   |
    | apache-airflow-providers-sqlite        | 3.9.0   |
    | astronomer-providers-logging           | 1.5.2   |
  </Tab>
</Tabs>

## Astro Runtime 11.9.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.27.0  |
    | apache-airflow-providers-celery          | 3.7.3   |
    | apache-airflow-providers-cncf-kubernetes | 8.3.4   |
    | apache-airflow-providers-common-compat   | 1.1.0   |
    | apache-airflow-providers-common-io       | 1.4.0   |
    | apache-airflow-providers-common-sql      | 1.15.0  |
    | apache-airflow-providers-datadog         | 3.6.1   |
    | apache-airflow-providers-elasticsearch   | 5.4.2   |
    | apache-airflow-providers-fab             | 1.2.2   |
    | apache-airflow-providers-ftp             | 3.10.1  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.12.0  |
    | apache-airflow-providers-imap            | 3.6.1   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.6.3   |
    | apache-airflow-providers-openlineage     | 1.8.0   |
    | apache-airflow-providers-postgres        | 5.11.3  |
    | apache-airflow-providers-redis           | 3.7.1   |
    | apache-airflow-providers-smtp            | 1.7.1   |
    | apache-airflow-providers-sqlite          | 3.8.2   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.2  |
    | astronomer-providers-logging             | 1.5.1   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.7.3   |
    | apache-airflow-providers-common-io     | 1.4.0   |
    | apache-airflow-providers-common-sql    | 1.15.0  |
    | apache-airflow-providers-elasticsearch | 5.4.2   |
    | apache-airflow-providers-fab           | 1.2.2   |
    | apache-airflow-providers-ftp           | 3.10.1  |
    | apache-airflow-providers-http          | 4.12.0  |
    | apache-airflow-providers-imap          | 3.6.1   |
    | apache-airflow-providers-mysql         | 5.6.3   |
    | apache-airflow-providers-postgres      | 5.11.3  |
    | apache-airflow-providers-smtp          | 1.7.1   |
    | apache-airflow-providers-sqlite        | 3.8.2   |
    | astronomer-providers-logging           | 1.5.1   |
  </Tab>
</Tabs>

## Astro Runtime 11.8.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.27.0  |
    | apache-airflow-providers-celery          | 3.7.3   |
    | apache-airflow-providers-cncf-kubernetes | 8.3.4   |
    | apache-airflow-providers-common-compat   | 1.1.0   |
    | apache-airflow-providers-common-io       | 1.4.0   |
    | apache-airflow-providers-common-sql      | 1.15.0  |
    | apache-airflow-providers-datadog         | 3.6.1   |
    | apache-airflow-providers-elasticsearch   | 5.4.2   |
    | apache-airflow-providers-fab             | 1.2.2   |
    | apache-airflow-providers-ftp             | 3.10.1  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.12.0  |
    | apache-airflow-providers-imap            | 3.6.1   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.6.3   |
    | apache-airflow-providers-openlineage     | 1.10.0  |
    | apache-airflow-providers-postgres        | 5.11.3  |
    | apache-airflow-providers-redis           | 3.7.1   |
    | apache-airflow-providers-smtp            | 1.7.1   |
    | apache-airflow-providers-sqlite          | 3.8.2   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.2  |
    | astronomer-providers-logging             | 1.5.1   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.7.3   |
    | apache-airflow-providers-common-io     | 1.4.0   |
    | apache-airflow-providers-common-sql    | 1.15.0  |
    | apache-airflow-providers-elasticsearch | 5.4.2   |
    | apache-airflow-providers-fab           | 1.2.2   |
    | apache-airflow-providers-ftp           | 3.10.1  |
    | apache-airflow-providers-http          | 4.12.0  |
    | apache-airflow-providers-imap          | 3.6.1   |
    | apache-airflow-providers-mysql         | 5.6.3   |
    | apache-airflow-providers-postgres      | 5.11.3  |
    | apache-airflow-providers-smtp          | 1.7.1   |
    | apache-airflow-providers-sqlite        | 3.8.2   |
    | astronomer-providers-logging           | 1.5.1   |
  </Tab>
</Tabs>

## Astro Runtime 11.7.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.26.0  |
    | apache-airflow-providers-celery          | 3.7.2   |
    | apache-airflow-providers-cncf-kubernetes | 8.3.3   |
    | apache-airflow-providers-common-io       | 1.3.2   |
    | apache-airflow-providers-common-sql      | 1.14.2  |
    | apache-airflow-providers-datadog         | 3.6.1   |
    | apache-airflow-providers-elasticsearch   | 5.4.1   |
    | apache-airflow-providers-fab             | 1.2.1   |
    | apache-airflow-providers-ftp             | 3.10.0  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.12.0  |
    | apache-airflow-providers-imap            | 3.6.1   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.6.2   |
    | apache-airflow-providers-openlineage     | 1.9.1   |
    | apache-airflow-providers-postgres        | 5.11.2  |
    | apache-airflow-providers-redis           | 3.7.1   |
    | apache-airflow-providers-smtp            | 1.7.1   |
    | apache-airflow-providers-sqlite          | 3.8.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.1  |
    | astronomer-providers-logging             | 1.5.1   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.7.2   |
    | apache-airflow-providers-common-io     | 1.3.2   |
    | apache-airflow-providers-common-sql    | 1.14.2  |
    | apache-airflow-providers-elasticsearch | 5.4.1   |
    | apache-airflow-providers-fab           | 1.2.1   |
    | apache-airflow-providers-ftp           | 3.10.0  |
    | apache-airflow-providers-http          | 4.12.0  |
    | apache-airflow-providers-imap          | 3.6.1   |
    | apache-airflow-providers-mysql         | 5.6.2   |
    | apache-airflow-providers-postgres      | 5.11.2  |
    | apache-airflow-providers-smtp          | 1.7.1   |
    | apache-airflow-providers-sqlite        | 3.8.1   |
    | astronomer-providers-logging           | 1.5.1   |
  </Tab>
</Tabs>

## Astro Runtime 11.6.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.25.0  |
    | apache-airflow-providers-celery          | 3.7.2   |
    | apache-airflow-providers-cncf-kubernetes | 8.3.2   |
    | apache-airflow-providers-common-io       | 1.3.2   |
    | apache-airflow-providers-common-sql      | 1.14.1  |
    | apache-airflow-providers-datadog         | 3.6.1   |
    | apache-airflow-providers-elasticsearch   | 5.4.1   |
    | apache-airflow-providers-fab             | 1.2.0   |
    | apache-airflow-providers-ftp             | 3.10.0  |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.12.0  |
    | apache-airflow-providers-imap            | 3.6.1   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.6.2   |
    | apache-airflow-providers-openlineage     | 1.8.0   |
    | apache-airflow-providers-postgres        | 5.11.2  |
    | apache-airflow-providers-redis           | 3.7.1   |
    | apache-airflow-providers-smtp            | 1.7.1   |
    | apache-airflow-providers-sqlite          | 3.8.1   |
    | astro-sdk-python                         | 1.8.1   |
    | astronomer-providers                     | 1.19.1  |
    | astronomer-providers-logging             | 1.5.1   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.7.2   |
    | apache-airflow-providers-common-io     | 1.3.2   |
    | apache-airflow-providers-common-sql    | 1.14.1  |
    | apache-airflow-providers-elasticsearch | 5.4.1   |
    | apache-airflow-providers-fab           | 1.2.0   |
    | apache-airflow-providers-ftp           | 3.10.0  |
    | apache-airflow-providers-http          | 4.12.0  |
    | apache-airflow-providers-imap          | 3.6.1   |
    | apache-airflow-providers-mysql         | 5.6.2   |
    | apache-airflow-providers-postgres      | 5.11.2  |
    | apache-airflow-providers-smtp          | 1.7.1   |
    | apache-airflow-providers-sqlite        | 3.8.1   |
    | astronomer-providers-logging           | 1.5.1   |
  </Tab>
</Tabs>

## Astro Runtime 11.5.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.24.0  |
    | apache-airflow-providers-celery          | 3.7.2   |
    | apache-airflow-providers-cncf-kubernetes | 8.3.1   |
    | apache-airflow-providers-common-io       | 1.3.2   |
    | apache-airflow-providers-common-sql      | 1.14.0  |
    | apache-airflow-providers-datadog         | 3.6.1   |
    | apache-airflow-providers-elasticsearch   | 5.4.1   |
    | apache-airflow-providers-fab             | 1.1.1   |
    | apache-airflow-providers-ftp             | 3.9.1   |
    | apache-airflow-providers-google          | 10.19.0 |
    | apache-airflow-providers-http            | 4.11.1  |
    | apache-airflow-providers-imap            | 3.6.1   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.6.1   |
    | apache-airflow-providers-openlineage     | 1.8.0   |
    | apache-airflow-providers-postgres        | 5.11.1  |
    | apache-airflow-providers-redis           | 3.7.1   |
    | apache-airflow-providers-smtp            | 1.7.1   |
    | apache-airflow-providers-sqlite          | 3.8.1   |
    | astro-sdk-python                         | 1.8.0   |
    | astronomer-providers                     | 1.19.1  |
    | astronomer-providers-logging             | 1.5.1   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.7.2   |
    | apache-airflow-providers-common-io     | 1.3.2   |
    | apache-airflow-providers-common-sql    | 1.14.0  |
    | apache-airflow-providers-elasticsearch | 5.4.1   |
    | apache-airflow-providers-fab           | 1.1.1   |
    | apache-airflow-providers-ftp           | 3.9.1   |
    | apache-airflow-providers-http          | 4.11.1  |
    | apache-airflow-providers-imap          | 3.6.1   |
    | apache-airflow-providers-mysql         | 5.6.1   |
    | apache-airflow-providers-postgres      | 5.11.1  |
    | apache-airflow-providers-smtp          | 1.7.1   |
    | apache-airflow-providers-sqlite        | 3.8.1   |
    | astronomer-providers-logging           | 1.5.1   |
  </Tab>
</Tabs>

## Astro Runtime 11.4.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.22.0  |
    | apache-airflow-providers-celery          | 3.7.0   |
    | apache-airflow-providers-cncf-kubernetes | 8.2.0   |
    | apache-airflow-providers-common-io       | 1.3.1   |
    | apache-airflow-providers-common-sql      | 1.13.0  |
    | apache-airflow-providers-datadog         | 3.6.0   |
    | apache-airflow-providers-elasticsearch   | 5.4.0   |
    | apache-airflow-providers-fab             | 1.1.0   |
    | apache-airflow-providers-ftp             | 3.9.0   |
    | apache-airflow-providers-google          | 10.18.0 |
    | apache-airflow-providers-http            | 4.11.0  |
    | apache-airflow-providers-imap            | 3.6.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.6.0   |
    | apache-airflow-providers-openlineage     | 1.7.1   |
    | apache-airflow-providers-postgres        | 5.11.0  |
    | apache-airflow-providers-redis           | 3.7.0   |
    | apache-airflow-providers-smtp            | 1.7.0   |
    | apache-airflow-providers-sqlite          | 3.8.0   |
    | astro-sdk-python                         | 1.8.0   |
    | astronomer-providers                     | 1.19.1  |
    | astronomer-providers-logging             | 1.4.7   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.7.0   |
    | apache-airflow-providers-common-io     | 1.3.1   |
    | apache-airflow-providers-common-sql    | 1.13.0  |
    | apache-airflow-providers-elasticsearch | 5.4.0   |
    | apache-airflow-providers-fab           | 1.1.0   |
    | apache-airflow-providers-ftp           | 3.9.0   |
    | apache-airflow-providers-http          | 4.11.0  |
    | apache-airflow-providers-imap          | 3.6.0   |
    | apache-airflow-providers-mysql         | 5.6.0   |
    | apache-airflow-providers-postgres      | 5.11.0  |
    | apache-airflow-providers-smtp          | 1.7.0   |
    | apache-airflow-providers-sqlite        | 3.8.0   |
    | astronomer-providers-logging           | 1.4.7   |
  </Tab>
</Tabs>

## Astro Runtime 11.3.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.20.0  |
    | apache-airflow-providers-celery          | 3.7.0   |
    | apache-airflow-providers-cncf-kubernetes | 8.0.1   |
    | apache-airflow-providers-common-io       | 1.3.1   |
    | apache-airflow-providers-common-sql      | 1.13.0  |
    | apache-airflow-providers-datadog         | 3.6.0   |
    | apache-airflow-providers-elasticsearch   | 5.4.0   |
    | apache-airflow-providers-fab             | 1.1.0   |
    | apache-airflow-providers-ftp             | 3.9.0   |
    | apache-airflow-providers-google          | 10.17.0 |
    | apache-airflow-providers-http            | 4.11.0  |
    | apache-airflow-providers-imap            | 3.6.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.6.0   |
    | apache-airflow-providers-openlineage     | 1.7.1   |
    | apache-airflow-providers-postgres        | 5.11.0  |
    | apache-airflow-providers-redis           | 3.7.0   |
    | apache-airflow-providers-smtp            | 1.7.0   |
    | apache-airflow-providers-sqlite          | 3.8.0   |
    | astro-sdk-python                         | 1.8.0   |
    | astronomer-providers                     | 1.19.0  |
    | astronomer-providers-logging             | 1.4.7   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.7.0   |
    | apache-airflow-providers-common-io     | 1.3.1   |
    | apache-airflow-providers-common-sql    | 1.13.0  |
    | apache-airflow-providers-elasticsearch | 5.4.0   |
    | apache-airflow-providers-fab           | 1.1.0   |
    | apache-airflow-providers-ftp           | 3.9.0   |
    | apache-airflow-providers-http          | 4.11.0  |
    | apache-airflow-providers-imap          | 3.6.0   |
    | apache-airflow-providers-mysql         | 5.6.0   |
    | apache-airflow-providers-postgres      | 5.11.0  |
    | apache-airflow-providers-smtp          | 1.7.0   |
    | apache-airflow-providers-sqlite        | 3.8.0   |
    | astronomer-providers-logging           | 1.4.7   |
  </Tab>
</Tabs>

## Astro Runtime 11.2.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.20.0  |
    | apache-airflow-providers-celery          | 3.6.2   |
    | apache-airflow-providers-cncf-kubernetes | 8.0.1   |
    | apache-airflow-providers-common-io       | 1.3.1   |
    | apache-airflow-providers-common-sql      | 1.12.0  |
    | apache-airflow-providers-datadog         | 3.5.1   |
    | apache-airflow-providers-elasticsearch   | 5.3.4   |
    | apache-airflow-providers-fab             | 1.0.4   |
    | apache-airflow-providers-ftp             | 3.8.0   |
    | apache-airflow-providers-google          | 10.17.0 |
    | apache-airflow-providers-http            | 4.10.1  |
    | apache-airflow-providers-imap            | 3.5.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-mysql           | 5.5.4   |
    | apache-airflow-providers-openlineage     | 1.7.0   |
    | apache-airflow-providers-postgres        | 5.10.2  |
    | apache-airflow-providers-redis           | 3.6.1   |
    | apache-airflow-providers-smtp            | 1.6.1   |
    | apache-airflow-providers-sqlite          | 3.7.1   |
    | astro-sdk-python                         | 1.8.0   |
    | astronomer-providers                     | 1.19.0  |
    | astronomer-providers-logging             | 1.4.7   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.6.2   |
    | apache-airflow-providers-common-io     | 1.3.1   |
    | apache-airflow-providers-common-sql    | 1.12.0  |
    | apache-airflow-providers-elasticsearch | 5.3.4   |
    | apache-airflow-providers-fab           | 1.0.4   |
    | apache-airflow-providers-ftp           | 3.8.0   |
    | apache-airflow-providers-http          | 4.10.1  |
    | apache-airflow-providers-imap          | 3.5.0   |
    | apache-airflow-providers-mysql         | 5.5.4   |
    | apache-airflow-providers-postgres      | 5.10.2  |
    | apache-airflow-providers-smtp          | 1.6.1   |
    | apache-airflow-providers-sqlite        | 3.7.1   |
    | astronomer-providers-logging           | 1.4.7   |
  </Tab>
</Tabs>

## Astro Runtime 11.1.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.20.0  |
    | apache-airflow-providers-celery          | 3.6.2   |
    | apache-airflow-providers-cncf-kubernetes | 8.0.1   |
    | apache-airflow-providers-common-io       | 1.3.1   |
    | apache-airflow-providers-common-sql      | 1.12.0  |
    | apache-airflow-providers-datadog         | 3.5.1   |
    | apache-airflow-providers-elasticsearch   | 5.3.4   |
    | apache-airflow-providers-fab             | 1.0.3   |
    | apache-airflow-providers-ftp             | 3.8.0   |
    | apache-airflow-providers-google          | 10.17.0 |
    | apache-airflow-providers-http            | 4.10.1  |
    | apache-airflow-providers-imap            | 3.5.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-openlineage     | 1.7.0   |
    | apache-airflow-providers-postgres        | 5.10.2  |
    | apache-airflow-providers-redis           | 3.6.1   |
    | apache-airflow-providers-smtp            | 1.6.1   |
    | apache-airflow-providers-sqlite          | 3.7.1   |
    | astro-sdk-python                         | 1.8.0   |
    | astronomer-providers                     | 1.19.0  |
    | astronomer-providers-logging             | 1.4.7   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.6.2   |
    | apache-airflow-providers-common-io     | 1.3.1   |
    | apache-airflow-providers-common-sql    | 1.12.0  |
    | apache-airflow-providers-elasticsearch | 5.3.4   |
    | apache-airflow-providers-fab           | 1.0.3   |
    | apache-airflow-providers-ftp           | 3.8.0   |
    | apache-airflow-providers-http          | 4.10.1  |
    | apache-airflow-providers-imap          | 3.5.0   |
    | apache-airflow-providers-postgres      | 5.10.2  |
    | apache-airflow-providers-smtp          | 1.6.1   |
    | apache-airflow-providers-sqlite        | 3.7.1   |
    | astronomer-providers-logging           | 1.4.7   |
  </Tab>
</Tabs>

## Astro Runtime 11.0.0

<Tabs>
  <Tab title="Standard">
    | Package Name                             | Version |
    | :--------------------------------------- | :------ |
    | apache-airflow-providers-amazon          | 8.19.0  |
    | apache-airflow-providers-celery          | 3.6.1   |
    | apache-airflow-providers-cncf-kubernetes | 8.0.1   |
    | apache-airflow-providers-common-io       | 1.3.0   |
    | apache-airflow-providers-common-sql      | 1.11.1  |
    | apache-airflow-providers-datadog         | 3.5.1   |
    | apache-airflow-providers-elasticsearch   | 5.3.3   |
    | apache-airflow-providers-fab             | 1.0.2   |
    | apache-airflow-providers-ftp             | 3.7.0   |
    | apache-airflow-providers-google          | 10.16.0 |
    | apache-airflow-providers-http            | 4.10.0  |
    | apache-airflow-providers-imap            | 3.5.0   |
    | apache-airflow-providers-microsoft-azure | 9.0.1   |
    | apache-airflow-providers-openlineage     | 1.6.0   |
    | apache-airflow-providers-postgres        | 5.10.2  |
    | apache-airflow-providers-redis           | 3.6.0   |
    | apache-airflow-providers-smtp            | 1.6.1   |
    | apache-airflow-providers-sqlite          | 3.7.1   |
    | astro-sdk-python                         | 1.8.0   |
    | astronomer-providers                     | 1.19.0  |
    | astronomer-providers-logging             | 1.4.7   |
  </Tab>

  <Tab title="Slim">
    | Package Name                           | Version |
    | :------------------------------------- | :------ |
    | apache-airflow-providers-celery        | 3.6.1   |
    | apache-airflow-providers-common-io     | 1.3.0   |
    | apache-airflow-providers-common-sql    | 1.11.1  |
    | apache-airflow-providers-elasticsearch | 5.3.3   |
    | apache-airflow-providers-fab           | 1.0.2   |
    | apache-airflow-providers-ftp           | 3.7.0   |
    | apache-airflow-providers-http          | 4.10.0  |
    | apache-airflow-providers-imap          | 3.5.0   |
    | apache-airflow-providers-postgres      | 5.10.2  |
    | apache-airflow-providers-smtp          | 1.6.1   |
    | apache-airflow-providers-sqlite        | 3.7.1   |
    | astronomer-providers-logging           | 1.4.7   |
  </Tab>
</Tabs>


# Astro Runtime release notes
Source: https://astronomer.io/docs/runtime/runtime-release-notes

A summary of the latest Astro Runtime features and functionality. Astro Runtime is a Docker image built by Astronomer that provides a differentiated Apache Airflow experience and execution framework.

<Tip>[Subscribe to Astro Runtime release notes](/docs/astro/release-notes-subscribe) to receive updates via RSS, email, or Slack.</Tip>

Astro Runtime is a Docker image built and published by Astronomer that extends the Apache Airflow project to provide a differentiated data orchestration experience. This document provides a summary of changes made to each available version of Astro Runtime.

To upgrade Astro Runtime, see either [Upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime) or [Upgrade Astro Private Cloud Runtime](/docs/runtime/manage-airflow-versions). For general product release notes, see [Astro Release Notes](/docs/astro/release-notes). If you have any questions or a bug to report, contact [Astronomer support](https://cloud.astronomer.io/open-support-request).

For release notes and provider package reference of Astro Runtime versions 10.x and older, see [Astro Runtime docs archive](https://github.com/astronomer/astronomer-docs-resources/tree/main/astro/runtime).

<Info>
  Because Astronomer has separate [maintenance life cycles](/docs/runtime/runtime-version-lifecycle-policy) for each major version of Astro Runtime, the same change can be introduced multiple times across major versions, resulting in multiple identical release notes. When a new major version releases, such as Astro Runtime 8.0.0, all changes from previously released versions are included in the new major version.

  If you're upgrading to receive a specific change, ensure the release note for the change appears either:

  * Within your target major version.
  * In any minor or patch version that was released before the first release (`X.0.0`) of your target major version. For example, a change in Astro Runtime 9.9.0, which released January 10 2024, is not guaranteed to appear in Runtime 10.0.0, which released December 8 2023, unless there is a release note for it in a subsequent Runtime 10 patch. However, a change in Astro Runtime 9.6.0, which released November 30 2023, is guaranteed to exist in Runtime 10.0.0 because 9.6.0 was released prior to 10.0.0.
</Info>

<Update label="Astro Runtime 3.3-5" description="August 21, 2026">
  * Airflow version: 3.3.1
  * Python versions: 3.12 - 3.14 (default: 3.14)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.3-5`

  ### Additional improvements

  * Upgrade to Airflow `3.3.1+astro.3`, which includes:
    * Bound the scheduler's deserialized Dag cache ([#71704](https://github.com/apache/airflow/pull/71704))
    * Stop the Dag processor warning on every normalized file path ([#71091](https://github.com/apache/airflow/pull/71091))
    * Fix Dag callbacks silently dropped when version inflation check blocks parsing ([#70987](https://github.com/apache/airflow/pull/70987))
</Update>

<Update label="Astro Runtime 3.3-4" description="August 18, 2026">
  * Airflow version: 3.3.1
  * Python versions: 3.12 - 3.14 (default: 3.14)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.3-4`

  ### Additional improvements

  * Upgrade to Airflow `3.3.1+astro.2`, which includes:
    * Fix cleared tasks getting stuck when a Dag run has no version ([#71696](https://github.com/apache/airflow/pull/71696))

  ### Security fixes

  * Fixed [GHSA-3496-9g83-7v6x](https://github.com/advisories/GHSA-3496-9g83-7v6x)
  * Fixed [GHSA-f2ff-p2ww-7p4p](https://github.com/advisories/GHSA-f2ff-p2ww-7p4p)
  * Fixed [GHSA-prg7-hcfm-mfcr](https://github.com/advisories/GHSA-prg7-hcfm-mfcr)
  * Fixed [GHSA-pwgv-4x5q-6m9f](https://github.com/advisories/GHSA-pwgv-4x5q-6m9f)
</Update>

<Update label="Astro Runtime 3.3-3" description="August 17, 2026">
  * Airflow version: 3.3.1
  * Python versions: 3.12 - 3.14 (default: 3.14)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.3-3`

  ### Additional improvements

  * Upgrade to Airflow `3.3.1+astro.1`
  * Upgrade to Apache Airflow TaskSDK to `1.3.1+astro.1`
  * Upgrade `astronomer-kubernetes-executor` to `10.21.0+astro.1`
  * Upgraded the minor and patch versions of some open-source provider packages. See [Astro Runtime 3.3-3 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-3-3).

  ### Security fixes

  * Fixed [CVE-2026-4360](https://avd.aquasec.com/nvd/cve-2026-4360)
  * Fixed [GHSA-cq5v-8q36-5273](https://github.com/advisories/GHSA-cq5v-8q36-5273)
  * Fixed [GHSA-g6cj-pr64-35w5](https://github.com/advisories/GHSA-g6cj-pr64-35w5)
  * Fixed [GHSA-jwv3-5hgf-82ww](https://github.com/advisories/GHSA-jwv3-5hgf-82ww)
  * Fixed [GHSA-m2h6-j472-rp4c](https://github.com/advisories/GHSA-m2h6-j472-rp4c)
  * Fixed [GHSA-mfx4-hv73-q22v](https://github.com/advisories/GHSA-mfx4-hv73-q22v)
  * Fixed [GHSA-mq44-7p77-q5h7](https://github.com/advisories/GHSA-mq44-7p77-q5h7)
</Update>

<Update label="Astro Runtime 3.3-2" description="July 09, 2026">
  * Airflow version: 3.3.0
  * Python versions: 3.12 - 3.14 (default: 3.14)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.3-2`

  ### Additional improvements

  * Upgrade to Airflow `3.3.0+astro.2`, which includes:
    * Fixed compatibility with the Astro CLI Dag integrity tests.
</Update>

<Update label="Astro Runtime 3.3-1" description="July 09, 2026">
  * Airflow version: 3.3.0
  * Python versions: 3.12 - 3.14 (default: 3.14)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.3-1`

  ### Introducing Airflow 3.3

  Astro Runtime 3.3-1 includes support for Apache Airflow 3.3, which introduces a number of significant new features and improvements, including:

  * **Task & Asset State Store (AIP-103):** Durable key-value state that persists across task retries and Dag runs without an external store.
  * **Language Task SDK (AIP-108):** Author individual tasks in Java or Go while keeping orchestration in Python, with native Airflow logging and XCom support.
  * **Pluggable retry policies (AIP-105):** Define custom retry logic, such as conditional retries based on exception type or custom backoff strategies.
  * **Expanded asset partitioning:** New partition mappers, time windows, and wait policies for controlling downstream fan-out.
  * **Dag Results API:** Designate task outputs as Dag results and retrieve them through the API.
  * **UI enhancements:** Asset and task state viewers, bulk actions for Dag runs, and a full-screen code viewer.

  For more information about the major changes and breaking changes in this release, see the [Airflow 3.3.0 blog post](https://airflow.apache.org/blog/airflow-3.3.0/) or the [Airflow release notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html).

  ### Additional improvements

  * Upgrade to Airflow `3.3.0+astro.1`.
  * Upgrade to Apache Airflow TaskSDK to `1.3.0+astro.1`.
  * Upgrade the default Python version to 3.14. Astro Runtime 3.3 supports Python 3.12, 3.13, and 3.14.
  * Upgrade `astronomer-kubernetes-executor` to `10.19.0+astro.1`.
  * Upgrade `astronomer-providers-logging` to `1.6.7`.
  * Upgraded the versions of many open-source provider packages. See [Astro Runtime 3.3-1 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-3-1).

  ### Security fixes

  * Fixed [GHSA-4xgf-cpjx-pc3j](https://github.com/advisories/GHSA-4xgf-cpjx-pc3j)
  * Fixed [GHSA-2943-9672-r45w](https://github.com/advisories/GHSA-2943-9672-r45w)
</Update>

<Update label="Astro Runtime 3.2-6" description="July 14, 2026">
  * Airflow version: 3.2.2
  * Python versions: 3.12 - 3.14 (default: 3.13)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.2-6`

  ### Additional improvements

  * Upgrade to Airflow `3.2.2+astro.3`, which includes:
    * Fix an OpenTelemetry metrics scheduler crash for non-ASCII Dag and task names ([#68023](https://github.com/apache/airflow/pull/68023))
    * Fix a DagProcessor crash by adding a missing `name_is_otel_safe()` guard to `gauge()` and `timer()` ([#68284](https://github.com/apache/airflow/pull/68284))
    * Do not deserialize `trigger_kwargs` when loading serialized Dags ([#66002](https://github.com/apache/airflow/pull/66002))
    * Remove the dead AIP-44 trigger-over-`BaseSerialization` path (`DAT.BASE_TRIGGER`) ([#68528](https://github.com/apache/airflow/pull/68528))
    * Remove trigger kwargs from the REST API response ([#67868](https://github.com/apache/airflow/pull/67868))
    * Fix a missing redaction in a Variable JSON value ([#67495](https://github.com/apache/airflow/pull/67495))
    * Apply per-file authorization to the dag-source endpoint ([#67662](https://github.com/apache/airflow/pull/67662))
    * Filter scheduling-dependencies graph edges by readable-Dag access ([#67627](https://github.com/apache/airflow/pull/67627))
    * Mask per-key secrets-backend-kwarg overrides on the Config API ([#67622](https://github.com/apache/airflow/pull/67622))
    * Fix a crash when tailing logs of a running task instance ([#69503](https://github.com/apache/airflow/pull/69503))
  * Upgrade to Apache Airflow TaskSDK to `1.2.2+astro.3`
  * Upgrade `astronomer-kubernetes-executor` to `10.19.0+astro.1`
  * Upgrade `astronomer-providers-logging` to `1.6.7`
  * Upgrade the `kubernetes` Python client to `36.0.2` and `kubernetes_asyncio` to `36.1.0`, fixing `NO_PROXY` handling for in-cluster Kubernetes API calls

  ### Security fixes

  * Fixed [GHSA-hg6j-4rv6-33pg](https://github.com/advisories/GHSA-hg6j-4rv6-33pg)
  * Fixed [GHSA-jg22-mg44-37j8](https://github.com/advisories/GHSA-jg22-mg44-37j8)
  * Fixed [GHSA-cx3h-4qpv-8hc9](https://github.com/advisories/GHSA-cx3h-4qpv-8hc9)
  * Fixed [GHSA-537c-gmf6-5ccf](https://github.com/advisories/GHSA-537c-gmf6-5ccf)
  * Fixed [GHSA-5rvq-cxj2-64vf](https://github.com/advisories/GHSA-5rvq-cxj2-64vf)
  * Fixed [GHSA-82w8-qh3p-5jfq](https://github.com/advisories/GHSA-82w8-qh3p-5jfq)
  * Fixed [GHSA-4fvr-rgm6-gqmc](https://github.com/advisories/GHSA-4fvr-rgm6-gqmc)
  * Fixed [GHSA-63hw-fmq6-xxg2](https://github.com/advisories/GHSA-63hw-fmq6-xxg2)
  * Fixed [GHSA-g3cq-j2xw-wf74](https://github.com/advisories/GHSA-g3cq-j2xw-wf74)
  * Fixed [GHSA-hpj7-wq8m-9hgp](https://github.com/advisories/GHSA-hpj7-wq8m-9hgp)
  * Fixed [GHSA-pw6j-qg29-8w7f](https://github.com/advisories/GHSA-pw6j-qg29-8w7f)
  * Fixed [GHSA-xcgm-r5h9-7989](https://github.com/advisories/GHSA-xcgm-r5h9-7989)
  * Fixed [GHSA-2fqr-mr3j-6wp8](https://github.com/advisories/GHSA-2fqr-mr3j-6wp8)
  * Fixed [GHSA-4m7w-qmgq-4wj5](https://github.com/advisories/GHSA-4m7w-qmgq-4wj5)
  * Fixed [GHSA-6jv3-5f52-599m](https://github.com/advisories/GHSA-6jv3-5f52-599m)
  * Fixed [GHSA-9x8q-7h8h-wcw9](https://github.com/advisories/GHSA-9x8q-7h8h-wcw9)
  * Fixed [GHSA-jp82-jpqv-5vv3](https://github.com/advisories/GHSA-jp82-jpqv-5vv3)
  * Fixed [GHSA-v9pg-7xvm-68hf](https://github.com/advisories/GHSA-v9pg-7xvm-68hf)
  * Fixed [GHSA-vffw-93wf-4j4q](https://github.com/advisories/GHSA-vffw-93wf-4j4q)
  * Fixed [GHSA-v3q9-hj7j-63hq](https://github.com/advisories/GHSA-v3q9-hj7j-63hq)
  * Fixed [CVE-2026-33264](https://nvd.nist.gov/vuln/detail/CVE-2026-33264)
  * Fixed [CVE-2026-48828](https://nvd.nist.gov/vuln/detail/CVE-2026-48828)
  * Fixed [CVE-2026-48891](https://nvd.nist.gov/vuln/detail/CVE-2026-48891)
  * Fixed [CVE-2026-48892](https://nvd.nist.gov/vuln/detail/CVE-2026-48892)
  * Fixed [CVE-2026-49296](https://nvd.nist.gov/vuln/detail/CVE-2026-49296)
  * Fixed [CVE-2026-49487](https://nvd.nist.gov/vuln/detail/CVE-2026-49487)
  * Fixed [PYSEC-2026-2132](https://osv.dev/vulnerability/PYSEC-2026-2132)
</Update>

<Update label="Astro Runtime 3.2-5" description="June 03, 2026">
  * Airflow version: 3.2.2
  * Python versions: 3.12 - 3.14 (default: 3.13)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.2-5`

  ### Additional improvements

  * Upgrade to Airflow `3.2.2+astro.1`, which includes:
    * Add `on_kill()` hook to `BaseTrigger` to handle user actions on triggers ([#65590](https://github.com/apache/airflow/pull/65590))
    * Fix dag processor callback cleanup for versioned bundle files ([#66484](https://github.com/apache/airflow/pull/66484))
    * Fix `Callback.handle_event` crash on OTel metrics with `dict` tag values ([#67527](https://github.com/apache/airflow/pull/67527))
    * Add a compatibility layer for import errors caused by `AirflowSecretsBackendAccessDenied` ([#67560](https://github.com/apache/airflow/pull/67560))
    * Fix Graph layout for TaskGroup tasks wired to external nodes ([#67720](https://github.com/apache/airflow/pull/67720))
    * Fix scheduler orphaned task reset logging crash ([#67822](https://github.com/apache/airflow/pull/67822))
  * Upgrade to Apache Airflow TaskSDK to `1.2.2+astro.1`

  ### Security fixes

  * Fixed [GHSA-2h4p-vjrc-8xpq](https://github.com/advisories/GHSA-2h4p-vjrc-8xpq)
  * Fixed [GHSA-mf9v-mfxr-j63j](https://github.com/advisories/GHSA-mf9v-mfxr-j63j)
  * Fixed [GHSA-qccp-gfcp-xxvc](https://github.com/advisories/GHSA-qccp-gfcp-xxvc)
  * Fixed [GHSA-pp6c-gr5w-3c5g](https://github.com/advisories/GHSA-pp6c-gr5w-3c5g)
  * Fixed [GHSA-65pc-fj4g-8rjx](https://github.com/advisories/GHSA-65pc-fj4g-8rjx)
  * Fixed [GHSA-g3jr-4jrm-jvqv](https://github.com/advisories/GHSA-g3jr-4jrm-jvqv)
  * Fixed [PYSEC-2026-24](https://osv.dev/vulnerability/PYSEC-2026-24)
  * Fixed [CVE-2025-62727](https://avd.aquasec.com/nvd/cve-2025-62727)
  * Fixed [GHSA-cq8v-f236-94qc](https://github.com/advisories/GHSA-cq8v-f236-94qc)
  * Fixed [GHSA-fp55-jw48-c537](https://github.com/advisories/GHSA-fp55-jw48-c537)
  * Fixed [GHSA-xx64-wwv2-hcqq](https://github.com/advisories/GHSA-xx64-wwv2-hcqq)
  * Fixed [PYSEC-2026-175](https://osv.dev/vulnerability/PYSEC-2026-175)
  * Fixed [PYSEC-2026-177](https://osv.dev/vulnerability/PYSEC-2026-177)
  * Fixed [PYSEC-2026-178](https://osv.dev/vulnerability/PYSEC-2026-178)
  * Fixed [PYSEC-2026-179](https://osv.dev/vulnerability/PYSEC-2026-179)

  ### Requires action

  The `apache-airflow-providers-smtp` provider is upgraded from `2.4.5` to `3.0.1` to fix [PYSEC-2026-24](https://osv.dev/vulnerability/PYSEC-2026-24). `SmtpHook` now validates the SMTP server's certificate against the system CA bundle during STARTTLS upgrades by default. If you point `SmtpHook` at a server with a self-signed or otherwise non-validating certificate, set the `ssl_context` field in your SMTP connection's extras to `none` to keep the previous behavior.
</Update>

<Update label="Astro Runtime 3.2-4" description="May 06, 2026">
  * Airflow version: 3.2.1
  * Python versions: 3.12 - 3.14 (default: 3.13)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.2-4`

  ### Additional improvements

  * Upgrade to Airflow `3.2.1+astro.2`, which includes:
    * Show the task ID attributes (`ti_id`, `task_id`, and so on) once, not on every log line ([#66036](https://github.com/apache/airflow/pull/66036))
    * Don't re-emit `logical_date` when the previous `data_interval` is zero-length ([#66132](https://github.com/apache/airflow/pull/66132))
    * Fix slow and incomplete trigger cleanup in the scheduler ([#66210](https://github.com/apache/airflow/pull/66210))
    * Fix missing autoincrement sequence on `callback_request` downgrade ([#65230](https://github.com/apache/airflow/pull/65230))
    * Fix triggerer crash when multiple triggers call sync SDK methods concurrently ([#66412](https://github.com/apache/airflow/pull/66412))
  * Upgrade to Apache Airflow TaskSDK to `1.2.1+astro.2`, which includes:
    * Fix triggerer crash when multiple triggers call sync SDK methods concurrently ([#66412](https://github.com/apache/airflow/pull/66412))
</Update>

<Update label="Astro Runtime 3.2-3" description="April 29, 2026">
  * Airflow version: 3.2.1
  * Python versions: 3.12 - 3.14 (default: 3.13)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.2-3`

  ### Additional improvements

  * Upgrade to Airflow `3.2.1+astro.1`
  * Upgrade to Apache Airflow TaskSDK to `1.2.1+astro.1`

  ### Security fixes

  * Fixed [GHSA-v92g-xgxw-vvmm](https://github.com/advisories/GHSA-v92g-xgxw-vvmm)
  * Fixed [GHSA-965h-392x-2mh5](https://github.com/advisories/GHSA-965h-392x-2mh5)
  * Fixed [GHSA-xgp8-3hg3-c2mh](https://github.com/advisories/GHSA-xgp8-3hg3-c2mh)

  ### Requires action

  If you use custom user roles, users with read-only access to Dags can no longer view the Dag list in the Airflow UI because the endpoint now requires additional permissions. Astronomer-provided roles aren't affected. See [Upgrade considerations: Runtime 3.2-3 and later](/docs/runtime/version-upgrade-considerations#runtime-3-2-3-and-later) for details.
</Update>

<Update label="Astro Runtime 3.2-2" description="April 16, 2026">
  * Airflow version: 3.2.0
  * Python versions: 3.12 - 3.14 (default: 3.13)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.2-2`

  ### Introducing Airflow 3.2

  Astro Runtime 3.2-2 includes support for Apache Airflow 3.2, which introduces Asset partitioning for granular pipeline orchestration, synchronous deadline alert callbacks, and continued progress toward full Task SDK separation. Airflow 3.2 includes the following changes:

  * Asset partitioning (AIP-76): Downstream Dags trigger only when the specific partition they depend on is updated, rather than firing on any partition change. This applies to date-partitioned S3 paths, Hive table partitions, BigQuery partitions, and other partitioned data stores. Includes `CronPartitionTimetable`, backfill support for partitioned Dags, and multi-asset partition scheduling.
  * Synchronous deadline alert callbacks: Building on the deadline alerts system introduced in Airflow 3.1, this release adds synchronous callback support. `SyncCallback` executes directly on the worker through the executor, with optional targeting of a specific executor through the `executor` parameter. You can also configure multiple deadline alerts per Dag.
  * Async `PythonOperator` support: `PythonOperator` now supports async callables. You can pass an async function as the `python_callable` and the operator correctly awaits it, enabling async I/O patterns without a custom operator.
  * UI enhancements:
    * Human-in-the-loop approval history with full audit trail
    * XCom management directly from the UI (add, edit, and delete)
    * Data redaction for sensitive fields in the UI and Public API
    * Segmented state bars for collapsed task groups and mapped tasks
    * One-click log copying, date range filters, and unified tooltips in Grid and Graph views
  * Performance: Rendered task instance fields cleanup is approximately 42 times faster for Dags with many mapped tasks. Retention is now based on the N most recent Dag runs instead of N most recent task executions.

  For more information about the changes in this release, see the [Airflow Blog](https://airflow.apache.org/blog/airflow-3.2.0/) or the [Airflow release notes](https://airflow.apache.org/docs/apache-airflow/3.2.0/release_notes.html).

  ### Additional improvements

  * Fixed an issue where Astro environment manager connections were not found in Airflow.
  * Upgraded to Airflow `3.2.0+astro.1`, which includes:
    * Asset partitioning
    * Synchronous deadline alert callbacks
    * Async `PythonOperator` support
  * Upgraded the RHEL UBI base image from UBI 9 to UBI 10, and changed the OS variant image tag suffix from `-ubi9` to `-ubi`. For more information, see [Operating system variant images](https://www.astronomer.io/docs/astro/runtime-image-architecture#operating-system-variant-images).
  * Upgraded the versions of many open-source provider packages. See [Astro Runtime 3.2-2 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-2-2).

  ### Security fixes

  * Fixed [GHSA-mj87-hwqh-73pj](https://github.com/advisories/GHSA-mj87-hwqh-73pj)
</Update>

<Update label="Astro Runtime 3.2-1" description="April 14, 2026">
  * Airflow version: 3.2.0
  * Python versions: 3.12 - 3.14 (default: 3.13)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.2-1`

  <Warning>
    **Restricted release**

    Astro Runtime 3.2-1 was restricted from use on April 16, 2026, after its initial release because of an issue where environment manager connections are not found. See [Restricted Runtime versions](/docs/runtime/runtime-version-lifecycle-policy#restricted-runtime-versions).
  </Warning>
</Update>

<Update label="Astro Runtime 3.1-19" description="August 17, 2026">
  * Airflow version: 3.1.8
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-19`

  ### Additional improvements

  * Upgrade to Airflow `3.1.8+astro.6`, which includes:
    * Fix a 500 error for event logs with a `NULL` `dttm` ([#68338](https://github.com/apache/airflow/pull/68338))
    * Restore human-readable owner names in the Audit Log ([#68833](https://github.com/apache/airflow/pull/68833), [#70583](https://github.com/apache/airflow/pull/70583))
    * Fix a FAB roles and users 500 error caused by connection hook metadata mocks ([#68421](https://github.com/apache/airflow/pull/68421))
    * Keep connection hook metadata working on images built without FAB
    * Show user display names for triggered runs and human-in-the-loop responses ([#70836](https://github.com/apache/airflow/pull/70836))
    * Fix a note being discarded when materializing an asset
    * Fix a metadata database migration failure when the `deadline` table is not empty ([#66016](https://github.com/apache/airflow/pull/66016))
  * Upgrade to Apache Airflow TaskSDK to `1.1.8+astro.6`
  * Upgrade `astronomer-kubernetes-executor` to `10.20.0+astro.1`
  * Upgraded the minor and patch versions of some open-source provider packages. See [Astro Runtime 3.1-19 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-19).

  ### Security fixes

  * Fixed [GHSA-cq5v-8q36-5273](https://github.com/advisories/GHSA-cq5v-8q36-5273)
  * Fixed [GHSA-g6cj-pr64-35w5](https://github.com/advisories/GHSA-g6cj-pr64-35w5)
  * Fixed [GHSA-jwv3-5hgf-82ww](https://github.com/advisories/GHSA-jwv3-5hgf-82ww)
  * Fixed [GHSA-m2h6-j472-rp4c](https://github.com/advisories/GHSA-m2h6-j472-rp4c)
  * Fixed [GHSA-mfx4-hv73-q22v](https://github.com/advisories/GHSA-mfx4-hv73-q22v)
  * Fixed [GHSA-mq44-7p77-q5h7](https://github.com/advisories/GHSA-mq44-7p77-q5h7)
</Update>

<Update label="Astro Runtime 3.1-18" description="July 27, 2026">
  * Airflow version: 3.1.8
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-18`

  ### Additional improvements

  * Upgrade to Airflow 3.1.8+astro.5
    * Do not deserialize `trigger_kwargs` when loading serialized DAGs ([#66002](https://github.com/apache/airflow/pull/66002))
    * Remove dead AIP-44 `trigger-over-BaseSerialization` path (`DAT.BASE_TRIGGER`) ([#68528](https://github.com/apache/airflow/pull/68528))
    * Remove trigger kwargs from the REST API response ([#67868](https://github.com/apache/airflow/pull/67868))
    * Fix the miss redact in variable Json value ([#67495](https://github.com/apache/airflow/pull/67495))
    * Apply per-file authorization to `dag-source` endpoint ([#67662](https://github.com/apache/airflow/pull/67662))
    * Filter scheduling-dependencies graph edges by readable-DAG access ([#67627](https://github.com/apache/airflow/pull/67627))
    * Mask per-key secrets-backend-kwarg overrides on the Config API ([#67622](https://github.com/apache/airflow/pull/67622))
    * Fix DAG processor callback cleanup for versioned bundle files ([#66484](https://github.com/apache/airflow/pull/66484))
    * Reduce API server memory usage by eliminating `SerializedDAG` loads on task start ([#60803](https://github.com/apache/airflow/pull/60803))
    * Add configurable LRU+TTL caching for API server DAG retrieval ([#60804](https://github.com/apache/airflow/pull/60804))
    * Respect dag processor config option to show parsing logs on stdout ([#65528](https://github.com/apache/airflow/pull/65528))
    * Fix duplicate deadline callbacks with HA scheduler replicas ([#64737](https://github.com/apache/airflow/pull/64737))
    * Fix `run on latest version` not applied on task clear ([#65835](https://github.com/apache/airflow/pull/65835), [#68336](https://github.com/apache/airflow/pull/68336))
  * Upgrade to Apache Airflow TaskSDK to 1.1.8+astro.5
  * Upgrade `astronomer-kubernetes-executor` to `10.19.0+astro.1`
  * Upgrade `astronomer-providers-logging` to `1.6.8`
  * Upgraded several open-source provider packages. See [Astro Runtime 3.1-18 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-18).

  ### Security fixes

  * Fixed [GHSA-2fqr-mr3j-6wp8](https://github.com/advisories/GHSA-2fqr-mr3j-6wp8)
  * Fixed [GHSA-4fvr-rgm6-gqmc](https://github.com/advisories/GHSA-4fvr-rgm6-gqmc)
  * Fixed [GHSA-4m7w-qmgq-4wj5](https://github.com/advisories/GHSA-4m7w-qmgq-4wj5)
  * Fixed [GHSA-63hw-fmq6-xxg2](https://github.com/advisories/GHSA-63hw-fmq6-xxg2)
  * Fixed [GHSA-9x8q-7h8h-wcw9](https://github.com/advisories/GHSA-9x8q-7h8h-wcw9)
  * Fixed [GHSA-g3cq-j2xw-wf74](https://github.com/advisories/GHSA-g3cq-j2xw-wf74)
  * Fixed [GHSA-xcgm-r5h9-7989](https://github.com/advisories/GHSA-xcgm-r5h9-7989)
  * Fixed [GHSA-hpj7-wq8m-9hgp](https://github.com/advisories/GHSA-hpj7-wq8m-9hgp)
  * Fixed [GHSA-3x9g-8vmp-wqvf](https://github.com/advisories/GHSA-3x9g-8vmp-wqvf)
  * Fixed [GHSA-mgf9-4vpg-hj56](https://github.com/advisories/GHSA-mgf9-4vpg-hj56)
  * Fixed [GHSA-pw6j-qg29-8w7f](https://github.com/advisories/GHSA-pw6j-qg29-8w7f)
  * Fixed [GHSA-cx3h-4qpv-8hc9](https://github.com/advisories/GHSA-cx3h-4qpv-8hc9)
  * Fixed [GHSA-537c-gmf6-5ccf](https://github.com/advisories/GHSA-537c-gmf6-5ccf)
  * Fixed [GHSA-5rvq-cxj2-64vf](https://github.com/advisories/GHSA-5rvq-cxj2-64vf)
  * Fixed [GHSA-6jv3-5f52-599m](https://github.com/advisories/GHSA-6jv3-5f52-599m)
  * Fixed [GHSA-v9pg-7xvm-68hf](https://github.com/advisories/GHSA-v9pg-7xvm-68hf)
  * Fixed [GHSA-vffw-93wf-4j4q](https://github.com/advisories/GHSA-vffw-93wf-4j4q)
  * Fixed [GHSA-v3q9-hj7j-63hq](https://github.com/advisories/GHSA-v3q9-hj7j-63hq)
  * Fixed [CVE-2026-7246](https://avd.aquasec.com/nvd/cve-2026-7246)
  * Fixed [CVE-2026-33264](https://avd.aquasec.com/nvd/cve-2026-33264)
  * Fixed [CVE-2026-49487](https://avd.aquasec.com/nvd/cve-2026-49487)
  * Fixed [CVE-2026-48892](https://avd.aquasec.com/nvd/cve-2026-48892)
  * Fixed [CVE-2026-48828](https://avd.aquasec.com/nvd/cve-2026-48828)
  * Fixed [CVE-2026-49296](https://avd.aquasec.com/nvd/cve-2026-49296)
  * Fixed [CVE-2026-48891](https://avd.aquasec.com/nvd/cve-2026-48891)
</Update>

<Update label="Astro Runtime 3.1-17" description="June 12, 2026">
  * Airflow version: 3.1.8
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-17`

  ### Additional improvements

  * Upgrade to Airflow 3.1.8+astro.4
    * Fix OTel metrics scheduler crash for non-ASCII DAG/task names ([#68023](https://github.com/apache/airflow/pull/68023))
    * Add missing `name_is_otel_safe()` guard to `gauge()` and `timer()` ([#68284](https://github.com/apache/airflow/pull/68284))
  * Upgrade to Apache Airflow TaskSDK to 1.1.8+astro.4
  * Upgrade `astronomer-providers-logging` to `1.6.6`
</Update>

<Update label="Astro Runtime 3.1-16" description="June 09, 2026">
  * Airflow version: 3.1.8
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-16`

  ### Additional improvements

  * Upgrade to Airflow 3.1.8+astro.3
    * Apply per-DAG audit log permission to event log detail endpoint ([#67112](https://github.com/apache/airflow/pull/67112))
    * Set JWT refresh cookie Secure flag when request is HTTPS ([#65348](https://github.com/apache/airflow/pull/65348))
    * Refuse to follow log symlinks that resolve outside the base log folder ([#65325](https://github.com/apache/airflow/pull/65325))
    * Validate SMTP server certificate on `STARTTLS` upgrade ([#65346](https://github.com/apache/airflow/pull/65346))
    * Update the `is_url_safe` method to reject URLs with `///` ([#65557](https://github.com/apache/airflow/pull/65557))
    * Filter external dependency nodes by readable DAGs in `structure_data` endpoint ([#65342](https://github.com/apache/airflow/pull/65342))
    * Check sensitive key names before applying recursion-depth cutoff in secrets masker ([#65912](https://github.com/apache/airflow/pull/65912))
    * Extend `DEFAULT_SENSITIVE_FIELDS` with common credential field names ([#66673](https://github.com/apache/airflow/pull/66673))
    * Fix log server path extraction to use `removeprefix` ([#66749](https://github.com/apache/airflow/pull/66749))
  * Upgrade to Apache Airflow TaskSDK to 1.1.8+astro.3
  * Upgrade `astronomer-kubernetes-executor` to `10.17.1+astro.1`
  * Upgrade `astronomer-providers-logging` to `1.6.5`
  * Upgraded several open-source provider packages. See [Astro Runtime 3.1-16 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-16).

  ### Security fixes

  * Fixed [CVE-2026-49298](https://avd.aquasec.com/nvd/cve-2026-49298)
  * Fixed [GHSA-xgmm-8j9v-c9wx](https://github.com/advisories/GHSA-xgmm-8j9v-c9wx)
  * Fixed [GHSA-w7vc-732c-9m39](https://github.com/advisories/GHSA-w7vc-732c-9m39)
  * Fixed [GHSA-fhv5-28vv-h8m8](https://github.com/advisories/GHSA-fhv5-28vv-h8m8)
  * Fixed [GHSA-jq35-7prp-9v3f](https://github.com/advisories/GHSA-jq35-7prp-9v3f)
  * Fixed [GHSA-993g-76c3-p5m4](https://github.com/advisories/GHSA-993g-76c3-p5m4)
  * Fixed [GHSA-65pc-fj4g-8rjx](https://github.com/advisories/GHSA-65pc-fj4g-8rjx)
  * Fixed [GHSA-hg6j-4rv6-33pg](https://github.com/advisories/GHSA-hg6j-4rv6-33pg)
  * Fixed [GHSA-jg22-mg44-37j8](https://github.com/advisories/GHSA-jg22-mg44-37j8)
  * Fixed [GHSA-4gg8-gxpx-9rph](https://github.com/advisories/GHSA-4gg8-gxpx-9rph)
  * Fixed [GHSA-3cv2-h65g-fgmm](https://github.com/advisories/GHSA-3cv2-h65g-fgmm)
  * Fixed [GHSA-3pv8-6f4r-ffg2](https://github.com/advisories/GHSA-3pv8-6f4r-ffg2)

  ### Requires action

  The `apache-airflow-providers-smtp` provider is upgraded from `2.4.5` to `3.0.1`, a major version upgrade. `SmtpHook` now validates the SMTP server's certificate against the system CA bundle during STARTTLS upgrades by default. If you point `SmtpHook` at a server with a self-signed or otherwise non-validating certificate, set the `ssl_context` field in your SMTP connection's extras to `none` to keep the previous behavior.
</Update>

<Update label="Astro Runtime 3.1-15" description="May 13, 2026">
  * Airflow version: 3.1.8
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-15`

  ### Additional improvements

  * Upgrade to Airflow 3.1.8+astro.2
    * Refactor xcom API to use shared serialisation constants ([#64148](https://github.com/apache/airflow/pull/64148))
    * Fix task-level audit logs missing success/running events in Airflow 3.1.x ([#58655](https://github.com/apache/airflow/pull/58655))
    * Don't crash the scheduler on a stray ADRQ ([#61932](https://github.com/apache/airflow/pull/61932))
    * Fix: use Dag form when materializing asset ([#64211](https://github.com/apache/airflow/pull/64211))
    * Improve xcom value handling in extra links API ([#61641](https://github.com/apache/airflow/pull/61641))
    * Updates exception to hide sql statements on constraint failure ([#63028](https://github.com/apache/airflow/pull/63028))
    * Add check for xcom permission when result is specified in DagRun wait API ([#64415](https://github.com/apache/airflow/pull/64415))
    * Use default max depth to redact Variable ([#63480](https://github.com/apache/airflow/pull/63480))
    * Fix: add RUN, `HITL_DETAIL`, `TASK_INSTANCE` entities to /dags endpoint as it returns nested entities in response ([#64822](https://github.com/apache/airflow/pull/64822))
    * Add additional permission check in asset materialization ([#63338](https://github.com/apache/airflow/pull/63338))
    * Don't re-emit `logical_date` when previous `data_interval` is zero-length ([#66132](https://github.com/apache/airflow/pull/66132))
    * Show the task ID attributes (`ti_id`, `task_id`, etc.) once, not on every log line ([#66036](https://github.com/apache/airflow/pull/66036))
    * Fix slow and incomplete trigger cleanup in scheduler ([#66210](https://github.com/apache/airflow/pull/66210))
    * Fix triggerer crash when multiple triggers call sync SDK methods concurrently ([#66412](https://github.com/apache/airflow/pull/66412))
    * Fix scheduler callback `bundle_version` when versioning disabled ([#66485](https://github.com/apache/airflow/pull/66485))
  * Upgrade to Apache Airflow TaskSDK to 1.1.8+astro.2
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.1-15 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-15).

  ### Requires action

  If you use custom user roles, users with read-only access to Dags can no longer view the Dag list in the Airflow UI because the endpoint now requires additional permissions. Astronomer-provided roles aren't affected. See [Upgrade considerations: Runtime 3.1-15 and later](/docs/runtime/version-upgrade-considerations#runtime-3-1-15-and-later) for details.

  ### Security fixes

  * Fixed [GHSA-mj87-hwqh-73pj](https://github.com/advisories/GHSA-mj87-hwqh-73pj)
  * Fixed [GHSA-mf9w-mj56-hr94](https://github.com/advisories/GHSA-mf9w-mj56-hr94)
  * Fixed [GHSA-v92g-xgxw-vvmm](https://github.com/advisories/GHSA-v92g-xgxw-vvmm)
  * Fixed [GHSA-2h4p-vjrc-8xpq](https://github.com/advisories/GHSA-2h4p-vjrc-8xpq)
  * Fixed [GHSA-pp6c-gr5w-3c5g](https://github.com/advisories/GHSA-pp6c-gr5w-3c5g)
  * Fixed [GHSA-mf9v-mfxr-j63j](https://github.com/advisories/GHSA-mf9v-mfxr-j63j)
  * Fixed [GHSA-qccp-gfcp-xxvc](https://github.com/advisories/GHSA-qccp-gfcp-xxvc)
</Update>

<Update label="Astro Runtime 3.1-14" description="March 18, 2026">
  * Airflow version: 3.1.8
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-14`

  ### Additional improvements

  * Upgrade to Airflow 3.1.8+astro.1
    * Filter on `dag_id` when querying `task_instance` (#63291) ([#63291](https://github.com/apache/airflow/pull/63291))
    * Fix: redact JWT token from worker logs for kubernetes executor ([#62964](https://github.com/apache/airflow/pull/62964))
  * Upgrade to Apache Airflow TaskSDK to 1.1.8+astro.1
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.1-14 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-14).

  ### Security fixes

  * Fixed [CVE-2026-32597](https://avd.aquasec.com/nvd/cve-2026-32597)
  * Fixed [CVE-2026-31958](https://avd.aquasec.com/nvd/cve-2026-31958)
  * Fixed [GHSA-78cv-mqj4-43f7](https://github.com/advisories/GHSA-78cv-mqj4-43f7)
  * Fixed [CVE-2026-30922](https://avd.aquasec.com/nvd/cve-2026-30922)
</Update>

<Update label="Astro Runtime 3.1-13" description="February 18, 2026">
  * Airflow version: 3.1.7
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-13`

  ### Additional improvements

  * Upgraded to Airflow `3.1.7+astro.2`
    * Deprecate `BackfillDetails` and use `DagAcccessEntity.Run` for backfill ([#61400](https://github.com/apache/airflow/pull/61400))
    * Fix list Dag versions permissions ([#61733](https://github.com/apache/airflow/pull/61733))
    * Add gunicorn support for API server with rolling worker restarts ([#60940](https://github.com/apache/airflow/pull/60940))
    * Remove explicit background color from filter buttons to fix pale appearance([#61457](https://github.com/apache/airflow/pull/61457))
    * Fix secrets masking in Rendered Templates for complex objects ([#61763](https://github.com/apache/airflow/pull/61763))
  * Upgraded Apache Airflow TaskSDK to `1.1.7+astro.2`
    * Make `conn_type` optional in task SDK Connection datamodel ([#61728](https://github.com/apache/airflow/pull/61728))
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.1-13 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-13).
</Update>

<Update label="Astro Runtime 3.1-12" description="February 04, 2026">
  * Airflow version: 3.1.7
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-12`

  ### Additional improvements

  * Upgraded to Airflow `3.1.7+astro.1`
    * Flatten grid structure endpoint memory consumption ([#61393](https://github.com/apache/airflow/pull/61393))
  * Upgraded Apache Airflow TaskSDK to `1.1.7+astro.1`
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.1-12 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-12).

  ### Security fixes

  * Fixed [CVE-2026-0994](https://avd.aquasec.com/nvd/2026/cve-2026-0994)
  * Fixed [CVE-2026-1703](https://avd.aquasec.com/nvd/2026/cve-2026-1703)
</Update>

<Update label="Astro Runtime 3.1-11" description="January 23, 2026">
  * Airflow version: 3.1.6
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-11`

  ### Additional improvements

  * Upgrade to Airflow `3.1.6+astro.2`
    * Avoid loading all `TaskInstances` when checking `DagVersion` in `write_dag` to fix DAG processor OOM.([#60962](https://github.com/apache/airflow/pull/60962))
    * Fix unnecessary DAG version churn when DAG file paths change ([#60799](https://github.com/apache/airflow/pull/60799))
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.1-11 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-11).

  ### Security fixes

  * Fixed [CVE-2026-23490](https://avd.aquasec.com/nvd/2026/cve-2026-23490)
</Update>

<Update label="Astro Runtime 3.1-10" description="January 13, 2026">
  * Airflow version: 3.1.6
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-10`

  ### Additional improvements

  * Upgraded to Airflow `3.1.6+astro.1`, which includes:
    * Bug fixes
  * Upgraded Apache Airflow TaskSDK to `1.1.6+astro.1`

  ### Security fixes

  * Fixed [CVE-2026-21441](https://avd.aquasec.com/nvd/CVE-2026-21441)
</Update>

<Update label="Astro Runtime 3.1-9" description="December 15, 2025">
  * Airflow version: 3.1.5
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-9`

  ### Additional improvements

  * Upgraded to Airflow `3.1.5+astro.1`, which includes:
    * Bug fixes
  * Upgraded Apache Airflow TaskSDK to `1.1.5+astro.1`
</Update>

<Update label="Astro Runtime 3.1-8" description="December 10, 2025">
  * Airflow version: 3.1.4
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-8`

  <Warning>
    **Restricted release**

    Astro Runtime 3.1-8 was restricted from use on December 16, 2025, after its initial release because the Airflow version it is based on, Apache Airflow 3.1.4, was yanked from the OSS Airflow project.
  </Warning>
</Update>

<Update label="Astro Runtime 3.1-7" description="December 04, 2025">
  * Airflow version: 3.1.3
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-7`

  ### Additional improvements

  * Upgraded to Airflow `3.1.3+astro.2`, which includes:
    * Fix TypeError in `parseStreamingLogContent` for non-string data  ([#58314](https://github.com/apache/airflow/pull/58314))
</Update>

<Update label="Astro Runtime 3.1-6" description="December 03, 2025">
  * Airflow version: 3.1.3
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-6`

  ### Additional improvements

  * Upgrade to Apache Airflow TaskSDK to `1.1.3+astro.2`
    * Redact secrets in rendered templates properly to not expose them on UI ([#58767](https://github.com/apache/airflow/pull/58767))
    * Mask secrets properly when using deprecated import path ([#58662](https://github.com/apache/airflow/pull/58662))
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.1-6 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-6).
  * Enforce `SQLAlchemy==1.4.54`, even when users extend the image
</Update>

<Update label="Astro Runtime 3.1-5" description="November 14, 2025">
  * Airflow version: 3.1.3
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-5`

  ### Additional improvements

  * Upgraded to Airflow `3.1.3+astro.1`, which includes:
    * Bug fixes
    * Fix atomicity issue in `SerializedDagModel.write_dag` preventing orphaned DagVersions ([#58281](https://github.com/apache/airflow/pull/58281))
  * Upgraded to Apache Airflow TaskSDK to `1.1.3+astro.1`
</Update>

<Update label="Astro Runtime 3.1-4" description="November 06, 2025">
  * Airflow version: 3.1.2
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-4`

  ### Additional improvements

  * Upgraded to Airflow `3.1.2+astro.1`, which includes:
    * Bug fixes
    * Ensure that DB migrations handles all kinds of NaN values in historical xcoms ([#57893](https://github.com/apache/airflow/pull/57893))
    * Revert "Fix text selection jumping in logs pane to match text editor behavior (#57309)" ([#57874](https://github.com/apache/airflow/pull/57874))
  * Upgraded to Apache Airflow TaskSDK to `1.1.2+astro.1`
    * Ensure task in the context is always correct ([#57892](https://github.com/apache/airflow/pull/57892))
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.1-4 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-4).

  ### Security fixes

  * Fixed [GHSA-pqhf-p39g-3x64](https://github.com/advisories/GHSA-pqhf-p39g-3x64)
</Update>

<Update label="Astro Runtime 3.1-3" description="October 27, 2025">
  * Airflow version: 3.1.1
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-3`

  ### Additional improvements

  * Upgraded to Airflow `3.1.1+astro.1`, which includes:
    * Bug fixes
  * Fix memory leak in Client via SSL context creation ([#57334](https://github.com/apache/airflow/pull/57334))
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.1-3 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-3).

  ### Security fixes

  * Fixed [CVE-2025-62611](https://nvd.nist.gov/vuln/detail/CVE-2025-62611)
</Update>

<Update label="Astro Runtime 3.1-2" description="October 09, 2025">
  * Airflow version: 3.1.0
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-2`

  <Warning>
    **Known Issue**

    This version of Runtime has a significant memory leak for Celery Workers. Astronomer recommends upgrading directly to Runtime 3.1-3 or switching to Astro Executor.
  </Warning>

  ### Additional improvements

  * Upgraded to Airflow `3.1.0+astro.2`, which includes:
    * Fix scheduler crash with email notifications ([#56429](https://github.com/apache/airflow/pull/56429))
    * Emit log stream stopped warning as ndjson ([#56474](https://github.com/apache/airflow/pull/56474))

  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.1-2 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-2).
</Update>

<Update label="Astro Runtime 3.1-1" description="September 26, 2025">
  * Airflow version: 3.1.0
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.1-1`

  <Warning>
    **Known Issue**

    This version of Runtime has a significant memory leak for Celery Workers. Astronomer recommends upgrading directly to Runtime 3.1-3 or switching to Astro Executor.
  </Warning>

  ### Introducing Airflow 3.1

  Astro Runtime 3.1-1 includes support for Apache Airflow 3.1, which includes a number of new features and improvements. Airflow 3.1 includes the following changes:

  * Human-in-the-loop: Bridge between fully automated processes and human expertise
  * Many UI enhancements:
    * Internationalization with 17 supported languages
    * Calendar and gantt views
    * Robust plugin support
  * Deadline alerts

  For more information about the major changes in this release, see the [Airflow Blog](https://airflow.apache.org/blog/airflow-3.1.0/) or the [Airflow release notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-3-1-0-2025-09-25).

  ### Additional improvements

  * Upgraded to Airflow `3.1.0+astro.1`, which includes:
    * New features and improvements in Airflow 3.1.0
    * Enhanced performance and stability
    * Bug fixes
  * Upgraded the versions of many open-source provider packages. See [Astro Runtime 3.1-1 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-1-1).

  ### Behavior change

  <Warning>This Runtime version introduces a new default behavior that could cause breaks depending on your configurations.</Warning>

  * The Astro Runtime no longer installs Apache Airflow MySQL and PostgreSQL providers by default. If you need these providers, you must install them explicitly. See [Add Airflow providers, Python packages, and operating system packages](/docs/cli/v1.43/add-providers-packages) for how to add Airflow providers to your Astro project.
</Update>

<Update label="Astro Runtime 3.0-16" description="June 22, 2026">
  * Airflow version: 3.0.6
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-16`

  ### Additional improvements

  * Upgrade to Airflow 3.0.6+astro.7
    * Apply per-DAG audit log permission to event log detail endpoint ([#67112](https://github.com/apache/airflow/pull/67112))
    * Refuse to follow log symlinks that resolve outside the base log folder ([#65325](https://github.com/apache/airflow/pull/65325))
    * Validate SMTP server certificate on `STARTTLS` upgrade ([#65346](https://github.com/apache/airflow/pull/65346))
    * Update the `is_url_safe` method to reject URLs with `///` ([#65557](https://github.com/apache/airflow/pull/65557))
    * Filter external dependency nodes by readable DAGs in `structure_data` endpoint ([#65342](https://github.com/apache/airflow/pull/65342))
    * Check sensitive key names before applying `recursion-depth` cutoff in secrets masker ([#65912](https://github.com/apache/airflow/pull/65912))
    * Extend `DEFAULT_SENSITIVE_FIELDS` with common credential field names ([#66673](https://github.com/apache/airflow/pull/66673))
    * Fix OTel metrics scheduler crash for non-ASCII DAG/task names ([#68023](https://github.com/apache/airflow/pull/68023))
    * Add missing `name_is_otel_safe()` guard to `gauge()` and `timer()` ([#68284](https://github.com/apache/airflow/pull/68284))
    * Remove Airflow 2 code path in executors ([#51009](https://github.com/apache/airflow/pull/51009))
  * Upgrade to Apache Airflow TaskSDK to 1.0.6+astro.5
  * Upgrade `astronomer-kubernetes-executor` to `10.18.0+astro.1`
  * Upgrade `astronomer-providers-logging` to `1.6.6`
  * Upgraded several open-source provider packages. See [Astro Runtime 3.0-16 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-0-16).

  ### Security fixes

  * Fixed [CVE-2026-49298](https://avd.aquasec.com/nvd/cve-2026-49298)
  * Fixed [GHSA-xgmm-8j9v-c9wx](https://github.com/advisories/GHSA-xgmm-8j9v-c9wx)
  * Fixed [GHSA-w7vc-732c-9m39](https://github.com/advisories/GHSA-w7vc-732c-9m39)
  * Fixed [GHSA-fhv5-28vv-h8m8](https://github.com/advisories/GHSA-fhv5-28vv-h8m8)
  * Fixed [GHSA-jq35-7prp-9v3f](https://github.com/advisories/GHSA-jq35-7prp-9v3f)
  * Fixed [GHSA-993g-76c3-p5m4](https://github.com/advisories/GHSA-993g-76c3-p5m4)
  * Fixed [GHSA-65pc-fj4g-8rjx](https://github.com/advisories/GHSA-65pc-fj4g-8rjx)
  * Fixed [GHSA-hg6j-4rv6-33pg](https://github.com/advisories/GHSA-hg6j-4rv6-33pg)
  * Fixed [GHSA-jg22-mg44-37j8](https://github.com/advisories/GHSA-jg22-mg44-37j8)
  * Fixed [GHSA-4gg8-gxpx-9rph](https://github.com/advisories/GHSA-4gg8-gxpx-9rph)
  * Fixed [GHSA-3cv2-h65g-fgmm](https://github.com/advisories/GHSA-3cv2-h65g-fgmm)
  * Fixed [GHSA-3pv8-6f4r-ffg2](https://github.com/advisories/GHSA-3pv8-6f4r-ffg2)
  * Fixed [GHSA-cx3h-4qpv-8hc9](https://github.com/advisories/GHSA-cx3h-4qpv-8hc9)
  * Fixed [GHSA-537c-gmf6-5ccf](https://github.com/advisories/GHSA-537c-gmf6-5ccf)
  * Fixed [GHSA-5rvq-cxj2-64vf](https://github.com/advisories/GHSA-5rvq-cxj2-64vf)
  * Fixed [GHSA-4fvr-rgm6-gqmc](https://github.com/advisories/GHSA-4fvr-rgm6-gqmc)
  * Fixed [GHSA-63hw-fmq6-xxg2](https://github.com/advisories/GHSA-63hw-fmq6-xxg2)
  * Fixed [GHSA-g3cq-j2xw-wf74](https://github.com/advisories/GHSA-g3cq-j2xw-wf74)
  * Fixed [GHSA-hpj7-wq8m-9hgp](https://github.com/advisories/GHSA-hpj7-wq8m-9hgp)
  * Fixed [GHSA-pw6j-qg29-8w7f](https://github.com/advisories/GHSA-pw6j-qg29-8w7f)
  * Fixed [GHSA-xcgm-r5h9-7989](https://github.com/advisories/GHSA-xcgm-r5h9-7989)
  * Fixed [GHSA-2fqr-mr3j-6wp8](https://github.com/advisories/GHSA-2fqr-mr3j-6wp8)
  * Fixed [GHSA-4m7w-qmgq-4wj5](https://github.com/advisories/GHSA-4m7w-qmgq-4wj5)
  * Fixed [GHSA-6jv3-5f52-599m](https://github.com/advisories/GHSA-6jv3-5f52-599m)
  * Fixed [GHSA-9x8q-7h8h-wcw9](https://github.com/advisories/GHSA-9x8q-7h8h-wcw9)
  * Fixed [GHSA-v9pg-7xvm-68hf](https://github.com/advisories/GHSA-v9pg-7xvm-68hf)
  * Fixed [GHSA-vffw-93wf-4j4q](https://github.com/advisories/GHSA-vffw-93wf-4j4q)

  ### Requires action

  The `apache-airflow-providers-smtp` provider is upgraded from `2.4.2` to `3.0.1`, a major version upgrade. `SmtpHook` now validates the SMTP server's certificate against the system CA bundle during STARTTLS upgrades by default. If you point `SmtpHook` at a server with a self-signed or otherwise non-validating certificate, set the `ssl_context` field in your SMTP connection's extras to `none` to keep the previous behavior.
</Update>

<Update label="Astro Runtime 3.0-15" description="May 14, 2026">
  * Airflow version: 3.0.6
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-15`

  ### Additional improvements

  * Upgrade to Airflow 3.0.6+astro.6
    * Add additional permission check in asset materialization ([#63338](https://github.com/apache/airflow/pull/63338))
    * Use default max depth to redact Variable ([#63480](https://github.com/apache/airflow/pull/63480))
    * Updates exception to hide sql statements on constraint failure ([#63028](https://github.com/apache/airflow/pull/63028))
    * Improve xcom value handling in extra links API ([#61641](https://github.com/apache/airflow/pull/61641))
    * Exclude JWT token from workload repr to prevent log exposure ([#62964](https://github.com/apache/airflow/pull/62964))
    * Add `access_key` and `connection_string` to `DEFAULT_SENSITIVE_FIELDS` ([#61580](https://github.com/apache/airflow/pull/61580))
    * Add readable dags checks for the dependencies endpoint ([#62046](https://github.com/apache/airflow/pull/62046))
    * Fix list dag versions permissions ([#61675](https://github.com/apache/airflow/pull/61675))
    * Fix invalid TypeVar inside Redactable constraints in `secrets_masker`
    * Add `proxy` and `proxies` to `DEFAULT_SENSITIVE_FIELDS` ([#59688](https://github.com/apache/airflow/pull/59688))
    * Don't re-emit `logical_date` when previous `data_interval` is zero-length ([#66132](https://github.com/apache/airflow/pull/66132))
    * Fix scheduler callback `bundle_version` when versioning disabled ([#66485](https://github.com/apache/airflow/pull/66485))
    * Ensure JWTValidator handles GUESS algorithm with JWKS ([#63115](https://github.com/apache/airflow/pull/63115))
    * Fix JWT token generation with unset issuer/audience config ([#61278](https://github.com/apache/airflow/pull/61278))
  * Upgrade to Apache Airflow TaskSDK to 1.0.6+astro.4

  ### Security fixes

  * Fixed [GHSA-2h4p-vjrc-8xpq](https://github.com/advisories/GHSA-2h4p-vjrc-8xpq)
  * Fixed [GHSA-jr27-m4p2-rc6r](https://github.com/advisories/GHSA-jr27-m4p2-rc6r)
  * Fixed [GHSA-27jp-wm6q-gp25](https://github.com/advisories/GHSA-27jp-wm6q-gp25)
  * Fixed [GHSA-29vq-49wr-vm6x](https://github.com/advisories/GHSA-29vq-49wr-vm6x)
  * Fixed [GHSA-6w46-j5rx-g56g](https://github.com/advisories/GHSA-6w46-j5rx-g56g)
  * Fixed [GHSA-gc5v-m9x4-r6x2](https://github.com/advisories/GHSA-gc5v-m9x4-r6x2)
  * Fixed [GHSA-mf9w-mj56-hr94](https://github.com/advisories/GHSA-mf9w-mj56-hr94)
  * Fixed [GHSA-v92g-xgxw-vvmm](https://github.com/advisories/GHSA-v92g-xgxw-vvmm)
  * Fixed [GHSA-5239-wwwm-4pmq](https://github.com/advisories/GHSA-5239-wwwm-4pmq)
  * Fixed [GHSA-pjjw-68hj-v9mw](https://github.com/advisories/GHSA-pjjw-68hj-v9mw)
  * Fixed [GHSA-6vgw-5pg2-w6jp](https://github.com/advisories/GHSA-6vgw-5pg2-w6jp)
  * Fixed [GHSA-752w-5fwx-jx9f](https://github.com/advisories/GHSA-752w-5fwx-jx9f)
  * Fixed [GHSA-pp6c-gr5w-3c5g](https://github.com/advisories/GHSA-pp6c-gr5w-3c5g)
  * Fixed [GHSA-wp53-j4wj-2cfg](https://github.com/advisories/GHSA-wp53-j4wj-2cfg)
  * Fixed [GHSA-mj87-hwqh-73pj](https://github.com/advisories/GHSA-mj87-hwqh-73pj)
  * Fixed [GHSA-mf9v-mfxr-j63j](https://github.com/advisories/GHSA-mf9v-mfxr-j63j)
  * Fixed [GHSA-qccp-gfcp-xxvc](https://github.com/advisories/GHSA-qccp-gfcp-xxvc)
</Update>

<Update label="Astro Runtime 3.0-14" description="January 23, 2026">
  * Airflow version: 3.0.6
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-14`

  ### Additional improvements

  * Upgrade to Airflow `3.0.6+astro.5`
    * Eager-load `DagRun` asset relationships before creating `DagRunContext` ([#59714](https://github.com/apache/airflow/pull/59714))
    * Avoid loading all `TaskInstances` when checking `DagVersion` in `write_dag` to fix DAG processor OOM.([#60962](https://github.com/apache/airflow/pull/60962))
  * Upgrade to Apache Airflow TaskSDK to 1.0.6+astro.3
    * Mask kwargs on illegal args ([#58252](https://github.com/apache/airflow/pull/58252))
    * Redact secrets in rendered templates properly to not expose them on UI ([#58767](https://github.com/apache/airflow/pull/58767))
    * Redact secrets in rendered templates properly when truncating it ([#59566](https://github.com/apache/airflow/pull/59566))
  * Upgrade `astronomer-kubernetes-executor` to `10.8.1+astro.1`
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.0-14 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-0-14).

  ### Security fixes

  * Fixed [CVE-2025-62727](https://nvd.nist.gov/vuln/detail/CVE-2025-62727) -  Starlette DoS via Range header merging in FileResponse
  * Fixed [CVE-2025-65995](https://nvd.nist.gov/vuln/detail/CVE-2025-65995) - Airflow: Disclosure of secrets to UI via kwargs
  * Fixed [CVE-2025-66388](https://nvd.nist.gov/vuln/detail/CVE-2025-66388) - Airflow: Secrets in rendered templates not redacted properly and exposed in the UI
  * Fixed [CVE-2026-21441](https://avd.aquasec.com/nvd/CVE-2026-21441)
  * Fixed [CVE-2025-69223](https://avd.aquasec.com/nvd/cve-2025-69223)
  * Fixed [CVE-2025-69227](https://avd.aquasec.com/nvd/cve-2025-69227)
  * Fixed [CVE-2025-69228](https://avd.aquasec.com/nvd/cve-2025-69228)
  * Fixed [CVE-2025-69229](https://avd.aquasec.com/nvd/cve-2025-69229)
  * Fixed [CVE-2025-69224](https://avd.aquasec.com/nvd/cve-2025-69224)
  * Fixed [CVE-2025-69225](https://avd.aquasec.com/nvd/cve-2025-69225)
  * Fixed [CVE-2025-69226](https://avd.aquasec.com/nvd/cve-2025-69226)
  * Fixed [CVE-2025-69230](https://avd.aquasec.com/nvd/cve-2025-69230)
  * Fixed [CVE-2026-21860](https://avd.aquasec.com/nvd/2026/cve-2026-21860)
  * Fixed [CVE-2026-23490](https://avd.aquasec.com/nvd/2026/cve-2026-23490)
</Update>

<Update label="Astro Runtime 3.0-13" description="October 30, 2025">
  * Airflow version: 3.0.6
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-13`

  ### Additional improvements

  * Upgraded to Airflow `3.0.6+astro.4`, which includes:
    * Fix DAG processor crash with pre-import module optimization ([#56773](https://github.com/apache/airflow/pull/56773))
    * Prevent unnecessary kubernetes client imports in workers ([#56692](https://github.com/apache/airflow/pull/56692))
  * Upgraded Apache Airflow TaskSDK to `1.0.6+astro.2`which includes:
    * Fix memory leak in remote logging connection cache ([#56695](https://github.com/apache/airflow/pull/56695))
    * Fix memory leak in Client via SSL context creation ([#57334](https://github.com/apache/airflow/pull/57334))
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.0-13 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-0-13).

  ### Security fixes

  * Fixed [CVE-2025-62611](https://nvd.nist.gov/vuln/detail/CVE-2025-62611)
  * Fixed [CVE-2025-62503](https://nvd.nist.gov/vuln/detail/CVE-2025-62503)
  * Fixed [CVE-2025-62402](https://nvd.nist.gov/vuln/detail/CVE-2025-62402)
  * Fixed [GHSA-pqhf-p39g-3x64](https://github.com/advisories/GHSA-pqhf-p39g-3x64)
</Update>

<Update label="Astro Runtime 3.0-12" description="September 24, 2025">
  * Airflow version: 3.0.6
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-12`

  <Warning>
    **Known Issue**

    This version of Runtime has a significant memory leak for Celery Workers. Astronomer recommends upgrading directly to Runtime 3.1-3 or switching to Astro Executor.
  </Warning>

  ### Additional improvements

  * Upgraded to Airflow `3.0.6+astro.3`, which includes:
    * Handle trigger calls to `get_connection` ([#55799](https://github.com/apache/airflow/pull/55799))
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.0-12 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-0-12).
</Update>

<Update label="Astro Runtime 3.0-11" description="September 17, 2025">
  * Airflow version: 3.0.6
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-11`

  <Warning>
    **Known Issue**

    This version of Runtime has a significant memory leak for Celery Workers. Astronomer recommends upgrading directly to Runtime 3.1-3 or switching to Astro Executor.
  </Warning>

  ### Additional improvements

  * Upgraded to Airflow `3.0.6+astro.2`, which includes:
    * Fix query client retry strategy ([#55528](https://github.com/apache/airflow/pull/55528))
    * Fix FAB related db downgrade issues ([#55738](https://github.com/apache/airflow/pull/55738)), ([#55231](https://github.com/apache/airflow/pull/55231))
    * Reduce default API server workers to 1 ([#55707](https://github.com/apache/airflow/pull/55707))
    * Remove `python_callable` as string from mapped operator in serialized dag ([#55288](https://github.com/apache/airflow/pull/55288))
    * Minor optimizations to serialized dag storage and retrieval to reduce size
</Update>

<Update label="Astro Runtime 3.0-10" description="August 29, 2025">
  * Airflow version: 3.0.6
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-10`

  <Warning>
    **Known Issue**

    This version of Runtime has a significant memory leak for Celery Workers. Astronomer recommends upgrading directly to Runtime 3.1-3 or switching to Astro Executor.
  </Warning>

  ### Additional improvements

  * Upgraded to Airflow `3.0.6+astro.1`. See [Airflow Release Notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-3-0-6-2025-08-29) for more information.
  * Upgraded Apache Airflow TaskSDK to `1.0.6+astro.1`.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.0-10 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-0-10).
</Update>

<Update label="Astro Runtime 3.0-9" description="August 21, 2025">
  * Airflow version: 3.0.5
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-9`

  <Warning>
    **Known Issue**

    This version of Runtime has a significant memory leak for Celery Workers. Astronomer recommends upgrading directly to Runtime 3.1-3 or switching to Astro Executor.
  </Warning>

  ### Restricted version

  Airflow 3.0.5 was `yanked` due to a bug with `conn.extra_dejson` masking that causes tasks to fail ([#54768](https://github.com/apache/airflow/issues/54768)).

  This means that the Astro Runtime 3.0-9 is also yanked. See [Restricted Runtime versions](/docs/runtime/runtime-version-lifecycle-policy#restricted-runtime-versions).
</Update>

<Update label="Astro Runtime 3.0-8" description="August 19, 2025">
  * Airflow version: 3.0.4
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-8`

  <Warning>
    **Known Issue**

    This version of Runtime has a significant memory leak for Celery Workers. Astronomer recommends upgrading directly to Runtime 3.1-3 or switching to Astro Executor.
  </Warning>

  ### Additional improvements

  * Upgrade to Airflow 3.0.4+astro.2, which includes:
    * Fix scheduler crashes with DetachedInstanceError when processing executor events. [#54334](https://github.com/apache/airflow/pull/54334)
    * Fix DetachedInstanceError when accessing DagRun.`created_dag_version`. [#54362](https://github.com/apache/airflow/pull/54362)
    * Fix custom XCom backends not being used when BaseXCom.`get_all`() is called. [#53814](https://github.com/apache/airflow/pull/53814)
    * Fix `xcom_pull` ignoring `include_prior_dates` parameter when `map_indexes` is not specified. [#53809](https://github.com/apache/airflow/pull/53809)
    * Restore `get_previous_dagrun` functionality for task context. [#53655](https://github.com/apache/airflow/pull/53655)
    * Fix log retrieval failures for in-progress tasks by properly configuring JWT authentication. [#54444](https://github.com/apache/airflow/pull/54444)
  * Upgrade to Apache Airflow Task SDK 1.0.4+astro.2, which includes:
    * Fix custom XCom backends not being used when BaseXCom.`get_all`() is called. [#53814](https://github.com/apache/airflow/pull/53814)
    * Fix `xcom_pull` ignoring `include_prior_dates` parameter when `map_indexes` is not specified. [#53809](https://github.com/apache/airflow/pull/53809)
    * Restore `get_previous_dagrun` functionality for task context. [#53655](https://github.com/apache/airflow/pull/53655)
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.0-8 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-0-8).
  * Fixed [CVE-2025-53643](https://nvd.nist.gov/vuln/detail/CVE-2025-53643) for AIOHTTP vulnerable by upgrading to version 3.12.14.
  * Fixed [CVE-2025-54368](https://nvd.nist.gov/vuln/detail/CVE-2025-54368) for uv by upgrading to version 0.8.6.
</Update>

<Update label="Astro Runtime 3.0-7" description="August 08, 2025">
  * Airflow version: 3.0.4
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-7`

  <Warning>
    **Known Issue**

    This version of Runtime has a significant memory leak for Celery Workers. Astronomer recommends upgrading directly to Runtime 3.1-3 or switching to Astro Executor.
  </Warning>

  ### Additional improvements

  * Upgraded to Airflow 3.0.4. See [Airflow Release Notes](https://airflow.apache.org/docs/apache-airflow/3.0.4/release_notes.html#airflow-3-0-4-2025-08-08) for more information.
  * Upgraded Apache Airflow TaskSDK to `1.0.4+astro.1`.
  * Upgraded `astronomer-logging-providers==1.6.4` which enables exporting the logs to secondary GCS bucket.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.0-7 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-0-7).
</Update>

<Update label="Astro Runtime 3.0-6" description="July 24, 2025">
  * Airflow version: 3.0.3
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-6`

  <Warning>
    **Known Issue**

    This version of Runtime has a significant memory leak for Celery Workers. Astronomer recommends upgrading directly to Runtime 3.1-3 or switching to Astro Executor.
  </Warning>

  ### Additional improvements

  * Upgraded to `Airflow 3.0.3+astro.2`, which includes:
    * Skips empty dag-run config rows and raises the SQL timeout for XCom migrations to avoid lock-timeout errors. [#50788](https://github.com/apache/airflow/pull/50788)
    * Fixes sensor skipping in 3.x branching operators by correcting the SkipMixin import path. [#53455](https://github.com/apache/airflow/pull/53455)
    * Triggers task-failure callbacks on the Dag processor when tasks are externally killed. [#53143](https://github.com/apache/airflow/pull/53143)
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.0-6 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-0-6).

  ### Security fixes

  * Fixed [CVE-2025-53528](https://avd.aquasec.com/nvd/cve-2025-53528)
</Update>

<Update label="Astro Runtime 3.0-5" description="July 14, 2025">
  * Airflow version: 3.0.3
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-5`

  ### Additional improvements

  * Upgraded to Airflow 3.0.3. See [Airflow Release Notes](https://airflow.apache.org/docs/apache-airflow/3.0.3/release_notes.html#airflow-3-0-3-2025-07-14) for more information.
  * Upgrade Apache Airflow TaskSDK to `1.0.3+astro.1`.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.0-5 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-0-5).
</Update>

<Update label="Astro Runtime 3.0-4" description="June 12, 2025">
  * Airflow version: 3.0.2
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-4`

  ### Bug fixes

  * Fixed various bugs in `AstroAuthManager`.
  * Fixed text alignment of **Back to Astro** button.
</Update>

<Update label="Astro Runtime 3.0-3" description="June 10, 2025">
  * Airflow version: 3.0.2
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-3`

  ### Additional improvements

  * Upgraded to Airflow 3.0.2. See [Airflow Release Notes](https://airflow.apache.org/docs/apache-airflow/3.0.2/release_notes.html#airflow-3-0-2-2025-06-10) for more information.
  * Upgrade to `Airflow 3.0.2+astro.1`.
  * Upgrade Apache Airflow TaskSDK to `1.0.2+astro.1`.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.0-3 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-0-3)
</Update>

<Update label="Astro Runtime 3.0-2" description="May 12, 2025">
  * Airflow version: 3.0.1
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-2`

  ### Additional improvements

  * Added `SCARF_NO_ANALYTICS=True` in Runtime images to disable telemetry by default.
  * Updated `astronomer-providers-logging` to `1.6.1`.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 3.0-2 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-0-2)

  ### Known issues

  Remote execution has a known incompatibility with Runtime 3.0-2. If you use remote execution, do not upgrade to or create Remote Deployments that use this version. A fix for the incompatibility will be released in v3.0-3.

  If you created a new remote Deployment with 3.0-2, you cannot rollback to 3.0-1. You must either create a new remote Deployment with 3.0-1 or wait until the 3.0-3 patch and then upgrade.

  If you upgraded a Deployment from 3.0-1 to 3.0-2, Astronomer advises you to downgrade your Deployments or rollback to 3.0-1 by using a Dockerfile change or by rolling back your Deployment in the Astro UI.

  * [Roll back to previous deploys](/docs/astro/deploy-history#what-happens-during-a-deploy-rollback)
  * [Roll back Deployments after a broken upgrade](/docs/astro/best-practices/upgrading-astro-runtime#roll-back-deployments-after-a-broken-upgrade)
</Update>

<Update label="Astro Runtime 3.0-1" description="April 22, 2025">
  * Airflow version: 3.0.0
  * Python versions: 3.11 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/runtime:3.0-1`

  ### Introducing Airflow 3.0

  Astro Runtime 3.0-1 includes same-day support for Apache Airflow 3.0, which includes a number of new features and improvements. Airflow 3.0 includes the following changes:

  * Support for dag versioning
  * Enhanced and improved React-based Airflow UI
  * Support for remote dag execution
  * Scheduler-managed backfill
  * Event-driven and asset-driven scheduling

  For more information about the major changes in this release, see the [Airflow release notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html).

  ### Behavior changes

  * New naming convention for Runtime release versions. Previously, versions were noted as `XX.X.X`, now versions are named as `XX.X-X`.
  * Astro Runtime for 3.0 and higher includes only the [provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-3-0-1) required to run on Astro. Previous versions contained additional providers, and if you are using those providers you must explicitly add them to your `requirements.txt`.
  * Runtime docs for versions that are past their **End of Basic Support** date, below v6.0.0, have been moved to the [Astronomer Docs Resources](https://github.com/astronomer/astronomer-docs-resources) archive.
  * The Docker registry URL for Runtime images has changed for Airflow 3.x. For details, see [Docker Registry URL changes](#docker-registry-url-changes).

  #### Pip package resolution

  With Runtime 3.0, Python package resolution has switched by default from pip to [uv](https://docs.astral.sh/uv/), which is intended as a fast [drop in replacement](https://docs.astral.sh/uv/pip/compatibility/) for pip. However, uv has a different behavior for packages that exist on [multiple indexes](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes), because pip's behavior is unsafe against dependency confusion attacks. If you rely on the pip behavior, you can add `# ASTRO_RUNTIME_USE_PIP` to your `requirements.txt` file to use pip instead of uv. You can also opt to install your dependencies in a separate `RUN` line on your Dockerfile instead of using Runtime's built-in pip installation.

  #### Docker Compose override changes

  When using `astro dev start`, you can specify a [docker-`compose.override.yml`](/docs/cli/v1.43/run-airflow-locally#override-the-astro-cli-docker-compose-file). If you specified any overrides for the `webserver` container, these break your ability to use `astro dev start` because there is no longer a `webserver` container. Replace all references to `webserver` with `api-server`.

  #### Docker Registry URL changes

  Starting with Airflow 3.x, Runtime Docker images are hosted at a new registry:

  ```text wrap theme={null}
  astrocrpublic.azurecr.io/runtime:<version>
  ```

  Airflow 2.x images are still available under the original registry:

  ```text wrap theme={null}
  quay.io/astronomer/astro-runtime:<version>
  ```

  but can also be pulled from the new domain as an alternative:

  ```text wrap theme={null}
  astrocrpublic.azurecr.io/astronomer/astro-runtime:<version>
  ```
</Update>

<Update label="Astro Runtime 13.9.0" description="August 03, 2026">
  * Airflow version: 2.11.2
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:13.9.0`

  ### Additional improvements

  * Upgrade to Airflow `2.11.2+astro.5` which includes:
    * Stop exposing trigger kwargs in the REST API response [#67868](https://github.com/apache/airflow/pull/67868)
    * Mask per-key secrets-backend-kwarg overrides on the Config API [#67622](https://github.com/apache/airflow/pull/67622)
    * Do not deserialize `trigger_kwargs` when loading serialized DAGs [#66002](https://github.com/apache/airflow/pull/66002)
    * Fix truncated details tabs and page overflow in the grid view [#68750](https://github.com/apache/airflow/pull/68750)
  * Upgrade `astronomer-providers-logging` to `1.6.8`
  * Upgrade `apache-airflow-providers-cncf-kubernetes` to `10.20.0`
  * Update `astronomer-kubernetes-executor` to `10.20.0+astro.1`
  * Upgraded the minor and patch versions of several open-source provider packages.

  ### Security fixes

  * Fixed [GHSA-2wc2-fm75-p42x](https://github.com/advisories/GHSA-2wc2-fm75-p42x)
  * Fixed [GHSA-836r-79rf-4m37](https://github.com/advisories/GHSA-836r-79rf-4m37)
  * Fixed [GHSA-8ppf-4f7h-5ppj](https://github.com/advisories/GHSA-8ppf-4f7h-5ppj)
  * Fixed [GHSA-hm4w-wwcw-mr6r](https://github.com/advisories/GHSA-hm4w-wwcw-mr6r)
  * Fixed [PYSEC-2026-2132](https://osv.dev/vulnerability/PYSEC-2026-2132)
  * Fixed [PYSEC-2026-3444](https://osv.dev/vulnerability/PYSEC-2026-3444)
  * Fixed [PYSEC-2026-3455](https://osv.dev/vulnerability/PYSEC-2026-3455)
  * Fixed [GHSA-h35f-9h28-mq5c](https://github.com/advisories/GHSA-h35f-9h28-mq5c)
</Update>

<Update label="Astro Runtime 13.8.0" description="June 18, 2026">
  * Airflow version: 2.11.2
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:13.8.0`

  ### Additional improvements

  * Upgrade to Airflow `2.11.2+astro.4` which includes:
    * Improve `/home` load time on deployments with many DAGs by skipping the per-DAG authorization filter for users with global DAG access
    * Fix OTel metrics crash for non-ASCII DAG/task names [#68023](https://github.com/apache/airflow/pull/68023)
    * Fix DagProcessor crash: add missing `name_is_otel_safe()` guard to `gauge()` and `timer()` [#68284](https://github.com/apache/airflow/pull/68284)
    * Extend `DEFAULT_SENSITIVE_FIELDS` with `webhook_url`, `bearer`, `dsn`, `auth_header`, and `service_key` [#66673](https://github.com/apache/airflow/pull/66673)
    * Check sensitive key names before applying recursion-depth cutoff in secrets masker [#65912](https://github.com/apache/airflow/pull/65912)
    * Refuse to follow log symlinks that resolve outside the base log folder [#65325](https://github.com/apache/airflow/pull/65325)
    * Validate SMTP server certificate on `STARTTLS` upgrade [#65346](https://github.com/apache/airflow/pull/65346)
  * Upgrade `astronomer-providers-logging` to `1.6.6`
  * Upgrade `apache-airflow-providers-cncf-kubernetes` to `10.18.0`

  ### Security fixes

  * Fixed [GHSA-65pc-fj4g-8rjx](https://github.com/advisories/GHSA-65pc-fj4g-8rjx)
  * Fixed [PYSEC-2026-24](https://osv.dev/vulnerability/PYSEC-2026-24)
  * Fixed [PYSEC-2026-113](https://osv.dev/vulnerability/PYSEC-2026-113)
  * Fixed [CVE-2026-45192](https://avd.aquasec.com/nvd/cve-2026-45192)
  * Fixed [CVE-2026-42358](https://avd.aquasec.com/nvd/cve-2026-42358)
  * Fixed [CVE-2026-40861](https://avd.aquasec.com/nvd/cve-2026-40861)
  * Fixed [CVE-2026-49267](https://avd.aquasec.com/nvd/cve-2026-49267)
  * Fixed [GHSA-29h4-r29x-hchv](https://github.com/advisories/GHSA-29h4-r29x-hchv)
  * Fixed [PYSEC-2026-175](https://osv.dev/vulnerability/PYSEC-2026-175)
  * Fixed [PYSEC-2026-177](https://osv.dev/vulnerability/PYSEC-2026-177)
  * Fixed [PYSEC-2026-178](https://osv.dev/vulnerability/PYSEC-2026-178)
  * Fixed [PYSEC-2026-179](https://osv.dev/vulnerability/PYSEC-2026-179)
  * Fixed [GHSA-hg6j-4rv6-33pg](https://github.com/advisories/GHSA-hg6j-4rv6-33pg)
  * Fixed [GHSA-jg22-mg44-37j8](https://github.com/advisories/GHSA-jg22-mg44-37j8)
  * Fixed [GHSA-cx3h-4qpv-8hc9](https://github.com/advisories/GHSA-cx3h-4qpv-8hc9)
  * Fixed [GHSA-537c-gmf6-5ccf](https://github.com/advisories/GHSA-537c-gmf6-5ccf)
  * Fixed [GHSA-pw6j-qg29-8w7f](https://github.com/advisories/GHSA-pw6j-qg29-8w7f)
  * Fixed [GHSA-4fvr-rgm6-gqmc](https://github.com/advisories/GHSA-4fvr-rgm6-gqmc)
  * Fixed [GHSA-63hw-fmq6-xxg2](https://github.com/advisories/GHSA-63hw-fmq6-xxg2)
  * Fixed [GHSA-g3cq-j2xw-wf74](https://github.com/advisories/GHSA-g3cq-j2xw-wf74)
  * Fixed [GHSA-hpj7-wq8m-9hgp](https://github.com/advisories/GHSA-hpj7-wq8m-9hgp)
  * Fixed [GHSA-xcgm-r5h9-7989](https://github.com/advisories/GHSA-xcgm-r5h9-7989)
  * Fixed [GHSA-2fqr-mr3j-6wp8](https://github.com/advisories/GHSA-2fqr-mr3j-6wp8)
  * Fixed [GHSA-4m7w-qmgq-4wj5](https://github.com/advisories/GHSA-4m7w-qmgq-4wj5)
  * Fixed [GHSA-9x8q-7h8h-wcw9](https://github.com/advisories/GHSA-9x8q-7h8h-wcw9)

  ### Breaking changes

  * `apache-airflow-providers-smtp` bumped 2.4.5 → 3.0.1 to fix [PYSEC-2026-24](https://osv.dev/vulnerability/PYSEC-2026-24). `SmtpHook` STARTTLS upgrades now validate the SMTP server's certificate against the system CA bundle by default. Deployments pointing `SmtpHook` at servers with self-signed or otherwise non-validating certs must set the `ssl_context` field in the SMTP connection extras to `"none"` to preserve the previous behavior.
  * `azure-datalake-store` bumped 0.0.53 → 1.0.1. Version 1.0.0 switched ADLS authentication to the generic Azure token credential (replacing the legacy `lib.auth`) and dropped end-of-life Python versions. Deployments using Azure Data Lake (ADLS) hooks/operators should validate authentication after upgrading. The `cryptography<47` cap that previously blocked upgrading `cryptography` to 48.0.1 for [GHSA-537c-gmf6-5ccf](https://github.com/advisories/GHSA-537c-gmf6-5ccf) came from `msal`, which this release bumps 1.32.3 → 1.37.0.
</Update>

<Update label="Astro Runtime 13.7.0" description="May 18, 2026">
  * Airflow version: 2.11.2
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:13.7.0`

  ### Additional improvements

  * Upgrade to Airflow `2.11.2+astro.3` which includes:
    * Skip dataset-triggered dags without SerializedDagModel [#63546](https://github.com/apache/airflow/pull/63546)
    * Prevent scheduler crash on AF3.2+ downgrade (interval timetables)
    * Fix log groups collapsing on auto-refresh in Logs view [#65378](https://github.com/apache/airflow/pull/65378)
    * Guard Grid task-group selections in legacy details flow
    * Add INFO-level logging to dataset scheduling path [#63958](https://github.com/apache/airflow/pull/63958)
  * Remove sunset Astronomer Registry link from Airflow UI [#3144](https://github.com/astronomer/astro-runtime/pull/3144)

  ### Security fixes

  * Fix 22 endorctl-flagged CVEs by bumping `aiohttp`, `urllib3`, `requests`, `microsoft-kiota-http`, `Mako`, `Authlib`, `Pygments`, `lxml`, `pytest`, and `pyarrow`

  ### Breaking changes

  * Bumped `snowflake-connector-python` 3.15.0 → 4.5.0 and `lxml` 5.3.2 → 6.1.0 to fix CVEs. Snowflake 4.x and lxml 6.x each drop deprecated APIs — review DAGs that use them directly.
</Update>

<Update label="Astro Runtime 13.6.0" description="March 19, 2026">
  * Airflow version: 2.11.2
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:13.6.0`

  ### Additional improvements

  * Upgrade to Airflow `2.11.2+astro.2` which includes:
    * Prevent stale zombie callbacks from failing newer task attempts [#63726](https://github.com/apache/airflow/pull/63726)

  ### Security fixes

  * Fixed [CVE-2026-28802](https://avd.aquasec.com/nvd/cve-2026-28802)
  * Fixed [CVE-2025-69219](https://avd.aquasec.com/nvd/cve-2025-69219) by upgrading `apache-airflow-providers-http` from 5.6.4 to 6.0.0
  * Fixed [CVE-2026-32597](https://avd.aquasec.com/nvd/cve-2026-32597)
  * Fixed [CVE-2026-31958](https://avd.aquasec.com/nvd/cve-2026-31958)
  * Fixed [GHSA-78cv-mqj4-43f7](https://github.com/advisories/GHSA-78cv-mqj4-43f7)
  * Fixed [CVE-2026-27962](https://avd.aquasec.com/nvd/cve-2026-27962)
  * Fixed [CVE-2026-28490](https://avd.aquasec.com/nvd/cve-2026-28490)
  * Fixed [CVE-2026-28498](https://avd.aquasec.com/nvd/cve-2026-28498)
  * Fixed [CVE-2026-30922](https://avd.aquasec.com/nvd/cve-2026-30922)
</Update>

<Update label="Astro Runtime 13.5.1" description="February 25, 2026">
  * Airflow version: 2.11.1
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:13.5.1`

  ### Additional improvements

  * Upgrade to Airflow `2.11.1+astro.2` which includes:
    * Fix webserver 500 error when upgrading to FAB provider 1.5.4 [#62412](https://github.com/apache/airflow/pull/62412)
    * Lazily import fs and `package_index` hook in providers manager [#62357](https://github.com/apache/airflow/pull/62356)
</Update>

<Update label="Astro Runtime 13.5.0" description="February 23, 2026">
  * Airflow version: 2.11.1
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:13.5.0`

  ### Restricted version

  Astro Runtime 13.5.0 has been restricted from use due to a known issue where the Airflow webserver returns 500 errors for existing user sessions when upgrading from previous versions. This is caused by an incompatibility between session cookies from earlier versions and the updated flask-session dependency included in `FAB 1.5.4`. See [Restricted Runtime versions](/docs/runtime/runtime-version-lifecycle-policy#restricted-runtime-versions).

  ### Additional improvements

  * Upgrade to Airflow `2.11.1+astro.1` which includes:
    * Mask proxy and proxies in logs [#59688](https://github.com/apache/airflow/pull/59688)
    * Fix permissions check in import error APIs [#60801](https://github.com/apache/airflow/pull/60801)
    * Fix import errors not showing on UI [#61163](https://github.com/apache/airflow/pull/61163)
    * Fix base url for Dag dependencies
    * Fix home page pagination when running filter is active

  ### Security fixes

  * Fixed [CVE-2025-68675](https://avd.aquasec.com/nvd/cve-2025-68675)
  * Fixed [CVE-2025-15467](https://avd.aquasec.com/nvd/cve-2025-15467)
  * Fixed [CVE-2025-69419](https://avd.aquasec.com/nvd/cve-2025-69419)
  * Fixed [CVE-2025-69421](https://avd.aquasec.com/nvd/cve-2025-69421)
  * Fixed [CVE-2025-68160](https://avd.aquasec.com/nvd/cve-2025-68160)
  * Fixed [CVE-2025-69418](https://avd.aquasec.com/nvd/cve-2025-69418)
  * Fixed [CVE-2025-69420](https://avd.aquasec.com/nvd/cve-2025-69420)
  * Fixed [CVE-2026-22795](https://avd.aquasec.com/nvd/cve-2026-22795)
  * Fixed [CVE-2026-22796](https://avd.aquasec.com/nvd/cve-2026-22796)
  * Fixed [CVE-2026-21226](https://avd.aquasec.com/nvd/cve-2026-21226)
  * Fixed [CVE-2026-22701](https://avd.aquasec.com/nvd/cve-2026-22701)
  * Fixed [CVE-2026-23949](https://avd.aquasec.com/nvd/cve-2026-23949)
  * Fixed [CVE-2026-1703](https://avd.aquasec.com/nvd/cve-2026-1703)
  * Fixed [CVE-2026-0994](https://avd.aquasec.com/nvd/cve-2026-0994)
  * Fixed [CVE-2026-23490](https://avd.aquasec.com/nvd/cve-2026-23490)
  * Fixed [CVE-2026-22702](https://avd.aquasec.com/nvd/cve-2026-22702)
  * Fixed [CVE-2026-24049](https://avd.aquasec.com/nvd/cve-2026-24049)
  * Fixed [CVE-2026-27199](https://avd.aquasec.com/nvd/cve-2026-27199)
  * Fixed [GHSA-27jp-wm6q-gp25](https://github.com/advisories/GHSA-27jp-wm6q-gp25)
</Update>

<Update label="Astro Runtime 13.4.0" description="January 13, 2026">
  * Airflow version: 2.11.0
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:13.4.0`

  ### Additional improvements

  * Upgrade to Airflow `2.11.0+astro.4` which includes:
    * Fix Redirection on Dag Dependencies Page
    * Fix downgrade for 2.9.0 audit log migration [#57131](https://github.com/apache/airflow/pull/57131)

  ### Security fixes

  * Fixed [CVE-2025-69223](https://avd.aquasec.com/nvd/cve-2025-69223)
  * Fixed [CVE-2025-69227](https://avd.aquasec.com/nvd/cve-2025-69227)
  * Fixed [CVE-2025-69228](https://avd.aquasec.com/nvd/cve-2025-69228)
  * Fixed [CVE-2025-69229](https://avd.aquasec.com/nvd/cve-2025-69229)
  * Fixed [CVE-2025-69224](https://avd.aquasec.com/nvd/cve-2025-69224)
  * Fixed [CVE-2025-69225](https://avd.aquasec.com/nvd/cve-2025-69225)
  * Fixed [CVE-2025-69226](https://avd.aquasec.com/nvd/cve-2025-69226)
  * Fixed [CVE-2025-69230](https://avd.aquasec.com/nvd/cve-2025-69230)
  * Fixed [CVE-2025-68146](https://avd.aquasec.com/nvd/cve-2025-68146)
  * Fixed [CVE-2025-68480](https://avd.aquasec.com/nvd/cve-2025-68480)
  * Fixed [CVE-2025-66418](https://avd.aquasec.com/nvd/cve-2025-66418)
  * Fixed [CVE-2025-66471](https://avd.aquasec.com/nvd/cve-2025-66471)
  * Fixed [CVE-2026-21441](https://avd.aquasec.com/nvd/cve-2026-21441)
  * Fixed [CVE-2025-68158](https://avd.aquasec.com/nvd/cve-2025-68158)
  * Fixed [CVE-2025-12758](https://github.com/advisories/GHSA-vghf-hv5q-vc2g)
</Update>

<Update label="Astro Runtime 13.3.0" description="December 03, 2025">
  * Airflow version: 2.11.0
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:13.3.0`

  ### Additional improvements

  * Updated `astronomer-kubernetes-executor` to version `10.8.1+astro.1`.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 13.3.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-13-3-0).
  * Upgrade `astronomer-airflow-scripts` to `0.0.8`

  ### Security fixes

  * Fixed [CVE-2025-59420](https://nvd.nist.gov/vuln/detail/CVE-2025-59420), [CVE-2026-61920](https://nvd.nist.gov/vuln/detail/CVE-2026-61920), and [CVE-2025-62706](https://nvd.nist.gov/vuln/detail/CVE-2025-62706) for Authlib by upgrading to version 1.6.5.
  * Fixed [CVE-2025-62611](https://nvd.nist.gov/vuln/detail/CVE-2025-62611) for aiomysql by upgrading to version 0.3.2.
  * Fixed [CVE-2025-8869](https://nvd.nist.gov/vuln/detail/CVE-2025-8869) for pip by upgrading to version 25.x.
  * Fixed [CVE-2025-6965](https://nvd.nist.gov/vuln/detail/CVE-2025-6965)
  * Fixed [CVE-2023-31484](https://nvd.nist.gov/vuln/detail/CVE-2023-31484)
  * Fixed [CVE-2025-40909](https://nvd.nist.gov/vuln/detail/CVE-2025-40909)
  * Fixed [CVE-2025-4802](https://nvd.nist.gov/vuln/detail/CVE-2025-4802)
  * Fixed [CVE-2025-8058](https://nvd.nist.gov/vuln/detail/CVE-2025-8058)
  * Fixed [CVE-2025-8714](https://nvd.nist.gov/vuln/detail/CVE-2025-8714)
  * Fixed [CVE-2025-8715](https://nvd.nist.gov/vuln/detail/CVE-2025-8715)
  * Fixed [CVE-2025-8713](https://nvd.nist.gov/vuln/detail/CVE-2025-8713)
  * Fixed [CVE-2023-52969](https://nvd.nist.gov/vuln/detail/CVE-2023-52969)
  * Fixed [CVE-2023-52970](https://nvd.nist.gov/vuln/detail/CVE-2023-52970)
  * Fixed [CVE-2023-52971](https://nvd.nist.gov/vuln/detail/CVE-2023-52971)
  * Fixed [CVE-2025-30693](https://nvd.nist.gov/vuln/detail/CVE-2025-30693)
  * Fixed [CVE-2025-30722](https://nvd.nist.gov/vuln/detail/CVE-2025-30722)
  * Fixed [CVE-2025-3576](https://nvd.nist.gov/vuln/detail/CVE-2025-3576)
  * Fixed [CVE-2025-9230](https://nvd.nist.gov/vuln/detail/CVE-2025-9230)
  * Fixed [CVE-2025-9232](https://nvd.nist.gov/vuln/detail/CVE-2025-9232)
</Update>

<Update label="Astro Runtime 13.2.0" description="September 04, 2025">
  * Airflow version: 2.11.0
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:13.2.0`

  ### Additional improvements

  * Upgrade to Airflow 2.11.0+astro.3 which includes:
    * Make CronTriggerTimetable startup behavior intuitive. [#41558](https://github.com/apache/airflow/pull/41558)
    * Timetable that runs on multiple cron expressions. [#46451](https://github.com/apache/airflow/pull/46451)
    * Fix MultipleCronTriggerTimetable deserialization. [#46886](https://github.com/apache/airflow/pull/46886)
  * Update astronomer-kubernetes-executor to 10.7.0+astro.1.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 13.2.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-13-2-0).

  ### Security fixes

  * Fixed [CVE-2025-53643](https://nvd.nist.gov/vuln/detail/CVE-2025-53643)
  * Fixed [CVE-2025-57804](https://nvd.nist.gov/vuln/detail/CVE-2025-57804)
  * Fixed [CVE-2024-47081](https://nvd.nist.gov/vuln/detail/CVE-2024-47081)
  * Fixed [CVE-2025-50181](https://nvd.nist.gov/vuln/detail/CVE-2025-50181)
  * Fixed [CVE-2025-50182](https://nvd.nist.gov/vuln/detail/CVE-2025-50182)
  * Fixed [CVE-2025-32988](https://avd.aquasec.com/nvd/cve-2025-32988)
  * Fixed [CVE-2025-32990](https://avd.aquasec.com/nvd/cve-2025-32990)
  * Fixed [CVE-2025-32989](https://avd.aquasec.com/nvd/cve-2025-32989)
  * Fixed [CVE-2025-6395](https://avd.aquasec.com/nvd/cve-2025-6395)
</Update>

<Update label="Astro Runtime 13.1.0" description="July 02, 2025">
  * Airflow version: 2.11.0
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:13.1.0`

  ### Additional improvements

  * Stop streaming task logs if the end of log mark is missing [#50715](https://github.com/apache/airflow/pull/50715)
  * Correctly treat re-queues on reschedule sensors as resets after each reschedule [#51410](https://github.com/apache/airflow/pull/51410)
  * Fix archival for cascading deletes by archiving dependent tables first [#51952](https://github.com/apache/airflow/pull/51952)
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 13.1.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-13-1-0).
</Update>

<Update label="Astro Runtime 13.0.0" description="May 20, 2025">
  * Airflow version: 2.11.0
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:13.0.0`

  ### Additional improvements

  * Upgrade to Airflow `2.11.0+astro.1`.
  * Update `astronomer-providers-logging` to `1.6.2`. This adds support for writing task logs to a secondary S3 bucket.
  * Fix `get_health` endpoint to return `None` for inactive standalone Dag processors, preventing them from being reported as unhealthy.
  * Upgraded the major, minor, and patch versions of several open-source provider packages. See [Astro Runtime 13.0.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-13-0-0). As this release includes major provider updates, this release introduces breaking changes in dags. These issues can be resolved by updating provider-specific changes in the dags.
</Update>

<Update label="Astro Runtime 12.12.0" description="March 09, 2026">
  * Airflow version: 2.10.5
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.12.0`

  ### Additional improvements

  * Upgrade to Airflow `2.10.5+astro.4` which includes:
    * Mask proxy and proxies in logs
    * Mask details while creating connections using json & uri [#61882](https://github.com/apache/airflow/pull/61882)
    * Fix redaction of illegal args [#61883](https://github.com/apache/airflow/pull/61883)
    * Fix permissions check in import error APIs [#60801](https://github.com/apache/airflow/pull/60801)
    * Fix import errors not showing on UI [#61163](https://github.com/apache/airflow/pull/61163)
    * Disable use of `LogTemplate` table by default [#61880](https://github.com/apache/airflow/pull/61880)
  * Updated `astronomer-logging-provider` to version `1.6.2`.

  ### Security fixes

  * Fixed [CVE-2025-69223](https://avd.aquasec.com/nvd/cve-2025-69223)
  * Fixed [CVE-2025-69227](https://avd.aquasec.com/nvd/cve-2025-69227)
  * Fixed [CVE-2025-69228](https://avd.aquasec.com/nvd/cve-2025-69228)
  * Fixed [CVE-2025-69229](https://avd.aquasec.com/nvd/cve-2025-69229)
  * Fixed [CVE-2025-69229](https://avd.aquasec.com/nvd/cve-2025-69229)
  * Fixed [CVE-2025-69229](https://avd.aquasec.com/nvd/cve-2025-69229)
  * Fixed [CVE-2025-69224](https://avd.aquasec.com/nvd/cve-2025-69224)
  * Fixed [CVE-2025-69225](https://avd.aquasec.com/nvd/cve-2025-69225)
  * Fixed [CVE-2025-69226](https://avd.aquasec.com/nvd/cve-2025-69226)
  * Fixed [CVE-2025-69230](https://avd.aquasec.com/nvd/cve-2025-69230)
  * Fixed [CVE-2025-68146](https://avd.aquasec.com/nvd/cve-2025-68146)
  * Fixed [CVE-2025-66418](https://avd.aquasec.com/nvd/cve-2025-66418)
  * Fixed [CVE-2025-66471](https://avd.aquasec.com/nvd/cve-2025-66471)
  * Fixed [CVE-2026-21441](https://avd.aquasec.com/nvd/cve-2026-21441)
  * Fixed [CVE-2026-22702](https://avd.aquasec.com/nvd/cve-2026-22702)
  * Fixed [CVE-2026-21226](https://avd.aquasec.com/nvd/cve-2026-21226)
  * Fixed [CVE-2026-22701](https://avd.aquasec.com/nvd/cve-2026-22701)
  * Fixed [CVE-2026-1703](https://avd.aquasec.com/nvd/cve-2026-1703)
  * Fixed [CVE-2026-0994](https://avd.aquasec.com/nvd/cve-2026-0994)
  * Fixed [CVE-2026-23490](https://avd.aquasec.com/nvd/cve-2026-23490)
  * Fixed [CVE-2026-22702](https://avd.aquasec.com/nvd/cve-2026-22702)
  * Fixed [CVE-2026-24049](https://avd.aquasec.com/nvd/cve-2026-24049)
  * Fixed [CVE-2026-23949](https://avd.aquasec.com/nvd/cve-2026-23949)
  * Fixed [GHSA-27jp-wm6q-gp25](https://github.com/advisories/GHSA-27jp-wm6q-gp25)
</Update>

<Update label="Astro Runtime 12.11.0" description="December 03, 2025">
  * Airflow version: 2.10.5
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.11.0`

  ### Additional improvements

  * Updated `astronomer-kubernetes-executor` to version `10.7.0+astro.1`.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 12.11.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-12-11-0).
  * Upgrade `astronomer-airflow-scripts` to `0.0.8`
</Update>

<Update label="Astro Runtime 12.10.0" description="July 02, 2025">
  * Airflow version: 2.10.5
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.10.0`

  ### Additional improvements

  * Stop streaming task logs if the end of log mark is missing [#50715](https://github.com/apache/airflow/pull/50715)
  * Correctly treat re-queues on reschedule sensors as resets after each reschedule [#51410](https://github.com/apache/airflow/pull/51410)
  * Fix archival for cascading deletes by archiving dependent tables first [#51952](https://github.com/apache/airflow/pull/51952)
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 12.10.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-12-10-0).
</Update>

<Update label="Astro Runtime 12.9.0" description="May 05, 2025">
  * Airflow version: 2.10.5
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.9.0`

  ### Additional improvements

  * Add `SCARF_NO_ANALYTICS=True` in Runtime images to disable telemetry by default.
  * Update `astronomer-providers-logging` to `1.6.0`.
  * Update `astronomer-kubernetes-executor` to `10.4.2+astro.1` to allow internal retries when a pending Kubernetes Pod is deleted.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 12.9.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-12-9-0)
</Update>

<Update label="Astro Runtime 12.8.0" description="April 10, 2025">
  * Airflow version: 2.10.5
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.8.0`

  ### Additional improvements

  * Upgraded `apache-airflow-providers-celery==3.10.0` to `apache-airflow-providers-celery==3.10.5`
  * Upgraded `celery==5.4.0` to `celery==5.5.0`.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 12.8.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-12-8-0)
</Update>

<Update label="Astro Runtime 12.7.1" description="February 19, 2025">
  * Airflow version: 2.10.5
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.7.1`

  ### Early access Airflow bug fixes

  * Improved `CronTriggerTimetable` startup behavior [#41558](https://github.com/apache/airflow/pull/41558)
  * Introduced `MultipleCronTriggerTimetable` that allows scheduling dag runs based on multiple cron expressions [#46451](https://github.com/apache/airflow/pull/46451)

  ### Additional improvements

  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 12.7.1 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-12-7-1)
</Update>

<Update label="Astro Runtime 12.7.0" description="February 11, 2025">
  * Airflow version: 2.10.5
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.7.0`

  ### Additional improvements

  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 12.7.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-12-7-0)
</Update>

<Update label="Astro Runtime 12.6.0" description="December 16, 2024">
  * Airflow version: 2.10.4
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.6.0`

  ### Additional improvements

  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 12.6.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-12-6-0)
</Update>

<Update label="Astro Runtime 12.5.0" description="December 02, 2024">
  * Airflow version: 2.10.3
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.5.0`

  ### RHEL UBI 9 images are officially supported

  RHEL UBI 9 images are now officially supported starting with version 12.5. In addition, the following changes have been made since the initial release:

  * **Removed Packages:** `glibc`, `krb5-libs`, `systemd-sysv`, `wget`, `lz4`, `libicu`
  * PostgreSQL installation is now sourced from the UBI repository instead of [download.postgresql.org](https://download.postgresql.org)
  * The EPEL repository has been removed from the UBI image

  ### Early access Airflow bug fixes

  * You can now retry tasks that are stuck in `queued` state [#43520](https://github.com/apache/airflow/pull/43520)

  ### Additional improvements

  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 12.5.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-12-5-0)
</Update>

<Update label="Astro Runtime 12.4.0" description="November 15, 2024">
  * Airflow version: 2.10.3
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.4.0`

  ### Additional improvements

  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 12.4.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-12-4-0)
</Update>

<Update label="Astro Runtime 12.3.0" description="November 06, 2024">
  * Airflow version: 2.10.3
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.3.0`

  ### Additional improvements

  * (Experimental) Introduced a new [RHEL UBI 9](/docs/runtime/runtime-image-architecture#operating-system-support) based image option.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 12.3.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-12-3-0)
</Update>

<Update label="Astro Runtime 12.2.0" description="October 16, 2024">
  * Airflow version: 2.10.2
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.2.0`

  ### Early access Airflow bug fixes

  * Fixed a bug where the executor would not clean up terminated task instances that go too long without a heartbeat [#43065](https://github.com/apache/airflow/pull/43065)
  * Fixed a bug that caused `StaleDataError` by using a different session for writing and deleting recent task instance failures [#42928](https://github.com/apache/airflow/pull/42928)

  ### Additional improvements

  * [Scarf telemetry collected by Airflow](https://airflow.apache.org/docs/apache-airflow/stable/faq.html#does-airflow-collect-any-telemetry-data) is disabled by default.
</Update>

<Update label="Astro Runtime 12.1.1" description="September 20, 2024">
  * Airflow version: 2.10.2
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.1.1`

  ### Additional improvements

  * Upgraded to Airflow 2.10.2. See [Airflow Release Notes](https://airflow.apache.org/docs/apache-airflow/2.10.2/release_notes.html#airflow-2-10-2-2024-09-18) for more information.
</Update>

<Update label="Astro Runtime 12.1.0" description="September 06, 2024">
  * Airflow version: 2.10.1
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.1.0`

  ### Additional improvements

  * Upgraded to Airflow 2.10.1. See [Airflow Release Notes](https://airflow.apache.org/docs/apache-airflow/2.10.1/release_notes.html#airflow-2-10-1-2024-09-05) for more information.
  * Updated the Airflow startup sequence to better isolate dag authors.
  * Upgraded the minor and patch versions of some open-source provider packages. See [Astro Runtime 12.1.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-12-1-0)

  ### Security fixes

  * Fixed [CVE-2024-45034](https://www.cve.org/CVERecord?id=CVE-2024-45034)
  * Fixed [CVE-2024-45498](https://www.cve.org/CVERecord?id=CVE-2024-45498)
</Update>

<Update label="Astro Runtime 12.0.0" description="August 16, 2024">
  * Airflow version: 2.10.0
  * Python versions: 3.10 - 3.12 (default: 3.12)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:12.0.0`

  ### Airflow 2.10.0

  Astro Runtime 12.0.0 includes same-day support for Apache Airflow 2.10, which includes a number of new features and improvements. Airflow 2.10 includes the following changes:

  * Adds decorators for task flow (`@skip_if`, `@run_if`) to make it easier to apply whether or not to skip a task.
  * You can now see `TaskInstance` **Try History** in the Airflow UI
  * Enable ending the task directly from the triggerer without going into the worker.
  * Extended dataset dependencies to support dynamic Dataset Event Emission and Dataset Creation.
  * A new object, `DatasetAlias`, is available to support dynamic Dataset Event Emission and Dataset Creation ([#40478](https://github.com/apache/airflow/pull/40478))
  * Implement accessors to read dataset events defined as inlet ([#39367](https://github.com/apache/airflow/pull/39367))

  For more information about the major changes in this release, see the [Airflow Blog](https://airflow.apache.org/blog/airflow-2.10.0/) or the [Airflow release notes](https://airflow.apache.org/docs/apache-airflow/2.10.0/release_notes.html#airflow-2-10-0-2024-08-15).

  ### Additional improvements

  * Updated OS to Debian 12.6 (bookworm)
  * Updated Python version to 3.12 in default image
  * Upgraded the minor and patch versions of some open-source provider packages. See [Astro Runtime 12.0.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-12-0-0)

  ### Behavior change

  * Since Airflow 2.10 uses Python 3.12, some Python modules have reached end of life, like `imp`, that might create errors in your dags if you still use them. See [Upgrade considerations: Runtime 12](/docs/runtime/version-upgrade-considerations#runtime-12-airflow-2-10) for more information.
</Update>

<Update label="Astro Runtime 11.20.0" description="October 13, 2025">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.20.0`

  ### Additional improvements

  * Upgraded to Airflow `2.9.3+astro.13`, which includes:
    * Fix `try_number` in backport of [#51410](https://github.com/apache/airflow/pull/51410)

  ### Security fixes

  * [CVE-2024-35255](https://www.cve.org/CVERecord?id=CVE-2024-35255)
  * [CVE-2024-47081](https://www.cve.org/CVERecord?id=CVE-2024-47081)
  * [CVE-2024-52304](https://www.cve.org/CVERecord?id=CVE-2024-52304)
  * [CVE-2024-37891](https://www.cve.org/CVERecord?id=CVE-2024-37891)
  * [CVE-2025-50181](https://www.cve.org/CVERecord?id=CVE-2025-50181)
  * [CVE-2025-50182](https://www.cve.org/CVERecord?id=CVE-2025-50182)
  * [CVE-2025-8869](https://www.cve.org/CVERecord?id=CVE-2025-8869)
  * [CVE-2024-56201](https://www.cve.org/CVERecord?id=CVE-2024-56201)
  * [CVE-2024-56326](https://www.cve.org/CVERecord?id=CVE-2024-56326)
  * [CVE-2025-27516](https://www.cve.org/CVERecord?id=CVE-2025-27516)
</Update>

<Update label="Astro Runtime 11.19.0" description="July 02, 2025">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.19.0`

  ### Additional improvements

  * Stop streaming task logs if the end of log mark is missing [#50715](https://github.com/apache/airflow/pull/50715)
  * Correctly treat re-queues on reschedule sensors as resets after each reschedule [#51410](https://github.com/apache/airflow/pull/51410)
  * Fix archival for cascading deletes by archiving dependent tables first [#51952](https://github.com/apache/airflow/pull/51952)
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 11.19.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-19-0).
</Update>

<Update label="Astro Runtime 11.18.0" description="May 05, 2025">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.18.0`

  ### Additional improvements

  * Add `SCARF_NO_ANALYTICS=True` in Runtime images to disable telemetry by default.
  * Upgraded `apache-airflow-providers-celery==3.10.0` to `apache-airflow-providers-celery==3.10.6`
  * Upgraded `celery==5.4.0` to `celery==5.5.0`.
  * Update `astronomer-kubernetes-executor` to `10.4.2+astro.1` to allow internal retries when a pending Kubernetes Pod is deleted.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 11.18.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-18-0)
</Update>

<Update label="Astro Runtime 11.17.0" description="February 26, 2025">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.17.0`

  ### Early access Airflow bug fixes

  * Improved `CronTriggerTimetable` startup behavior [#41558](https://github.com/apache/airflow/pull/41558)
  * Introduced `MultipleCronTriggerTimetable` that allows scheduling dag runs based on multiple cron expressions [#46451](https://github.com/apache/airflow/pull/46451)
  * Mask connection details when creating connections using JSON or URI to ensure consistency with other methods [#46595](https://github.com/apache/airflow/pull/46595)

  ### Additional improvements

  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 11.17.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-17-0)
</Update>

<Update label="Astro Runtime 11.16.0" description="January 21, 2025">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.16.0`

  ### Early access Airflow bug fixes

  * Fixed a bug where the executor would not clean up terminated task instances that go too long without a heartbeat [#42932](https://github.com/apache/airflow/pull/42932)

  ### Additional improvements

  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 11.16.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-16-0)
</Update>

<Update label="Astro Runtime 11.15.1" description="December 06, 2024">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.15.1`

  ### Early access Airflow bug fixes

  * Restore `add_input_dataset` and `add_output_dataset` in NoOpCollector for backward compatibility. [#44681](https://github.com/apache/airflow/pull/44681)
</Update>

<Update label="Astro Runtime 11.15.0" description="December 02, 2024">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.15.0`

  ### Early access Airflow bug fixes

  * You can now retry tasks that are stuck in `queued` state [#43520](https://github.com/apache/airflow/pull/43520)

  ### Additional improvements

  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 11.15.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-15-0)
</Update>

<Update label="Astro Runtime 11.14.0" description="November 15, 2024">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.14.0`

  ### Early access Airflow bug fixes

  * Improved handling value masking of the set variable [#43123](https://github.com/apache/airflow/pull/43123)
  * Masked configuration values that are irrelevant to the dag author [#43040](https://github.com/apache/airflow/pull/43040)

  ### Additional improvements

  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 11.14.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-14-0)
</Update>

<Update label="Astro Runtime 11.13.0" description="October 31, 2024">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.13.0`

  ### Additional improvements

  * (Experimental) Introduced a new [RHEL UBI 9](/docs/runtime/runtime-image-architecture#operating-system-support) based image option.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 11.13.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-13-0)
</Update>

<Update label="Astro Runtime 11.12.0" description="October 25, 2024">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.12.0`

  ### Early access Airflow bug fixes

  * Revert "Fix: Dags are not marked as stale if the dags folder change" [#42197](https://github.com/apache/airflow/pull/42197)
  * Revert "Handle Example dags case when checking for missing files" [#42193](https://github.com/apache/airflow/pull/42193)

  ### Additional improvements

  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 11.12.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-12-0)
</Update>

<Update label="Astro Runtime 11.11.0" description="October 03, 2024">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.11.0`

  ### Early access Airflow bug fixes

  * Fixed a bug in the check served logs logic that caused the UI to show an erroneous 404 error if the user was looking at the logs for a non-running try [#41272](https://github.com/apache/airflow/pull/41272)
  * Fixed a bug where clicking on a `run_id` in a `task_instance` or `dag_run` list incorrectly opened a different `run_id` [#42138](https://github.com/apache/airflow/pull/42138)

  ### Additional improvements

  * Add logging around listener.
  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 11.11.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-11-0)
</Update>

<Update label="Astro Runtime 11.10.1" description="September 06, 2024">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.10.1`

  ### Additional improvements

  * Updated the Airflow startup sequence to better isolate dag authors.
  * Included open-source provider packages reference. See [Astro Runtime 11.10.1 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-10-0)

  ### Security fixes

  * Fixed [CVE-2024-45034](https://www.cve.org/CVERecord?id=CVE-2024-45034)
</Update>

<Update label="Astro Runtime 11.10.0" description="September 02, 2024">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.10.0`

  ### Additional improvements

  * Upgraded the minor and patch versions of several open-source provider packages. See [Astro Runtime 11.10.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-10-0)

  ### Bug fixes

  * Resolved a dag parsing issue where dags were not marked as stale if the `AIRFLOW__CORE__DAGS_FOLDER` was changed. ([#41433](https://github.com/apache/airflow/pull/41433))
  * LocalTaskJob no longer fails on heartbeat due to temporary database connection losses. ([#41704](https://github.com/apache/airflow/pull/41704))
</Update>

<Update label="Astro Runtime 11.9.0" description="August 15, 2024">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.9.0`

  ### Additional improvements

  * Downgrade `apache-airflow-providers-openlineage` to `1.8.0` to prevent scheduler OOM with complex dags
</Update>

<Update label="Astro Runtime 11.8.0" description="August 09, 2024">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.8.0`

  ### Additional improvements

  * Fixed the Tasks API endpoint for scenarios when a dag doesn't have a `start_date` ([#40878](https://github.com/apache/airflow/pull/40878))
  * Added validation for the project URL that comes from installed providers, before displaying the URL in views ([#40933](https://github.com/apache/airflow/pull/40933))
  * Upgraded the minor and patch versions of some open-source provider packages. See [Astro Runtime 11.8.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-8-0)
</Update>

<Update label="Astro Runtime 11.7.0" description="July 17, 2024">
  * Airflow version: 2.9.3
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.7.0`

  ### Airflow 2.9.3

  Astro Runtime 11.7.0 includes same-day support for Apache Airflow 2.9.3. Airflow 2.9.3 contains a number of bug fixes and new features including:

  * The time unit for `scheduled_duration` and `queued_duration` metrics has changed to milliseconds instead of seconds [#37936](https://github.com/apache/airflow/pull/37936)
  * Support for OpenTelemetry metrics on Airflow are now considered **Stable**, and was previously added in Airflow version 2.7.0 as **Experimental** [#40286](https://github.com/apache/airflow/pull/40286)

  For more information, see the [Apache Airflow release notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-2-9-3-2024-07-15).

  ### Additional improvements

  * Upgraded the minor and patch versions of some Astro open source provider packages. See [Astro Runtime 11.7.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-7-0)

  ### Security fixes

  * Fixed [CVE-2024-6345](https://www.cvedetails.com/cve/CVE-2024-6345/)
  * Fixed [CVE-2024-39863](https://www.cve.org/CVERecord?id=CVE-2024-39863)
  * Fixed [CVE-2024-39877](https://www.cve.org/CVERecord?id=CVE-2024-39877)
</Update>

<Update label="Astro Runtime 11.6.0" description="June 28, 2024">
  * Airflow version: 2.9.2
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.6.0`

  ### Early access Airflow bug fixes

  * Fixed a bug where FAB config options, such as `[fab] update_fab_perms`, were not checking for values in the deprecated webserver config section. For example, `[webserver] update_fab_perms` ([#40317](https://github.com/apache/airflow/pull/40317))

  ### Additional improvements

  * Upgraded the minor and patch versions of some Astro open source provider packages. See [Astro Runtime 11.6.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-6-0)
</Update>

<Update label="Astro Runtime 11.5.0" description="June 11, 2024">
  * Airflow version: 2.9.2
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.5.0`

  ### Airflow 2.9.2

  Astro Runtime 11.5.0 includes same-day support for Apache Airflow 2.9.2. Airflow 2.9.2 contains a number of bug fixes including:

  * Resolved a bug where valid dags that worked in Airflow 2.8 and had outlet datasets with specific URIs stopped working depending on how the URI was formatted [(#39670)](https://github.com/apache/airflow/pull/39670)
  * Resolved an issue where the object storage XCOM backend did not serialize correctly, causing custom XCOM backends to sometimes fail [(#39313)](https://github.com/apache/airflow/pull/39313)

  For more information, see the [Apache Airflow release notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html#airflow-2-9-2-2024-06-10).

  ### Additional improvements

  * Upgraded the minor and patch versions of some Astro open source provider packages. See [Astro Runtime 11.5.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-5-0)

  ### Security fixes

  * [CVE-2024-25142](https://www.cve.org/CVERecord?id=CVE-2024-25142)
</Update>

<Update label="Astro Runtime 11.4.0" description="May 28, 2024">
  * Airflow version: 2.9.1
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.4.0`

  ### Additional improvements

  * Upgraded some OSS providers' minor and patch versions. See [Astro Runtime 11.4.0 provider packages](https://www.astronomer.io/docs/astro/runtime-provider-reference#astro-runtime-11-4-0)
  * Added a centralized reference page with all OSS provider package versions listed for each Astro Runtime version. See [Provider package reference](https://www.astronomer.io/docs/astro/runtime-provider-reference)
</Update>

<Update label="Astro Runtime 11.3.0" description="May 06, 2024">
  * Airflow version: 2.9.1
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.3.0`

  ### Early access Airflow bug fixes

  * Fixed a bug affecting custom actions in Airflow plugins that prevents users from running an Astro Runtime environment locally for Astro Runtime versions `11.0.0`-`11.2.0`. Deployments running these versions on Astro are not affected. To continue using `11.0.0`-`11.2.0` locally, set `AIRFLOW__ASTRONOMER__UPDATE_CHECK_INTERVAL=0` in your Astro project `.env` file ([#39421](https://github.com/apache/airflow/pull/39421))

  ### Additional improvements

  * Upgraded some OSS providers' minor and patch versions

  ### Security fixes

  * [CVE-2024-30251](https://www.cve.org/CVERecord?id=CVE-2024-30251)
</Update>

<Update label="Astro Runtime 11.2.0" description="April 26, 2024">
  * Airflow version: 2.9.0
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.2.0`

  <Warning>Due to an [issue related to using custom FAB actions in Airflow plugins](https://github.com/apache/airflow/issues/39144), you might experience an error when you run this version of Astro Runtime locally using the Astro CLI. To resolve this issue, either upgrade directly to Astro Runtime 11.3.0 or set `AIRFLOW__ASTRONOMER__UPDATE_CHECK_INTERVAL=0` in your Astro project `.env` file.</Warning>

  ### Early access Airflow bug fixes

  * Fixed a bug where `airflow db migrate` would throw an error ([#39246](https://github.com/apache/airflow/pull/39246))

  ### Additional improvements

  * Added the [`apache-airflow-providers-mysql`](https://airflow.apache.org/docs/apache-airflow-providers-mysql/stable/index.html) provider
  * Upgraded some OSS providers' minor and patch versions

  ### Security fixes

  * [CVE-2024-4340](https://www.cve.org/CVERecord?id=CVE-2024-4340)
</Update>

<Update label="Astro Runtime 11.1.0" description="April 19, 2024">
  * Airflow version: 2.9.0
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.1.0`

  <Warning>Due to an [issue related to using custom FAB actions in Airflow plugins](https://github.com/apache/airflow/issues/39144), you might experience an error when you run this version of Astro Runtime locally using the Astro CLI. To resolve this issue, either upgrade directly to Astro Runtime 11.3.0 or set `AIRFLOW__ASTRONOMER__UPDATE_CHECK_INTERVAL=0` in your Astro project `.env` file.</Warning>

  ### Additional improvements

  * Updated `sqlparse` to `0.5.0`.
  * Upgraded [Gunicorn](https://gunicorn.org/) to `22.0.0`.
  * Added functionality for using plugins to generate custom menu items in the Airflow UI. This feature will be fully available on Astro in a future release.
</Update>

<Update label="Astro Runtime 11.0.0" description="April 08, 2024">
  * Airflow version: 2.9.0
  * Python versions: 3.9 - 3.11 (default: 3.11)
  * Runtime image: `astrocrpublic.azurecr.io/astronomer/astro-runtime:11.0.0`

  <Warning>Due to an [issue related to using custom FAB actions in Airflow plugins](https://github.com/apache/airflow/issues/39144), you might experience an error when you run this version of Astro Runtime locally using the Astro CLI. To resolve this issue, either upgrade directly to Astro Runtime 11.3.0 or set `AIRFLOW__ASTRONOMER__UPDATE_CHECK_INTERVAL=0` in your Astro project `.env` file.</Warning>

  ### Airflow 2.9.0

  Astro Runtime 11.0.0 includes same-day support for Apache Airflow 2.9, which includes a number of new features and improvements. Airflow 2.9 includes the following changes:

  * New data-aware scheduling lets you use conditional logic (AND / OR) to schedule dags.
  * You can now create your own labels for dynamically mapped tasks with templates, which makes it easier to search through mapped task instances.
  * External XCom backends can now be configured to use object storage.
  * Delivered several significant improvements to the Airflow UI. For example, you can now filter, view, and create datasets through the Airflow UI.
  * New Listener API methods are considered stable and suitable for use in production.
  * Added the ability to automatically pause a dag after a pre-defined number of sequentially failed runs.
  * Dataset URIs are validated when you enter them, and must conform to the rules set in AIP-60. See the [Dataset documentation](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/datasets.html) for more information.

  For more information about the major changes in this release, see the [Airflow Blog](https://airflow.apache.org/blog/airflow-2.9.0/) or the [Airflow release notes](https://airflow.apache.org/docs/apache-airflow/2.9.0/release_notes.html#airflow-2-9-0-2024-04-08).

  #### Upgrade to Python 3.12

  Airflow now supports Python 3.12. However, [Pendulum](https://pendulum.eustace.io/) 2 does not support Python 3.12. If you upgrade to Python 3.12 and want to use Airflow, you also need to upgrade to Pendulum 3.

  Refer to the [Airflow release notes](https://airflow.apache.org/docs/apache-airflow/2.9.0/release_notes.html#official-support-for-python-3-12-38025) for more information about any limitations in Python 3.12 support.

  #### Bug fixes

  * Fixed a bug where after a task failed, and no longer exists in a dag, you can now still access details about the dag in the Grid View of the Airflow UI.
  * Fixed a bug where Airflow would show `failed_upstream` when a dynamically mapped task was `skipped`.
  * In the Python task decorator, you can only have `None` as the default parameter for context parameters.
</Update>


# Astro Runtime maintenance and lifecycle policy
Source: https://astronomer.io/docs/runtime/runtime-version-lifecycle-policy

Learn how Astronomer releases and maintains versions of Astro Runtime.

Astro Runtime is a production ready, data orchestration tool based on Apache Airflow that is distributed as a Docker image and is required by all Astronomer products. It is intended to provide organizations with improved functionality, reliability, efficiency, and performance. Deploying Astro Runtime is a requirement if your organization is using Astro.

Astronomer maintenance and lifecycle policies are part of the distribution. These policies define:

* The maintenance window for specific Astro Runtime versions.
* The frequency of updates.
* The length of time for which support requests will be accepted.

## Astro Runtime maintenance policy

All Runtime versions supporting Airflow 3.0 and newer (Runtime v3.x) receive two years of maintenance in the form of bug fixes, followed by six months of Basic Support. Previously, Runtime releases had differing amounts of time in maintenance based on support policy at the time. This is reflected in the following tables.

Within the maintenance window of each Astro Runtime version, the following is true:

* A set of Docker images corresponding to that version is available for download on [Quay.io](https://quay.io/repository/astronomer/astro-runtime?tab=tags) or `astrocrpublic.azurecr.io/runtime`.
* Astronomer will regularly publish bug or security fixes identified as high-priority.
* Support for paying customers running a maintained version of Astro Runtime is provided by [Astronomer Support](https://cloud.astronomer.io/open-support-request).
* A user can create a new Deployment with the Astro UI, API, or Astro CLI with any maintained `major.minor` version pair of Runtime. For new Deployments, the Astro UI assumes the latest patch.
* When upgrading a Deployment, Astro only allows upgrades to the latest patch version of a given minor version, or to a version already in use within the Organization.

When the maintenance window for a given version of Runtime ends, the following is true:

* New Deployments cannot be created on Astro with an unmaintained version of Runtime. Versions that are no longer maintained will not render as an option in the Deployment creation process from the Astro UI, API, or Astro CLI.
* The Deployment view of the Astro UI will show a warning that encourages the user to upgrade if the Deployment is running an unmaintained version.
* The latest version of the Astro CLI will show a warning if a user pushes an Astro Runtime image to Astronomer that corresponds to an unmaintained version.

Astronomer will not interrupt service for Deployments running Astro Runtime versions that are no longer in maintenance. Unmaintained versions of Astro Runtime are available for local development and testing with the Astro CLI.

### Bug fixes and patch versions

For Runtime versions based on Airflow 3, Astronomer releases patch versions for every in maintenance minor version. These patches are delivered as `-patch`, for example `3.0-2` followed by `3.0-3`. Patches include both Astronomer-specific fixes as well as Apache Airflow community bug fixes. Patches do not need to be explicitly specified and are automatically applied when performing an image build through the Astronomer tagging system. If you have not performed an image update for your Deployment since the most recent patch and are experiencing an issue that could potentially be resolved with a patch, Astronomer Support might instruct you to perform an image update.

For Runtime 11, Runtime 12, and Runtime 13, which are based on Airflow 2, Astronomer only delivers bug fixes through new `minor.patch` versions. If you report an issue with a maintained Astro Runtime image that is not on the latest `minor.patch` version, Astronomer Support might ask that you upgrade your Astro Runtime version to see if that resolves the issue. For example, if you report an issue occurring on a Deployment running Astro Runtime 12.0.0, Astronomer support might ask you to first upgrade to the latest `12.minor.patch` version before troubleshooting your issue any further. If the issue still persists after upgrading, any fixes to that issue will be delivered in a new minor or patch release. Astronomer strives to provide backwards compatibility for all upgrades within the version. For example, you can upgrade directly from 11.0.0 to 11.16.0 and expect no breaking changes unless otherwise stated in documentation.

### Basic Support

Astronomer Support will continue to accept support tickets for an additional period after a Runtime version becomes unmaintained. The ability of Astronomer Support to do deep dives on unmaintained versions of Astro Runtime is very limited, so you may be asked to upgrade to a maintained version if a solution cannot be straightforwardly found on your current version. Astronomer is not obligated to answer questions regarding a Deployment that is running a version that is past its Basic Support window. Astronomer Support strongly recommends remaining on maintained versions of Astro Runtime and limiting the use of Basic Support, as Astronomer will not release any patches for Basic Support versions whatsoever, even in the case of a critical security vulnerability.

### End of maintenance date

Maintenance is discontinued the last day of the month for a given version. For example, if the maintenance window for a version of Astro Runtime is January - June of a given year, that version will be maintained by Astronomer until the last day of June.

### Restricted runtime versions

There are some restricted versions of the Astro runtime that have known bugs and can't be used even if you enabled [deprecated Astro Runtime versions](/docs/runtime/upgrade-astro-runtime#run-a-deprecated-astro-runtime-version).

#### Yanked versions

| Runtime version | Yanked reason                                                          |
| --------------- | ---------------------------------------------------------------------- |
| 8.0.0           | Does not apply cluster policies defined through pluggy.                |
| 9.0.0 - 9.6.0   | Has significant scheduling performance issues.                         |
| 11.0.0 - 11.1.0 | Has db migration issues when you have running triggers.                |
| 13.5.0          | Webserver returns 500 errors for existing user sessions after upgrade. |
| 3.0-9           | Underlying Airflow version, 3.0.5, was yanked by OSS community.        |
| 3.1-8           | Underlying Airflow version, 3.1.4, was yanked by OSS community.        |
| 3.2-1           | Environment manager connections are not found.                         |

## Security

Astronomer continuously checks for available security fixes for software used in Astro Runtime. This process includes scanning language dependencies, container images, and open source threat intelligence sources. When a security fix is available, Astronomer evaluates potential risks for organizations using Astro Runtime and determines deployment priority. Low-priority fixes are deployed following the regular maintenance policy as described in [Astro Runtime maintenance policy](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-maintenance-policy).

If a vulnerability is not yet addressed in a third-party dependency and no official fix is available, Astronomer attempts to address the vulnerability or its impact with environmental mitigations. Whenever possible, Astronomer collaborates with the upstream project to support a timely delivery of the official fix. This process also covers images publicly available on [Quay.io](https://quay.io/repository/astronomer/astro-runtime?tab=tags) and provides context for their vulnerability scanning results.

Astro Runtime releases may or may not include a fix to a Common Vulnerability and Exposure (CVE). When you use a particular Astro Runtime version the day of release, the CVE fixes that version contains might be undisclosed. For security purposes, a description of these fixes will be retroactively added to release notes only once the CVE resolution is published or announced by the upstream project.

If you identify a vulnerability that results in relevant risk for your organization, contact [Astronomer security](mailto:security@astronomer.io).

### Backport policy for bug and security fixes

* **Functional bugs:** When Astronomer identifies a significant functional bug in Astro Runtime, a fix is backported to all maintained versions. To avoid the impact of previously identified bugs, Astronomer recommends that you consistently upgrade Astro Runtime to the latest version.

* **Security vulnerabilities:** When Astronomer identifies a significant security vulnerability in Astro Runtime, a fix is backported and made available as a patch version for all versions in maintenance. A significant security issue is defined as an issue with significant impact and exploitability.

Occasionally, Astronomer might deviate from the defined response policy and backport a bug or security fix to releases other than in-maintenance versions.

## Astro Runtime lifecycle schedule

The following table contains the exact lifecycle for each published version of Astro Runtime.

| Runtime version                                           | Airflow version | Current Support | Release date       | End of maintenance date | End of basic support |
| --------------------------------------------------------- | --------------- | --------------- | ------------------ | ----------------------- | -------------------- |
| [3.3](/docs/runtime/runtime-release-notes#astro-runtime-3-3-1) | 3.3             | Maintenance     | July 9, 2026       | July 2028               | January 2029         |
| [3.2](/docs/runtime/runtime-release-notes#astro-runtime-3-2-1) | 3.2             | Maintenance     | April 14, 2026     | April 2028              | October 2028         |
| [3.1](/docs/runtime/runtime-release-notes#astro-runtime-3-1-1) | 3.1             | Maintenance     | September 26, 2025 | September 2027          | March 2028           |
| [3.0](/docs/runtime/runtime-release-notes#astro-runtime-3-0-1) | 3.0             | Maintenance     | April 22, 2025     | April 2027              | October 2027         |
| [13](/docs/runtime/runtime-release-notes#astro-runtime-13-0-0) | 2.11            | Maintenance     | May 20, 2025       | April 2027              | April 2028           |
| [12](/docs/runtime/runtime-release-notes#astro-runtime-12-0-0) | 2.10            | Basic Support   | August 16, 2024    | February 2026           | February 2027        |
| [11](/docs/runtime/runtime-release-notes#astro-runtime-11-0-0) | 2.9             | Basic Support   | April 8, 2024      | October 2025            | October 2026         |

If you have any questions or concerns, contact [Astronomer support](https://cloud.astronomer.io/open-support-request).

### Astro Runtime Python version images

Starting with Astro Runtime 9, Astronomer maintains different Astro Runtime images for each [supported Python version](/docs/runtime/runtime-image-architecture#python-versioning).
If a Python version goes out of support while that Runtime version is in support, Astronomer will not publish a Runtime version for that Python version.

## Legacy Astro Runtime versions

The following table contains all major Runtime releases that are no longer maintained. Support policies at the time of many of these releases was different, and the dates reflect the time maintenance and support was ended.

<Info>
  Runtime Version 4.2.9 is the lowest deprecated Runtime version that can be used with Astro.
</Info>

| Runtime version                                                                                              | Airflow version | Release date       | End of maintenance date | End of basic support |
| ------------------------------------------------------------------------------------------------------------ | --------------- | ------------------ | ----------------------- | -------------------- |
| [4](https://github.com/astronomer/astronomer-docs-resources/tree/main/astro/runtime/4.0.0) (LTS)             | 2.2             | March 10, 2022     | September 2023          | September 2024       |
| [5](https://github.com/astronomer/astronomer-docs-resources/tree/main/astro/runtime/5.0.0) (LTS)             | 2.3             | April 30, 2022     | April 2024              | December 2024        |
| [6](https://github.com/astronomer/astronomer-docs-resources/tree/main/astro/runtime/6.0.0-airflow-2.4) (LTS) | 2.4             | September 19, 2022 | March 2024              | December 2024        |
| [7](https://github.com/astronomer/astronomer-docs-resources/tree/main/astro/runtime/7.0.0-airflow-2.5)       | 2.5             | December 3, 2022   | July 2023               | July 2024            |
| [8](https://github.com/astronomer/astronomer-docs-resources/tree/main/astro/runtime/8.0.0-airflow-2.6)       | 2.6             | April 30, 2023     | November 2023           | October 2025         |
| [9](https://github.com/astronomer/astronomer-docs-resources/tree/main/astro/runtime/9.0.0-airflow-2.7)       | 2.7             | August 18, 2023    | January 2025            | January 2026         |
| [10](https://github.com/astronomer/astronomer-docs-resources/tree/main/astro/runtime/10.0.0-airflow-2.8)     | 2.8             | December 18, 2023  | June 2024               | June 2025            |


# Astro: Upgrade Runtime
Source: https://astronomer.io/docs/runtime/upgrade-astro-runtime

Learn how to upgrade the Astro Runtime version on your Deployments. To take advantage of new features and bug and security fixes, upgrade Astro Runtime when a new version becomes available.

New versions of Astro Runtime are released regularly to support new Astro and Apache Airflow functionality. To take advantage of new features and bug and security fixes, Astronomer recommends upgrading Astro Runtime as new versions are available.

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/install-cli).
* An [Astro project](/docs/cli/v1.43/get-started-cli).
* An [Astro Deployment](/docs/astro/create-deployment).

<Info>If you're only upgrading a local Airflow environment, you don't need an Astro Deployment and you can skip steps 7-9.</Info>

<Note>These instructions cover upgrading Astro Runtime versions within the same major Airflow version. For example, Airflow 2.x → 2.y or Airflow 3.x → 3.y.
If you are upgrading from Airflow 2 to Airflow 3, follow the [Airflow 3 upgrade guide](/docs/astro/airflow3/upgrade-af3) instead.</Note>

## Step 1: Review upgrade considerations

Astro upgrades can include breaking changes, especially when you're upgrading to a new major version. Check the [upgrade considerations](/docs/runtime/version-upgrade-considerations) for your upgrade version to anticipate any breaking changes or upgrade-specific instructions before you proceed.

<Warning>
  On Airflow 3, Runtime upgrades that include a database migration (and Runtime downgrades) can temporarily disrupt task execution and Deployment availability during the upgrade window. Airflow 3 workers depend on the API server, so when the Airflow operator recreates components during a migration, in-flight tasks may fail heartbeats until the new API server pods are healthy. Plan upgrades during a maintenance window, pause Dags beforehand, or ensure your tasks tolerate retries. For details, see [Airflow 3 runtime upgrades and task disruption](/docs/runtime/version-upgrade-considerations#airflow-3-runtime-upgrades-and-task-disruption).
</Warning>

<a />

## Step 2: (Optional) Pin provider package versions

Major Astro Runtime upgrades can include major upgrades to built-in provider packages. These package upgrades can sometimes include breaking changes for your Dags. See the [Apache Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers/packages-ref.html) for a list of all available provider packages and their release notes.

For the most stable upgrade path, Astronomer recommends pinning all provider package versions from your current Runtime version before upgrading. To check the version of all provider packages installed in your Runtime version, run:

```sh wrap theme={null}
docker run --rm quay.io/astronomer/astro-runtime:<current-runtime-version> pip freeze | grep apache-airflow-providers
```

After reviewing this list, pin the version for each provider package in your Astro project `requirements.txt` file. For example, Runtime 7.4.1 uses version 4.0.0 of `apache-airflow-providers-databricks`. To pin this version of the Databricks provider package when you upgrade to a later version of Runtime, you add the following line to your `requirements.txt` file:

```text wrap theme={null}
apache-airflow-providers-databricks==4.0.0
```

## Step 3: (Optional) Run upgrade tests with the Astro CLI

You can use the Astro CLI to anticipate and address problems before upgrading to a newer version of Astro Runtime. Before you upgrade, run the following command to run tests against the version of Astro Runtime you're upgrading to:

```bash wrap theme={null}
astro dev upgrade-test --runtime-version <upgraded-runtime-version>
```

The Astro CLI then generates test results in your Astro project that identify dependency conflicts and import errors that you would experience using the new Astro Runtime version. Review these results and make the recommended changes to reduce the risk of your project generating errors after you upgrade. For more information about using this command and the test results, see [Test before an Astro Runtime upgrade](/docs/cli/v1.43/test-your-astro-project-locally#test-before-an-astro-runtime-upgrade).

## Step 4: Update your Dockerfile

<Note>These are instructions for upgrading minor versions on Airflow 2 or Airflow 3. For upgrading to Airflow 3, see the [Airflow 3 upgrade guide](/docs/astro/airflow3/upgrade-af3).</Note>

1. In your Astro project, open your `Dockerfile`.
2. Change the Docker image in the `FROM` statement of your `Dockerfile` to a new version of Astro Runtime. For Airflow 2-based Runtimes, specify the major, minor, and patch version of Astro Runtime. For Airflow 3-based Runtimes, specify only the major and minor version. Exclude the patch version to automatically use the latest available patch.

<Info>
  **Airflow version–specific registry URLs**

  Airflow 2.x-based Runtime versions and Airflow 3.x-based Runtime versions have different registry URLs. See [Docker Registry URLs](/docs/runtime/runtime-image-architecture#container-registry-urls) for information on which URL to pull from.
</Info>

3. Save the changes to your `Dockerfile`.

## Step 5: Test Astro Runtime locally

Astronomer recommends testing new versions of Astro Runtime locally before upgrading a Deployment on Astro.

1. Open your project directory in your terminal and run `astro dev restart`. This restarts the Docker containers for the Airflow webserver, scheduler, triggerer, and Postgres metadata database.

2. Access the Airflow UI of your local environment by navigating to `http://localhost:8080` in your browser.

3. Confirm that your local upgrade was successful by scrolling to the end of any page. The new Astro Runtime version is listed in the footer as well as the version of Airflow it is based on.

   <Frame>
     <img alt="Runtime Version banner - Local" />
   </Frame>

4. (Optional) Run Dags locally to ensure that all of your code works as expected. If you encounter errors after your upgrade, it's possible that your new Astro Runtime version includes a breaking provider package change. If you encounter one of these breaking changes, follow the steps in [Pin provider package versions](#pin-provider-package) to check your provider package versions and, if required, pin the provider package version from your previous Runtime version in your `requirements.txt` file.

## Step 6: (Optional) Upgrade and test provider packages

If you pinned provider package versions before your upgrade, upgrade your provider packages by changing the pinned version in your `requirements.txt` file. Test each provider package upgrade locally before deploying to Astro.

## Step 7: Deploy to Astronomer

To push your upgraded project to an Astro Deployment, run:

```sh wrap theme={null}
astro deploy
```

For more information about deploying to Astro, see [Deploy code](/docs/astro/deploy-code).

<Warning>
  After you upgrade a Deployment on Astro to a new version of Astro Runtime, the only way to downgrade is to [roll back to a previous deploy](/docs/astro/deploy-history). If you attempt to downgrade a Deployment by updating your Dockerfile, the Astro CLI produces an error and your request to deploy does not succeed.

  Generally speaking, Deployment rollbacks to lower Runtime versions are recommended only when your current code isn't working as expected. This is because rollbacks to lower Runtime versions can result in your Deployment losing data from the metadata database. For more information, see [What happens during a deploy rollback](/docs/astro/deploy-history#what-happens-during-a-deploy-rollback).
</Warning>

## Step 8: Confirm your upgrade on Astro

1. In the Astro UI, click **Deployments**, then select a Deployment.
2. Click **Open Airflow**.
3. In the Airflow UI, scroll to the bottom of any page. You should see your new Runtime version in the footer:

   <Frame>
     <img alt="Runtime Version banner - Astro" />
   </Frame>

   You will also see an **Image tag** for your deploy. This tag is shown only for Deployments on Astro and is not generated for changes in a local environment.

## Step 9: (Remote Execution Deployments only) Upgrade Remote Execution Agents

If your Deployment uses [Remote Execution](/docs/astro/execution-mode), you must upgrade your Remote Execution Agents to a version compatible with your new Astro Runtime.

<Info>
  Agent and Runtime version compatibility is critical for Remote Execution Deployments.
  Always select an [agent image](/docs/astro/agent-images) that matches your Deployment's Astro Runtime version, never newer.
  The Airflow version on Remote Execution Agents must be **equal to or lower than** the Airflow version configured for the orchestration plane.
</Info>

1. Refer to the [Remote Execution Agent release notes](/docs/astro/agent-release-notes) to identify the correct Agent version for your Astro Runtime version.
2. Find the base image tags in the [Remote Execution Agent image reference](/docs/astro/agent-images).
3. Build and push an updated remote agent image, or pull a prebuilt image if suitable for your environment. For detailed instructions, see [Build and deploy Remote Execution Agent images](/docs/astro/deploy-project-remote-execution).
4. Update your Agent deployment. For example, modify the agent image reference in your Helm `values.yaml` file, then run a `helm upgrade`.
5. Confirm your agents are running the new version and healthy in the Astro UI.

For full upgrade and maintenance instructions, see [Remote Execution Agents](/docs/astro/remote-execution-configure-agents#manage-remote-execution-agents).

## Upgrade considerations

Consider the following when you upgrade Astro Runtime:

* All versions of the Astro CLI support all versions of Astro Runtime. There are no dependencies between the two products.
* Upgrading to certain versions of Runtime might result in extended upgrade times or otherwise disruptive changes to your environment. To learn more, see [Version-specific upgrade considerations](/docs/runtime/version-upgrade-considerations).
* Astro only allows upgrades to the latest patch version of a given minor version, or to a version that is already in use within your Organization. If you attempt to upgrade to a non-latest patch version, you receive the following error:

  ```text wrap theme={null}
  You can only upgrade to either the latest patch version of any given minor version or a version already used within your organization
  ```
* You can't downgrade a Deployment on Astro to a lower version of Astro Runtime unless you [roll back to a previous deploy](/docs/astro/deploy-history).

To stay up to date on the latest versions of Astro Runtime, see [Astro Runtime release notes](/docs/runtime/runtime-release-notes). For more information on Astro Runtime versioning and support, see [Astro Runtime versioning and lifecycle policy](/docs/runtime/runtime-version-lifecycle-policy). For differences in registry URLs between Airflow 2.x-based and Airflow 3.x-based Runtime versions, see [Docker Registry URLs](/docs/runtime/runtime-image-architecture#container-registry-urls).

### Run a deprecated Astro Runtime version

If you're migrating to Astro from OSS Airflow or Astro Hybrid, where your Deployments run using an older version of Airflow or a deprecated version of the Astro Runtime, you can now use stepwise migration to complete your migration in stages. First, you can create Deployments with a deprecated version of the Astro Runtime using the Astro API to move to an Astro Hosted environment. Then, you can upgrade your code to the most up-to-date version of the Astro Runtime.

Deployments that run deprecated Astro Runtime versions will only receive support as defined in the [Astro Runtime maintenance policy](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-maintenance-policy), and Astronomer might advise you to upgrade your Runtime to the latest version of Astro to resolve performance issues or bugs. Runtime version 4.2.9 is the lowest deprecated Runtime version that can be used with Astro.

<Info>When you choose a deprecated version of Astro Runtime, Astronomer recommends always choosing the latest patch version that includes your desired Airflow version. Some earlier patch versions of Astro Runtime include unexpected behavior and can't be used even as a deprecated version. For more information about which versions are affected, see the [Version upgrade considerations](/docs/runtime/version-upgrade-considerations).</Info>

1. Contact your Astronomer account team and request the ability to run deprecated versions of Astro Runtime.

2. Create a [Personal Access Token](https://cloud.astronomer.io/token) (PAT) to use for authentication to Astro.

   <Info>You only need a PAT to create the first Deployment in your Organization running a deprecated version of Astro Runtime. You can then create all subsequent Deployments using either a Workspace or Organization API token with permissions to create a Deployment.</Info>

3. Create a Deployment using the [Astro API](/docs/astro/api/v-1/deployment/create-a-deployment). In your request, specify your PAT in the authorization header, and your deprecated Astro Runtime version using the `astroRuntimeVersions` property.

After you create the Deployment, you can manage the Deployment using any interface, including the Astro CLI and [Astro UI](/docs/astro/deployment-settings). You can also create additional Deployments with deprecated Runtimes using the Astro API and either a Workspace or Organization API token.


# Version upgrade considerations
Source: https://astronomer.io/docs/runtime/version-upgrade-considerations

Review upgrade considerations for specific Astro Runtime versions, including breaking changes, database migrations, and known issues.

Some Astro Runtime versions require specific upgrade considerations, such as breaking changes, database migrations, or known issues. If an Astro Runtime version isn't listed in this section, then no specific upgrade actions are required for that version.

## Airflow 3 Runtime upgrades and task disruption

Astro Runtime upgrades on Airflow 3 that include a database migration (and Runtime downgrades) can temporarily disrupt task execution and deployment availability during the upgrade window. Unlike Airflow 2, where workers connected directly to Postgres through PgBouncer, Airflow 3 workers depend on the API server. When the Airflow operator tears down and recreates Airflow components during a migration, the API server is replaced and worker pods that are mid-task can fail heartbeats until the new API server pods are healthy.

In practice, this means:

* Running tasks may fail or be marked as zombies during the upgrade window because workers can't reach the API server to send heartbeats.
* Multiple task instances across multiple worker pods can be affected, since this is a platform-side disruption rather than a single-task issue.
* Health checks against the API server can disappear entirely during the outage window and only resume after the new pods come up.

Astronomer recommends scheduling Airflow 3 Runtime upgrades during a maintenance window, pausing Dags before the upgrade, or otherwise tolerating retries for tasks that may be running when you start the upgrade. The Airflow operator doesn't wait for worker pods to scale to zero before recreating the API server, because worker pods can take up to 24 hours to terminate and waiting would make upgrades and downgrades take an unacceptably long time.

## Runtime 3.2 (Airflow 3.2)

### Runtime 3.2-3 and later

If you use custom user roles, users with read-only access to Dags can no longer view the Dag list in the Airflow UI because the endpoint now requires additional permissions to return aggregated data from multiple entities. Astronomer-provided roles already include these permissions and aren't affected. This change addresses [CVE-2026-38743](https://www.cve.org/CVERecord?id=CVE-2026-38743).

Update your custom user roles to include read access for Dag Runs, Task Instances, and HITL Details for users who should still be able to view the Dag list in the Airflow UI.

## Runtime 3.1 (Airflow 3.1)

### Runtime 3.1-15 and later

If you use custom user roles, users with read-only access to Dags can no longer view the Dag list in the Airflow UI because the endpoint now requires additional permissions to return aggregated data from multiple entities. Astronomer-provided roles already include these permissions and aren't affected. This change addresses [CVE-2026-38743](https://www.cve.org/CVERecord?id=CVE-2026-38743).

Update your custom user roles to include read access for Dag Runs, Task Instances, and HITL Details for users who should still be able to view the Dag list in the Airflow UI.

### Runtime 3.1-1 to 3.1-2

<Warning>
  This Runtime version has a significant memory leak with Celery workers. Astronomer highly recommends upgrading to 3.1-3 or updating your Deployment to use Astro Executor.
</Warning>

## Runtime 3.0 (Airflow 3.0)

### Runtime 3.0-6 to 3.0-12

<Warning>
  This Runtime version has a significant memory leak with Celery workers. Astronomer highly recommends upgrading to 3.1-3 or updating your Deployment to use Astro Executor.
</Warning>

### Runtime 3.0-9

This Runtime version is restricted and you can't upgrade to it or create Astro Deployments using it. Airflow 3.0.5 was `yanked` due to a bug with `conn.extra_dejson` masking that causes tasks to fail ([#54768](https://github.com/apache/airflow/issues/54768)), so the underlying Airflow version isn't available for use.

### Runtime 3.0-2

The Remote Execution Agent based on Airflow 3.0-1 has a known incompatibility with Astro Runtime 3.0-2. If you use remote execution, don't upgrade to or create Remote Deployments that use the combination of Remote Execution Agent 1.0.0 and Astro Runtime 3.0-2. A fix for the incompatibility shipped in Astro Runtime 3.0-3.

If you created a new Remote Deployment with 3.0-2, you can't roll back to 3.0-1. You must either create a new Remote Deployment with 3.0-1 or upgrade to 3.0-3 or later.

If you upgraded a Deployment from 3.0-1 to 3.0-2, Astronomer advises you to downgrade your Deployments or roll back to 3.0-1 by using a Dockerfile change or by rolling back your Deployment in the Astro UI.

* [Roll back to previous deploys](/docs/astro/deploy-history#what-happens-during-a-deploy-rollback)
* [Roll back Deployments after a broken upgrade](/docs/astro/best-practices/upgrading-astro-runtime#roll-back-deployments-after-a-broken-upgrade)

## Runtime 12 (Airflow 2.10)

### Upgrade to Python 3.12

Starting with Python version 3.12, Python has removed the module [`imp`](https://docs.python.org/3.11/library/imp.html). This can cause errors for your Dags if you use `imp` to access materials using the `import` statement. Python recommends migrating to [`importlib`](https://docs.python.org/3.11/library/importlib.html#module-importlib) instead.

See [How do I migrate from `imp`](https://discuss.python.org/t/how-do-i-migrate-from-imp/27885) guidance from Python for remediation steps.

## Runtime 11 (Airflow 2.9)

### Restricted versions of Runtime 11

You can't create new Deployments or upgrade to the following versions of Runtime 11, which are `yanked`. These [restricted runtime versions](/docs/runtime/runtime-version-lifecycle-policy#restricted-runtime-versions) prevent you from upgrading to or creating a Deployment with a version that contains a known limitation or bug.

* 11.0.0
* 11.1.0

### Bug affecting users running a local Astro Runtime environment

In Airflow 2.9, a bug affecting custom actions in Airflow plugins ([#39421](https://github.com/apache/airflow/pull/39421)) prevented users from running the Astro Runtime environment locally for Astro Runtime versions 11.0.0, 11.1.0, and 11.2.0. Deployments running these versions on Astro aren't affected.

To continue using these versions of Astro Runtime locally, set `AIRFLOW__ASTRONOMER__UPDATE_CHECK_INTERVAL=0` in your Astro project `.env` file.

## Runtime 9 (Airflow 2.7)

### Restricted versions of Runtime 9

You can't create new Deployments or upgrade to the following versions of Runtime 9, which are `yanked`. These [restricted runtime versions](/docs/runtime/runtime-version-lifecycle-policy#restricted-runtime-versions) prevent you from upgrading to or creating a Deployment with a version that contains a known limitation or bug.

* 9.0.0
* 9.1.0
* 9.2.0
* 9.3.0
* 9.4.0
* 9.5.0
* 9.6.0

### Connection testing in the Airflow UI disabled by default

In Airflow 2.7, connection testing in the Airflow UI is disabled by default. Astronomer doesn't recommend reenabling the feature unless you fully trust all users with edit or delete permissions for Airflow connections.

To reenable the feature, set the following environment variable in your Astro project Dockerfile:

```dockerfile wrap theme={null}
ENV AIRFLOW__CORE__TEST_CONNECTION=Enabled
```

### Upgrade to Python 3.11

The base distribution of Astro Runtime 9 uses Python 3.11 by default. Some provider packages, such as `apache-airflow-providers-apache-hive`, aren't compatible with Python 3.11.

To continue using these packages with a compatible version of Python, upgrade to the [Astro Runtime Python distribution](/docs/runtime/runtime-image-architecture#python-version-images) for your desired Python version.

## Runtime 8 (Airflow 2.6)

Astro Runtime version 8.0.0 is a restricted version of the Astro Runtime (`yanked`), which means you can't create Deployments on Astro with this runtime version. These [restricted runtime versions](/docs/runtime/runtime-version-lifecycle-policy#restricted-runtime-versions) prevent you from upgrading to or creating a Deployment with a version that contains a known limitation or bug.

### Breaking change to `apache-airflow-providers-cncf-kubernetes` in version 8.4.0

Astro Runtime 8.4.0 upgrades `apache-airflow-providers-cncf-kubernetes` to 7.0.0, which includes a breaking change by [removing some deprecated features from `KubernetesHook`](https://github.com/apache/airflow/commit/a1f5a5425e65c40e9baaf5eb4faeaed01cee3569). If you are using any of these features, either pin your current version of `apache-airflow-providers-cncf-kubernetes` to your `requirements.txt` file or ensure that you don't use any of the deprecated features before upgrading.

### Upgrade directly to Astro Runtime 8.1

Astro Runtime 8.0 introduced a number of bugs and dependency conflicts which were subsequently fixed in Runtime 8.1. As a result, Astro Runtime 8.0 isn't available in the Astro UI and no longer supported by Astronomer. To use Airflow 2.6, upgrade directly to Runtime 8.1.

### Package dependency conflicts

Astro Runtime 8 includes fewer default dependencies than previous versions. Specifically, the following provider packages are no longer installed by default:

* `apache-airflow-providers-apache-hive`
* `apache-airflow-providers-apache-livy`
* `apache-airflow-providers-databricks`
* `apache-airflow-providers-dbt-cloud`
* `apache-airflow-providers-microsoft-mssql`
* `apache-airflow-providers-sftp`
* `apache-airflow-providers-snowflake`

If your Dags depend on any of these provider packages, add the provider packages to your Astro project `requirements.txt` file before upgrading. You can also [pin specific provider package versions](/docs/runtime/upgrade-astro-runtime#pin-provider-package) to ensure that none of your provider packages change after upgrading.

### Upgrade to Python 3.10

Astro Runtime 8 uses Python 3.10. If you use provider packages that don't yet support Python 3.10, use one of the following options to stay on Python 3.9:

* Run your tasks using the `KubernetesPodOperator` or `PythonVirtualenvOperator`. You can configure the environment that these tasks run in to use Python 3.9.
* Use the [`astronomer-provider-venv`](https://github.com/astronomer/astro-provider-venv) to configure a custom virtual environment that you can apply to individual tasks.

### Provider incompatibilities

There is an incompatibility between Astro Runtime 8 and the following provider packages installed together:

* `apache-airflow-providers-cncf-kubernetes==6.1.0`
* `apache-airflow-providers-google==10.0.0`

That can be resolved by pinning `apache-airflow-providers-google==10.9.0` or greater in your `requirements.txt` file.

This incompatibility breaks the `GKEStartPodOperator`. This operator inherits from the `KubernetesPodOperator`, but then overrides the hook attribute with the `GKEPodHook`. In the included version of the `cncf-kubernetes` providers package, the `KubernetesPodOperator` uses a new method, `get_xcom_sidecar_container_resources`. This method is present in the `KubernetesHook`, but not the `GKEPodHook`. Therefore, when it is called it causes the task execution to break.

## Runtime 6 (Airflow 2.4)

Smart Sensors were deprecated in Airflow 2.2.4 and removed in Airflow 2.4.0. If your organization is still using Smart Sensors, you'll need to start using deferrable operators. See [Deferrable operators](/docs/learn/deferrable-operators).

## Runtime 5 (Airflow 2.3)

Astro Runtime 5.0.0, based on Airflow 2.3, includes changes to the schema of the Airflow metadata database. When you first upgrade to Runtime 5.0.0, consider the following:

* Upgrading to Runtime 5.0.0 can take 10 to 30 minutes or more depending on the number of task instances that have been recorded in the metadata database throughout the lifetime of your Deployment on Astro.
* Once you upgrade successfully to Runtime 5, you might see errors in the Airflow UI that warn you of incompatible data in certain tables of the database. For example:

  ```txt wrap theme={null}
  Airflow found incompatible data in the `dangling_rendered_task_instance_fields` table in your metadata database, and moved...
  ```

  These warnings have no impact on your tasks or Dags and can be ignored. If you want to remove these warning messages from the Airflow UI, contact [Astronomer support](https://cloud.astronomer.io/open-support-request). If requested, Astronomer can drop incompatible tables from your metadata database.

For more information on Airflow 2.3, see ["Apache Airflow 2.3.0 is here"](https://airflow.apache.org/blog/airflow-2.3.0/) or the [Airflow 2.3.0 changelog](https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html#airflow-2-3-0-2022-04-30).


# astro deployment
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment

Manage details about Deployments.

<Info>
  The behavior and format of these commands differs depending on what Astronomer
  product you're using.
</Info>

Use `astro deployment` commands to manage all details about Deployments, including Deployment resources, Airflow objects, and Astro Runtime versioning.

<CardGroup>
  <Card title="astro deployment airflow upgrade" href="/cli/v1.45/astro-deployment-airflow-upgrade">
    View documentation for `astro deployment airflow upgrade`.
  </Card>

  <Card title="astro deployment airflow-variable" href="/cli/v1.45/astro-deployment-airflow-variable-list">
    View documentation for `astro deployment airflow-variable`.
  </Card>

  <Card title="astro deployment adopt" href="/cli/v1.45/astro-deployment-adopt">
    View documentation for `astro deployment adopt`.
  </Card>

  <Card title="astro deployment connection" href="/cli/v1.45/astro-deployment-connection-list">
    View documentation for `astro deployment connection`.
  </Card>

  <Card title="astro deployment create" href="/cli/v1.45/astro-deployment-create">
    View documentation for `astro deployment create`.
  </Card>

  <Card title="astro deployment delete" href="/cli/v1.45/astro-deployment-delete">
    View documentation for `astro deployment delete`.
  </Card>

  <Card title="astro deployment hibernate" href="/cli/v1.45/astro-deployment-hibernate">
    View documentation for `astro deployment hibernate`.
  </Card>

  <Card title="astro deployment inspect" href="/cli/v1.45/astro-deployment-inspect">
    View documentation for `astro deployment inspect`.
  </Card>

  <Card title="astro deployment list" href="/cli/v1.45/astro-deployment-list">
    View documentation for `astro deployment list`.
  </Card>

  <Card title="astro deployment logs" href="/cli/v1.45/astro-deployment-logs">
    View documentation for `astro deployment logs`.
  </Card>

  <Card title="astro deployment pool" href="/cli/v1.45/astro-deployment-pool-list">
    View documentation for `astro deployment pool`.
  </Card>

  <Card title="astro deployment runtime upgrade" href="/cli/v1.45/astro-deployment-runtime-upgrade">
    View documentation for `astro deployment runtime upgrade`.
  </Card>

  <Card title="astro deployment service account" href="/cli/v1.45/astro-deployment-service-account">
    View documentation for `astro deployment service account`.
  </Card>

  <Card title="astro deployment team" href="/cli/v1.45/astro-deployment-team">
    View documentation for `astro deployment team`.
  </Card>

  <Card title="astro deployment token" href="/cli/v1.45/astro-deployment-token-list">
    View documentation for `astro deployment token`.
  </Card>

  <Card title="astro deployment unadopt" href="/cli/v1.45/astro-deployment-unadopt">
    View documentation for `astro deployment unadopt`.
  </Card>

  <Card title="astro deployment update" href="/cli/v1.45/astro-deployment-update">
    View documentation for `astro deployment update`.
  </Card>

  <Card title="astro deployment user" href="/cli/v1.45/astro-deployment-user">
    View documentation for `astro deployment user`.
  </Card>

  <Card title="astro deployment variable" href="/cli/v1.45/astro-deployment-variable-list">
    View documentation for `astro deployment variable`.
  </Card>

  <Card title="astro deployment wake up" href="/cli/v1.45/astro-deployment-wake-up">
    View documentation for `astro deployment wake up`.
  </Card>

  <Card title="astro deployment worker-queue" href="/cli/v1.45/astro-deployment-worker-queue-create">
    View documentation for `astro deployment worker-queue`.
  </Card>
</CardGroup>


# astro deployment adopt
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-adopt

Adopt an existing operator-managed Airflow custom resource into Astro Private Cloud (Astro Private Cloud only).

<Info>This command is available only if you're authenticated to an Astro Private Cloud installation.</Info>

Adopt an existing operator-managed Airflow custom resource into Astro Private Cloud. Adopting creates the corresponding Deployment record in Astro Private Cloud. It also updates the custom resource. It reconfigures the ingress and the authentication for the Airflow UI. With `--use-apc-logging`, it also updates the logging configuration. The Deployment's namespace and its metadata database don't change.

## Usage

```sh wrap theme={null}
astro deployment adopt --cluster-id=<cluster-id> --name=<cr-name> --namespace=<cr-namespace>
```

## Options

| Option                       | Description                                                                                              | Possible Values                                |
| ---------------------------- | -------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
| `--accept-incompatibilities` | Adopt the Deployment even if the custom resource has fields with no Astro Private Cloud representation   | `true` or `false`. The default value is `true` |
| `--cluster-id` (*Required*)  | The ID of the cluster the Airflow custom resource is running on                                          | Any valid cluster ID                           |
| `--description`              | A description for the adopted Deployment                                                                 | Any string                                     |
| `-l`, `--label`              | A label for the adopted Deployment. If omitted, Astro Private Cloud uses the `--name` value              | Any string                                     |
| `--name` (*Required*)        | The `metadata.name` of the existing Airflow custom resource                                              | Any string                                     |
| `--namespace` (*Required*)   | The Kubernetes namespace of the existing Airflow custom resource                                         | Any string                                     |
| `--use-apc-logging`          | Route the adopted Deployment's logs through Astro Private Cloud logging                                  | None                                           |
| `--use-apc-registry`         | Use the Astro Private Cloud in-cluster registry for the adopted Deployment. You must pre-sync its images | None                                           |

## Examples

```sh wrap theme={null}
astro deployment adopt --cluster-id=cmra8iy730008e519f0hma7nx --name=ce04 --namespace=ce04 --use-apc-logging --use-apc-registry --label="My custom label" --description="Adopted from standalone operator"
```

## Related commands

* [`astro deployment unadopt`](/docs/cli/v1.45/astro-deployment-unadopt)
* [`astro deployment list`](/docs/cli/v1.45/astro-deployment-list)


# astro deployment airflow upgrade
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-airflow-upgrade

Upgrade Airflow (Astro Private Cloud only).

<Info>This command is available only if you're authenticated to an Astro Private Cloud installation.</Info>

Initializes the Airflow version upgrade process on any Airflow Deployment on Astronomer. See [Upgrade Airflow on Astro Private Cloud](https://www.astronomer.io/docs/software/manage-airflow-versions)

## Usage

Run `astro deployment airflow upgrade --deployment-id` to initialize the Airflow upgrade process. To finalize the Airflow upgrade process, complete all of the steps in [Upgrade Airflow on Astro Private Cloud](https://www.astronomer.io/docs/software/manage-airflow-versions).

If you do not specify `--desired-airflow-version`, this command creates a list of available Airflow versions that you can select. The Astro CLI lists only the available Airflow versions that are later than the version currently specified in your `Dockerfile`.

## Options

| Option                      | Description                                                                                                                       | Possible values               |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- |
| `--cancel`                  | Cancel the upgrade                                                                                                                | None                          |
| `--deployment-id`           | The ID of the Deployment that you want to upgrade the Airflow version. Run `astro deployment list` to retrieve your Deployment ID | Any Deployment ID             |
| `--desired-airflow-version` | The Airflow version you're upgrading to. For example, `2.2.0`                                                                     | Any supported Airflow version |

## Examples

```sh wrap theme={null}
# Upgrade to Airflow 2.4
$ astro deployment airflow --deployment-id telescopic-sky-4599 --desired-airflow-version 2.2.0
```

## Related commands

* [`astro deployment runtime upgrade`](/docs/cli/v1.45/astro-deployment-runtime-upgrade)
* [`astro deployment runtime migrate`](/docs/cli/v1.45/astro-deployment-runtime-migrate)


# astro deployment airflow-variable copy
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-airflow-variable-copy

Copy an Airflow variable from a Deployment.

<Info>
  This command is only available on Astro.
</Info>

Copy Airflow variables from one Astro Deployment to another. Airflow variables are stored in the target Deployment's metadata database and appear in the Airflow UI.

## Usage

```sh wrap theme={null}
astro deployment airflow-variable copy
```

This command only copies Airflow variables that were configured through the Airflow UI or Airflow REST API

<Tip>
  This command is recommended for automated workflows. To run this command in an automated process such as a [CI/CD pipeline](/docs/astro/set-up-ci-cd), you can generate an API token, then specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

  ```bash wrap theme={null}
  export ASTRO_API_TOKEN=<your-token>
  ```

  See [Organization](/docs/astro/organization-api-tokens), [Workspace](/docs/astro/workspace-api-tokens), and [Deployment](/docs/astro/deployment-api-tokens) API token documentation for more details about ways to use API tokens.
</Tip>

## Options

| Option                | Description                                                                                                                                   | Possible Values           |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-s`,`--source-id`    | The ID of the Deployment to copy Airflow variables from.                                                                                      | Any valid Deployment ID   |
| `-n`, `--source-name` | The name of the Deployment from which to copy Airflow variables. Use as an alternative to `<source-id>`.                                      | Any valid Deployment name |
| `-t`, `--target-id`   | The ID of the Deployment to receive the copied Airflow variables                                                                              |                           |
| `--target-name`       | The name of the Deployment to receive the copied Airflow variables. Use as an alternative to `<target-id>`.                                   | Any valid Deployment name |
| `-w`,`--workspace-id` | Specify to copy Airflow variables to a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed. | Any valid Workspace ID    |

## Examples

```bash wrap theme={null}
# copy airflow variables stored in the Deployment with an ID of cl03oiq7d80402nwn7fsl3dmv to a deployment with an ID of cl03oiq7d80402nwn7fsl3dcd
astro deployment airflow-variable copy --source-id cl03oiq7d80402nwn7fsl3dmv --target-id cl03oiq7d80402nwn7fsl3dcd

# copy airflow variables stored in the Deployment "My Deployment" to another Deployment "My Other Deployment"
astro deployment airflow-variable copy --source-name="My Deployment" --target-name="My Other Deployment"
```

## Related commands

* [`astro deployment airflow variable create`](/docs/cli/v1.45/astro-deployment-airflow-variable-create)
* [`astro deployment airflow variable update`](/docs/cli/v1.45/astro-deployment-airflow-variable-update)


# astro deployment airflow-variable create
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-airflow-variable-create

Create an Airflow variable in a Deployment.

<Info>
  This command is only available on Astro.
</Info>

Create Airflow variables on a Deployment. Airflow variables are stored in the Deployment's metadata database and appear in the Airflow UI.

## Usage

```bash wrap theme={null}
astro deployment airflow-variable create
```

<Tip>
  This command is recommended for automated workflows. To run this command in an automated process such as a [CI/CD pipeline](/docs/astro/set-up-ci-cd), you can generate an API token, then specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

  ```bash wrap theme={null}
  export ASTRO_API_TOKEN=<your-token>
  ```

  See [Organization](/docs/astro/organization-api-tokens), [Workspace](/docs/astro/workspace-api-tokens), and [Deployment](/docs/astro/deployment-api-tokens) API token documentation for more details about ways to use API tokens.
</Tip>

## Options

| Option                 | Description                                                                                                                          | Possible Values           |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------- |
| `-d`,`--deployment-id` | The ID of the Deployment where you want to create Airflow variables.                                                                 | Any valid Deployment ID   |
| `--deployment-name`    | The name of the Deployment where you want to create Airflow variables. Use as an alternative to `<deployment-id>`.                   | Any valid Deployment name |
| `-w`,`--workspace-id`  | Create Airflow variables in a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed. | Any valid Workspace ID    |
| `-k`,`--key`           | The Airflow variable key. Required.                                                                                                  | string                    |
| `-v`,`--value`         | The Airflow variable value. Required.                                                                                                | string                    |
| `--description`        | The Airflow variable description.                                                                                                    | string                    |

## Examples

```bash wrap theme={null}
# create airflow variable called my-variable stored in the Deployment with an ID of cl03oiq7d80402nwn7fsl3dmv
astro deployment airflow-variable create --deployment-id cl03oiq7d80402nwn7fsl3dmv --key my-variable --value VAR

# create airflow-variables stored in the Deployment "My Deployment"
astro deployment airflow-variable create --deployment-name="My Deployment" --key my-variable --value VAR
```

## Related commands

* [`astro deployment airflow variable copy`](/docs/cli/v1.45/astro-deployment-airflow-variable-copy)
* [`astro deployment airflow variable update`](/docs/cli/v1.45/astro-deployment-airflow-variable-update)


# astro deployment airflow-variable list
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-airflow-variable-list

List variables in an Airflow Deployment.

<Info>
  This command is only available on Astro.
</Info>

List the Airflow variables stored in a Deployment's metadata database.

## Usage

```sh wrap theme={null}
astro deployment airflow-variable list
```

This command only lists Airflow variables that were configured through the Airflow UI or otherwise stored in the Airflow metadata database.

## Options

| Option                   | Description                                                                                                                         | Possible Values           |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-d`,`--deployment-id`   | The ID of the Deployment to list Airflow variables for.                                                                             | Any valid Deployment ID   |
| `-n`,`--deployment-name` | The name of the Deployment to list Airflow variables for. Use as an alternative to `<deployment-id>`.                               | Any valid Deployment name |
| `-w`,`--workspace-id`    | List Airflow variables for a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed. | Any valid Workspace ID    |

## Output

| Output        | Description                                   | Data Type |
| ------------- | --------------------------------------------- | --------- |
| `KEY`         | The `key` of the variable's `key:value` pair. | String    |
| `DESCRIPTION` | The optional description of the variable.     | String    |

## Examples

```bash wrap theme={null}
# List airflow variables stored in the Deployment with an ID of cl03oiq7d80402nwn7fsl3dmv
astro deployment airflow-variable list --deployment-id cl03oiq7d80402nwn7fsl3dmv

# List airflow variables stored in the Deployment "My Deployment"
astro deployment airflow-variable list --deployment-name="My Deployment"
```

## Related commands

* [`astro deployment airflow-variable create`](/docs/cli/v1.45/astro-deployment-airflow-variable-create)
* [`astro deployment airflow-variable update`](/docs/cli/v1.45/astro-deployment-airflow-variable-update)


# astro deployment airflow-variable update
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-airflow-variable-update

Update an existing Deployment Airflow variable.

<Info>
  This command is only available on Astro.
</Info>

Update the value for a Deployment's Airflow variable.

## Usage

```sh wrap theme={null}
astro deployment airflow-variable update
```

<Tip>
  This command is recommended for automated workflows. To run this command in an automated process such as a [CI/CD pipeline](/docs/astro/set-up-ci-cd), you can generate an API token, then specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

  ```bash wrap theme={null}
  export ASTRO_API_TOKEN=<your-token>
  ```

  See [Organization](/docs/astro/organization-api-tokens), [Workspace](/docs/astro/workspace-api-tokens), and [Deployment](/docs/astro/deployment-api-tokens) API token documentation for more details about ways to use API tokens.
</Tip>

## Options

| Option                 | Description                                                                                                                           | Possible Values           |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-d`,`--deployment-id` | The ID of the Deployment where you want to update an Airflow variable.                                                                | Any valid Deployment ID   |
| `--deployment-name`    | The name of the Deployment where you want to update Airflow variables. Use as an alternative to `<deployment-id>`.                    | Any valid Deployment name |
| `-w`,`--workspace-id`  | Update Airflow variables for a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed. | Any valid Workspace ID    |
| `-k`,`--key`           | The Airflow variable key. Required.                                                                                                   | string                    |
| `-v`,`--value`         | The Airflow variable value. Required.                                                                                                 | string                    |
| `--description`        | The Airflow variable description.                                                                                                     | string                    |

## Examples

```bash wrap theme={null}
# update airflow-variable called my-airflow-variable stored in the Deployment with an ID of cl03oiq7d80402nwn7fsl3dmv
astro deployment airflow-variable update --deployment-id cl03oiq7d80402nwn7fsl3dmv --key my-variable --value VAR

# update airflow-variables stored in the Deployment "My Deployment"
astro deployment airflow-variable update --deployment-name="My Deployment" --key my-variable --value VAR

## Related commands

- [`astro deployment airflow-variable create`](astro-deployment-airflow-variable-create)
- [`astro deployment airflow-variable list`](astro-deployment-airflow-variable-list)
```


# astro deployment connection copy
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-connection-copy

Copy an Airflow connection from a Deployment.

<Info>
  This command is only available on Astro.
</Info>

<Warning>This command copies **all** connections from one instance to another, but does not copy encrypted data, like passwords, or some `extras`, such as when they contain private keys.</Warning>

Copy Airflow connections from one Astro Deployment to another. Airflow connections are stored in the target Deployment's metadata database and appear in the Airflow UI.

## Usage

```sh wrap theme={null}
astro deployment connection copy
```

This command only copies Airflow connections that were configured through the Airflow UI or otherwise stored in the Airflow metadata database.

<Tip>
  This command is recommended for automated workflows. To run this command in an automated process such as a [CI/CD pipeline](/docs/astro/set-up-ci-cd), you can generate an API token, then specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

  ```bash wrap theme={null}
  export ASTRO_API_TOKEN=<your-token>
  ```

  See [Organization](/docs/astro/organization-api-tokens), [Workspace](/docs/astro/workspace-api-tokens), and [Deployment](/docs/astro/deployment-api-tokens) API token documentation for more details about ways to use API tokens.
</Tip>

## Options

| Option                | Description                                                                                                                                     | Possible Values           |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-s`,`--source-id`    | The ID of the Deployment to copy Airflow connections from.                                                                                      | Any valid Deployment ID   |
| `-n`, `--source-name` | The name of the Deployment from which to copy Airflow connections. Use as an alternative to `<source-id>`.                                      | Any valid Deployment name |
| `-t`, `--target-id`   | The ID of the Deployment to receive the copied Airflow connections                                                                              |                           |
| `--target-name`       | The name of the Deployment to receive the copied Airflow connections. Use as an alternative to `<target-id>`.                                   | Any valid Deployment name |
| `-w`,`--workspace-id` | Specify to copy Airflow connections to a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed. | Any valid Workspace ID    |

## Examples

```bash wrap theme={null}
# copy connections stored in the Deployment with an ID of cl03oiq7d80402nwn7fsl3dmv to a deployment with an ID of cl03oiq7d80402nwn7fsl3dcd
astro deployment connection copy --source-id cl03oiq7d80402nwn7fsl3dmv --target cl03oiq7d80402nwn7fsl3dcd

# copy connections stored in the Deployment "My Deployment" to another Deployment "My Other Deployment"
astro deployment connection copy --source-name="My Deployment" --target-name="My Other Deployment"
```

## Related commands

* [`astro deployment connection create`](/docs/cli/v1.45/astro-deployment-connection-create)
* [`astro deployment connection update`](/docs/cli/v1.45/astro-deployment-connection-update)


# astro deployment connection create
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-connection-create

Create an Airflow connection in a Deployment.

<Info>
  This command is only available on Astro.
</Info>

Create an Airflow connection in a Deployment's metadata database.

## Usage

```sh wrap theme={null}
astro deployment connection create
```

## Options

| Option                 | Description                                                                                                                      | Possible Values           |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-d`,`--deployment-id` | The ID of the Deployment where you want to create a connection.                                                                  | Any valid Deployment ID   |
| `--deployment-name`    | The name of the Deployment where you want to create a connection. Use as an alternative to `<deployment-id>`.                    | Any valid Deployment name |
| `-w`,`--workspace-id`  | Create a connection for a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed. | Any valid Workspace ID    |
| `-i`,`--conn-id`       | The connection ID. Required.                                                                                                     | string                    |
| `-t`,`--conn-type`     | The connection type. Required.                                                                                                   | string                    |
| `--description`        | The connection description.                                                                                                      | string                    |
| `--extra`              | The extra field configuration, defined as a stringified JSON object.                                                             | string                    |
| `--host`               | The connection host.                                                                                                             | string                    |
| `--login`              | The connection login or username.                                                                                                | string                    |
| `--password`           | The connection password.                                                                                                         | string                    |
| `--port`               | The connection port.                                                                                                             | string                    |
| `--schema`             | The connection schema.                                                                                                           | string                    |

## Examples

```bash wrap theme={null}
# create connection called my-connection stored in the Deployment with an ID of cl03oiq7d80402nwn7fsl3dmv
astro deployment connection create --deployment-id cl03oiq7d80402nwn7fsl3dmv --conn-id my-connection --conn-type http

# create connections stored in the Deployment "My Deployment"
astro deployment connection create --deployment-name="My Deployment" --conn-id my-connection --conn-type http
```

## Related commands

* [`astro deployment connection list`](/docs/cli/v1.45/astro-deployment-connection-list)
* [`astro deployment connection update`](/docs/cli/v1.45/astro-deployment-connection-update)


# astro deployment connection list
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-connection-list

List Airflow connections in a Deployment.

<Info>
  This command is only available on Astro.
</Info>

List the Airflow connections stored in a Deployment's metadata database.

## Usage

```sh wrap theme={null}
astro deployment connection list
```

This command only lists Airflow connections that were configured through the Airflow UI or otherwise stored in the Airflow metadata database.

## Options

| Option                 | Description                                                                                                                  | Possible Values           |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-d`,`--deployment-id` | The ID of the Deployment for which to list connections                                                                       | Any valid Deployment ID   |
| `--deployment-name`    | The name of the Deployment for which to list connections. Use as an alternative to `<deployment-id>`.                        | Any valid Deployment name |
| `-w`,`--workspace-id`  | List connections for a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed | Any valid Workspace ID    |

## Output

| Output          | Description                            | Data Type |
| --------------- | -------------------------------------- | --------- |
| `CONNECTION ID` | The ID you assigned to the connection. | String    |
| `CONN TYPE`     | The Airflow connection type.           | String    |

## Examples

```bash wrap theme={null}
# List connections stored in the Deployment with an ID of cl03oiq7d80402nwn7fsl3dmv
astro deployment connection list --deployment-id cl03oiq7d80402nwn7fsl3dmv

# List connections stored in the Deployment "My Deployment"
astro deployment connection list --deployment-name="My Deployment"
```

## Related commands

* [`astro deployment connection create`](/docs/cli/v1.45/astro-deployment-connection-create)
* [`astro deployment connection update`](/docs/cli/v1.45/astro-deployment-connection-update)


# astro deployment connection update
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-connection-update

Update an Airflow connection in a Deployment.

<Info>
  This command is only available on Astro.
</Info>

Update the value for a Deployment's Airflow variable.

## Usage

```sh wrap theme={null}
astro deployment connection update
```

<Tip>
  This command is recommended for automated workflows. To run this command in an automated process such as a [CI/CD pipeline](/docs/astro/set-up-ci-cd), you can generate an API token, then specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

  ```bash wrap theme={null}
  export ASTRO_API_TOKEN=<your-token>
  ```

  See [Organization](/docs/astro/organization-api-tokens), [Workspace](/docs/astro/workspace-api-tokens), and [Deployment](/docs/astro/deployment-api-tokens) API token documentation for more details about ways to use API tokens.
</Tip>

## Options

| Option                 | Description                                                                                                                      | Possible Values           |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-d`,`--deployment-id` | The ID of the Deployment where you want to create a connection.                                                                  | Any valid Deployment ID   |
| `--deployment-name`    | The name of the Deployment where you want to create a connection. Use as an alternative to `<deployment-id>`.                    | Any valid Deployment name |
| `-w`,`--workspace-id`  | Create a connection for a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed. | Any valid Workspace ID    |
| `-i`,`--conn-id`       | The connection ID. Required.                                                                                                     | string                    |
| `-t`,`--conn-type`     | The connection type. Required.                                                                                                   | string                    |
| `--description`        | The connection description.                                                                                                      | string                    |
| `--extra`              | The extra field configuration, defined as a stringified JSON object.                                                             | string                    |
| `--host`               | The connection host.                                                                                                             | string                    |
| `--login`              | The connection login or username.                                                                                                | string                    |
| `--password`           | The connection password.                                                                                                         | string                    |
| `--port`               | The connection port.                                                                                                             | string                    |
| `--schema`             | The connection schema.                                                                                                           | string                    |

## Examples

```bash wrap theme={null}
# update connection called my-connection stored in the Deployment with an ID of cl03oiq7d80402nwn7fsl3dmv
astro deployment connection update --deployment-id cl03oiq7d80402nwn7fsl3dmv --conn-id my-connection --conn-type http

# update connections stored in the Deployment "My Deployment"
astro deployment connection update --deployment-name="My Deployment" --conn-id my-connection --conn-type http
```

## Related commands

* [`astro deployment connection create`](/docs/cli/v1.45/astro-deployment-connection-create)
* [`astro deployment connection list`](/docs/cli/v1.45/astro-deployment-connection-list)


# astro deployment create
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-create

Create a Deployment.

<Info>
  The behavior and format of this command differs depending on what Astronomer
  product you're using. Use the following tabs to change product contexts.
</Info>

<Tabs>
  <Tab title="Astro">
    Create a Deployment on Astro. This command is functionally identical to using the Astro UI to [create a Deployment](/docs/astro/create-deployment).

    ## Usage

    ```sh wrap theme={null}
    astro deployment create
    ```

    When you use `astro deployment create`, it creates a Deployment with a default Worker Queue that uses default worker types.

    Some Deployment configurations, including worker queue and worker type, can be set only by using the `--deployment-file` flag to apply a Deployment file. See [Manage Deployments as code](/docs/astro/manage-deployments-as-code).

    ## Options

    | Option                        | Description                                                                                                                                                                                                                                                     | Possible Values                                                                                                                                                                                                                                                                                    |
    | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | `--allowed-ip-address-ranges` | If Remote Execution is enabled, limit the Deployment's incoming traffic to the Remote Agents in your environment.                                                                                                                                               | A comma-separated list of CIDR blocks, such as `192.168.0.0/24,10.0.0.0/16`.                                                                                                                                                                                                                       |
    | `-a`, `--high-availability`   | Enables high availability for the Deployment. The default is `disable`.                                                                                                                                                                                         | Either `enable` or `disable`.                                                                                                                                                                                                                                                                      |
    | `-c`, `--cluster-id`          | (Dedicated clusters only) The cluster in which you want to create a Deployment.                                                                                                                                                                                 | A valid cluster ID.                                                                                                                                                                                                                                                                                |
    | `--cicd-enforcement`          | Specify that the Deployment can only accept code deploys from API tokens. Note that in CLI versions before 1.23, this flag was `--enforce-cicd`.                                                                                                                | None                                                                                                                                                                                                                                                                                               |
    | `-d`,`--description`          | The description for the Deployment.                                                                                                                                                                                                                             | Any string. Multiple-word descriptions should be specified in quotations (`"`)                                                                                                                                                                                                                     |
    | `--dag-deploy`                | Enables DAG-only deploys for the Deployment. The default is `disable`.                                                                                                                                                                                          | Either `enable` or `disable`.                                                                                                                                                                                                                                                                      |
    | `--default-task-pod-cpu`      | The default task Pod CPUs to use for the Deployment. See [Customize a task's Kubernetes Pod](/docs/astro/kubernetes-executor#customize-a-task’s-kubernetes-pod) for more information.                                                                                | A numeric value such as `0.25`. This number cannot exceed the values you configured in your [Default pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources) in the UI or in your Deployment configuration file.                                                            |
    | `--default-task-pod-memory`   | The default task pod memory to use for the Deployment in Gi. See [Customize a task's Kubernetes Pod](/docs/astro/kubernetes-executor#customize-a-task’s-kubernetes-pod) for more information.                                                                        | A numeric value followed by `Gi`, such as `0.5Gi`. This number cannot exceed the values you configured in your [Default pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources) in the UI or in your Deployment configuration file.                                         |
    | `--deployment-file`           | Location of the template file that specifies the configuration of the new Deployment. File can be in either JSON or YAML format. See [Create a Deployment with a Deployment File](/docs/astro/manage-deployments-as-code#create-a-deployment-using-a-template-file). | A valid file path for YAML or JSON template file                                                                                                                                                                                                                                                   |
    | `-f`,`--force`                | Create the Deployment without prompting for confirmation.                                                                                                                                                                                                       | None                                                                                                                                                                                                                                                                                               |
    | `-e`,`--executor`             | The executor to use for the Deployment.                                                                                                                                                                                                                         | AstroExecutor, CeleryExecutor, or KubernetesExecutor                                                                                                                                                                                                                                               |
    | `-m`,`--development-mode`     | Set to 'enable' to enable development-only features such as hibernation. When enabled, the Deployment does not have guaranteed uptime SLAs.                                                                                                                     | `enable` or `disable`                                                                                                                                                                                                                                                                              |
    | `-n`,`--name`                 | The name of the Deployment.                                                                                                                                                                                                                                     | Any string. Multiple-word descriptions should be specified in quotations                                                                                                                                                                                                                           |
    | `-p`, `--cloud-provider`      | The Cloud Provider to use for your Deployment. The default is `azure`.                                                                                                                                                                                          | Possible values are `azure`, `aws`, and `gcp` for both standard and dedicated clusters.                                                                                                                                                                                                            |
    | `--region`                    | The region where you want to host the Deployment.                                                                                                                                                                                                               | The code for any [supported region](/docs/astro/resource-reference-hosted#standard-cluster-regions)                                                                                                                                                                                                     |
    | `--remote-execution-enabled`  | Enables [Remote Execution](/docs/astro/execution-mode#remote-execution) for the Deployment.                                                                                                                                                                          | None                                                                                                                                                                                                                                                                                               |
    | `--resource-quota-cpu`        | The Deployment's CPU resource quota for Kubernetes Pods. See [Customize a task's Kubernetes Pod](/docs/astro/kubernetes-executor#customize-a-task’s-kubernetes-pod) for more information.                                                                            | A numeric value such as `10`. This number cannot exceed the values you configured in your [Default pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources) in the UI or in your [Deployment file](/docs/astro/deployment-file-reference) configuration file.                     |
    | `--resource-quota-memory`     | The Deployment's memory resource quota for Kubernetes Pods in Gi. See [Customize a task's Kubernetes Pod](/docs/astro/kubernetes-executor#customize-a-task’s-kubernetes-pod) for more information.                                                                   | A numeric value followed by `Gi`, such as `20Gi`. This number cannot exceed the values you configured in your [Default pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources) in the UI or in your [Deployment file](/docs/astro/deployment-file-reference) configuration file. |
    | `-s`,`--scheduler-size`       | The size of scheduler for the Deployment. The default is `small`.                                                                                                                                                                                               | Either `small`, `medium`, `large`, or `extra_large`. `extra_large` schedulers require a minimum Astro Runtime version of 9.7.0.                                                                                                                                                                    |
    | `--task-log-bucket`           | If Remote Execution is enabled, specify the cloud storage bucket for task log storage. You can [configure your Deployment to show the stored logs directly in the Airflow UI](/docs/astro/remote-task-logs-af-ui).                                                   | A valid cloud storage bucket name, such as `gs://my-task-logs` or `s3://my-task-logs`.                                                                                                                                                                                                             |
    | `--task-log-url-pattern`      | If Remote Execution is enabled, specify the URL template to link to task logs stored in an external logging provider from the Airflow UI.                                                                                                                       | A string URL template. See [Configure external logging provider for Remote Execution Deployments](/docs/astro/remote-task-logs-external#configure-external-logging-provider-for-remote-execution-deployments).                                                                                          |
    | `--type`                      | The type of cluster you want to run the Deployment on. The default is `standard`.                                                                                                                                                                               | Either `dedicated` or `standard`.                                                                                                                                                                                                                                                                  |
    | `-v`,`--runtime-version`      | The Astro Runtime version for the Deployment                                                                                                                                                                                                                    | Any supported version of Astro Runtime. Major, minor, and patch versions must be specified.                                                                                                                                                                                                        |
    | `--wait`                      | Wait for the new Deployment to have a [healthy](/docs/astro/deployment-health-incidents) status before completing the command.                                                                                                                                       | None                                                                                                                                                                                                                                                                                               |
    | `--wait-time`                 | Time to wait for the Deployment to become healthy before ending the command. Can only be used with `--wait=true`.                                                                                                                                               | A time duration amount, such as `10s` or `1m11s`.                                                                                                                                                                                                                                                  |
    | `--workload-identity`         | The workload identity to use for the Deployment. You must use this flag with `--cloud-provider`.                                                                                                                                                                | A valid AWS `arn` or a GCP service account.                                                                                                                                                                                                                                                        |
    | `--workspace-id`              | The Workspace in which to create a Deployment. If not specified, your current Workspace is assumed.                                                                                                                                                             | Any valid Workspace ID                                                                                                                                                                                                                                                                             |

    ## Examples

    ```bash wrap theme={null}
    # CLI prompts you for a Deployment name and cluster
    astro deployment create

    # Create a Deployment with all required information specified. The CLI will not prompt you for more information
    astro deployment create -d="My Deployment Description" --name="My Deployment Name" --cluster-id="ckwqkz36200140ror6axh8p19"

    # Specify the new Deployment's configuration with a yaml file
    astro deployment create --deployment-file deployment.yaml

    # Create a deployment on Astro using a standard cluster
    astro deployment create --name="my-gcp-deployment" --region="us-central1"

    # Create a deployment on Astro using a dedicated cluster
    astro deployment create --name="my-gcp-deployment" --cluster-type="dedicated" --cluster-id="clj123n1311p901muj9hwpgjb"
    ```
  </Tab>

  <Tab title="APC">
    Create a Deployment on Astro Private Cloud. This command is functionally identical to using the UI to create a Deployment.

    ## Usage

    ```sh wrap theme={null}
    astro deployment create
    ```

    ## Options

    | Option                        | Description                                                                                                                                                        | Possible Values                                                                                                                     |
    | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- |
    | `-a`,`--airflow-version`      | The Astronomer Certified version to use for the Deployment                                                                                                         | Any supported version of Astronomer Certified                                                                                       |
    | `-c`, `--cloud-role`          | An AWS or GCP IAM role to append to your Deployment's webserver, scheduler, and worker Pods                                                                        | Any string                                                                                                                          |
    | `-t`, `--dag-deployment-type` | The dag deploy method for the Deployment                                                                                                                           | Can be either `image`, `git_sync`, or `volume`. The default is `image`                                                              |
    | `-d`,`--description`          | The description for the Deployment                                                                                                                                 | Any string. Multiple-word descriptions should be specified in quotations (`"`)                                                      |
    | `-e`, `--executor`            | The executor type for the Deployment                                                                                                                               | `local`, `celery`, or `kubernetes`. The default value is `celery`                                                                   |
    | `-b`, `--git-branch-name`     | The branch name of the git repo to sync your Deployment from. Must be specified with `--dag-deployment-type=git_sync`                                              | Any valid git branch name                                                                                                           |
    | `-u`, `--git-repository-url`  | The URL for the git repository to sync your Deployment from. Must be specified with `--dag-deployment-type=git_sync`                                               | Any valid git repository URL                                                                                                        |
    | `-v`, `--git-revision`        | The commit reference of the branch that you want to sync with your Deployment. Must be specified with `--dag-deployment-type=git_sync`                             | Any valid git revision                                                                                                              |
    | `--known-hosts`               | The public key for your Git provider, which can be retrieved using `ssh-keyscan -t rsa <provider-domain>`. Must be specified with `--dag-deployment-type=git_sync` | Any valid public key                                                                                                                |
    | `-l`,`--label`                | The label for your Deployment                                                                                                                                      | Any string                                                                                                                          |
    | `--mode`                      | Creates an Operator-mode Deployment. Requires Astro Private Cloud 2.1.0 or later.                                                                                  | `helm` or `operator`. The default value is `helm`                                                                                   |
    | `-n`,`--nfs-location`         | The location for an NFS volume mount. Must be specified with `--dag-deployment-type=volume`.                                                                       | An NFS volume mount specified as: `<IP>:/<path>`. Input is automatically prepended with `nfs:/` - do not include this in your input |
    | `-r`,`--release-name`         | A custom release name for the Deployment. See [Customize release names](https://www.astronomer.io/docs/software/configure-deployment#customize-release-names)      | Any string of alphanumeric and hyphen characters                                                                                    |
    | `--runtime-version`           | The Astro Runtime version for the Deployment                                                                                                                       | Any supported version of Astro Runtime. Major, minor, and patch versions must be specified.                                         |
    | `--ssh-key`                   | The SSH private key for your Git repository. Must be specified with `--dag-deployment-type=git_sync`                                                               | Any valid SSH key                                                                                                                   |
    | `-s`,`--sync-interval`        | The time interval between checks for updates in your Git repository, in seconds. Must be specified with `--dag-deployment-type=git_sync`                           | Any integer                                                                                                                         |
    | `-t`,`--triggerer-replicas`   | Number of replicas to use for the Airflow triggerer                                                                                                                | Any integer between 0 - 2. The default value is 1.                                                                                  |
    | `--workspace-id`              | The Workspace in which to create a Deployment. If not specified, your current Workspace is assumed                                                                 | Any valid Workspace ID                                                                                                              |

    ## Examples

    ```sh wrap theme={null}
    $ astro deployment create
    # CLI prompts you for a Deployment name

    $ astro deployment create -l="My Deployment label" --workspace-id="ckwqkz36200140ror6axh8p19"
    # Create a Deployment in a separate Workspace. The CLI will not prompt you for more information
    ```
  </Tab>
</Tabs>

## Related commands

* [`astro deployment delete`](/docs/cli/v1.45/astro-deployment-delete)
* [`astro deployment list`](/docs/cli/v1.45/astro-deployment-list)


# astro deployment delete
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-delete

Delete a Deployment.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change product contexts.</Info>

<Tabs>
  <Tab title="Astro">
    Delete a Deployment on Astro. This command is functionally identical to deleting a Deployment in the Astro UI.

    When you run `astro deployment delete`, you are prompted to select from a list of Deployments that you can access in your Workspace. You can bypass this prompt and specify a Deployment name or ID in the command. To retrieve a Deployment ID, open your Deployment in the Astro UI and copy the value in the **ID** section of the Deployment page. You can also run `astro deployment list` to find a Deployment ID or name.

    <Info>To complete this action, [Workspace Owner](/docs/astro/user-permissions#workspace-roles) permissions are required.</Info>

    ## Usage

    ```sh wrap theme={null}
    astro deployment delete
    ```

    ## Options

    | Option              | Description                                                                       | Possible Values           |
    | ------------------- | --------------------------------------------------------------------------------- | ------------------------- |
    | `<deployment-id>`   | The ID of the Deployment to delete                                                | Any valid Deployment ID   |
    | `-f`,`--force`      | Do not include a confirmation prompt before deleting the Deployment               | None                      |
    | `--workspace-id`    | Specify a Workspace to delete a Deployment outside of your current Workspace      | Any valid Workspace ID    |
    | `--deployment-name` | The name of the Deployment to delete. Use as an alternative to `<deployment-id>`. | Any valid Deployment name |

    ## Examples

    ```sh wrap theme={null}
    $ astro deployment delete
    # CLI prompts you for a Deployment to delete
    $ astro deployment delete ckvvfp9tf509941drl4vela81n -f
    # Force delete a Deployment without a confirmation prompt
    $ astro deployment delete --deployment-name="My deployment"
    # Delete a Deployment by specifying its name.
    ```
  </Tab>

  <Tab title="APC">
    Delete a Deployment on Astro Private Cloud. This command is functionally identical to deleting a Deployment with the UI.

    <Note>Astro Private Cloud now hard-deletes Deployments by default. The `-h`, `--hard` flag is deprecated because it has no effect, and Astronomer plans to remove it in a future release.</Note>

    ## Usage

    ```sh wrap theme={null}
    astro deployment delete
    ```

    ## Options

    | Option                         | Description                                                                                                                                                                                                                                           | Possible Values         |
    | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
    | `<deployment-id>` (*Required*) | The ID of the Deployment to delete                                                                                                                                                                                                                    | Any valid Deployment ID |
    | `-h`,`--hard` (*Deprecated*)   | Previously deleted all infrastructure and records for this Deployment. No longer needed because hard delete is now the default. See [Hard delete a Deployment](https://www.astronomer.io/docs/software/configure-deployment#hard-delete-a-deployment) | None                    |
  </Tab>
</Tabs>

## Related commands

* [`astro deployment create`](/docs/cli/v1.45/astro-deployment-create)
* [`astro deployment list`](/docs/cli/v1.45/astro-deployment-list)


# astro deployment hibernate
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-hibernate

Hibernate a Deployment.

[Hibernate an Astro development Deployment](/docs/astro/deployment-resources#hibernate-a-development-deployment) for a set amount of time. Overrides any existing hibernation schedule and sets the Deployment to hibernate for a specific duration or until a specific date.

## Usage

```sh wrap theme={null}
astro deployment hibernate [one of --until/ --for/ --remove-override]
```

## Options

| Option                   | Description                                                           | Possible Values                                      |
| ------------------------ | --------------------------------------------------------------------- | ---------------------------------------------------- |
| `-n`,`--deployment-name` | The name of the development Deployment to hibernate.                  | Any valid Deployment name                            |
| `-u, --until`            | Specify the hibernation period using an end date and time.            | Any future date in the format `YYYY-MM-DDT00:00:00Z` |
| `-d, --for`              | Specify the hibernation period using a duration.                      | Any amount of time in the format `XhYm`              |
| `-r, --remove-override`  | Remove any existing override and resume regular hibernation schedule. | None                                                 |
| `-f, --force`            | The CLI will not prompt to confirm before hibernating the Deployment. | None                                                 |

## Related commands

* [`astro deployment wake-up`](/docs/cli/v1.45/astro-deployment-wake-up)


# astro deployment inspect
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-inspect

Inspect a Deployment.

<Info>
  This command is only available on Astro.
</Info>

Inspect an Astro Deployment. This command returns a YAML or JSON representation of a Deployment's current configuration and state as shown in the Astro UI. When the `--key` flag is used, it returns only the values specified with the flag.

For more information about how to use Deployment files, see [Manage Deployments as Code](/docs/astro/manage-deployments-as-code).

## Usage

```sh wrap theme={null}
astro deployment inspect
```

When using the `--key` flag, specify the complete path of the key you want to return the value for, excluding `deployment`. For example, to return the `cluster_id` for a specific Deployment, you would run:

```sh wrap theme={null}
astro deployment inspect -n <deployment-name> --key metadata.cluster_id
```

See [Template file contents](/docs/astro/deployment-file-reference) for all possible values to return.

## Options

| Option                     | Description                                                                                                                                                                                                                                                                                                                                             | Possible Values                        |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- |
| `<deployment-id>`          | The ID of the Deployment to inspect.                                                                                                                                                                                                                                                                                                                    | Any valid Deployment ID                |
| `-n`, `--deployment-name`  | Name of the Deployment to inspect. Use as an alternative to `<deployment-id>`.                                                                                                                                                                                                                                                                          | Any valid Deployment name              |
| `-k`, `--key`              | Return only a specific configuration key for a Deployment. For example `--key configuration.cluster_id` to get a Deployment's cluster ID.                                                                                                                                                                                                               | Any valid Deployment configuration key |
| `-o`, `--output`           | Output format can be one of: YAML or JSON. By default, inspecting a Deployment returns a file in YAML format.                                                                                                                                                                                                                                           | `yaml` or `json`                       |
| `--show-workload-identity` | Return the workload identity values that are set for the Deployment.                                                                                                                                                                                                                                                                                    | None                                   |
| `-t`, `--template`         | Generate a Deployment template file for the inspected Deployment. A template file is a configuration file that includes all information about a Deployment at the time of inspection, except for its name, description field, and unique metadata. See [Manage with Deployment files](/docs/astro/manage-deployments-as-code) to learn more about templates. | None                                   |
| `--workspace-id`           | Specify a Workspace to run this command for a Deployment that is outside of your current Workspace.                                                                                                                                                                                                                                                     | Any valid Workspace ID                 |

## Examples

```sh wrap theme={null}
# Shows a list of Deployments to inspect and prompts you to choose one
$ astro deployment inspect

# Shows a specific Deployment's configuration
$ astro deployment inspect <deployment-id>

# Shows a specific Deployment's health status
$ astro deployment inspect <deployment-id> --key metadata.status

# Save the current state of a Deployment to a YAML Deployment file
$ astro deployment inspect <deployment-id> > deployment.yaml

# Save a Deployment as a JSON template file
$ astro deployment inspect <deployment-id> --template -o json > deployment.json
```

## Related commands

* [`astro deployment list`](/docs/cli/v1.45/astro-deployment-list)
* [`astro deployment create`](/docs/cli/v1.45/astro-deployment-create)


# astro deployment list
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-list

List all Deployments in a Workspace.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

List all Deployments within your current Workspace.

## Usage

```sh wrap theme={null}
astro deployment list
```

## Options

| Option           | Description                                                                                          | Possible Values                |
| ---------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------ |
| `-a`,`--all`     | Show Deployments across all Workspaces that you have access to.                                      | None                           |
| `--json`         | Output the response as JSON. Shorthand for `--output json`. Mutually exclusive with `-o`/`--output`. | None                           |
| `-o`,`--output`  | Format the output. The default is `table`.                                                           | `table`, `json`, or `template` |
| `--template`     | Format the output using a Go template. Use with `--output template`.                                 | Any valid Go template string   |
| `--workspace-id` | Specify a Workspace to list Deployments outside of your current Workspace                            | Any valid Workspace ID         |

## Examples

```sh wrap theme={null}
$ astro deployment list --all
# Shows Deployments from all Workspaces that you're authenticated to
```

## Output

| Output               | Description                                                                   | Data Type                                  |
| -------------------- | ----------------------------------------------------------------------------- | ------------------------------------------ |
| `NAME`               | The name of the Deployment.                                                   | String                                     |
| `NAMESPACE`          | The Deployment's Kubernetes namespace.                                        | String                                     |
| `CLUSTER`            | The name of the Astro cluster where the Deployment runs.                      | String                                     |
| `DEPLOYMENT ID`      | The Deployment ID                                                             | String                                     |
| `RUNTIME VERSION`    | The Deployment's Astro Runtime version and its corresponding Airflow version. | String. (`X.X.X (based on Airflow X.X.X)`) |
| `DAG DEPLOY ENABLED` | Whether the Deployment supports dag deploys.                                  | Boolean                                    |

## Related commands

* [`astro login`](/docs/cli/v1.45/astro-login)
* [`astro deploy`](/docs/cli/v1.45/astro-deploy)


# astro deployment logs
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-logs

Show Airflow component logs for a Deployment.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change product contexts.</Info>

<Tabs>
  <Tab title="Astro">
    Show [Airflow component logs](/docs/astro/view-logs#view-airflow-component-logs-in-the-astro-ui) over the last 24 hours for a given Deployment on Astro. These are the same logs that appear in the **Logs** tab of the Astro UI.

    ## Usage

    ```sh wrap theme={null}
    astro deployment logs
    ```

    <Info>When you filter logs using the command flags `--error`, `--warn`, `--info`, and `--key-word`, you can specify only one filter flag per command.</Info>

    ## Options

    | Option                    | Description                                                                                                                                                                  | Possible Values                                                                                                           |
    | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
    | `<deployment-id>`         | The ID of the Deployment to show logs for                                                                                                                                    | Any valid Deployment ID                                                                                                   |
    | `-n`, `--deployment-name` | The name of the Deployment to show logs for. Use as an alternative to `<deployment-id>`.                                                                                     | Any valid Deployment name                                                                                                 |
    | `-e`,`--error`            | Show only logs with a log level of `ERROR`                                                                                                                                   | None                                                                                                                      |
    | `-i`,`--info`             | Show only logs with a log level of `INFO`                                                                                                                                    | None                                                                                                                      |
    | `--keyword`               | Search the Deployment logs for a specific keyword or phrase.                                                                                                                 | None                                                                                                                      |
    | `-c`,`--log-count`        | The number of log lines to show. The default is `500`. The Astro CLI paginates the underlying API to return the full requested number of lines.                              | Any integer. If you request a number of log lines that exceeds the total number of logs, then it shows all existing logs. |
    | `--component`             | Show logs from one or more components by name, passed as a repeatable flag or a comma-separated list. Use as an alternative to flags such as `--scheduler` or `--triggerer`. | Any valid component name, for example `scheduler` or `dag-processor`                                                      |
    | `--dag-processor`         | Show logs from the Dag processor                                                                                                                                             | None                                                                                                                      |
    | `--scheduler`             | Show logs from the scheduler                                                                                                                                                 | None                                                                                                                      |
    | `--triggerer`             | Show logs from the triggerer                                                                                                                                                 | None                                                                                                                      |
    | `-w`,`--warn`             | Show only logs with a log level of `WARNING`                                                                                                                                 | None                                                                                                                      |
    | `--webserver`             | Show logs from the webserver                                                                                                                                                 | None                                                                                                                      |
    | `--workers`               | Show logs from the workers                                                                                                                                                   | None                                                                                                                      |
    | `--workspace-id`          | Specify a Workspace to show logs for a Deployment outside of your current Workspace                                                                                          | Any valid Workspace ID                                                                                                    |

    ## Examples

    ```sh wrap theme={null}
    $ astro deployment logs
    # CLI prompts you for a Deployment to view logs for

    $ astro deployment logs cl03oiq7d80402nwn7fsl3dmv
    # View logs for a specific Deployment

    $ astro deployment logs --deployment-name="My Deployment" --error --log-count=25
    # Show only the last 25 error-level logs

    $ astro deployment logs cl03oiq7d80402nwn7fsl3dmv --dag-processor
    # Show Dag processor logs for a specific Deployment

    $ astro deployment logs cl03oiq7d80402nwn7fsl3dmv --component scheduler,dag-processor
    # Show logs from multiple components in a single command
    ```
  </Tab>

  <Tab title="APC">
    Show Airflow component logs over the last 24 hours for a given Deployment. These logs are the same logs that appear in the **Logs** tab of the UI.

    ## Usage

    Run one of the following commands depending on which logs you want to stream:

    ```sh wrap theme={null}
    astro deployment logs <deployment-id> scheduler
    astro deployment logs <deployment-id> webserver
    astro deployment logs <deployment-id> workers
    astro deployment logs <deployment-id> triggerer
    ```

    ## Options

    | Option                             | Description                                                        | Possible values                                         |
    | ---------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------- |
    | `<deployment-id>` (\_Required)     | The ID of the Deployment for which you want to view logs           | Any valid Deployment ID                                 |
    | `<airflow-component>` (\_Required) | The Airflow component for which you want to view logs              | `scheduler`, `webserver`, `workers`, or `triggerer`     |
    | `--follow`                         | Subscribes to watch more logs                                      | None                                                    |
    | `--search`                         | Searches for the specified string within the logs you're following | Any string                                              |
    | `--since`                          | Limits past logs to those generated in the lookback window         | Lookback time in `h` or `m` (for example, `5m` or `2h`) |

    ## Examples

    ```sh wrap theme={null}
    # Return logs for last five minutes of webserver logs.
    $ astro deployment logs webserver example-deployment-uuid

    # Return logs from airflow workers for the last 5 minutes with a given search term, and subscribe to view more as they are generated.
    $ astro deployment logs workers example-deployment-uuid --follow --search "some search terms"

    # Return logs from airflow webserver for last 25 minutes.
    $ astro deployment logs webserver example-deployment-uuid --since 25m
    ```
  </Tab>
</Tabs>

## Related commands

* [`astro dev logs`](/docs/cli/v1.45/astro-dev-logs)
* [`astro dev run`](/docs/cli/v1.45/astro-dev-run)
* [`astro dev ps`](/docs/cli/v1.45/astro-dev-ps)


# astro deployment pool copy
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-pool-copy

Copy an Airflow pool from a Deployment.

<Info>
  This command is only available on Astro.
</Info>

Copy Airflow pools from one Astro Deployment to another. Airflow pools are stored in the target Deployment's metadata database and appear in the Airflow UI.

## Usage

```sh wrap theme={null}
astro deployment pool copy
```

This command only copies Airflow pools that were configured through the Airflow UI or otherwise stored in the Airflow metadata database.

<Tip>
  This command is recommended for automated workflows. To run this command in an automated process such as a [CI/CD pipeline](/docs/astro/set-up-ci-cd), you can generate an API token, then specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

  ```bash wrap theme={null}
  export ASTRO_API_TOKEN=<your-token>
  ```

  See [Organization](/docs/astro/organization-api-tokens), [Workspace](/docs/astro/workspace-api-tokens), and [Deployment](/docs/astro/deployment-api-tokens) API token documentation for more details about ways to use API tokens.
</Tip>

## Options

| Option                | Description                                                                                                                               | Possible Values           |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-s`,`--source-id`    | The ID of the Deployment to copy Airflow pools from.                                                                                      | Any valid Deployment ID   |
| `-n`, `--source-name` | The name of the Deployment from which to copy Airflow pools. Use as an alternative to `<source-id>`.                                      | Any valid Deployment name |
| `-t`, `--target-id`   | The ID of the Deployment to receive the copied Airflow pools                                                                              |                           |
| `--target-name`       | The name of the Deployment to receive the copied Airflow pools. Use as an alternative to `<target-id>`.                                   | Any valid Deployment name |
| `-w`,`--workspace-id` | Specify to copy Airflow pools to a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed. | Any valid Workspace ID    |

## Examples

```bash wrap theme={null}
# copy pools stored in the Deployment with an ID of cl03oiq7d80402nwn7fsl3dmv to a deployment with an ID of cl03oiq7d80402nwn7fsl3dcd
astro deployment pool copy --source-id cl03oiq7d80402nwn7fsl3dmv --target cl03oiq7d80402nwn7fsl3dcd

# copy pools stored in the Deployment "My Deployment" to another Deployment "My Other Deployment"
astro deployment pool copy --source-name="My Deployment" --target-name="My Other Deployment"
```

## Related commands

* [`astro deployment pool create`](/docs/cli/v1.45/astro-deployment-pool-create)
* [`astro deployment pool update`](/docs/cli/v1.45/astro-deployment-pool-update)


# astro deployment pool create
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-pool-create

Create Airflow pools in a Deployment.

<Info>
  This command is only available on Astro.
</Info>

Create Airflow pools in a Deployment. Airflow pools are stored in the Deployment's metadata database and appear in the Airflow UI.

## Usage

```sh wrap theme={null}
astro deployment pool create
```

<Tip>
  This command is recommended for automated workflows. To run this command in an automated process such as a [CI/CD pipeline](/docs/astro/set-up-ci-cd), you can generate an API token, then specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

  ```bash wrap theme={null}
  export ASTRO_API_TOKEN=<your-token>
  ```

  See [Organization](/docs/astro/organization-api-tokens), [Workspace](/docs/astro/workspace-api-tokens), and [Deployment](/docs/astro/deployment-api-tokens) API token documentation for more details about ways to use API tokens.
</Tip>

## Options

| Option                 | Description                                                                                                                      | Possible Values           |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-d`,`--deployment-id` | The ID of the Deployment where you want to create Airflow pools.                                                                 | Any valid Deployment ID   |
| `--deployment-name`    | The name of the Deployment where you want to create Airflow pools. Use as an alternative to `<deployment-id>`.                   | Any valid Deployment name |
| `-w`,`--workspace-id`  | Create Airflow pools in a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed. | Any valid Workspace ID    |
| `--name`               | The Airflow pool name. Required.                                                                                                 | Any string                |
| `-v`,`--slots`         | Number of airflow pool slots. Required.                                                                                          | Any integer               |
| `--description`        | The pool description.                                                                                                            | Any string                |

## Examples

```bash wrap theme={null}
# create pool called my-pool stored in the Deployment with an ID of cl03oiq7d80402nwn7fsl3dmv
astro deployment pool create --deployment-id cl03oiq7d80402nwn7fsl3dmv --name my-pool --slots 10

# create pool stored in the Deployment "My Deployment"
astro deployment pool create --deployment-name="My Deployment" --name my-pool --slots 10
```

## Related commands

* [`astro deployment pool copy`](/docs/cli/v1.45/astro-deployment-pool-copy)
* [`astro deployment pool update`](/docs/cli/v1.45/astro-deployment-pool-update)


# astro deployment pool list
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-pool-list

List Airflow pools in a Deployment.

<Info>
  This command is only available on Astro.
</Info>

List the Airflow pools stored in a Deployment's metadata database.

## Usage

```sh wrap theme={null}
astro deployment pool list
```

This command only lists Airflow pools that were configured through the Airflow UI or otherwise stored in the Airflow metadata database.

## Options

| Option                 | Description                                                                                                                     | Possible Values           |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-d`,`--deployment-id` | The ID of the Deployment to list Airflow pools for.                                                                             | Any valid Deployment ID   |
| `--deployment-name`    | The name of the Deployment to list Airflow pools for. Use as an alternative to `<deployment-id>`.                               | Any valid Deployment name |
| `-w`,`--workspace-id`  | List Airflow pools for a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed. | Any valid Workspace ID    |

## Output

| Output  | Description                                  | Data Type |
| ------- | -------------------------------------------- | --------- |
| `NAME`  | The name of the Airflow pool                 | String    |
| `SLOTS` | The total number of worker slots in the pool | String    |

## Examples

```bash wrap theme={null}
# List pools stored in the Deployment with an ID of cl03oiq7d80402nwn7fsl3dmv
astro deployment pool list --deployment-id cl03oiq7d80402nwn7fsl3dmv

# List pools stored in the Deployment "My Deployment"
astro deployment pool list --deployment-name="My Deployment"
```

## Related commands

* [`astro deployment pool create`](/docs/cli/v1.45/astro-deployment-pool-create)
* [`astro deployment pool update`](/docs/cli/v1.45/astro-deployment-pool-update)


# astro deployment pool update
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-pool-update

Update a Deployment's Airflow pool.

<Info>
  This command is only available on Astro.
</Info>

Update the value for a Deployment's Airflow pool.

## Usage

```sh wrap theme={null}
astro deployment airflow-pool update
```

<Tip>
  This command is recommended for automated workflows. To run this command in an automated process such as a [CI/CD pipeline](/docs/astro/set-up-ci-cd), you can generate an API token, then specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

  ```bash wrap theme={null}
  export ASTRO_API_TOKEN=<your-token>
  ```

  See [Organization](/docs/astro/organization-api-tokens), [Workspace](/docs/astro/workspace-api-tokens), and [Deployment](/docs/astro/deployment-api-tokens) API token documentation for more details about ways to use API tokens.
</Tip>

## Options

| Option                 | Description                                                                                                                      | Possible Values           |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-d`,`--deployment-id` | The ID of the Deployment where you want to create Airflow pools.                                                                 | Any valid Deployment ID   |
| `--deployment-name`    | The name of the Deployment where you want to create Airflow pools. Use as an alternative to `<deployment-id>`.                   | Any valid Deployment name |
| `-w`,`--workspace-id`  | Create Airflow pools in a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed. | Any valid Workspace ID    |
| `--name`               | The Airflow pool name. Required.                                                                                                 | Any string                |
| `-v`,`--slots`         | Number of airflow pool slots. Required.                                                                                          | Any integer               |
| `--description`        | The pool description.                                                                                                            | Any string                |

## Examples

```bash wrap theme={null}
# update pool called my-pool stored in the Deployment with an ID of cl03oiq7d80402nwn7fsl3dmv
astro deployment pool update --deployment-id cl03oiq7d80402nwn7fsl3dmv --name my-pool --slots 10

# update pools stored in the Deployment "My Deployment"
astro deployment pool update --deployment-name="My Deployment" --name my-pool --slots 10

```

## Related commands

* [`astro deployment pool create`](/docs/cli/v1.45/astro-deployment-pool-create)
* [`astro deployment pool list`](/docs/cli/v1.45/astro-deployment-pool-list)


# astro deployment runtime migrate
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-runtime-migrate

Migrate an Astro Private Cloud Deployment to Astro Runtime.

<Info>This command is available only if you're authenticated to an Astro Private Cloud installation.</Info>

Initializes the Runtime migration upgrade process on any Deployment on Astro Private Cloud.

## Usage

Run `astro deployment runtime migrate --deployment-id=<deployment-id>` to initialize the Runtime upgrade process. To finalize the Runtime upgrade process, complete all of the steps in [Migrate to Astro Runtime](/docs/astro-private-cloud/v-0-36/migrate-to-runtime).

The Astro CLI lists only the available Runtime versions that are later than the version currently specified in your `Dockerfile`.

## Options

| Option                         | Description                                                                                                             | Possible values   |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `--cancel`                     | Cancel the migration                                                                                                    | None              |
| `--deployment-id` (*Required*) | The ID of the Deployment which you want to migrate to Runtime. To find your Deployment ID, run `astro deployment list`. | Any Deployment ID |

## Related commands

* [`astro deployment airflow upgrade`](/docs/cli/v1.45/astro-deployment-airflow-upgrade)
* [`astro deployment runtime upgrade`](/docs/cli/v1.45/astro-deployment-runtime-upgrade)


# astro deployment runtime upgrade
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-runtime-upgrade

Upgrade a Deployment (Astro Private Cloud only).

<Info>This command is available only if you're authenticated to an Astro Private Cloud installation.</Info>

Initializes the Runtime version upgrade process on any Deployment on Astro Private Cloud.

## Usage

Run `astro deployment runtime upgrade --deployment-id=<deployment-id>` to initialize the Runtime upgrade process. To finalize the Runtime upgrade process, complete all of the steps in [Upgrade Apache Airflow on Astro Private Cloud](https://www.astronomer.io/docs/software/manage-airflow-versions).

## Options

| Option                         | Description                                                                                                                 | Possible values               |
| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------- | ----------------------------- |
| `--cancel`                     | Cancel the upgrade                                                                                                          | None                          |
| `--deployment-id` (*Required*) | The ID of the Deployment for which you want to upgrade Runtime. Run `astro deployment list ` to retrieve the Deployment ID. | Any Deployment ID             |
| `--desired-runtime-version`    | The Runtime version you're upgrading to. For example, `2.2.0`.                                                              | Any supported Runtime version |

## Examples

```text wrap theme={null}
# Upgrade to Runtime 6.0.0
$ astro deployment runtime --deployment-id telescopic-sky-4599 --desired-runtime-version 6.0.0
```

## Related commands

* [`astro deployment airflow upgrade`](/docs/cli/v1.45/astro-deployment-airflow-upgrade)
* [`astro deployment runtime migrate`](/docs/cli/v1.45/astro-deployment-runtime-migrate)


# astro deployment service-account
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-service-account

Manage service accounts (Astro Private Cloud only).

<Info>This command is available only if you're authenticated to an Astro Private Cloud installation.</Info>

Manage Deployment-level service accounts, which you can use to configure a CI/CD pipeline or otherwise interact with the Astronomer Houston API.

## Usage

This command includes three subcommands: `create`, `delete`, and `list`

```sh wrap theme={null}
# Creates a Deployment-level service account
astro deployment service-account create --deployment-id=<your-deployment-id> --label=<your-service-account-label>

# Deletes a Deployment-level service account
astro deployment service-account delete <your-service-account-label>

# Shows the name, ID, and API token for each service account in a specific Deployment.
astro deployment service-account list
```

## Options

| Option                                                 | Description                                                     | Possible Values                                                        |
| ------------------------------------------------------ | --------------------------------------------------------------- | ---------------------------------------------------------------------- |
| `--category`                                           | The category for the new service account as displayed in the UI | Any string. The default value is `default`.                            |
| `--deployment-id` (Required for `create` and `delete`) | The Deployment you're creating a service account for            | Any Deployment ID                                                      |
| `--label` (Required for `create`)                      | The name or label for the new service account                   | Any string                                                             |
| `--role`                                               | The User Role for the new service account                       | Can be either `viewer`, `editor`, or `admin`. The default is `viewer`. |


# astro deployment team
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-team

Manage Deployment Teams.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change between product contexts.</Info>

Manage Deployment-level Teams.

<Tabs>
  <Tab title="Astro">
    ## Usage

    This command includes four subcommands: `add`, `list`, `update`, and `remove`.

    ```sh wrap theme={null}
    astro deployment team add --deployment-id=<your-deployment-id> <team-id> --role --workspace-id
    astro deployment team list --deployment-id=<deployment-id> <team-id> --role --workspace-id
    astro deployment team update --deployment-id=<your-deployment-id> <team-id> --role --workspace-id
    astro deployment team remove --deployment-id=<your-deployment-id> <team-id> --role --workspace-id
    ```

    To find a Team ID using the Astro CLI, run `astro organization team list`.

    To find a Team ID in the Astro UI, click **Organization Settings** > **Access Management** > **Teams**. Search for your Team in the **Teams** table and copy its **ID**. The ID should look something like `clk17xqgm124q01hkrgilsr49`.

    ## Options

    | Option                         | Description                                 | Possible Values                           |
    | ------------------------------ | ------------------------------------------- | ----------------------------------------- |
    | `--deployment-id` (*Required*) | The Deployment for the Team                 | Any valid Deployment ID                   |
    | `<team-id>` (*Required*)       | The Team's ID                               | Any valid Team ID                         |
    | `--role`                       | The role for the team.                      | `DEPLOYMENT_ADMIN` or a custom role name. |
    | `--workspace-id`               | The Workspace from which to remove the user | Any valid Workspace ID                    |

    ### Additional options for `list`

    | Option          | Description                                                                                          | Possible Values                |
    | --------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------ |
    | `--json`        | Output the response as JSON. Shorthand for `--output json`. Mutually exclusive with `-o`/`--output`. | None                           |
    | `-o`,`--output` | Format the output. The default is `table`.                                                           | `table`, `json`, or `template` |
    | `--template`    | Format the output using a Go template. Use with `--output template`.                                 | Any valid Go template string   |
  </Tab>

  <Tab title="APC">
    ## Usage

    This command includes four subcommands: `add`, `list`, `update`, and `remove`:

    ```sh wrap theme={null}
    astro deployment team add --deployment-id=<your-deployment-id> <team-id>
    astro deployment team list <deployment-id>
    astro deployment team update --deployment-id=<your-deployment-id> <team-id>
    astro deployment team remove --deployment-id=<your-deployment-id> <team-id>
    ```

    You can retrieve a Team's ID in one of two ways:

    * Access the Team in the UI and copy the last part of the URL in your web browser. For example, if your Team is located at `BASEDOMAIN.astronomer.io/w/cx897fds98csdcsdafasdot8g7/team/cl4iqjamcnmfgigl4852flfgulye`, your Team ID is `cl4iqjamcnmfgigl4852flfgulye`.
    * Run [`astro workspace team list`](/docs/cli/v1.45/astro-workspace-team-list) and copy the value in the `ID` column.

    ## Options

    | Option                         | Description                       | Possible Values                                                                                                            |
    | ------------------------------ | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
    | `--deployment-id` (*Required*) | The Deployment for the Team       | Any valid Deployment ID                                                                                                    |
    | `<team-id>` (*Required*)       | The Team's ID                     | Any valid Team ID                                                                                                          |
    | `--role`                       | The Team's role in the Deployment | Possible values are either `DEPLOYMENT_VIEWER`, `DEPLOYMENT_EDITOR`, or `DEPLOYMENT_ADMIN`. Default is `DEPLOYMENT_VIEWER` |
  </Tab>
</Tabs>


# astro deployment token create
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-token-create

Create a Deployment API token.

<Info>
  This command is only available on Astro.
</Info>

Create a [Deployment API Token](/docs/astro/deployment-api-tokens).

See [Authenticate an automation tool](/docs/astro/automation-authentication) to use your API token in an automated process.

## Usage

```sh wrap theme={null}
astro deployment token create --deployment-id=<deployment-id>
```

## Options

| Option                | Description                                                                                                    | Possible Values              |
| --------------------- | -------------------------------------------------------------------------------------------------------------- | ---------------------------- |
| `-c`,`--clean-output` | Print only the token as output. For use of the command in scripts.                                             | `True` or `False`            |
| `-d`,`--description`  | Description of the token. If the description contains a space, specify the entire description within quotes "" | String                       |
| `-e`,`--expiration`   | Expiration of the token in days. If the flag isn't used the token won't have an expiration.                    | An integer between 1 and 365 |
| `-n`,`--name`         | The token's name. If the name contains a space, specify the entire name within quotes `""`.                    | String                       |
| `-r`,`--role`         | The role for the token. Possible values are `DEPLOYMENT_ADMIN` or a custom role name.                          | Any valid Deployment role    |

## Examples

```bash wrap theme={null}
# create a deployment token
astro deployment token create --deployment-id=clukapi6r000008l58530cg8i
```

## Related commands

* [`astro deployment token list`](/docs/cli/v1.45/astro-deployment-token-list)
* [`astro deployment token rotate`](/docs/cli/v1.45/astro-deployment-token-rotate)
* [`astro deployment token update`](/docs/cli/v1.45/astro-deployment-token-update)


# astro deployment token delete
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-token-delete

Delete a Deployment API token.

<Info>
  This command is only available on Astro.
</Info>

Delete a [Deployment API Token](/docs/astro/deployment-api-tokens).

See [Authenticate an automation tool](/docs/astro/automation-authentication) to use your API token in an automated process.

## Usage

```sh wrap theme={null}
astro deployment token delete --deployment-id=<deployment-id>
```

## Options

| Option         | Description                                                                                                 | Possible Values |
| -------------- | ----------------------------------------------------------------------------------------------------------- | --------------- |
| `-f`,`--force` | Delete or remove the API token without showing a warning                                                    | N/A             |
| `-t`,`--name`  | The name of the token to be deleted. If the name contains a space, specify the entire name within quotes "" | String          |

## Examples

```bash wrap theme={null}
# Delete a Deployment API token
astro deployment token delete --deployment-id=clukapi6r000008l58530cg8i
```

## Related commands

* [`astro deployment token list`](/docs/cli/v1.45/astro-deployment-token-list)
* [`astro deployment token rotate`](/docs/cli/v1.45/astro-deployment-token-rotate)
* [`astro deployment token update`](/docs/cli/v1.45/astro-deployment-token-update)


# astro deployment token list
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-token-list

List your Deployment API tokens.

<Info>
  This command is only available on Astro.
</Info>

List all [API tokens](/docs/astro/deployment-api-tokens) belonging to a Deployment.

To use an API token in an automated process, see [Authenticate an automation tool](/docs/astro/automation-authentication).

## Usage

```sh wrap theme={null}
astro deployment token list --deployment-id=<deployment-id>
```

## Options

| Option            | Description                          | Possible Values                                                           |
| ----------------- | ------------------------------------ | ------------------------------------------------------------------------- |
| `--deployment-id` | The Deployment to list tokens for.   | A valid Deployment ID                                                     |
| `--verbosity`     | The log level.                       | `debug`, `info`, `warn`, `error`, `fatal`, or `panic`. Default is `warn`. |
| `--workspace-id`  | The Workspace ID for the Deployment. | A valid Workspace ID                                                      |

## Output

| Output        | Description                                                                  | Data Type |
| ------------- | ---------------------------------------------------------------------------- | --------- |
| `ID`          | The token ID.                                                                | String    |
| `NAME`        | The name of the token.                                                       | String    |
| `DESCRIPTION` | The description of the API Token.                                            | String    |
| `SCOPE`       | Whether the API Token is scoped to a Deployment, Workspace, or Organization. | String    |
| `CREATED`     | How long ago the token was created in days.                                  | String    |
| `CREATED BY`  | The name of the user entity who created the token.                           | String    |

## Examples

```bash wrap theme={null}
# List tokens for a single Deployment
astro deployment token list --deployment-id=clukapi6r000008l58530cg8i
```

## Related commands

* [`astro deployment token create`](/docs/cli/v1.45/astro-deployment-token-create)
* [`astro deployment token update`](/docs/cli/v1.45/astro-deployment-token-update)
* [`astro deployment token rotate`](/docs/cli/v1.45/astro-deployment-token-rotate)


# astro deployment token organization-token
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-token-organization-token

Scope an Organization token to a specific Deployment.

<Info>
  This command is only available on Astro.
</Info>

Manage Organization-level API tokens within a specific Deployment. See [Assign an Organization or Workspace API token to a Deployment](/docs/astro/deployment-api-tokens#assign-an-organization-or-workspace-api-token-to-a-deployment).

## astro deployment organization-token add

Add an Organization API token to a Deployment and grant it Deployment-specific permissions.

### Usage

```sh wrap theme={null}
astro deployment organization-token add --deployment-id=<my-deployment-id> --role=DEPLOYMENT_ADMIN --workspace-id=<workspace-id>
```

### Options

| Option                   | Description                                               | Valid Values                                                                          |
| ------------------------ | --------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `--deployment-id`        | The ID of the Deployment where you want to manage tokens. | Any Deployment ID.                                                                    |
| `-n`, `--org-token-name` | The name of the Organization API token.                   | Any string. If the name contains a space, specify the entire name within quotes `""`. |
| `-r`, `--role`           | The role the API token has in the Deployment.             | `DEPLOYMENT_ADMIN` or a custom role name.                                             |
| `--workspace-id`         | The Workspace to which the Deployment belongs.            | Any Workspace ID.                                                                     |

### Example

```sh wrap theme={null}
astro deployment organization-token add --deployment-id=clvduhrvd000008l842ohcpvb --role=DEPLOYMENT_ADMIN --org-token-name="My org token"
```

## astro deployment organization-token list

List all Organization API tokens that are assigned to a specific Deployment.

### Usage

```sh wrap theme={null}
astro deployment organization-token list --deployment-id=<your-deployment-id>
```

### Options

| Option            | Description                                      | Valid Values       |
| ----------------- | ------------------------------------------------ | ------------------ |
| `--deployment-id` | The ID of the Deployment to list API tokens for. | Any Deployment ID. |
| `--workspace-id`  | The Workspace to which the Deployment belongs.   | Any Workspace ID.  |

### Output

| Output            | Description                                      | Data Type |
| ----------------- | ------------------------------------------------ | --------- |
| `ID`              | The API token ID.                                | String    |
| `NAME`            | The name of the API token.                       | String    |
| `DESCRIPTION`     | The API token description.                       | String    |
| `SCOPE`           | The original scope of the API token.             | String    |
| `DEPLOYMENT_ROLE` | The API token's role in the Deployment.          | String    |
| `CREATED`         | How long ago the API token was created, in days. | String    |
| `CREATED BY`      | The name of the user who created the API token.  | String    |

## astro deployment organization-token remove

Remove an Organization API token from a Deployment.

### Usage

```sh wrap theme={null}
astro deployment organization-token remove --deployment-id=<my-deployment-id>
```

### Options

| Option             | Description                                                                    | Valid Values       |
| ------------------ | ------------------------------------------------------------------------------ | ------------------ |
| `--deployment-id`  | The Deployment ID you want to remove an API token from.                        | Any Deployment ID. |
| `--org-token-name` | The name of the Organization API token you want to remove from the Deployment. | Any string.        |
| `--workspace-id`   | The Workspace to which the Deployment belongs.                                 | Any Workspace ID.  |

### Example

```sh wrap theme={null}
astro deployment organization-token remove --deployment-id=clvduhrvd000008l842ohcpvb --org-token-name="My org token"
```

## astro deployment organization-token update

Update the role an Organization API token has within a Deployment.

### Usage

```sh wrap theme={null}
astro deployment organization-token update --deployment-id=<my-deployment-id> --role=DEPLOYMENT_ADMIN
```

### Options

| Option                   | Description                                                                  | Valid Values                              |
| ------------------------ | ---------------------------------------------------------------------------- | ----------------------------------------- |
| `--deployment-id`        | The ID of the Deployment where you want to update an Organization API token. | Any Deployment ID.                        |
| `-r`, `--role`           | The Deployment role that you want to assign to the token.                    | `DEPLOYMENT_ADMIN` or a custom role name. |
| `-n`, `--org-token-name` | The name of the Organization API token that you want to update.              | Any string.                               |
| `--workspace-id`         | The Workspace to which the Deployment belongs.                               | Any Workspace ID.                         |

### Example

```sh wrap theme={null}
astro deployment organization-token update --deployment-id=clvduhrvd000008l842ohcpvb --org-token-name="My org token" --role=DEPLOYMENT_ADMIN
```


# astro deployment token rotate
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-token-rotate

Rotate your Deployment API tokens.

<Info>
  This command is only available on Astro.
</Info>

Rotate an Astro [Deployment API Token](/docs/astro/deployment-api-tokens).

To use your API token in an automated process, see [Authenticate an automation tool](/docs/astro/automation-authentication).

## Usage

```sh wrap theme={null}
astro deployment token rotate <token-id> --deployment-id=<deployment-id> [flags]
```

## Options

| Option                | Description                                                                                                  | Possible Values |
| --------------------- | ------------------------------------------------------------------------------------------------------------ | --------------- |
| `-c`,`--clean-output` | Print only the token as output. Use when writing scripts that run the command.                               | N/A             |
| `-f`, `--force`       | Rotate the Deployment API token without showing a warning.                                                   | N/A             |
| `-t`,`--name`         | The name of the token to be rotated. If the name contains a space, specify the entire name within quotes "". | String          |

## Examples

```bash wrap theme={null}
astro deployment token rotate --deployment-id=clukapi6r000008l58530cg8i --name "My token"
```

## Related commands

* [`astro deployment token create`](/docs/cli/v1.45/astro-deployment-token-create)
* [`astro deployment token list`](/docs/cli/v1.45/astro-deployment-token-list)
* [`astro deployment token update`](/docs/cli/v1.45/astro-deployment-token-update)


# astro deployment token update
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-token-update

Update your Deployment API tokens.

<Info>
  This command is only available on Astro.
</Info>

Update an Astro [Deployment API Token](/docs/astro/deployment-api-tokens).

To use your API token in an automated process, see [Authenticate an automation tool](/docs/astro/automation-authentication).

## Usage

```sh wrap theme={null}
astro deployment token update --deployment-id=MOCK_DEP_ID
```

## Options

| Option                | Description                                                                                                           | Possible Values            |
| --------------------- | --------------------------------------------------------------------------------------------------------------------- | -------------------------- |
| `-d`, `--description` | Updated description of the token. If the description contains a space, specify the entire description in quotes `""`. | String                     |
| `-t`,`--name`         | The current name of the token. If the name contains a space, specify the entire name within quotes `""`.              | String                     |
| `-n`, `--new-name`    | The token's new name. If the new name contains a space, specify the entire name within quotes "".                     | String                     |
| `-r`, `--role`        | The role for the token. Possible values are `DEPLOYMENT_ADMIN` or a custom role name.                                 | Any valid Deployment role. |

## Examples

```bash wrap theme={null}
# The CLI prompts you to input a role for a token with Token ID assigned to a specific Deployment
astro deployment token update <token-id> --deployment-id=clukapi6r000008l58530cg8i

# The CLI prompts you to input a role for a token identified by its name
astro deployment token update --deployment-id=clukapi6r000008l58530cg8i --name="Token name"

# The CLI prompts you to select the token from a list and input a role
astro deployment token update --deployment-id=clukapi6r000008l58530cg8i

# This command assigns a token with the specified TOKEN_ID the role `Deployment Admin` to a Deployment with the following ID.
astro deployment token update TOKEN_ID --deployment-id=clukapi6r000008l58530cg8i --role=DEPLOYMENT_ADMIN
```

## Related commands

* [`astro deployment token create`](/docs/cli/v1.45/astro-deployment-token-create)
* [`astro deployment token list`](/docs/cli/v1.45/astro-deployment-token-list)
* [`astro deployment token rotate`](/docs/cli/v1.45/astro-deployment-token-rotate)


# astro deployment token workspace-token
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-token-workspace-token

Scope a Workspace token to a specific Deployment.

<Info>
  This command is only available on Astro.
</Info>

Manage Workspace-level API tokens within a specific Deployment. See [Assign an Organization or Workspace API token to a Deployment](/docs/astro/deployment-api-tokens#assign-an-organization-or-workspace-api-token-to-a-deployment).

## astro deployment workspace-token add

Add a Workspace API token to a Deployment and grant it Deployment-specific permissions.

### Usage

```sh wrap theme={null}
astro deployment workspace-token add --deployment-id=<my-deployment-id> --role=DEPLOYMENT_ADMIN --workspace-token-name=<workspace-token-name>
```

### Options

| Option                         | Description                                        | Valid Values                                                                          |
| ------------------------------ | -------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `--deployment-id`              | The Deployment ID where you want to manage tokens. | Any Deployment ID.                                                                    |
| `-r`, `--role`                 | The role the API token has in the Deployment.      | `DEPLOYMENT_ADMIN` or a custom role name.                                             |
| `-n`, `--workspace-token-name` | The Workspace API token name.                      | Any string. If the name contains a space, specify the entire name within quotes `""`. |
| `--workspace-id`               | The Workspace to which the Deployment belongs.     | Any Workspace ID.                                                                     |

### Example

```sh wrap theme={null}
astro deployment workspace-token add --deployment-id=clvduhrvd000008l842ohcpvb  --role=DEPLOYMENT_ADMIN --workspace-token-name="My workspace token"
```

## astro deployment workspace-token list

List all Workspace API tokens that are assigned to a specific Deployment.

### Usage

```sh wrap theme={null}
astro deployment workspace-token list --deployment-id=<your-deployment-id> --workspace-id=<your-workspace-ic>
```

### Options

| Option            | Description                                      | Valid Values       |
| ----------------- | ------------------------------------------------ | ------------------ |
| `--deployment-id` | The ID of the Deployment to list API tokens for. | Any Deployment ID. |
| `--workspace-id`  | The Workspace to which the Deployment belongs.   | Any Workspace ID.  |

### Output

| Output            | Description                                      | Data Type |
| ----------------- | ------------------------------------------------ | --------- |
| `ID`              | The API token ID.                                | String    |
| `NAME`            | The name of the API token.                       | String    |
| `DESCRIPTION`     | The API token description.                       | String    |
| `SCOPE`           | The original scope of the API token.             | String    |
| `DEPLOYMENT_ROLE` | The API token's role in the Deployment.          | String    |
| `CREATED`         | How long ago the API token was created, in days. | String    |
| `CREATED BY`      | The name of the user who created the API token.  | String    |

### Example

```sh wrap theme={null}
astro deployment workspace-token list --deployment-id=clvduhrvd000008l842ohcpvb
```

## astro deployment workspace-token remove

Remove a Workspace API token from a Deployment.

### Usage

```sh wrap theme={null}
astro deployment workspace-token remove --deployment-id=<my-deployment-id> --workspace-token-name=<workspace-token-name> --workspace-id=<my-workspace-id>
```

### Options

| Option                         | Description                                                                 | Valid Values       |
| ------------------------------ | --------------------------------------------------------------------------- | ------------------ |
| `--deployment-id`              | The Deployment ID you want to remove an API token from.                     | Any Deployment ID. |
| `-n`, `--workspace-token-name` | The name of the Workspace API token you want to remove from the Deployment. | Any string.        |
| `--workspace-id`               | The Workspace to which the Deployment belongs.                              | Any Workspace ID.  |

### Example

```sh wrap theme={null}
astro deployment workspace-token remove --deployment-id=clvduhrvd000008l842ohcpvb --workspace-token-name="My workspace token" --workspace-id=clvdwt4z3000008l60ofb6347
```

## astro deployment workspace-token update

Update the role a Workspace API token has within a Deployment.

### Usage

```sh wrap theme={null}
astro deployment workspace-token update --workspace-token-name=<workspace-token-name> --deployment-id=<my-deployment-id> --role=DEPLOYMENT_ADMIN
```

### Options

| Option                         | Description                                                   | Valid Values                              |
| ------------------------------ | ------------------------------------------------------------- | ----------------------------------------- |
| `--deployment-id`              | The Deployment ID you want to scope a Workspace API token to. | Any Deployment ID.                        |
| `-r`, `--role`                 | The Deployment role that you want to assign to the token.     | `DEPLOYMENT_ADMIN` or a custom role name. |
| `-n`, `--workspace-token-name` | The name of the Workspace API token you want to update.       | Any string.                               |
| `--workspace-id`               | The Workspace to which the Deployment belongs.                | Any Workspace ID.                         |

### Example

```sh wrap theme={null}
astro deployment workspace-token update --deployment-id=clvduhrvd000008l842ohcpvb --role=DEPLOYMENT_ADMIN --workspace-token-name="My workspace token"
```


# astro deployment unadopt
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-unadopt

Release an adopted Deployment back to operator-only management (Astro Private Cloud only).

<Info>This command is available only if you're authenticated to an Astro Private Cloud installation.</Info>

Release an adopted Deployment back to operator-only management. This command removes the Deployment record from Astro Private Cloud. It doesn't delete the Airflow custom resource, its namespace, or its metadata database.

<Warning>You can't undo this action from the CLI.</Warning>

## Usage

```sh wrap theme={null}
astro deployment unadopt --deployment-id=<deployment-id>
```

## Options

| Option                               | Description                                 | Possible Values         |
| ------------------------------------ | ------------------------------------------- | ----------------------- |
| `-d`, `--deployment-id` (*Required*) | The ID of the adopted Deployment to release | Any valid Deployment ID |

## Examples

```sh wrap theme={null}
astro deployment unadopt --deployment-id=cmrccaqdz0014hy190k24tivc
```

## Related commands

* [`astro deployment adopt`](/docs/cli/v1.45/astro-deployment-adopt)
* [`astro deployment list`](/docs/cli/v1.45/astro-deployment-list)


# astro deployment update
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-update

Update a Deployment.

<Info>
  The behavior and format of this command differs depending on what Astronomer
  product you're using. Use the following tabs to change product contexts.
</Info>

<Tabs>
  <Tab title="Astro">
    Update the configuration for a Deployment on Astro.

    <Info>
      To update existing worker queues or to create new queues for an existing
      Deployment, you must update your Deployment by using the `--deployment-file`
      flag to update a [Deployment file](/docs/astro/manage-deployments-as-code).
    </Info>

    ## Usage

    ```sh wrap theme={null}
    astro deployment update <deployment-id> <flags>
    ```

    <Tip>
      This command is recommended for automated workflows. To run this command in an automated process such as a [CI/CD pipeline](/docs/astro/set-up-ci-cd), you can generate an API token, then specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

      ```bash wrap theme={null}
      export ASTRO_API_TOKEN=<your-token>
      ```

      See [Organization](/docs/astro/organization-api-tokens), [Workspace](/docs/astro/workspace-api-tokens), and [Deployment](/docs/astro/deployment-api-tokens) API token documentation for more details about ways to use API tokens.
    </Tip>

    ## Options

    | Option                         | Description                                                                                                                                                                                                                                                                                                                                               | Possible Values                                                                                                                                                                                                                                                                                    |
    | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
    | `<deployment-id>` (*Required*) | The ID of the Deployment to update                                                                                                                                                                                                                                                                                                                        | Any valid Deployment ID                                                                                                                                                                                                                                                                            |
    | `--allowed-ip-address-ranges`  | If Remote Execution is enabled, limit the Deployment's incoming traffic to the Remote Agents in your environment.                                                                                                                                                                                                                                         | A comma-separated list of CIDR blocks, such as `192.168.0.0/24,10.0.0.0/16`.                                                                                                                                                                                                                       |
    | `--cicd-enforcement`           | Specify that the Deployment can only accept code deploys from API tokens. Note that in CLI versions before 1.23, this flag was `--enforce-cicd`.                                                                                                                                                                                                          | None                                                                                                                                                                                                                                                                                               |
    | `--dag-deploy`                 | Enable or disable dag-only deploys for the Deployment.                                                                                                                                                                                                                                                                                                    | Either `enable` or `disable`. Contact [Astronomer support](https://cloud.astronomer.io/open-support-request) before using `disable` to disable the feature.                                                                                                                                        |
    | `--default-task-pod-cpu`       | The default task Pod CPUs to use for the Deployment. See [Customize a task's Kubernetes Pod](/docs/astro/kubernetes-executor#customize-a-task’s-kubernetes-pod) for more information.                                                                                                                                                                          | A numeric value such as `0.25`. This number cannot exceed the values you configured in your [Default pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources) in the UI or in your Deployment configuration file.                                                            |
    | `--default-task-pod-memory`    | The default task pod memory to use for the Deployment in Gi. See [Customize a task's Kubernetes Pod](/docs/astro/kubernetes-executor#customize-a-task’s-kubernetes-pod) for more information.                                                                                                                                                                  | A numeric value followed by `Gi`, such as `0.5Gi`. This number cannot exceed the values you configured in your [Default pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources) in the UI or in your Deployment configuration file.                                         |
    | `--deployment-file`            | A location of the Deployment file to update the Deployment with. The file format can be JSON or YAML. See [Create a Deployment with a Deployment File](/docs/astro/manage-deployments-as-code).                                                                                                                                                                | A valid file path to any YAML or JSON Deployment file                                                                                                                                                                                                                                              |
    | `--deployment-name`            | The name of the Deployment to update. Use as an alternative to `<deployment-id>`.                                                                                                                                                                                                                                                                         | Any valid Deployment name                                                                                                                                                                                                                                                                          |
    | `-d`,`--description`           | A description of the Deployment                                                                                                                                                                                                                                                                                                                           | Any string. Multiple-word descriptions should be specified in quotations (`"`)                                                                                                                                                                                                                     |
    | `-m`, `--development-mode`     | Whether the Deployment is for development mode only. If set to `disable`, the Deployment can be considered production for the purposes of Astro support case priority, but development-only features such as hibernation aren't available. You can't update this value to `enable` for existing non-development Deployments. Set to `disable` by default. | `disable` or `enable`                                                                                                                                                                                                                                                                              |
    | `-e`,`--executor`              | The executor to use for the Deployment                                                                                                                                                                                                                                                                                                                    | CeleryExecutor or KubernetesExecutor                                                                                                                                                                                                                                                               |
    | `-f`,`--force`                 | Force a Deployment update                                                                                                                                                                                                                                                                                                                                 | None                                                                                                                                                                                                                                                                                               |
    | `-n`,`--name`                  | The Deployment's name                                                                                                                                                                                                                                                                                                                                     | Any string. Multiple-word descriptions should be specified in quotations.                                                                                                                                                                                                                          |
    | `--resource-quota-cpu`         | The Deployment's CPU resource quota for Kubernetes Pods. See [Customize a task's Kubernetes Pod](/docs/astro/kubernetes-executor#customize-a-task’s-kubernetes-pod) for more information.                                                                                                                                                                      | A numeric value such as `10`. This number cannot exceed the values you configured in your [Default pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources) in the UI or in your [Deployment file](/docs/astro/deployment-file-reference) configuration file.                     |
    | `--resource-quota-memory`      | The Deployment's memory resource quota for Kubernetes Pods in Gi. See [Customize a task's Kubernetes Pod](/docs/astro/kubernetes-executor#customize-a-task’s-kubernetes-pod) for more information.                                                                                                                                                             | A numeric value followed by `Gi`, such as `20Gi`. This number cannot exceed the values you configured in your [Default pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources) in the UI or in your [Deployment file](/docs/astro/deployment-file-reference) configuration file. |
    | `--scheduler-size`             | The size of the Scheduler for the Deployment.                                                                                                                                                                                                                                                                                                             | Possible values can be `small`, `medium`, `large`, or `extra_large`. `extra_large` schedulers require a minimum Astro Runtime version of 9.7.0.                                                                                                                                                    |
    | `-s`,`--scheduler-au`          | The number of AU to allocate towards the Deployment's Scheduler(s). The default is`5`.                                                                                                                                                                                                                                                                    | Integer between `0` and `24`                                                                                                                                                                                                                                                                       |
    | `-r`,`--scheduler-replicas`    | The number of scheduler replicas for the Deployment. The default is `1`.                                                                                                                                                                                                                                                                                  | Integer between `0` and `4`                                                                                                                                                                                                                                                                        |
    | `--task-log-bucket`            | If Remote Execution is enabled, specify the cloud storage bucket for task log storage. You can configure your Deployment to [show the stored logs directly in the Airflow UI](/docs/astro/remote-task-logs-af-ui).                                                                                                                                             | A valid cloud storage bucket name, such as `gs://my-task-logs` or `s3://my-task-logs`.                                                                                                                                                                                                             |
    | `--task-log-url-pattern`       | If Remote Execution is enabled, specify the URL template to link to task logs stored in an external logging provider from the Airflow UI.                                                                                                                                                                                                                 | A string URL template. See [Configure external logging provider for Remote Execution Deployments](/docs/astro/remote-task-logs-external#configure-external-logging-provider-for-remote-execution-deployments).                                                                                          |
    | `--workload-identity`          | The workload identity to use for the Deployment.                                                                                                                                                                                                                                                                                                          | A valid AWS `arn` or a GCP service account.                                                                                                                                                                                                                                                        |
    | `-w`,`--workspace-id`          | Specify a Workspace to update a Deployment outside of your current Workspace                                                                                                                                                                                                                                                                              | Any valid Workspace ID                                                                                                                                                                                                                                                                             |

    ## Examples

    ```sh wrap theme={null}
    # Update a Deployment's name and description
    $ astro deployment update cl03oiq7d80402nwn7fsl3dmv -d="My Deployment Description" --name="My Deployment Name"

    # Force update a Deployment
    $ astro deployment update cl03oiq7d80402nwn7fsl3dmv -d="My Deployment Description" --force

    # Update the Deployment according to the configurations specified in the YAML Deployment file
    $ astro deployment update --deployment-file deployment.yaml
    ```
  </Tab>

  <Tab title="APC">
    Create a Deployment on Astro Private Cloud. This command is functionally identical to using the UI to create a Deployment.

    ## Usage

    ```sh wrap theme={null}
    astro deployment update <deployment-id>
    ```

    ## Options

    | Option                         | Description                                                                                                                                                        | Possible Values                                                                                                                     |
    | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- |
    | `<deployment-id>` (*Required*) | The ID for the Deployment to update                                                                                                                                | Any valid Deployment ID                                                                                                             |
    | `-a`,`--airflow-version`       | The version of Astronomer Certified to use for the Deployment                                                                                                      | Any supported version of Astronomer Certified                                                                                       |
    | `-c`, `--cloud-role`           | An AWS or GCP IAM role to append to your Deployment's webserver, scheduler, and worker Pods                                                                        | Any string                                                                                                                          |
    | `-t`, `--dag-deployment-type`  | The dag deploy method for the Deployment                                                                                                                           | Can be either `image`, `git_sync`, or `volume`. The default is `image`                                                              |
    | `-d`,`--description`           | The description for the Deployment                                                                                                                                 | Any string. Multiple-word descriptions should be specified in quotations (`"`)                                                      |
    | `-e`, `--executor`             | The executor type for the Deployment                                                                                                                               | `local`, `celery`, or `kubernetes`. The default value is `celery`                                                                   |
    | `-b`, `--git-branch-name`      | The branch name of the git repo to sync your Deployment from. Must be specified with `--dag-deployment-type=git_sync`                                              | Any valid git branch name                                                                                                           |
    | `-u`, `--git-repository-url`   | The URL for the git repository to sync your Deployment from. Must be specified with `--dag-deployment-type=git_sync`                                               | Any valid git repository URL                                                                                                        |
    | `-v`, `--git-revision`         | The commit reference of the branch that you want to sync with your Deployment. Must be specified with `--dag-deployment-type=git_sync`                             | Any valid git revision                                                                                                              |
    | `--known-hosts`                | The public key for your Git provider, which can be retrieved using `ssh-keyscan -t rsa <provider-domain>`. Must be specified with `--dag-deployment-type=git_sync` | Any valid public key                                                                                                                |
    | `-l`,`--label`                 | The label for your Deployment                                                                                                                                      | Any string                                                                                                                          |
    | `-n`,`--nfs-location`          | The location for an NFS volume mount. Must be specified with `--dag-deployment-type=volume`                                                                        | An NFS volume mount specified as: `<IP>:/<path>`. Input is automatically prepended with `nfs:/` - do not include this in your input |
    | `-r`,`--release-name`          | A custom release name for the Deployment. See [Customize release names](https://www.astronomer.io/docs/software/configure-deployment#customize-release-names)      | Any string of alphanumeric and hyphen characters                                                                                    |
    | `--runtime-version`            | The Astro Runtime version for the Deployment                                                                                                                       | Any supported version of Astro Runtime. Major, minor, and patch versions must be specified                                          |
    | `--ssh-key`                    | The SSH private key for your Git repository. Must be specified with `--dag-deployment-type=git_sync`                                                               | Any valid SSH key                                                                                                                   |
    | `-s`,`--sync-interval`         | The time interval between checks for updates in your Git repository, in seconds. Must be specified with `--dag-deployment-type=git_sync`                           | Any integer                                                                                                                         |
    | `-t`,`--triggerer-replicas`    | Number of replicas to use for the Airflow triggerer                                                                                                                | Any integer between 0 - 2. The default value is 1                                                                                   |
    | `--workload-identity`          | The workload identity to use for the Deployment. You must use this flag with `--cloud-provider`.                                                                   | A valid AWS `arn` or GCP `service account`                                                                                          |
    | `--workspace-id`               | The Workspace in which to create a Deployment. If not specified, your current Workspace is assumed                                                                 | Any valid Workspace ID                                                                                                              |

    ## Examples

    ```sh wrap theme={null}
    $ astro deployment update telescopic-sky-4599 --executor kubernetes
    # Change the executor for a Deployment

    $ astro deployment update telescopic-sky-4599 -l="My Deployment label" --workspace-id="ckwqkz36200140ror6axh8p19"
    # Update a Deployment in a separate Workspace. The CLI will not prompt you for more information
    ```
  </Tab>
</Tabs>

## Related commands

* [`astro deployment delete`](/docs/cli/v1.45/astro-deployment-delete)
* [`astro deployment list`](/docs/cli/v1.45/astro-deployment-list)


# astro deployment user
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-user

Manage Deployment users.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change between product contexts.</Info>

Manage Deployment-level users.

<Tabs>
  <Tab title="Astro">
    ## Usage

    This command has several subcommands. Read the following sections to learn how to use each subcommand.

    ### astro deployment user add

    Give an existing user in a Workspace access to a Deployment within that Workspace.

    You must be a Deployment Admin for the given Deployment to run this command.

    #### Usage

    ```sh wrap theme={null}
    astro deployment user add --email=<user-email-address> --deployment-id=<user-deployment-id> --workspace-id=<user-workspace-id> --role=<user-role>
    ```

    #### Options

    | Option                         | Description                                         | Possible values                                                                                    |
    | ------------------------------ | --------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
    | `--deployment-id` (*Required*) | The ID for the Deployment that the user is added to | Any valid Deployment ID                                                                            |
    | `-e`,`--email` (*Required*)    | The user's email                                    | Any valid email address                                                                            |
    | `--role` (*Required*)          | The role assigned to the user                       | Possible values are `DEPLOYMENT_ADMIN` or the custom role name. The default is `DEPLOYMENT_ADMIN`. |
    | `--workspace-id`               | The workspace assigned to the Deployment            | Any valid Workspace ID                                                                             |

    ### astro deployment user list

    View a list of all Workspace users who have access to a given Deployment.

    #### Usage

    ```sh wrap theme={null}
    astro deployment user list --deployment-id=<deployment-id>
    ```

    #### Options

    | Option                       | Description                                                                                          | Possible values                |
    | ---------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------ |
    | `--deployment-id` (Required) | The Deployment whose users you want to view                                                          | Any valid Deployment ID        |
    | `--json`                     | Output the response as JSON. Shorthand for `--output json`. Mutually exclusive with `-o`/`--output`. | None                           |
    | `-o`,`--output`              | Format the output. The default is `table`.                                                           | `table`, `json`, or `template` |
    | `--template`                 | Format the output using a Go template. Use with `--output template`.                                 | Any valid Go template string   |
    | `--workspace-id`             | The Workspace whose users you want to view                                                           | Any valid Workspace ID         |

    ### astro deployment user remove

    Remove access to a Deployment for an existing Workspace user, the command line prompts you for the user email address or user ID for the user you want to remove. To grant that same user a different set of permissions instead, modify their existing Deployment-level role by running `astro deployment user update`.

    You must be a Deployment Admin for the given Deployment to run this command.

    #### Usage

    ```sh wrap theme={null}
    astro deployment user remove --deployment-id=<deployment-id> --email=<user-email-address> --workspace-id=<workspace-id>
    ```

    #### Options

    | Option                       | Description                                  | Possible values         |
    | ---------------------------- | -------------------------------------------- | ----------------------- |
    | `--deployment-id` (Required) | The Deployment from which to remove the user | Any valid Deployment ID |
    | `-e`,`--email` (*Required*)  | The user's email                             | Any valid email address |
    | `--workspace-id`             | The Workspace from which to remove the user  | Any valid Workspace ID  |

    ### astro deployment user update

    Update a user's role in a given Deployment.

    #### Usage

    ```sh wrap theme={null}
    astro deployment user update --email=<email=address> --deployment-id=<deployment-id> --role=
    ```

    #### Options

    | Option                       | Description                          | Possible values                                                                                          |
    | ---------------------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------- |
    | `--deployment-id` (Required) | The Deployment that you're searching | Any valid Deployment ID.                                                                                 |
    | `-e`,`--email` (*Required*)  | The user's email                     | Any valid email address                                                                                  |
    | `--role`                     | The role for the user.               | Possible values are `DEPLOYMENT_ADMIN` or the custom role name. The default value is `DEPLOYMENT_ADMIN`. |

    #### Related documentation

    * [Manage User Permissions on Astronomer](/docs/astro-private-cloud/v-0-37/workspace-permissions)
  </Tab>

  <Tab title="APC">
    ## Usage

    This command has several subcommands. Read the following sections to learn how to use each subcommand.

    ### astro deployment user add

    Give an existing user in a Workspace access to a Deployment within that Workspace.

    You must be a Deployment Admin for the given Deployment to run this command.

    #### Usage

    ```sh wrap theme={null}
    astro deployment user add --email=<user-email-address> --deployment-id=<user-deployment-id> --role<user-role>
    ```

    #### Options

    | Option                         | Description                                         | Possible values                                                             |
    | ------------------------------ | --------------------------------------------------- | --------------------------------------------------------------------------- |
    | `--deployment-id` (*Required*) | The ID for the Deployment that the user is added to | Any valid Deployment ID                                                     |
    | `-e`,`--email` (*Required*)    | The user's email                                    | Any valid email address                                                     |
    | `--role` (*Required*)          | The role assigned to the user                       | 1DEPLOYMENT\_ADMIN1 or the custom role name. Default is `DEPLOYMENT_ADMIN`. |

    #### Related documentation

    * [Manage User Permissions on Astronomer](/docs/astro-private-cloud/v-0-37/workspace-permissions)

    ### astro deployment user remove

    Remove access to a Deployment for an existing Workspace user. To grant that same user a different set of permissions instead, modify their existing Deployment-level role by running `astro deployment user update`.

    You must be a Deployment Admin for the given Deployment to run this command.

    #### Usage

    ```sh wrap theme={null}
    astro deployment user remove --deployment-id=<deployment-id> --email=<user-email-address>
    ```

    #### Options

    | Option                       | Description                                  | Possible values         |
    | ---------------------------- | -------------------------------------------- | ----------------------- |
    | `--email` (Required)         | The user's email                             | Any valid email address |
    | `--deployment-id` (Required) | The Deployment from which to remove the user | Any valid Deployment ID |

    #### Related documentation

    * [Manage User Permissions on Astronomer](/docs/astro-private-cloud/v-0-37/workspace-permissions)

    ### astro deployment user list

    View a list of all Workspace users who have access to a given Deployment.

    #### Usage

    ```sh wrap theme={null}
    astro deployment user list --deployment-id=<deployment-id>
    ```

    #### Options

    | Option                       | Description                                 | Possible values         |
    | ---------------------------- | ------------------------------------------- | ----------------------- |
    | `--deployment-id` (Required) | The Deployment that you're searching in     | Any valid Deployment ID |
    | `--email`                    | The email for the user you're searching for | Any valid email address |
    | `--name`                     | The name of the user to search for          | Any string              |

    #### Related documentation

    * [Manage User Permissions on Astronomer](/docs/astro-private-cloud/v-0-37/workspace-permissions)

    ### astro deployment user update

    Update a user's role in a given Deployment.

    #### Usage

    ```sh wrap theme={null}
    astro deployment user update --deployment-id=<deployment-id>
    ```

    #### Options

    | Option                       | Description                          | Possible values                                                                                                                |
    | ---------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ |
    | `--deployment-id` (Required) | The Deployment that you're searching | Any valid Deployment ID.                                                                                                       |
    | `--role`                     | The role for the user.               | Possible values are `DEPLOYMENT_VIEWER`, `DEPLOYMENT_EDITOR`, or `DEPLOYMENT_ADMIN`. The default value is `DEPLOYMENT_VIEWER`. |

    #### Related documentation

    * [Manage User Permissions on Astronomer](/docs/astro-private-cloud/v-0-37/workspace-permissions)
  </Tab>
</Tabs>


# astro deployment variable create
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-variable-create

Reference documentation for astro deployment variable create.

<Info>
  This command is only available on Astro.
</Info>

For a given Deployment on Astro, create environment variables in the Astro CLI by supplying either a key and value pair or a file (for example, `.env`) with a list of keys and values. This command is functionally identical to creating an environment variable in the Astro UI. See [Set Environment Variables on Astro](/docs/astro/environment-variables).

## Usage

```sh wrap theme={null}
astro deployment variable create <key>=<value>
```

<Tip>
  This command is recommended for automated workflows. To run this command in an automated process such as a [CI/CD pipeline](/docs/astro/set-up-ci-cd), you can generate an API token, then specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

  ```bash wrap theme={null}
  export ASTRO_API_TOKEN=<your-token>
  ```

  See [Organization](/docs/astro/organization-api-tokens), [Workspace](/docs/astro/workspace-api-tokens), and [Deployment](/docs/astro/deployment-api-tokens) API token documentation for more details about ways to use API tokens.
</Tip>

## Options

| Option                 | Description                                                                                                                                                                                                                                                                                             | Possible Values           |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-d`,`--deployment-id` | The ID of the Deployment in which to create environment variable(s).                                                                                                                                                                                                                                    | Any valid Deployment ID   |
| `--deployment-name`    | The name of the Deployment in which to create environment variable(s). Use as an alternative to `<deployment-id>`.                                                                                                                                                                                      | Any valid Deployment name |
| `-e`,`--env`           | The path to a file that contains a list of environment variables. If a filepath isn't specified, this looks for a `.env` file in your current directory. If `.env` doesn't exist, this flag will create it for you                                                                                      | Any valid filepath        |
| `-l`,`--load`          | Export new environment variables from your Astro project's `.env` file to the Deployment. This is an alternative to creating an environment variable by manually specifying `--key` and `--value`. By default, this flag exports all new environment variables based on the file specified with `--env` | \`\`                      |
| `-s`,`--secret`        | Set the value of the new environment variable as secret                                                                                                                                                                                                                                                 | \`\`                      |
| `-w`,`--workspace-id`  | Create or update an environment variable for a Deployment that is not in your current Workspace. If this is not specified, your current Workspace is assumed                                                                                                                                            | Any valid Workspace ID    |

## Examples

```sh wrap theme={null}
# Create a new secret environment variable
$ astro deployment variable create --deployment-id cl03oiq7d80402nwn7fsl3dmv AIRFLOW__SECRETS__BACKEND_KWARGS=<my-secret-value> --secret

# Create multiple environment variables for a Deployment at once by specifying multiple keys
$ astro deployment variable create AIRFLOW__CORE__PARALLELISM=32 MAX_ACTIVE_TASKS_PER_DAG=16 --deployment-id cl03oiq7d80402nwn7fsl3dmv

# Create multiple environment variables for a Deployment at once by loading them from a .env file
$ astro deployment variable create --deployment-name="My Deployment" --load --env .env.dev
```

## Related commands

* [`astro deployment variable list`](/docs/cli/v1.45/astro-deployment-variable-list)
* [`astro deployment variable update`](/docs/cli/v1.45/astro-deployment-variable-update)


# astro deployment variable list
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-variable-list

List Deployment environment variables.

<Info>
  This command is only available on Astro.
</Info>

For a given Deployment on Astro, list its running environment variables in your terminal. To test these environment variables locally without having to manually copy them, you can also use this command to save them in a local `.env` file.

If an existing `.env` file already exists in your current directory, `--save` will append environment variables to the bottom of that file. It will not override or replace its contents. If `.env` does not exist, `--save` will create the file for you.

If an environment variable value is set as secret, the CLI will list only its key.

## Usage

```sh wrap theme={null}
astro deployment variable list
```

<Tip>
  This command is recommended for automated workflows. To run this command in an automated process such as a [CI/CD pipeline](/docs/astro/set-up-ci-cd), you can generate an API token, then specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

  ```bash wrap theme={null}
  export ASTRO_API_TOKEN=<your-token>
  ```

  See [Organization](/docs/astro/organization-api-tokens), [Workspace](/docs/astro/workspace-api-tokens), and [Deployment](/docs/astro/deployment-api-tokens) API token documentation for more details about ways to use API tokens.
</Tip>

## Options

| Option                   | Description                                                                                                                            | Possible Values           |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-d`,`--deployment-id`   | The ID of the Deployment for which to list environment variables                                                                       | Any valid Deployment ID   |
| `-n`,`--deployment-name` | The name of the Deployment for which to list environment variable(s). Use as an alternative to `<deployment-id>`.                      | Any valid Deployment name |
| `-e`,`--env`             | The file to save environment variables to when using `--save`. Defaults to `.env` in your current directory.                           | Any valid file path       |
| `-k`,`--key`             | List only the environment variable associated with this key. If not specified, all environment variables are listed                    | Any string                |
| `-s`,`--save`            | Save environment variables to a local `.env` file                                                                                      | \`\`                      |
| `-w`,`--workspace-id`    | List environment variables for a Deployment that is not in your current Workspace. If not specified, your current Workspace is assumed | Any valid Workspace ID    |

## Examples

```sh wrap theme={null}
# Save all environment variables currently running on an Astro Deployment to the `.env` file in your current directory
$ astro deployment variable list --deployment-id cl03oiq7d80402nwn7fsl3dmv --save

# Save only a single environment variable from a Deployment on Astro to a `.env` file that is outside of your current directory
$ astro deployment variable list --deployment-name="My Deployment" --key AIRFLOW__CORE__PARALLELISM --save --env /users/documents/my-astro-project/.env
```

## Related commands

* [`astro deployment variable create`](/docs/cli/v1.45/astro-deployment-variable-create)
* [`astro deployment variable update`](/docs/cli/v1.45/astro-deployment-variable-update)


# astro deployment variable update
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-variable-update

Update Deployment environment variables.

<Info>
  This command is only available on Astro.
</Info>

For a given Deployment on Astro, use `astro deployment variable update` to update the value of an existing environment variable with the Astro CLI. To do so, you can either:

* Manually enter a new `key=value` pair for an existing key directly in the command.
* Modify the value of one or more environment variables in a `.env` file and load that file with `--load`.

This command is functionally identical to editing and saving the value of an existing environment variable in the Astro UI. For more information on environment variables, see [Set environment variables on Astro](/docs/astro/manage-env-vars).

## Usage

```sh wrap theme={null}
astro deployment variable update
```

<Tip>
  This command is recommended for automated workflows. To run this command in an automated process such as a [CI/CD pipeline](/docs/astro/set-up-ci-cd), you can generate an API token, then specify the `ASTRO_API_TOKEN` environment variable in the system running the Astro CLI:

  ```bash wrap theme={null}
  export ASTRO_API_TOKEN=<your-token>
  ```

  See [Organization](/docs/astro/organization-api-tokens), [Workspace](/docs/astro/workspace-api-tokens), and [Deployment](/docs/astro/deployment-api-tokens) API token documentation for more details about ways to use API tokens.
</Tip>

## Options

| Option                 | Description                                                                                                                                                                                                                                                                                             | Possible Values           |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `-d`,`--deployment-id` | The ID of the Deployment whose environment variable(s) you want to update.                                                                                                                                                                                                                              | Any valid Deployment ID   |
| `--deployment-name`    | The name of the Deployment whose environment variable(s) you want to update. Use as an alternative to `<deployment-id>`.                                                                                                                                                                                | Any valid Deployment name |
| `-e`,`--env`           | The path to a file that contains a list of environment variables. If a filepath isn't specified, this looks for a `.env` file in your current directory. If `.env` doesn't exist, this flag will create it for you                                                                                      | Any valid filepath        |
| `-l`,`--load`          | Export updated environment variables from your Astro project's `.env` file to the Deployment. This is an alternative to updating an environment variable by manually specifying `--key` and `--value`. By default, this flag updates all environment variables based on the file specified with `--env` | \`\`                      |
| `-s`,`--secret`        | Set the value of the updated environment variable as secret                                                                                                                                                                                                                                             | \`\`                      |
| `-w`,`--workspace-id`  | Update an environment variable for a Deployment that is not in your current Workspace. If this is not specified, your current Workspace is assumed                                                                                                                                                      | Any valid Workspace ID    |

## Examples

```sh wrap theme={null}
# Update an existing environment variable and set as secret
$ astro deployment variable update --deployment-id cl03oiq7d80402nwn7fsl3dmv AIRFLOW__SECRETS__BACKEND_KWARGS=<my-new-secret-value> --secret

# Update multiple environment variables for a Deployment at once by loading them from a .env file
$ astro deployment variable update --deployment-id cl03oiq7d80402nwn7fsl3dmv --load --env .env.dev
```

## Related commands

* [`astro deployment variable create`](/docs/cli/v1.45/astro-deployment-variable-create)
* [`astro deployment variable list`](/docs/cli/v1.45/astro-deployment-variable-list)


# astro deployment wake-up
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-wake-up

Wake up a Deployment.

Wake up an Astro development Deployment from [hibernation](/docs/astro/deployment-resources#hibernate-a-development-deployment). Overrides any existing hibernation schedule and sets the Deployment to run for a specific duration or until a specific time.

## Usage

```sh wrap theme={null}
astro deployment wake-up [one of --until/ --for/ --remove-override]
```

## Options

| Option                   | Description                                                         | Possible Values                                      |
| ------------------------ | ------------------------------------------------------------------- | ---------------------------------------------------- |
| `-n`,`--deployment-name` | The name of the development Deployment to wake up.                  | Any valid Deployment name                            |
| `-u, --until`            | Specify the awake period using an end date and time.                | Any future date in the format `YYYY-MM-DDT00:00:00Z` |
| `-d, --for`              | Specify the awake period using a duration.                          | Any amount of time in the format `XhYm`              |
| `-r, --remove-override`  | Remove any overrides and resume regular hibernation schedule.       | None                                                 |
| `-f, --force`            | The CLI will not prompt to confirm before waking up the Deployment. | None                                                 |

## Related commands

* [`astro deployment hibernate`](/docs/cli/v1.45/astro-deployment-hibernate)


# astro deployment worker-queue create
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-worker-queue-create

Create a Deployment worker queue.

<Info>
  This command is only available on Astro.
</Info>

Create a [worker queue](/docs/astro/configure-worker-queues) in a Deployment on Astro. This command is functionally identical to creating a worker queue in the Astro UI.

## Usage

```sh wrap theme={null}
astro deployment worker-queue create
```

## Options

| Option                 | Description                                                                                                      | Possible Values                                                       |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| `--concurrency`        | The maximum number of tasks that each worker can run                                                             | Any integer from 1 to 64                                              |
| `-d`,`--deployment-id` | The ID of the Deployment where you want to create the worker queue                                               | Any valid Deployment ID                                               |
| `--deployment-name`    | The name of the Deployment where you want to create the worker queue. Use as an alternative to `<deployment-id>` | Any valid Deployment name                                             |
| `--max-count`          | The maximum worker count of the worker queue                                                                     | Any integer from 0 to 30                                              |
| `--min-count`          | The minimum worker count of the worker queue                                                                     | Any integer from 0 to 30                                              |
| `-n`,`--name`          | The name of the worker queue                                                                                     | Any string                                                            |
| `-t`,`--worker-type`   | The worker type of the worker queue                                                                              | Any worker type enabled on the cluster in which the Deployment exists |

## Examples

```sh wrap theme={null}
astro deployment worker-queue create --deployment-id cl03oiq7d80402nwn7fsl3dmv
# Creates a new worker queue for a Deployment with ID `cl03oiq7d80402nwn7fsl3dmv`. The Astro CLI prompts you for configuration information.

astro deployment worker-queue create --concurrency 20 --max-count 10 --min-count 2 --name "My worker queue" --worker-type "m5d.8xlarge"
# Creates a new worker queue with specified configurations. The Astro CLI prompts you for Deployment information.
```

## Related commands

* [`astro deployment worker-queue update`](/docs/cli/v1.45/astro-deployment-worker-queue-update)
* [`astro deployment worker-queue delete`](/docs/cli/v1.45/astro-deployment-worker-queue-delete)


# astro deployment worker-queue delete
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-worker-queue-delete

Delete a Deployment worker queue.

<Info>
  This command is only available on Astro.
</Info>

Delete an existing [worker queue](/docs/astro/configure-worker-queues) in a Deployment on Astro.

## Usage

```sh wrap theme={null}
astro deployment worker-queue delete
```

## Options

| Option                 | Description                                                                                                  | Possible Values           |
| ---------------------- | ------------------------------------------------------------------------------------------------------------ | ------------------------- |
| `-d`,`--deployment-id` | The ID of the Deployment whose worker queue you want to delete                                               | Any valid Deployment ID   |
| `--deployment-name`    | The name of the Deployment whose worker queue you want to delete. Use as an alternative to `<deployment-id>` | Any valid Deployment name |
| `-f` `--force`         | Skip prompting the user to confirm the deletion                                                              | \`\`                      |
| `-n`,`--name`          | The name of the worker queue to delete                                                                       | Any string                |

## Related commands

* [`astro deployment worker-queue update`](/docs/cli/v1.45/astro-deployment-worker-queue-update)
* [`astro deployment worker-queue create`](/docs/cli/v1.45/astro-deployment-worker-queue-create)


# astro deployment worker-queue update
Source: https://astronomer.io/docs/cli/v1.45/astro-deployment-worker-queue-update

Update a Deployment worker queue.

<Info>
  This command is only available on Astro.
</Info>

Update the settings for an existing [worker queue](/docs/astro/configure-worker-queues) in a Deployment on Astro. This is functionally identical to updating the settings of a worker queue in the Astro UI.

## Usage

```sh wrap theme={null}
astro deployment worker-queue update
```

## Options

| Option                 | Description                                                                                                  | Possible Values                                                       |
| ---------------------- | ------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- |
| `--concurrency`        | The maximum number of tasks that each worker can run                                                         | Any integer from 1 to 64                                              |
| `-d`,`--deployment-id` | The ID of the Deployment whose worker queue you want to update                                               | Any valid Deployment ID                                               |
| `--deployment-name`    | The name of the Deployment whose worker queue you want to update. Use as an alternative to `<deployment-id>` | Any valid Deployment name                                             |
| `-f` `--force`         | Skip prompting the user to confirm the update                                                                | \`\`                                                                  |
| `--max-count`          | The maximum worker count of the worker queue                                                                 | Any integer from 0 to 30                                              |
| `--min-count`          | The minimum worker count of the worker queue                                                                 | Any integer from 0 to 30                                              |
| `-n`,`--name`          | The name of the worker queue                                                                                 | Any string                                                            |
| `-t`,`--worker-type`   | The worker type of the worker queue                                                                          | Any worker type enabled on the cluster in which the Deployment exists |

## Examples

```sh wrap theme={null}
astro deployment worker-queue update --deployment-id cl03oiq7d80402nwn7fsl3dmv --name="Updated name"
# Update a worker queue's name in a specified Deployment.

astro deployment worker-queue update --concurrency 20 --max-count 10 --min-count 2 --name "My worker queue" --worker-type "m5d.8xlarge"
# Update a new worker queue in a Deployment. The CLI prompts you to specify a Deployment and worker queue to update
```

## Related commands

* [`astro deployment worker-queue create`](/docs/cli/v1.45/astro-deployment-worker-queue-create)
* [`astro deployment worker-queue delete`](/docs/cli/v1.45/astro-deployment-worker-queue-delete)


# astro dev
Source: https://astronomer.io/docs/cli/v1.45/astro-dev

Manage your project and interact with your local Airflow environment.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Use `astro dev` commands to manage your Astro project and interact with your local Airflow environment.

<CardGroup>
  <Card title="astro dev bash" href="/cli/v1.45/astro-dev-bash">
    View documentation for `astro dev bash`.
  </Card>

  <Card title="astro dev build" href="/cli/v1.45/astro-dev-build">
    View documentation for `astro dev build`.
  </Card>

  <Card title="astro dev init" href="/cli/v1.45/astro-dev-init">
    View documentation for `astro dev init`.
  </Card>

  <Card title="astro dev kill" href="/cli/v1.45/astro-dev-kill">
    View documentation for `astro dev kill`.
  </Card>

  <Card title="astro dev logs" href="/cli/v1.45/astro-dev-logs">
    View documentation for `astro dev logs`.
  </Card>

  <Card title="astro dev object export" href="/cli/v1.45/astro-dev-object-export">
    View documentation for `astro dev object export`.
  </Card>

  <Card title="astro dev object import" href="/cli/v1.45/astro-dev-object-import">
    View documentation for `astro dev object import`.
  </Card>

  <Card title="astro dev parse" href="/cli/v1.45/astro-dev-parse">
    View documentation for `astro dev parse`.
  </Card>

  <Card title="astro dev proxy" href="/cli/v1.45/astro-dev-proxy">
    View documentation for `astro dev proxy`.
  </Card>

  <Card title="astro dev ps" href="/cli/v1.45/astro-dev-ps">
    View documentation for `astro dev ps`.
  </Card>

  <Card title="astro dev pytest" href="/cli/v1.45/astro-dev-pytest">
    View documentation for `astro dev pytest`.
  </Card>

  <Card title="astro dev restart" href="/cli/v1.45/astro-dev-restart">
    View documentation for `astro dev restart`.
  </Card>

  <Card title="astro dev run" href="/cli/v1.45/astro-dev-run">
    View documentation for `astro dev run`.
  </Card>

  <Card title="astro dev start" href="/cli/v1.45/astro-dev-start">
    View documentation for `astro dev start`.
  </Card>

  <Card title="astro dev stop" href="/cli/v1.45/astro-dev-stop">
    View documentation for `astro dev stop`.
  </Card>

  <Card title="astro dev upgrade test" href="/cli/v1.45/astro-dev-upgrade-test">
    View documentation for `astro dev upgrade test`.
  </Card>
</CardGroup>


# astro dev bash
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-bash

Run a bash command in an Airflow component's Docker container.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Run a bash command in a locally running Docker container for an Airflow component. This command is equivalent to running `docker exec -it <container-id>`.

## Usage

In a locally running Astro project, run:

```sh wrap theme={null}
astro dev bash
```

By default, the command execs into the scheduler container and prompts you to run a bash command. To run a command in a different container, you have to specify a different container flag.

## Options

| Option              | Description                                           | Possible Values |
| ------------------- | ----------------------------------------------------- | --------------- |
| `-p`, `--postgres`  | Run a bash command in the metadata database container | None            |
| `-s`,`--scheduler`  | Run a bash command in the scheduler container         | None            |
| `-t`, `--triggerer` | Run a bash command in the triggerer container         | None            |
| `-w`, `--webserver` | Run a bash command in the webserver container         | None            |

## Examples

```sh wrap theme={null}
$ astro dev bash --webserver
$ ls -al
# View all files in the webserver container

$ astro dev bash --scheduler
$ pip-freeze | grep pymongo
# Check the version of the pymongo package running in the scheduler
```

## Related commands

* [`astro dev start`](/docs/cli/v1.45/astro-dev-start)
* [`astro dev run`](/docs/cli/v1.45/astro-dev-run)
* [`astro dev ps`](/docs/cli/v1.45/astro-dev-ps)


# astro dev build
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-build

Build your Astro project into a Docker image.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Build your Astro project into a Docker image without starting a local Airflow environment. Use this command to verify that your project builds successfully before deployment or when you need only the image build step.

This command is not available in standalone mode.

## Usage

```bash wrap theme={null}
astro dev build
```

## Options

| Option               | Description                                                                                                                                                              | Possible Values                                                                                                                          |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `--build-secret`     | Run `docker build --secret` to mount a secret value to your Docker image. Repeat the flag to mount more than one secret. Replaces the deprecated `--build-secrets` flag. | `id=<your-secret-id>, src=<path-to-secret> .` See [Docker documentation](https://docs.docker.com/build/building/secrets/#secret-mounts). |
| `-i`, `--image-name` | Tag a pre-built custom image as the project image.                                                                                                                       | A valid Docker image name                                                                                                                |
| `--no-cache`         | Do not use cache when building your Astro project into a Docker image.                                                                                                   | None                                                                                                                                     |

## Examples

```bash wrap theme={null}
$ astro dev build
# Build the project image

$ astro dev build --no-cache
# Build the project image without using Docker cache

$ astro dev build --image-name my-custom-image:latest
# Tag a pre-built image as the project image
```

## Related commands

* [`astro dev start`](/docs/cli/v1.45/astro-dev-start)
* [`astro deploy`](/docs/cli/v1.45/astro-deploy)


# astro dev init
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-init

Initialize an Astro project.

<Info>
  The behavior and format of this command are the same for both Astro and Astro
  Private Cloud.
</Info>

Initialize an [Astro project](/docs/cli/v1.45/develop-project#create-an-astro-project) in an empty local directory. An Astro project contains the set of files necessary to run Airflow, including dedicated folders for your dag files, plugins, and dependencies. An Astro project can be either run locally with `astro dev start` or pushed to Astronomer with `astro deploy`.

<Tabs>
  <Tab title="Astro">
    ### Usage

    ```sh wrap theme={null}
    astro dev init <project-name>
    ```

    ### Options

    | Option                       | Description                                                                                                                                                                            | Possible Values                    |
    | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
    | `<project-name>`             | Optional name for your Astro project. Creates and initializes a directory with the specified name.                                                                                     | Any string                         |
    | `-a`, `--airflow-version`    | Version of Airflow you want to create an Astro project with. If not specified, latest is assumed. You can change this version in your Dockerfile at any time.                          | String (Airflow version)           |
    | `-v`, `--runtime-version`    | Astro Runtime version to use for project initialization.                                                                                                                               | Any supported Runtime version      |
    | `-n`, `--name`               | Name for your Astro project.                                                                                                                                                           | Any string                         |
    | `-f`, `--force`              | Initialize the project without confirmation, even in a non-empty directory.                                                                                                            | None                               |
    | `--from-template`            | Specify a getting started [template](https://github.com/astronomer/templates) to use as the base for your Astro project. Opens an interactive menu to select from available templates. | N/A (interactive selection)        |
    | `--remote-execution-enabled` | Enable Remote Execution mode. Prompts for remote Docker repository, generates client image files and config.                                                                           | N/A                                |
    | `--remote-image-repository`  | Provide the remote image repository for client images. If omitted, prompts interactively with a warning if unset.                                                                      | Remote Docker image repository URI |

    ### Examples

    ```sh wrap theme={null}
    $ astro dev init
    # Initializes default project

    $ astro dev init my-airflow-project
    # Creates and initializes a directory named 'my-airflow-project'

    $ astro dev init --name=MyProject
    # Generates `config.yaml` file with `name=MyProject`

    $ astro dev init --airflow-version=2.10.5
    # Initializes project with Airflow 2.10.5

    $ astro dev init --airflow-version=3.1.3
    # Initializes project with Airflow 3.1.3

    $ astro dev init --runtime-version=13.2.0
    # Initializes project with Runtime 13.2.0

    $ astro dev init --runtime-version=3.1-5
    # Initializes project with Runtime 3.1-5

    $ astro dev init --from-template
    # Initializes project with an interactive template selection menu
    ```

    ## Remote Execution Mode

    Enabling remote execution with the `--remote-execution-enabled` flag results in the following behavior:

    * The CLI prompts you to provide a remote client image repository, unless you specify it with `--remote-image-repository`.
    * If you do not provide a repository, you receive a warning. You can also configure the remote image repository later via the `astro config set remote.client_registry <remote-image-repository>` CLI command
    * A new entry in `.astro/config.yaml` is created or updated with your remote client image repository.
    * These new files are generated for customizing dependencies and build steps for remote execution client images:
      * `Dockerfile.client`
      * `requirements-client.txt`
      * `packages-client.txt`

    ### Examples

    ```sh wrap theme={null}
    $ astro dev init --remote-execution-enabled
    # Initialize with remote execution and prompt for repo

    $ astro dev init --remote-execution-enabled --remote-image-repository=images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-4-python-3.12-astro-agent-1.2.1
    # Initialize with remote execution using vanilla Astronomer agent image

    $ astro dev init --remote-execution-enabled --remote-image-repository=123456789012.dkr.ecr.us-east-1.amazonaws.com/remote-execution-custom-agent:1.0.0
    # Initialize with remote execution using AWS ECR

    $ astro dev init --remote-execution-enabled --remote-image-repository=nexus.company.com:8083/remote-execution-custom-agent:1.0.0
    # Initialize with remote execution using Nexus registry

    ```
  </Tab>

  <Tab title="APC">
    ## Usage

    ```sh wrap theme={null}
    astro dev init <project-name>
    ```

    ## Options

    | Option                    | Description                                                                                                                                                                            | Possible Values               |
    | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- |
    | `<project-name>`          | Optional name for your Astro project. Creates and initializes a directory with the specified name.                                                                                     | Any string                    |
    | `-a`, `--airflow-version` | Version of Airflow you want to create an Astro project with. If not specified, latest is assumed. You can change this version in your Dockerfile at any time.                          | String (Airflow version)      |
    | `-v`, `--runtime-version` | Astro Runtime version to use for project initialization.                                                                                                                               | Any supported Runtime version |
    | `-n`, `--name`            | Name for your Astro project.                                                                                                                                                           | Any string                    |
    | `-f`, `--force`           | Initialize the project without confirmation, even in a non-empty directory.                                                                                                            | None                          |
    | `--from-template`         | Specify a getting started [template](https://github.com/astronomer/templates) to use as the base for your Astro project. Opens an interactive menu to select from available templates. | N/A (interactive selection)   |

    ## Examples

    ```sh wrap theme={null}
    $ astro dev init
    # Initializes default project

    $ astro dev init my-airflow-project
    # Creates and initializes a directory named 'my-airflow-project'

    $ astro dev init --name=MyProject
    # Generates `config.yaml` file with `name=MyProject`

    $ astro dev init --airflow-version=2.10.5
    # Initializes project with Airflow 2.10.5

    $ astro dev init --airflow-version=3.1.3
    # Initializes project with Airflow 3.1.3

    $ astro dev init --runtime-version=13.2.0
    # Initializes project with Runtime 13.2.0 (Airflow 2.x)

    $ astro dev init --runtime-version=3.1-5
    # Initializes project with Runtime 3.1-5 (Airflow 3.x)

    $ astro dev init --from-template
    # Initializes project with an interactive template selection menu
    ```
  </Tab>
</Tabs>

## Related commands

* [`astro dev restart`](/docs/cli/v1.45/astro-dev-restart)
* [`astro dev run`](/docs/cli/v1.45/astro-dev-run)
* [`astro dev logs`](/docs/cli/v1.45/astro-dev-logs)


# astro dev kill
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-kill

Force-stop and remove all locally running Airflow containers.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Force-stop and remove all running containers for your local Airflow environment. Unlike [`astro dev stop`](/docs/cli/v1.45/astro-dev-stop), which only pauses running containers, `astro dev kill` deletes all data associated with your local Postgres database. This includes Airflow connections, logs, and task history.

For more information, read [Hard reset your local environment](/docs/cli/v1.45/run-airflow-locally#hard-reset-your-local-environment) or [Build and run a project locally](/docs/cli/v1.45/run-airflow-locally).

## Usage

```sh wrap theme={null}
astro dev kill
```

<Note>
  In standalone mode, this command stops the Airflow processes and removes the virtual environment, database, and logs. If you started your environment with `astro dev start --standalone` without setting `dev.mode` to `standalone`, pass `--standalone` so this command resets your standalone environment.
</Note>

## Related commands

* [`astro dev start`](/docs/cli/v1.45/astro-dev-start)
* [`astro dev stop`](/docs/cli/v1.45/astro-dev-stop)


# astro dev logs
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-logs

Show logs for Airflow components.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Show webserver, scheduler, and triggerer logs from your local Airflow environment.

## Usage

```sh wrap theme={null}
astro dev logs
```

## Options

| Option             | Description                                                                     | Possible Values |
| ------------------ | ------------------------------------------------------------------------------- | --------------- |
| `--api-server`     | Show only API server logs. In standalone mode, `--webserver` maps to this flag. | None            |
| `--dag-processor`  | Show only dag processor logs.                                                   | None            |
| `-f`,`--follow`    | Continue streaming most recent log output to your terminal.                     | None            |
| `-s`,`--scheduler` | Show only scheduler logs.                                                       | None            |
| `-t`,`--triggerer` | Show only triggerer logs.                                                       | None            |
| `-w`,`--webserver` | Show only webserver logs. In standalone mode, this maps to API server logs.     | None            |

## Examples

```sh wrap theme={null}
$ astro dev logs
# Show the most recent logs from both the Airflow webserver and Scheduler

$ astro dev logs --follow
# Stream all new webserver and scheduler logs to the terminal

$ astro dev logs --follow --scheduler
# Stream only new scheduler logs to the terminal
```

## Related commands

* [`astro dev ps`](/docs/cli/v1.45/astro-dev-ps)
* [`astro dev run`](/docs/cli/v1.45/astro-dev-run)


# astro dev object export
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-object-export

Export Airflow objects from a local Airflow environment.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Export Airflow variables, connections, and pools from a locally running environment to a local file and format of your choice. By default, the command exports all Airflow objects to the `airflow_settings.yaml` file in your Astro project.

## Usage

After starting your local Airflow environment with `astro dev start`, run:

```sh wrap theme={null}
astro dev object export
```

By default, the command exports all variables, connections, and pools as YAML configurations to `airflow_settings.yaml`.

## Options

| Option                 | Description                                                                                                                                                 | Possible Values    |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| `--compose`            | Export the Docker Compose file used to start Airflow locally.                                                                                               |                    |
| `--compose-file`       | The location Docker Compose file to export. The default is `compose.yaml`.                                                                                  | Any valid filepath |
| `-c`,`--connections`   | Export connections to a given local file                                                                                                                    | \`\`               |
| `-e`,`--env`           | Location of the file to export Airflow objects to as Astro environment variables. Use this option only with `--env-export`. The default file path is `env`. | Any valid filepath |
| `-n`,`--env-export`    | Export Airflow objects as Astro environment variables.                                                                                                      | \`\`               |
| `-p`,`--pools`         | Export pools to a given local file                                                                                                                          | \`\`               |
| `-s`,`--settings-file` | Location of the file to export Airflow objects to as YAML configuration. The default file path is `airflow_settings.yaml`.                                  | Any valid filepath |
| `-v`,`--variables`     | Export variables to a given local file                                                                                                                      | \`\`               |

## Examples

```sh wrap theme={null}
astro dev object export --pools
# Exports only pools from the local Airflow environment to `airflow_settings.yaml`

astro dev object export --env-export --env="myairflowenv.env"
# Exports all Airflow objects from the local Airflow environment as
# Astro variables to a file in the project named `myairflowenv.env`
```

## Related commands

* [`astro dev object import`](/docs/cli/v1.45/astro-dev-object-import)
* [`astro deployment variable create`](/docs/cli/v1.45/astro-deployment-variable-create)
* [`astro deployment variable update`](/docs/cli/v1.45/astro-deployment-variable-update)


# astro dev object import
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-object-import

Import Airflow objects from a configuration file.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Import Airflow variables, connections, and pools from a configuration file to a locally running Airflow environment.

## Usage

After starting your local Airflow environment with `astro dev start`, run:

```sh wrap theme={null}
astro dev object import
```

By default, the command imports all variables, connections, and pools from `airflow_settings.yaml` to your project. You do not need to restart your environment for these changes to take effect.

## Options

| Option                 | Description                                                                                                     | Possible Values    |
| ---------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------ |
| `-c`,`--connections`   | Import connections from a given local file                                                                      | \`\`               |
| `-p`,`--pools`         | Import pools from a given local file                                                                            | \`\`               |
| `-s`,`--settings-file` | Location of the file from which to import Airflow objects. The default file path is `~/.airflow_settings.yaml`. | Any valid filepath |
| `-v`,`--variables`     | Import variables from a given local file                                                                        | \`\`               |

## Examples

```sh wrap theme={null}
astro dev object import --pools
# Imports pools from `airflow_settings.yaml` to a locally running Airflow environment

astro dev object import --settings-file="myairflowobjects.yaml"
# Imports all Airflow objects from `myairflowobjects.yaml` to a locally running Airflow environment
```

## Related commands

* [`astro dev object export`](/docs/cli/v1.45/astro-dev-object-export)
* [`astro deployment variable create`](/docs/cli/v1.45/astro-deployment-variable-create)
* [`astro deployment variable update`](/docs/cli/v1.45/astro-deployment-variable-update)


# astro dev parse
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-parse

Parse the dags to check for errors in a local Airflow environment.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Parse the dags in a locally hosted Astro project to quickly check them for errors. Parse tests are defined in the `.astro/test_dag_integrity_default.py` file of your Astro project. For more information about testing dags locally, read [Test your Astro project locally](/docs/cli/v1.45/test-your-astro-project-locally).

## Usage

```bash wrap theme={null}
astro dev parse
```

## Options

| Option               | Description                                                                                                                                                              | Possible Values                                                                                                                          |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `--build-secret`     | Run `docker build --secret` to mount a secret value to your Docker image. Repeat the flag to mount more than one secret. Replaces the deprecated `--build-secrets` flag. | `id=<your-secret-id>, src=<path-to-secret> .` See [Docker documentation](https://docs.docker.com/build/building/secrets/#secret-mounts). |
| `-e`, `--env`        | The filepath to your environment variables. (The default is `.env`)                                                                                                      | Any valid filepath within your Astro project                                                                                             |
| `-i`, `--image-name` | The name of a pre-built custom Docker image to use with your project. The image must be available on your local machine.                                                 | A valid name for a pre-built Docker image based on Astro Runtime                                                                         |

## Examples

```bash wrap theme={null}
# Parse dags
astro dev parse

# Specify alternative environment variables
astro dev parse --env=myAlternativeEnvFile.env
```

## Related commands

* [`astro dev pytest`](/docs/cli/v1.45/astro-dev-pytest)
* [`astro dev start`](/docs/cli/v1.45/astro-dev-start)
* [`astro deploy`](/docs/cli/v1.45/astro-deploy)


# astro dev proxy
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-proxy

Manage the built-in reverse proxy for local Airflow development.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Manage the built-in reverse proxy for local Airflow development.

The built-in reverse proxy routes `<project>.localhost:6563` to the correct local Airflow instance, letting you run multiple Airflow projects simultaneously without manually configuring ports. The proxy starts automatically when you run `astro dev start` and stops when the last project stops.

When working in a git worktree, the hostname includes both the worktree name and the repository name: `<worktree>.<repo>.localhost:6563`.

To disable the proxy for a single start command, use `astro dev start --no-proxy`.

## astro dev proxy status

Show all running Airflow projects and their proxy routes.

```bash wrap theme={null}
astro dev proxy status
```

If the proxy daemon has stopped unexpectedly, this command automatically restarts it and restores all routes from `~/.astro/proxy/routes.json`.

## astro dev proxy stop

Stop the reverse proxy daemon.

```bash wrap theme={null}
astro dev proxy stop
```

## Related commands

* [`astro dev start`](/docs/cli/v1.45/astro-dev-start)
* [`astro dev stop`](/docs/cli/v1.45/astro-dev-stop)


# astro dev ps
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-ps

List all running Docker containers in your local Airflow environment.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

List all Docker containers running in your local Airflow environment, including the Airflow Webserver, Scheduler, and Postgres database. It outputs metadata for each running container, including `Container ID`, `Created`, `Status`, and `Ports`.

This command works similarly to [`docker ps`](https://docs.docker.com/engine/reference/commandline/ps/) and can only be run from a directory that is running an Astro project.

## Usage

```sh wrap theme={null}
astro dev ps
```

## Options

| Option          | Description                                                                                          | Possible Values                |
| --------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------ |
| `--json`        | Output the response as JSON. Shorthand for `--output json`. Mutually exclusive with `-o`/`--output`. | None                           |
| `-o`,`--output` | Format the output. The default is `table`.                                                           | `table`, `json`, or `template` |
| `--template`    | Format the output using a Go template. Use with `--output template`.                                 | Any valid Go template string   |

## Related commands

* [`astro dev logs`](/docs/cli/v1.45/astro-dev-logs)
* [`astro dev run`](/docs/cli/v1.45/astro-dev-run)


# astro dev pytest
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-pytest

Run unit tests with pytest for dags in a local Airflow environment.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Run unit tests for your data pipelines on Astro with `pytest`, a testing framework for Python. When you run this command, the Astro CLI creates a local Python environment that includes your dag code, dependencies, and Astro Runtime Docker image. The CLI then runs any pytests in the `tests` directory of your Astro project and shows you the results of those tests in your terminal.

The command runs `pytest` in a container. If your test generates artifacts, such as code coverage reports, you can output the artifacts to the `include` folder of your Astro project so they can be accessed after the test has finished.

For more information on this functionality, see [Test your Astro project locally](/docs/cli/v1.45/test-your-astro-project-locally).

<Info>This command requires Astro Runtime version `3.1.1`+. For more information, see [Astro Runtime Release Notes](/docs/runtime/runtime-release-notes#astro-runtime-3-1-1).</Info>

## Usage

```sh wrap theme={null}
astro dev pytest
```

## Options

| Option               | Description                                                                                                                                                              | Possible Values                                                                                                                          |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `<pytest-filepath>`  | The filepath to an alternative pytest file or directory. Must be within the `tests` directory                                                                            | Any valid filepath within the `tests` directory                                                                                          |
| `-a`, `--args`       | Arguments to pass to pytest. Surround the args in quotes.                                                                                                                | Any set of pytest command arguments surrounded by quotes                                                                                 |
| `--build-secret`     | Run `docker build --secret` to mount a secret value to your Docker image. Repeat the flag to mount more than one secret. Replaces the deprecated `--build-secrets` flag. | `id=<your-secret-id>, src=<path-to-secret> .` See [Docker documentation](https://docs.docker.com/build/building/secrets/#secret-mounts). |
| `-e`, `--env`        | The filepath to your environment variables. The default is `.env`)                                                                                                       | Any valid filepath within your Astro project                                                                                             |
| `-i`, `--image-name` | The name of a pre-built custom Docker image to use with your project. The image must be available on your local machine.                                                 | A valid name for a pre-built Docker image based on Astro Runtime                                                                         |

## Examples

```bash wrap theme={null}
# Specify env file at root of Astro project
astro dev pytest --env=myAlternativeEnvFile.env

# Specify an argument for pytest
astro dev pytest --args "–-cov-config path"

# Generate a coverage report in the include/coverage.xml file
astro dev pytest --args "--cov --cov-report xml:include/coverage.xml"
```

## Related commands

* [`astro dev init`](/docs/cli/v1.45/astro-dev-init)
* [`astro dev start`](/docs/cli/v1.45/astro-dev-start)
* [`astro deploy`](/docs/cli/v1.45/astro-deploy)


# astro dev restart
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-restart

Restart a local Airflow environment.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Stop your Airflow environment, rebuild your Astro project into a Docker image, and restart your Airflow environment with the new Docker image.

This command can be used to rebuild an Astro project and run it locally. For more information, read [Build and run a project locally](/docs/cli/v1.45/run-airflow-locally).

## Usage

```sh wrap theme={null}
astro dev restart
```

<Note>
  In standalone mode, this command recreates the virtual environment instead of rebuilding a Docker image. If you started your environment with `astro dev start --standalone` without setting `dev.mode` to `standalone`, pass `--standalone` so this command restarts your standalone environment.
</Note>

## Options

| Option               | Description                                                                                                                                                              | Possible Values                                                                                                                          |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `--build-secret`     | Run `docker build --secret` to mount a secret value to your Docker image. Repeat the flag to mount more than one secret. Replaces the deprecated `--build-secrets` flag. | `id=<your-secret-id>, src=<path-to-secret> .` See [Docker documentation](https://docs.docker.com/build/building/secrets/#secret-mounts). |
| `-e`,`--env`         | Path to your environment variable file. Default is `.env`                                                                                                                | Valid filepaths                                                                                                                          |
| `-i`, `--image-name` | The name of a pre-built custom Docker image to use with your project. The image must be available on your local machine.                                                 | A valid name for a pre-built Docker image based on Astro Runtime                                                                         |
| `-k`, `--kill`       | Kill all running containers and remove all data before restarting. Equivalent to running `astro dev kill` followed by `astro dev start`.                                 | None                                                                                                                                     |

## Examples

```sh wrap theme={null}
$ astro dev restart --env=/users/username/documents/myfile.env
```

## Related commands

* [`astro dev start`](/docs/cli/v1.45/astro-dev-start)
* [`astro dev stop`](/docs/cli/v1.45/astro-dev-stop)
* [`astro dev kill`](/docs/cli/v1.45/astro-dev-kill)
* [`astro dev init`](/docs/cli/v1.45/astro-dev-init)
* [`astro dev run`](/docs/cli/v1.45/astro-dev-run)
* [`astro dev logs`](/docs/cli/v1.45/astro-dev-logs)


# astro dev run
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-run

Run Airflow CLI commands in a local Airflow environment.

Run [Airflow CLI commands](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html) in your local Airflow environment. This command is for local development only and cannot be applied to Deployments running on Astro.

## Usage

```sh wrap theme={null}
astro dev run <airflow-command>
```

## Examples

```sh wrap theme={null}
$ astro dev run users create --role Admin --username admin --email <your-email-address> --firstname <your-first-name> --lastname <your-last-name> --password admin
# Create a user in your local Airflow environment using the `airflow user create` Airflow CLI command

$ astro dev run connections export - --file-format=env --serialization-format=json
# Export connections in your local Airflow environment to STDOUT in a JSON format

$ astro dev run connections export - --file-format=env
# Export connections in your local Airflow environment to STDOUT in the default URI format
```

## Related commands

* [`astro dev logs`](/docs/cli/v1.45/astro-dev-logs)
* [`astro dev ps`](/docs/cli/v1.45/astro-dev-ps)


# astro dev start
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-start

Build your Astro project and start a local Airflow environment.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Start a local Airflow environment in Docker mode (default) or [standalone mode](#standalone-mode) (without Docker).

For more information, see [Build and run a project locally](/docs/cli/v1.45/run-airflow-locally).

## Usage

```sh wrap theme={null}
astro dev start
```

## Options

| Option                  | Description                                                                                                                                                                                                                                                                                                                                                                                                                        | Possible Values                                                                                                                          |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `--build-secret`        | Run `docker build --secret` to mount a secret value to your Docker image. Repeat the flag to mount more than one secret. Docker mode only. Replaces the deprecated `--build-secrets` flag.                                                                                                                                                                                                                                         | `id=<your-secret-id>, src=<path-to-secret> .` See [Docker documentation](https://docs.docker.com/build/building/secrets/#secret-mounts). |
| `--compose-file`        | The location of a custom Docker Compose file to use for starting Airflow on Docker. Docker mode only.                                                                                                                                                                                                                                                                                                                              | Any valid filepath                                                                                                                       |
| `--deployment-id`       | Specifies a Deployment whose Environment Manager configurations you want to use locally. When Airflow builds locally, Astro populates the Airflow metadata database with the Airflow objects specified from the Deployment Environment Manager in the Astro UI. Local development access to connections must be enabled first. See [Use Airflow connections hosted on Astro in a local environment](/docs/cli/v1.45/local-connections). | Any valid Deployment ID                                                                                                                  |
| `--docker`              | Run in Docker mode. Use to override the `dev.mode` project config for a single command.                                                                                                                                                                                                                                                                                                                                            | None                                                                                                                                     |
| `-e`,`--env`            | Path to your environment variable file. Default is `.env`                                                                                                                                                                                                                                                                                                                                                                          | Valid filepaths                                                                                                                          |
| `-f`, `--foreground`    | Run Airflow in the foreground instead of as a background process. Standalone mode only.                                                                                                                                                                                                                                                                                                                                            | None                                                                                                                                     |
| `-i`, `--image-name`    | The name of a pre-built custom Docker image to use with your project. The image must be available on your local machine. Docker mode only.                                                                                                                                                                                                                                                                                         | A valid name for a pre-built Docker image based on Astro Runtime                                                                         |
| `-n`, `--no-browser`    | Starts a local Airflow environment without opening a web browser for the Airflow UI. Docker mode only.                                                                                                                                                                                                                                                                                                                             | None                                                                                                                                     |
| `--no-cache`            | Do not use cache when building your Astro project into a Docker image. Docker mode only.                                                                                                                                                                                                                                                                                                                                           | None                                                                                                                                     |
| `--no-proxy`            | Disable the built-in reverse proxy and use classic fixed-port behavior.                                                                                                                                                                                                                                                                                                                                                            | None                                                                                                                                     |
| `-p`, `--port`          | Port for the Airflow API server. Standalone mode only.                                                                                                                                                                                                                                                                                                                                                                             | Any valid port number                                                                                                                    |
| `-s`, `--settings-file` | Settings file from which to import Airflow objects. Default is `airflow_settings.yaml`.                                                                                                                                                                                                                                                                                                                                            | Any valid path to an Airflow settings file                                                                                               |
| `--standalone`          | Run in standalone mode without Docker. Use to override the `dev.mode` project config for a single command. See [Standalone mode](#standalone-mode).                                                                                                                                                                                                                                                                                | None                                                                                                                                     |
| `--wait`                | Amount of time to wait for the webserver to get healthy before timing out. The default is 1 minute for most machines and 5 minutes for Apple M1 machines.                                                                                                                                                                                                                                                                          | Time in minutes defined as `<integer>m` and time in seconds defined as `<integer>s`                                                      |
| `-workspace-id`         | Specifies a Workspace whose Environment Manager configurations you want to use locally. When Airflow builds locally, Astro populates the Airflow metadata database with the Airflow objects to all Deployments in the Workspace.                                                                                                                                                                                                   | Any valid Workspace ID                                                                                                                   |

## Examples

```sh wrap theme={null}
$ astro dev start --env=/users/username/documents/myfile.env
```

<Info>
  The following error can sometimes occur when the CLI tries to build your Astro Runtime image using Podman:

  ```bash wrap theme={null}
  WARN[0010] SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format
  ```

  You can resolve this issue by exporting the `BUILDAH_FORMAT` [environment variable](/docs/astro/environment-variables) to Podman:

  ```dockerfile wrap theme={null}
  export BUILDAH_FORMAT=docker
  ```
</Info>

## Standalone mode

Standalone mode runs Airflow directly on your machine in a virtual environment, without Docker. Use it when Docker isn't available or when you want a lighter-weight local setup.

To set standalone as the default mode for a project:

```bash wrap theme={null}
astro config set dev.mode standalone
```

To use standalone mode for a single command:

```bash wrap theme={null}
astro dev start --standalone
```

The `--standalone` flag applies to a single command. If you start your environment with the flag instead of setting `dev.mode`, you must also pass `--standalone` to `astro dev stop`, `astro dev restart`, and `astro dev kill`. Without the flag, these commands run in Docker mode and don't act on your standalone environment.

To switch back to Docker mode:

```bash wrap theme={null}
astro config set dev.mode docker
```

The following commands are not available in standalone mode:

* `astro dev build`
* `astro dev upgrade-test`
* `astro dev compose-export`

<Note>
  `astro dev run --standalone` is not supported. To run Airflow CLI commands in standalone mode, set standalone as the default mode first:

  ```bash wrap theme={null}
  astro config set dev.mode standalone
  astro dev run dags list
  ```
</Note>

### Dockerfile handling in standalone mode

Standalone mode runs Airflow in a virtual environment instead of building a Docker image, so it doesn't run the build instructions in your `Dockerfile`. It ignores instructions such as `RUN` and `COPY`. If your project relies on these instructions to install system packages or add files to the image, adjust your project before you run it in standalone mode.

Standalone mode reads the `FROM` instruction to determine your Astro Runtime version, and therefore the Airflow version, for your environment. It supports both Airflow 2 (Astro Runtime 13.x) and Airflow 3 (Astro Runtime 3.x) projects.

## Related commands

* [`astro dev restart`](/docs/cli/v1.45/astro-dev-restart)
* [`astro dev stop`](/docs/cli/v1.45/astro-dev-stop)
* [`astro dev kill`](/docs/cli/v1.45/astro-dev-kill)
* [`astro dev init`](/docs/cli/v1.45/astro-dev-init)
* [`astro dev run`](/docs/cli/v1.45/astro-dev-run)
* [`astro dev logs`](/docs/cli/v1.45/astro-dev-logs)


# astro dev stop
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-stop

Pause all local Airflow Docker containers.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Pause all Docker containers running your local Airflow environment. Unlike `astro dev kill`, this command does not prune mounted volumes and delete data associated with your local Postgres database. If you run this command, Airflow connections and task history will be preserved.

This command can be used regularly with `astro dev start` to apply changes to your Astro project as you test and troubleshoot dags. For more information, read [Build and run a project locally](/docs/cli/v1.45/run-airflow-locally).

## Usage

```sh wrap theme={null}
astro dev stop
```

<Note>
  In standalone mode, this command stops the Airflow processes running on your machine. If you started your environment with `astro dev start --standalone` without setting `dev.mode` to `standalone`, pass `--standalone` so this command stops your standalone environment:

  ```bash wrap theme={null}
  astro dev stop --standalone
  ```
</Note>

## Related commands

* [`astro dev start`](/docs/cli/v1.45/astro-dev-start)
* [`astro dev restart`](/docs/cli/v1.45/astro-dev-restart)
* [`astro dev kill`](/docs/cli/v1.45/astro-dev-kill)


# astro dev upgrade-test
Source: https://astronomer.io/docs/cli/v1.45/astro-dev-upgrade-test

Test your Astro project before upgrading to a new Astro Runtime version.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Test your local Astro project against a new version of Astro Runtime to prepare for an upgrade. Specifically, this command will run the following tests:

* Identify major and minor version changes of the Python packages in your upgrade version.
* Identify dag import errors that will appear after you upgrade.

See [Test before upgrading your Astro project](/docs/cli/v1.45/test-your-astro-project-locally) for more detailed information about usage and test results.

## Usage

```bash wrap theme={null}
astro dev upgrade-test
```

By default, the command runs all three available tests on your project against the latest version of Astro Runtime.

## Options

| Option                       | Description                                                                                                                                                                                                                                     | Possible Values                                                                                                                          |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `-a`, `--airflow-version`    | The equivalent of Airflow you want to upgrade to. The default is the latest available version. Note that the Astro CLI will still test against an Astro Runtime image based on the Airflow version you specify.                                 | Any valid [Airflow version](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html).                                   |
| `--build-secret`             | Run `docker build --secret` to mount a secret value to your Docker image. Repeat the flag to mount more than one secret. Replaces the deprecated `--build-secrets` flag.                                                                        | `id=<your-secret-id>, src=<path-to-secret> .` See [Docker documentation](https://docs.docker.com/build/building/secrets/#secret-mounts). |
| `-d`, `--dag-test`           | Only run dag tests. These tests check whether your dags will generate import errors after you upgrade.                                                                                                                                          | None                                                                                                                                     |
| `-i`, `--deployment-id`      | Specify a Deployment ID to test with an image from an Astro Deployment instead of the image listed in your Astro project Dockerfile.                                                                                                            | Any valid Deployment ID.                                                                                                                 |
| `-n`, `--image-name`         | A custom image with an upgraded image tag to test against. The CLI creates a new Dockerfile in your project named `upgrade-test-<old version>--<new version>/Dockerfile` that includes your upgraded image, then tests against that Dockerfile. | A valid image URL                                                                                                                        |
| `-v`, `--runtime-version`    | The version of Astro Runtime you want to upgrade to. The default is the latest available version.                                                                                                                                               | Any valid [Astro runtime version](/docs/runtime/runtime-release-notes).                                                                       |
| `--use-astronomer-certified` | Test against an Astronomer Certified distribution of Airflow. Must be used with `--airflow-version`.                                                                                                                                            | None                                                                                                                                     |
| `--version-test`             | Only run version tests. These tests show you how the versions of your dependencies will change after you upgrade.                                                                                                                               | None                                                                                                                                     |
| `--fix`                      | Automatically fix linting issues identified by the upgrade test using ruff.                                                                                                                                                                     | None                                                                                                                                     |
| `--lint-deprecations`        | Include deprecation warnings in lint tests (set to false by default).                                                                                                                                                                           | true, false                                                                                                                              |

## Examples

* Run all tests before upgrading to the latest version of Astro Runtime:

  ```bash wrap theme={null}
  astro dev upgrade-test
  ```

* Test an Astro project file against a the Astro Runtime distribution of a Airflow 2.6.3:

  ```bash wrap theme={null}
  astro dev upgrade-test --airflow-version 2.6.3
  ```

* Include deprecation warnings in lint tests (exports to `ruff-lint-results.txt`):
  ```bash wrap theme={null}
  astro dev upgrade-test --lint-deprecations=true
  ```

* Test only dependency version changes against the Astro Runtime distribution for Airflow 2.6.3:

  ```bash wrap theme={null}
  astro dev upgrade-test --airflow-version 2.6.3 --provider-check
  ```

* Test a custom image for an upgrade to a custom versioned distribution of Astro Runtime:

  ```bash wrap theme={null}
  astro dev upgrade-test --image-name quay.io/example-organization/new-example-organization-image:1.0.7
  ```

  If you were upgrading from `quay.io/example-organization/new-example-organization-image:1.0.5`, for example, the Astro CLI creates a new Dockerfile located in `upgrade-test-1.0.5--1.0.7/Dockerfile` that contains the image you specified and tests against it.

## Related commands

* [`astro dev pytest`](/docs/cli/v1.45/astro-dev-pytest)
* [`astro dev parse`](/docs/cli/v1.45/astro-dev-parse)


# astro env
Source: https://astronomer.io/docs/cli/v1.45/astro-env

Manage Astro Environment Manager objects from the Astro CLI.

<Info>
  This command is only available on Astro.
</Info>

Use `astro env` commands to manage Environment Manager objects from your terminal. These objects include environment variables, Airflow connections, Airflow variables, and metrics exports. You can scope each object to a Workspace or a Deployment, and you can link a Workspace-scoped object to specific Deployments.

These commands are distinct from [`astro deployment variable`](/docs/cli/v1.45/astro-deployment-variable-create), which writes directly to a single Deployment, and [`astro deployment connection`](/docs/cli/v1.45/astro-deployment-connection-create), which writes to a Deployment's Airflow metadata database. Use `astro env` for objects that you want to share across Deployments or manage at the Workspace level.

<CardGroup>
  <Card title="astro env variable" href="/cli/v1.45/astro-env-variable">
    View documentation for `astro env variable`.
  </Card>

  <Card title="astro env connection" href="/cli/v1.45/astro-env-connection">
    View documentation for `astro env connection`.
  </Card>

  <Card title="astro env airflow-variable" href="/cli/v1.45/astro-env-airflow-variable">
    View documentation for `astro env airflow-variable`.
  </Card>

  <Card title="astro env metrics-export" href="/cli/v1.45/astro-env-metrics-export">
    View documentation for `astro env metrics-export`.
  </Card>
</CardGroup>


# astro env airflow-variable
Source: https://astronomer.io/docs/cli/v1.45/astro-env-airflow-variable

Manage Environment Manager Airflow variables from the Astro CLI.

<Info>
  This command is only available on Astro.
</Info>

Use `astro env airflow-variable` commands to list, create, update, and delete Airflow variables managed through the Astro Environment Manager. You can scope an Airflow variable to a Workspace or a Deployment, and you can link a Workspace-scoped variable to specific Deployments.

These commands are distinct from [`astro deployment airflow-variable`](/docs/cli/v1.45/astro-deployment-airflow-variable-create), which writes directly to a single Deployment's Airflow metadata database.

<CardGroup>
  <Card title="astro env airflow-variable create" href="/cli/v1.45/astro-env-airflow-variable-create">
    View documentation for `astro env airflow-variable create`.
  </Card>

  <Card title="astro env airflow-variable delete" href="/cli/v1.45/astro-env-airflow-variable-delete">
    View documentation for `astro env airflow-variable delete`.
  </Card>

  <Card title="astro env airflow-variable get" href="/cli/v1.45/astro-env-airflow-variable-get">
    View documentation for `astro env airflow-variable get`.
  </Card>

  <Card title="astro env airflow-variable list" href="/cli/v1.45/astro-env-airflow-variable-list">
    View documentation for `astro env airflow-variable list`.
  </Card>

  <Card title="astro env airflow-variable update" href="/cli/v1.45/astro-env-airflow-variable-update">
    View documentation for `astro env airflow-variable update`.
  </Card>
</CardGroup>


# astro env airflow-variable create
Source: https://astronomer.io/docs/cli/v1.45/astro-env-airflow-variable-create

Create one or more Environment Manager Airflow variables.

<Info>
  This command is only available on Astro.
</Info>

Create a single Airflow variable with `--key` and `--value`, or bulk-create variables from a dotenv file with `--from-file`. When you use `--from-file`, the `--secret` and `--auto-link` flags apply to every entry in the file.

<Note>Bulk import with `--from-file` supports POSIX-style keys only. To create an Airflow variable whose key isn't POSIX-style, such as `team.config` or `dag-name`, use `--key` and `--value` instead.</Note>

## Usage

```bash wrap theme={null}
astro env airflow-variable create
```

## Options

| Option            | Description                                                                                                                                         | Possible Values             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| `-k`,`--key`      | The variable key. Required unless you use `--from-file`.                                                                                            | Any string                  |
| `-v`,`--value`    | The variable value. If omitted, the CLI reads the value from stdin when piped, or prompts for it with the input hidden.                             | Any string                  |
| `-s`,`--secret`   | Mark the variable as secret.                                                                                                                        | None                        |
| `--from-file`     | Bulk-create variables from a dotenv file with one `KEY=VALUE` per line. Pass `-` to read from stdin. Mutually exclusive with `--key` and `--value`. | Any valid file path, or `-` |
| `--auto-link`     | Workspace scope only. Automatically link the variable to all Deployments in the Workspace, including future ones.                                   | None                        |
| `--workspace-id`  | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context.                            | Any valid Workspace ID      |
| `--deployment-id` | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                                                 | Any valid Deployment ID     |

## Examples

```bash wrap theme={null}
# Create a Workspace Airflow variable
astro env airflow-variable create --workspace-id <workspace-id> --key MY_VAR --value some-value

# Bulk-create Airflow variables from a dotenv file
astro env airflow-variable create --workspace-id <workspace-id> --from-file vars.env
```

## Related commands

* [`astro env airflow-variable update`](/docs/cli/v1.45/astro-env-airflow-variable-update)
* [`astro env airflow-variable list`](/docs/cli/v1.45/astro-env-airflow-variable-list)
* [`astro env airflow-variable delete`](/docs/cli/v1.45/astro-env-airflow-variable-delete)


# astro env airflow-variable delete
Source: https://astronomer.io/docs/cli/v1.45/astro-env-airflow-variable-delete

Delete an Environment Manager Airflow variable.

<Info>
  This command is only available on Astro.
</Info>

Delete an Airflow variable, identified by its ID or key.

## Usage

```bash wrap theme={null}
astro env airflow-variable delete <id-or-key>
```

## Options

| Option            | Description                                                                                                              | Possible Values              |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------- |
| `<id-or-key>`     | The ID or key of the Airflow variable to delete.                                                                         | Any valid variable ID or key |
| `-y`,`--yes`      | Skip the confirmation prompt.                                                                                            | None                         |
| `--workspace-id`  | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context. | Any valid Workspace ID       |
| `--deployment-id` | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                      | Any valid Deployment ID      |

## Examples

```bash wrap theme={null}
# Delete a Workspace Airflow variable by key, skipping the confirmation prompt
astro env airflow-variable delete MY_VAR --workspace-id <workspace-id> --yes
```

## Related commands

* [`astro env airflow-variable create`](/docs/cli/v1.45/astro-env-airflow-variable-create)
* [`astro env airflow-variable list`](/docs/cli/v1.45/astro-env-airflow-variable-list)


# astro env airflow-variable get
Source: https://astronomer.io/docs/cli/v1.45/astro-env-airflow-variable-get

Get a single Environment Manager Airflow variable.

<Info>
  This command is only available on Astro.
</Info>

Get a single Airflow variable by its ID or key.

## Usage

```bash wrap theme={null}
astro env airflow-variable get <id-or-key>
```

## Options

| Option              | Description                                                                                                              | Possible Values              |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------- |
| `<id-or-key>`       | The ID or key of the Airflow variable to get.                                                                            | Any valid variable ID or key |
| `--format`          | The output format. Defaults to `table`.                                                                                  | `table`, `json`, or `yaml`   |
| `--include-secrets` | Surface the secret value in the output. Requires an Organization policy that allows it.                                  | None                         |
| `--workspace-id`    | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context. | Any valid Workspace ID       |
| `--deployment-id`   | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                      | Any valid Deployment ID      |

## Examples

```bash wrap theme={null}
# Get a Workspace Airflow variable by key
astro env airflow-variable get MY_VAR --workspace-id <workspace-id>
```

## Related commands

* [`astro env airflow-variable list`](/docs/cli/v1.45/astro-env-airflow-variable-list)
* [`astro env airflow-variable update`](/docs/cli/v1.45/astro-env-airflow-variable-update)


# astro env airflow-variable list
Source: https://astronomer.io/docs/cli/v1.45/astro-env-airflow-variable-list

List Environment Manager Airflow variables.

<Info>
  This command is only available on Astro.
</Info>

List the Airflow variables for a scope. At Deployment scope, the list includes variables linked from the Workspace unless you set `--resolve-linked` to `false`.

## Usage

```bash wrap theme={null}
astro env airflow-variable list
```

## Options

| Option              | Description                                                                                                                                                                | Possible Values             |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| `--format`          | The output format. Defaults to `table`.                                                                                                                                    | `table`, `json`, or `yaml`  |
| `--output`          | Write the output to a file. Use `-` for stdout. Defaults to `-`.                                                                                                           | Any valid file path, or `-` |
| `--include-secrets` | Surface secret values in the output. Requires an Organization policy that allows it.                                                                                       | None                        |
| `--resolve-linked`  | Include objects linked from another scope, such as a Workspace variable resolved at Deployment scope. Enabled by default. Set to `false` to return addressable object IDs. | `true` (default), `false`   |
| `--workspace-id`    | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context.                                                   | Any valid Workspace ID      |
| `--deployment-id`   | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                                                                        | Any valid Deployment ID     |

## Examples

```bash wrap theme={null}
# List Workspace Airflow variables
astro env airflow-variable list --workspace-id <workspace-id>

# List the Airflow variables a Deployment resolves, including those linked from the Workspace
astro env airflow-variable list --deployment-id <deployment-id> --resolve-linked
```

## Related commands

* [`astro env airflow-variable get`](/docs/cli/v1.45/astro-env-airflow-variable-get)
* [`astro env airflow-variable create`](/docs/cli/v1.45/astro-env-airflow-variable-create)


# astro env airflow-variable update
Source: https://astronomer.io/docs/cli/v1.45/astro-env-airflow-variable-update

Set the value of an Environment Manager Airflow variable.

<Info>
  This command is only available on Astro.
</Info>

Set the value of an Airflow variable. By default this command upserts: if the key doesn't exist, the CLI creates it. Pass `--strict` to fail when the key is missing. Use `--from-file` to bulk-upsert variables from a dotenv file.

## Usage

```bash wrap theme={null}
astro env airflow-variable update [<id-or-key>]
```

## Options

| Option            | Description                                                                                                                                | Possible Values              |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------- |
| `<id-or-key>`     | The ID or key of the Airflow variable to update. Omit this argument when you use `--from-file`.                                            | Any valid variable ID or key |
| `-v`,`--value`    | The new variable value. If omitted, the CLI reads the value from stdin when piped, or prompts for it with the input hidden.                | Any string                   |
| `-s`,`--secret`   | When the variable doesn't exist and the CLI creates it, mark it as secret. This has no effect when updating an existing variable.          | None                         |
| `--strict`        | Fail if the variable doesn't exist instead of creating it.                                                                                 | None                         |
| `--from-file`     | Bulk-upsert variables from a dotenv file. Pass `-` to read from stdin. Mutually exclusive with `--value` and the positional `<id-or-key>`. | Any valid file path, or `-`  |
| `--auto-link`     | Workspace scope only. Automatically link the variable to all Deployments in the Workspace, including future ones.                          | None                         |
| `--workspace-id`  | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context.                   | Any valid Workspace ID       |
| `--deployment-id` | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                                        | Any valid Deployment ID      |

## Examples

```bash wrap theme={null}
# Update a Workspace Airflow variable's value
astro env airflow-variable update MY_VAR --workspace-id <workspace-id> --value new-value

# Bulk-upsert Airflow variables from a dotenv file
astro env airflow-variable update --workspace-id <workspace-id> --from-file vars.env
```

## Related commands

* [`astro env airflow-variable create`](/docs/cli/v1.45/astro-env-airflow-variable-create)
* [`astro env airflow-variable get`](/docs/cli/v1.45/astro-env-airflow-variable-get)
* [`astro env airflow-variable list`](/docs/cli/v1.45/astro-env-airflow-variable-list)


# astro env connection
Source: https://astronomer.io/docs/cli/v1.45/astro-env-connection

Manage Environment Manager Airflow connections from the Astro CLI.

<Info>
  This command is only available on Astro.
</Info>

Use `astro env connection` commands to list, create, update, and delete Airflow connections managed through the Astro Environment Manager. You can scope a connection to a Workspace or a Deployment, and you can link a Workspace-scoped connection to specific Deployments. To manage these connections in the Astro UI instead, see [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).

<CardGroup>
  <Card title="astro env connection create" href="/cli/v1.45/astro-env-connection-create">
    View documentation for `astro env connection create`.
  </Card>

  <Card title="astro env connection delete" href="/cli/v1.45/astro-env-connection-delete">
    View documentation for `astro env connection delete`.
  </Card>

  <Card title="astro env connection get" href="/cli/v1.45/astro-env-connection-get">
    View documentation for `astro env connection get`.
  </Card>

  <Card title="astro env connection list" href="/cli/v1.45/astro-env-connection-list">
    View documentation for `astro env connection list`.
  </Card>

  <Card title="astro env connection update" href="/cli/v1.45/astro-env-connection-update">
    View documentation for `astro env connection update`.
  </Card>
</CardGroup>


# astro env connection create
Source: https://astronomer.io/docs/cli/v1.45/astro-env-connection-create

Create an Environment Manager Airflow connection.

<Info>
  This command is only available on Astro.
</Info>

Create an Airflow connection managed through the Astro Environment Manager. You can scope the connection to a Workspace or a Deployment.

## Usage

```bash wrap theme={null}
astro env connection create
```

## Options

| Option            | Description                                                                                                                              | Possible Values                   |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `-k`,`--key`      | The connection key. Required.                                                                                                            | Any string                        |
| `-t`,`--type`     | The connection type, for example `postgres` or `http`.                                                                                   | Any valid Airflow connection type |
| `--host`          | The connection host.                                                                                                                     | Any string                        |
| `-l`,`--login`    | The connection login or username.                                                                                                        | Any string                        |
| `-p`,`--password` | The connection password. If omitted alongside `--type`, the CLI reads it from stdin when piped, or prompts for it with the input hidden. | Any string                        |
| `--port`          | The connection port.                                                                                                                     | Any valid port number             |
| `--schema`        | The connection schema.                                                                                                                   | Any string                        |
| `--extra`         | Extra configuration, defined as a stringified JSON object.                                                                               | A stringified JSON object         |
| `--auto-link`     | Workspace scope only. Automatically link the connection to all Deployments in the Workspace, including future ones.                      | None                              |
| `--workspace-id`  | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context.                 | Any valid Workspace ID            |
| `--deployment-id` | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                                      | Any valid Deployment ID           |

## Examples

```bash wrap theme={null}
# Create a Postgres connection scoped to a Workspace
astro env connection create --workspace-id <workspace-id> --key db_main --type postgres --host db.example.com --port 5432 --login admin

# Create a Workspace connection auto-linked to every Deployment, reading the password from stdin
echo "$DB_PASSWORD" | astro env connection create --workspace-id <workspace-id> --key db_main --type postgres --host db.example.com --port 5432 --login admin --schema public --auto-link
```

## Related commands

* [`astro env connection update`](/docs/cli/v1.45/astro-env-connection-update)
* [`astro env connection list`](/docs/cli/v1.45/astro-env-connection-list)
* [`astro env connection delete`](/docs/cli/v1.45/astro-env-connection-delete)


# astro env connection delete
Source: https://astronomer.io/docs/cli/v1.45/astro-env-connection-delete

Delete an Environment Manager Airflow connection.

<Info>
  This command is only available on Astro.
</Info>

Delete an Airflow connection, identified by its ID or key.

## Usage

```bash wrap theme={null}
astro env connection delete <id-or-key>
```

## Options

| Option            | Description                                                                                                              | Possible Values                |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------ |
| `<id-or-key>`     | The ID or key of the connection to delete.                                                                               | Any valid connection ID or key |
| `-y`,`--yes`      | Skip the confirmation prompt.                                                                                            | None                           |
| `--workspace-id`  | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context. | Any valid Workspace ID         |
| `--deployment-id` | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                      | Any valid Deployment ID        |

## Examples

```bash wrap theme={null}
# Delete a Workspace connection by key, skipping the confirmation prompt
astro env connection delete db_main --workspace-id <workspace-id> --yes
```

## Related commands

* [`astro env connection create`](/docs/cli/v1.45/astro-env-connection-create)
* [`astro env connection list`](/docs/cli/v1.45/astro-env-connection-list)


# astro env connection get
Source: https://astronomer.io/docs/cli/v1.45/astro-env-connection-get

Get a single Environment Manager Airflow connection.

<Info>
  This command is only available on Astro.
</Info>

Get a single Airflow connection by its ID or key.

## Usage

```bash wrap theme={null}
astro env connection get <id-or-key>
```

## Options

| Option              | Description                                                                                                              | Possible Values                |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------ |
| `<id-or-key>`       | The ID or key of the connection to get.                                                                                  | Any valid connection ID or key |
| `--format`          | The output format. Defaults to `table`.                                                                                  | `table`, `json`, or `yaml`     |
| `--include-secrets` | Surface secret values in the output. Requires an Organization policy that allows it.                                     | None                           |
| `--workspace-id`    | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context. | Any valid Workspace ID         |
| `--deployment-id`   | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                      | Any valid Deployment ID        |

## Examples

```bash wrap theme={null}
# Get a Workspace connection by key
astro env connection get db_main --workspace-id <workspace-id>

# Get a connection as JSON
astro env connection get db_main --workspace-id <workspace-id> --format json
```

## Related commands

* [`astro env connection list`](/docs/cli/v1.45/astro-env-connection-list)
* [`astro env connection update`](/docs/cli/v1.45/astro-env-connection-update)


# astro env connection list
Source: https://astronomer.io/docs/cli/v1.45/astro-env-connection-list

List Environment Manager Airflow connections.

<Info>
  This command is only available on Astro.
</Info>

List the Airflow connections for a scope. At Deployment scope, the list includes connections linked from the Workspace unless you set `--resolve-linked` to `false`.

## Usage

```bash wrap theme={null}
astro env connection list
```

## Options

| Option              | Description                                                                                                                                                                  | Possible Values             |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| `--format`          | The output format. Defaults to `table`.                                                                                                                                      | `table`, `json`, or `yaml`  |
| `--output`          | Write the output to a file. Use `-` for stdout. Defaults to `-`.                                                                                                             | Any valid file path, or `-` |
| `--include-secrets` | Surface secret values in the output. Requires an Organization policy that allows it.                                                                                         | None                        |
| `--resolve-linked`  | Include objects linked from another scope, such as a Workspace connection resolved at Deployment scope. Enabled by default. Set to `false` to return addressable object IDs. | `true` (default), `false`   |
| `--workspace-id`    | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context.                                                     | Any valid Workspace ID      |
| `--deployment-id`   | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                                                                          | Any valid Deployment ID     |

## Examples

```bash wrap theme={null}
# List Workspace connections
astro env connection list --workspace-id <workspace-id>

# List the connections a Deployment resolves, including those linked from the Workspace
astro env connection list --deployment-id <deployment-id> --resolve-linked
```

## Related commands

* [`astro env connection get`](/docs/cli/v1.45/astro-env-connection-get)
* [`astro env connection create`](/docs/cli/v1.45/astro-env-connection-create)


# astro env connection update
Source: https://astronomer.io/docs/cli/v1.45/astro-env-connection-update

Update an Environment Manager Airflow connection.

<Info>
  This command is only available on Astro.
</Info>

Update an existing Airflow connection, identified by its ID or key. Provide only the fields you want to change.

## Usage

```bash wrap theme={null}
astro env connection update <id-or-key>
```

## Options

| Option            | Description                                                                                                                              | Possible Values                   |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `<id-or-key>`     | The ID or key of the connection to update.                                                                                               | Any valid connection ID or key    |
| `-t`,`--type`     | The connection type, for example `postgres` or `http`.                                                                                   | Any valid Airflow connection type |
| `--host`          | The connection host.                                                                                                                     | Any string                        |
| `-l`,`--login`    | The connection login or username.                                                                                                        | Any string                        |
| `-p`,`--password` | The connection password. If omitted alongside `--type`, the CLI reads it from stdin when piped, or prompts for it with the input hidden. | Any string                        |
| `--port`          | The connection port.                                                                                                                     | Any valid port number             |
| `--schema`        | The connection schema.                                                                                                                   | Any string                        |
| `--extra`         | Extra configuration, defined as a stringified JSON object.                                                                               | A stringified JSON object         |
| `--auto-link`     | Workspace scope only. Automatically link the connection to all Deployments in the Workspace, including future ones.                      | None                              |
| `--workspace-id`  | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context.                 | Any valid Workspace ID            |
| `--deployment-id` | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                                      | Any valid Deployment ID           |

## Examples

```bash wrap theme={null}
# Update only the host of a Workspace connection
astro env connection update db_main --workspace-id <workspace-id> --type postgres --host db-new.example.com
```

## Related commands

* [`astro env connection create`](/docs/cli/v1.45/astro-env-connection-create)
* [`astro env connection get`](/docs/cli/v1.45/astro-env-connection-get)
* [`astro env connection list`](/docs/cli/v1.45/astro-env-connection-list)


# astro env metrics-export
Source: https://astronomer.io/docs/cli/v1.45/astro-env-metrics-export

Manage Environment Manager metrics exports from the Astro CLI.

<Info>
  This command is only available on Astro.
</Info>

Use `astro env metrics-export` commands to list, create, update, and delete metrics exports managed through the Astro Environment Manager. A metrics export pushes Deployment metrics to a remote endpoint, such as a Prometheus remote-write target. You can scope a metrics export to a Workspace or a Deployment, and you can link a Workspace-scoped export to specific Deployments.

<CardGroup>
  <Card title="astro env metrics-export create" href="/cli/v1.45/astro-env-metrics-export-create">
    View documentation for `astro env metrics-export create`.
  </Card>

  <Card title="astro env metrics-export delete" href="/cli/v1.45/astro-env-metrics-export-delete">
    View documentation for `astro env metrics-export delete`.
  </Card>

  <Card title="astro env metrics-export get" href="/cli/v1.45/astro-env-metrics-export-get">
    View documentation for `astro env metrics-export get`.
  </Card>

  <Card title="astro env metrics-export list" href="/cli/v1.45/astro-env-metrics-export-list">
    View documentation for `astro env metrics-export list`.
  </Card>

  <Card title="astro env metrics-export update" href="/cli/v1.45/astro-env-metrics-export-update">
    View documentation for `astro env metrics-export update`.
  </Card>
</CardGroup>


# astro env metrics-export create
Source: https://astronomer.io/docs/cli/v1.45/astro-env-metrics-export-create

Create an Environment Manager metrics export.

<Info>
  This command is only available on Astro.
</Info>

Create a metrics export that pushes Deployment metrics to a remote endpoint, such as a Prometheus remote-write target. You can scope the metrics export to a Workspace or a Deployment.

## Usage

```bash wrap theme={null}
astro env metrics-export create
```

## Options

| Option               | Description                                                                                                                                                         | Possible Values                   |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `-k`,`--key`         | The metrics export key. Required.                                                                                                                                   | Any string                        |
| `--endpoint`         | The remote endpoint to push metrics to.                                                                                                                             | Any valid URL                     |
| `--exporter-type`    | The exporter type, for example `PROMETHEUS`.                                                                                                                        | Any valid exporter type           |
| `--auth-type`        | The authentication type for the endpoint.                                                                                                                           | `BASIC`, `AUTH_TOKEN`, or `SIGV4` |
| `--username`         | The username for `BASIC` authentication.                                                                                                                            | Any string                        |
| `--password`         | The password for `BASIC` authentication. If omitted alongside `--auth-type BASIC`, the CLI reads it from stdin when piped, or prompts for it with the input hidden. | Any string                        |
| `--basic-token`      | The bearer or authentication token for `AUTH_TOKEN` authentication.                                                                                                 | Any string                        |
| `--sigv4-assume-arn` | The AWS IAM role to assume for `SIGV4` authentication.                                                                                                              | Any valid IAM role ARN            |
| `--sigv4-sts-region` | The AWS STS region for `SIGV4` authentication.                                                                                                                      | Any valid AWS region              |
| `--header`           | A request header in `KEY=VALUE` form. Repeatable.                                                                                                                   | `KEY=VALUE`                       |
| `--label`            | A metric label in `KEY=VALUE` form. Repeatable.                                                                                                                     | `KEY=VALUE`                       |
| `--auto-link`        | Workspace scope only. Automatically link the metrics export to all Deployments in the Workspace, including future ones.                                             | None                              |
| `--workspace-id`     | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context.                                            | Any valid Workspace ID            |
| `--deployment-id`    | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                                                                 | Any valid Deployment ID           |

## Examples

```bash wrap theme={null}
# Create a Prometheus metrics export with basic authentication
astro env metrics-export create --workspace-id <workspace-id> \
  --key prom_main \
  --endpoint https://prom.example.com/api/v1/write \
  --exporter-type PROMETHEUS \
  --auth-type BASIC --username scraper --password "$SCRAPER_PASSWORD"
```

## Related commands

* [`astro env metrics-export update`](/docs/cli/v1.45/astro-env-metrics-export-update)
* [`astro env metrics-export list`](/docs/cli/v1.45/astro-env-metrics-export-list)
* [`astro env metrics-export delete`](/docs/cli/v1.45/astro-env-metrics-export-delete)


# astro env metrics-export delete
Source: https://astronomer.io/docs/cli/v1.45/astro-env-metrics-export-delete

Delete an Environment Manager metrics export.

<Info>
  This command is only available on Astro.
</Info>

Delete a metrics export, identified by its ID or key.

## Usage

```bash wrap theme={null}
astro env metrics-export delete <id-or-key>
```

## Options

| Option            | Description                                                                                                              | Possible Values                    |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------------- |
| `<id-or-key>`     | The ID or key of the metrics export to delete.                                                                           | Any valid metrics export ID or key |
| `-y`,`--yes`      | Skip the confirmation prompt.                                                                                            | None                               |
| `--workspace-id`  | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context. | Any valid Workspace ID             |
| `--deployment-id` | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                      | Any valid Deployment ID            |

## Examples

```bash wrap theme={null}
# Delete a Workspace metrics export by key, skipping the confirmation prompt
astro env metrics-export delete prom_main --workspace-id <workspace-id> --yes
```

## Related commands

* [`astro env metrics-export create`](/docs/cli/v1.45/astro-env-metrics-export-create)
* [`astro env metrics-export list`](/docs/cli/v1.45/astro-env-metrics-export-list)


# astro env metrics-export get
Source: https://astronomer.io/docs/cli/v1.45/astro-env-metrics-export-get

Get a single Environment Manager metrics export.

<Info>
  This command is only available on Astro.
</Info>

Get a single metrics export by its ID or key.

## Usage

```bash wrap theme={null}
astro env metrics-export get <id-or-key>
```

## Options

| Option              | Description                                                                                                              | Possible Values                    |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------------- |
| `<id-or-key>`       | The ID or key of the metrics export to get.                                                                              | Any valid metrics export ID or key |
| `--format`          | The output format. Defaults to `table`.                                                                                  | `table`, `json`, or `yaml`         |
| `--include-secrets` | Surface secret values in the output. Requires an Organization policy that allows it.                                     | None                               |
| `--workspace-id`    | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context. | Any valid Workspace ID             |
| `--deployment-id`   | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                      | Any valid Deployment ID            |

## Examples

```bash wrap theme={null}
# Get a Workspace metrics export by key
astro env metrics-export get prom_main --workspace-id <workspace-id>
```

## Related commands

* [`astro env metrics-export list`](/docs/cli/v1.45/astro-env-metrics-export-list)
* [`astro env metrics-export update`](/docs/cli/v1.45/astro-env-metrics-export-update)


# astro env metrics-export list
Source: https://astronomer.io/docs/cli/v1.45/astro-env-metrics-export-list

List Environment Manager metrics exports.

<Info>
  This command is only available on Astro.
</Info>

List the metrics exports for a scope. At Deployment scope, the list includes metrics exports linked from the Workspace unless you set `--resolve-linked` to `false`.

## Usage

```bash wrap theme={null}
astro env metrics-export list
```

## Options

| Option              | Description                                                                                                                                                                      | Possible Values             |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| `--format`          | The output format. Defaults to `table`.                                                                                                                                          | `table`, `json`, or `yaml`  |
| `--output`          | Write the output to a file. Use `-` for stdout. Defaults to `-`.                                                                                                                 | Any valid file path, or `-` |
| `--include-secrets` | Surface secret values in the output. Requires an Organization policy that allows it.                                                                                             | None                        |
| `--resolve-linked`  | Include objects linked from another scope, such as a Workspace metrics export resolved at Deployment scope. Enabled by default. Set to `false` to return addressable object IDs. | `true` (default), `false`   |
| `--workspace-id`    | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context.                                                         | Any valid Workspace ID      |
| `--deployment-id`   | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                                                                              | Any valid Deployment ID     |

## Examples

```bash wrap theme={null}
# List Workspace metrics exports
astro env metrics-export list --workspace-id <workspace-id>
```

## Related commands

* [`astro env metrics-export get`](/docs/cli/v1.45/astro-env-metrics-export-get)
* [`astro env metrics-export create`](/docs/cli/v1.45/astro-env-metrics-export-create)


# astro env metrics-export update
Source: https://astronomer.io/docs/cli/v1.45/astro-env-metrics-export-update

Update an Environment Manager metrics export.

<Info>
  This command is only available on Astro.
</Info>

Update an existing metrics export, identified by its ID or key. Provide only the fields you want to change.

## Usage

```bash wrap theme={null}
astro env metrics-export update <id-or-key>
```

## Options

| Option               | Description                                                                                                                                                         | Possible Values                    |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
| `<id-or-key>`        | The ID or key of the metrics export to update.                                                                                                                      | Any valid metrics export ID or key |
| `--endpoint`         | The remote endpoint to push metrics to.                                                                                                                             | Any valid URL                      |
| `--exporter-type`    | The exporter type, for example `PROMETHEUS`.                                                                                                                        | Any valid exporter type            |
| `--auth-type`        | The authentication type for the endpoint.                                                                                                                           | `BASIC`, `AUTH_TOKEN`, or `SIGV4`  |
| `--username`         | The username for `BASIC` authentication.                                                                                                                            | Any string                         |
| `--password`         | The password for `BASIC` authentication. If omitted alongside `--auth-type BASIC`, the CLI reads it from stdin when piped, or prompts for it with the input hidden. | Any string                         |
| `--basic-token`      | The bearer or authentication token for `AUTH_TOKEN` authentication.                                                                                                 | Any string                         |
| `--sigv4-assume-arn` | The AWS IAM role to assume for `SIGV4` authentication.                                                                                                              | Any valid IAM role ARN             |
| `--sigv4-sts-region` | The AWS STS region for `SIGV4` authentication.                                                                                                                      | Any valid AWS region               |
| `--header`           | A request header in `KEY=VALUE` form. Repeatable.                                                                                                                   | `KEY=VALUE`                        |
| `--label`            | A metric label in `KEY=VALUE` form. Repeatable.                                                                                                                     | `KEY=VALUE`                        |
| `--auto-link`        | Workspace scope only. Automatically link the metrics export to all Deployments in the Workspace, including future ones.                                             | None                               |
| `--workspace-id`     | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context.                                            | Any valid Workspace ID             |
| `--deployment-id`    | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                                                                 | Any valid Deployment ID            |

## Examples

```bash wrap theme={null}
# Update the labels on a Workspace metrics export
astro env metrics-export update prom_main --workspace-id <workspace-id> --label env=prod --label team=data
```

## Related commands

* [`astro env metrics-export create`](/docs/cli/v1.45/astro-env-metrics-export-create)
* [`astro env metrics-export get`](/docs/cli/v1.45/astro-env-metrics-export-get)
* [`astro env metrics-export list`](/docs/cli/v1.45/astro-env-metrics-export-list)


# astro env variable
Source: https://astronomer.io/docs/cli/v1.45/astro-env-variable

Manage Environment Manager environment variables from the Astro CLI.

<Info>
  This command is only available on Astro.
</Info>

Use `astro env variable` commands to list, create, update, delete, and export environment variables managed through the Astro Environment Manager. You can scope a variable to a Workspace or a Deployment, and you can link a Workspace-scoped variable to specific Deployments. To manage these variables in the Astro UI instead, see [Create environment variables in the Astro UI](/docs/astro/create-and-link-environment-variables).

<CardGroup>
  <Card title="astro env variable create" href="/cli/v1.45/astro-env-variable-create">
    View documentation for `astro env variable create`.
  </Card>

  <Card title="astro env variable delete" href="/cli/v1.45/astro-env-variable-delete">
    View documentation for `astro env variable delete`.
  </Card>

  <Card title="astro env variable export" href="/cli/v1.45/astro-env-variable-export">
    View documentation for `astro env variable export`.
  </Card>

  <Card title="astro env variable get" href="/cli/v1.45/astro-env-variable-get">
    View documentation for `astro env variable get`.
  </Card>

  <Card title="astro env variable link" href="/cli/v1.45/astro-env-variable-link">
    View documentation for `astro env variable link`.
  </Card>

  <Card title="astro env variable list" href="/cli/v1.45/astro-env-variable-list">
    View documentation for `astro env variable list`.
  </Card>

  <Card title="astro env variable update" href="/cli/v1.45/astro-env-variable-update">
    View documentation for `astro env variable update`.
  </Card>
</CardGroup>


# astro env variable create
Source: https://astronomer.io/docs/cli/v1.45/astro-env-variable-create

Create one or more Environment Manager environment variables.

<Info>
  This command is only available on Astro.
</Info>

Create a single environment variable with `--key` and `--value`, or bulk-create variables from a dotenv file with `--from-file`. When you use `--from-file`, the `--secret` and `--auto-link` flags apply to every entry in the file.

## Usage

```bash wrap theme={null}
astro env variable create
```

## Options

| Option            | Description                                                                                                                                         | Possible Values             |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
| `-k`,`--key`      | The variable key. Required unless you use `--from-file`.                                                                                            | Any string                  |
| `-v`,`--value`    | The variable value. If omitted, the CLI reads the value from stdin when piped, or prompts for it with the input hidden.                             | Any string                  |
| `-s`,`--secret`   | Mark the variable as secret.                                                                                                                        | None                        |
| `--from-file`     | Bulk-create variables from a dotenv file with one `KEY=VALUE` per line. Pass `-` to read from stdin. Mutually exclusive with `--key` and `--value`. | Any valid file path, or `-` |
| `--auto-link`     | Workspace scope only. Automatically link the variable to all Deployments in the Workspace, including future ones.                                   | None                        |
| `--workspace-id`  | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context.                            | Any valid Workspace ID      |
| `--deployment-id` | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                                                 | Any valid Deployment ID     |

## Examples

```bash wrap theme={null}
# Create a Workspace variable
astro env variable create --workspace-id <workspace-id> --key DBT_PROFILES_DIR --value /opt/profiles

# Create a secret Workspace variable and auto-link it to every Deployment
astro env variable create --workspace-id <workspace-id> --key API_TOKEN --value $TOKEN --secret --auto-link

# Bulk-create variables from a dotenv file
astro env variable create --workspace-id <workspace-id> --from-file .env
```

## Related commands

* [`astro env variable update`](/docs/cli/v1.45/astro-env-variable-update)
* [`astro env variable list`](/docs/cli/v1.45/astro-env-variable-list)
* [`astro env variable export`](/docs/cli/v1.45/astro-env-variable-export)


# astro env variable delete
Source: https://astronomer.io/docs/cli/v1.45/astro-env-variable-delete

Delete an Environment Manager environment variable.

<Info>
  This command is only available on Astro.
</Info>

Delete an environment variable, identified by its ID or key.

## Usage

```bash wrap theme={null}
astro env variable delete <id-or-key>
```

## Options

| Option            | Description                                                                                                              | Possible Values              |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------ | ---------------------------- |
| `<id-or-key>`     | The ID or key of the variable to delete.                                                                                 | Any valid variable ID or key |
| `-y`,`--yes`      | Skip the confirmation prompt.                                                                                            | None                         |
| `--workspace-id`  | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context. | Any valid Workspace ID       |
| `--deployment-id` | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                      | Any valid Deployment ID      |

## Examples

```bash wrap theme={null}
# Delete a Workspace variable by key, skipping the confirmation prompt
astro env variable delete DBT_PROFILES_DIR --workspace-id <workspace-id> --yes
```

## Related commands

* [`astro env variable create`](/docs/cli/v1.45/astro-env-variable-create)
* [`astro env variable list`](/docs/cli/v1.45/astro-env-variable-list)


# astro env variable export
Source: https://astronomer.io/docs/cli/v1.45/astro-env-variable-export

Export Environment Manager environment variables in dotenv format.

<Info>
  This command is only available on Astro.
</Info>

Export the environment variables for a scope as `KEY=VALUE` lines suitable for a `.env` file. The output round-trips with the `--from-file` flag on [`astro env variable create`](/docs/cli/v1.45/astro-env-variable-create) and [`astro env variable update`](/docs/cli/v1.45/astro-env-variable-update), so you can promote variables between Workspaces or pull them into a local Astro project.

By default, the CLI masks secret values. Pass `--include-secrets` to include them, subject to an Organization policy that allows it. When you use `--include-secrets`, the CLI prints a sensitive-data warning to stderr so the warning never lands in the exported file.

## Usage

```bash wrap theme={null}
astro env variable export
```

## Options

| Option              | Description                                                                                                              | Possible Values             |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------ | --------------------------- |
| `--output`          | Write the output to a file. Use `-` for stdout. Defaults to `-`.                                                         | Any valid file path, or `-` |
| `--include-secrets` | Surface secret values in the export. Requires an Organization policy that allows it.                                     | None                        |
| `--workspace-id`    | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context. | Any valid Workspace ID      |
| `--deployment-id`   | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                      | Any valid Deployment ID     |

## Examples

```bash wrap theme={null}
# Export Workspace variables to a local .env file
astro env variable export --workspace-id <workspace-id> > .env

# Include secret values in the export
astro env variable export --workspace-id <workspace-id> --include-secrets > .env

# Promote variables from one Workspace to another
astro env variable export --workspace-id <dev-workspace-id> | \
  astro env variable update --workspace-id <prod-workspace-id> --from-file -
```

## Related commands

* [`astro env variable list`](/docs/cli/v1.45/astro-env-variable-list)
* [`astro env variable create`](/docs/cli/v1.45/astro-env-variable-create)
* [`astro env variable update`](/docs/cli/v1.45/astro-env-variable-update)


# astro env variable get
Source: https://astronomer.io/docs/cli/v1.45/astro-env-variable-get

Get a single Environment Manager environment variable.

<Info>
  This command is only available on Astro.
</Info>

Get a single environment variable by its ID or key.

## Usage

```bash wrap theme={null}
astro env variable get <id-or-key>
```

## Options

| Option              | Description                                                                                                              | Possible Values                      |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------ |
| `<id-or-key>`       | The ID or key of the variable to get.                                                                                    | Any valid variable ID or key         |
| `--format`          | The output format. Defaults to `table`.                                                                                  | `table`, `json`, `yaml`, or `dotenv` |
| `--include-secrets` | Surface the secret value in the output. Requires an Organization policy that allows it.                                  | None                                 |
| `--workspace-id`    | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context. | Any valid Workspace ID               |
| `--deployment-id`   | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                      | Any valid Deployment ID              |

## Examples

```bash wrap theme={null}
# Get a Workspace variable by key
astro env variable get DBT_PROFILES_DIR --workspace-id <workspace-id>

# Get a variable as JSON
astro env variable get DBT_PROFILES_DIR --workspace-id <workspace-id> --format json
```

## Related commands

* [`astro env variable list`](/docs/cli/v1.45/astro-env-variable-list)
* [`astro env variable update`](/docs/cli/v1.45/astro-env-variable-update)


# astro env variable link
Source: https://astronomer.io/docs/cli/v1.45/astro-env-variable-link

Manage Deployment links for Workspace environment variables.

<Info>
  This command is only available on Astro.
</Info>

Use `astro env variable link` commands to manage the per-Deployment links of a Workspace-scoped environment variable. A link attaches a Workspace variable to a specific Deployment and can carry a per-Deployment value override. You can also exclude a Deployment from a variable that automatically links to every Deployment in the Workspace. Identify the variable with either `--variable-id` or `--variable-key`.

<CardGroup>
  <Card title="astro env variable link create" href="/cli/v1.45/astro-env-variable-link-create">
    View documentation for `astro env variable link create`.
  </Card>

  <Card title="astro env variable link delete" href="/cli/v1.45/astro-env-variable-link-delete">
    View documentation for `astro env variable link delete`.
  </Card>

  <Card title="astro env variable link list" href="/cli/v1.45/astro-env-variable-link-list">
    View documentation for `astro env variable link list`.
  </Card>
</CardGroup>


# astro env variable link create
Source: https://astronomer.io/docs/cli/v1.45/astro-env-variable-link-create

Link a Workspace environment variable to a Deployment.

<Info>
  This command is only available on Astro.
</Info>

Attach a Workspace-scoped environment variable to a specific Deployment. When you pass `--value`, that value overrides the Workspace default for the linked Deployment only. The Workspace value and other Deployments are unaffected. Pass `--exclude` to opt a Deployment out of a variable that automatically links to every Deployment in the Workspace.

This command is idempotent. If the link doesn't exist, the CLI creates it. If the link already exists, the CLI replaces the override when you pass `--value`. To remove an existing override, delete the link with [`astro env variable link delete`](/docs/cli/v1.45/astro-env-variable-link-delete), then recreate it without `--value`.

## Usage

```bash wrap theme={null}
astro env variable link create
```

## Options

| Option            | Description                                                                                                                          | Possible Values         |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- |
| `--variable-key`  | The key of the Workspace variable to link. Provide either this or `--variable-id`.                                                   | Any string              |
| `--variable-id`   | The ID of the Workspace variable to link. Provide either this or `--variable-key`.                                                   | Any valid variable ID   |
| `--deployment-id` | The ID of the Deployment to link. Required.                                                                                          | Any valid Deployment ID |
| `--value`         | An override value to use for the linked Deployment. Only the linked Deployment uses this value. Mutually exclusive with `--exclude`. | Any string              |
| `--exclude`       | Exclude the Deployment from an auto-linked variable instead of linking it. Mutually exclusive with `--value`.                        | None                    |
| `--workspace-id`  | Set the Workspace scope. Defaults to your current Workspace from CLI context.                                                        | Any valid Workspace ID  |

## Examples

```bash wrap theme={null}
# Link a Workspace variable to a Deployment
astro env variable link create --variable-key DATABASE_URL --workspace-id <workspace-id> --deployment-id <deployment-id>

# Link with a per-Deployment override value
astro env variable link create --variable-key DATABASE_URL --workspace-id <workspace-id> --deployment-id <deployment-id> --value "postgres://prod-host:5432/app"

# Exclude a Deployment from an auto-linked variable
astro env variable link create --variable-key LOG_LEVEL --workspace-id <workspace-id> --deployment-id <deployment-id> --exclude
```

## Related commands

* [`astro env variable link list`](/docs/cli/v1.45/astro-env-variable-link-list)
* [`astro env variable link delete`](/docs/cli/v1.45/astro-env-variable-link-delete)
* [`astro env variable create`](/docs/cli/v1.45/astro-env-variable-create)


# astro env variable link delete
Source: https://astronomer.io/docs/cli/v1.45/astro-env-variable-link-delete

Remove a link between a Workspace environment variable and a Deployment.

<Info>
  This command is only available on Astro.
</Info>

Remove an explicit link, or an exclude, between a Workspace-scoped environment variable and a Deployment. Identify the variable with either `--variable-key` or `--variable-id`.

## Usage

```bash wrap theme={null}
astro env variable link delete
```

## Options

| Option            | Description                                                                          | Possible Values         |
| ----------------- | ------------------------------------------------------------------------------------ | ----------------------- |
| `--variable-key`  | The key of the Workspace variable to unlink. Provide either this or `--variable-id`. | Any string              |
| `--variable-id`   | The ID of the Workspace variable to unlink. Provide either this or `--variable-key`. | Any valid variable ID   |
| `--deployment-id` | The ID of the Deployment to unlink. Required.                                        | Any valid Deployment ID |
| `--exclude`       | Remove an exclude instead of a link.                                                 | None                    |
| `--workspace-id`  | Set the Workspace scope. Defaults to your current Workspace from CLI context.        | Any valid Workspace ID  |

## Examples

```bash wrap theme={null}
# Remove a link between a Workspace variable and a Deployment
astro env variable link delete --variable-key DATABASE_URL --workspace-id <workspace-id> --deployment-id <deployment-id>

# Remove an exclude so the auto-linked variable applies to the Deployment again
astro env variable link delete --variable-key LOG_LEVEL --workspace-id <workspace-id> --deployment-id <deployment-id> --exclude
```

## Related commands

* [`astro env variable link create`](/docs/cli/v1.45/astro-env-variable-link-create)
* [`astro env variable link list`](/docs/cli/v1.45/astro-env-variable-link-list)


# astro env variable link list
Source: https://astronomer.io/docs/cli/v1.45/astro-env-variable-link-list

Show the Deployments a Workspace environment variable is linked to.

<Info>
  This command is only available on Astro.
</Info>

Show every Deployment that a Workspace-scoped environment variable is linked to or excluded from. Identify the variable with either `--variable-key` or `--variable-id`.

## Usage

```bash wrap theme={null}
astro env variable link list
```

## Options

| Option              | Description                                                                          | Possible Values            |
| ------------------- | ------------------------------------------------------------------------------------ | -------------------------- |
| `--variable-key`    | The key of the Workspace variable. Provide either this or `--variable-id`.           | Any string                 |
| `--variable-id`     | The ID of the Workspace variable. Provide either this or `--variable-key`.           | Any valid variable ID      |
| `--format`          | The output format. Defaults to `table`.                                              | `table`, `json`, or `yaml` |
| `--include-secrets` | Surface secret values in the output. Requires an Organization policy that allows it. | None                       |
| `--workspace-id`    | Set the Workspace scope. Defaults to your current Workspace from CLI context.        | Any valid Workspace ID     |

## Examples

```bash wrap theme={null}
# Show every Deployment a Workspace variable is linked to or excluded from
astro env variable link list --variable-key DATABASE_URL --workspace-id <workspace-id>
```

## Related commands

* [`astro env variable link create`](/docs/cli/v1.45/astro-env-variable-link-create)
* [`astro env variable link delete`](/docs/cli/v1.45/astro-env-variable-link-delete)


# astro env variable list
Source: https://astronomer.io/docs/cli/v1.45/astro-env-variable-list

List Environment Manager environment variables.

<Info>
  This command is only available on Astro.
</Info>

List the environment variables for a scope. At Deployment scope, the list includes variables linked from the Workspace unless you set `--resolve-linked` to `false`.

## Usage

```bash wrap theme={null}
astro env variable list
```

## Options

| Option              | Description                                                                                                                                                                | Possible Values                      |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| `--format`          | The output format. Defaults to `table`.                                                                                                                                    | `table`, `json`, `yaml`, or `dotenv` |
| `--output`          | Write the output to a file. Use `-` for stdout. Defaults to `-`.                                                                                                           | Any valid file path, or `-`          |
| `--include-secrets` | Surface secret values in the output. Requires an Organization policy that allows it.                                                                                       | None                                 |
| `--resolve-linked`  | Include objects linked from another scope, such as a Workspace variable resolved at Deployment scope. Enabled by default. Set to `false` to return addressable object IDs. | `true` (default), `false`            |
| `--workspace-id`    | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context.                                                   | Any valid Workspace ID               |
| `--deployment-id`   | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                                                                        | Any valid Deployment ID              |

## Examples

```bash wrap theme={null}
# List Workspace variables
astro env variable list --workspace-id <workspace-id>

# List the variables a Deployment resolves, including those linked from the Workspace
astro env variable list --deployment-id <deployment-id> --resolve-linked
```

## Related commands

* [`astro env variable get`](/docs/cli/v1.45/astro-env-variable-get)
* [`astro env variable export`](/docs/cli/v1.45/astro-env-variable-export)
* [`astro env variable create`](/docs/cli/v1.45/astro-env-variable-create)


# astro env variable update
Source: https://astronomer.io/docs/cli/v1.45/astro-env-variable-update

Set the value of an Environment Manager environment variable.

<Info>
  This command is only available on Astro.
</Info>

Set the value of an environment variable. By default this command upserts: if the key doesn't exist, the CLI creates it. Pass `--strict` to fail when the key is missing. Use `--from-file` to bulk-upsert variables from a dotenv file.

The platform API doesn't allow toggling the secret flag on an existing variable. To change whether an existing variable is secret, delete it and recreate it.

## Usage

```bash wrap theme={null}
astro env variable update [<id-or-key>]
```

## Options

| Option            | Description                                                                                                                                | Possible Values              |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------- |
| `<id-or-key>`     | The ID or key of the variable to update. Omit this argument when you use `--from-file`.                                                    | Any valid variable ID or key |
| `-v`,`--value`    | The new variable value. If omitted, the CLI reads the value from stdin when piped, or prompts for it with the input hidden.                | Any string                   |
| `-s`,`--secret`   | When the variable doesn't exist and the CLI creates it, mark it as secret. This has no effect when updating an existing variable.          | None                         |
| `--strict`        | Fail if the variable doesn't exist instead of creating it.                                                                                 | None                         |
| `--from-file`     | Bulk-upsert variables from a dotenv file. Pass `-` to read from stdin. Mutually exclusive with `--value` and the positional `<id-or-key>`. | Any valid file path, or `-`  |
| `--auto-link`     | Workspace scope only. Automatically link the variable to all Deployments in the Workspace, including future ones.                          | None                         |
| `--workspace-id`  | Set the Workspace scope. Mutually exclusive with `--deployment-id`. Defaults to your current Workspace from CLI context.                   | Any valid Workspace ID       |
| `--deployment-id` | Set the Deployment scope. Mutually exclusive with `--workspace-id`.                                                                        | Any valid Deployment ID      |

## Examples

```bash wrap theme={null}
# Update a Workspace variable's value
astro env variable update DBT_PROFILES_DIR --workspace-id <workspace-id> --value /etc/profiles

# Bulk-upsert variables from a dotenv file
astro env variable update --workspace-id <workspace-id> --from-file .env

# Fail if the variable doesn't already exist
astro env variable update DBT_PROFILES_DIR --workspace-id <workspace-id> --value /etc/profiles --strict
```

## Related commands

* [`astro env variable create`](/docs/cli/v1.45/astro-env-variable-create)
* [`astro env variable get`](/docs/cli/v1.45/astro-env-variable-get)
* [`astro env variable list`](/docs/cli/v1.45/astro-env-variable-list)


# astro ide
Source: https://astronomer.io/docs/cli/v1.45/astro-ide

Manage Astro IDE projects from the Astro CLI.

<Info>
  These commands are only available on Astro.
</Info>

# Manage Astro IDE projects from the CLI

The `astro ide` command group provides basic management for [Astro IDE](/docs/astro/ide-overview) projects through the Astro CLI. You can list all available projects, import an existing project to your local environment, or export a local project to Astro IDE.

<CardGroup>
  <Card title="astro ide project list" href="/cli/v1.45/astro-ide-project-list">
    View documentation for `astro ide project list`.
  </Card>

  <Card title="astro ide project import" href="/cli/v1.45/astro-ide-project-import">
    View documentation for `astro ide project import`.
  </Card>

  <Card title="astro ide project export" href="/cli/v1.45/astro-ide-project-export">
    View documentation for `astro ide project export`.
  </Card>
</CardGroup>


# astro ide project export
Source: https://astronomer.io/docs/cli/v1.45/astro-ide-project-export

Export Astro IDE projects.

<Info>
  These commands are only available on Astro.
</Info>

Export your current local project to Astro IDE.

## Usage

```sh wrap theme={null}
astro ide project export
```

## Description

Upload your local project to Astro IDE. You can create a new Astro IDE project or select an existing one from the list to update with your local files.

## Related commands

* [`astro ide project import`](/docs/cli/v1.45/astro-ide-project-import)
* [`astro ide project list`](/docs/cli/v1.45/astro-ide-project-list)


# astro ide project import
Source: https://astronomer.io/docs/cli/v1.45/astro-ide-project-import

Import an existing Astro IDE project.

<Info>
  These commands are only available on Astro.
</Info>

## astro ide project import

Import an existing Astro IDE project into your local environment for development or editing.

### Usage

```sh wrap theme={null}
astro ide project import
```

### Options

| Option         | Description                                | Possible Values |
| -------------- | ------------------------------------------ | --------------- |
| `--session-id` | The ID of the Astro IDE session to import. | String          |
| `-h`, `--help` | Help for the command.                      | N/A             |

### Description

Prompts you to select from the list of Astro IDE projects you have access to and imports the selected project into your current directory.

## Related commands

* [`astro ide project export`](/docs/cli/v1.45/astro-ide-project-export)
* [`astro ide project list`](/docs/cli/v1.45/astro-ide-project-list)


# astro ide project list
Source: https://astronomer.io/docs/cli/v1.45/astro-ide-project-list

List Astro IDE projects.

<Info>
  These commands are only available on Astro.
</Info>

List all Astro IDE projects available to your user and workspace.

## Usage

```sh wrap theme={null}
astro ide project list
```

## Output

| Output | Description                                     | Data Type |
| ------ | ----------------------------------------------- | --------- |
| `NAME` | The name of the Astro IDE project.              | String    |
| `ID`   | The unique identifier of the Astro IDE project. | String    |

## Description

Displays the names and unique identifiers of all Astro IDE projects you have access to in your current workspace.

## Related commands

* [`astro ide project export`](/docs/cli/v1.45/astro-ide-project-export)
* [`astro ide project import`](/docs/cli/v1.45/astro-ide-project-import)


# astro login
Source: https://astronomer.io/docs/cli/v1.45/astro-login

Reference documentation for astro login.

<Info>This command is an alias for [`astro auth login`](/docs/cli/v1.45/astro-auth-login). The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change product contexts.</Info>

<Tabs>
  <Tab title="Astro">
    Authenticate to Astro. After you run this command, the CLI prompts you for your login email address. Using the provided email address, the CLI assumes your organization and redirects you to a web browser where you can log in to the Astro UI. After you log in, the CLI automatically recognizes this and authenticates your account.

    If you're running the Astro CLI on a headless system or in an environment without browser access (such as a remote server, Docker container, or CI/CD pipeline), use the `--login-link` or `--token-login` flags to authenticate without needing a local browser.

    ## Usage

    ```sh wrap theme={null}
    astro login
    ```

    ## Options

    | Option                | Description                                                                                                                                                                                                                                                                                                                                              | Possible Values                                               |
    | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
    | `-l`, `--login-link`  | Force the CLI to print a login URL instead of automatically opening a browser. Copy the URL and open it on any device that has browser access to complete authentication. This is useful for headless environments, remote servers, or SSH sessions where a browser is not available                                                                     | None                                                          |
    | `-t`, `--token-login` | Authenticate by providing a token directly, without any browser interaction. Generate a token at `cloud.astronomer.io/token` from a device with browser access, then pass it to this flag. This is the recommended approach for fully headless systems and non-interactive environments such as CI/CD pipelines, Docker containers, or automated scripts | A valid authentication token from `cloud.astronomer.io/token` |

    ## Examples

    ```sh wrap theme={null}
    astro login
    # The CLI automatically opens the Astro UI in a web browser, which prompts you to log in.

    astro login --login-link
    # The CLI prints a login URL instead of opening a browser.
    # Use this on headless systems or SSH sessions: copy the URL to a device with a browser to authenticate.

    astro login --token-login <your-token>
    # Authenticate without any browser interaction.
    # Generate a token at cloud.astronomer.io/token and pass it directly.
    # Recommended for CI/CD pipelines, Docker containers, and other non-interactive environments.
    ```
  </Tab>

  <Tab title="APC">
    Authenticate to Astro Private Cloud. After you run this command, the CLI prompts you to either enter a username and password or retrieve an OAuth token from `<basedomain>/token`.

    ## Usage

    ```sh wrap theme={null}
    astro login <basedomain>
    ```

    ## Options

    | Option               | Description                                                                              | Possible Values |
    | -------------------- | ---------------------------------------------------------------------------------------- | --------------- |
    | `-l`, `--login-link` | Generate a login link to login on a separate device for cloud CLI login                  | None            |
    | `-o`, `--oauth`      | Skip the prompt for local authentication, proceed directly to OAuth token authentication | None            |

    ## Examples

    ```sh wrap theme={null}
    astro login mycompany.astronomer.io
    # The CLI prompts you for a username and password, or to leave the prompt empty for OAuth authentication

    astro login mycompany.astronomer.io -o
    # The CLI does not prompt you for a username and password and instead directly prompts you for an OAuth login token
    ```
  </Tab>
</Tabs>

### Related commands

* [`astro logout`](/docs/cli/v1.45/astro-logout)
* [`astro deploy`](/docs/cli/v1.45/astro-deploy)


# astro logout
Source: https://astronomer.io/docs/cli/v1.45/astro-logout

Reference documentation for astro logout.

<Info>This command is an alias for [`astro auth logout`](/docs/cli/v1.45/astro-auth-logout). The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Log out of the Astro CLI. This command does not affect your current web browser login session in either Astro or Astro Private Cloud.

## Usage

```sh wrap theme={null}
astro logout
```

## Related commands

* [`astro login`](/docs/cli/v1.45/astro-login)


# astro organization
Source: https://astronomer.io/docs/cli/v1.45/astro-organization

Manage users and their Organization-level permissions.

<Info>
  This command is only available on Astro.
</Info>

Use `astro organization` commands to manage users and their Organization-level permissions.

<CardGroup>
  <Card title="astro organization audit logs" href="/cli/v1.45/astro-organization-audit-logs">
    View documentation for `astro organization audit logs`.
  </Card>

  <Card title="astro organization list" href="/cli/v1.45/astro-organization-list">
    View documentation for `astro organization list`.
  </Card>

  <Card title="astro organization switch" href="/cli/v1.45/astro-organization-switch">
    View documentation for `astro organization switch`.
  </Card>

  <Card title="astro organization role list" href="/cli/v1.45/astro-organization-role-list">
    View documentation for `astro organization role list`.
  </Card>

  <Card title="astro organization team create" href="/cli/v1.45/astro-organization-team-create">
    View documentation for `astro organization team create`.
  </Card>

  <Card title="astro organization team delete" href="/cli/v1.45/astro-organization-team-delete">
    View documentation for `astro organization team delete`.
  </Card>

  <Card title="astro organization team list" href="/cli/v1.45/astro-organization-team-list">
    View documentation for `astro organization team list`.
  </Card>

  <Card title="astro organization team update" href="/cli/v1.45/astro-organization-team-update">
    View documentation for `astro organization team update`.
  </Card>

  <Card title="astro organization team user" href="/cli/v1.45/astro-organization-team-user">
    View documentation for `astro organization team user`.
  </Card>

  <Card title="astro organization token create" href="/cli/v1.45/astro-organization-token-create">
    View documentation for `astro organization token create`.
  </Card>

  <Card title="astro organization token delete" href="/cli/v1.45/astro-organization-token-delete">
    View documentation for `astro organization token delete`.
  </Card>

  <Card title="astro organization token list" href="/cli/v1.45/astro-organization-token-list">
    View documentation for `astro organization token list`.
  </Card>

  <Card title="astro organization token roles" href="/cli/v1.45/astro-organization-token-roles">
    View documentation for `astro organization token roles`.
  </Card>

  <Card title="astro organization token rotate" href="/cli/v1.45/astro-organization-token-rotate">
    View documentation for `astro organization token rotate`.
  </Card>

  <Card title="astro organization token update" href="/cli/v1.45/astro-organization-token-update">
    View documentation for `astro organization token update`.
  </Card>

  <Card title="astro organization user invite" href="/cli/v1.45/astro-organization-user-invite">
    View documentation for `astro organization user invite`.
  </Card>

  <Card title="astro organization user list" href="/cli/v1.45/astro-organization-user-list">
    View documentation for `astro organization user list`.
  </Card>

  <Card title="astro organization user update" href="/cli/v1.45/astro-organization-user-update">
    View documentation for `astro organization user update`.
  </Card>
</CardGroup>


# astro organization audit-logs
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-audit-logs

Reference documentation for the astro organization audit-logs command.

<Info>
  This command is only available on Astro.
</Info>

Astro audit logs record administrative activities and events in your Organization. You can use the audit logs to determine who did what, where, and when. This command allows you to export audit logs in a GZIP format for your entire Organization.

See [Export Astro audit logs](/docs/astro/audit-logs) for more information about exporting audit logs as well as a reference describing the different included fields.

## Usage

```bash wrap theme={null}
astro organization audit-logs <options>
```

Or

```bash wrap theme={null}
astro organization al <options>
```

## Options

| Option   | Description                                                                           | Valid Values |
| -------- | ------------------------------------------------------------------------------------- | ------------ |
| `export` | Export your Organization audit logs in GZIP. Requires Organization Owner permissions. | N/A          |

## `export` flags

| Flag                        | Description                                                                                       | Valid Values                     |
| --------------------------- | ------------------------------------------------------------------------------------------------- | -------------------------------- |
| `-i`, `--include`           | Number of days in the past to start exporting logs from. Minimum: 1. Maximum: 90. Default is `1`. | An integer between `1` and `90`. |
| `-n`, `--organization-name` | The name of the Organization to export audit logs for.                                            | Any valid Organization name.     |
| `-o`, `--output-file`       | Path to a file for storing exported audit logs.                                                   | Any valid file path.             |

## Examples

Export audit logs for your Organization:

```bash wrap theme={null}
astro organization audit-logs export --organization-name="<your-organization-name>"
```

Export the last 30 days of audit logs to a file:

```bash wrap theme={null}
astro organization audit-logs export --include 30 --output-file audit-logs.gz
```

## Related commands

* [`astro organization list`](/docs/cli/v1.45/astro-organization-list)
* [`astro organization role list`](/docs/cli/v1.45/astro-organization-role-list)


# astro organization list
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-list

Reference documentation for the astro organization list command.

<Info>
  This command is only available on Astro.
</Info>

View a list of Organizations that you can access on Astro and their IDs.

## Usage

Run `astro organization list` to view a list of Organizations that you can access on Astro and their IDs. Only the Organizations you have been invited to by an Organization Owner appear on this list.

## Options

| Option          | Description                                                                                          | Possible Values                |
| --------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------ |
| `--json`        | Output the response as JSON. Shorthand for `--output json`. Mutually exclusive with `-o`/`--output`. | None                           |
| `-o`,`--output` | Format the output. The default is `table`.                                                           | `table`, `json`, or `template` |
| `--template`    | Format the output using a Go template. Use with `--output template`.                                 | Any valid Go template string   |

## Output

| Output | Description                                            | Data Type |
| ------ | ------------------------------------------------------ | --------- |
| `NAME` | The name of the Organizations that you have access to. | String    |
| `ID`   | The Organization ID.                                   | String    |

## Related commands

* [`astro login`](/docs/cli/v1.45/astro-login)
* [`astro organization switch`](/docs/cli/v1.45/astro-organization-switch)


# astro organization role list
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-role-list

Reference documentation for the astro organization role list command.

<Info>
  This command is only available on Astro.
</Info>

View a list of all roles in an Organization.

## Usage

Run `astro organization role list` to view a list of the custom and default user roles in your Organization and their IDs. By default, only custom roles are displayed. To create a custom role, see [Create and assign custom Deployment roles](/docs/astro/customize-deployment-roles)

If you want to view the Astro-defined user roles and their IDs, you need to add the `--include-default-roles` flag. For example,

```sh wrap theme={null}

astro organization role list --include-default-roles

```

## Options

| Option                    | Description                                                                          | Possible Values                  |
| ------------------------- | ------------------------------------------------------------------------------------ | -------------------------------- |
| `--include-default-roles` | Display the Astro-defined [user roles](/docs/astro/user-permissions) in the organization. | Any of the available user roles. |

## Output

| Output        | Description                     | Data Type |
| ------------- | ------------------------------- | --------- |
| `NAME`        | The name of the role.           | String    |
| `ID`          | The custom role ID.             | String    |
| `DESCRIPTION` | A description of the user role. | String    |


# astro organization switch
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-switch

Switch your current Organization.

<Info>
  This command is only available on Astro.
</Info>

Switch the Astro Organization where you're currently working.

## Usage

Run `astro organization switch` to switch between Organizations. In the Astro CLI, you can first run `astro organization list` to see a list of Organizations that you can access and their IDs.

| Option              | Description                                                   | Possible Values                                          |
| ------------------- | ------------------------------------------------------------- | -------------------------------------------------------- |
| `<organization-id>` | Specify the Astro Organization ID that you want to switch to. | Any valid Organization ID                                |
| `--workspace-id`    | Select a Workspace when switching organizations.              | Any valid Workspace ID within the specified Organization |

## Related commands

* [`astro login`](/docs/cli/v1.45/astro-login)
* [`astro organization list`](/docs/cli/v1.45/astro-organization-list)


# Astro CLI documentation archive
Source: https://astronomer.io/docs/cli/archive

Links to documentation for retired versions of the Astro CLI.

This document lists the Astro CLI documentation sets that Astronomer has retired from the documentation site.

While only maintained documentation sets appear under the Astro CLI documentation menu, Astronomer preserves all versions for historical reference. Documentation for Astro CLI versions v1.37 and earlier no longer receives updates and is not rendered on the Astronomer documentation site. You can still read these docs in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources).

To use the latest Astro CLI features and documentation, Astronomer recommends [upgrading to the latest version of the Astro CLI](/docs/cli/v1.45/install-cli).

## Retired versions

Documentation for the following Astro CLI versions is available in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources):

* v1.37
* v1.36
* v1.35
* v1.34

If you notice an error or misleading information in any part of Astronomer's documentation, [create a GitHub issue](https://github.com/astronomer/docs/issues) or contact [Astronomer support](https://support.astronomer.io).


# Add Airflow providers, Python packages, and operating system packages
Source: https://astronomer.io/docs/cli/v1.45/add-providers-packages

Airflow providers, Python packages, and operating system packages.

Most dags need additional Python or OS-level packages to run. You need to add Python packages, including [Airflow Providers](https://airflow.apache.org/docs/apache-airflow-providers/), to your Astro project's `requirements.txt` file, and OS-level packages to the project's `packages.txt` file.

There are two primary kinds of Python packages that you might need to add to your Astro project:

* **Non-provider Python libraries**. If you’re using Airflow for a data science project, for example, you might use a data science library such as [pandas](https://pandas.pydata.org/) or [NumPy (`numpy`)](https://numpy.org/).
* **Airflow Providers**. Airflow Providers are Python packages that contain relevant Airflow modules for a third-party service. For example, `apache-airflow-providers-amazon` includes the hooks, operators, and integrations you need to access services on Amazon Web Services (AWS) with Airflow. See [Provider packages](https://airflow.apache.org/docs/apache-airflow-providers/).

Adding the name of a package to the `packages.txt` or `requirements.txt` files of your Astro project installs the package to your Airflow environment.

1. Add the package name to your Astro project. If it’s a Python package, add it to `requirements.txt`. If it’s an OS-level package, add it to `packages.txt`. The latest version of the package that’s publicly available is installed by default.

   To pin a version of a package, use the following syntax:

   ```text wrap theme={null}
   <package-name>==<version>
   ```

   For example, to install NumPy version 1.23.0, add the following to your `requirements.txt` file:

   ```text wrap theme={null}
   numpy==1.23.0
   ```

2. [Restart your local environment](/docs/cli/v1.45/run-airflow-locally#restart-a-local-airflow-environment).

3. Confirm that your package was installed:

   ```sh wrap theme={null}
   astro dev bash --scheduler "pip freeze | grep <package-name>"
   ```

To learn more about the format of the `requirements.txt` file, see [Requirements File Format](https://pip.pypa.io/en/stable/reference/requirements-file-format/#requirements-file-format) in pip documentation. To browse Python libraries, see [PyPi](https://pypi.org/). To browse Airflow providers, see the [Astronomer Registry](https://registry.astronomer.io/providers/).


# Use `.airflowignore` to ignore files in your Astro project
Source: https://astronomer.io/docs/cli/v1.45/airflowignore



You can create an `.airflowignore` file in the `dags` directory of your Astro project to identify the files to ignore when you deploy to Astro or develop locally. This can be helpful if your team has a single Git repository that contains dags for multiple projects.

The `.airflowignore` file and the files listed in it must be in the same `dags` directory of your Astro project. The Airflow scheduler does not parse the files or directories listed in `.airflowignore` and the Airflow UI does not show dags listed in the file.

For more information about `.airflowignore`, see [`.airflowignore` in the Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#airflowignore). To learn more about the code deploy process, see [What happens during a code deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy).

## Setup

1. In the `dags` directory of your Astro project, create a new file named `.airflowignore`.

2. List the files or sub-directories you want ignored when you push code to Astro or when you are developing locally. You should list the path for each file or directory relative to the `dags` directory. For example:

   ```text wrap theme={null}
   mydag.py
   data-team-dags
   some-dags/ignore-this-dag.py
   ```

   You can also use regular expressions to specify groups of files. See the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#airflowignore) for more information about usage.

3. Save your changes locally or deploy to Astro.

   Your local Airflow environment automatically updates as soon as you save your changes to `.airflowignore`. To apply your change in Astro, you need to deploy. See [Deploy code](/docs/astro/deploy-code).


# astro organization team create
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-team-create

Create a new Team in your Organization.

<Info>
  This command is only available on Astro.
</Info>

Create a new Team in your Organization.

## Usage

```sh wrap theme={null}
astro organization team create --name "<team-name>"
```

## Options

| Option          | Description                          | Valid Values                                                                                   |
| --------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------- |
| `--name`        | The Team's name.                     | String surrounded by quotation marks.                                                          |
| `--description` | A description for the Team.          | String surrounded by quotation marks.                                                          |
| `--role`        | The Team's role in the Organization. | Valid values are `ORGANIZATION_MEMBER`, `ORGANIZATION_BILLING_ADMIN`, or `ORGANIZATION_OWNER`. |

## Examples

```sh wrap theme={null}
# Invite a user to your Organization
astro organization team create --name "Billing Admins" --role ORGANIZATION_BILLING_ADMIN
```

## Related commands

* [`astro organization user update`](/docs/cli/v1.45/astro-organization-user-update)
* [`astro workspace user add`](/docs/cli/v1.45/astro-workspace-user-add)


# astro organization team delete
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-team-delete

Delete a Team from your Organization.

<Info>
  This command is only available on Astro.
</Info>

Delete a Team from your Organization.

## Usage

```sh wrap theme={null}
astro organization team delete <team-id>
```

## Options

| Option          | Description                                                             | Valid Values |
| --------------- | ----------------------------------------------------------------------- | ------------ |
| `-f`, `--force` | Skip the confirmation prompt for Identity Provider (IdP)-managed teams. | None         |

To find a Team ID using the Astro CLI, run `astro organization team list`.

To find a Team ID in the Astro UI, click **Organization Settings** > **Access Management** > **Teams**. Search for your Team in the **Teams** table and copy its **ID**. The ID should look something like `clk17xqgm124q01hkrgilsr49`.

## Related commands

* [`astro organization user update`](/docs/cli/v1.45/astro-organization-user-update)
* [`astro workspace user add`](/docs/cli/v1.45/astro-workspace-user-add)


# astro organization team list
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-team-list

List all Teams in your current Organization.

<Info>
  This command is only available on Astro.
</Info>

List all Teams in your current Organization.

## Usage

```sh wrap theme={null}
astro organization team list
```

## Options

| Option          | Description                                                                                          | Possible Values                |
| --------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------ |
| `--json`        | Output the response as JSON. Shorthand for `--output json`. Mutually exclusive with `-o`/`--output`. | None                           |
| `-o`,`--output` | Format the output. The default is `table`.                                                           | `table`, `json`, or `template` |
| `--template`    | Format the output using a Go template. Use with `--output template`.                                 | Any valid Go template string   |

## Output

| Column        | Description                                                      | Data type                                                                                          |
| ------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `ID`          | The Team ID in the Organization.                                 | String                                                                                             |
| `NAME`        | Team name.                                                       | String                                                                                             |
| `DESCRIPTION` | The description for the Team.                                    | String                                                                                             |
| `ORG ROLE`    | The Team's role in the Organization.                             | Possible values are `ORGANIZATION_MEMBER`, `ORGANIZATION_BILLING_ADMIN`, and `ORGANIZATION_OWNER`. |
| `IDP MANAGED` | Whether the Team is managed through an identity provider.        | Boolean                                                                                            |
| `CREATE DATE` | The date and time that the Team was created in the Organization. | Date (`YYYY-MM-DDTHH:MM:SSZ`)                                                                      |

## Related commands

* [`astro workspace team add`](/docs/cli/v1.45/astro-workspace-team-add)
* [`astro organization team create`](/docs/cli/v1.45/astro-organization-team-create)
* [`astro workspace switch`](/docs/cli/v1.45/astro-workspace-switch)


# astro organization team update
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-team-update

Update a Team in your Organization.

<Info>
  This command is only available on Astro.
</Info>

Update a Team in your Organization.

## Usage

```sh wrap theme={null}
astro organization team update <flags>
```

## Options

| Option          | Description                                                             | Valid Values                                                                                       |
| --------------- | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `<team-id>`     | The ID for the Team you want to update.                                 | A valid Team ID.                                                                                   |
| `--name`        | The Team's name.                                                        | String surrounded by quotation marks.                                                              |
| `--description` | A description for the Team.                                             | String surrounded by quotation marks.                                                              |
| `-f`, `--force` | Skip the confirmation prompt for Identity Provider (IdP)-managed teams. | None                                                                                               |
| `-r`, `--role`  | The new role for the team.                                              | Possible values are `ORGANIZATION_MEMBER`, `ORGANIZATION_BILLING_ADMIN`, and `ORGANIZATION_OWNER`. |

## Related commands

* [`astro organization user update`](/docs/cli/v1.45/astro-organization-user-update)
* [`astro workspace user add`](/docs/cli/v1.45/astro-workspace-user-add)


# astro organization team user
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-team-user

Manage users in an Astro Team.

<Info>
  This command is only available on Astro.
</Info>

Manage users in an Astro Team.

## Usage

This command has several subcommands

### astro organization user team add

Add a user to a Team.

#### Usage

```sh wrap theme={null}
astro organization team user add
```

#### Options

| Option          | Description                                                             | Valid Values     |
| --------------- | ----------------------------------------------------------------------- | ---------------- |
| `--team-id`     | The ID of the Team where you want to add the user.                      | A valid Team ID. |
| `--user-id`     | The user's ID.                                                          | A valid user ID. |
| `-f`, `--force` | Skip the confirmation prompt for Identity Provider (IdP)-managed teams. | None             |

### astro organization user team list

List all users in a Team

#### Usage

```sh wrap theme={null}
astro organization team user list
```

### astro organization user team remove

Remove a user from a Team.

#### Usage

```sh wrap theme={null}
astro organization team user remove
```

#### Options

| Option          | Description                                                             | Valid Values     |
| --------------- | ----------------------------------------------------------------------- | ---------------- |
| `--team-id`     | The ID of the Team where you want to remove the user.                   | A valid Team ID. |
| `--user-id`     | The user's ID.                                                          | A valid user ID. |
| `-f`, `--force` | Skip the confirmation prompt for Identity Provider (IdP)-managed teams. | None             |

## Related commands

* [`astro organization user update`](/docs/cli/v1.45/astro-organization-user-update)
* [`astro workspace user add`](/docs/cli/v1.45/astro-workspace-user-add)


# astro organization token create
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-token-create

Create an Organization API token.

<Info>
  This command is only available on Astro.
</Info>

Create an Organization API token.

## Usage

```sh wrap theme={null}
astro organization token create
```

## Options

| Option           | Description                                                                | Valid Values                                                                                             |
| ---------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `--clean-output` | Print only the token as output. Use this flag in automated workflows.      | None.                                                                                                    |
| `--description`  | The description for the token.                                             | Any string surrounded by quotations.                                                                     |
| `--expiration`   | The expiration date for the token. By default there is no expiration date. | Any integer between 1 and 3650, used to represent days.                                                  |
| `--name`         | The name for the token.                                                    | Any string surrounded by quotations.                                                                     |
| `--role`         | The token's role in the Organization.                                      | Possible values are either `ORGANIZATION_MEMBER`, `ORGANIZATION_BILLING_ADMIN`, or `ORGANIZATION_OWNER`. |

## Examples

```sh wrap theme={null}
astro organization token create --name "My Org Owner API token" --role ORGANIZATION_OWNER
```

## Related commands

* [astro organization token update](/docs/cli/v1.45/astro-organization-token-update)
* [astro organization token rotate](/docs/cli/v1.45/astro-organization-token-rotate)
* [astro organization switch](/docs/cli/v1.45/astro-organization-switch)


# astro organization token delete
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-token-delete

Delete an Organization API token.

<Info>
  This command is only available on Astro.
</Info>

Delete an Organization API token.

## Usage

```sh wrap theme={null}
astro organization token delete
```

## Options

| Option    | Description                                               | Valid Values                         |
| --------- | --------------------------------------------------------- | ------------------------------------ |
| `--force` | Delete or remove the API token without showing a warning. | None.                                |
| `--name`  | The name of the token to delete.                          | Any string surrounded by quotations. |

## Related commands

* [astro organization token update](/docs/cli/v1.45/astro-organization-token-update)
* [astro organization token rotate](/docs/cli/v1.45/astro-organization-token-rotate)
* [astro organization switch](/docs/cli/v1.45/astro-organization-switch)


# astro organization token list
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-token-list

List all Organization API tokens.

<Info>
  This command is only available on Astro.
</Info>

List all Organization API tokens.

## Usage

```sh wrap theme={null}
astro organization token list
```

## Related commands

* [astro organization token update](/docs/cli/v1.45/astro-organization-token-update)
* [astro organization token rotate](/docs/cli/v1.45/astro-organization-token-rotate)
* [astro organization switch](/docs/cli/v1.45/astro-organization-switch)


# astro organization token roles
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-token-roles

List an Organization API token's roles.

<Info>
  This command is only available on Astro.
</Info>

List the roles that an Organization API token has throughout the Organization, including Workspace roles.

## Usage

```sh wrap theme={null}
astro organization token roles <token-id>
```

Retrieve the ID for a token by running `astro organization token list` or viewing Organization tokens from the Astro UI.


# astro organization token rotate
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-token-rotate

Rotate an Organization API token.

<Info>
  This command is only available on Astro.
</Info>

Rotate an Organization API token.

## Usage

```sh wrap theme={null}
astro organization token rotate <flags>
```

## Options

| Option           | Description                                                           | Valid Values                         |
| ---------------- | --------------------------------------------------------------------- | ------------------------------------ |
| `--clean-output` | Print only the token as output. Use this flag in automated workflows. | None.                                |
| `--force`        | Rotate the token without showing a warning.                           | None.                                |
| `--name`         | The name for the token.                                               | Any string surrounded by quotations. |

## Examples

```sh wrap theme={null}
astro organization token rotate --name "My token" --force
```

## Related commands

* [astro organization token update](/docs/cli/v1.45/astro-organization-token-update)
* [astro organization token delete](/docs/cli/v1.45/astro-organization-token-delete)
* [astro organization switch](/docs/cli/v1.45/astro-organization-switch)


# astro organization token update
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-token-update

Update an Organization API token.

<Info>
  This command is only available on Astro.
</Info>

Update an Organization API token.

## Usage

```sh wrap theme={null}
astro organization token update <flags>
```

## Options

| Option           | Description                                                                | Valid Values                                                                                             |
| ---------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `--clean-output` | Print only the token as output. Use this flag in automated workflows.      | None.                                                                                                    |
| `--description`  | The description for the token.                                             | Any string surrounded by quotations.                                                                     |
| `--expiration`   | The expiration date for the token. By default there is no expiration date. | Any integer between 1 and 3650, used to represent days.                                                  |
| `--name`         | The current name for the token.                                            | Any string surrounded by quotations.                                                                     |
| `--new-name`     | The updated name for the token.                                            | Any string surrounded by quotations.                                                                     |
| `--role`         | The token's role in the Organization.                                      | Possible values are either `ORGANIZATION_MEMBER`, `ORGANIZATION_BILLING_ADMIN`, or `ORGANIZATION_OWNER`. |

## Examples

```sh wrap theme={null}
astro organization token update --new-name "My updated API token" --role ORGANIZATION_MEMBER
```

## Related commands

* [astro organization token rotate](/docs/cli/v1.45/astro-organization-token-rotate)
* [astro organization token delete](/docs/cli/v1.45/astro-organization-token-delete)
* [astro organization switch](/docs/cli/v1.45/astro-organization-switch)


# astro organization user invite
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-user-invite

Invite users to your Organization.

<Info>
  This command is only available on Astro.
</Info>

Invite users to your current Astro Organization.

<Warning>This command will replace `astro user invite` in Astro CLI v1.15.0. Any instances in your projects or automation where you use `astro user invite` needs to be updated to `astro organization user invite` before the release of Astro CLI v1.15.0.</Warning>

## Usage

Run `astro organization user invite` to invite a new user to your Astronomer Organization. You can use `astro organization user invite` to invite multiple users to an Organization at a time. By default, new users are added as an `ORGANIZATION_MEMBER`. See [Add a group of users to Astro using the Astro CLI](/docs/astro/manage-organization-users#add-a-group-of-users-to-astro-using-the-astro-cli).

You must add new users to an Astro Organization before you can add them to specific Astro Workspaces. See [`astro workspace user add`](/docs/cli/v1.45/astro-workspace-user-add).

## Options

| Option    | Description                                                                                                                                                                             | Valid Values                                                                                   |
| --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| `<email>` | Specify the email for the user you want to invite or update. Use only with `astro organization user update` and `astro organization user invite`.                                       | Any valid email                                                                                |
| `--role`  | The user's role in the Organization. Use only with `astro organization user update` and `astro organization user invite`.  By default, new users are added as an `ORGANIZATION_MEMBER`. | Valid values are `ORGANIZATION_MEMBER`, `ORGANIZATION_BILLING_ADMIN`, or `ORGANIZATION_OWNER`. |

## Examples

```sh wrap theme={null}
# Invite a user to your Organization
astro organization user invite user@cosmicenergy.org --role ORGANIZATION_BILLING_ADMIN
```

## Related commands

* [`astro organization user update`](/docs/cli/v1.45/astro-organization-user-update)
* [`astro workspace user add`](/docs/cli/v1.45/astro-workspace-user-add)


# astro organization user list
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-user-list

Manage users in your Organization.

<Info>
  This command is only available on Astro.
</Info>

Manage users in your current Astro Organization.

## Usage

Run `astro organization user list` to list all users, their email, ID, organization role, and account creation date in your Organization.

## Options

| Option          | Description                                                                                          | Possible Values                |
| --------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------ |
| `--json`        | Output the response as JSON. Shorthand for `--output json`. Mutually exclusive with `-o`/`--output`. | None                           |
| `-o`,`--output` | Format the output. The default is `table`.                                                           | `table`, `json`, or `template` |
| `--template`    | Format the output using a Go template. Use with `--output template`.                                 | Any valid Go template string   |

## Output

| Output              | Description                                                                                                                                                  | Data Type                     |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------- |
| `FULLNAME`          | The full name of the user.                                                                                                                                   | String                        |
| `EMAIL`             | The email address associated with the user account.                                                                                                          | String                        |
| `ID`                | The user ID.                                                                                                                                                 | String                        |
| `ORGANIZATION ROLE` | The level of permissions granted to the user at the Organization level. Can be `ORGANIZATION_MEMBER`, `ORGANIZATION_BILLING_ADMIN`, or `ORGANIZATION_OWNER`. | String                        |
| `IDP MANAGED`       | Whether or not the user is managed through an identity provider (IdP).                                                                                       | Boolean                       |
| `CREATE DATE`       | The date the user profile was created.                                                                                                                       | Date (`YYYY-MM-DDTHH:MM:SSZ`) |

## Related commands

* [`astro organization user update`](/docs/cli/v1.45/astro-organization-user-update)


# astro organization user update
Source: https://astronomer.io/docs/cli/v1.45/astro-organization-user-update

Update users in your Organization.

<Info>
  This command is only available on Astro.
</Info>

Update user Organization roles in your current Astro Organization.

## Usage

Run `astro organization user update` to update a user's Organization role. The CLI prompts you for the user email address associated with their account.

## Options

| Option    | Description                                                                                                                                       | Valid Values                                                                                                                                 |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `<email>` | Specify the email for the user you want to invite or update. Use only with `astro organization user update` and `astro organization user invite`. | Any valid email                                                                                                                              |
| `--role`  | The user's role in the Organization. Use only with `astro organization user update` and `astro organization user invite`.                         | Valid values are either `ORGANIZATION_MEMBER`, `ORGANIZATION_BILLING_ADMIN`, or `ORGANIZATION_OWNER`.   The default is `ORGANIZATION_MEMBER` |

## Examples

```sh wrap theme={null}
# Invite a user to your Organization
astro organization user invite user@cosmicenergy.org --role ORGANIZATION_BILLING_ADMIN

# Update a user's role. The CLI prompts you for the user's email
astro organization user update --role ORGANIZATION_MEMBER
```

## Related commands

* [`astro login`](/docs/cli/v1.45/astro-login)
* [`astro organization user invite`](/docs/cli/v1.45/astro-organization-user-invite)
* [`astro organization user list`](/docs/cli/v1.45/astro-organization-user-list)


# astro otto
Source: https://astronomer.io/docs/cli/v1.45/astro-otto

Reference documentation for astro otto.

<Info>
  This command is only available on Astro.
</Info>

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

Launch Otto, Astronomer's data engineering agent, in your terminal. Otto helps you author, upgrade, debug, and manage Airflow Dags with deep context about your project and environment.

The `astro otto` command handles binary management, authentication, and local Airflow discovery automatically. All flags and arguments passed to `astro otto` are forwarded directly to Otto. For more information about Otto's capabilities, see the [Otto overview](/docs/astro/otto-overview).

## Usage

```bash wrap theme={null}
astro otto [flags/args forwarded to Otto]
astro otto update
astro otto version
```

When run without a prompt, Otto opens an interactive terminal user interface. When a prompt is provided, Otto runs in one-shot mode and exits after responding.

## Subcommands

| Subcommand           | Description                                            |
| -------------------- | ------------------------------------------------------ |
| `astro otto update`  | Update the Otto binary to the latest version           |
| `astro otto version` | Print the installed Otto version and check for updates |

The first time you run `astro otto`, the CLI downloads the Otto binary to `~/.astro/bin/otto`. On subsequent launches, the CLI checks for newer releases and applies them automatically before launching the agent. Set `otto.auto_update` to `false` to opt out and apply updates manually with `astro otto update`.

<Note>
  Otto versions independently of the Astro CLI. You don't need to upgrade the Astro CLI to pick up new Otto features or fixes. Use `astro otto update` to pull the latest Otto release at any time.
</Note>

## Options

All options are forwarded to Otto.

| Option                     | Description                                                                                                                                                 |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--mode <mode>`            | Output mode: `interactive` (default), `text`, or `json`                                                                                                     |
| `--persona <name>`         | Run the main agent as a named persona. Built-ins are `explorer` and `reviewer`. See [Personas](#personas)                                                   |
| `--continue`, `-c`         | Resume the most recent session                                                                                                                              |
| `--resume`, `-r`           | Open an interactive picker to choose a previous session                                                                                                     |
| `--session <path>`         | Open a specific session file                                                                                                                                |
| `--no-session`             | Run without saving session history                                                                                                                          |
| `--stream`                 | Force streaming on in text mode                                                                                                                             |
| `--no-stream`              | Disable streaming in text mode                                                                                                                              |
| `--output-schema`          | Require a structured final answer matching a JSON schema. Accepts a raw JSON string or `@path/to/file.json`. Headless modes only                            |
| `--allowed-tools <names>`  | Restrict active tools to a comma-separated allowlist                                                                                                        |
| `--permission-mode <mode>` | Set the starting permission mode: `default`, `acceptEdits`, `confirmEdits`, `plan`, or `bypassPermissions`. See [Otto permissions](/docs/astro/otto-permissions) |
| `--skip-permissions`       | Disable the permission layer for the session. Coerces the mode to `bypassPermissions`; bypass-immune safety checks still fire                               |
| `--extension <name>`       | Enable a bundled extension. Repeatable. See [Otto extensions](/docs/astro/otto-extensions)                                                                       |
| `--no-extension <name>`    | Disable a bundled extension. Repeatable                                                                                                                     |
| `--model <id>`             | Override the default model                                                                                                                                  |
| `--provider <name>`        | Override the model provider                                                                                                                                 |
| `--list-models [search]`   | List available models, optionally filtered                                                                                                                  |
| `--version`, `-v`          | Print the Otto version and exit                                                                                                                             |

## Usage modes

`astro otto` supports three modes via `--mode`:

* **`interactive`** (default): Full TUI with streaming output, tool call rendering, and session persistence. In a session, use `/` commands like `/airflow` (project context), `/skills` (browse and load skills), `/model` (switch the active model), `/permissions`, `/extensions`, `/bootstrap`, and `/remember`.
* **`text`**: Plain-text response on stdout. Useful for piping to other tools or shell scripts. Streams to a TTY by default and suppresses streaming when piped or redirected. Override with `--stream` or `--no-stream`.
* **`json`**: Newline-delimited JSON event stream covering messages, tool calls, tool results, and session lifecycle. Useful for programmatic integration.

### Sessions

Otto persists every session to disk by default. Use `--continue` to resume the most recent, `--resume` to pick from an interactive list, or `--session <path>` to open a specific file. Session files are stored as JSONL at `~/.astro/otto/sessions/`. Pass `--no-session` for ephemeral runs that don't save history, which is useful in CI.

### Headless mode flags

`--output-schema` (in `text` and `json` modes only) requires Otto to return a final answer matching a JSON schema. It accepts a raw JSON string or `@path/to/file.json`. `--allowed-tools <names>` restricts active tools to a comma-separated allowlist.

### Model selection

Otto supports models from OpenAI, Anthropic, and Google through the Astronomer Gateway. The exact set available to your Organization is fetched at runtime.

In an interactive session, run `/model` to browse the current list and switch mid-session. To pin a model at launch or in headless modes, pass `--model <id>` and optionally `--provider <name>`. Run `astro otto --list-models [search]` to list what's available from your shell.

### Personas

Use `--persona <name>` to run the main agent as a named persona. A persona sets the system prompt, tool allowlist, tier (which maps to a model), permission mode, and output schema when the persona declares one. Otto ships two built-in personas:

* **`explorer`**: Tuned for read-only exploration of a project.
* **`reviewer`**: Tuned for code review. The Otto review action runs `astro otto --persona reviewer` to review pull requests and merge requests. See [Review code with Otto](/docs/astro/otto-code-review).

Explicit `--model`, `--allowed-tools`, `--permission-mode`, and `--output-schema` flags override the persona's defaults. A persona-supplied output schema is dropped in interactive mode, which has no structured-output surface.

## Configuration

`astro otto` sets `ASTRO_TOKEN`, `ASTRO_DOMAIN`, `ASTRO_ORGANIZATION`, and the local `AIRFLOW_*` variables automatically. For the full list of environment variables, config files, and settings precedence, see [Otto settings](/docs/astro/otto-settings).

## Examples

```bash wrap theme={null}
# Launch the interactive TUI
astro otto

# Start the TUI with an initial prompt
astro otto "summarize this Airflow project"

# One-shot question in text mode
astro otto --mode text "describe the Dags in this project"

# One-shot without saving session history
astro otto --mode text --no-session "what Airflow version am I running?"

# Resume the most recent session
astro otto --continue

# List available models
astro otto --list-models

# Use a smaller model
astro otto --model gpt-5.4-mini

# Update Otto to the latest version
astro otto update

# Check the installed Otto version
astro otto version
```

## Related commands

* [`astro login`](/docs/cli/v1.45/astro-login)
* [`astro dev start`](/docs/cli/v1.45/astro-dev-start)
* [`astro deploy`](/docs/cli/v1.45/astro-deploy)


# astro remote
Source: https://astronomer.io/docs/cli/v1.45/astro-remote

Build and deploy remote client images for Remote Execution.

<Info>
  This command is only available on Astro.
</Info>

The `astro remote` command group enables workflows for building and deploying remote client images as part of Remote Execution Deployments.

## astro remote deploy

Builds and deploys a remote client image from your Astro project to a specified remote registry.

### Usage

```sh wrap theme={null}
astro remote deploy
```

### Description

Builds and deploys a remote client image from your Astro project to a specified remote registry. The command blocks deployment if your client image uses a newer version of Astro Runtime than the target Deployment environment to ensure compatibility.

### Options

| Option               | Description                                                                                                                                                                                                                    | Possible Values                                           |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- |
| `--build-secret`     | Mimics `docker build --secret` flag. Repeat the flag to mount more than one secret. Replaces the deprecated `--build-secrets` flag. See [Build Secrets](https://docs.docker.com/build/building/secrets/) for more information. | String array Example input: `id=mysecret,src=secrets.txt` |
| `-h`, `--help`       | Help for the command.                                                                                                                                                                                                          | N/A                                                       |
| `-i`, `--image-name` | Name of a custom image to deploy, or image name with custom tag. The image must be present on the local machine.                                                                                                               | String (image name)                                       |
| `--platform`         | Target platform for client image build. Defaults to host machine platform.                                                                                                                                                     | String. For example, `linux/amd64` or`linux/arm64`        |

### Examples

Deploy with a specific platform:

```sh wrap theme={null}
$ astro remote deploy --platform linux/amd64,linux/arm64
```

Deploy a pre-built image:

```sh wrap theme={null}
$ astro remote deploy --image-name my-custom-image:tag

```

Deploy with build secrets:

```sh wrap theme={null}
$ astro remote deploy --build-secret id=mysecret,src=secrets.txt
```


# astro run
Source: https://astronomer.io/docs/cli/v1.45/astro-run

Reference documentation for astro run.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Trigger a single dag run in a local Airflow environment and see task success or failure in your terminal. This command compiles your dag and runs it in a single Airflow worker container based on your Astro project configurations.

For more information, see [Test your Astro project locally](/docs/cli/v1.45/test-your-astro-project-locally).

## Usage

```sh wrap theme={null}
astro run <dag-id>
```

## Options

| Option                  | Description                                                                                                                                              | Possible Values                                                                                      |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `-d`, `--dag-file`      | The location of your dag file. When you specify this flag, only the specified dag is parsed by the Astro CLI. All other dags in the project are ignored. | Any valid dag file in your `dags` directory.                                                         |
| `-e`,`--env`            | Path to an alternative environment variable file. The default is `.env` in your current Astro project.                                                   | Any valid filepath.                                                                                  |
| `--execution-date`      | The execution date for the dag run.                                                                                                                      | An execution date formatted as either `YYYY-MM-DD`, `YYYY-MM-DDTHH:MM:SS`. or `YYYY-MM-DD HH:MM:SS`. |
| `--no-cache`            | Build your Astro project into a Docker image without using cache.                                                                                        | None.                                                                                                |
| `-s`, `--settings-file` | An alternative settings file from which Airflow objects are imported. The default is `airflow_settings.yaml` in your current Astro project.              | Any valid filepath.                                                                                  |

## Examples

```sh wrap theme={null}
# Run a dag with an alternative set of environment variables
$ astro run example_dag_basic --env dev.env
```

## Related commands

* [`astro dev start`](/docs/cli/v1.45/astro-dev-start)
* [`astro dev restart`](/docs/cli/v1.45/astro-dev-restart)
* [`astro dev stop`](/docs/cli/v1.45/astro-dev-stop)
* [`astro deploy`](/docs/cli/v1.45/astro-deploy)


# astro team
Source: https://astronomer.io/docs/cli/v1.45/astro-team

Reference documentation for astro team.

<Info>This command is available only if you're authenticated to an Astro Private Cloud installation.</Info>

Manage system-level Teams on Astro Private Cloud. See [Import identity provider groups into Astro Private Cloud](https://www.astronomer.io/docs/software/import-idp-groups).

## Usage

This command has several subcommands. Read the following sections to learn how to use each subcommand.

### astro team get

View information for a single Team.

#### Usage

```sh wrap theme={null}
astro team get <team-id>
```

You can retrieve a Team's ID in one of two ways:

* Access the Team in the UI and copy the last part of the URL in your web browser. For example, if your Team is located at `BASEDOMAIN.astronomer.io/w/cx897fds98csdcsdafasdot8g7/team/cl4iqjamcnmfgigl4852flfgulye`, your Team ID would be `cl4iqjamcnmfgigl4852flfgulye`.
* Run [`astro team list`](#astro-team-list) and copy the value in the `ID` column.

#### Options

| Option          | Description                         | Possible Values |
| --------------- | ----------------------------------- | --------------- |
| `-a`, `--all`   | View all information about the Team | None            |
| `-r`, `--roles` | View role details for the Team      | None            |
| `-u`, `--users` | View all users in the Team          | None            |

### astro team list

List all Teams on Astro Private Cloud.

#### Usage

```sh wrap theme={null}
astro team list
```

#### Options

| Option              | Description                                                                                 | Possible Values |
| ------------------- | ------------------------------------------------------------------------------------------- | --------------- |
| `-a`, `--all`       | View all information about the Team                                                         | None            |
| `-p` `--paginated ` | Paginate the list of Teams. If `--page-size` is not specified, the default page size is 20. | None            |
| `-s` `--page-size`  | The page size for paginated lists.                                                          | Any integer     |

### astro team update

Update an Astro Team's system-level role.

#### Usage

```sh wrap theme={null}
astro team update <team-id> --role=<system-role>
```

You can retrieve a Team's ID in one of two ways:

* Access the Team in the UI and copy the last part of the URL in your web browser. For example, if your Team is located at `BASEDOMAIN.astronomer.io/w/cx897fds98csdcsdafasdot8g7/team/cl4iqjamcnmfgigl4852flfgulye`, your Team ID is `cl4iqjamcnmfgigl4852flfgulye`.
* Run [`astro team list`](#astro-team-list) and copy the value in the `ID` column.

#### Options

| Option        | Description                  | Possible Values                                                          |
| ------------- | ---------------------------- | ------------------------------------------------------------------------ |
| `-r` `--role` | The Team's system-level role | Possible values are `SYSTEM_VIEWER`, `SYSTEM_EDITOR`, or `SYSTEM_ADMIN`. |


# astro telemetry
Source: https://astronomer.io/docs/cli/v1.45/astro-telemetry

Manage anonymous telemetry for the Astro CLI.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Use `astro telemetry` commands to manage anonymous usage telemetry for the Astro CLI. Run `astro telemetry` without a subcommand to show the current telemetry status.

Telemetry is enabled by default. You can also disable telemetry by setting the `ASTRO_TELEMETRY_DISABLED` environment variable to `1`.

## Usage

```sh wrap theme={null}
astro telemetry
```

<CardGroup>
  <Card title="astro telemetry enable" href="/cli/v1.45/astro-telemetry-enable">
    View documentation for `astro telemetry enable`.
  </Card>

  <Card title="astro telemetry disable" href="/cli/v1.45/astro-telemetry-disable">
    View documentation for `astro telemetry disable`.
  </Card>
</CardGroup>


# astro telemetry disable
Source: https://astronomer.io/docs/cli/v1.45/astro-telemetry-disable

Disable anonymous telemetry for the Astro CLI.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Disable anonymous usage telemetry for the Astro CLI. You can also disable telemetry by setting the `ASTRO_TELEMETRY_DISABLED` environment variable to `1`.

## Usage

```sh wrap theme={null}
astro telemetry disable
```

## Related commands

* [astro telemetry enable](/docs/cli/v1.45/astro-telemetry-enable)


# astro telemetry enable
Source: https://astronomer.io/docs/cli/v1.45/astro-telemetry-enable

Enable anonymous telemetry for the Astro CLI.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Enable anonymous usage telemetry for the Astro CLI. Telemetry is enabled by default.

## Usage

```sh wrap theme={null}
astro telemetry enable
```

## Related commands

* [astro telemetry disable](/docs/cli/v1.45/astro-telemetry-disable)


# astro user create
Source: https://astronomer.io/docs/cli/v1.45/astro-user-create

Reference documentation for astro user create.

<Info>This command is available only if you're authenticated to an Astro Private Cloud installation.</Info>

Create a new user profile on Astro Private Cloud.

## Usage

```sh wrap theme={null}
astro user create
```

## Options

| Option             | Description               | Possible Values         |
| ------------------ | ------------------------- | ----------------------- |
| `-e`, `--email`    | The email for the user    | Any valid email address |
| `-p`, `--password` | The password for the user | Any string              |


# astro version
Source: https://astronomer.io/docs/cli/v1.45/astro-version

Reference documentation for astro version.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Show the running version of the Astro CLI. Note that this command does not show whether your CLI is up to date. To check the latest version of the Astro CLI, see the [CLI Release Notes](/docs/cli/v1.45/release-notes).

## Usage

```sh wrap theme={null}
astro version
```


# astro workspace
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace

Manage Workspaces and Workspace-level user permissions.

Use `astro workspace` commands to manage Workspaces and Workspace-level user permissions.

<CardGroup>
  <Card title="astro workspace create" href="/cli/v1.45/astro-workspace-create">
    View documentation for `astro workspace create`.
  </Card>

  <Card title="astro workspace delete" href="/cli/v1.45/astro-workspace-delete">
    View documentation for `astro workspace delete`.
  </Card>

  <Card title="astro workspace list" href="/cli/v1.45/astro-workspace-list">
    View documentation for `astro workspace list`.
  </Card>

  <Card title="astro workspace service account" href="/cli/v1.45/astro-workspace-service-account">
    View documentation for `astro workspace service account`.
  </Card>

  <Card title="astro workspace switch" href="/cli/v1.45/astro-workspace-switch">
    View documentation for `astro workspace switch`.
  </Card>

  <Card title="astro workspace team add" href="/cli/v1.45/astro-workspace-team-add">
    View documentation for `astro workspace team add`.
  </Card>

  <Card title="astro workspace team list" href="/cli/v1.45/astro-workspace-team-list">
    View documentation for `astro workspace team list`.
  </Card>

  <Card title="astro workspace team remove" href="/cli/v1.45/astro-workspace-team-remove">
    View documentation for `astro workspace team remove`.
  </Card>

  <Card title="astro workspace team update" href="/cli/v1.45/astro-workspace-team-update">
    View documentation for `astro workspace team update`.
  </Card>

  <Card title="astro workspace token add" href="/cli/v1.45/astro-workspace-token-add">
    View documentation for `astro workspace token add`.
  </Card>

  <Card title="astro workspace token create" href="/cli/v1.45/astro-workspace-token-create">
    View documentation for `astro workspace token create`.
  </Card>

  <Card title="astro workspace token list" href="/cli/v1.45/astro-workspace-token-list">
    View documentation for `astro workspace token list`.
  </Card>

  <Card title="astro workspace token organization token" href="/cli/v1.45/astro-workspace-token-organization-token">
    View documentation for `astro workspace token organization token`.
  </Card>

  <Card title="astro workspace token rotate" href="/cli/v1.45/astro-workspace-token-rotate">
    View documentation for `astro workspace token rotate`.
  </Card>

  <Card title="astro workspace token update" href="/cli/v1.45/astro-workspace-token-update">
    View documentation for `astro workspace token update`.
  </Card>

  <Card title="astro workspace update" href="/cli/v1.45/astro-workspace-update">
    View documentation for `astro workspace update`.
  </Card>

  <Card title="astro workspace user add" href="/cli/v1.45/astro-workspace-user-add">
    View documentation for `astro workspace user add`.
  </Card>

  <Card title="astro workspace user list" href="/cli/v1.45/astro-workspace-user-list">
    View documentation for `astro workspace user list`.
  </Card>

  <Card title="astro workspace user remove" href="/cli/v1.45/astro-workspace-user-remove">
    View documentation for `astro workspace user remove`.
  </Card>

  <Card title="astro workspace user update" href="/cli/v1.45/astro-workspace-user-update">
    View documentation for `astro workspace user update`.
  </Card>
</CardGroup>


# astro workspace create
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-create

Create an Astro Workspace.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change product contexts.</Info>

Create a Workspace.

<Tabs>
  <Tab title="Astro">
    ## Usage

    ```sh wrap theme={null}
    astro workspace create <options>
    ```

    ## Options

    | Option            | Description                                                               | Valid Values  |
    | ----------------- | ------------------------------------------------------------------------- | ------------- |
    | `--description`   | The description for the Workspace.                                        | Any string    |
    | `--enforce-ci-cd` | Determines whether users are required to use an API token to deploy code. | `ON` or `OFF` |
    | `--name`          | The name for the Workspace.                                               | Any string    |

    ## Examples

    ```sh wrap theme={null}
    $ astro workspace create --name "My Deployment" --enforce-ci-cd ON
    ```

    ## Related commands

    * [`astro workspace update`](/docs/cli/v1.45/astro-workspace-update)
    * [`astro workspace delete`](/docs/cli/v1.45/astro-workspace-delete)
    * [`astro workspace user update`](/docs/cli/v1.45/astro-workspace-user-update)
  </Tab>

  <Tab title="APC">
    ## Usage

    ```sh wrap theme={null}
    astro workspace create <options>
    ```

    ## Options

    | Option          | Description                        | Valid Values |
    | --------------- | ---------------------------------- | ------------ |
    | `--description` | The description for the Workspace. | Any string   |
    | `--label`       | The label for the Workspace.       | Any string   |

    ## Examples

    ```sh wrap theme={null}
    $ astro workspace create --label "My Deployment" --description "My new Deployment"
    ```

    ## Related commands

    * [`astro workspace update`](/docs/cli/v1.45/astro-workspace-update)
    * [`astro workspace delete`](/docs/cli/v1.45/astro-workspace-delete)
    * [`astro workspace user update`](/docs/cli/v1.45/astro-workspace-user-update)
  </Tab>
</Tabs>


# astro workspace delete
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-delete

Delete an Astro Workspace.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Delete an Astro Workspace.

## Usage

```sh wrap theme={null}
astro workspace delete <workspace-id>
```

You can find a Workspace's ID by running `astro workspace list`. Alternatively, in Astro, you can find a Workspace ID by opening the Workspace and going to **Workspace Settings** > **General** in the Astro UI. If you don't provide a Workspace ID, the CLI prompts you to pick from a list of Workspaces that you belong to in your current Organization.

## Related commands

* [`astro workspace update`](/docs/cli/v1.45/astro-workspace-update)
* [`astro deployment delete`](/docs/cli/v1.45/astro-deployment-delete)


# astro workspace list
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-list

List Workspaces.

<Info>The behavior and format of this command are the same for both Astro and Astro Private Cloud.</Info>

Generates a list of all Workspaces within your current Organization that you have access to.

## Usage

Run `astro workspace list` to see the name and Workspace ID for each Workspace to which you have access.

## Options

| Option          | Description                                                                                          | Possible Values                |
| --------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------ |
| `--json`        | Output the response as JSON. Shorthand for `--output json`. Mutually exclusive with `-o`/`--output`. | None                           |
| `-o`,`--output` | Format the output. The default is `table`.                                                           | `table`, `json`, or `template` |
| `--template`    | Format the output using a Go template. Use with `--output template`.                                 | Any valid Go template string   |

## Output

| Output | Description                                                              | Data Type |
| ------ | ------------------------------------------------------------------------ | --------- |
| `NAME` | The name of the Workspaces in your Organization that you have access to. | String    |
| `ID`   | The Workspace ID.                                                        | String    |

## Related commands

* [`astro workspace switch`](/docs/cli/v1.45/astro-workspace-switch)


# astro workspace service-account
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-service-account

Manage Workspace service accounts (Astro Private Cloud only).

<Info>This command is available only if you're authenticated to an Astro Private Cloud installation.</Info>

## Usage

This command has several subcommands.

### astro workspace service-account create

Creates a service account for a given Workspace.

#### Usage

```sh wrap theme={null}
astro workspace service-account create --workspace-id=<your-workspace> --label=<your-label>
```

#### Options

| Option                      | Description                                                     | Possible Values                                                                                                         |
| --------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `--workspace-id` (Required) | The Workspace you're creating a service account for.            | Any valid Workspace ID                                                                                                  |
| `--label` (Required)        | A label for the service account.                                | Any string                                                                                                              |
| `--category`                | The Category for the service account. The default is `Not set`. | Any string                                                                                                              |
| `role`                      | The User Role for the service account.                          | Possible values are `WORKSPACE_VIEWER`, `WORKSPACE_EDITOR`, `WORKSPACE_ADMIN`. The default value is `WORKSPACE_VIEWER`. |

#### Related documentation

* [Manage Workspaces and Deployments on Astronomer](/docs/astro-private-cloud/v-0-37/manage-workspaces)

### astro workspace service-account delete

Deletes a service account for a given Workspace.

#### Usage

```sh wrap theme={null}
astro workspace service-account delete <your-service-account-id>
```

#### Options

| Option           | Description                                                                                                                                                                                                                               | Possible Values        |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
| `--workspace-id` | The Workspace in which you want to delete a service account. If this flag is used instead of specifying `<your-service-account-id>`, you'll be prompted to select a service account from a list of all service accounts on the Workspace. | Any valid Workspace ID |

#### Related documentation

* [Manage Workspaces and Deployments on Astronomer](/docs/astro-private-cloud/v-0-37/manage-workspaces)

### astro workspace service-account get

Shows the name, ID, and API token for each service account on a given Workspace.

#### Usage

Run `astro deployment service-account get <service-account-id> --workspace-id=<your-workspace-id>` to get information on a single service account within a Workspace. To see a list of all service accounts on a Workspace, run `astro deployment service-account get --workspace-id=<your-workspace-id>`.

#### Options

| Option           | Description                                                                                                                       | Possible Values        |
| ---------------- | --------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
| `--workspace-id` | The Workspace you're getting the service account from. Use this flag as an alternative to specifying `<your-service-account-id>`. | Any valid Workspace ID |

#### Related documentation

* [Manage Workspaces and Deployments on Astronomer](/docs/astro-private-cloud/v-0-37/manage-workspaces)


# astro workspace switch
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-switch

Switch between Workspaces.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change product contexts.</Info>

Switch between Workspaces.

<Tabs>
  <Tab title="Astro">
    ## Usage

    ```sh wrap theme={null}
    astro workspace switch
    ```

    ## Options

    | Option                      | Description                                                                                         | Valid Values           |
    | --------------------------- | --------------------------------------------------------------------------------------------------- | ---------------------- |
    | `<workspace_id>` (Optional) | The ID of the Workspace you want to switch to. If not specified, the CLI prompts you to choose one. | Any valid Workspace ID |

    ## Example

    Run `astro workspace switch <workspace-id>` to switch between Workspaces.

    ```sh wrap theme={null}
    astro workspace switch clqw6uskr000008l9370y04jd
    ```

    You can find a Workspace's ID by running `astro workspace list`, or by opening your Workspace and going to **Workspace Settings** > **General** in the Astro UI. On Astro, if you don't provide a Workspace ID, the CLI prompts you to pick from a list of Workspaces that you belong to in your current Organization.

    ## Related commands

    * [`astro workspace list`](/docs/cli/v1.45/astro-workspace-list)
  </Tab>

  <Tab title="APC">
    ## Usage

    ```sh wrap theme={null}
    astro workspace switch <options>
    ```

    You can find a Workspace's ID by running `astro workspace list`.

    ## Options

    | Option              | Description                                                                                  | Valid Values           |
    | ------------------- | -------------------------------------------------------------------------------------------- | ---------------------- |
    | `<workspace_id>`    | The ID of the workspace you want to switch to. Otherwise, the CLI prompts you to choose one. | Any valid Workspace ID |
    | `-p`, `--paginated` | Choose whether or not to paginate the list of available Workspaces.                          | `TRUE` or `FALSE`      |
    | `-s`, `--page-size` | The length of the list per page when paginate is set to `TRUE`.                              | Any integer            |

    ## Related commands

    * [`astro workspace list`](/docs/cli/v1.45/astro-workspace-list)
  </Tab>
</Tabs>


# astro workspace team add
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-team-add

Add a Team to your Workspace.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change between product contexts.</Info>

Add a Team to your current Workspace and grant it a Workspace role.

<Tabs>
  <Tab title="Astro">
    ## Usage

    ```sh wrap theme={null}
    astro workspace team add <options>
    ```

    If you run the command with no options specified, the CLI lists Teams in your Organization and prompts you to select one. It then adds the Team to your current Workspace with the Workspace Member role.

    To add a Team to a specific Workspace, specify the Workspace using the `--workspace-id` flag. To find a Workspace ID, run `astro workspace list`.

    #### Options

    | Option           | Description                                                               | Valid Values                                                                                                                                                 |
    | ---------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
    | `--team-id`      | The ID for the Team to add to a Workspace. Bypasses the selection prompt. | Any valid Team ID. You can find available Team ID's by running `astro organization team list`.                                                               |
    | `-r`, `--role`   | The Team's role in the Workspace.                                         | Possible values are `WORKSPACE_ACCESSOR`, `WORKSPACE_MEMBER`, `WORKSPACE_AUTHOR`, `WORKSPACE_OPERATOR`, or `WORKSPACE_OWNER`. Default is `WORKSPACE_MEMBER`. |
    | `--workspace-id` | The ID for the Workspace where you want to add the Team.                  | Any valid Workspace ID. Default is the current Workspace context you are working in.                                                                         |
  </Tab>

  <Tab title="APC">
    Manage Astro Private Cloud [Teams](https://www.astronomer.io/docs/software/import-idp-groups).

    ## Usage

    If you want to add a team to the current Workspace with the default role of Workspace Member, you can run `astro workspace team add <team-id>`, with the team ID that you want to add.

    ```sh wrap theme={null}
    astro workspace team add <team-id>
    ```

    You can retrieve a Team's ID in one of two ways:

    * Access the Team in the UI and copy the last part of the URL in your web browser. For example, if your Team is located at `BASEDOMAIN.astronomer.io/w/cx897fds98csdcsdafasdot8g7/team/cl4iqjamcnmfgigl4852flfgulye`, your Team ID would be `cl4iqjamcnmfgigl4852flfgulye`.
    * Run `astro organization team list` and copy the value in the ID column

    #### Options

    | Option                   | Description                       | Possible Values                                                                                                                        |
    | ------------------------ | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
    | `<team-id>` (*Required*) | The Team's ID                     | Any valid Team ID                                                                                                                      |
    | `--role`                 | The Team's role in the Workspace. | Possible values are `WORKSPACE_MEMBER`, `WORKSPACE_AUTHOR`, `WORKSPACE_OPERATOR`, or `WORKSPACE_OWNER`. Default is `WORKSPACE_MEMBER`. |
  </Tab>
</Tabs>

## Related commands

* [`astro workspace team remove`](/docs/cli/v1.45/astro-workspace-team-remove)
* [`astro organization team create`](/docs/cli/v1.45/astro-organization-team-create)
* [`astro workspace switch`](/docs/cli/v1.45/astro-workspace-switch)


# astro workspace team list
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-team-list

List Teams in a Workspace.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change product contexts.</Info>

List all Teams in your current Workspace, as well as their level of user permissions within the Workspace.

<Tabs>
  <Tab title="Astro">
    ## Usage

    ```sh wrap theme={null}
    astro workspace team list
    ```

    ## Options

    | Option          | Description                                                                                          | Possible Values                |
    | --------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------ |
    | `--json`        | Output the response as JSON. Shorthand for `--output json`. Mutually exclusive with `-o`/`--output`. | None                           |
    | `-o`,`--output` | Format the output. The default is `table`.                                                           | `table`, `json`, or `template` |
    | `--template`    | Format the output using a Go template. Use with `--output template`.                                 | Any valid Go template string   |

    ## Output

    | Output        | Description                                                      | Data Type                     |
    | ------------- | ---------------------------------------------------------------- | ----------------------------- |
    | `ID`          | The Team ID.                                                     | String                        |
    | `Role`        | The Team's role in the Workspace.                                | String                        |
    | `Name`        | The name of the Team.                                            | String                        |
    | `Description` | The Team description.                                            | String                        |
    | `Create Date` | The date and time that the Team was created in the Organization. | Date (`YYYY-MM-DDTHH:MM:SSZ`) |

    ## Related commands

    * [`astro workspace team add`](/docs/cli/v1.45/astro-workspace-team-add)
    * [`astro organization team create`](/docs/cli/v1.45/astro-organization-team-create)
    * [`astro organization team list`](/docs/cli/v1.45/astro-organization-team-list)
    * [`astro workspace switch`](/docs/cli/v1.45/astro-workspace-switch)
  </Tab>

  <Tab title="APC">
    ## Usage

    ```sh wrap theme={null}
    astro workspace team list <options>
    ```

    ## Options

    | Option           | Description                               | Valid Values           |
    | ---------------- | ----------------------------------------- | ---------------------- |
    | `<workspace_id>` | The Workspace you want to list Teams for. | Any valid Workspace ID |

    ## Output

    | Output        | Description                                                      | Data Type                     |
    | ------------- | ---------------------------------------------------------------- | ----------------------------- |
    | `ID`          | The Team ID.                                                     | String                        |
    | `Role`        | The Team's role in the Workspace.                                | String                        |
    | `Name`        | The name of the Team.                                            | String                        |
    | `Description` | The Team description.                                            | Boolean                       |
    | `Create Date` | The date and time that the Team was created in the Organization. | Date (`YYYY-MM-DDTHH:MM:SSZ`) |

    ## Related commands

    * [`astro workspace team add`](/docs/cli/v1.45/astro-workspace-team-add)
    * [`astro organization team create`](/docs/cli/v1.45/astro-organization-team-create)
    * [`astro organization team list`](/docs/cli/v1.45/astro-organization-team-list)
    * [`astro workspace switch`](/docs/cli/v1.45/astro-workspace-switch)
  </Tab>
</Tabs>


# astro workspace team remove
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-team-remove

Remove Teams from a Workspace.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change between product contexts.</Info>

Remove a Team from your current Workspace.

<Tabs>
  <Tab title="Astro">
    ## Usage

    ```sh wrap theme={null}
    astro workspace team remove
    ```

    When you remove a team from Astro, the CLI displays a list of Teams in the Workspace and prompts you to select the team to remove.
  </Tab>

  <Tab title="APC">
    ## Usage

    ```sh wrap theme={null}
    astro workspace team remove <team-id> --workspace-id <workspace-id>
    ```

    You can retrieve a Team's ID in one of two ways:

    * Access the Team in the UI and copy the last part of the URL in your web browser. For example, if your Team is located at `BASEDOMAIN.astronomer.io/w/cx897fds98csdcsdafasdot8g7/team/cl4iqjamcnmfgigl4852flfgulye`, your Team ID would be `cl4iqjamcnmfgigl4852flfgulye`.
    * Run [`astro team list`](/docs/cli/v1.45/astro-team#astro-team-list) and copy the value in the `ID` column.

    #### Options

    | Option                        | Description                 | Possible Values                                                       |
    | ----------------------------- | --------------------------- | --------------------------------------------------------------------- |
    | `<team-id>`                   | The ID for the Team.        | Any valid Team ID. To retrieve a Team ID, run `astro workspace list`. |
    | `--workspace-id` (*Required*) | The Workspace for the Team. | Any valid Workspace ID                                                |
  </Tab>
</Tabs>

## Related commands

* [`astro workspace team add`](/docs/cli/v1.45/astro-workspace-team-add)
* [`astro organization team remove`](/docs/cli/v1.45/astro-organization-team-delete)


# astro workspace team update
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-team-update

Update a Team in a Workspace.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change between product contexts.</Info>

Update a Team's permissions in a given Workspace.

<Tabs>
  <Tab title="Astro">
    ## Usage

    ```sh wrap theme={null}
    astro workspace team update <team-id> --workspace-id <workspace-id> --role=<system-role>
    ```

    To find a Team ID using the Astro CLI, run `astro workspace team list`.

    To find a Team ID in the Astro UI, click **Organization Settings** > **Access Management** > **Teams**. Search for your Team in the **Teams** table and copy its **ID**. The ID should look something like `clk17xqgm124q01hkrgilsr49`.

    #### Options

    | Option           | Description                                              | Possible Values                                                                                                               |
    | ---------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
    | `--workspace-id` | The Workspace for the Team. Use to override CLI prompts. | Any valid Workspace ID.                                                                                                       |
    | `<team-id>`      | The ID for the Team. Use to override CLI prompts.        | Any valid Team ID. To retrieve a Team ID, run `astro workspace team list`.                                                    |
    | `--role`         | The Team's role in the Workspace.                        | Possible values are `WORKSPACE_ACCESSOR`,`WORKSPACE_MEMBER`, `WORKSPACE_AUTHOR`, `WORKSPACE_OPERATOR`, and `WORKSPACE_OWNER`. |
  </Tab>

  <Tab title="APC">
    ## Usage

    ```sh wrap theme={null}
    astro workspace team update <team-id> --workspace-id <workspace-id> --role=<system-role>
    ```

    To find a Team ID using the Astro CLI, run `astro workspace team list`. You can also access the Team in the UI and copy the last part of the URL in your web browser. For example, if your Team is located at `BASEDOMAIN.astronomer.io/w/cx897fds98csdcsdafasdot8g7/team/cl4iqjamcnmfgigl4852flfgulye`, your Team ID would be `cl4iqjamcnmfgigl4852flfgulye`.

    #### Related documentation

    * [Import identity provider groups into Astro Private Cloud](https://www.astronomer.io/docs/software/import-idp-groups).

    #### Options

    | Option                        | Description                       | Possible Values                                                                                                  |
    | ----------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
    | `--workspace-id` (*Required*) | The Workspace for the Team        | Any valid Workspace ID.                                                                                          |
    | `<team-id>` (*Required*)      | The Team's ID.                    | Any valid Team ID.                                                                                               |
    | `--role`                      | The Team's role in the Workspace. | Possible values are `WORKSPACE_VIEWER`, `WORKSPACE_EDITOR`, or `WORKSPACE_ADMIN`. Default is `WORKSPACE_VIEWER`. |
  </Tab>
</Tabs>

## Related commands

* [`astro workspace team remove`](/docs/cli/v1.45/astro-workspace-team-remove)
* [`astro organization team create`](/docs/cli/v1.45/astro-organization-team-create)
* [`astro workspace switch`](/docs/cli/v1.45/astro-workspace-switch)


# astro workspace token add
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-token-add

Add an Organization API token to your current Workspace.

<Info>
  This command is only available on Astro.
</Info>

Add an Organization API token to your current Workspace and grant it Workspace permissions.

## Usage

```sh wrap theme={null}
astro workspace token add
```

## Options

| Option             | Description                                                               | Valid Values                                                                              |
| ------------------ | ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `--org-token-name` | The name of the Organization API token you want to add to your Workspace. | Any string enclosed in quotations                                                         |
| `--role`           | The Workspace role to grant to the Organization API token.                | One of `WORKSPACE_MEMBER`, `WORKSPACE_AUTHOR`, `WORKSPACE_OPERATOR` or `WORKSPACE_OWNER`. |

## Related commands

* [astro workspace token update](/docs/cli/v1.45/astro-workspace-token-update)
* [astro workspace token rotate](/docs/cli/v1.45/astro-workspace-token-rotate)
* [astro workspace switch](/docs/cli/v1.45/astro-workspace-switch)


# astro workspace token create
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-token-create

Create a Workspace API token.

<Info>
  This command is only available on Astro.
</Info>

Create a Workspace API token in your current Workspace.

## Usage

```sh wrap theme={null}
astro workspace token create
```

## Options

| Option           | Description                                                                | Valid Values                                                                               |
| ---------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `--clean-output` | Print only the token as output. Use this flag in automated workflows.      | None                                                                                       |
| `--description`  | The description for the token                                              | Any string surrounded by quotations                                                        |
| `--expiration`   | The expiration date for the token. By default there is no expiration date. | Any integer between 1 and 3650, used to represent days                                     |
| `--name`         | The name for the token.                                                    | Any string surrounded by quotations                                                        |
| `--role`         | The token's role in the Workspace.                                         | One of `WORKSPACE_MEMBER`, `WORKSPACE_AUTHOR`, `WORKSPACE_OPERATOR`, or `WORKSPACE_OWNER`. |

## Examples

```sh wrap theme={null}
astro workspace token create --name "My production API token" --role WORKSPACE_MEMBER
```

## Related commands

* [astro workspace token update](/docs/cli/v1.45/astro-workspace-token-update)
* [astro workspace token rotate](/docs/cli/v1.45/astro-workspace-token-rotate)
* [astro workspace switch](/docs/cli/v1.45/astro-workspace-switch)


# astro workspace token delete
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-token-delete

Delete a Workspace API token.

<Info>
  This command is only available on Astro.
</Info>

Delete a Workspace API token or remove an Organization API token in your current Workspace.

## Usage

```sh wrap theme={null}
astro workspace token delete
```

## Options

| Option    | Description                                               | Valid Values                        |
| --------- | --------------------------------------------------------- | ----------------------------------- |
| `--force` | Delete or remove the API token without showing a warning. | None                                |
| `--name`  | The name of the token to delete.                          | Any string surrounded by quotations |

## Related commands

* [astro workspace token update](/docs/cli/v1.45/astro-workspace-token-update)
* [astro workspace token rotate](/docs/cli/v1.45/astro-workspace-token-rotate)
* [astro workspace switch](/docs/cli/v1.45/astro-workspace-switch)


# astro workspace token list
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-token-list

List Workspace API tokens.

<Info>
  This command is only available on Astro.
</Info>

List all Workspace API tokens in your current Workspace.

## Usage

```sh wrap theme={null}
astro workspace token list
```

## Related commands

* [astro workspace token update](/docs/cli/v1.45/astro-workspace-token-update)
* [astro workspace token rotate](/docs/cli/v1.45/astro-workspace-token-rotate)
* [astro workspace switch](/docs/cli/v1.45/astro-workspace-switch)


# astro workspace token organization-token
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-token-organization-token

Scope an Organization token to a specific Workspace.

<Info>
  This command is only available on Astro.
</Info>

Manage Organization-level API tokens within a specific Workspace. See [Assign an Organization API token to a Workspace](/docs/astro/workspace-api-tokens#assign-an-organization-api-token-to-a-workspace).

There are four sub-commands for managing a Workspace-scoped Organization token.

## astro workspace organization-token add

Add an Organization API token to a Workspace and grant it Workspace-specific permissions.

### Usage

```sh wrap theme={null}
astro workspace organization-token add --org-token-name=<organization-token-name> --role=<workspace-role>
```

### Options

| Option                   | Description                                  | Valid Values                                                                              |
| ------------------------ | -------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `-n`, `--org-token-name` | The name of the Organization API token.      | Any string. If the name contains a space, specify the entire name within quotes `""`.     |
| `-r`, `--role`           | The role the API token has in the Workspace. | One of `WORKSPACE_MEMBER`, `WORKSPACE_AUTHOR`, `WORKSPACE_OPERATOR` or `WORKSPACE_OWNER`. |
| `--workspace-id`         | The Workspace to add the token to.           | Any Workspace ID.                                                                         |

### Example

```sh wrap theme={null}
astro workspace organization-token add --org-token-name="My Organization" --role=WORKSPACE_OWNER
```

## astro workspace organization-token list

List all Organization API tokens that are assigned to a specific Workspace.

### Usage

```sh wrap theme={null}
astro workspace organization-token list
```

### Options

| Option           | Description                                   | Valid Values      |
| ---------------- | --------------------------------------------- | ----------------- |
| `--workspace-id` | The Workspace to which the API tokens belong. | Any Workspace ID. |

### Output

| Output           | Description                                      | Data Type |
| ---------------- | ------------------------------------------------ | --------- |
| `ID`             | The API token ID.                                | String    |
| `NAME`           | The name of the API token.                       | String    |
| `DESCRIPTION`    | The API token description.                       | String    |
| `SCOPE`          | The original scope of the API token.             | String    |
| `WORKSPACE_ROLE` | The API token's role in the Workspace.           | String    |
| `CREATED`        | How long ago the API token was created, in days. | String    |
| `CREATED BY`     | The name of the user who created the API token.  | String    |

## astro workspace organization-token remove

Remove an Organization API token from a Workspace.

### Usage

```sh wrap theme={null}
astro workspace organization-token remove
```

### Options

| Option                  | Description                                                                   | Valid Values      |
| ----------------------- | ----------------------------------------------------------------------------- | ----------------- |
| `-n`,`--org-token-name` | The name of the Organization API token you want to remove from the Workspace. | Any string.       |
| `--workspace-id`        | Workspace where you want to remove an API token.                              | Any Workspace ID. |

### Example

```sh wrap theme={null}
astro workspace organization-token remove
```

## astro workspace organization-token update

Update the role an Organization API token has within a Workspace.

### Usage

```sh wrap theme={null}
astro workspace organization-token update --workspace-id=<workspace-id> --role=<workspace-role>
```

### Options

| Option                   | Description                                                | Valid Values                                                                              |
| ------------------------ | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `-n`, `--org-token-name` | The name of the Organization API token you want to update. | Any string.                                                                               |
| `-r`, `--role`           | The Workspace role that you want to assign to the token.   | One of `WORKSPACE_MEMBER`, `WORKSPACE_AUTHOR`, `WORKSPACE_OPERATOR` or `WORKSPACE_OWNER`. |
| `--workspace-id`         | The Workspace where you want to update the token.          | Any Workspace ID.                                                                         |

### Example

```sh wrap theme={null}
astro workspace organization-token add --workspace-id=clvdx7z3c000008kv5tdw5tc5 --org-token-name="My organization token" --role=WORKSPACE_AUTHOR
```


# astro workspace token rotate
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-token-rotate

Rotate a Workspace API token.

<Info>
  This command is only available on Astro.
</Info>

Rotate a Workspace API token in your current Workspace.

## Usage

```sh wrap theme={null}
astro workspace token rotate <flags>
```

## Options

| Option           | Description                                                           | Valid Values                        |
| ---------------- | --------------------------------------------------------------------- | ----------------------------------- |
| `--clean-output` | Print only the token as output. Use this flag in automated workflows. | None                                |
| `--force`        | Rotate the token without showing a warning                            | None                                |
| `--name`         | The name for the token.                                               | Any string surrounded by quotations |

## Examples

```sh wrap theme={null}
astro workspace token rotate --name "My token" --force
```

## Related commands

* [astro workspace token update](/docs/cli/v1.45/astro-workspace-token-update)
* [astro workspace token delete](/docs/cli/v1.45/astro-workspace-token-delete)
* [astro workspace switch](/docs/cli/v1.45/astro-workspace-switch)


# astro workspace token update
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-token-update

Update a Workspace API token.

<Info>
  This command is only available on Astro.
</Info>

Update a Workspace API token in your current Workspace.

## Usage

```sh wrap theme={null}
astro workspace token update <flags>
```

## Options

| Option           | Description                                                                | Valid Values                                                                                                   |
| ---------------- | -------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `--clean-output` | Print only the token as output. Use this flag in automated workflows.      | None                                                                                                           |
| `--description`  | The description for the token                                              | Any string surrounded by quotations                                                                            |
| `--expiration`   | The expiration date for the token. By default there is no expiration date. | Any integer between 1 and 3650, used to represent days                                                         |
| `--name`         | The current name for the token.                                            | Any string surrounded by quotations                                                                            |
| `--new-name`     | The updated name for the token.                                            | Any string surrounded by quotations                                                                            |
| `--role`         | The token's role in the Workspace.                                         | Possible values are either `WORKSPACE_MEMBER`, `WORKSPACE_AUTHOR`, `WORKSPACE_OPERATOR`, or `WORKSPACE_OWNER`. |

## Examples

```sh wrap theme={null}
astro workspace token update --new-name "My updated API token" --role WORKSPACE_MEMBER
```

## Related commands

* [astro workspace token rotate](/docs/cli/v1.45/astro-workspace-token-rotate)
* [astro workspace token delete](/docs/cli/v1.45/astro-workspace-token-delete)
* [astro workspace switch](/docs/cli/v1.45/astro-workspace-switch)


# astro workspace update
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-update

Update an Astro Workspace.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change between product contexts.</Info>

Update an Astro Workspace.

<Tabs>
  <Tab title="Astro">
    ## Usage

    ```sh wrap theme={null}
    astro workspace update <workspace-id> <options>
    ```

    You can find a Workspace's ID by running `astro workspace list`, or by opening your Workspace and going to **Workspace Settings** > **General** in the Astro UI. If you do not provide a Workspace ID, the CLI prompts you to pick from a list of Workspaces that you belong to in your current Organization.

    ## Options

    | Option            | Description                                                               | Valid Values  |
    | ----------------- | ------------------------------------------------------------------------- | ------------- |
    | `--description`   | The description for the Workspace.                                        | Any string    |
    | `--enforce-ci-cd` | Determines whether users are required to use an API token to deploy code. | `ON` or `OFF` |
    | `--name`          | The name for the Workspace.                                               | Any string    |

    ## Examples

    ```sh wrap theme={null}
    $ astro workspace update --name "My Deployment" --enforce-ci-cd OFF
    ```

    ## Related commands

    * [`astro workspace create`](/docs/cli/v1.45/astro-workspace-create)
    * [`astro deployment update`](/docs/cli/v1.45/astro-deployment-update)
  </Tab>

  <Tab title="APC">
    ## Usage

    ```sh wrap theme={null}
    astro workspace update <workspace-id> <options>
    ```

    You can find a Workspace's ID by running `astro workspace list`. If you do not provide a Workspace ID, the CLI prompts you to pick from a list of Workspaces that you belong to in your current Organization.

    ## Options

    | Option          | Description                        | Valid Values |
    | --------------- | ---------------------------------- | ------------ |
    | `--description` | The description for the Workspace. | Any string   |
    | `--label`       | The label for the Workspace.       | Any string   |

    ## Examples

    ```sh wrap theme={null}
    $ astro workspace update --label "My Deployment"
    ```

    ## Related commands

    * [`astro workspace create`](/docs/cli/v1.45/astro-workspace-create)
    * [`astro deployment update`](/docs/cli/v1.45/astro-deployment-update)
  </Tab>
</Tabs>


# astro workspace user add
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-user-add

Add a user to a Workspace.

<Info>The behavior and format of this command differs depending on which Astronomer product you're using. Use the following tabs to change product contexts.</Info>

<Tabs>
  <Tab title="Astro">
    Add an existing Organization user to your current Astronomer Workspace.

    You must be a Workspace Owner to perform this action.

    You can use this command to invite multiple users to a Workspace at a time. See [Add a group of users to Astro using the Astro CLI](/docs/astro/manage-organization-users#add-a-group-of-users-to-astro-using-the-astro-cli).

    ## Usage

    ```sh wrap theme={null}
    astro workspace user add <email>
    ```

    ## Options

    | Option            | Description                                                          | Valid Values                                                                                                                         |
    | ----------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
    | `<email-address>` | The email address of the user that you want to add to the Workspace. | Any valid email address                                                                                                              |
    | `--role`          | The user's role in the Workspace.                                    | Possible values are either `WORKSPACE_ACCESSOR`, `WORKSPACE_MEMBER`, `WORKSPACE_AUTHOR`, `WORKSPACE_OPERATOR`, or `WORKSPACE_OWNER`. |

    ## Related commands

    * [`astro workspace user update`](/docs/cli/v1.45/astro-workspace-user-update)
    * [`astro organization user invite`](/docs/cli/v1.45/astro-organization-user-invite)
    * [`astro organization user update`](/docs/cli/v1.45/astro-organization-user-update)
  </Tab>

  <Tab title="APC">
    Creates a new user in your current Workspace. If the user has already authenticated to Astronomer, they will automatically be granted access to the Workspace. If the user does not have an account on Astronomer, they will receive an email invitation to the platform.

    ## Usage

    ```sh wrap theme={null}
    astro workspace user add --email <user-email-address>
    ```

    ## Options

    | Option                 | Description                                                                                                                                           | Possible Values                                                                                                           |
    | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
    | `--email` (*Required*) | The user's email                                                                                                                                      | Any valid email address                                                                                                   |
    | `--workspace-id`       | The Workspace that the user is added to. Specify this flag if you want to create a user in a Workspace that is different than your current Workspace. | Any valid Workspace ID                                                                                                    |
    | `--role`               | The role assigned to the user.                                                                                                                        | Possible values are `WORKSPACE_MEMBER`, `WORKSPACE_OPERATOR`, and `WORKSPACE_OWNER`. Default value is `WORKSPACE_MEMBER`. |

    ## Related documentation

    * [Manage Workspaces and Deployments on Astronomer](/docs/astro-private-cloud/v-0-37/manage-workspaces)
    * [Manage User Permissions on Astronomer](/docs/astro-private-cloud/v-0-37/workspace-permissions)
  </Tab>
</Tabs>


# astro workspace user list
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-user-list

List Workspace users.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change product contexts.</Info>

<Tabs>
  <Tab title="Astro">
    List all users with access to your current Workspace.

    ## Usage

    ```sh wrap theme={null}
    astro workspace user list
    ```

    ## Options

    | Option          | Description                                                                                          | Possible Values                |
    | --------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------ |
    | `--json`        | Output the response as JSON. Shorthand for `--output json`. Mutually exclusive with `-o`/`--output`. | None                           |
    | `-o`,`--output` | Format the output. The default is `table`.                                                           | `table`, `json`, or `template` |
    | `--template`    | Format the output using a Go template. Use with `--output template`.                                 | Any valid Go template string   |

    ## Output

    | Output           | Description                                                                                                                                                                                      | Data Type                     |
    | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------- |
    | `FULLNAME`       | The full name of the user.                                                                                                                                                                       | String                        |
    | `EMAIL`          | The email address associated with the user account.                                                                                                                                              | String                        |
    | `ID`             | The user ID.                                                                                                                                                                                     | String                        |
    | `WORKSPACE ROLE` | The level of permissions granted to the user in this Workspace. Possible values can be `WORKSPACE_ACCESSOR`, `WORKSPACE_MEMBER`, `WORKSPACE_AUTHOR`, `WORKSPACE_OPERATOR`, or `WORKSPACE_OWNER`. | String                        |
    | `CREATE DATE`    | The date the user profile was created.                                                                                                                                                           | Date (`YYYY-MM-DDTHH:MM:SSZ`) |

    ## Related commands

    * [`astro workspace user update`](/docs/cli/v1.45/astro-workspace-user-update)
    * [`astro workspace user remove`](/docs/cli/v1.45/astro-workspace-user-remove)
    * [`astro organization user list`](/docs/cli/v1.45/astro-organization-user-list)
  </Tab>

  <Tab title="APC">
    Outputs a list of all users with access to your current Workspace.

    ## Usage

    ```sh wrap theme={null}
    astro workspace user list
    ```

    ## Options

    | Option           | Description                                                                                                                                                     | Possible Values        |
    | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
    | `--workspace-id` | The Workspace for which you want to list users. Specify this flag if you want to search for users in a Workspace that is different than your current Workspace. | Any valid Workspace ID |
    | `--email`        | The email address for the user you're searching for.                                                                                                            | Any string             |
    | `--name`         | The name of the user to search for.                                                                                                                             | Any string             |
    | `--paginated `   | Paginate the list of users. If `--page-size` is not specified, the default page size is 20.                                                                     | None                   |
    | `--page-size`    | The page size for paginated lists.                                                                                                                              | Any integer            |

    ## Related documentation

    * [Manage Workspaces and Deployments on Astronomer](/docs/astro-private-cloud/v-0-37/manage-workspaces)
    * [Manage User Permissions on Astronomer](/docs/astro-private-cloud/v-0-37/workspace-permissions)
  </Tab>
</Tabs>


# astro workspace user remove
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-user-remove

Remove a Workspace user.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change product contexts.</Info>

<Tabs>
  <Tab title="Astro">
    Remove a user from your current Workspace.

    ## Usage

    ```sh wrap theme={null}
    astro workspace user remove
    ```

    ## Options

    | Option    | Description                                | Possible Values |
    | --------- | ------------------------------------------ | --------------- |
    | `<email>` | The email for the user you want to remove. | Any valid email |

    ## Related commands

    * [`astro workspace user add`](/docs/cli/v1.45/astro-workspace-user-add)
    * [`astro organization user update`](/docs/cli/v1.45/astro-organization-user-update)
  </Tab>

  <Tab title="APC">
    Removes a user from your current Workspace.

    ## Usage

    ```sh wrap theme={null}
    astro workspace user remove --email <user-email-address>
    ```

    ## Options

    | Option                 | Description       | Possible Values         |
    | ---------------------- | ----------------- | ----------------------- |
    | `--email` (*Required*) | The user's email. | Any valid email address |
  </Tab>
</Tabs>


# astro workspace user update
Source: https://astronomer.io/docs/cli/v1.45/astro-workspace-user-update

Update a Workspace user.

<Info>The behavior and format of this command differs depending on what Astronomer product you're using. Use the following tabs to change product contexts.</Info>

<Tabs>
  <Tab title="Astro">
    Update the role of an existing user in your current Workspace. The CLI prompts you for the user's email.

    ## Usage

    ```sh wrap theme={null}
    astro workspace user update --role <user-role>
    ```

    ## Options

    | Option                | Description                                                  | Valid Values                                                                                                               |
    | --------------------- | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- |
    | `<email>`             | The email address of the user whose role you want to update. | Any valid email                                                                                                            |
    | `--role` (*Required*) | The user's role in the Workspace.                            | Valid values are `WORKSPACE_ACCESSOR`, `WORKSPACE_MEMBER`, `WORKSPACE_OPERATOR`, `WORKSPACE_AUTHOR`, or `WORKSPACE_OWNER`. |

    ## Related commands

    * [`astro workspace user add`](/docs/cli/v1.45/astro-workspace-user-add)
    * [`astro organization user update`](/docs/cli/v1.45/astro-organization-user-update)
  </Tab>

  <Tab title="APC">
    Update a user's permissions in your current Astronomer Workspace.

    ## Usage

    ```sh wrap theme={null}
    astro workspace user update --email <user-email-address>
    ```

    ## Options

    | Option                 | Description                           | Possible Values                                                                                                       |
    | ---------------------- | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
    | `--email` (*Required*) | The user's email.                     | Any valid email address                                                                                               |
    | `--role`               | The role you're updating the user to. | Possible values are `WORKSPACE_VIEWER`, `WORKSPACE_EDITOR`, or `WORKSPACE_ADMIN`. Default value is `WORKSPACE_VIEWER` |

    #### Related documentation

    * [Manage Workspaces and Deployments on Astronomer](/docs/astro-private-cloud/v-0-37/manage-workspaces)
    * [Manage User Permissions on Astronomer](/docs/astro-private-cloud/v-0-37/workspace-permissions)
  </Tab>
</Tabs>


# Authenticate Astro to AWS
Source: https://astronomer.io/docs/cli/v1.45/authenticate-to-aws

Instructions for authenticating your Astro project to AWS.

#### Prerequisites

* A user account on AWS with access to AWS cloud resources.
* The [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
* The [Astro CLI](/docs/cli/v1.45/overview).
* An [Astro project](/docs/cli/v1.45/develop-project#create-an-astro-project).

#### Retrieve AWS user credentials locally

Run the following command to obtain your user credentials locally:

```text wrap theme={null}
aws configure
```

This command prompts you for your Access Key Id, Secret Access Key, Region, and output format. If you log into AWS using single sign-on (SSO), run `aws configure sso` instead.

The AWS CLI then stores your credentials in two separate files:

* `.aws/config`
* `.aws/credentials`

The location of these files depends on your operating system:

* Linux: `/home/<username>/.aws`
* Mac: `/Users/<username>/.aws`
* Windows: `%UserProfile%/.aws`

#### Configure your Astro project

<Info>For Airflow 3, use the provided `docker-compose.override.yml`. For Airflow 2, replace `api-server` with `webserver` and remove the `dag-processor` block.</Info>

The Astro CLI runs Airflow in a Docker-based environment. To give Airflow access to your credential files, you'll mount the `.aws` folder as a volume in Docker.

1. In your Astro project, create a file named `docker-compose.override.yml` with the following configuration:

<Tabs>
  <Tab title="Mac">
    ```yaml wrap theme={null}
    version: "3.1"
    services:
      scheduler:
        volumes:
          - /Users/<username>/.aws:/home/astro/.aws:rw
      api-server:
        volumes:
          - /Users/<username>/.aws:/home/astro/.aws:rw
      triggerer:
        volumes:
          - /Users/<username>/.aws:/home/astro/.aws:rw
      dag-processor:
        volumes:
          - /Users/<username>/.aws:/home/astro/.aws:rw

    ```
  </Tab>

  <Tab title="Linux">
    ```yaml wrap theme={null}
    version: "3.1"
    services:
      scheduler:
        volumes:
          - /home/<username>/.aws:/home/astro/.aws:rw
      api-server:
        volumes:
          - /home/<username>/.aws:/home/astro/.aws:rw
      triggerer:
        volumes:
          - /home/<username>/.aws:/home/astro/.aws:rw
      dag-processor:
        volumes:
          - /home/<username>/.aws:/home/astro/.aws:rw

    ```
  </Tab>

  <Tab title="Windows">
    ```yaml wrap theme={null}
    version: "3.1"
    services:
      scheduler:
        volumes:
          - /c/Users/<username>/.aws:/home/astro/.aws:rw
      api-server:
        volumes:
          - /c/Users/<username>/.aws:/home/astro/.aws:rw
      triggerer:
        volumes:
          - /c/Users/<username>/.aws:/home/astro/.aws:rw
      dag-processor:
        volumes:
          - /c/Users/<username>/.aws:/home/astro/.aws:rw

    ```
  </Tab>
</Tabs>

<info>
  Depending on your Docker configurations, you might have to make your `.aws` folder accessible to Docker. To do this, open **Preferences** in Docker Desktop and go to **Resources** → **File Sharing**. Add the full path of your `.aws` folder to the list of shared folders.
</info>

2. In your Astro project's `.env` file, add the following environment variables. Make sure that the volume path is the same as the one you configured in the `docker-compose.override.yml`.

```text wrap theme={null}
AWS_CONFIG_FILE=/home/astro/.aws/config
AWS_SHARED_CREDENTIALS_FILE=/home/astro/.aws/credentials
```

When you run Airflow locally, all AWS connections without defined credentials automatically fall back to your user credentials when connecting to AWS. Airflow applies and overrides user credentials for AWS connections in the following order:

* Mounted user credentials in the `~/.aws/config` file.
* Configurations in `aws_access_key_id`, `aws_secret_access_key`, and `aws_session_token`.
* An explicit username & password provided in the connection.

For example, if you completed the configuration in this document and then created a new AWS connection with its own username and password, Airflow would use those credentials instead of the credentials in `~/.aws/config`.

## Test your credentials with a secrets backend

Now that Airflow has access to your user credentials, you can use them to connect to your cloud services. Use the following example setup to test your credentials by pulling values from different secrets backends.

1. Create a secret for an Airflow variable or connection in AWS Secrets Manager. All Airflow variables and connection keys must be prefixed with the following strings respectively:

   * `airflow/variables/<my_variable_name>`
   * `airflow/connections/<my_connection_id>`

   For example when adding the secret variable `my_secret_var` you will need to give the secret the name `airflow/variables/my_secret_var`.

   When setting the secret type, choose `Other type of secret` and select the `Plaintext` option. If you're creating a connection URI or a non-dict variable as a secret, remove the brackets and quotations that are pre-populated in the plaintext field.

2. Add the following environment variables to your Astro project `.env` file. For additional configuration options, see the [Apache Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-secrets-manager.html). Make sure to specify your `region_name`.

   ```text wrap theme={null}
   AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
   AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "region_name": "<your-aws-region>"}
   ```

3. Run the following command to start Airflow locally:

   ```sh wrap theme={null}
   astro dev start
   ```

4. Access the Airflow UI at `localhost:8080` and create an Airflow AWS connection named `aws_standard` with no credentials. See [Connections](/docs/learn/connections).

   When you use this connection in your dag, it will fall back to using your configured user credentials.

5. Add a dag which uses the secrets backend to your Astro project `dags` directory. You can use the following example dag to retrieve `<my_variable_name>` and `<my_connection_id>` from the secrets backend and print it to the terminal:

   ```python wrap theme={null}
   from airflow.models.dag import DAG
   from airflow.hooks.base import BaseHook
   from airflow.models import Variable
   from airflow.decorators import task
   from datetime import datetime

   with DAG(
       'example_secrets_dag',
       start_date=datetime(2022, 1, 1),
       schedule=None
   ):

       @task
       def print_var():
           my_var = Variable.get("<my_variable_name>")
           print(f"My secret variable is: {my_var}") # secrets will be masked in the logs!

           conn = BaseHook.get_connection(conn_id="<my_connection_id>")
           print(f"My secret connection is: {conn.get_uri()}") # secrets will be masked in the logs!

       print_var()
   ```

6. In the Airflow UI, unpause your dag and click **Play** to trigger a dag run.

7. View logs for your dag run. If the connection was successful, your masked secrets appear in your logs. See [Airflow logging](/docs/learn/logging).

<Frame>
  <img alt="Secrets in logs" />
</Frame>


# Authenticate Astro to Azure
Source: https://astronomer.io/docs/cli/v1.45/authenticate-to-azure

Instructions for authenticating your Astro project to Azure.

#### Prerequisites

* A user account on Azure with access to Azure cloud resources.
* The [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli).
* The [Astro CLI](/docs/cli/v1.45/overview).
* An [Astro project](/docs/cli/v1.45/develop-project#create-an-astro-project).
* If you're using Windows, [Windows Subsystem Linux](https://learn.microsoft.com/en-us/windows/wsl/install).

#### Retrieve Azure user credentials locally

Run the following command to obtain your user credentials locally:

```sh wrap theme={null}
az login
```

The CLI provides you with a link to a webpage where you authenticate to your Azure account. Once you complete the login, the CLI stores your user credentials in your local Azure configuration folder. The developer account credentials are used in place of the credentials associated with the Registered Application (Service Principal) in Microsoft Entra ID.

The default location of the Azure configuration folder depends on your operating system:

* Linux: `$HOME/.azure/`
* Mac: `/Users/<username>/.azure`
* Windows: `%USERPROFILE%/.azure/`

#### Configure your Astro project

<Info>For Airflow 3, use the provided `docker-compose.override.yml`. For Airflow 2, replace `api-server` with `webserver` and remove the `dag-processor` block.</Info>

The Astro CLI runs Airflow in a Docker-based environment. To give Airflow access to your credential files, mount the `.azure` folder as a volume in Docker.

1. In your Astro project, create a file named `docker-compose.override.yml` with the following configuration:

<Tabs>
  <Tab title="Mac">
    ```yaml wrap theme={null}
    version: "3.1"
    services:
      scheduler:
        volumes:
          - /Users/<username>/.azure:/usr/local/airflow/.azure:rw
      api-server:
        volumes:
          - /Users/<username>/.azure:/usr/local/airflow/.azure:rw
      triggerer:
        volumes:
          - /Users/<username>/.azure:/usr/local/airflow/.azure:rw
      dag-processor:
        volumes:
          - /Users/<username>/.azure:/usr/local/airflow/.azure:rw
    ```
  </Tab>

  <Tab title="Windows and Linux">
    ```yaml wrap theme={null}
    version: "3.1"
    services:
      scheduler:
        volumes:
          - /home/<username>/.azure:/usr/local/airflow/.azure
      api-server:
        volumes:
          - /home/<username>/.azure:/usr/local/airflow/.azure
      triggerer:
        volumes:
          - /home/<username>/.azure:/usr/local/airflow/.azure
      dag-processor:
        volumes:
          - /home/<username>/.azure:/usr/local/airflow/.azure
    ```

    <info>
      In Azure CLI versions 2.30.0 and later on Windows systems, credentials generated by the CLI are saved in an encrypted file and cannot be accessed from Astro Runtime Docker containers. See [MSAL-based Azure CLI](https://learn.microsoft.com/en-us/cli/azure/msal-based-azure-cli).

      To work around this limitation on a Windows computer, use Windows Subsystem Linux (WSL) when completing this setup.

      If you installed the Azure CLI both in Windows and WSL, make sure that the `~/.azure` file path in your volume points to the configuration file for the Azure CLI installed in WSL.
    </info>
  </Tab>
</Tabs>

2. Add the following lines after the `FROM` line in your `Dockerfile` to install the Azure CLI inside your Astro Runtime image:

```dockerfile wrap theme={null}
# FROM ...
USER root
RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash
USER ASTRO
```

<info>
  If you're using an Apple M1 Mac, you must use the`linux/amd64` distribution of Astro Runtime. Replace the first line in the `Dockerfile` of your Astro project with:

  ```text wrap theme={null}
  FROM --platform=linux/amd64 quay.io/astronomer/astro-runtime:<version>
  ```
</info>

3. Add the following environment variable to your `.env` file. Make sure the file path is the same volume location you configured in `docker-compose.override.yml`:

```text wrap theme={null}
AZURE_CONFIG_DIR=/usr/local/airflow/.azure
```

When you run Airflow locally, all Azure connections without defined credentials automatically fall back to your user credentials when connecting to Azure. Airflow applies and overrides user credentials for Azure connections in the following order:

* Mounted user credentials in `/~/.azure`.
* Configurations in `azure_client_id`, `azure_tenant_id`, and `azure_client_secret`.
* An explicit username & password provided in the connection.

For example, if you completed the configuration in this document and then created a new Azure connection with its own username and password, Airflow would use those credentials instead of the credentials in `~/.azure/config`.

## Test your credentials with a secrets backend

Now that Airflow has access to your user credentials, you can use them to connect to your cloud services. Use the following example setup to test your credentials by pulling values from different secrets backends.

1. Create a secret for an Airflow variable or connection in Azure Key Vault. All Airflow variables and connection keys must be prefixed with the following strings respectively:

   * `airflow-variables-<my_variable_name>`
   * `airflow-connections-<my_connection_name>`

   For example, to use a secret named `mysecretvar` in your dag, you must name the secret `airflow-variables-mysecretvar`.

   You will need to store your connection in [URI format](/docs/learn/connections#define-connections-with-environment-variables).

2. In your Astro project, add the following line to Astro project `requirements.txt` file:

   ```text wrap theme={null}
   apache-airflow-providers-microsoft-azure
   ```

3. Add the following environment variables to your Astro project `.env` file. For additional configuration options, see the [Apache Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/secrets-backends/azure-key-vault.html). Make sure to specify your `vault_url`.

   ```text wrap theme={null}
   AIRFLOW__SECRETS__BACKEND=airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend
   AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": "<your-vault-url>"}
   ```

   By default, this setup requires that you prefix any secret names in Key Vault with `airflow-connections` or `airflow-variables`. If you don't want to use prefixes in your Key Vault secret names, set the values for `"connections_prefix"` and `"variables_prefix"` to `""` within `AIRFLOW__SECRETS__BACKEND_KWARGS`. The `vault_url` can be found on the overview page of your Key vault under `Vault URI`.

4. Run the following command to start Airflow locally:

   ```sh wrap theme={null}
   astro dev start
   ```

5. Access the Airflow UI at `localhost:8080` and create an Airflow Azure connection named `azure_standard` with no credentials. See [Connections](/docs/learn/connections).

   When you use this connection in your dag, it will fall back to using your configured user credentials.

6. Add a dag  which uses the secrets backend to your Astro project `dags` directory. You can use the following example dag to retrieve a value from `airflow/variables` and print it to the terminal:

   ```python wrap theme={null}
   from airflow.models.dag import DAG
   from airflow.hooks.base import BaseHook
   from airflow.models import Variable
   from airflow.decorators import task
   from datetime import datetime

   with DAG(
       'example_secrets_dag',
       start_date=datetime(2022, 1, 1),
       schedule=None
   ):

       @task
       def print_var():
           my_var = Variable.get("mysecretvar")
           print(f"My secret variable is: {my_var}")

           conn = BaseHook.get_connection(conn_id="mysecretconnection")
           print(f"My secret connection is: {conn.get_uri()}")

       print_var()
   ```

7. In the Airflow UI, unpause your dag and click **Play** to trigger a dag run.

8. View logs for your dag run. If the connection was successful, your masked secrets appear in your logs. See [Airflow logging](/docs/learn/logging).

<Frame>
  <img alt="Secrets in logs" />
</Frame>


# Authenticate to cloud services with user credentials
Source: https://astronomer.io/docs/cli/v1.45/authenticate-to-clouds

Configure the Astro CLI to use locally stored user credentials as the default for all connections to your cloud.

When you develop Apache Airflow dags locally with the Astro CLI, testing with local data is the easiest way to get started. For more complex data pipelines, you might need to test dags locally with data that's stored in your organization's cloud, such as secret values in a secrets backend service.

To access data on the cloud while developing locally with the Astro CLI, export your cloud account user credentials to a secure configuration file and mount that file in the Docker containers running your local Airflow environment. After you configure this file, you can connect to your cloud without needing to configure additional credentials in Airflow connections. Airflow inherits all permissions from your cloud account and uses them to access your cloud.

* [Authenticate to AWS guide](/docs/cli/v1.45/authenticate-to-aws)
* [Authenticate to GCP guide](/docs/cli/v1.45/authenticate-to-gcp)
* [Authenticate to Azure guide](/docs/cli/v1.45/authenticate-to-azure)


# Authenticate Astro to GCP
Source: https://astronomer.io/docs/cli/v1.45/authenticate-to-gcp

Instructions for authenticating your Astro project to GCP.

#### Prerequisites

* A user account on GCP with access to GCP cloud resources.
* The [Google Cloud SDK](https://cloud.google.com/sdk/docs/install-sdk).
* The [Astro CLI](/docs/cli/v1.45/overview).
* An [Astro project](/docs/cli/v1.45/develop-project#create-an-astro-project).
* Optional. Access to a secrets backend hosted on GCP, such as GCP Secret Manager.

#### Retrieve GCP user credentials locally

Run the following command to obtain your user credentials locally:

```sh wrap theme={null}
gcloud auth application-default login
```

The SDK provides a link to a webpage where you can log in to your Google Cloud account. After you complete your login, the SDK stores your user credentials in a file named `application_default_credentials.json`.

The location of this file depends on your operating system:

* Linux: `$HOME/.config/gcloud/application_default_credentials.json`
* Mac: `/Users/<username>/.config/gcloud/application_default_credentials.json`
* Windows: `%APPDATA%/gcloud/application_default_credentials.json`

#### Configure your Astro project

<Info>For Airflow 3, use the provided `docker-compose.override.yml`. For Airflow 2, replace `api-server` with `webserver` and remove the `dag-processor` block.</Info>

The Astro CLI runs Airflow in a Docker-based environment. To give Airflow access to your credential file, mount it as a Docker volume.

1. In your Astro project, create a file named `docker-compose.override.yml` to your project with the following configuration:

<Tabs>
  <Tab title="Mac">
    ```yaml wrap theme={null}
    version: "3.1"
    services:
      scheduler:
        volumes:
          - /Users/<username>/.config/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
      api-server:
        volumes:
          - /Users/<username>/.config/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
      triggerer:
        volumes:
          - /Users/<username>/.config/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
      dag-processor:
        volumes:
          - /Users/<username>/.config/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
    ```
  </Tab>

  <Tab title="Linux">
    ```yaml wrap theme={null}
    version: "3.1"
    services:
      scheduler:
        volumes:
          - /home/<username>/.config/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
      api-server:
        volumes:
          - /home/<username>/.config/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
      triggerer:
        volumes:
          - /home/<username>/.config/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
      dag-processor:
        volumes:
          - /home/<username>/.config/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
    ```
  </Tab>

  <Tab title="Windows">
    ```yaml wrap theme={null}
    version: "3.1"
    services:
      scheduler:
        volumes:
          - /c/Users/<username>/AppData/Roaming/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
      api-server:
        volumes:
          - /c/Users/<username>/AppData/Roaming/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
      triggerer:
        volumes:
          - /c/Users/<username>/AppData/Roaming/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
      dag-processor:
        volumes:
          - /c/Users/<username>/AppData/Roaming/gcloud/application_default_credentials.json:/usr/local/airflow/gcloud/application_default_credentials.json:rw
    ```
  </Tab>
</Tabs>

2. In your Astro project's `.env` file, add the following environment variable. Ensure that this volume path is the same as the one you configured in `docker-compose.override.yml`.

```text wrap theme={null}
GOOGLE_APPLICATION_CREDENTIALS=/usr/local/airflow/gcloud/application_default_credentials.json
```

When you run Airflow locally, all GCP connections without defined credentials automatically fall back to your user credentials when connecting to GCP. Airflow applies and overrides user credentials for GCP connections in the following order:

* Mounted user credentials in the `/~/gcloud/` folder
* Configurations in `gcp_keyfile_dict`
* An explicit username & password provided in the connection

For example, if you completed the configuration in this document and then created a new GCP connection with its own username and password, Airflow would use those credentials instead of the credentials in `~/gcloud/application_default_credentials.json`.

## Test your credentials with a secrets backend

Now that Airflow has access to your user credentials, you can use them to connect to your cloud services. Use the following example setup to test your credentials by pulling values from different secrets backends.

1. Create a secret for an Airflow variable or connection in GCP Secret Manager. You can do this using the Google Cloud Console or the gcloud CLI. All Airflow variables and connection keys must be prefixed with the following strings respectively:

   * `airflow-variables-<my_variable_name>`
   * `airflow-connections-<my_connection_name>`

   For example when adding the secret variable `my_secret_var` you will need to give the secret the name `airflow-variables-my_secret_var`.

2. Add the following environment variables to your Astro project `.env` file. For additional configuration options, see the [Apache Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/secrets-backends/google-cloud-secret-manager-backend.html). Make sure to specify your `project_id`.

   ```text wrap theme={null}
   AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
   AIRFLOW__SECRETS__BACKEND_KWARGS={"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "project_id": "<my-project-id>"}
   ```

3. Run the following command to start Airflow locally:

   ```sh wrap theme={null}
   astro dev start
   ```

4. Access the Airflow UI at `localhost:8080` and create an Airflow GCP connection named `gcp_standard` with no credentials. See [Connections](/docs/learn/connections).

   When you use this connection in your dag, it will fall back to using your configured user credentials.

5. Add a dag  which uses the secrets backend to your Astro project `dags` directory. You can use the following example dag to retrieve `<my_variable_name>` and `<my_connection_id>` from the secrets backend and print it to the terminal:

   ```python wrap theme={null}
   from airflow.models.dag import DAG
   from airflow.hooks.base import BaseHook
   from airflow.models import Variable
   from airflow.decorators import task
   from datetime import datetime

   with DAG(
       'example_secrets_dag',
       start_date=datetime(2022, 1, 1),
       schedule=None
   ):

       @task
       def print_var():
           my_var = Variable.get("<my_variable_name>")
           print(f"My secret variable is: {my_var}")

           conn = BaseHook.get_connection(conn_id="<my_connection_name>")
           print(f"My secret connection is: {conn.get_uri()}")

       print_var()
   ```

6. In the Airflow UI, unpause your dag and click **Play** to trigger a dag run.

7. View logs for your dag run. If the connection was successful, your masked secrets appear in your logs. See [Airflow logging](/docs/learn/logging).

<Frame>
  <img alt="Secrets in logs" />
</Frame>


# Configure the Astro CLI
Source: https://astronomer.io/docs/cli/v1.45/configure-cli

Learn how to modify project-level settings by updating the .astro/config.yaml file.

Every Astro project includes a file called `.astro/config.yaml` that supports various project-level settings, including:

* The name of your Astro project.
* The port for the Airflow webserver and Postgres metadata database.
* The username and password for accessing the Postgres metadata database.

In most cases, you only need to modify these settings in the case of debugging and troubleshooting the behavior of Airflow components in your local environment.

## Set a configuration

You can set Astro configurations at two different scopes: global and project.

For project-specific configurations, Astro stores the settings in a file named `.astro/config.yaml` in your project directory. This file is generated when you run `astro dev init` in your project folder. Global configurations, however, are stored in a central location with the Astro executable. If a configuration is set in a project configuration file, it always overrides any of the same configuration in the global configuration file.

### Set project configurations

Run the following command in an Astro project to set a configuration for that project:

```sh wrap theme={null}
astro config set <configuration-option> <value>
```

This command applies your configuration to `.astro/config.yaml` in your current Astro project. Configurations do not persist between Astro projects.

For example, to update the port of your local Airflow webserver to 8081 from the default of 8080, run:

```sh wrap theme={null}
astro config set webserver.port 8081
```

### Set global configurations

Use the `--global` flag with `astro config set [option] --global` to set a configuration for all current and new Astro projects on your machine. Your global Astro CLI configuration is saved in a central location on your machine. For example, the configurations are stored in `~/.astro/config.yaml` on Mac OS and `/home//.astro/config.yaml` on Linux.

Note that you can override global configurations by setting a [project-level configuration](#set-project-configurations). If an Astro project contains an `.astro/config.yaml` file, any configuration values in that file take precedence over the same global configuration values.

The following example shows how to set the local Airflow webserver port to `8081` for all Astro projects on your local machine:

```sh wrap theme={null}
astro config set -g webserver.port 8081
```

## Available CLI configurations

<Info>The Astronomer product you're using determines the format and behavior of the configuration commands. Select one of the following tabs to change product contexts.</Info>

<Tabs>
  <Tab title="Astro">
    | Option                    | Description                                                                                                                                                                                                                                                                                      | Default value        | Valid values                                                                    |
    | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------- | ------------------------------------------------------------------------------- |
    | `airflow.expose_port`     | Determines whether to expose the webserver and postgres database of a local Airflow environment to all connected networks.                                                                                                                                                                       | `false`              | `true`, `false`                                                                 |
    | `api-server.port`         | The port for the Airflow 3 apiserver in your local Airflow environment.                                                                                                                                                                                                                          | `8080`               | Any available port                                                              |
    | `cloud.api.protocol`      | The type of protocol to use when calling the Airflow API in a local Airflow environment.                                                                                                                                                                                                         | `https`              | `http`, `https`                                                                 |
    | `cloud.api.port`          | The port to use when calling the Airflow API in a local environment.                                                                                                                                                                                                                             | `443`                | Any available port                                                              |
    | `cloud.api.ws_protocol`   | The type of WebSocket (ws) protocol to use when calling the Airflow API in a local Airflow environment.                                                                                                                                                                                          | `wss`                | `ws`, `wss`                                                                     |
    | `container.binary`        | The name of the container engine. Astro CLI will attempt to auto-detect the container runtime to use, checking for `docker` then `podman` in that order.                                                                                                                                         | Empty string         | `docker` or `podman`                                                            |
    | `context`                 | The context for your Astro project.                                                                                                                                                                                                                                                              | Empty string         | Any available [context](/docs/cli/v1.45/astro-context-list)                          |
    | `cosmos_boost.pre_deploy` | Determines whether the Cosmos Boost pre-deploy step runs during `astro deploy` and `astro dbt deploy`. Use [`astro dbt cleanup`](/docs/cli/v1.45/astro-dbt-cleanup) to remove existing artifacts after you disable this option.                                                                       | `false`              | `true`, `false`                                                                 |
    | `dev.build_secrets`       | One or more Docker build secrets to apply to every `astro dev` command, so you don't need to pass `--build-secret` each time. Use a newline-delimited string to set more than one secret.                                                                                                        | Empty string         | A valid build secret string, such as `id=<your-secret-id>,src=<path-to-secret>` |
    | `dev.mode`                | Determines whether `astro dev` commands run a Docker-based or standalone (no Docker) local Airflow environment. Setting this once means you don't need to pass `--standalone` on every command.                                                                                                  | `docker`             | `docker`, `standalone`                                                          |
    | `disable_astro_run`       | Determines whether to disable `astro run` commands and exclude `astro-run-dag` from any images built by the CLI.                                                                                                                                                                                 | `false`              | `true`, `false`                                                                 |
    | `disable_env_objects`     | Determines whether the Astro CLI pulls connections set in the Astro UI to your local environment. When set to `true`, connections are not pulled to the local environment. Set to `false` to import connections from the Astro UI for local development. Can be set globally with the `-g` flag. | `true`               | `true`, `false`                                                                 |
    | `duplicate_volumes`       | Determines if the Astro CLI creates duplicate volumes when running Airflow locally.                                                                                                                                                                                                              | `true`               | `true` or `false`                                                               |
    | `local.registry`          | The location of your local Docker container running Airflow.                                                                                                                                                                                                                                     | `localhost:5555`     | Any available port                                                              |
    | `otto.auto_update`        | Determines whether the Astro CLI updates the Otto binary automatically on launch when a newer release is available. When `false`, the CLI prints an upgrade hint instead and you apply updates manually with `astro otto update`.                                                                | `true`               | `true`, `false`                                                                 |
    | `postgres.user`           | The username for the Postgres metadata database.                                                                                                                                                                                                                                                 | `postgres`           | Any string                                                                      |
    | `postgres.password`       | The password for the Postgres metadata database.                                                                                                                                                                                                                                                 | `postgres`           | Any string                                                                      |
    | `postgres.host`           | The hostname for the Postgres metadata database.                                                                                                                                                                                                                                                 | `postgres`           | Any string                                                                      |
    | `postgres.port`           | The port for the Postgres metadata database.                                                                                                                                                                                                                                                     | `5432`               | Any available port                                                              |
    | `postgres.repository`     | Image repository to pull the Postgres image from                                                                                                                                                                                                                                                 | `docker.io/postgres` | Any Postgres image in a repository                                              |
    | `postgres.tag`            | The tag for your Postgres image                                                                                                                                                                                                                                                                  | `12.6`               | Any valid image tag                                                             |
    | `project.name`            | The name of your Astro project.                                                                                                                                                                                                                                                                  | Empty string         | Any string                                                                      |
    | `show_warnings`           | Determines whether warning messages appear when starting a local Airflow environment. For example, when set to `true`, you'll receive warnings when a new version of Astro Runtime is available and when your Astro project doesn't have any dags.                                               | `true`               | `true`, `false`                                                                 |
    | `skip_parse`              | Determines whether the CLI parses dags before pushing code to a Deployment.                                                                                                                                                                                                                      | `false`              | `true`, `false`                                                                 |
    | `upgrade_message`         | Determines whether a message indicating the availability of a new Astro CLI version displays in the Astro CLI.                                                                                                                                                                                   | `true`               | `true`, `false`                                                                 |
    | `webserver.port`          | The port for the webserver in your local Airflow environment.                                                                                                                                                                                                                                    | `8080`               | Any available port                                                              |
  </Tab>

  <Tab title="APC">
    | Option                    | Description                                                                                                                                                                                                                                        | Default value | Valid values                        |
    | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ----------------------------------- |
    | `houston.dial_timeout`    | The time in seconds to wait for a Houston connection.                                                                                                                                                                                              | `10`          | Any integer                         |
    | `houston.skip_verify_tls` | Determines whether the Transport Layer Security (TLS) certificate is verified when connecting to Houston.                                                                                                                                          | `false`       | `true`, `false`                     |
    | `interactive`             | Determines whether responses are paginated in the Astro CLI when pagination is supported.                                                                                                                                                          | `false`       | `true`, `false`                     |
    | `page_size`               | Determines the size of the paginated response when `interactive` is set to `true`.                                                                                                                                                                 | `20`          | Any integer                         |
    | `postgres.user`           | The username for the Postgres metadata database.                                                                                                                                                                                                   | `postgres`    | Any string                          |
    | `postgres.password`       | The password for the Postgres metadata database.                                                                                                                                                                                                   | `postgres`    | Any string                          |
    | `postgres.host`           | The hostname for the Postgres metadata database.                                                                                                                                                                                                   | `postgres`    | Any string                          |
    | `postgres.port`           | The port for the Postgres metadata database.                                                                                                                                                                                                       | `5432`        | Any available port                  |
    | `project.name`            | The name of your Astro project.                                                                                                                                                                                                                    | Empty string  | Any string                          |
    | `sha_as_tag`              | Determines whether the SHA Digest value is used when making a call to the `updateDeploymentImage` endpoint in Houston, instead of the Runtime version tag.                                                                                         | `false`       | `true`, `false`                     |
    | `show_warnings`           | Determines whether warning messages appear when starting a local Airflow environment. For example, when set to `true`, you'll receive warnings when a new version of Astro Runtime is available and when your Astro project doesn't have any dags. | `true`        | `true`, `false`                     |
    | `upgrade_message`         | Determines whether a message indicating the availability of a new Astro CLI version displays in the Astro CLI.                                                                                                                                     | `true`        | `true`, `false`                     |
    | `verbosity`               | Determines the Astro CLI log level type.                                                                                                                                                                                                           | `warning`     | `debug`, `info`, `warning`, `error` |
    | `webserver.port`          | The port for the webserver in your local Airflow environment.                                                                                                                                                                                      | `8080`        | Any available port                  |
  </Tab>
</Tabs>


# Customize your Astro project Dockerfile
Source: https://astronomer.io/docs/cli/v1.45/customize-dockerfile



By default, the Astro project Dockerfile only includes a `FROM` statement that specifies your Astro Runtime version. However, you can extend your Dockerfile to use a different distribution or run additional buildtime arguments. Use this document to learn which Dockerfile customizations are supported both locally and on Astro.

## Prerequisites

* An [Astro project](/docs/cli/v1.45/develop-project#create-an-astro-project)

## Use an alternative Astro Runtime distribution

Starting with Astro Runtime 9, each version of Astro Runtime has a separate distribution for each currently supported Python version. Use an alternative Python distribution if any of your dependencies require a Python version other than the [default Runtime Python version](/docs/runtime/runtime-image-architecture#python-versioning).

<Info>Replace `<registry-url>` with the Docker registry URL. Airflow 2.x-based Runtime versions and Airflow 3.x-based Runtime versions have different registry URLs. See [Docker Registry URLs](/docs/runtime/runtime-image-architecture#container-registry-urls) for information on which URL to pull from.</Info>

To use a specific Python distribution, update the first line in your Astro project `Dockerfile` to reference the required distribution:

```text wrap theme={null}
FROM <registry-url><runtime-version>-python-<python-version>
```

For example, to use Python 3.10 with Astro Runtime version 9.0.0, you update the first line of your Dockerfile to the following:

```text wrap theme={null}
FROM <registry-url>9.0.0-python-3.10
```

## Run commands on build

To run additional commands as your Astro project is built into a Docker image, add them to your `Dockerfile` as `RUN` commands. These commands run as the last step in the image build process.

<Info>Replace `<registry-url>` with the Docker registry URL. Airflow 2.x-based Runtime versions and Airflow 3.x-based Runtime versions have different registry URLs. See [Docker Registry URLs](/docs/runtime/runtime-image-architecture#container-registry-urls) for information on which URL to pull from.</Info>

For example, if you want to run `ls` when your image builds, your `Dockerfile` would look like this:

```text wrap theme={null}
FROM <registry-url>3.1-5
RUN ls
```

This is supported both on Astro and in the context of local development.

## Add a CA certificate to an Astro Runtime image

If you need your Astro Deployment to communicate securely with a remote service using a certificate signed by an untrusted or internal certificate authority (CA), you need to add the CA certificate to the trust store inside your Astro project's Docker image.

<Info>Replace `<registry-url>` with the Docker registry URL. Airflow 2.x-based Runtime versions and Airflow 3.x-based Runtime versions have different registry URLs. See [Docker Registry URLs](/docs/runtime/runtime-image-architecture#container-registry-urls) for information on which URL to pull from.</Info>

1. In your Astro project `Dockerfile`, add the CA certificate to the base Runtime image:

   ```docker wrap theme={null}
   # Use the base Astro Runtime image so we can setup our CA cert before installing any packages
   FROM <registry-url><runtime-version>-base

   # Switch to root for setup
   USER root

   # Add internal CA certificate
   COPY <internal-ca.crt> /usr/local/share/ca-certificates/<internal-ca.crt>
   RUN update-ca-certificates

   # Install system packages if listed in packages.txt
   COPY packages.txt .
   RUN /usr/local/bin/install-system-packages

   # Install Python dependencies
   COPY requirements.txt .
   RUN /usr/local/bin/install-python-dependencies

   # Switch back to astro user
   USER astro

   # Copy project into image
   COPY --chown=astro:0 . .
   ```

   <Info>When using a base image, make sure to install the system level and python packages.</Info>

2. (Optional) Add additional `COPY` statements before the `RUN update-ca-certificates` stanza for each CA certificate your organization is using for external access.

3. [Restart your local environment](/docs/cli/v1.45/run-airflow-locally#restart-a-local-airflow-environment) or deploy to Astro. See [Deploy code](/docs/astro/deploy-code).


# Develop your Astro project
Source: https://astronomer.io/docs/cli/v1.45/develop-project



An Astro project contains all of the files necessary to test and run dags in a local Airflow environment and on Astro. This guide provides information about adding and organizing Astro project files, including:

* Adding dags
* Setting environment variables
* Applying changes
* Running on-build commands

For information about adding Airflow Providers, Python, and OS-level packages, see [Add Airflow providers and packages](/docs/cli/v1.45/add-providers-packages).

For information about running your Astro project in a local Airflow, see [Run Airflow locally](/docs/cli/v1.45/run-airflow-locally).

<Tip>As you add to your Astro project, Astronomer recommends reviewing the [Astronomer Registry](https://registry.astronomer.io/), a library of Airflow modules, providers, and dags that serve as the building blocks for data pipelines.</Tip>

## Prerequisites

* The [Astro CLI](/docs/cli/v1.45/overview)

## Create an Astro project

In an empty folder, run the following command to create an Astro project:

```sh wrap theme={null}
astro dev init
```

This command generates the following files in your directory:

```text wrap theme={null}
.
├── .env # Local environment variables
├── dags # Where your dags go
│   └── exampledag.py # Example dag that showcases a simple ETL data pipeline
├── Dockerfile # For the Astro Runtime Docker image, environment variables, and overrides
├── include # For any other files you'd like to include
├── plugins # For any custom or community Airflow plugins
├── tests # For any dag unit test files to be run with pytest
│   └── test_dag_example.py # Test that checks for basic errors in your dags
├── airflow_settings.yaml # For your Airflow connections, variables and pools (local only)
├── packages.txt # For OS-level packages
└── requirements.txt # For Python packages
```

You can use the [`--from-template`](/docs/cli/v1.45/astro-dev-init#options) option with `astro dev init` to initialize an Astro project based off of a [template](https://github.com/astronomer/templates/tree/main). Each template has different contents unique to the use case they are made for, but the directory structures are the same.

Use the rest of this document to understand how to interact with each of these folders and files.

## Add dags

In Apache Airflow, data pipelines are defined in Python code as Directed Acyclic Graphs (dags). A dag is a collection of tasks and dependencies between tasks that are defined as code. See [Introduction to Airflow dags](/docs/learn/dags) for an introduction to dags, and [Dag Writing on Astro](/docs/astro/best-practices/dag-writing-on-astro) for information on tools to help you write and develop dags.

Dags are stored in the `dags` folder of your Astro project. To add a dag to your project:

1. Add the `.py` file to the `dags` folder.
2. Save your changes. If you're using a Mac, use **Command-S**.
3. Refresh your Airflow browser.

<Tip>Use the `astro run <dag-id>` command to run and debug a dag from the command line without starting a local Airflow environment. This is an alternative to testing your entire Astro project with the Airflow webserver and scheduler. See [Test your Astro project locally](/docs/cli/v1.45/test-your-astro-project-locally).</Tip>

## Add utility files

Airflow dags sometimes require utility files to run workflows. This can include:

* SQL files.
* Custom Airflow operators.
* Python functions.

When more than one dag in your Astro project needs a certain function or query, creating a shared utility file helps make your dags idempotent, more readable, and minimizes the amount of code you have in each dag.

You can store utility files in the `/dags` directory of your Astro project. In most cases, Astronomer recommends adding your utility files to the `/dags` directory and organizing them into sub-directories based on whether they're needed for a single dag or for multiple dags.

In the following example, the `dags` folder includes both types of utility files:

```text wrap theme={null}
└── dags
    ├── my_dag
    │   ├── my_dag.py
    │   └── my_dag_utils.py  # specific dag utils
    └── utils
        └── common_utils.py # common utils
```

1. To add utility files which are shared between all your dags, create a folder named `utils` in the `dags` directory of your Astro project. To add utility files only for a specific dag, create a new folder in `dags` to store both your dag file and your utility file.
2. Add your utility files to the folder you created.
3. Reference your utility files in your dag code.
4. Apply your changes. If you're developing locally, refresh the Airflow UI in your browser.

Utility files in the `/dags` directory will not be parsed by Airflow, so you don't need to specify them in `.airflowignore` to prevent parsing. If you're using [dag-only deploys](/docs/astro/deploy-dags) on Astro, changes to this folder are deployed when you run `astro deploy --dags` and do not require rebuilding your Astro project into a Docker image and restarting your Deployment.

<Info>
  **Airflow 3 import paths**

  In Airflow 3 on Astro, bare imports like `import utils` no longer work due to Dag bundle isolation. Use fully qualified imports instead, such as `import dags.utils`. See [PYTHONPATH doesn’t include the Dags folder](/docs/astro/airflow3/upgrade-af3#pythonpath-doesn’t-include-the-dags-folder) for details.
</Info>

## Add Airflow connections, pools, variables

Airflow connections connect external applications such as databases and third-party services to Apache Airflow. See [Manage connections in Apache Airflow](/docs/learn/connections#airflow-connection-basics) or [Apache Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html).

To add Airflow [connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html), [pools](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/pools.html), and [variables](https://airflow.apache.org/docs/apache-airflow/stable/howto/variable.html) to your local Airflow environment, you have the following options:

* Use the Airflow UI. In **Admin**, click **Connections**, **Variables** or **Pools**, and then add your values. These values are stored in the metadata database and are deleted when you run the [`astro dev kill` command](/docs/cli/v1.45/astro-dev-kill), which can sometimes be used for troubleshooting.
* Modify the `airflow_settings.yaml` file of your Astro project. This file is included in every Astro project and permanently stores your values in plain-text. To prevent you from committing sensitive credentials or passwords to your version control tool, Astronomer recommends adding this file to `.gitignore`.
* Use the Astro UI to create connections that can be shared across Deployments in a Workspace. These connections are not visible in the Airflow UI. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).
* Use a secret backend, such as AWS Secrets Manager, and access the secret backend locally. See [Configure an external secrets backend on Astro](/docs/astro/secrets-backend).

When you add Airflow objects to the Airflow UI of a local environment or to your `airflow_settings.yaml` file, your values can only be used locally. When you deploy your project to a Deployment on Astro, the values in this file are not included.

Astronomer recommends using the `airflow_settings.yaml` file so that you don’t have to manually redefine these values in the Airflow UI every time you restart your project. To ensure the security of your data, Astronomer recommends [configuring a secrets backend](/docs/astro/secrets-backend).

## Add test data or files for local testing

Use the `include` folder of your Astro project to store files for testing locally, such as test data or a dbt project file. The files in your `include` folder are included in your deploys to Astro, but they are not parsed by Airflow. Therefore, you don't need to specify them in `.airflowignore` to prevent parsing.

If you're running Airflow locally, apply your changes by refreshing the Airflow UI.

### Configure `airflow_settings.yaml` (Local development only)

The `airflow_settings.yaml` file includes a template with the default values for all possible configurations. To add a connection, variable, or pool, replace the default value with your own.

1. Open the `airflow_settings.yaml` file and replace the default value with your own.

   ```yaml wrap theme={null}

   airflow:
     connections: ## conn_id and conn_type are required
       - conn_id: my_new_connection
         conn_type: postgres
         conn_host: 123.0.0.4
         conn_schema: airflow
         conn_login: user
         conn_password: pw
         conn_port: 5432
         conn_extra:
     pools: ## pool_name and pool_slot are required
       - pool_name: my_new_pool
         pool_slot: 5
         pool_description:
     variables: ## variable_name and variable_value are required
       - variable_name: my_variable
         variable_value: my_value

   ```

2. Save the modified `airflow_settings.yaml` file in your code editor. If you use a Mac computer, for example, use **Command-S**.

3. Import these objects to the Airflow UI. Run:

   ```sh wrap theme={null}
   astro dev object import
   ```

4. In the Airflow UI, click **Connections**, **Pools**, or **Variables** to see your new or modified objects.

5. Optional. To add another connection, pool, or variable, you append it to this file within its corresponding section. To create another variable, add it under the existing `variables` section of the same file. For example:

```yaml wrap theme={null}

variables:
  - variable_name: <my-variable-1>
    variable_value: <my-variable-value>
  - variable_name: <my-variable-2>
    variable_value: <my-variable-value-2>

```

## Set environment variables locally

For local development, Astronomer recommends setting environment variables in your Astro project’s `.env` file. You can then push your environment variables from the `.env` file to a Deployment on Astro. To manage environment variables in the Astro UI, see [Environment variables](/docs/astro/environment-variables).

If your environment variables contain sensitive information or credentials that you don’t want to expose in plain-text, you can add your `.env` file to `.gitignore` when you deploy these changes to your version control tool.

1. Open the `.env` file in your Astro project directory.

2. Add your environment variables to the `.env` file or run `astro deployment variable list --save` to copy environment variables from an existing Deployment to the file.

   Use the following format when you set environment variables in your `.env` file:

   ```text wrap theme={null}
   KEY=VALUE
   ```

   Environment variables should be in all-caps and not include spaces.

3. [Restart your local environment](/docs/cli/v1.45/run-airflow-locally#restart-a-local-airflow-environment).

4. Run the following command to confirm that your environment variables were applied locally:

   ```sh wrap theme={null}
   astro dev bash --scheduler "/bin/bash && env"
   ```

   These commands output all environment variables that are running locally. This includes environment variables set on Astro Runtime by default.

5. Optional. Run `astro deployment variable create --load` or `astro deployment variable update --load` to export environment variables from your `.env` file to a Deployment. You can view and modify the exported environment variables in the Astro UI page for your Deployment.

<Info>
  For local environments, the Astro CLI generates an `airflow.cfg` file at runtime based on the environment variables you set in your `.env` file. You can’t create or modify `airflow.cfg` in an Astro project.

  To view your local environment variables in the context of the generated Airflow configuration, run:

  ```sh wrap theme={null}
  astro dev bash --scheduler "/bin/bash && cat airflow.cfg"
  ```

  These commands output the contents of the generated `airflow.cfg` file, which lists your environment variables as human-readable configurations with inline comments.
</Info>

### Use multiple .env files

The Astro CLI looks for `.env` by default, but if you want to specify multiple files, make `.env` a top-level directory and create sub-files within that folder.

A project with multiple `.env` files might look like the following:

```text wrap theme={null}
my_project
├── Dockerfile
├── dags
│   └── my_dag
├── plugins
│   └── my_plugin
├── airflow_settings.yaml
└── .env
    ├── dev.env
    └── prod.env
```

## Add Airflow plugins

If you need to build a custom view in the Airflow UI or build an application on top of the Airflow metadata database, you can use Airflow plugins. To use an Airflow plugin, add your plugin files to the `plugins` folder of your Astro project. To apply changes from this folder to a local Airflow environment, [restart your local environment](/docs/cli/v1.45/run-airflow-locally#restart-a-local-airflow-environment).

To learn more about Airflow plugins and how to build them, see [Airflow Plugins](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/plugins.html) in Airflow documentation or the Astronomer [Airflow plugins](/docs/learn/using-airflow-plugins) guide.

## Unsupported project configurations

You can't use `airflow.cfg` or `airflow_local_settings.py` files in an Astro project. `airflow_local_settings.py` has no effect on Astro Deployments, and `airflow.cfg` has no effect on local environments and Astro Deployments.

An alternative to using `airflow.cfg` is to set Airflow environment variables in your `.env` file. See [Set environment variables locally](#set-environment-variables-locally).

## See also

For more advanced project configurations, see:

* [Customize your Astro project Dockerfile](/docs/cli/v1.45/customize-dockerfile)
* [Install Python packages from private sources](/docs/cli/v1.45/private-python-packages)


# Install the Astro CLI
Source: https://astronomer.io/docs/cli/v1.45/install-cli

Instructions for installing, upgrading, and uninstalling the Astro command-line interface (CLI).

There are multiple ways to install, upgrade, and uninstall the Astro CLI. Install the Astro CLI to start running Airflow locally or to manage Astro from your terminal.

<Info>If you can't install the Astro CLI on your local machine, you can still run Astro CLI commands and deploy to Astro using [GitHub Actions](/docs/astro/first-dag-github-actions).</Info>

## Prerequisites

<Tabs>
  <Tab title="Mac">
    * [Homebrew](https://brew.sh/).

    Starting with version 1.32.0, the Astro CLI Homebrew tap installs Podman by default on macOS to run Airflow locally. If your organization uses Docker and you want to skip installing Podman, install from the Astronomer tap and pass `--without-podman` (macOS only):

    ```sh wrap theme={null}
    brew tap astronomer/tap
    brew install astronomer/tap/astro --without-podman
    ```

    <Note>`brew install astro` (Homebrew core) does not support `--without-podman` and will install Podman.</Note>
  </Tab>

  <Tab title="Windows with winget">
    * (Optional) The winget command line tool is supported on Windows 10 1709 (build 16299) or later, and is bundled with Windows 11 and modern versions of Windows 10 by default as the App Installer. If you're running an earlier version of Windows 10 and you don't have the App Installer installed, you can download it from the [Microsoft Store](https://apps.microsoft.com/store/detail/app-installer/9NBLGGH4NNS1?hl=en-ca\&gl=ca). If you've installed the App Installer previously, make sure you're using the latest version before running commands.

    * Windows Subsystem for Linux (WSL) 2 enabled: See [Enable the Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install), [WSL basic commands](https://learn.microsoft.com/en-us/windows/wsl/basic-commands), and [Troubleshooting WSL 2](https://learn.microsoft.com/en-us/windows/wsl/troubleshooting#error-0x80370102-the-virtual-machine-could-not-be-started-because-a-required-feature-is-not-installed).
      * After you enable WSL, run:
      ```text wrap theme={null}
       wsl --update
       wsl --install --no-distribution
      ```

    * The latest version of the Windows [App Installer](https://apps.microsoft.com/store/detail/app-installer/9NBLGGH4NNS1?hl=en-ca\&gl=ca).

    * Windows 10 1709 (build 16299) or later or Windows 11.

    Starting with 1.32.0, the Astro CLI installs Podman, by default, as its container management engine for running Airflow locally. If your organization uses Docker to run and manage containers, you can opt out of installing Podman with the Astro CLI, allowing the Astro CLI to use your existing runtime setup. To opt out, use `winget install -e --id Astronomer.Astro --skip-dependencies`.
  </Tab>

  <Tab title="Windows (Manual)">
    * Windows Subsystem for Linux (WSL) 2 enabled: See [Enable the Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install), [WSL basic commands](https://learn.microsoft.com/en-us/windows/wsl/basic-commands), and [Troubleshooting WSL 2](https://learn.microsoft.com/en-us/windows/wsl/troubleshooting#error-0x80370102-the-virtual-machine-could-not-be-started-because-a-required-feature-is-not-installed).
      * After you enable WSL, run:
      ```text wrap theme={null}
       wsl --update
       wsl --install --no-distribution
      ```
    * Windows 10 or Windows 11.
    * A container service like [Podman](https://podman.io/docs/installation) or [Docker Engine](https://docs.docker.com/engine/install/).
  </Tab>

  <Tab title="Linux">
    * A container service like [Podman](https://podman.io/docs/installation) or [Docker Engine](https://docs.docker.com/engine/install/).
  </Tab>
</Tabs>

## Installation

<Tabs>
  <Tab title="Mac">
    To install the latest version of the Astro CLI, run the following command:

    ```sh wrap theme={null}
    brew install astro
    ```

    To install a specific version of the Astro CLI, specify the version you want to install at the end of the command. For example, to install Astro CLI version 1.36.0, you would run the following command:

    ```sh wrap theme={null}
    brew install astro@1.36.0
    ```

    If you specify only a major version, this command installs the latest minor or patch version available for the major version. For a list of all available versions, see the [CLI release notes](/docs/cli/v1.45/release-notes).

    To verify that the correct Astro CLI version was installed, run:

    ```sh wrap theme={null}
    astro version
    ```
  </Tab>

  <Tab title="Windows with winget">
    <Warning>
      You can use the Windows Package Manager winget command-line tool to install the Astro CLI. To install an older version of the Astro CLI, you'll need to follow the Manual Windows installation process.
    </Warning>

    1. Open Windows PowerShell as an administrator and then run the following command:

       ```sh wrap theme={null}
       winget install -e --id Astronomer.Astro
       ```

       To install a specific version of the Astro CLI, specify the version you want to install at the end of the command. For example, running the following command specifies the latest available version of the Astro CLI:

       ```sh wrap theme={null}
       winget install -e --id Astronomer.Astro -v 1.36.0
       ```

    2. Run `astro version` to confirm the Astro CLI is installed properly.
  </Tab>

  <Tab title="Windows (Manual)">
    <Warning>
      Astronomer recommends using the Windows Package Manager winget command-line tool to install the Astro CLI.
    </Warning>

    1. Go to the [Releases page](https://github.com/astronomer/astro-cli/releases) of the Astro CLI GitHub repository, scroll to a CLI version, and then download the `.exe` file that matches the CPU architecture of your machine.

       For example, to install v1.18.2 of the Astro CLI on a Windows machine with an AMD 64 architecture, download `astro_1.18.2_windows_amd64.exe`.

    2. Rename the file to `astro.exe`.

    3. Add the filepath for the directory containing the new `astro.exe` as a PATH environment variable. For example, if `astro.exe` is stored in `C:\Users\username\astro.exe`, you add `C:\Users\username` as your PATH environment variable. To learn more about configuring the PATH environment variable, see [How do I set or change the PATH system variable?](https://www.java.com/en/download/help/path.html).

    4. Restart your machine.
  </Tab>

  <Tab title="Linux">
    Run the following command to install the latest version of the Astro CLI directly to `PATH`:

    ```sh wrap theme={null}
    curl -sSL install.astronomer.io | sudo bash -s
    ```

    To install a specific version of the CLI, specify the version number as a flag at the end of the command. For example, to install the most recent release of the CLI, you would run:

    ```sh wrap theme={null}
    curl -sSL install.astronomer.io | sudo bash -s -- v1.36.0
    ```

    If you specify only a major version, this command installs the latest minor or patch version available for the major version. If you specify only a major version, this command installs the latest minor or patch version available for the major version. For a list of all available versions, see the [CLI release notes](/docs/cli/v1.45/release-notes).
  </Tab>
</Tabs>

## Resolve installation issues

<Tabs>
  <Tab title="Mac">
    Follow this procedure when Homebrew fails to install the latest Astro CLI version or the error `No formulae or casks found for astro@<major.minor.patch-version>` appears. To troubleshoot other Homebrew issues, see [Common Issues](https://docs.brew.sh/Common-Issues) in the Homebrew documentation.

    1. If the install process is not working, run the following command to update Homebrew and all package definitions (formulae):

       ```sh wrap theme={null}
       brew update
       ```

    2. Re-run the installation again:

       ```sh wrap theme={null}
       brew install astro
       ```

    3. If this is the first time you're installing the CLI and updating Homebrew doesn't work, check to see if `astronomer/tap` is in your [Homebrew tap list](https://docs.brew.sh/Taps):

       ```sh wrap theme={null}
       brew tap astronomer/tap
       ```

       If `astronomer/tap` is in your tap list and you still can't install the Astro CLI, you might be experiencing a different issue. See [Common Issues](https://docs.brew.sh/Common-Issues) in the Homebrew documentation.

    4. Run the following command to install the Astronomer CLI:

       ```sh wrap theme={null}
       brew install astro
       ```

    ### Troubleshooting `--without-podman`

    #### Error: No available formula or cask with the name "astronomer/tap/astro"

    Run `brew tap astronomer/tap`, then retry the install.

    #### Error: invalid option: `--without-podman`

    You're installing the Homebrew core formula. Use the fully qualified tap formula:

    ```sh wrap theme={null}
    brew install astronomer/tap/astro --without-podman
    ```

    #### Already installed core formula

    Uninstall it, then reinstall from the tap:

    ```sh wrap theme={null}
    brew uninstall astro
    brew install astronomer/tap/astro --without-podman
    ```
  </Tab>

  <Tab title="Windows with winget">
    If an error message appears indicating that the term winget is not recognized as an internal or external command when you attempt to run winget commands, see this [troubleshooting document](https://github.com/microsoft/winget-cli/tree/master/doc/troubleshooting#common-issues) provided by Microsoft.

    If you're still struggling to install the Astro CLI with winget, retry the install using the alternative instructions in [Windows (Manual)](/docs/cli/v1.45/install-cli?tab=windows-manual#installation).
  </Tab>
</Tabs>


# Run Airflow locally
Source: https://astronomer.io/docs/cli/v1.45/local-airflow-overview

Work with your Astro project in a local environment by running Airflow and dags locally.

Running Airflow locally with the Astro CLI lets you preview and debug dag changes before deploying to production. In a local Airflow environment, you can fix issues with your dags without consuming infrastructure resources or waiting on code deploy processes.

The Astro CLI supports multiple modes for running Airflow locally:

* **Container mode (default):** Uses Podman or Docker to run Airflow components in containers. All tasks run locally in the scheduler container using the [local executor](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/local.html).
* **Standalone mode:** Runs Airflow directly on your machine in a virtual environment, without Docker or Podman. This can enable faster iteration in development as you don't have to wait for containers to rebuild. Use `astro dev start --standalone` or set it as your default with `astro config set dev.mode standalone`. See [astro dev start](/docs/cli/v1.45/astro-dev-start) for all available options.

See the following documentation to get started:

* [Run Airflow locally](/docs/cli/v1.45/run-airflow-locally)
* [Test your dags](/docs/cli/v1.45/test-your-astro-project-locally)
* [Troubleshoot locally](/docs/cli/v1.45/troubleshoot-locally)
* [Sync Deployment connections from Astro](/docs/cli/v1.45/local-connections)


# Use Airflow connections hosted on Astro in a local environment
Source: https://astronomer.io/docs/cli/v1.45/local-connections

Use dags locally with Airflow connections that you created in the Astro Environment Manager.

When you create Airflow connections for a Deployment on Astro with the [Environment Manager](/docs/astro/create-and-link-connections), you can also use them to test dags locally. This is the easiest way to share connection details between a Deployment on Astro and your local Airflow environment.

Using connections from the Astro Environment Manager in a local Airflow environment means that instead of creating a connection twice or manually importing connection details, you can configure the Astro CLI to use any connections that are already configured for a particular Deployment or Workspace. The CLI then adds your connection details directly to your local Airflow metadata database, so you don't need to manage a `.env` file, secrets backend, or `airflow_settings.yaml` file to keep connection information consistent between your local environment and your Deployments on Astro.

To use connections from the Environment Manager locally, start a local project using `astro dev start`, and then specify either the Workspace or Deployment that you want to import connections from. When you start your project with these settings, the Astro CLI fetches the necessary connections from Astro. Then, after the local Airflow containers start, the Astro CLI populates the metadata database with the connections. This ensures that the connections are encrypted in the metadata database and not easily accessible by an end user.

Connection details in the Astro Environment Manager are not visible from the Airflow UI when you run your project locally. Instead, they are synced using the Astro CLI for the Workspace and Deployment that you want to work with.

### Prerequisites

* The latest version of the [Astro CLI](/docs/cli/v1.45/install-cli)
* Either a Workspace or Deployment with at least one connection [configured through the Astro Environment Manager in the Astro UI](/docs/astro/create-and-link-connections)
* A local [Astro Project](/docs/cli/v1.45/develop-project#create-an-astro-project)
* Astro Runtime 9.3.0 or greater
* `WORKSPACE_AUTHOR`, `WORKSPACE_OPERATOR`, or `WORKSPACE_OWNER` user permissions
* An internet connection

### Setup

1. Enable local development access to connections created in the Astro UI.

   ```zsh wrap theme={null}
   # -g sets this config globally
   astro config set -g disable_env_objects false
   ```

2. Log in to Astro.

   ```zsh wrap theme={null}
   astro login
   ```

3. Retrieve the ID of either the Workspace or Deployment that you want to import connections from.

   ```zsh wrap theme={null}
   # Retrieve Workspace IDs
   astro workspace list
   # Retrieve Deployment IDs
   astro deployment list
   ```

4. Start your project locally specifying the Deployment or Workspace ID from Step 3.

   * **Using connections linked to all Deployments in a Workspace**

   ```zsh wrap theme={null}
   astro dev start --workspace-id [workspace-id]
   ```

   * **Using Deployment-level connections**

   ```zsh wrap theme={null}
   astro dev start --deployment-id [deployment-id]
   ```

   <Info>
     If you see the error `Error: showSecrets on organization with id is not allowed`, your [Organization Owner](/docs/astro/user-permissions#organization-roles) needs to enable **Environment Secrets Fetching** in the **Organization Settings** on the Astro UI before you can use your connections locally. See [Configure environment secrets fetching for the Astro Environment Manager](/docs/astro/organization-settings#configure-environment-secrets-fetching-for-the-astro-environment-manager).
   </Info>

Congratulations! You set up your local Airflow environment with the Astro CLI to use connections created in the Astro Environment Manager. Now, you can run and test dags locally that automatically use these connections without any additional setup.


# Install Python packages from private sources
Source: https://astronomer.io/docs/cli/v1.45/private-python-packages



Python packages can be installed into your image from both public and private sources. To install packages listed on private PyPI indices or a private git-based repository, you need to complete additional configuration in your project.

Depending on where your private packages are stored, use one of the following setups to install these packages to an Astro project by customizing your Runtime image.

<Info>Deploying a custom Runtime image with a CI/CD pipeline requires additional configurations. For an example implementation, see the GitHub Actions CI/CD templates for [Astro](/docs/astro/ci-cd-templates/github-actions-template) and [Astro Private Cloud](https://www.astronomer.io/docs/software/ci-cd#github-actions-cicd).</Info>

## Setup

<Tabs>
  <Tab title="Private GitHub Repo">
    #### Install Python packages from private GitHub repositories

    This topic provides instructions for building your Astro project with Python packages from a private GitHub repository. Although GitHub is used in these examples, the same approach works with any hosted Git repository.

    <Tabs>
      <Tab title="Using .netrc build secrets">
        Use the `--build-secret` flag to pass `.netrc` credentials at build time. This approach lets you add private packages to your `requirements.txt` file without custom Dockerfile configuration.

        <Info>This method requires Astro Runtime 3.1-14+, 3.0-15+, or later than 13.5.1.</Info>

        #### Prerequisites

        * The [Astro CLI](/docs/cli/v1.45/overview)
        * An [Astro project](/docs/cli/v1.45/develop-project#create-an-astro-project)
        * Custom Python packages that are [installable with pip](https://packaging.python.org/en/latest/tutorials/packaging-projects/)
        * A private GitHub repository for each of your custom Python packages
        * The [GitHub CLI](https://cli.github.com/) or a [GitHub personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens)

        #### Step 1: Specify the private repository in your project

        Add your private packages to your `requirements.txt` file using HTTPS URLs in the following format:

        ```text wrap theme={null}
        git+https://github.com/<your-github-organization-name>/<your-private-repository>.git
        ```

        For example, to install `mypackage1` and `mypackage2` from `myorganization`:

        ```text wrap theme={null}
        git+https://github.com/myorganization/mypackage1.git
        git+https://github.com/myorganization/mypackage2.git
        ```

        #### Step 2: Configure credentials

        Define the `NETRC_CONTENT` environment variable in your shell profile (`.bashrc` or `.zshrc`):

        ```bash wrap theme={null}
        export NETRC_CONTENT="machine github.com login oauth2 password $(gh auth token)"
        ```

        If you don't use the GitHub CLI, replace `$(gh auth token)` with a [GitHub personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) that has access to your private repositories.

        #### Step 3: Run with build secrets

        Pass the `--build-secret` flag when running Astro CLI commands:

        ```bash wrap theme={null}
        astro dev start --build-secret id=netrc,env=NETRC_CONTENT
        ```

        To run tests:

        ```bash wrap theme={null}
        astro dev pytest --build-secret id=netrc,env=NETRC_CONTENT
        ```

        The `--build-secret` flag securely provides the `.netrc` content during the Docker build without storing credentials in the image. Alternatively, set the `dev.build_secrets` [configuration](/docs/cli/v1.45/configure-cli) once to apply it to every `astro dev` command without repeating the flag.

        #### Deploy with GitHub Actions

        To build your image in a GitHub Actions workflow, pass the `.netrc` content as a build secret:

        ```yaml wrap theme={null}
        - name: Build image
          uses: docker/build-push-action@v4
          with:
            context: .
            secrets: |
              netrc=machine github.com login oauth2 password ${{ secrets.GITHUB_TOKEN }}
        ```

        #### Deploy with the Astronomer deploy action

        To deploy using the [Astronomer deploy action](https://github.com/astronomer/deploy-action), pass the `.netrc` content through `build-secrets` and set the `NETRC_CONTENT` environment variable:

        ```yaml wrap theme={null}
        - name: Deploy to Astro
          uses: astronomer/deploy-action@v0.x
          with:
            build-secrets: id=netrc,env=NETRC_CONTENT
            # ... deployment-id, etc.
          env:
            NETRC_CONTENT: "machine github.com login oauth2 password ${{ secrets.GITHUB_TOKEN }}"
        ```
      </Tab>

      <Tab title="Using SSH keys">
        <Info>The following setup has been validated with only a single SSH key. You might need to modify this setup when using more than one SSH key per Docker image.</Info>

        #### Prerequisites

        * The [Astro CLI](/docs/cli/v1.45/overview)
        * An [Astro project](/docs/cli/v1.45/develop-project#create-an-astro-project)
        * Custom Python packages that are [installable with pip](https://packaging.python.org/en/latest/tutorials/packaging-projects/)
        * A private GitHub repository for each of your custom Python packages
        * A [GitHub SSH private key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) authorized to access your private GitHub repositories

        This setup assumes that each custom Python package is hosted within its own private GitHub repository. Installing multiple custom packages from a single private GitHub repository is not supported.

        <Warning>If your organization enforces SAML single sign-on (SSO), you must first authorize your key to be used with that authentication method. See [Authorizing an SSH key for use with SAML single sign-on](https://docs.github.com/en/enterprise-cloud@latest/authentication/authenticating-with-saml-single-sign-on/authorizing-an-ssh-key-for-use-with-saml-single-sign-on).</Warning>

        #### Step 1: Specify the private repository in your project

        To add a Python package from a private repository to your Astro project, specify the Secure Shell (SSH) URL for the repository in a new `private-requirements.txt` file. Use the following format for the SSH URL:

        ```text wrap theme={null}
        git+ssh://git@github.com/<your-github-organization-name>/<your-private-repository>.git
        ```

        For example, to install `mypackage1` and `mypackage2` from `myorganization`, add the following to your `private-requirements.txt` file:

        ```text wrap theme={null}
        git+ssh://git@github.com/myorganization/mypackage1.git
        git+ssh://git@github.com/myorganization/mypackage2.git
        ```

        This example assumes that the name of each of your Python packages is identical to the name of its corresponding GitHub repository. In other words,`mypackage1` is both the name of the package and the name of the repository.

        #### Step 2: Update Dockerfile

        1. (Optional) Copy and save any existing build steps in your `Dockerfile`.

        2. Add the following to your `packages.txt` file:

           ```bash wrap theme={null}
           openssh-client
           git
           ```

        3. In your Dockerfile, add the following instructions:

           ```docker wrap theme={null}
           USER root
           RUN mkdir -p -m 0700 ~/.ssh && \
               echo "github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl" >> ~/.ssh/known_hosts

           COPY private-requirements.txt .
           RUN --mount=type=ssh,id=github pip install --no-cache-dir --requirement private-requirements.txt
           USER astro

           ENV PATH="/home/astro/.local/bin:$PATH"
           ```

           In order, these instructions:

           * Switch to `root` user for SSH setup and installation from private repository
           * Add the fingerprint for GitHub to `known_hosts`
           * Copy your `private-requirements.txt` file into the image
           * Install Python-level packages from your private repository as specified in your `private-requirements.txt` file. This securely mounts your SSH key at build time, ensuring that the key itself is not stored in the resulting Docker image filesystem or metadata.
           * Switch back to `astro` user
           * Add the user bin directory to `PATH`

        <Info>
          See GitHub's documentation for all available [SSH key fingerprints](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints).

          If your repository isn't hosted on GitHub, replace the fingerprint with one from where the package is hosted. Use `ssh-keyscan` to generate the fingerprint.
        </Info>

        #### Step 3: Build a custom Docker image

        1. Run the following command to automatically generate a unique image name:

           ```sh wrap theme={null}
           image_name=astro-$(date +%Y%m%d%H%M%S)
           ```

        2. Run the following command to create a new Docker image from your `Dockerfile`. Replace `<ssh-key>` with your SSH private key file name.

           ```sh wrap theme={null}
           DOCKER_BUILDKIT=1 docker build -f Dockerfile --progress=plain --ssh=github="$HOME/.ssh/<ssh-key>" -t $image_name .
           ```

        3. (Optional) Test your Dags locally. See [Restart your local environment](/docs/cli/v1.45/run-airflow-locally#restart-a-local-airflow-environment).

        4. Deploy the image to Astro using the Astro CLI:

           ```sh wrap theme={null}
           astro deploy --image-name $image_name
           ```

        Your Astro project can now use Python packages from your private GitHub repository.
      </Tab>
    </Tabs>
  </Tab>

  <Tab title="Private PyPi Index">
    #### Install Python packages from a private PyPI index

    Installing Python packages on Astro from a private PyPI index is required for organizations that deploy a [private PyPI server (`private-pypi`)](https://pypi.org/project/private-pypi/) as a secure layer between pip and a Python package storage backend, such as GitHub, AWS, or a local file system or managed service.

    To complete this setup, you'll specify your privately hosted Python packages in `requirements.txt`.

    #### Prerequisites

    * An [Astro project](/docs/cli/v1.45/develop-project#create-an-astro-project)
    * A private PyPI index with a corresponding username and password

    #### Add Python packages to your Astro project

    To install a Python package from a private PyPI index, add the package name and version to the `requirements.txt` file of your Astro project.

    If you don't specify a version, Astro installs the latest version. Use the same syntax for installing private packages that you use when you add public packages from [PyPI](https://pypi.org). Your `requirements.txt` file can contain both publicly accessible and private packages.

    At the beginning of the `requirements.txt` file, add `--index-url https://myuser:example.com/api/pypi/pypi/simple` for Airflow 3 based projects and `--extra-index-url https://myuser:example.com/api/pypi/pypi/simple` for Airflow 2 based projects. Replace this example URL with your index's location.

    #### Example `requirements.txt`

    ```requirements wrap theme={null}
    --index-url https://myuser:example.com/api/pypi/pypi/simple
    pandas
    my-custom-package-company
    apache-airflow-providers-snowflake
    ```

    <Warning>
      If your organization configures pip globally with `/etc/pip.conf`, ensure that the following is included:

      ```text wrap theme={null}
      [global]
      constraint = /etc/pip-constraints.txt
      ```

      The `constraint` setting ensures that dependencies are pinned to versions compatible with your Airflow version on Astro. Omitting the `constraint` setting can lead to compatibility issues.
    </Warning>

    Your Astro project can now use Python packages from your private PyPi index.
  </Tab>
</Tabs>


# Astro CLI release and lifecycle policy
Source: https://astronomer.io/docs/cli/v1.45/release-lifecycle-policy



Astronomer regularly releases new versions of the Astro CLI that include new integrations with Astronomer products and local development features. To encourage users to regularly update the Astro CLI and to ensure stability across Astronomer products, the Astro CLI has a maintenance release lifecycle policy.

## Versioning and release channels

The Astro CLI uses semantic versioning, where each version of the CLI includes a major, minor, and patch number. The format of an Astro CLI version is `major.minor.patch`.

Each version of the Astro CLI belongs to one of two release channels:

* **Stable**: Releases in the stable channel are up to date and compatible with all current Astro and Astro Private Cloud functionality. The Astronomer team tests all new features against all stable Astro CLI releases.
* **Deprecated**: Releases in the deprecated channel are not guaranteed to work with all Astro and Astro Private Cloud functionality.

Go to [updates.astronomer.io](https://updates.astronomer.io/astro-cli) to see which versions are currently stable and deprecated.

Specifically, when a release is in the stable channel, the following is true:

* The version is tested regularly against new versions of Astronomer APIs.
* All Astro, Astro Private Cloud, and local use cases are supported.
* The release includes `stable` metadata in `https://updates.astronomer.io/astro-cli`.
* The release's binary is available on GitHub.
* The release can be installed from [install.astronomer.io](http://install.astronomer.io), `brew install astro` and, `winget install astro`.

When a release is in the deprecated channel, the following is true:

* Astro and Astro Private Cloud functionality can still work, but is not guaranteed to work.
* If you're an Astronomer customer and reach out to support while using a deprecated version of the Astro CLI, Astronomer support will recommend that you upgrade the CLI as a first step.
* The release includes `deprecated` metadata in `https://updates.astronomer.io/astro-cli`.
* The release's binary is available on GitHub.
* The release can be installed from [install.astronomer.io](http://install.astronomer.io), `brew install astro` and, `winget install astro`.

## Restricted versions

There are some restricted versions of the Astro CLI that have known bugs and cannot be used without fatal errors. For these versions of the Astro CLI, the following is true:

* Astro automatically blocks you from interacting with Astro resources with a restricted release version.
* The release does not appear in `https://updates.astronomer.io/astro-cli`.
* This version is not available to install from [install.astronomer.io](http://install.astronomer.io), `brew install astro`, and `winget install astro`.

The following versions have been restricted from use:

* Astro CLI v1.35.0

## Astro CLI maintenance policy

The stable release channel contains only the latest patches of the three most recent minor versions of the Astro CLI.

For example, consider a circumstance where the three most recent minor versions of the Astro CLI are 1.31, 1.32, and 1.33 Based on this maintenance policy:

* The latest patch versions for 1.31, 1.32, and 1.33 are all stable. These would be the only available stable releases for the Astro CLI.
* Say 1.33.1 is the current stable version and Astronomer releases version 1.33.2. This means 1.33.2 becomes the new stable version and version 1.33.1 becomes deprecated. This release has no impact on the channels for the 1.31 and 1.32 releases.
* If 1.34 releases, all versions of 1.31 are marked as deprecated. This release would have no impact on the channels of 1.32 and 1.33 releases.


# Astro CLI release notes
Source: https://astronomer.io/docs/cli/v1.45/release-notes

A record of the latest Astro command-line interface (CLI) features and bug fixes.

<Tip>[Subscribe to Astro CLI release notes](/docs/astro/release-notes-subscribe) to receive updates via RSS, email, or Slack.</Tip>

This document provides a summary of all changes made to the [Astro CLI](/docs/cli/v1.45/overview). For general product release notes, go to [Astro Release Notes](/docs/astro/release-notes). If you have any questions or a bug to report, contact [Astronomer support](https://cloud.astronomer.io/open-support-request).

* **Stable versions**: 1.45.0, 1.44.0, and 1.43.1. See [Astro CLI release and lifecycle policy](/docs/cli/v1.45/release-lifecycle-policy) for more information about support for CLI versions.

<Update label="Astro CLI 1.45.0" description="August 13, 2026">
  ### Slim, field-filtered dbt manifest at deploy time

  When the [Cosmos Boost pre-deploy step](/docs/cli/v1.45/configure-cli) is enabled, `astro deploy` and `astro dbt deploy` now also write a slim, field-filtered `.astro/manifest.slim.json` next to each dbt manifest. Cosmos Boost can load this smaller file instead of the full `manifest.json` to reduce Dag-parse time and memory on large dbt projects. This is on by default whenever `cosmos_boost.pre_deploy` is enabled. To opt out, set `ASTRO_COSMOS_BOOST_SLIM_MANIFEST_ENABLED=false`.

  [`astro dbt cleanup`](/docs/cli/v1.45/astro-dbt-cleanup) now removes both `.astro/dbt_metadata.json` and `.astro/manifest.slim.json`.

  ### `--dags-path` is no longer a hidden flag on `astro deploy`

  The [`--dags-path`](/docs/cli/v1.45/astro-deploy) flag, which deploys Dags from a path other than your project's `dags` directory, is now documented. Combining `--dags` with `--dags-path` no longer requires running the command from an Astro project directory.

  ### Bug fixes

  * `astro deploy` now correctly parses Remote Execution Agent client image tags that include a distro segment (for example, `-ubi9-`) instead of failing with a parse error. When multiple distro variants are available, the CLI prefers the Debian-based image by default.
</Update>

<Update label="Astro CLI 1.44.0" description="August 6, 2026">
  ### New `astro dbt cleanup` command and Cosmos Boost pre-deploy step

  Set `cosmos_boost.pre_deploy` to `true` to enable an opt-in Cosmos Boost pre-deploy step:

  ```bash wrap theme={null}
  astro config set cosmos_boost.pre_deploy true
  ```

  When enabled, [`astro deploy`](/docs/cli/v1.45/astro-deploy) and [`astro dbt deploy`](/docs/cli/v1.45/astro-dbt-deploy) generate a `.astro/dbt_metadata.json` file next to each dbt project and dbt manifest in your deploy payload before it leaves your machine. Cosmos Boost uses this file as a cache key instead of hashing your project tree at every Dag parse. This step is disabled by default and has no effect on your deploy while disabled. It doesn't apply to deploys that use `--image-name` with a prebuilt image.

  Use the new [`astro dbt cleanup`](/docs/cli/v1.45/astro-dbt-cleanup) command to remove Cosmos Boost artifacts from one or more paths.

  ### Adopt operator-managed Deployments on Astro Private Cloud

  <Info>
    **Public Preview**

    This feature is in Public Preview for Astro Private Cloud.
  </Info>

  Use the new [`astro deployment adopt`](/docs/cli/v1.45/astro-deployment-adopt) command to bring an Airflow custom resource that the standalone operator created into Astro Private Cloud management. The command creates the matching Deployment record in Astro Private Cloud. It also updates the custom resource. It reconfigures the ingress and the authentication for the Airflow UI. The Deployment's namespace and its metadata database don't change.

  ```bash wrap theme={null}
  astro deployment adopt --cluster-id=<cluster-id> --name=<cr-name> --namespace=<cr-namespace>
  ```

  Use `--use-apc-logging` to route the adopted Deployment's logs through Astro Private Cloud logging. This flag updates the logging configuration in the custom resource. Use `--use-apc-registry` to use the Astro Private Cloud in-cluster registry. If you use `--use-apc-registry`, you must pre-sync the Deployment's images.

  Use the new [`astro deployment unadopt`](/docs/cli/v1.45/astro-deployment-unadopt) command to release an adopted Deployment back to operator-only management. The command removes the Deployment record from Astro Private Cloud. It doesn't delete the Airflow custom resource, its namespace, or its metadata database.

  Both commands require Astro Private Cloud 2.1.0 or later.

  ### Additional improvements

  * [`astro api`](/docs/cli/v1.45/astro-api) `describe` now expands nested `$ref` schemas fully instead of stopping at the first reference. Both `describe` and `ls` support a new `--json` flag for machine-readable output.
  * Added a repeatable `--build-secret` flag to [`astro dev build`](/docs/cli/v1.45/astro-dev-build), [`astro dev start`](/docs/cli/v1.45/astro-dev-start), [`astro dev restart`](/docs/cli/v1.45/astro-dev-restart), [`astro dev parse`](/docs/cli/v1.45/astro-dev-parse), [`astro dev pytest`](/docs/cli/v1.45/astro-dev-pytest), [`astro dev upgrade-test`](/docs/cli/v1.45/astro-dev-upgrade-test), and [`astro deploy`](/docs/cli/v1.45/astro-deploy) to pass multiple Docker build secrets. The existing `--build-secrets` flag is deprecated because passing more than one secret to it silently mounted only the last one.
  * Added a `dev.build_secrets` configuration key so you can set a build secret once instead of passing `--build-secret` on every `astro dev` command. Set it with `astro config set dev.build_secrets "<secret>"` or `astro config set --global dev.build_secrets "<secret>"`.
  * `astro deploy` to Astro Private Cloud Hybrid Deployments now injects an Airflow 3-compatible monitoring Dag. Previously, only Airflow 2 monitoring Dags were supported.
  * New projects created with [`astro dev init`](/docs/cli/v1.45/astro-dev-init) now add `.astro/worktrees/` to `.gitignore`.
  * Added a `--mode` flag to [`astro deployment create`](/docs/cli/v1.45/astro-deployment-create) for Astro Private Cloud to create Operator-mode Deployments. This flag requires Astro Private Cloud 2.1.0 or later.
  * Deprecated the `-h`, `--hard` flag on [`astro deployment delete`](/docs/cli/v1.45/astro-deployment-delete) for Astro Private Cloud. Astro Private Cloud now hard-deletes Deployments by default, so the flag has no effect. Astronomer plans to remove the flag in a future release.

  ### Bug fixes

  * Fixed a bug where a successful interactive re-login after a failed token refresh was immediately overwritten, which left you signed out even though sign-in succeeded.
  * Fixed [`astro dev pytest`](/docs/cli/v1.45/astro-dev-pytest) and [`astro dev parse`](/docs/cli/v1.45/astro-dev-parse) to read container exit codes as integers instead of comparing them as substrings. Previously, exit code 10 was reported as a pass and exit code 130 was reported as a test failure.
  * Fixed [`astro deploy`](/docs/cli/v1.45/astro-deploy) to return an error instead of exiting successfully when it fails to save the Deployment ID to your project configuration.
  * Fixed `astro deployment update` on Astro Private Cloud to return an error instead of exiting successfully when you specify an invalid `--executor` value.
  * Fixed an issue where the Astro CLI created scaffolded projects, templates, and other files and folders with world-writable permissions. Directories now default to `0o755`, files to `0o644`, and files that can contain credentials to `0o600`.
  * Fixed [`astro api airflow`](/docs/cli/v1.45/astro-api-airflow) with `--paginate` and `--slurp` to page through the full result set for Airflow REST endpoints, such as `get_dags`. Previously, results were capped at the first page.
  * Fixed commands that accept `--workspace-id` to honor the flag before requiring a current Workspace context. This fixes non-interactive authentication with an Organization-scoped API token.
  * Fixed `astro organization audit-logs export` to write a valid gzip file. A regression in 1.43.0 caused the command to write uncompressed data to a file with a `.gz` extension.
  * Fixed [`astro organization switch`](/docs/cli/v1.45/astro-organization-switch) and [`astro workspace switch`](/docs/cli/v1.45/astro-workspace-switch) to find a target Organization, Workspace, or Deployment that isn't on the first page of results.
</Update>

<Update label="Astro CLI 1.43.1" description="July 2, 2026">
  ### Bug fixes

  * Fixed [`astro deployment update`](/docs/cli/v1.45/astro-deployment-update) with `--deployment-file` against Kubernetes-executor Deployments. A regression in 1.43.0 caused the command to send an empty `worker_queues` field, which returned an `Invalid field values` error.
  * Fixed [`astro deployment airflow-variable copy`](/docs/cli/v1.45/astro-deployment-airflow-variable-copy), [`astro deployment connection copy`](/docs/cli/v1.45/astro-deployment-connection-copy), and [`astro deployment pool copy`](/docs/cli/v1.45/astro-deployment-pool-copy) to retry write requests after a transient server error and to update existing values instead of failing when you rerun a copy after a partial failure.
  * Fixed [`astro dev start`](/docs/cli/v1.45/astro-dev-start) `--standalone` to show sign-in instructions for Airflow 2 projects after startup. On Linux, Apache Airflow writes a random password for the default `admin` user to a file, and the CLI didn't previously tell you where to find it.
</Update>

<Update label="Astro CLI 1.43.0" description="June 23, 2026">
  <Warning>
    **Known issue (resolved in 1.43.1)**

    Astro CLI 1.43.0 fails to run [`astro deployment update`](/docs/cli/v1.45/astro-deployment-update) with `--deployment-file` against Kubernetes-executor Deployments. The command sends an empty `worker_queues` field, which returns an `Invalid field values` error.

    **Impact:**
    You cannot update a Kubernetes-executor Deployment using a Deployment file on Astro CLI 1.43.0.

    **Solution:**
    Upgrade to Astro CLI 1.43.1 or later. This issue is fixed in Astro CLI version 1.43.1.
  </Warning>

  ### New `astro env` command tree

  A new [`astro env`](/docs/cli/v1.45/astro-env) command tree manages Astro Environment Manager objects from your terminal, including environment variables, Airflow connections, Airflow variables, and metrics exports. You can scope each object to a Workspace or a Deployment, link Workspace-scoped objects to specific Deployments, and bulk-import variables from a dotenv file.

  * [`astro env variable`](/docs/cli/v1.45/astro-env-variable) manages environment variables and exports them to a `.env` file with [`astro env variable export`](/docs/cli/v1.45/astro-env-variable-export).
  * [`astro env variable link`](/docs/cli/v1.45/astro-env-variable-link) links a Workspace variable to specific Deployments, with optional per-Deployment value overrides.
  * [`astro env connection`](/docs/cli/v1.45/astro-env-connection) manages Airflow connections.
  * [`astro env airflow-variable`](/docs/cli/v1.45/astro-env-airflow-variable) manages Airflow variables.
  * [`astro env metrics-export`](/docs/cli/v1.45/astro-env-metrics-export) manages metrics exports.

  These commands are distinct from [`astro deployment variable`](/docs/cli/v1.45/astro-deployment-variable-create) and [`astro deployment connection`](/docs/cli/v1.45/astro-deployment-connection-create), which write directly to a single Deployment. To manage the same objects in the Astro UI, see [Create environment variables in the Astro UI](/docs/astro/create-and-link-environment-variables) and [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).

  ### Dag processor and component filters for `astro deployment logs`

  [`astro deployment logs`](/docs/cli/v1.45/astro-deployment-logs) now accepts a `--dag-processor` flag to show Dag processor logs. A new `--component` flag shows logs from one or more components by name. The `--component` flag is repeatable and accepts a comma-separated list, so you can opt into new log sources without a CLI release.

  ### Additional improvements

  * The deployment template file used by [`astro deployment create`](/docs/cli/v1.45/astro-deployment-create) and [`astro deployment update`](/docs/cli/v1.45/astro-deployment-update) now supports the `podEphemeralStorage` field.
  * [`astro otto`](/docs/cli/v1.45/astro-otto) now surfaces its `update` and `version` subcommands in `--help`.

  ### Bug fixes

  * Fixed [`astro deploy`](/docs/cli/v1.45/astro-deploy) to preserve symlinks and honor `.dockerignore` in the image build context.
  * Fixed [`astro deploy`](/docs/cli/v1.45/astro-deploy) to preserve the trailing newline in your `.dockerignore` after building the image.
  * Fixed [`astro deploy`](/docs/cli/v1.45/astro-deploy) to skip the Dag-only deploy check on Remote Execution Deployments, which previously blocked custom image deploys to those Deployments.
  * Fixed [`astro dev start`](/docs/cli/v1.45/astro-dev-start) standalone mode to set `PYTHONPATH` to match the Astro Runtime image.
  * Fixed [`astro dev start`](/docs/cli/v1.45/astro-dev-start) `--standalone` for Airflow 2 projects whose Dockerfile uses a `-slim` Astro Runtime image. The CLI now strips the `-slim` suffix when fetching constraints, so the command no longer fails with an HTTP 404 error.
  * Fixed [`astro dev start`](/docs/cli/v1.45/astro-dev-start) in Docker Compose mode so that bare host-environment passthrough resolves. Previously, an `environment: [VAR]` entry without a value in a `docker-compose.override.yml` file was ignored.
  * Fixed [`astro dev init`](/docs/cli/v1.45/astro-dev-init) to add `.astro/*.local.yml` and `.astro/*.local.yaml` to `.gitignore`, so local config files are no longer committed unintentionally.
  * Fixed local Astro project setup with the Podman container engine on Windows.
  * Fixed local project discovery on Windows. The CLI now writes `routes.json` even when the proxy daemon is unavailable, so you can reach local projects by hostname.
</Update>

<Update label="Astro CLI 1.42.1" description="May 20, 2026">
  ### Bug fixes

  * Fixed [`astro deploy`](/docs/cli/v1.45/astro-deploy) with `--image-name` against Deployments that have Dag-only deploys disabled, including Remote Execution Deployments. A regression in 1.42.0 caused these deploys to fail with a `DAG-only deploys are not enabled for this Deployment` error.
  * Fixed `astro dev start` with the Podman container engine on Windows. Native Podman commands no longer receive an unsupported `npipe://` connection value, so local project setup completes successfully.
</Update>

<Update label="Astro CLI 1.42.0" description="April 30, 2026">
  <Warning>
    **Known issue (resolved in 1.42.1)**

    Astro CLI 1.42.0 fails to deploy with [`astro deploy`](/docs/cli/v1.45/astro-deploy) `--image-name` against Deployments that have Dag-only deploys disabled, including Remote Execution Deployments. The deploy returns a `DAG-only deploys are not enabled for this Deployment` error.

    **Impact:**
    You cannot push a custom image to a Remote Execution Deployment, or to any Deployment without Dag-only deploys enabled, using `--image-name` on Astro CLI 1.42.0.

    **Solution:**
    Upgrade to Astro CLI 1.42.1 or later. This issue is fixed in Astro CLI version 1.42.1.
  </Warning>

  ### New `astro otto` command

  A new [`astro otto`](/docs/cli/v1.45/astro-otto) command launches [Otto](/docs/astro/otto-overview), Astronomer's data engineering agent, in your terminal. The CLI handles Otto binary downloads, version management, and authentication automatically.

  ### Airflow 2 support in standalone mode

  Standalone mode now supports Airflow 2 (Astro Runtime 13.x) projects in addition to Airflow 3. Run `astro dev start --standalone` from any Astro project to start a local Airflow environment without Docker, regardless of the Airflow major version.

  ### `--json` and `--output` flags for list commands

  Most list commands now accept `--json`, `--output` (`-o`), and `--template` flags to produce machine-readable output for scripts and CI/CD pipelines. The default table output is unchanged.

  The following commands support the new flags:

  * [`astro dev ps`](/docs/cli/v1.45/astro-dev-ps)
  * [`astro deployment list`](/docs/cli/v1.45/astro-deployment-list)
  * [`astro deployment user list`](/docs/cli/v1.45/astro-deployment-user)
  * [`astro deployment team list`](/docs/cli/v1.45/astro-deployment-team)
  * [`astro workspace list`](/docs/cli/v1.45/astro-workspace-list)
  * [`astro workspace user list`](/docs/cli/v1.45/astro-workspace-user-list)
  * [`astro workspace team list`](/docs/cli/v1.45/astro-workspace-team-list)
  * [`astro organization list`](/docs/cli/v1.45/astro-organization-list)
  * [`astro organization user list`](/docs/cli/v1.45/astro-organization-user-list)
  * [`astro organization team list`](/docs/cli/v1.45/astro-organization-team-list)

  For example:

  ```bash wrap theme={null}
  astro workspace list --json
  astro deployment list -o template --template '{{range .Deployments}}{{.Name}} ({{.DeploymentID}}){{"\n"}}{{end}}'
  ```

  `--json` and `--output` are mutually exclusive.

  ### Faster local environment startup

  `astro dev start` and [`astro dev object import`](/docs/cli/v1.45/astro-dev-object-import) now apply variables, connections, and pools from `airflow_settings.yaml` through the Airflow REST API instead of running a separate Airflow CLI command per item. A typical settings file with several connections, variables, and pools that previously took more than a minute now completes in a few seconds.

  ### Astro Private Cloud 2.0 support

  The CLI now supports Astro Private Cloud 2.0. Deployment, image update, and Dag-only deploy operations against Astro Private Cloud 2.0. Compatibility with earlier Astro Private Cloud versions is preserved.

  ### Additional improvements

  * `astro dev init` and `astro dev start --standalone` now skip Astro Runtime versions marked as yanked when picking a default image tag, so new projects no longer pin to a yanked Runtime version. This does not affect projects that already pin a floating tag in their Dockerfile, because Docker resolves floating tags through the registry.
  * The `--spec-url` flag on [`astro api cloud`](/docs/cli/v1.45/astro-api-cloud) now accepts local file paths in addition to HTTP URLs. Supported forms include absolute paths, relative paths, paths starting with `~/`, and `file://` URIs. Local specs are read fresh on every command invocation, and the `--spec-token-env-var` flag is ignored for local files.
  * [`astro deployment logs`](/docs/cli/v1.45/astro-deployment-logs) with `--log-count` now paginates results so the command returns the full requested number of log lines. Previously, results were capped at the API's per-page limit regardless of the requested count.
  * The Astro CLI now reports the local development mode (`docker` or `standalone`) in anonymous telemetry. To opt out of telemetry, run [`astro telemetry disable`](/docs/cli/v1.45/astro-telemetry-disable) or set the `ASTRO_TELEMETRY_DISABLED` environment variable to `1`.

  ### Bug fixes

  * Fixed `astro dev start --standalone` to use your project's `plugins/` folder for `plugins_folder`. Previously, `plugins_folder` defaulted to `.astro/standalone/plugins/`, so plugins from your project were not loaded.
  * Fixed [`astro deploy`](/docs/cli/v1.45/astro-deploy) with `--image-name` to skip Dockerfile parsing. Projects that build their image in a separate pipeline and reference it through `--image-name` no longer fail when no `Dockerfile` is present in the project directory.
</Update>

<Update label="Astro CLI 1.41.0" description="April 20, 2026">
  ### New `astro auth` command

  A new [`astro auth`](/docs/cli/v1.45/astro-auth) command group provides commands for authenticating to Astro or Astro Private Cloud:

  * [`astro auth login`](/docs/cli/v1.45/astro-auth-login): Authenticate to Astro or Astro Private Cloud. The existing [`astro login`](/docs/cli/v1.45/astro-login) command is now an alias for this command.
  * [`astro auth logout`](/docs/cli/v1.45/astro-auth-logout): Log out of the Astro CLI. The existing [`astro logout`](/docs/cli/v1.45/astro-logout) command is now an alias for this command.
  * [`astro auth token`](/docs/cli/v1.45/astro-auth-token): Print the current authentication token to standard output. This is useful for using the token in scripts or CI/CD pipelines. Use the `--domain` flag to print the token for a specific context domain.

  ### New `astro api registry` command

  A new [`astro api registry`](/docs/cli/v1.45/astro-api-registry) command lets you make HTTP requests to the Airflow Provider Registry API directly from the Astro CLI. No authentication is required. The command supports endpoint discovery with `ls` and `describe`, `jq` filters, Go template output, and a `--generate` flag to produce equivalent `curl` commands.

  ### Additional improvements

  * Added the `-f` (`--force`) flag to [`astro dev init`](/docs/cli/v1.45/astro-dev-init) to initialize a project without confirmation, even in a non-empty directory.
  * Added the `-f` (`--force`) flag to [`astro deployment create`](/docs/cli/v1.45/astro-deployment-create) to create a Deployment without prompting for confirmation.
  * Added the `-f` (`--force`) flag to [`astro organization team update`](/docs/cli/v1.45/astro-organization-team-update), [`astro organization team delete`](/docs/cli/v1.45/astro-organization-team-delete), and the `astro organization team user` [`add`](/docs/cli/v1.45/astro-organization-team-user) and [`remove`](/docs/cli/v1.45/astro-organization-team-user) subcommands to skip confirmation prompts for Identity Provider (IdP)-managed teams.
  * The `-f` (`--force`) flag on [`astro deploy`](/docs/cli/v1.45/astro-deploy) now also skips the "No Dags found" warning in addition to parse errors and uncommitted changes.
  * Added Swagger 2.0 support and custom spec URL for `astro api cloud`.
  * Fixed `astro dev run` to propagate non-zero exit codes from the executed command.
  * Fixed `astro dev restart` with `--allow-existing` to preserve the virtual environment in standalone mode.
  * Fixed `astro dev start` in standalone mode to add `include/` to `PYTHONPATH`.
  * Standardized error messages for missing required flags across commands.
  * Fixed the default image ordering for `astro-agent` in Airflow version resolution.
  * Improved help text with examples and descriptions across commands.
</Update>

<Update label="Astro CLI 1.40.1" description="March 12, 2026">
  ### Bug fixes

  * Fixed the `astro dev pytest` command to use `docker cp` instead of bind-mounts for copying test files into the container.
</Update>

<Update label="Astro CLI 1.40.0" description="March 10, 2026">
  ### New `astro dev build` command

  A new [`astro dev build`](/docs/cli/v1.45/astro-dev-build) command builds your Astro project into a Docker image without starting a local Airflow environment. Use this command to verify that your project builds successfully before deployment. The command supports `--no-cache` and `--image-name` flags.

  ### New `astro api` command

  A new [`astro api`](/docs/cli/v1.45/astro-api) command lets you make authenticated API requests to Astronomer services directly from the Astro CLI:

  * [`astro api airflow`](/docs/cli/v1.45/astro-api-airflow): Make requests to the Airflow REST API for a local or deployed Airflow instance. The CLI automatically detects the Airflow version and resolves the OpenAPI spec. Supports endpoint discovery via `ls` and `describe`, pagination, `jq` filters, Go template output, and a `--curl` flag to generate equivalent `curl` commands.
  * [`astro api cloud`](/docs/cli/v1.45/astro-api-cloud): Make requests to the Astro platform API using your current context's bearer token. Supports the same output formatting and discovery features as `astro api airflow`.

  ### Standalone mode for Docker-free local development

  You can now run a local Airflow environment without Docker using standalone mode. Standalone mode runs Airflow directly on your machine using a virtual environment. This initial release supports Airflow 3 projects only.

  To use standalone mode, set it as the default for a project:

  ```bash wrap theme={null}
  astro config set dev.mode standalone
  ```

  Or use the `--standalone` flag to override for a single command:

  ```bash wrap theme={null}
  astro dev start --standalone
  ```

  All existing `astro dev` commands work in standalone mode, including `run`, `bash`, `parse`, `pytest`, `object import`, and `object export`. The `build`, `upgrade-test`, and `compose-export` commands are not available in standalone mode.

  ### Built-in reverse proxy for multi-instance local development

  When you run `astro dev start`, a built-in reverse proxy now routes `<project>.localhost:6563` to the correct local Airflow instance. This lets you run multiple Airflow projects simultaneously without manually configuring ports. The first project keeps the default port (8080), and subsequent projects are assigned random ports automatically.

  To view all running projects and their routes, run [`astro dev proxy status`](/docs/cli/v1.45/astro-dev-proxy). To disable the proxy for a single start, use `astro dev start --no-proxy`.

  When working in a git worktree, the URL includes both the worktree and repository names: `<worktree>.<repo>.localhost:6563`.

  ### New `astro telemetry` command

  A new [`astro telemetry`](/docs/cli/v1.45/astro-telemetry) command lets you manage anonymous usage telemetry for the Astro CLI. Telemetry is enabled by default and sends anonymous usage data using a fire-and-forget pattern with no impact on CLI responsiveness. Use [`astro telemetry disable`](/docs/cli/v1.45/astro-telemetry-disable) or set the `ASTRO_TELEMETRY_DISABLED` environment variable to `1` to opt out.

  ### Removed `astro registry` command

  The `astro registry` command and its subcommands (`dag add`, `provider add`) have been removed.

  ### Additional improvements

  * Added the `--kill` (`-k`) flag to [`astro dev restart`](/docs/cli/v1.45/astro-dev-restart). When used, this flag kills all running containers and removes all data before restarting. This is equivalent to running `astro dev kill` followed by `astro dev start`.
  * Added the `--fix` flag to [`astro dev upgrade-test`](/docs/cli/v1.45/astro-dev-upgrade-test) to automatically fix linting issues identified by the upgrade test using ruff.
  * Added the `--no-dags-base-dir` flag to [`astro deploy`](/docs/cli/v1.45/astro-deploy) for use with `--dags`. By default, Dag-only deploys place files in a `dags/` folder prefix. With `--no-dags-base-dir`, files are placed at the bundle root for compatibility with Airflow 3, which adds the bundle root to `sys.path` instead of the `dags/` folder.
  * Added `--api-server` and `--dag-processor` component filter flags to [`astro dev logs`](/docs/cli/v1.45/astro-dev-logs).
  * Astro IDE commands now display Workspace and project names instead of IDs in command output.
  * Fixed an issue that prevented organization switching by ID when the organization didn't appear in the initial paginated list.
  * Fixed Podman builds by passing the machine socket through correctly.
</Update>

<Update label="Astro CLI 1.39.0" description="February 17, 2026">
  ### Additional improvements

  * The Astro CLI now strips newline characters from entries in the dag integrity exceptions file, preventing parse failures caused by trailing newlines.
  * You can now use the `--session-id` flag with [`astro ide project import`](/docs/cli/v1.45/astro-ide-project-import) to specify a session ID when importing an Astro IDE project.
  * `astro remote deploy` now blocks deployment of client images with an Astro Runtime version newer than the target Deployment's Astro Runtime version to ensure compatibility.
  * Added support for Astro Private Cloud 1.1.
</Update>

<Update label="Astro CLI 1.38.1" description="November 4, 2025">
  ### Additional improvements

  * You can now import Astro IDE projects to non-empty directories with user confirmation.

  ### Bug fixes

  * Fixed `astro remote deploy` to correctly install dependencies from `requirements-client.txt` and `packages-client.txt` during image build.
</Update>

<Update label="Astro CLI 1.38.0" description="October 27, 2025">
  <Warning>
    **Known issue (resolved in 1.38.1)**

    Astro CLI 1.38.0 installs Remote Execution Agent Python and system dependencies from `requirements.txt` and `packages.txt` instead of `requirements-client.txt` and `packages-client.txt` when you run `astro remote deploy`. As a result, dependencies meant for the client may not be installed on the Remote Execution Agent as expected.

    **Impact:**
    If your remote execution depends on packages listed only in `requirements-client.txt` or `packages-client.txt`, these will not be installed, which can lead to missing dependencies or failures at runtime.

    **Solution:**
    Upgrade to Astro CLI 1.38.1 or later. This issue is fixed in Astro CLI version 1.38.1.
  </Warning>

  ### Enable remote execution with `astro dev init`

  You can now create a remote execution project using [`astro dev init`](/docs/cli/v1.45/astro-dev-init#remote-execution-mode). Setting remote execution mode prompts you to add the resources necessary for configuring your remote execution configuration.

  ### Behavior Changes

  <Warning>These changes only apply to Astro users. Please check the [CLI command reference](/docs/cli/v1.45/reference) for more information about CLI commands for Astro Private Cloud.</Warning>

  * Enabled `astro remote deploy` command by default
  * Deprecated `--force-upgrade-to-af3` flag from `astro deploy`
  * Added `--wait-time` flag for `astro deployment create`, `astro deploy`, `astro dbt deploy` commands, to allow you to configure wait time for the operation to finish. The default wait time is now 5 minutes.
</Update>

<Update label="Astro CLI 1.37.0" description="October 14, 2025">
  ### Support for Astro Private Cloud 1.0

  Added support for [Astro Private Cloud 1.0](/docs/astro-private-cloud/v-1-x/astro-private-cloud-overview), including the ability to push images and dags to separate data plane clusters.

  ### Additional improvements

  * The Deployment `get`, `list`, and `update` commands now display a new `REMOTE EXECUTION` column that indicates if Remote Execution is enabled for a Deployment.
  * References to **Astronomer Software** have been updated to **Astro Private Cloud** in the CLI and documentation.
  * A new `--workspace-id` flag has been added to `organization switch` for selecting a Workspace programmatically.

  ### Bug fixes

  * Update Airflow client to update `default_pool` correctly by sending only slots and `include_deferred` with an `update_mask`.
  * Ignore hidden files during Astro IDE project export.
</Update>

<Update label="Astro CLI 1.36.0" description="September 10, 2025">
  ### New Astro IDE commands

  <Info>
    **Public Preview**

    This feature is in [Public Preview](/docs/astro/feature-previews).
  </Info>

  Added the following commands so that you can work with the [Astro IDE](/docs/astro/ide-overview) from the Astro CLI:

  * [`astro ide project export`](/docs/cli/v1.45/astro-ide-project-export): Export your current local project to Astro IDE.
  * [`astro ide project import`](/docs/cli/v1.45/astro-ide-project-import): Import an existing Astro IDE project into your local environment for development or editing.
  * [`astro ide project list`](/docs/cli/v1.45/astro-ide-project-list): List all Astro IDE projects you have access to.

  ### Additional improvements

  Enhanced error handling during Docker image pushes in `astro deploy`. When a 403 Forbidden error occurs, the CLI now displays actionable troubleshooting guidance, including steps for clearing Docker credentials, disabling the `containerd` snapshotter, and a link to detailed support documentation.

  ### Bug fixes

  * Fixed an issue where the CLI would silently not load connections from the specified Astro Deployment or Workspace if the settings file was missing. The CLI now loads Astro connections regardless of whether the settings file, default `airflow_settings.yaml`, is present.
</Update>

<Update label="Astro CLI 1.35.1" description="July 31, 2025">
  ### Improved Airflow 3 support

  Astro CLI 1.35.1 introduces full compatibility with Apache Airflow 3, including support for the Astro Executor and Remote Execution Deployments on Astro. This release also adds the ability to manage connections, variables, and pools for Airflow 3 environments.

  #### New flags for `astro deployment create` and `astro deployment update`

  You can now use the new `AstroExecutor` option as an executor type for Airflow 3 Deployments. When using `AstroExecutor`, you can enable Remote Execution by setting the `--remote-execution-enabled` flag when you use `astro deployment create`. If Remote Execution is enabled, you can also configure additional Remote Execution mode Deployment settings with the following flags:

  * `--allowed-ip-address-ranges`: Limit the Deployment's incoming traffic to the Remote Agents in your environment.
  * `--task-log-bucket`: Specify the cloud storage bucket for task log storage.
  * `--task-log-url-pattern`: Specify the URL template to link to task logs stored in an external logging provider from the Airflow UI.

  ### Behavior changes

  * Astro CLI 1.35.1 removes support for pools in Airflow Versions 2.6 and below. Airflow 2.6 corresponds to Runtime Version 8, which has reached end of maintenance date.
  * Astro CLI 1.35.1 now interprets `--webserver` as `--apiserver` for Airflow 3 environments to reflect the updated component architecture.
</Update>

<Update label="Astro CLI 1.35.0" description="July 1, 2025">
  ### Restricted release

  The Astro CLI 1.35.0 was restricted from use on July 1, 2025, after its initial release. After upgrading to this version, v1.35.0 caused automatic, unintended updates to worker queues. Astro automatically blocks any requests coming from this CLI version.

  See the [Astro CLI Release and lifecycle policy](/docs/cli/v1.45/release-lifecycle-policy) for more information about different release channels and a list of restricted CLI versions.
</Update>

<Update label="Astro CLI 1.34.1" description="May 27, 2025">
  ### Additional improvements

  * Updated the `astro dev init` command to no longer require a container runtime during project initialization. This means you do not need to have something like Docker or Podman installed locally on your machine to run `astro dev init` succesfully.
  * Improved project linting.
  * Updated linting in `astro dev upgrade-test` to lint the entire project, not just the `dags/` directory, so issues in other directories like `include/` are now also detected.
</Update>

<Update label="Astro CLI 1.34.0" description="April 22, 2025">
  ### Airflow 3 support

  Astro CLI 1.34.0 adds support for Apache Airflow 3, including local development with Airflow 3 and the ability to test your project locally for upgrade incompatibilities.

  See [Upgrade to Airflow 3](/docs/astro/airflow3/upgrade-af3) for steps to upgrade your Astro project from Airflow 2 to Airflow 3.

  ### Bug fixes

  * Fixed an issue where the appropriate container engine for Airflow objects commands was not being used. Now, logic is in place to dynamically select Docker or Podman for Airflow object handling.
</Update>

<Update label="Astro CLI 1.33.2" description="March 17, 2025">
  ### Bug fixes

  * Software users pulling images from internal registries no longer see a warning when their Astro Project’s Dockerfile path differs from the public defaults.
</Update>

<Update label="Astro CLI 1.33.1" description="February 20, 2025">
  ### Bug fixes

  * Logs for `astro dev parse` and `astro dev pytest` commands are now displayed at the warn log level by default, making them visible without the `--verbosity=debug` flag for easier troubleshooting.
</Update>

<Update label="Astro CLI 1.33.0" description="January 16, 2025">
  ### New flags for [`astro deploy`](/docs/cli/v1.45/astro-deploy) on Astronomer Software

  You can now use the `--image` flag with `astro deploy` to trigger an image-only deploy on [Astronomer Software](/docs/cli/v1.45/astro-deploy).

  You can also specify an image with the `--image-name` for a `astro deploy` on Astronomer Software. If you specify an image, you must also specify the image's Runtime version with the `--runtime-version` flag. You can also use the new `--remote` flag with `--image-name` to directly point the deployment to the remote image and skip pushing the image.

  ### Orbstack Container Runtime Engine Support for Mac

  You can now use the Orbstack Container Runtime Engine with the Astro CLI on macOS. If Orbstack is running, containers start seamlessly. If not, the Orbstack app launches automatically, and containers start in the background. You can close the app while it continues running. This feature is available for macOS only. Windows and Linux users will need to start their runtime manually.

  ### Additional improvements

  * Introduced a new configuration option [`sha_as_tag`](/docs/cli/v1.45/configure-cli) for Astronomer Software to use the SHA Digest value when making a call to the `updateDeploymentImage` endpoint in Houston, instead of the Runtime version tag.

  ### Bug fixes

  * Fixed an issue where `deployments list --all` would not work on Astronomer Software.
</Update>

<Update label="Astro CLI 1.32.1" description="January 8, 2025">
  ### Additional improvements

  * Dag-only and dbt deploys now use gzip compression for tar bundles, significantly reducing deploy times for large bundles to cloud Deployments.
</Update>

<Update label="Astro CLI 1.32.0" description="December 17, 2024">
  ### Astro CLI now includes pre-configured Podman container runtime

  You can now create local Astro projects without manually installing a container management engine, on all supported operating systems. When you install the Astro CLI with Homebrew or Winget, it includes a pre-configured Podman container runtime. If you don't want to install Podman with the Astro CLI, you can opt out during installation, allowing the Astro CLI to use your existing runtime setup. See [Install the Astro CLI](/docs/cli/v1.45/install-cli).

  ### Additional improvements

  * You can now specify the project name as an optional positional argument in [`astro dev init`](/docs/cli/v1.45/astro-dev-init), simplifying project initialization by automatically creating and initializing a new directory.

  ### Behavior changes

  * Previously in the CLI configuration file, the `container.binary` parameter for the Astro CLI was set to `docker` by default. Now, the Astro CLI will attempt to auto-detect the container runtime to use, checking for `docker` then `podman` in that order. You can override the auto-detection with [`astro config set container.binary`](/docs/cli/v1.45/switch-container-management).
</Update>

<Update label="Astro CLI 1.31.0" description="November 22, 2024">
  ### New `--from-template` flag to setup template based project

  You can now specify a template name such as `etl`, `dbt-on-astro`, `generative-ai`, or `learning-airflow` with the `--from-template` flag for [`astro dev init`](/docs/cli/v1.45/astro-dev-init) and the CLI sets up an Astro project based on the provided template.

  ### New `--show-workload-identity` flag to fetch workload identity of Deployment

  You can now fetch the [workload identity](/docs/astro/authorize-deployments-to-your-cloud#what-is-workload-identity) value of a Deployment with the `--show-workload-identity` flag for [`astro deployment inspect`](/docs/cli/v1.45/astro-deployment-inspect).

  ### Additional improvements

  * Added the ability for the CLI to autodetect the container runtime binary to use for the commands that utilize containers.
  * Streamlined the behavior of `astro dev start` for Podman users.

  ### Behavior changes

  * You no longer need to explicitly set Podman as your container management engine for the Astro CLI when you are configuring the Astro CLI to use Podman, as the CLI now autodetects the container runtime binary to use for the commands that use containers.

  ### Bug fixes

  * Fixed an issue where Astro created all the worker queues with the configuration of the first queue in the list when using the Deployment config file to create or update a Deployment.
</Update>

<Update label="Astro CLI 1.30.0" description="October 16, 2024">
  ### New description flag for deploy on Astronomer Software

  You can now add custom deploy descriptions during the [`astro deploy`](/docs/cli/v1.45/astro-deploy) CLI command on Astronomer Software to enhance traceability and clarity in the deploy process. If you don't provide a description, the system will automatically assign a default description based on the code deploy type. For example, the default description for a dag-only deploy would be `Deployed via <astro deploy --dags>`.

  ### `astro deployment create` to create in Azure by default for Astro

  <Danger>
    **Breaking Change**

    Azure is now the default cloud provider when creating a deployment on Astro with the [`astro deployment create`](/docs/cli/v1.45/astro-deployment-create) CLI command.

    Use the `--cloud-provider` flag to specify `aws`, `gcp`, or `azure` as the cloud provider for the Deployment.
  </Danger>

  ### Additional improvements

  * You can now create Astro Hosted Deployments on Azure via Astro CLI.
</Update>

<Update label="Astro CLI 1.29.0" description="September 9, 2024">
  ### New flags for create and update Deployments

  * You can now add Extra Large schedulers to your Deployments when you create or update them with the new `extra_large` option for the `--scheduler` flag. This requires a minimum Astro Runtime version of 9.7.0.
  * You can also customize the workload identity when updating (AWS and GCP Hosted only) or creating (AWS Hosted only) Deployments directly with the command line with the `--workload-identity` flag or by adding it to the Deployment config file with the `workload_identity` field.

  See [`astro deployment create`](/docs/cli/v1.45/astro-deployment-create) and [`astro deployment update`](/docs/cli/v1.45/astro-deployment-update) for more information.
</Update>

<Update label="Astro CLI 1.28.1" description="July 25, 2024">
  ### Bug fixes

  * Fixed a bug where the CLI ignored files when bundling dags for code deploys.
</Update>

<Update label="Astro CLI 1.28.0" description="July 24, 2024">
  ### Work with dbt projects on Astro

  <Note>
    **Labs**

    This feature is in [Labs](/docs/astro/feature-previews). Contact your account team to enable this feature.
  </Note>

  You can now deploy a dbt project to Astro using the Astro CLI. See the following reference pages for more information:

  * [astro dbt deploy](/docs/cli/v1.45/astro-dbt-deploy)
  * [astro dbt delete](/docs/cli/v1.45/astro-dbt-delete)

  ### Additional improvements

  * Added the ability to switch Workspaces by using the Workspace name instead of the `workspace-id`. See [astro workspace switch](/docs/cli/v1.45/astro-workspace-switch).

  ### Bug fixes

  * Fixed a bug where the command `astro deployment inspect --key configuration.is_development_mode` was printing wrong the value.
</Update>

<Update label="Astro CLI 1.27.1" description="May 16, 2024">
  ### Bug fixes

  * Fixed an issue where the API token expiration check was causing login failures with API tokens that did not have an expiration date.
</Update>

<Update label="Astro CLI 1.27.0" description="May 16, 2024">
  ### New flags for the Deployment logs commands:

  You can now filter logs for specific Deployment components using the following new flags for `astro deployment logs`:

  * `--webserver`
  * `--scheduler`
  * `--triggerer`
  * `--worker`

  ### Exclude dag files from parse test

  You can now exclude dag files from being tested when you run `astro dev parse`.

  All new Astro projects that you create with `astro dev init` now include a file named `.astro/dag_integrity_exceptions.txt`. Add the names of dags to this file to exclude them from being tested when you run `astro dev parse`. This allows you to exclude dags that you know will not pass your tests.

  To use this feature in an existing Astro project, delete the `.astro/test_dag_integrity_default.py` file  in your Astro project, then run `astro dev init`. After you run this command, the Astro CLI creates a new default test file along with a `.astro/dag_integrity_exceptions.txt` text file.

  ### Additional improvements

  * Astro projects no longer have to include dags in order to run `astro deploy --image`.
  * You can now append `2>/dev/null | head` to commands to disregard upgrade messages. For example, running `astro completion bash 2>/dev/null | head` ensures that the resulting bash script remains unaffected by the upgrade message.
  * You can now use the `--development-mode disable` flag with `astro deployment update` to turn off [development mode](https://docs.astronomer.io/astro/deployment-resources#hibernate-a-development-deployment) for an existing Deployment. Note that you still cannot turn on development mode for an existing Deployment.

  ### Bug fixes

  * Fixed an issue where you couldn't create two Deployments with identical names across different Workspaces.
  * The `upgrade-test` command now returns the correct error code, ensuring accurate feedback during testing and CI/CD.
</Update>

<Update label="Astro CLI 1.26.0" description="April 24, 2024">
  ### New commands to assign Organization and Workspace API tokens at different levels

  You can now use the Astro CLI to manage Organization and Workspace API tokens at the Workspace and Deployment level using the following commands:

  * [`astro deployment token organization-token`](/docs/cli/v1.45/astro-deployment-token-organization-token)
  * [`astro deployment token workspace-token`](/docs/cli/v1.45/astro-deployment-token-workspace-token)
  * [`astro workspace token organization-token`](/docs/cli/v1.45/astro-workspace-token-organization-token)

  For more information about this feature, see:

  * [Assign an Organization or Workspace API token to a Deployment](/docs/astro/deployment-api-tokens#assign-an-organization-or-workspace-api-token-to-a-deployment).
  * [Assign an Organization API token to a Workspace](/docs/astro/workspace-api-tokens#assign-an-organization-api-token-to-a-workspace)

  ### Bug fixes

  * Fixed an issue where existing secret environment variables in a Deployment file could be applied to the Deployment with an empty value. The secret variable value now persists.
  * Fixed an issue where `astro deployment inspect` generated an incorrect Airflow API URL.
  * Fixed a bug that caused some input checks for `astro deployment upgrade-checks` to fail against valid inputs.
</Update>

<Update label="Astro CLI 1.25.0" description="March 28, 2024">
  ### Manage Deployment API tokens with the Astro CLI

  You can now manage [Deployment API tokens](/docs/astro/deployment-api-tokens) using the following CLI commands:

  * `astro deployment token create`
  * `astro deployment token list`
  * `astro deployment token update`
  * `astro deployment token rotate`
  * `astro deployment token delete`

  ### Additional improvements

  * Updated the example dags that the Astro CLI creates when you run `astro dev init`.
  * The CLI now tells you if your API token is invalid.

  ### Bug fixes

  * Fixed an issue with `deployment variable create` where it would cut off the new variables value at the first "=" character.
  * Fixed an issue where running a deployment command in a workspace without a deployment caused an error. Now users will be asked if they want to create a deployment if one does not exist.
  * Fixed an issue where `astro dev start —deployment-id` was not creating local connections correctly in some scenarios.
  * Fixed an issue where `astro deployment` commands could only list 20 deployments. Now the commands will list up to 1000 for each workspace.
</Update>

<Update label="Astro CLI 1.24.1" description="February 29, 2024">
  ### Bug fixes

  * Fixed an issue where the Astro CLI would experience a code panic if you tried to set a hibernation schedule for a Deployment that didn't exist.
  * Fixed an issue where the Astro CLI would send and retrieve hibernation schedules for non-development Deployments.
</Update>

<Update label="Astro CLI 1.24.0" description="February 27, 2024">
  ### Support for hibernating development Deployments

  You can now use the Astro CLI to hibernate or wake up a development Deployment. These commands work well in automated processes where a Deployment requires flexibility for when it hibernates. Note that you can hibernate a Deployment only if you enable **Development Mode** when you [create the Deployment](/docs/cli/v1.45/astro-deployment-create).

  Use the following new commands to hibernate development Deployments regardless of their existing hibernation schedule:

  * [`astro deployment hibernate`](/docs/cli/v1.45/astro-deployment-hibernate)
  * [`astro deployment wake-up`](/docs/cli/v1.45/astro-deployment-wake-up)

  Additionally, you can create new development Deployments and configure long-term hibernation schedules for them using `astro deployment create`.

  ### Additional improvements

  * You can now configure a custom workload identity when you create a Deployment using a Deployment file.
  * Added support for the upcoming custom role management feature on Astro

  ### Bug fixes

  * Fixed an issue where `astro deployment variable list --save` didn't format secret environment variables correctly.
  * Fixed an issue where you couldn't update a Deployment with a Deployment file using a Deployment API token.
</Update>

<Update label="Astro CLI 1.23.0" description="February 14, 2024">
  ### Changes to existing CLI command flags

  The following flags have been updated, but will continue to work with a deprecation notice until the v1.25.0 release of the Astro CLI:

  * `astro deployment logs --key-word` ia a new flag that allows you to search your Deployment's logs for an exact key word or phrase.
  * `astro deployment create --cluster-type` is now `astro deployment create --type`.
  * `astro deployment create --enforce-cicd` is now `astro deployment create --cicd-enforcement`.

  ### Kubernetes worker configurations are now consistent with the Astro Cloud UI

  You can now use Deployment files or the Astro CLI to create or update Kubernetes worker configurations.

  [Deployment files](/docs/astro/deployment-file-reference) now include some new and updated fields for Deployment configuration to match the options available in the Astro UI. This also allows you to create or update Kubernetes worker configurations directly, instead of requiring you to update the worker resources by changing the Kubernetes worker queue configuration.

  You can now use the `default_task_pod_cpu`, `default_task_pod_memory`, `default_worker_type`, `resource_quota_cpu`, and `resource_quota_memory` fields in a Deployment file to update your Kubernetes workers instead of creating or updating a Kubernetes default worker queue.

  With this new functionality, the following commands to update a Deployment running the Kubernetes executor continue to work, but display a deprecation notice:

  * `astro deployment worker-queue create`
  * `astro deployment worker-queue update`

  Instead, you can use the new `--default-task-pod-cpu`, `--default-task-pod-memory`, `--resource-quota-cpu`, or `--resource-quota-memory` flags with [`astro deployment create`](/docs/cli/v1.45/astro-deployment-create) and [`astro deployment update`](/docs/cli/v1.45/astro-deployment-update) to edit your Kubernetes worker configuration using the Astro CLI.

  ### Changes to Deployment file configurations

  The following changes have been made to the format of [Deployment files](/docs/astro/deployment-file-reference):

  * You no longer have to specify a `cluster_name` for standard Deployment files.
  * `scheduler_size` is no longer case sensitive.
  * Possible values for `cloud_provider` are now `gcp`, `aws`, and `azure`. This input is not case sensitive.
  * Possible values for `deployment_type` now include `standard`, `dedicated`, and `hybrid` in addition to the existing values of `hosted_shared`, `hosted_dedicated`, and `hosted_standard`. This input is not case sensitive
  * Possible values for for the `executor` field are now include `celery` and `kubernetes`. `CeleryExecutor` and `KubernetesExecutor` still work. This input is not case sensitive, so, for example, `celeryexecutor` still works.
  * (*Astro Hosted only*) `default_task_pod_cpu`, `default_task_pod_memory`, `resource_quota_cpu`, and `resource_quota_memory` are new fields for Astro Hosted deployments.
  * (*Astro Hybrid only*)`default_worker_type` is a new field for Hybrid deployments that use the Kubernetes executor.

  ### Additional improvements

  * You can now trigger a dag-only deploy on Astronomer Software using `astro deploy --dags`. See [Deploy dags on Astronomer Software](https://www.astronomer.io/docs/software/deploy-dags).
  * `astro deployment logs --key-word` is a new flag that allows you to search your audit logs for an exact key word or phrase.
  * If you log in to Astro from the CLI, you need to select a Deployment when you deploy code. Previously, the Astro CLI used auto-select to automatically choose a Deployment for code deploys based on the CLI context. Now, by default, the CLI does not auto-selects the Deployments where your code deploys when you use it. However there are the following exceptions:
    * If you log in to Astro with an API token using the `ASTRO_API_TOKEN`, `ASTRONOMER_KEY_ID`, or `ASTRONOMER_KEY_SECRET` environment variables, auto-select is enabled. This is important because it ensures that if you have CI/CD scripts that rely on auto-select, they will continue to work.
    * There is a new config, `auto_select`. If `auto-select` is set to `true` in the config file, auto-select is always enabled.

  ### Bug fixes

  * Fixed an issue where `astro dev pytest --args` and `astro dev pytest --build-secrets` could fail.
</Update>

<Update label="Astro CLI 1.22.0" description="January 24, 2024">
  ### New flag to mount secrets to Astro project image

  Use the new `--build-secrets` flag with the following commands to mount a secret value to an Astro project image:

  * `astro deploy`
  * `astro dev parse`
  * `astro dev pytest`
  * `astro dev restart`
  * `astro dev start`
  * `astro dev upgrade test`

  This flag is equivalent to running [`docker build --secret`](https://docs.docker.com/build/building/secrets/#secret-mounts) for your Astro Runtime image build. Use this flag to simplify build steps for customizing the Astro Runtime image, for example when you need to [install Python packages from a private source](/docs/cli/v1.45/private-python-packages?tab=pypi#install-python-packages-from-a-private-pypi-index) .
</Update>

<Update label="Astro CLI 1.21.0" description="December 4, 2023">
  ### New command to deploy only images

  You can use the new `astro deploy --image` command to deploy only the image to you Deployment. Previously, you could either complete a full code deploy with `astro deploy` or only update your dags with a dag-only deploy. See [Trigger an image-only deploy](/docs/astro/deploy-dags#trigger-an-image-only-deploy) for more information.

  ### Bug fixes

  * Fixed a bug so you can now use a deployment file to create a Kubernetes deployment with a custom worker queue.
</Update>

<Update label="Astro CLI 1.20.1" description="November 8, 2023">
  ### Bug fixes

  * Fixed an issue where `astro deployment airflow-variable`, `astro deployment connection`, and `astro deployment pool` commands were returning the error `failed to decode response from API`.
</Update>

<Update label="Astro CLI 1.20.0" description="November 7, 2023">
  ### Bug fixes

  * Fixed an issue where `astro workspace users list` occasionally failed to return a table.
  * Fixed an issue introduced in version 1.19.4 where you could not deploy a custom image to Astro.
</Update>

<Update label="Astro CLI 1.19.4" description="November 1, 2023">
  ### Additional improvements

  * The Astro CLI now shows a warning if you attempt to deploy a project with an empty `dags` folder to Astro. To remove this warning along with all other CLI warnings, run `astro config set show_warnings false`.

  ### Bug fixes

  * Fixed an issue where `astro deployment create` sometimes showed an invalid Runtime version error for valid Runtime versions.
</Update>

<Update label="Astro CLI 1.19.3" description="October 12, 2023">
  ### Additional improvements

  * Sample test `test/dags/test_dag_integrity.py` was renamed to `test/dags/test_dag_example.py` to highlight that this test is an example.

  ### Bug fixes

  * Fixed an issue where CI/CD pipelines were unable to use Deployments as Code to create Deployment Previews for Deployments using the Kubernetes executor.
  * Fixed an issue where the CLI was asking users to select from the wrong regions when creating a Deployment on an AWS cluster.
  * Fixed an issue where secret variables values were being printed to local logs.
</Update>

<Update label="Astro CLI 1.19.2" description="September 14, 2023">
  ### Additional improvements

  * When you run `astro dev upgrade-test`, the generated HTML report for dag tests now shows how many dags passed and failed the test.

  ### Bug fixes

  * Fixed an issue where you couldn't create a Deployment on a dedicated cluster using a Deployment file and API token.
  * Fixed an issue where the Deployment URL that appears after you run `astro deploy` was not formatted properly.
  * Fixed an issue where `astro dev upgrade-test` would occasionally output that it was testing an upgrade to the latest version of Astro Runtime, even if it wasn't.
  * Fixed an issue where `astro dev upgrade-test` didn't produce an HTML report for dag tests.
</Update>

<Update label="Astro CLI 1.19.1" description="August 30, 2023">
  ### Bug fixes

  * Fixed an issue where dags would fail to parse correctly when running `astro dev parse` or `astro deploy`, resulting in a command execution failure.
</Update>

<Update label="Astro CLI 1.19.0" description="August 29, 2023">
  ### Additional improvements

  * You can now grant Astro users the `WORKSPACE_AUTHOR` role.
  * You can now run an Astro project from the same directory an Apache Airflow project.
  * `astro deployment inspect` now shows you a Deployment's workload identity.

  ### Bug fixes

  * Fixed an issue where some dags could be missed during `astro deploy` when dag-only deploys are enabled.
  * Fixed an issue where `astro dev pytest` would incorrectly fail when testing an Astro project within a CI/CD process.
  * Fixed an issue where you couldn't update a Deployment on a standard cluster using a Deployment file.
</Update>

<Update label="Astro CLI 1.18.2" description="August 10, 2023">
  ### Bug fixes

  * Fixed an issue where running `astro deployment create` on Astro Hosted would create Deployments where dag-only deploys were turned off by default.
</Update>

<Update label="Astro CLI 1.18.1" description="August 4, 2023">
  ### Bug fixes

  * Fixed an issue where `astro run` didn't work properly.
</Update>

<Update label="Astro CLI 1.18.0" description="August 3, 2023">
  ### New command to test Astro projects before you upgrade

  You can use the new `astro dev upgrade-test` command to anticipate and address problems before upgrading to a newer version of Astro Runtime. The command runs several test which let you determine whether an upgrade will result in major dependency changes and import errors, allowing you to fix the problems before you upgrade. See [Test your Astro project locally](/docs/cli/v1.45/test-your-astro-project-locally) for more information.

  ### Additional improvements

  * You can now specify the `--description` flag with `astro deploy` to add a description for your deploy. You can use this description to let other users know why you made a deploy or what changes a deploy contains.
  * You can now specify the `--role` flag with `astro organization team create/update` to update a Team's Organization-level role.
  * You can now specify the `--execution-date` flag with `astro run` to trigger a dag run for a specific execution date.
  * You can now specify the `--verbose` flag with `astro run` to stream all logs to your terminal after the dag run triggers.

  ### Bug fixes

  * Fixed an issue where `astro deployment inspect` was showing the wrong value for a Deployment’s workload identity on Astro Hosted.
  * Fixed an issue were `astro dev restart` would occasionally not work.
</Update>

<Update label="Astro CLI 1.17.1" description="July 12, 2023">
  ### Bug fixes

  * Fixed an issue were some Astro Hosted Deployment updates triggered by the Astro CLI were not working.
</Update>

<Update label="Astro CLI 1.17.0" description="July 6, 2023">
  ### Manage Organization API tokens with the Astro CLI

  You can now manage [Organization API tokens](/docs/astro/organization-api-tokens) using the following CLI commands:

  * [`astro organization token create`](/docs/cli/v1.45/astro-organization-token-create)
  * [`astro organization token roles`](/docs/cli/v1.45/astro-organization-token-roles)
  * [`astro organization token list`](/docs/cli/v1.45/astro-organization-token-list)
  * [`astro organization token update`](/docs/cli/v1.45/astro-organization-token-update)
  * [`astro organization token rotate`](/docs/cli/v1.45/astro-organization-token-rotate)
  * [`astro organization token delete`](/docs/cli/v1.45/astro-organization-token-delete)

  ### Download resources from the Astronomer Registry to your Astro project

  You can now use the following commands to download resources from the Astronomer Registry:

  * `astro registry dag add`
  * `astro registry provider add`

  The Astro registry contains dags and provider packages that are ready to use out of the box. Use these commands to quickly add resources to an Astro project, which you can then run locally or on Astro.

  ### Additional improvements

  * You can now create Deployments on AWS standard clusters.
  * If you belong to only one Workspace, the Astro CLI now uses that Workspace by default for all commands.

  ### Bug fixes

  * Fixed an issue where the Astro CLI could not retrieve the health status of a Deployment.
  * Fixed an issue where you could not set `worker_concurrency` to 0 in a Deployment file.
</Update>

<Update label="Astro CLI 1.16.2" description="June 30, 2023">
  ### Bug fixes

  * Fixed an issue where the `isHighAvailability` and `CICDEnforcement` fields in Deployment files were not processed correctly.
</Update>

<Update label="Astro CLI 1.16.1" description="June 13, 2023">
  ### Manage Teams using the Astro CLI

  You can now manage [Astro Teams](/docs/astro/manage-teams) using the following CLI commands:

  * [`astro workspace team add`](/docs/cli/v1.45/astro-workspace-team-add)
  * [`astro workspace team list`](/docs/cli/v1.45/astro-workspace-team-list)
  * [`astro workspace team update`](/docs/cli/v1.45/astro-workspace-team-update)
  * [`astro workspace team remove`](/docs/cli/v1.45/astro-workspace-team-remove)
  * [`astro organization team create`](/docs/cli/v1.45/astro-organization-team-create)
  * [`astro organization team list`](/docs/cli/v1.45/astro-organization-team-list)
  * [`astro organization team update`](/docs/cli/v1.45/astro-organization-team-update)
  * [`astro organization team delete`](/docs/cli/v1.45/astro-organization-team-delete)
  * [`astro organization team user remove`](/docs/cli/v1.45/astro-organization-team-user)
  * [`astro organization team user add`](/docs/cli/v1.45/astro-organization-team-user)
  * [`astro organization team user list`](/docs/cli/v1.45/astro-organization-team-user)

  You can use these commands in automated workflows with [Workspace API tokens](/docs/astro/workspace-api-tokens) and [Organization API tokens](/docs/astro/organization-api-tokens).

  ### Manage Workspace API tokens with the Astro CLI

  You can now manage [Workspace API tokens](/docs/astro/workspace-api-tokens) using the following CLI commands:

  * [`astro workspace token create`](/docs/cli/v1.45/astro-workspace-token-create)
  * [`astro workspace token add`](/docs/cli/v1.45/astro-workspace-token-add)
  * [`astro workspace token list`](/docs/cli/v1.45/astro-workspace-token-list)
  * [`astro workspace token update`](/docs/cli/v1.45/astro-workspace-token-update)
  * [`astro workspace token rotate`](/docs/cli/v1.45/astro-workspace-token-rotate)
  * [`astro workspace token delete`](/docs/cli/v1.45/astro-workspace-token-delete)

  These commands can be used to manage API tokens as part of an automated workflow.

  ### Additional improvements

  * You can now specify the `--cluster-type "dedicated"` flag when using `astro deployment create` to create a Deployment on a dedicated cluster in Astro Hosted.
  * You can now retrieve a Deployment's Workload Identity when using `astro deployment inspect`.
  * You can now specify the `--enforce-cicd` flag with `astro deployment create` and `astro deployment update` to [enforce CI/CD](/docs/astro/deployment-details#enforce-ci/cd-deploys) on a given Deployment.
  * You can now [manage Deployments as code](/docs/astro/manage-deployments-as-code) on Astro Hosted.
</Update>

<Update label="Astro CLI 1.15.1" description="May 19, 2023">
  ### Bug fixes

  * Fixed an issue where you could not create a Deployment on a standard Hosted cluster.
</Update>

<Update label="Astro CLI 1.15.0" description="May 18, 2023">
  ### New commands to manage Airflow resources on Deployments

  Use the following new Astro CLI commands to manage your Airflow variables, pools, connections, on Astro Deployments. These commands are particularly useful for automating the creation of new Deployments based on old ones, as you can now transfer all Airflow resources from a source Deployment to a target Deployment:

  * [`astro deployment connection list`](/docs/cli/v1.45/astro-deployment-connection-list)
  * [`astro deployment connection create`](/docs/cli/v1.45/astro-deployment-connection-create)
  * [`astro deployment connection update`](/docs/cli/v1.45/astro-deployment-connection-update)
  * [`astro deployment connection copy`](/docs/cli/v1.45/astro-deployment-connection-copy)
  * [`astro deployment airflow-variable list`](/docs/cli/v1.45/astro-deployment-airflow-variable-list)
  * [`astro deployment airflow-variable create`](/docs/cli/v1.45/astro-deployment-airflow-variable-create)
  * [`astro deployment airflow-variable update`](/docs/cli/v1.45/astro-deployment-airflow-variable-update)
  * [`astro deployment airflow-variable copy`](/docs/cli/v1.45/astro-deployment-airflow-variable-copy)
  * [`astro deployment pool list`](/docs/cli/v1.45/astro-deployment-pool-list)
  * [`astro deployment pool create`](/docs/cli/v1.45/astro-deployment-pool-create)
  * [`astro deployment pool update`](/docs/cli/v1.45/astro-deployment-pool-update)
  * [`astro deployment pool copy`](/docs/cli/v1.45/astro-deployment-pool-copy)

  ### Additional improvements

  * You can now use the `--args` flag to specify pytest arguments to run with `astro dev pytest`. For example, you can run `astro dev pytest --args "-p pytest_cov"` to plugin the `pytest_cov` plugin with your pyests.
  * You can now use Organization API tokens to automate Astro CLI tokens. Specify the Organization API token using the environment variable `ASTRO_API_TOKEN` in the environment where you run the Astro CLI.
  * You can now create a custom Docker/Podman compose file for your Astro project with the command `astro dev object export --compose`. After you modify the file, you can use it to start your project with `astro dev start --compose-file <compose-file-location>`.
  * You can now set `postgres.repository` and `postgres.tag` with `astro config set`. You can use these configurations to customize the postgres database used in your local Airflow environments.
  * The Astro CLI now automatically trims quotation marks from the beginning and end of environment variables being pushed to Astro.
  * The command `astro user invite` has been deprecated.

  ### Bug fixes

  * Fixed an issue were `astro deployment variable create/update` was not producing error when it failed to create an environment variable.
  * Fixed an issue were Podman deploys were failing if the user didn't have the Docker CLI installed.
</Update>

<Update label="Astro CLI 1.14.1" description="April 20, 2023">
  ### Bug fixes

  * Fixed an issue where `astro workspace user list` didn't work when using a Workspace API token.
</Update>

<Update label="Astro CLI 1.14.0" description="April 19, 2023">
  ### New commands to manage Astro Workspaces

  You can now manage Astro Workspaces from the Astro CLI using the following new commands:

  * [`astro workspace create`](/docs/cli/v1.45/astro-workspace-create)
  * [`astro workspace update`](/docs/cli/v1.45/astro-workspace-update)
  * [`astro workspace delete`](/docs/cli/v1.45/astro-workspace-delete)

  To automate Workspace management, you can run these commands using a [Workspace API token](/docs/astro/workspace-api-tokens).
</Update>

<Update label="Astro CLI 1.13.2" description="April 11, 2023">
  ### Bug fixes

  * Fixed an issue where the CLI added the `dags` folder to `.dockerignore` whenever an image build was interrupted, resulting in dags not being deployed on the next image build.
</Update>

<Update label="Astro CLI 1.13.0" description="March 30, 2023">
  <Warning>The command `astro user invite` will be deprecated in Astro CLI v1.15.0. Any use of this command in your projects or automation needs to be updated to [`astro organization user invite`](/docs/cli/v1.45/astro-organization-user-invite) before Astro CLI v1.15.0 is released.</Warning>

  ### New flag `--clean-output` for Deployment commands

  You can now use the `-—clean-output` flag with the following commands to make sure that any output comes only from the command itself.

  * `astro deployment inspect`
  * `astro deployment create`
  * `astro deployment update`

  This is helpful for users automating actions with deployment files, like using the Deploy Action template with [Github Actions](/docs/astro/ci-cd-templates/template-overview).

  ### New environment variable `ASTRO_HOME`

  The new environment variable `ASTRO_HOME` allows you to change the directory where the Astro CLI stores its global config file. This can be useful in environments where the CLI doesn’t have access to the HOME directory.

  ### Additional improvements

  * The command `astro login` won’t ask for email input in the command line anymore. You can now provide your email address in the browser when you log in.
</Update>

<Update label="Astro CLI 1.12.1" description="March 22, 2023">
  ### Bug fixes

  * Fixed an issue where you couldn't authenticate to the Astro from the Astro CLI using single sign-on (SSO).
</Update>

<Update label="Astro CLI 1.12.0" description="March 22, 2023">
  ### Additional improvements

  * You can now expose your local Airflow webserver and postgres database to all networks you're connected to using the following command:

    ```sh wrap theme={null}
    astro config set airflow.expose_port true
    ```

  * When you trigger a dag deploy to Astro, the CLI now includes the name of the dag bundle version that it pushed. You can use this name to verify that your Deployment uses the correct version of your dags after a deploy.

  * If you add the environment variable `ASTRO_API_TOKEN=<workspace-api-token>` to your environment, the Astro CLI will use the specified Workspace API token to perform Workspace and Deployment actions without requiring you to log in.

  * You can now disable [`astro run`](/docs/cli/v1.45/astro-run) commands and exclude `astro-run-dag` from any images built by the CLI using the following command:

    ```sh wrap theme={null}
    astro config set disable_astro_run true
    ```

  * In new Astro projects, `requirements.txt` now includes a commented list of the pre-installed provider packages on Astro Runtime.

  ### Bug fixes

  * Fixed an issue where the default dag integrity test would sometimes generate an error for valid uses of `os.getenv(key,default)`.
  * Fixed bugs in the default Astro project dags.
</Update>

<Update label="Astro CLI 1.11.0" description="February 27, 2023">
  ### Support for Podman

  You can now configure the Astro CLI to run Airflow locally and deploy to Astro using [Podman](https://podman.io/). Podman is an alternative container engine to Docker that doesn't require root access and orchestrates containers without using a centralized daemon.

  To configure the Astro CLI to use Podman, see [Run the Astro CLI using Podman](/docs/cli/v1.45/use-podman).

  ### Bug fixes

  * Fixed an issue where you couldn't run Astro CLI commands with a Deployment API key if you logged out of your personal account using `astro logout`.
  * Fixed an issue where you couldn't set the minimum worker count for a worker queue to zero.
  * Fixed an issue where running `astro deploy` would not return an error when you specified a Deployment name that didn't exist.
  * Fixed an issue where you could not update a Deployment with a file using a Deployment API key.
</Update>

<Update label="Astro CLI 1.10.0" description="February 2, 2023">
  ### New commands to manage Astro users

  To help you manage users in your Organization, Astro CLI 1.10.0 includes the following new commands:

  * `astro organization user invite`: Invite a new user to your Astronomer Organization.
  * `astro organization user update`: Update a user's Organization role.
  * `astro organization user list`: List all users in your Organization.
  * `astro workspace user add`: Add a user to a Workspace.
  * `astro workspace user update`: Update a user's role in a Workspace.
  * `astro workspace user list`: List all users in a Workspace.
  * `astro workspace user remove`: Remove a user from a Workspace.

  <Info>`astro organization user invite` is identical to the existing `astro user invite` command. `astro user invite` will be deprecated in a future release.</Info>

  For more information, see the [`astro organization`](/docs/cli/v1.45/astro-organization-user-invite) and [`astro workspace`](/docs/cli/v1.45/astro-workspace-user-add) command references.
</Update>

<Update label="Astro CLI 1.9.0" description="January 13, 2023">
  ### Manage Astro Deployments as code

  Astro CLI version 1.9 includes three new commands that make it possible to programmatically create and update Deployments:

  * `astro deployment inspect --template`: Create a template file in YAML for an existing Deployment. This template file includes all information about the Deployment in its current state, including worker queue configurations, environment variables, and Astro Runtime version.
  * `astro deployment create --deployment-file`: Create a new Deployment with the configurations specified in a template file.
  * `astro deployment update --deployment-file`: Update an existing Deployment based on the values in a Deployment file.

  You can use template and Deployment files to define Astro Deployments as code. For example, if your team regularly creates and deletes Deployments for testing, you can use template files to avoid manually copying configurations in the Astro UI. For more information, see [Astro CLI command reference](/docs/cli/v1.45/astro-deployment-create).

  ### New `--dag-file` flag for `astro run`

  By default, the `astro run` command parses all of the dags in your `dags` directory even if you are only running one dag.

  In Astro CLI 1.9, you can instead use the `--dag-file` flag to run a specific dag file without parsing all other dags in your directory. Specifying an individual dag file makes it easier to troubleshoot errors for that dag and results in faster execution of the command.

  ### Additional improvements

  * When you run Airflow locally, you no longer need to enter credentials to log in to the Airflow UI.
  * When you run Airflow locally, you can now access to the Airflow UI **Configurations** page (**Admin** > **Configurations**). This page shows the current configuration for your environment, including environment variables and Astro Runtime defaults.
  * The Astro CLI now reminds you when a new version of the Astro CLI is available. To turn this feature off, run `astro config set -g upgrade_message false`.
</Update>

<Update label="Astro CLI 1.8.4" description="December 12, 2022">
  ### Additional improvements

  * The `__pycache__/` directory is now included in the `.gitignore` file of an Astro project by default. `__pycache__/` includes compiled versions of dag and Python files that are automatically generated and should not be committed to Git.
  * Clarified the message that appears when you run `astro deployment update --dag-deploy enable` and dag-only deploys were already enabled for the Deployment.

  ### Bug fixes

  * Fixed an issue related to the [SQLAlchemy connection](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#sql-alchemy-conn) (\[`sql_alchemy_conn`]) in local Airflow environments. Now, users running Airflow 2.3 or above do not see deprecation warnings for SQLAlchemy in logs for locally running Airflow components.
</Update>

<Update label="Astro CLI 1.8.3" description="November 28, 2022">
  ### Additional improvements

  * Improved error handling for `astro login`.
  * Added minor performance improvements to `astro run`

  ### Bug fixes

  * Fixed an issue where `astro run` could not locate `airflow_settings.yaml` when running a local Airflow environment.
  * Fixed an issue were the Airflow settings file created by `astro dev object export` was not compatible with `astro run`.
</Update>

<Update label="Astro CLI 1.8.1" description="November 23, 2022">
  ### Bug fixes

  * Fixed an issue where you could not use `astro deploy` if you did not have an `.env` file in your Astro project.
</Update>

<Update label="Astro CLI 1.8.0" description="November 23, 2022">
  ### New `astro run` command

  You can now use the `astro run` command to run and debug a dag from the command line without starting a local Airflow environment. When you run the command, the CLI compiles your dag and runs it in a single Airflow worker container based on your Astro project configurations. You can see task success or failure, as well as task logs, directly in your terminal.

  This command is an alternative to running `astro dev restart` every time you make a change to your dag. Running dags without a scheduler or webserver improves the speed at which you can develop and test data pipelines.

  To learn more, see [Test your Astro project locally](/docs/cli/v1.45/test-your-astro-project-locally).

  ### Additional improvements

  * When you run `astro deploy` with an empty `dags` folder, the CLI excludes your `dags` folder when building and pushing an image of your project to Astro. This lets you manage your dags and project files in separate repositories when using [dag-only deploys](/docs/astro/deploy-dags).
  * The `deployment inspect` command now includes a `dag-deploy-enabled` field, and the fields are now ordered in logical groupings instead of by alphabetical order.

  ### Bug fixes

  * Fixed an issue where configurations specified in the `docker-compose.override.yaml` file of an Astro project were not properly applied.
  * Fixed an issue where `astro login` didn’t recognize some valid domains.
</Update>

<Update label="Astro CLI 1.7.0" description="November 9, 2022">
  ### Deploy only dags with `astro deploy -—dags`

  Use `astro deploy -—dags` with the Astro CLI to push only the `dags` directory of your Astro project to a Deployment on Astro. This is an additional option to `astro deploy`, which pushes all files in your Astro project every time you deploy your code to Astro.

  Deploying only dags:

  * Is significantly faster than running `astro deploy` when you only make changes to the `dags` directory.
  * Does not cause your workers and schedulers to terminate and restart every time you make a change to a dag and does not result in downtime for your Deployment.
  * Enables your team to create separate CI/CD processes for deploying dags and deploying other changes to your Astro project.

  When you make changes to other files in your Astro project that aren't in the `dags` directory, the `astro deploy` command is still required.

  To use this feature, you must enable it for each Deployment. See [Deploy dags only](/docs/astro/deploy-dags). For example CI/CD workflows with this feature enabled, see [CI/CD](/docs/astro/set-up-ci-cd).

  ### New `astro deployment inspect` command

  You can now run `astro deployment inspect` to return a Deployment's current state and configuration as a JSON or YAML object. This includes worker queue settings, Astro Runtime version, and more. Use this command to quickly understand the state of your Deployment as code and as an alternative to viewing it in the Astro UI.

  For more information, see the [CLI command reference](/docs/cli/v1.45/astro-deployment-inspect).

  ### Additional improvements

  * The outputs for `astro dev parse` and `astro dev pytest` commands have improved legibility by no longer including Docker container logs.
  * The `astro organization switch` command now includes a `-—login-link` flag that you can use to manually log in if you don't have access to a web browser.
  * You can now provide either an Organization name or ID when running `astro organization switch`.
  * `astro dev start` now times out if the Airflow webserver does not become healthy within a set period of time. Use the `-—wait` flag to specify a wait time in seconds or minutes.

  ### Bug fixes

  * Fixed an issue where `astro deploy` with `colima` was failing due to an issue with registry authentication
  * Fixed an issue where `astro deployment list` didn't display the Workspace ID for a Deployment
</Update>

<Update label="Astro CLI 1.6.1" description="November 3, 2022">
  ### Bug fixes

  * Fixed an issue where authenticating to Astronomer Software with `interactive=true` in your CLI configuration resulted in a 502 error.
</Update>

<Update label="Astro CLI 1.6.0" description="September 28, 2022">
  ### New commands to manage Airflow objects

  You can use the new `astro dev object` commands to better manage Airflow connections, variables, and pools between your local testing environment and Astro Deployments.

  * `astro dev object import` imports connections, variables, and pools from your Astro project `airflow_settings.yaml` into your locally running Airflow environment.
  * `astro dev object export` exports connections, variables, and pools from your local airflow database to a file of your choosing. specify the `--env-export` flag to export Airflow connections and variables to your `.env` file as Astro environment variables.

  These commands enable you to:

  * Update objects in a locally running Airflow environment without restarting it.
  * Quickly move Airflow objects from a local testing environment to an Astro Deployment.

  ### New commands to configure worker queues on Astro

  You can now manage create, delete, and update worker queues on an Astro Deployment with the following new commands:

  * `astro deployment worker-queue create` creates a new worker queue in a Deployment.
  * `astro deployment worker-queue update` updates an existing worker queue.
  * `astro deployment worker-queue delete` deletes an existing worker queue.

  ### New commands to manage Organization

  If you belong to multiple Astro Organizations, you can now use the CLI to switch between your Organizations:

  * `astro organization list` lists all Organizations you belong to
  * `astro organization switch` allows you to switch between Organizations

  To use these commands, you must be authenticated to your primary Organization through the CLI.

  ### Additional improvements

  * The Astro CLI for Windows is now distributed as an `.exe` file.
  * You can now define connections in the `conn_extra` field of `airflow_settings.yaml` as YAML blocks instead of stringified JSON objects.
  * You can now use the `--settings-file` flag with `astro dev start` to load and update Airflow objects in your environment from the configuration file of your choosing.

  ### Bug fixes

  * Fixed an issue where the Astro CLI generated incorrect URLs for the Deployment dashboard
  * Improved error handling and messaging when the Astro CLI doesn't recognize the image in a project's Dockerfile
</Update>

<Update label="Astro CLI 1.5.1" description="September 23, 2022">
  ### Bug fixes

  * Fixes an issue where you could not push a deprecated version of Astro Runtime to a Deployment, even if that Deployment was already running that version. Instead of blocking deploys, the Astro CLI now shows only a warning.
</Update>

<Update label="Astro CLI 1.5.0" description="September 2, 2022">
  ### Additional improvements

  * You can now use a new `--deployment-name` flag with all `astro deployment` commands to specify a Deployment by its name instead of its Deployment ID.
  * You can now use a new `--wait` flag with `astro deployment create` to have the command wait until the new Deployment is healthy before completing.
  * You can now use a new `--no-browser` flag with `astro dev start` if you don't want the Airflow UI to automatically open in a new tab on your browser when you run the command.
  * The `astro dev restart` command no longer opens a new tab in your browser for the Airflow UI. When you use this command to apply changes to your dags, the Airflow UI should already be open.

  ### Bug fixes

  * Fixed an issue where some environment variable values could be truncated when using `astro deployment variable create --load`.
  * Fixed an issue where users with access to more than one Astro Organization could only log in to their primary Organization. Now, users can authenticate to multiple Organizations with a [token login](/docs/cli/v1.45/astro-login). Native support for organization commands is coming soon.
</Update>

<Update label="Astro CLI 1.4.0" description="August 18, 2022">
  ### New command to bash into local Airflow containers

  You can now run bash commands in any locally running Airflow container using `astro dev bash`. You can use this to:

  * Verify the packages installed in your Airflow environment.
  * Run python commands and test python functions in your Airflow environment.
  * Explore the local Airflow metadata database with a simple `postgres` command.

  For more information, see the [CLI command reference](/docs/cli/v1.45/astro-dev-bash).

  ### New command to invite a user to an Astro Organization

  You can invite new users to an Astro Organization with the new `astro user invite` command. Previously, you could only invite users to Astro with the Astro UI.

  For more information, see the [CLI command reference](/docs/cli/v1.45/astro-organization-user-invite).

  ### Additional improvements

  * Create multiple environment variables more easily by passing a list of key and value pairs to `astro deployment variable create` and `astro deployment variable update`. For example, `astro deployment variable create KEY1=VAL1 KEY2=VAL2` creates variables for `KEY1` and `KEY2`. You can still create environment variables from a file with the `--load` flag.
  * If Docker Desktop isn't already running on your machine, the CLI automatically starts it when you run `astro dev start`. Previously, the CLI showed an error and forced users to manually start Docker. Note that this feature only works on Mac OS.
  * The Airflow UI now automatically opens in your default web browser after you run `astro dev start` as soon as the Airflow webserver is ready. Previously, you had to wait for the webserver to be ready and manually open or refresh your web browswer.
</Update>

<Update label="Astro CLI 1.3.0" description="July 19, 2022">
  ### Deploy a custom Docker image with new `--image-name` flag

  You can now deploy your Astro project with a custom Docker image by running `astro deploy --image-name <custom-image>`, as long as the image is based on Astro Runtime and is available in a local Docker registry. Customizing your Runtime image lets you securely mount additional files and arguments in your project, which is required for setups such as [installing Python packages from private sources](/docs/cli/v1.45/private-python-packages).

  Using this flag, you can automate deploying custom Runtime images from a CI/CD pipeline. You can also separate your build and deploy workflows in different pipelines.

  The `--image-name` flag is also available for the following local development commands:

  * `astro dev start`
  * `astro dev restart`
  * `astro dev parse`
  * `astro dev pytest`

  For more information about this command, see the [CLI command reference](/docs/cli/v1.45/astro-deploy).

  ### New token login method for Astro

  Astro CLI users can now log into Astro on a machine that does not have access to a browser by running `astro login --token-login`. This is an alternative to `astro login`, which automatically opens the Astro UI in a browser on your machine.

  If you run the command with this flag, the CLI provides a link to the Astro UI that you can manually open in a web browser. You then copy an authentication token from the UI and enter it in the CLI. If you're using a browserless machine with the Astro CLI, this enables you to log in. For a browserless login, you can open the link and copy the token on a separate machine from the one running the Astro CLI.

  For more information about this command, see the [CLI command reference](/docs/cli/v1.45/astro-login).

  ### Skip parsing dags before deploys

  By default, `astro deploy` automatically parses the dags in your Astro project for syntax and import errors. To develop more quickly, you can now configure the Astro CLI to automatically skip parsing dags before a deploy by updating one of the following configurations:

  * Add `skip_parse: true` to your `.astro/config.yaml` file.
  * Add `ASTRONOMER_SKIP_PARSE=true` as an environment variable to your local environment or CI/CD pipeline.

  For more information on parsing dags, see [Test your Astro project locally](/docs/cli/v1.45/test-your-astro-project-locally). For more information about deploying to Astro, see [Deploy code](/docs/astro/deploy-code).

  ### Additional improvements

  * Upgraded the CLI to Go version 1.18, which includes improvements to both performance and the development experience. See the [Go Blog](https://go.dev/blog/go1.18).

  ### Bug fixes

  * Fixed an issue where parsing dags during a deploy would kill a local project
  * Fixed an issue where `astro dev parse` failed on dags using the `SnowflakeOperator`. If you use the `SnowflakeOperator`, delete `.astro/test_dag_integrity_default.py` from the `tests` directory of your Astro project and run `astro dev init` with the Astro CLI. This command will create a new file in your project that does not have this issue.
</Update>

<Update label="Astro CLI 1.2.0" description="June 28, 2022">
  ### Bug fixes

  * Fixed an issue where `astro deploy` would kill a running project
</Update>

<Update label="Astro CLI 1.1.0" description="June 13, 2022">
  ### Deployment API keys now work with Deployment commands

  You can now run the following commands with a Deployment API key:

  * `astro deploy`
  * `astro deployment list`
  * `astro deployment logs`
  * `astro deployment update`
  * `astro deployment delete`
  * `astro deployment variable list`
  * `astro deployment variable create`
  * `astro deployment variable update`

  Previously, you could run only the `astro deploy` command with a Deployment API key. For more information on API keys.

  ### Easier way to determine Deployment ID on Deployment commands

  The Astro CLI now follows a new process to determine which Deployment to run a command against. Specifically:

  * The Astro CLI first checks if a Deployment ID is specified as an argument to the command. For example, `astro deployment update <deployment-id>`.
  * If not found, it checks for a Deployment ID in the `./astro/config.yaml` file of your Astro project. In this file, you can set up to one Deployment ID as default. This is an alternative to manually specifying it or using a Deployment API key.
  * If only one Deployment exists in your Workspace, the CLI automatically runs the command for that Deployment without requiring that you specify its Deployment ID.
  * If a Deployment API key is set as an OS-level environment variable on your machine or in a CI/CD pipeline, the CLI automatically runs the command for that Deployment without requiring a Deployment ID.
  * If multiple Deployments exist in your Workspace and a Deployment API key is not found, the CLI will prompt you to select a Deployment from a list of all Deployments in that Workspace.
  * If the Astro CLI doesn't detect a Deployment across your system, it will prompt you to create one.

  These changes make it easier to run and automate Deployment-level commands with the Astro CLI. Most notably, it means that you no longer need to specify a Deployment ID in cases where it can be automatically implied by our system.

  If your CI/CD pipelines currently define one or more Deployment IDs, you may remove those IDs and their corresponding environment variables as they are no longer required. For up-to-date CI/CD templates, see [Automate code deploys with CI/CD](/docs/astro/set-up-ci-cd).

  ### Bug fixes

  * Fixed an issue where only Workspace Admins could create Deployments
</Update>

<Update label="Astro CLI 1.0.1" description="June 6, 2022">
  ### Bug fixes

  * Fixed an issue where `astro deploy`, `astro dev parse`, and `astro dev pytest` failed for some users
</Update>

<Update label="Astro CLI 1.0.0" description="June 2, 2022">
  ### A shared CLI for all Astronomer users

  The Astro CLI is now a single CLI executable built for all Astronomer products. This new generation of the CLI optimizes for a consistent local experience with Astro Runtime as well as the ability to more easily upgrade to Astro from other products hosted on Astronomer.

  To establish a shared framework between products, the Astro CLI now uses a single `astro` executable:

  ```sh wrap theme={null}
  # Before upgrade
  astrocloud dev init

  # After upgrade
  astro dev init
  ```

  Additionally, some commands have been standardized so that they can be shared between Astro and Astronomer Software users. As part of this change, `astro auth login` and `astro auth logout` have been renamed `astro login` and `astro logout`:

  ```sh wrap theme={null}
  # Before upgrade
  astrocloud auth login

  # After upgrade
  astro login
  ```

  For Astro users, these are the only changes to existing CLI functionality. All other commands will continue to work as expected. We strongly recommend that all users upgrade.

  <Danger>
    **Possible Breaking Change**

    If you currently have CI/CD pipelines that install the `astrocloud` executable of the Astro CLI, we encourage you to update them to use the latest version of `astro` to ensure reliability. All `astrocloud` commands will continue to work for some time but will be deprecated by Astronomer soon.

    For updated CI/CD examples, see [CI/CD](/docs/astro/set-up-ci-cd).
  </Danger>

  ### New Command To Set Astro Project Configurations

  You can now use `astro config get` and `astro config set` to retrieve and modify the configuration of your Astro project as defined in the `.astro/config.yaml` file. The configuration in this file contains details about how your project runs in a local Airflow environment, including your Postgres username and password, your webserver port, and your project name.

  For more information about these commands, see the [CLI command reference](/docs/cli/v1.45/astro-config-set).

  ### New Command To Switch Between Astronomer Contexts

  You can now use `astro context list` and `astro context switch` to show all the Astronomer contexts that you have access to and switch between them. An Astronomer context is defined as a base domain that you can use to access either Astro or an installation of Astronomer Software. A domain will appear as an available context if you have authenticated to it at least once.

  This command is primarily for users who need to work in both Astro and Astronomer Software installations. If you're an Astro user with no ties to Astronomer Software, ignore this command. For more information, see the [CLI command reference ](/docs/cli/v1.45/astro-context-switch).

  For more information about these commands, see the [CLI command reference ](/docs/cli/v1.45/astro-context-switch).

  ### Additional improvements

  * Astro CLI documentation has been refactored. You can now find all information about the CLI, including installation steps and the command reference, under the [Astro CLI tab](/docs/cli/v1.45/overview).
  * The nonfunctional `--update` flag has been removed from `astro deployment variable create`. To update existing environment variables for a given Deployment, use `astro deployment variable update` instead.
</Update>

<Update label="1.5.0 (`astrocloud`)" description="April 28, 2022">
  ### New command to update Deployment environment variables

  A new `astro deployment variable update` command allows you to more easily update an existing environment variable by typing a new value directly into your command line or adding the updated variable to a `.env` file.

  This command replaces the `—update` flag that was previously released with the `astro deployment variable create` command. For more information, see the [Astro CLI command reference](/docs/cli/v1.45/astro-deployment-variable-create).

  ### Additional improvements

  * When you run `astro workspace switch`, you can now specify a `<workspace-id>` as part of the command and avoid the prompt to manually select a Workspace
  * You now need to provide an email address only the first time you run `astro login`. After you run that command once successfully, the Astro CLI will cache your email address in your `config.yaml` file and not prompt you to enter it again
  * The `astro deploy` and `astro dev start` commands will now inform you if there is a new version of Astro Runtime available

  ### Bug fixes

  * Fixed an issue were the `astro deployment variable create —load` command would fail if the specified `.env` file had a comment (e.g. `#  <comment>`) in it
  * Fixed an issue were Deployment API keys would not work locally for some users
</Update>

<Update label="1.4.0 (`astrocloud`)" description="April 14, 2022">
  ### New command to create and update environment variables

  `astro deployment variable create` is a new Astro CLI command that allows you to create and update [environment variables](/docs/astro/environment-variables) for a Deployment on Astro. New environment variables can be loaded from a file (e.g. `.env`) or specified as inputs to the CLI command itself. If you already set environment variables via a `.env` file locally, this command allows you to set environment variables on Astro from that file as well. More generally, this command makes it easy to automate creating or modifying environment variables instead of setting them manually in the Astro UI.

  For more information about this command and its options, see the [Astro CLI command reference](/docs/cli/v1.45/astro-deployment-variable-create).

  ### New command to list and save Deployment environment variables

  You can now list existing environment variables for a given Deployment and save them to a local `.env` file with a new `astro deployment variable list` command. This command makes it easy to export existing environment variables for a given Deployment on Astro and test dags with them in a local Airflow environment.

  For more information about this command and its options, see the [Astro CLI command reference](/docs/cli/v1.45/astro-deployment-variable-list).

  ### Additional improvements

  * You can now specify a custom image name in your Astro project's `Dockerfile` as long as the image is based on an existing Astro Runtime image
</Update>

<Update label="1.3.4 (`astrocloud`)" description="April 11, 2022">
  ### Additional improvements

  * Improved the performance of `astro dev start`
  * When you successfully push code to a Deployment with `astro deploy`, the CLI now provides URLs for accessing the Deployment's Cloud UI and Airflow UI pages.
</Update>

<Update label="1.3.3 (`astrocloud`)" description="March 31, 2022">
  ### Additional improvements

  * The `astro dev start` command should now be \~30 seconds faster
  * When `astro dev parse` results in an error, the error messages now specify which dags they apply to
  * If your dags don't pass the basic unit test that's included in your Astro project (`test_dag_integrity.py` ), running them with `astro dev pytest` will now provide more information about which part of your code caused an error

  ### Bug fixes

  * Fixed an issue where running `astro dev parse/pytest` would occasionally result in an "orphaned containers" warning
  * Fixed an issue where `astro dev parse/pytest` would crash when parsing projects with a large number of dags
  * Fixed an issue were some `docker-compose.override.yml` files would cause `astro dev parse/pytest` to stop working
</Update>

<Update label="1.3.2 (`astrocloud`)" description="March 17, 2022">
  <Info>Astro CLI 1.3.2 is a direct patch replacement for 1.3.1, which is no longer available for download because it includes a critical bug related to `astro dev parse/pytest`. If you are currently using Astro CLI 1.3.1, then we recommend upgrading to 1.3.2+ as soon as possible to receive important bug fixes.</Info>

  ### Support for identity-based login flow

  To better integrate with Astro's identity-based login flow, the CLI now prompts you for your login email after you run `astro login`. Based on your email, the CLI assumes your Astro Organization and automatically brings you to your Organization's login flow via web browser.

  ### Additional improvements

  * `astro deploy` now builds and tests only one image per deploy. This should result in improved deployment times in CI/CD pipelines which use this command.
  * The `test` directory generated by `astro dev init` now includes more example pytests.

  ### Bug fixes

  * Partially fixed `dev parse` permission errors on WSL. To fully fix this issue for an Astro project, you must delete the project's existing `.astro` directory and rerun `astro dev init`.
  * Fixed an issue where running `astro dev parse/pytest` while a local Airflow environment was running would crash the Airflow environment. This issue was introduced in Astro CLI 1.3.1, which is no longer available for download.
</Update>

<Update label="1.3.0 (`astrocloud`)" description="March 3, 2022">
  ### New command to parse dags for errors

  `astro dev parse` is a new Astro CLI command that allows you to run a basic test against your Astro project to ensure that essential aspects of your code are properly formatted. This includes the dag integrity test that is run with `astro dev pytest`, which checks that your dags are able to to render in the Airflow UI.

  This command was built to replace the need to constantly run `astro dev restart` during troubleshooting to see if your dags render in the Airflow UI. Now, you can quickly run `astro dev parse` and see import and syntax errors directly in your terminal without having to restart all Airflow services locally. For more complex testing, we still recommend using `astro dev pytest`, which allows you to run other custom tests in your project.

  For more information about `astro dev parse`, see the [CLI command reference](/docs/cli/v1.45/astro-dev-parse). For more guidance on testing dags locally, see [Test dags locally](/docs/cli/v1.45/test-your-astro-project-locally#unit-test-dags).

  ### `astro deploy` parses dags by default

  To better protect your Deployments from unexpected errors, `astro deploy` now automatically applies tests from `astro dev parse` to your Astro project before completing the deploy process. If any of these tests fail, the CLI will not push your code to Astro.

  For more information about `astro deploy`, see [CLI command reference](/docs/cli/v1.45/astro-deploy).

  <Danger>
    **Breaking Change**

    For Deployments running Astro Runtime 4.1.0+, `astro deploy` will no longer complete the code push to your Deployment if your dags contain basic errors. If any files in your Astro project contain these errors, then certain deploys might stop working after you upgrade the Astro CLI to 1.3.0.

    To maintain the CLI's original behavior, use `astro deploy --force`. This command forces a deploy even if errors are detected in your dags.
  </Danger>

  ### New command to update Deployment configurations

  You can now use `astro deployment update` to update certain configurations for an existing Astro Deployment directly from the Astro CLI. The configurations that you can update are:

  * Deployment name
  * Deployment description
  * Scheduler resources
  * Scheduler replicas
  * Worker resources

  This is the same set of configurations that you can modify with the **Edit Configuration** view in the Astro UI. For more information on modifying a Deployment, see [Deployment settings](/docs/astro/deployment-settings). For more information about this command, see [CLI command reference](/docs/cli/v1.45/astro-deployment-update).
</Update>

<Update label="1.2.0 (`astrocloud`)" description="February 25, 2022">
  ### Deploy to Astro with Deployment API keys for simpler CI/CD

  You can now use Deployment API keys to run `astro deploy` either from the CLI directly or via a CI/CD script. This update simplifies deploying code to Astro via CI/CD.

  With an existing Deployment API key, you can set `ASTRONOMER_KEY_ID` and `ASTRONOMER_KEY_SECRET` as OS-level environment variables. From there, you can now configure a CI/CD pipeline that:

  * Installs the Astro CLI.
  * Runs `astro deploy`.

  When `astro deploy` is run, the CLI will now automatically look for and use the Deployment API key credentials that were set as environment variables to authorize and complete a code push.

  Previously, any script that automated code pushes to Astro had to include a series of `cURL` requests to the Cloud API and could not use Deployment API keys to run an Astro CLI command. If your existing CI/CD pipelines still utilize this method, we recommend replacing those commands with an Astro CLI-based workflow. For more information and guiding examples, see [CI/CD](/docs/astro/set-up-ci-cd).

  ### New command to run dag unit tests with pytest

  You can now run custom unit tests for all dags in your Astro project with `astro dev pytest`, a new Astro CLI command that uses [pytest](https://docs.pytest.org/en/7.0.x/index.html), a common testing framework for Python. As part of this change, new Astro projects created via `astro dev init` now include a `tests` directory, which includes one example pytest built by Astronomer.

  When you run this command, the Astro CLI creates a local Python environment that includes your dag code, dependencies, and Astro Runtime Docker image. The CLI then runs any pytests in the `tests` directory and shows you the results of those tests in your terminal. You can add as many custom tests to this directory as you'd like.

  For example, you can use this command to run tests that check for:

  * Python and Airflow syntax errors.
  * Import errors.
  * Dependency conflicts.
  * Unique dag IDs.

  These tests don't require a fully functional Airflow environment in order to execute, which makes this Astro CLI command the fastest and easiest way to test dags locally.

  In addition to running tests locally, you can also run pytest as part of the Astro deploy process. To do so, specify the `--pytest` flag when running `astro deploy`. This ensures that your code push to Astro automatically fails if any dags do not pass all pytests specified in the `tests` directory of your Astro project. For more information, see [Test your Astro project locally](/docs/cli/v1.45/test-your-astro-project-locally).

  ### New command to view Deployment scheduler Logs

  If you prefer to troubleshoot dags and monitor your Deployments from the command line, you can now run `astro deployment logs`, a new Astro CLI command that allows you to view the same scheduler logs that appear in the **Logs** tab of the Astro UI.

  When you run this command, all scheduler logs emitted by a Deployment over the last 24 hours appear in your terminal. Similarly to the Astro UI, you can filter logs by log level using command flags. For more information about this command, see the [CLI command reference](/docs/cli/v1.45/astro-deployment-logs).

  ### New commands to create and delete Deployments on Astro

  You can now use the Astro CLI to create and delete Deployments on Astro with two new commands:

  * `astro deployment create`
  * `astro deployment delete`

  These commands are functionally identical to the [Deployment configuration](/docs/astro/deployment-settings) and deletion process in the Astro UI. For more information, see the [CLI command reference](/docs/cli/v1.45/astro-deployment-create).
</Update>

<Update label="1.1.0 (`astrocloud`)" description="February 17, 2022">
  ### New `astro dev restart` command to test local changes

  For users making quick and continuous changes to an Astro project locally, the Astro CLI now supports a new `astro dev restart` command. This command makes local testing significantly easier and is equivalent to running `astro dev stop` followed by `astro dev start`.

  ### Support for the triggerer in local Airflow environments

  The Astro CLI now supports the Apache Airflow [triggerer component](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html?) in a local environment. This means that you can test dags that use [deferrable operators](/docs/learn/deferrable-operators) locally before pushing them to a Deployment on Astronomer. Additionally, triggerer logs appear alongside webserver and scheduler logs when you run `astro dev logs`.

  The triggerer will only be created in local environments running Astro Runtime 4.0.0+.

  ### Additional improvements

  * Postgres has been upgraded from 12.2 to [12.6](https://www.postgresql.org/docs/12/release-12-6.html) for local Airflow environments.
</Update>

<Update label="1.0.0 (`astrocloud`)" description="February 3, 2022">
  ### Introducing the Astro CLI

  The Astro CLI (`astrocloud`) is now generally available as the official command-line tool for Astro. It is a direct replacement of the previously released `./astro` executable.

  The Astro CLI sets the foundation for more robust functionality in the future and includes several significant improvements to both the local development experience as well as use cases specific to Astro. These changes are summarized in the following sections.

  The Astro CLI can be installed via Homebrew. Commands take the form of:

  ```sh wrap theme={null}
  astro <command> # E.g. `astro dev start`
  ```

  We strongly recommend that all users install the Astro CLI and delete the `./astro` executable from local directories as soon as possible. For guidelines, read [Install the Astro CLI](/docs/cli/v1.45/install-cli). As of February 2022, `./astro` will no longer be maintained by our team. With that said, the release of the Astro CLI does not have any impact on your existing Deployments or dags.

  ### New authentication flow

  The Astro CLI introduces an easy way to authenticate. Instead of requiring that users manually pass authentication tokens, the new CLI consists of a simple, browser-based login process.

  Built with refresh tokens, the Astro CLI also does not require that users re-authenticate every 24 hours, as was the case with `./astro`. As long as you remain authenticated via the Astro UI, your session via the Astro CLI will remain valid. You can expect to be asked to re-authenticate only once every few months instead of on a daily basis.

  ### Improved local development

  Astro CLI 1.0.0 includes several improvements to the local development experience:

  * You can now run `astrocloud dev start` with [Docker Buildkit](https://docs.docker.com/develop/develop-images/build_enhancements/) enabled. This resolves a [common issue](https://forum.astronomer.io/t/buildkit-not-supported-by-daemon-error-command-docker-build-t-airflow-astro-bcb837-airflow-latest-failed-failed-to-execute-cmd-exit-status-1/857) where users with Docker Buildkit enabled could not run this command.
  * After running `astrocloud dev start`, the CLI no shows you the status of the webserver container as it spins up on your local machine. This makes it easier to know whether the Airflow UI is unavailable because the Airflow webserver container is still spinning up.

  ### Additional improvements

  * `astrocloud deploy` now shows a list of your Deployments in the order by which they were created instead of at random.
</Update>

<Update label="1.0.4 (`./astro`)" description="December 9, 2021">
  ### Improved example dags

  The Astro CLI is built to enable developers to learn about, test, automate, and make the most of Apache Airflow both locally and on Astro. To that end, we've updated the CLI with two example dags that will be present for all users in the `/dags` folder that is automatically generated by `astro dev init`.

  The file names are:

  * `example-dag-basic.py`
  * `example-dag-advanced.py`

  The basic dag showcases a simple ETL data pipeline and the advanced dag showcases a series of more powerful Airflow features, including the TaskFlow API, jinja templating, branching and more. Both dags can be deleted at any time.

  ### Bug fixes

  Fixed a broken documentation link and outdated description in the `airflow_settings.yaml` file, which you can use to programmatically set Airflow connections, variables, and pools locally.
</Update>

<Update label="1.0.3 (`./astro`)" description="November 5, 2021">
  * Bug Fix: Fixed an issue where users saw errors related to S3 in webserver logs when running locally (e.g. `Failed to verify remote log exists s3:///`).
</Update>

<Update label="1.0.2 (`./astro`)" description="October 25, 2021">
  * Improved help text throughout the CLI
</Update>

<Update label="1.0.1 (`./astro`)" description="October 15, 2021">
  * This release contains changes exclusively related to the Astro CLI developer experience.
</Update>

<Update label="1.0.0 (`./astro`)" description="September 28, 2021">
  * Improvement: `./astro dev init` now always pulls the latest version of Astro Runtime for new projects. This means that you no longer have to upgrade the CLI in order to take advantage of a new Runtime release. Note that you still need to manually [upgrade Runtime](/docs/runtime/upgrade-astro-runtime) for existing projects.
  * Improvement: Updated error messages throughout the CLI to be more clear and useful
</Update>

<Update label="0.2.9-beta (`./astro`)" description="September 20, 2021">
  * Improvement: Bumped the default Astro Runtime version for new projects to [`3.0.2`](/docs/runtime/runtime-release-notes#astro-runtime-3-0-2)
  * Improvement: You can now use `./astro dev run` to run Airflow CLI commands
  * Improvement: You can now use `./astro dev logs` to show logs for the Airflow scheduler and webserver when developing locally
</Update>

<Update label="0.2.8-beta (`./astro`)" description="August 31, 2021">
  * Improvement: Bumped the default Astro Runtime version for new projects to [`3.0.0`](/docs/runtime/runtime-release-notes#astro-runtime-3-0-1)
  * Improvement: Updated help text throughout the CLI
  * Improvement: Projects created with `./astro dev init` now include a README file
</Update>

<Update label="0.2.7-beta (`./astro`)" description="July 31, 2021">
  * Bug Fix: Fixed an issue where users could not push dags to Deployments on Astro via the CLI.
</Update>

<Update label="0.2.6-beta (`./astro`)" description="July 30, 2021">
  * Improvement: You can now run `./astro login` without specifying a domain (`astronomer.io` is always assumed).
</Update>


# Run your Astro project in a local Airflow environment with the CLI
Source: https://astronomer.io/docs/cli/v1.45/run-airflow-locally

Run commands in your local Airflow environment.

Running Airflow locally with the Astro CLI can be an easy way to preview and debug dag changes quickly before deploying your code to Astro. By locally running your dags, you can fix issues with your dags without consuming infrastructure resources or waiting on code deploy processes.

This document explains how to use the Astro CLI to start a local Airflow environment on your computer and interact with your Astro project. To learn more about unit testing for your dags or testing project dependencies when changing Python or Astro Runtime versions, see [Test your project locally](/docs/cli/v1.45/test-your-astro-project-locally).

You can find common issues and resolutions in the [troubleshoot a local environment](/docs/cli/v1.45/troubleshoot-locally) section.

## Start a local Airflow environment

To begin running your project in a local Airflow environment, run:

```bash wrap theme={null}
astro dev start
```

<Tabs>
  <Tab title="Container mode (default)">
    This command builds your project and spins up 4 containers on your machine, each for a different Airflow component.
  </Tab>

  <Tab title="Standalone mode">
    To run Airflow without Docker or Podman, use standalone mode:

    ```bash wrap theme={null}
    astro dev start --standalone
    ```

    This command runs Airflow directly on your machine in a virtual environment. To make standalone mode the default for your project, run:

    ```bash wrap theme={null}
    astro config set dev.mode standalone
    ```

    All existing `astro dev` commands work in standalone mode, including `run`, `bash`, `parse`, `pytest`, `object import`, and `object export`. The `build`, `upgrade-test`, and `compose-export` commands are not available in standalone mode. See [astro dev start](/docs/cli/v1.45/astro-dev-start) for all available options.

    Standalone mode doesn't build a Docker image. It reads the `FROM` instruction in your `Dockerfile` to set the Astro Runtime and Airflow version, but ignores build instructions such as `RUN` and `COPY`. For details, see [Standalone mode](/docs/cli/v1.45/astro-dev-start#standalone-mode).

    The `--standalone` flag applies to a single command. If you haven't set `dev.mode` to `standalone`, pass `--standalone` to `astro dev stop`, `astro dev restart`, and `astro dev kill` as well.
  </Tab>
</Tabs>

After the command completes, you can access your project's Airflow UI at `https://localhost:8080/`.

## Restart a local Airflow environment

Restarting your Airflow environment rebuilds your project and restarts your local Airflow components. In container mode, this rebuilds the Docker image and restarts containers. In standalone mode, this recreates the virtual environment. Restart your environment to apply changes from specific files in your project, or to troubleshoot issues that occur when your project is running.

To restart your local Airflow environment, run:

```sh wrap theme={null}
astro dev restart
```

Alternatively, you can run `astro dev stop` to stop your environment without restarting, then run `astro dev start` when you want to restart.

## Stop a local Airflow environment

Run the following command to stop your local Airflow environment.

```sh wrap theme={null}
astro dev stop
```

Unlike [`astro dev kill`](#hard-reset-your-local-environment), this command does not prune mounted volumes and delete data associated with your local Postgres database. If you run this command, Airflow connections and task history will be preserved.

Use this command when you're finished testing Airflow and you want to stop running its components locally.

## View Airflow component logs

You can use the Astro CLI to view logs for your local Airflow environment's webserver, scheduler, and triggerer. This is useful if you want to troubleshoot a specific task instance, or if your local environment does not run properly after a code change.

To view component logs in a local Airflow environment, run:

```sh wrap theme={null}
astro dev logs
```

See the [Astro CLI reference guide](/docs/cli/v1.45/astro-dev-logs) for more details and options.

## Apply changes to a running project

If you update dag code for an Astro project that's currently running locally, the Astro CLI automatically applies your changes to your environment. However, to update other files, you must restart your environment to apply your changes.

Specifically, you must restart your environment to apply changes for any of the following files:

* `packages.txt`
* `Dockerfile`
* `requirements.txt`
* `airflow_settings.yaml`

To restart your local Airflow environment, run:

```sh wrap theme={null}
astro dev restart
```

## Run Airflow CLI commands

To run [Apache Airflow CLI](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html) commands locally, run the following:

```sh wrap theme={null}
astro dev run <airflow-cli-command>
```

For example, the Airflow CLI command for listing connections is `airflow connections list`. To run this command with the Astro CLI, you would run `astro dev run connections list` instead.

`astro dev run` executes Airflow CLI commands in your local Airflow environment. In container mode, this is the equivalent of running `docker exec` in local containers.

<Info>You can only use `astro dev run` in a local Airflow environment. To automate Airflow actions on Astro, you can use the [Airflow REST API](/docs/astro/airflow-api). For example, you can make a request to the [`dagRuns` endpoint](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/post_dag_run) to trigger a dag run programmatically, which is equivalent to running `astro dev run dags trigger` in the Astro CLI.</Info>

## Make requests to the Airflow REST API locally

Make requests to the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) in a local Airflow environment with HTTP basic access authentication. This can be useful for testing and troubleshooting API calls before executing them in a Deployment on Astro.

To make local requests with cURL or Python, you only need the username and password for your local user. Both of these values are `admin` by default. They are the same credentials for logging into the Airflow UI, and they're listed when you run `astro dev start`.

To make requests to the Airflow REST API in a Deployment on Astro, see [Airflow API](/docs/astro/airflow-api).

### Airflow 3

Example GET Dags request:

#### cURL

```bash wrap theme={null}

curl -X POST "http://localhost:8080/auth/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \               
  -d "username=admin&password=admin
{"access_token":"eyJhbGciOiJIUzUx..."}

curl -X GET "http://localhost:8080/api/v2/dags" \
  -H "Accept: application/json" \                                      
  -H "Authorization: Bearer eyJhbGciOiJIUzUx..."

```

#### Python

```python wrap theme={null}
import requests

# First request: get auth token
auth_response = requests.post(
    "http://localhost:8080/auth/token",
    headers={"Content-Type": "application/x-www-form-urlencoded"},
    data={"username": "admin", "password": "admin"}
)
token = auth_response.json().get("access_token")

# Second request: use the token to list DAGs
dags_response = requests.get(
    "http://localhost:8080/api/v2/dags",
    headers={
        "Accept": "application/json",
        "Authorization": f"Bearer {token}"
    }
)
print(dags_response.status_code, dags_response.text)

```

### Airflow 2

Example GET Dags request:

#### cURL

```bash wrap theme={null}
curl -X GET localhost:8080/api/v1/<endpoint> --user "admin:admin"
```

#### Python

```python wrap theme={null}
import requests

response = requests.get(
   url="http://localhost:8080/api/v1/dags",
   auth=("admin", "admin")
)
```

## Hard reset your local environment

In most cases, [restarting your local project](/docs/cli/v1.45/run-airflow-locally#restart-a-local-airflow-environment) is sufficient for testing and making changes to your project. However, it is sometimes necessary to reset your environment and metadata database for testing purposes. To do so, run the following command:

```sh wrap theme={null}
astro dev kill
```

In container mode, this command forces your running containers to stop and deletes all data associated with your local Postgres metadata database, including Airflow connections, logs, and task history. In standalone mode, this command stops Airflow processes and removes the virtual environment and local database.

## Override the Astro CLI Docker Compose file

<Note>Docker Compose overrides are only available in container mode. They are not supported in standalone mode.</Note>

The Astro CLI uses a default set of [Docker Compose](https://docs.docker.com/compose/) configurations to define and run local Airflow components. For advanced testing cases, you might need to override these default configurations. For example, you might need to:

* Add extra containers to mimic services that your Airflow environment needs to interact with locally, such as an SFTP server.
* Change the volumes mounted to any of your local containers.

<Info>The Astro CLI does not support overrides to environment variables that are required globally. For the list of environment variables that Astro enforces, see [Global environment variables](/docs/astro/platform-variables). To learn more about environment variables, read [Environment variables](/docs/astro/environment-variables).</Info>

1. Reference the Astro CLI's default [Docker Compose file](https://github.com/astronomer/astro-cli/blob/main/airflow/include/airflow2/composeyml.go.tmpl) (`composeyml.go.tmpl`) and determine one or more configurations to override.
   <Info>For Airflow 3 Deployments, reference the Astro CLI's default [Airflow 3 Docker Compose file](https://github.com/astronomer/astro-cli/blob/main/airflow/include/airflow3/composeyml.go.tmpl).</Info>
2. Add a `docker-compose.override.yml` file at the top level of your Astro project.
3. Specify your new configuration values in `docker-compose.override.yml` file using the same format as in `composeyml.go.tmpl`.

Common use cases are:

* Mounting a volume with additional files
* Running an additional service to simulate your production environment

### Example: Mounting a volume with additional files

To add another volume mount for a directory named `custom_dependencies`, add the following to your `docker-compose.override.yml` file:

```yaml wrap theme={null}
services:
  scheduler:
    volumes:
      - /home/astronomer_project/custom_dependencies:/usr/local/airflow/custom_dependencies:ro
```

Run the following command to see the directory in your scheduler container:

```sh wrap theme={null}
astro dev bash --scheduler "ls -al"
```

### Example: Running an additional service to simulate your production environment

To run HashiCorp Vault to simulate a production environment that uses a HashiCorp Vault secrets backend:

```yaml expandable wrap theme={null}
services:
  vault:
    image: hashicorp/vault:1.21
    networks:
      - airflow
    ports:
      - "8200:8200"
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: "root"
      VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200"
    cap_add:
      - IPC_LOCK
    command: server -dev
    volumes:
      - vault-data:/vault/file
    healthcheck:
      test: [ "CMD-SHELL", "VAULT_ADDR=http://127.0.0.1:8200 vault status >/dev/null 2>&1" ]
      interval: 1s
      timeout: 2s
      retries: 60

  # Optional: pre-fill Vault with secrets
  vault-load-data:
    image: hashicorp/vault:1.21
    networks:
      - airflow
    depends_on:
      vault:
        condition: service_healthy
    environment:
      VAULT_ADDR: http://vault:8200
      VAULT_TOKEN: root
    restart: "no"
    entrypoint: ["/bin/sh", "-lc"]
    command: |
      '
      set -e
      # Only write if secret is not present yet
      vault kv get -format=json secret/variables/my_api_key >/dev/null 2>&1 || vault kv put secret/variables/my_api_key value="super-secret-api-key"
      echo "Data loading done."
      '

volumes:
  vault-data:
```

Configure Airflow to use the additional Vault service as a secrets backend:

1. In `.env`, configure:

```text wrap theme={null}
AIRFLOW__SECRETS__BACKEND='airflow.providers.hashicorp.secrets.vault.VaultBackend'
AIRFLOW__SECRETS__BACKEND_KWARGS='{"url": "http://vault:8200", "token": "root", "mount_point": "secret", "kv_engine_version": 2, "connections_path": "connections", "variables_path": "variables", "config_path": "config", "verify": false}'
```

2. Install `apache-airflow-providers-hashicorp` in your `requirements.txt`.
3. Manually add secrets via [http://localhost:8200](http://localhost:8200) (token `root`), or test the provided variable `my_api_key`.


# Switch between Podman and Docker
Source: https://astronomer.io/docs/cli/v1.45/switch-container-management

Instructions for switching between Podman and Docker as the container management engine for the Astro CLI.

The Astro CLI automatically runs Podman containers whenever you run a command that requires them. You can switch between Podman and Docker as your container management engine to run Astro CLI commands.

## Prerequisites

<Tabs>
  <Tab title="Docker">
    * Microsoft Hyper-V enabled. See [Install Hyper-V On Windows](https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/quick-start/enable-hyper-v) or [Step-By-Step: Enabling Hyper-V on Windows 11](https://techcommunity.microsoft.com/t5/educator-developer-blog/step-by-step-enabling-hyper-v-for-use-on-windows-11/ba-p/3745905).
    * The latest version of the Windows [App Installer](https://apps.microsoft.com/store/detail/app-installer/9NBLGGH4NNS1?hl=en-ca\&gl=ca).
    * Windows 10 1709 (build 16299) or later or Windows 11.
  </Tab>

  <Tab title="Podman">
    * Windows Subsystem for Linux (WSL) 2 enabled: See [Enable the Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install), [WSL basic commands](https://learn.microsoft.com/en-us/windows/wsl/basic-commands), and [Troubleshooting WSL 2](https://learn.microsoft.com/en-us/windows/wsl/troubleshooting#error-0x80370102-the-virtual-machine-could-not-be-started-because-a-required-feature-is-not-installed).
      * After you enable WSL, run:
      ```text wrap theme={null}
       wsl --update
       wsl --install --no-distribution
      ```
    * Windows 10 or Windows 11.
    * A container service like [Podman](https://podman.io/docs/installation) or [Docker Engine](https://docs.docker.com/engine/install/).
  </Tab>
</Tabs>

## Switch between Podman and Docker

To run CLI commands in Docker containers, run the following command:

```sh wrap theme={null}
astro config set container.binary docker -g
```

If you need to switch back to using Podman again, run the following command:

```sh wrap theme={null}
astro config set container.binary podman -g
```

<Tip>If you want to switch between Docker and Podman on a specific project without changing the global configuration, run these commands without the `-g`.</Tip>


# Test your Astro project
Source: https://astronomer.io/docs/cli/v1.45/test-your-astro-project-locally

Check your Airflow dags for errors before you run them locally or deploy to Astro.

One of the Astro CLI's main features is the ability to run Astro projects in a local Airflow environment. It additionally includes commands that you can use to test and debug dags both inside and outside of a locally running Airflow environment.

Use the following document to learn more about how you can test locally with the Astro CLI before deploying your code changes to a production environment.

## Run a dag with `astro run`

Use the `astro run` command to run a dag from the command line. When you run the command, the CLI compiles your dag and runs it in a single Airflow worker container based on your Astro project configurations, including your `Dockerfile`, dag utility files, Python requirements, and environment variables. You can review task logs and task status in your terminal without opening the Airflow UI. Running dags without a scheduler or webserver can help reduce the time required to develop and test data pipelines.

To run a dag located within your local `/dags` directory, run:

```bash wrap theme={null}
astro run <dag-id>
```

You can only run one dag at a time. All the tasks in your dag run sequentially. Any errors produced by your code while parsing or running your dag appear in the command line. For more information about this command, see the [CLI command reference](/docs/cli/v1.45/astro-run).

## Unit test dags

You can run dag unit tests with the following Astro CLI commands to quickly test code:

* `astro dev parse`
* `astro dev pytest`

These commands don't require a running Airflow environment, meaning you can test dags without deploying to Astro or running Airflow locally. However, these commands do require Docker.

### Parse dags

To quickly parse your dags, run:

```sh wrap theme={null}
astro dev parse
```

This command parses your dags to ensure that they don't contain any basic syntax or import errors and that they can successfully render in the Airflow UI.

`astro dev parse` is a more convenient but less customizable version of `astro dev pytest`. If you don't have any specific test files that you want to run on your dags, Astronomer recommends using `astro dev parse` as your primary testing tool. For more information about this command, see the [CLI command reference](/docs/cli/v1.45/astro-dev-parse).

### Run tests with pytest

To run unit tests on your Astro project, run:

```sh wrap theme={null}
astro dev pytest
```

This command runs all tests in your project's `tests` directory with [pytest](https://docs.pytest.org/en/7.0.x/index.html#), a testing framework for Python. With pytest, you can test custom Python code and operators locally without having to start a local Airflow environment.

The `tests` directory in your Astro project includes an example dag test called `test_dag_example.py`. This test checks that:

* All Airflow tasks have required arguments.
* Dag IDs are unique across the Astro project.
* Dags have no cycles.
* There are no general import or syntax errors.

This test is just an example of the kinds of pytests one could run to test their dags. You may want to alter this test or create new ones that better fit the context of your dags. `astro dev pytest` will run any pytest file that you add to the `tests` directory. For more information about this command, see the [CLI command reference](/docs/cli/v1.45/astro-dev-pytest).

## Test before an Astro Runtime upgrade

You can use [`astro dev upgrade-test`](/docs/cli/v1.45/astro-dev-upgrade-test) to test your local Astro project against a new version of Astro Runtime to prepare for an upgrade. By default, the command runs the following tests in order to create reports that can help you determine whether your upgrade will be successful:

* **Dependency test**: Identify the packages that have been added, removed, or changed in the upgrade version.
* **Dag test**: Identify Python dag `import` errors in the upgrade version.

To run these tests, open your Astro project and run:

```sh wrap theme={null}
astro dev upgrade-test
```

If the tests are successful, the Astro CLI creates a folder in your Astro project called `upgrade-test-<your-current-version>--<your-upgrade-version>`. The folder will contain the following reports:

* `pip_freeze_<current-version>`: The output of the `pip freeze` with your current version.
* `pip_freeze_<upgrade-version>`: The output of the `pip freeze` with your upgrade version.
* `dependency_compare.txt`: The result of the dependency test.
* `Dockerfile`: The updated file used in the upgrade test.
* `dag-test-results.html`: The results of the dag test.

Use the test results to fix any major package changes or broken dags before you upgrade. Refer to the Airflow and Provider package release notes to assist in upgrading your dags. After you resolve all conflicts and dag import errors, you can [upgrade Astro Runtime](/docs/runtime/upgrade-astro-runtime) and [deploy your project](/docs/astro/deploy-dags) to an Astro Deployment.

<Info>When you rerun the test for the same project and upgrade version, all the files in the test results folder will be updated. To keep results for a particular test, change the folder name before rerunning the command.</Info>

<Tip>If you're testing a local project before deploying to Astro, you can test more accurately by adding  `--deployment-id` flag and specifying your Deployment ID. The Astro CLI uses the image currently running in your Deployment to test against the upgrade version. Note that this flag will use your local dags and dependencies against your Astro Deployment's image with the upgrade version of runtime specified.</Tip>

Read the following sections to learn more about the contents of each test report. For more information about the command's settings, see the [CLI reference guide](/docs/cli/v1.45/astro-dev-upgrade-test).

### Dependency test

To prepare for an upgrade, it's helpful to identify all Python packages which will modified as a result of the upgrade. You can do this using the dependency test. When you run the test, the Astro CLI generates a report called `dependency_compare.txt` in `upgrade-test-<current-version>--<upgrade-version>`. The report shows all Airflow providers and packages that have been removed, added, or updated.

When you read the results of this test, pay close attention to the `Major Updates` section. Major updates to Python packages are more likely to cause your dags to fail. Visit the changelog for any providers listed in this section (for example, the [HTTP provider changelog](https://airflow.apache.org/docs/apache-airflow-providers-http/stable/changelog.html)) to see if the major upgrade will affect your environment. You should also pay attention to anything listed under `Unknown Updates`. These are updates that Astro CLI could not categorize, which can include major upgrades that might cause dags to break.

To run only the dag test against the latest version of Astro Runtime, run the following command in your Astro Project:

```bash wrap theme={null}
astro dev upgrade-test --version-test
```

### Dag test

When you upgrade, any Python packages that changed can generate import errors and cause your dags to break. These import errors are visible in the UI after you upgrade, but you can address them before upgrading by running the dag test.

This test uses the [`astro dev parse`](/docs/cli/v1.45/astro-dev-parse) command against the upgrade version and produces a report called `dag-test-report.html` in `upgrade-test-<current-version>--<upgrade-version>`. This HTML report lists the dags that will have import errors, along with the first error encountered if you complete an upgrade. You can use this report along with the dependency test report to fix errors in your dags before your upgrade.

To run only the dag test against the latest version of Astro Runtime, run the following command in your Astro Project:

```bash wrap theme={null}
astro dev upgrade-test --dag-test
```

## See also

* [Debug dags](/docs/learn/debugging-dags)
* [`astro dev pytest`](/docs/cli/v1.45/astro-dev-pytest)


# Troubleshoot a local Airflow environment
Source: https://astronomer.io/docs/cli/v1.45/troubleshoot-locally

Address and resolve common issues with local development.

Use the following topics to resolve common issues with running an Astro project in a local Airflow environment.

## Troubleshoot `KubernetesPodOperator` issues

View local Kubernetes logs to troubleshoot issues with Pods that are created by the `KubernetesPodOperator`. See [Test and Troubleshoot the `KubernetesPodOperator` Locally](/docs/learn/kubepod-operator#run-the-kubernetespodoperator-locally).

## Troubleshoot dependency errors

When dependency errors occur, the error message that is returned often doesn't contain enough information to help you resolve the error. To retrieve additional error information, you can review individual operating system or python package dependencies inside your local Docker containers.

For example, if your `packages.txt` file contains several packages and you receive build errors after running `astro dev start`, you can enter the container and install the packages manually to review additional information about the errors.

1. Open your Astro project `packages.txt` file and remove the references to the packages that are returning error messages.

2. Run the following command to build your Astro project into a Docker image and start a local Docker container for each Airflow component:

   ```sh wrap theme={null}
   astro dev start
   ```

3. Run the following command to open a bash terminal in your scheduler container:

   ```sh wrap theme={null}
   astro dev bash --scheduler
   ```

4. In the bash terminal for your container, run the following command to install a package and review any error messages that are returned:

   ```bash wrap theme={null}
   apt-get install <package-name>
   ```

   For example, to install the GNU Compiler Collection (GCC) compiler, you would run:

   ```bash wrap theme={null}
   apt-get install gcc
   ```

5. Open your Astro project `packages.txt` file and add the package references you removed in Step 1 individually until you find the package that is the source of the error.

## New dags aren't visible in the Airflow UI

Make sure that no dags have duplicate `dag_ids`. When two dags use the same `dag_id`, the newest dag won't appear in the Airflow UI and you won't receive an error message.

By default, the Airflow scheduler scans the `dags` directory of your Astro project for new files every 300 seconds (5 minutes). For this reason, it might take a few minutes for new dags to appear in the Airflow UI. Changes to existing dags appear immediately.

To have the scheduler check for new dags more frequently, you can set the [`AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-dir-list-interval) environment variable to less than 300 seconds. If you have less than 200 dags in a Deployment, it's safe to set `AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL` to `30` (30 seconds). See [Set environment variables](/docs/astro/environment-variables) for how to set this on Astro.

In Astro Runtime 7.0 and later, the Airflow UI **Code** page includes a **Parsed at** value which shows when a dag was last parsed. This value can help you determine when a dag was last rendered in the Airflow UI. To view the **Parsed at** value in the Airflow UI, click **Dags**, select a dag, and then click **Code**. The **Parsed at** value appears at the top of the dag code pane.

## Dags running slowly

If your Astro project contains many dags or tasks, then you might experience performance issues in your local Airflow environment.

To improve the performance of your environment, you can:

* Adjust CPU and memory resource allocation in your Docker configuration. Be aware that increasing Docker resource allocation might decrease the performance of your computer.
* Modify Airflow-level environment variables, including concurrency and parallelism. See [Scaling out Airflow](/docs/learn/airflow-scaling-workers).

Generating dags dynamically can also decrease the performance of your local Airflow environment, though it's a common authoring pattern for advanced use cases. For more information, see [Dynamically Generating dags in Airflow](/docs/learn/dynamically-generating-dags). If your dags continue to run slowly and you can't scale Docker or Airflow any further, Astronomer recommends pushing your project to a Deployment on Astro that's dedicated to testing.

<Tip>
  If you don't have enough Docker resources allocated to your local Airflow environment, you might see tasks fail and exit with this error:

  ```text wrap theme={null}
  Task exited with return code Negsignal.SIGKILL
  ```

  If you see this error, increase the CPU and memory allocated to Docker. If you're using Docker Desktop, you can do this by opening Docker Desktop and going to **Preferences** > **Resources** > **Advanced**. See [Change Docker Desktop preferences on Mac](https://docs.docker.com/desktop/settings/mac/).

  If you are using Podman, you can run `podman machine set --cpus 4 --memory 4096`. See [Podman commands reference](https://docs.podman.io/en/latest/markdown/podman-machine-set.1.html) for more details.
</Tip>

## Astro project won't load after running `astro dev start`

If you're running the Astro CLI on a Mac computer that's built with the Apple M1 chip, your Astro project might take more than 5 mins to start after running `astro dev start`. This is a current limitation of Astro Runtime and the Astro CLI.

If your project won't load, it might also be because your webserver or scheduler is unhealthy. In this case, you might need to debug your containers.

1. After running `astro dev start`, retrieve a list of running containers by running `astro dev ps`.
2. If the webserver and scheduler containers exist but are unhealthy, check their logs by running:

   ```sh wrap theme={null}
   $ astro dev logs --webserver
   $ astro dev logs --scheduler
   ```
3. (Optional) Run the following command to prune all unused Docker objects including volumes and free disk space:

   ```bash wrap theme={null}
   docker system prune --volumes
   ```

   See [`docker system prune`](https://docs.docker.com/config/pruning/#prune-everything) for more information about pruning.
   These logs should help you understand why your webserver or scheduler is unhealthy. Possible reasons why these containers might be unhealthy include:

* Not enough Docker resources.
* A failed Airflow or Astro Runtime version upgrade.
* Misconfigured Dockerfile or Docker override file.
* Misconfigured Airflow settings including `packages.txt` or `requirements.txt`.

## Ports are not available for my local Airflow webserver

By default, the Astro CLI uses port `8080` for the Airflow webserver and port `5432` for the Airflow metadata database in a local Airflow environment. If these ports are already in use on your local computer, an error message similar to the following appears:

```text wrap theme={null}
Error: error building, (re)creating or starting project containers: Error response from daemon: Ports are not available: exposing port TCP 0.0.0.0:5432 → 0.0.0.0:0: listen tcp 0.0.0.0:5432: bind: address already in use
```

To resolve a port availability error, you have the following options:

* Stop all running Docker containers and restart your local environment using `astro dev restart`.
* Change the default ports for these components. For example, you can use `astro config set webserver.port 8081` for the webserver and `astro config set postgres.port 5433` for Postgres. See [Configure CLI](/docs/cli/v1.45/configure-cli) for all available configurations.

### Stop all running Docker containers

1. Run `docker ps` to identify the Docker containers running on your computer.
2. Copy the values in the `CONTAINER ID` column.
3. Select one of the following options:

   * Run `docker stop <container_id>` to stop a specific Docker container. Replace `<container_id>` with one of the values you copied in step 2.
   * Run `docker stop $(docker ps -q)` to stop all running Docker containers.

### Change the default port assignment

If port 8080 or 5432 are in use on your machine by other services, the Airflow webserver and metadata database won't be able to start. To run these components on different ports, run the following commands in your Astro project:

```bash wrap theme={null}
astro config set webserver.port <available-port>
astro config set postgres.port <available-port>
```

For example, to use 8081 for your webserver port and 5435 for your database port, you would run the following commands:

```bash wrap theme={null}
astro config set webserver.port 8081
astro config set postgres.port 5435
```


# Uninstall the Astro CLI
Source: https://astronomer.io/docs/cli/v1.45/uninstall-cli

Instructions for uninstalling the Astro command-line interface (CLI).

You can uninstall the Astro CLI from your machine if you no longer need it or if you want to perform a clean reinstallation of the CLI.

## Uninstall CLI

<Tabs>
  <Tab title="Mac">
    To uninstall the Astro CLI on Mac, run:

    ```sh wrap theme={null}
    brew uninstall astro
    ```
  </Tab>

  <Tab title="Windows with winget">
    To uninstall an older version of the Astro CLI, you'll need to follow the [alternate Windows uninstall process](/docs/cli/v1.45/uninstall-cli?tab=windows-manual).

    To uninstall the Astro CLI, open Windows PowerShell as an administrator and run the following command:

    ```sh wrap theme={null}
    winget uninstall -e --id Astronomer.Astro
    ```
  </Tab>

  <Tab title="Windows (Manual)">
    To uninstall the Astro CLI on Windows:

    1. Delete the filepath for `astro.exe` from your Windows PATH environment variable.
    2. Delete `astro.exe`.
  </Tab>

  <Tab title="Linux">
    Run the following command to uninstall the Astro CLI on Linux:

    ```sh wrap theme={null}
    sudo rm /usr/local/bin/astro
    ```
  </Tab>
</Tabs>


# Upgrade the Astro CLI
Source: https://astronomer.io/docs/cli/v1.45/upgrade-cli

Instructions for upgrading the Astro command-line interface (CLI).

The Astro CLI contains no breaking changes between minor versions within the same major version. Therefore, Astronomer recommends always using the latest minor version of the Astro CLI within your major version.

## Upgrade the existing CLI

<Tabs>
  <Tab title="Mac">
    Run the following command to upgrade the Astro CLI to the latest version:

    ```sh wrap theme={null}
    brew upgrade astro
    ```
  </Tab>

  <Tab title="Windows with winget">
    To upgrade the Astro CLI to the latest version, open Windows PowerShell as an administrator and run the following command:

    ```sh wrap theme={null}
    winget install -e --id Astronomer.Astro
    ```

    Note that if you're upgrading from Astro CLI version 1.5.1 or earlier to a later Astro CLI version, you still need to install the upgrade version manually.
  </Tab>

  <Tab title="Windows (Manual)">
    1. Delete the existing `astro.exe` file on your machine.

    2. Go to the [Releases page](https://github.com/astronomer/astro-cli/releases) of the Astro CLI GitHub repository, scroll to a CLI version, and then download the `.exe` file that matches the CPU architecture of your machine.

    For example, to upgrade to v1.0.0 of the Astro CLI on a Windows machine with an AMD 64 architecture, you download `astro_1.0.0-converged_windows_amd64.exe`.

    3. Rename the file to `astro.exe`.

    4. Add the filepath for the directory containing the new `astro.exe` as a PATH environment variable. For example, if `astro.exe` was stored in `C:\Users\username\astro.exe`, you would add `C:\Users\username` as your PATH environment variable. To learn more about configuring the PATH environment variable, see [Java documentation](https://www.java.com/en/download/help/path.html).

    5. Restart your machine.
  </Tab>

  <Tab title="Linux">
    To upgrade the Astro CLI to the latest version, run the following command:

    ```sh wrap theme={null}
    curl -sSL install.astronomer.io | sudo bash -s
    ```
  </Tab>
</Tabs>


# Podman for the Astro CLI
Source: https://astronomer.io/docs/cli/v1.45/use-podman



The Astro CLI requires a container management engine to run Apache Airflow components on your local machine and deploy to Astro. For example, the `astro dev start` and `astro deploy` commands both require containers.

Starting with version 1.32.0, the Astro CLI is packaged with Podman as its container management engine for running Airflow locally, when installed by Homebrew or Winget.

## Configure the Astro CLI to use Podman

<Tabs>
  <Tab title="Mac">
    Set up Podman on a Mac operating system so you can run Apache Airflow locally and deploy to Astro with Podman containers.

    ### Prerequisites

    * Podman 3 or later. See [Getting started with Podman](https://podman.io/get-started/).
    * A running Podman machine with at least 4 GiB of RAM. To confirm that Podman is running, run `podman ps`.

    <Tip>
      If you receive an error after running `podman ps`, there is likely a problem with your Podman connection. You might need to set the system-level `DOCKER_HOST` environment variable to be the location of your Podman service socket:

      1. Run the following command to identify the connection URI for `podman-machine-default`:

         ```sh wrap theme={null}
         podman system connection ls
         ```

         The output should look like the following:

         ```text wrap theme={null}
         podman-machine-default*      /Users/user/.ssh/podman-machine-default  ssh://core@localhost:54523/run/user/1000/podman/podman.sock
         podman-machine-default-root  /Users/user/.ssh/podman-machine-default  ssh://root@localhost:54523/run/podman/podman.sock
         ```

      2. Copy the value in the `URI` column from `podman-machine-default*`. This is typically `unix:///run/podman/podman.sock`, but it can vary based on your installation.

      3. Set your `DOCKER_HOST` environment variable to the value of the URI.
    </Tip>

    ### Setup

    1. Run the following command to confirm that Podman has access to Astro images at `docker.io`:

       ```sh wrap theme={null}
       podman run --rm -it postgres:12.6 whoami
       ```

       If this command fails, use [Podman Desktop](https://podman-desktop.io/) to change Podman's default image registry location to `docker.io`. See [Provide pre-defined registries](https://podman-desktop.io/blog/podman-desktop-release-0.11#provide-pre-defined-registries-1201).
  </Tab>

  <Tab title="WSL2 on Windows">
    Set up Podman on Windows so you can run Apache Airflow locally and deploy to Astro with Podman containers.

    ### Prerequisites

    * Podman 3 or later installed on Windows Subsystem for Linux version 2 (WSL 2) using Ubuntu 22.04 or later. See [Install Linux on Windows with WSL](https://learn.microsoft.com/en-us/windows/wsl/install) and [Getting started with Podman](https://podman.io/get-started/).
    * A running Podman machine with at least 4 GiB of RAM. To confirm that Podman is running, run `podman ps` in your Linux terminal.
    * The Astro CLI Linux distribution installed on WSL 2. See [Install the Astro CLI on Linux](/docs/cli/v1.45/install-cli#linux).

    <Tip>
      If you receive an error after running `podman ps`, there is likely a problem with your Podman connection. You might need to set the system-level `DOCKER_HOST` environment variable to be the location of your Podman service socket:

      1. In a WSL 2 terminal, run the following command to identify the connection URI for `podman-machine-default`:

         ```sh wrap theme={null}
         podman system connection ls
         ```

         The output should look like the following:

         ```text wrap theme={null}
         podman-machine-default*      /Users/user/.ssh/podman-machine-default  ssh://core@localhost:54523/run/user/1000/podman/podman.sock
         podman-machine-default-root  /Users/user/.ssh/podman-machine-default  ssh://root@localhost:54523/run/podman/podman.sock
         ```

      2. Copy the value in the `URI` column from `podman-machine-default*`. This is typically `unix:///run/podman/podman.sock`, but it can vary based on your installation.

      3. Set your `DOCKER_HOST` environment variable to the value of the URI.
    </Tip>

    ### Setup

    1. In a WSL 2 terminal, run the following command to confirm that Podman has access to Astro images at `docker.io`:

       ```sh wrap theme={null}
       podman run --rm -it postgres:12.6 whoami
       ```

       If this command fails, run the following command to change Podman's default image registry location to `docker.io`:

       ```sh wrap theme={null}
       cat << EOF | sudo tee -a /etc/containers/registries.conf.d/shortnames.conf
       "postgres" = "docker.io/postgres"
       EOF
       ```
  </Tab>

  <Tab title="Linux">
    Set up Podman on Linux so you can run Apache Airflow locally and deploy to Astro with Podman containers.

    ### Prerequisites

    * Podman 3 or later. See [Getting started with Podman](https://podman.io/get-started/).
    * A running Podman machine with at least 4 GiB of RAM. To confirm that Podman is running, run `podman ps`.

    <Tip>
      If you receive an error after running `podman ps`, there is likely a problem with your Podman connection. You might need to set the system-level `DOCKER_HOST` environment variable to be the location of your Podman service socket:

      1. Run the following command to identify the connection URI for `podman-machine-default`:

         ```sh wrap theme={null}
         podman system connection ls
         ```

         The output should look like the following:

         ```text wrap theme={null}
         podman-machine-default*      /Users/user/.ssh/podman-machine-default  ssh://core@localhost:54523/run/user/1000/podman/podman.sock
         podman-machine-default-root  /Users/user/.ssh/podman-machine-default  ssh://root@localhost:54523/run/podman/podman.sock
         ```

      2. Copy the value in the `URI` column from `podman-machine-default*`. This is typically `unix:///run/podman/podman.sock`, but it can vary based on your installation.

      3. Set your `DOCKER_HOST` environment variable to the value of the URI.
    </Tip>

    ### Setup

    1. Run the following command to confirm that Podman has access to Astro images at `docker.io`:

       ```sh wrap theme={null}
       podman run --rm -it postgres:12.6 whoami
       ```

       If this command fails, run the following command to change Podman's default image registry location to `docker.io`:

       ```sh wrap theme={null}
       cat << EOF | sudo tee -a /etc/containers/registries.conf.d/shortnames.conf
       "postgres" = "docker.io/postgres"
       EOF
       ```
  </Tab>
</Tabs>

<Accordion title="For CLI versions 1.30.0 and earlier">
  Run the following command to set Podman as your container management engine for the Astro CLI:

  ```sh wrap theme={null}
  astro config set -g container.binary podman
  ```

  If you're using Podman 3, additionally run the following command:

  ```sh wrap theme={null}
  astro config set -g duplicate_volumes false
  ```
</Accordion>

## Troubleshooting

#### SHELL is not supported for OCI image format

```bash wrap theme={null}
WARN[0010] SHELL is not supported for OCI image format, [/bin/bash -o pipefail -e -u -x -c] will be ignored. Must use `docker` format
```

This error can occur when the CLI tries to build your Astro Runtime image using Podman.

To resolve this issue, run the following command to set the `BUILDAH_FORMAT` environment variable on your machine:

```bash wrap theme={null}
export BUILDAH_FORMAT=docker
```

#### Cannot connect to the Docker daemon

```bash wrap theme={null}
Error: error creating docker-compose project: Cannot connect to the Docker daemon at unix:///Users/[YOUR.USER]/.docker/run/docker.sock. Is the docker daemon running?
```

Ensure **Docker Compatibility** is enabled in **Settings > Preferences > Experimental (Docker Compatibility)** and **Docker CLI Context** is set to `unix://var/run/docker.sock`.

#### "docker-credential-desktop": executable file not found

```bash wrap theme={null}
Error: error reading credentials: error getting credentials - err: exec: "docker-credential-desktop": executable file not found in $PATH, out: ``
```

This error can occur when Docker used to be installed, but no longer exists on the system. When podman runs, it is checking `$HOME/.docker/config.json` for registered credentials stores.

To resolve the issue, delete the corresponding Docker configuration at `$HOME/.docker` from the system or install Docker again.

Further info can be found [here](https://forums.docker.com/t/docker-credential-desktop-exe-executable-file-not-found-in-path-using-wsl2/100225).


# Astro CLI documentation archive
Source: https://astronomer.io/docs/cli/archive

Links to documentation for retired versions of the Astro CLI.

This document lists the Astro CLI documentation sets that Astronomer has retired from the documentation site.

While only maintained documentation sets appear under the Astro CLI documentation menu, Astronomer preserves all versions for historical reference. Documentation for Astro CLI versions v1.37 and earlier no longer receives updates and is not rendered on the Astronomer documentation site. You can still read these docs in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources).

To use the latest Astro CLI features and documentation, Astronomer recommends [upgrading to the latest version of the Astro CLI](/docs/cli/v1.45/install-cli).

## Retired versions

Documentation for the following Astro CLI versions is available in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources):

* v1.37
* v1.36
* v1.35
* v1.34

If you notice an error or misleading information in any part of Astronomer's documentation, [create a GitHub issue](https://github.com/astronomer/docs/issues) or contact [Astronomer support](https://support.astronomer.io).


# Astro CLI documentation archive
Source: https://astronomer.io/docs/cli/archive

Links to documentation for retired versions of the Astro CLI.

This document lists the Astro CLI documentation sets that Astronomer has retired from the documentation site.

While only maintained documentation sets appear under the Astro CLI documentation menu, Astronomer preserves all versions for historical reference. Documentation for Astro CLI versions v1.37 and earlier no longer receives updates and is not rendered on the Astronomer documentation site. You can still read these docs in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources).

To use the latest Astro CLI features and documentation, Astronomer recommends [upgrading to the latest version of the Astro CLI](/docs/cli/v1.45/install-cli).

## Retired versions

Documentation for the following Astro CLI versions is available in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources):

* v1.37
* v1.36
* v1.35
* v1.34

If you notice an error or misleading information in any part of Astronomer's documentation, [create a GitHub issue](https://github.com/astronomer/docs/issues) or contact [Astronomer support](https://support.astronomer.io).


# Astro CLI documentation archive
Source: https://astronomer.io/docs/cli/archive

Links to documentation for retired versions of the Astro CLI.

This document lists the Astro CLI documentation sets that Astronomer has retired from the documentation site.

While only maintained documentation sets appear under the Astro CLI documentation menu, Astronomer preserves all versions for historical reference. Documentation for Astro CLI versions v1.37 and earlier no longer receives updates and is not rendered on the Astronomer documentation site. You can still read these docs in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources).

To use the latest Astro CLI features and documentation, Astronomer recommends [upgrading to the latest version of the Astro CLI](/docs/cli/v1.45/install-cli).

## Retired versions

Documentation for the following Astro CLI versions is available in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources):

* v1.37
* v1.36
* v1.35
* v1.34

If you notice an error or misleading information in any part of Astronomer's documentation, [create a GitHub issue](https://github.com/astronomer/docs/issues) or contact [Astronomer support](https://support.astronomer.io).


# Astro CLI documentation archive
Source: https://astronomer.io/docs/cli/archive

Links to documentation for retired versions of the Astro CLI.

This document lists the Astro CLI documentation sets that Astronomer has retired from the documentation site.

While only maintained documentation sets appear under the Astro CLI documentation menu, Astronomer preserves all versions for historical reference. Documentation for Astro CLI versions v1.37 and earlier no longer receives updates and is not rendered on the Astronomer documentation site. You can still read these docs in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources).

To use the latest Astro CLI features and documentation, Astronomer recommends [upgrading to the latest version of the Astro CLI](/docs/cli/v1.45/install-cli).

## Retired versions

Documentation for the following Astro CLI versions is available in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources):

* v1.37
* v1.36
* v1.35
* v1.34

If you notice an error or misleading information in any part of Astronomer's documentation, [create a GitHub issue](https://github.com/astronomer/docs/issues) or contact [Astronomer support](https://support.astronomer.io).


# Astro CLI documentation archive
Source: https://astronomer.io/docs/cli/archive

Links to documentation for retired versions of the Astro CLI.

This document lists the Astro CLI documentation sets that Astronomer has retired from the documentation site.

While only maintained documentation sets appear under the Astro CLI documentation menu, Astronomer preserves all versions for historical reference. Documentation for Astro CLI versions v1.37 and earlier no longer receives updates and is not rendered on the Astronomer documentation site. You can still read these docs in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources).

To use the latest Astro CLI features and documentation, Astronomer recommends [upgrading to the latest version of the Astro CLI](/docs/cli/v1.45/install-cli).

## Retired versions

Documentation for the following Astro CLI versions is available in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources):

* v1.37
* v1.36
* v1.35
* v1.34

If you notice an error or misleading information in any part of Astronomer's documentation, [create a GitHub issue](https://github.com/astronomer/docs/issues) or contact [Astronomer support](https://support.astronomer.io).


# Astro CLI documentation archive
Source: https://astronomer.io/docs/cli/archive

Links to documentation for retired versions of the Astro CLI.

This document lists the Astro CLI documentation sets that Astronomer has retired from the documentation site.

While only maintained documentation sets appear under the Astro CLI documentation menu, Astronomer preserves all versions for historical reference. Documentation for Astro CLI versions v1.37 and earlier no longer receives updates and is not rendered on the Astronomer documentation site. You can still read these docs in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources).

To use the latest Astro CLI features and documentation, Astronomer recommends [upgrading to the latest version of the Astro CLI](/docs/cli/v1.45/install-cli).

## Retired versions

Documentation for the following Astro CLI versions is available in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources):

* v1.37
* v1.36
* v1.35
* v1.34

If you notice an error or misleading information in any part of Astronomer's documentation, [create a GitHub issue](https://github.com/astronomer/docs/issues) or contact [Astronomer support](https://support.astronomer.io).


# Create an Agent API token
Source: https://astronomer.io/docs/astro/api/v-1/agent-token/create-an-agent-api-token

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/deployments/{deploymentId}/agent-tokens
Create an Agent API token for remote workers.



# Delete an Agent API Token
Source: https://astronomer.io/docs/astro/api/v-1/agent-token/delete-an-agent-api-token

/astro/api/v-1/openapi.yaml delete /organizations/{organizationId}/deployments/{deploymentId}/agent-tokens/{agentTokenId}
Delete an Agent API Token.



# Get an Agent API Token
Source: https://astronomer.io/docs/astro/api/v-1/agent-token/get-an-agent-api-token

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/deployments/{deploymentId}/agent-tokens/{agentTokenId}
Get an Agent API Token.



# List Agent API Tokens
Source: https://astronomer.io/docs/astro/api/v-1/agent-token/list-agent-api-tokens

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/deployments/{deploymentId}/agent-tokens
List Agent API Tokens.



# Bulk create allowed IP address ranges
Source: https://astronomer.io/docs/astro/api/v-1/allowed-ip-address-range/bulk-create-allowed-ip-address-ranges

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/allowed-ip-address-ranges/bulk-create
Create up to 1000 allowed IP address ranges for the organization in one request. The whole batch is created atomically — if any item fails validation or conflicts with an existing range, no rows are inserted. This endpoint is NOT idempotent: a retry after an unacknowledged 2xx response may return 409. Clients should call ListAllowedIpAddressRanges to verify state before retrying.



# Bulk delete allowed IP address ranges
Source: https://astronomer.io/docs/astro/api/v-1/allowed-ip-address-range/bulk-delete-allowed-ip-address-ranges

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/allowed-ip-address-ranges/bulk-delete
Delete up to 1000 allowed IP address ranges for the organization in one request. The whole batch is deleted atomically. Unknown or duplicate IDs are accepted and ignored; matching rows for this organization are deleted.



# Create an allowed IP address range
Source: https://astronomer.io/docs/astro/api/v-1/allowed-ip-address-range/create-an-allowed-ip-address-range

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/allowed-ip-address-ranges
Create an allowed IP address range that constrains which IP addresses can be used to interact with your Astro Organization using APIs.



# Delete an allowed IP address range
Source: https://astronomer.io/docs/astro/api/v-1/allowed-ip-address-range/delete-an-allowed-ip-address-range

/astro/api/v-1/openapi.yaml delete /organizations/{organizationId}/allowed-ip-address-ranges/{allowedIpAddressRangeId}
Delete an allowed IP address range.



# List allowed IP address ranges
Source: https://astronomer.io/docs/astro/api/v-1/allowed-ip-address-range/list-allowed-ip-address-ranges

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/allowed-ip-address-ranges
List allowed IP address ranges.



# Create an API token
Source: https://astronomer.io/docs/astro/api/v-1/api-token/create-an-api-token

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/tokens
Create an API token. An API token is an alphanumeric token that grants programmatic access to Astro for automated workflows. An API token can be scoped to an Organization or a Workspace.



# Delete an API token
Source: https://astronomer.io/docs/astro/api/v-1/api-token/delete-an-api-token

/astro/api/v-1/openapi.yaml delete /organizations/{organizationId}/tokens/{tokenId}
Delete an API token. When you delete an API token, make sure that no existing automation workflows are using it. After it's deleted, an API token cannot be recovered.



# Get an API token
Source: https://astronomer.io/docs/astro/api/v-1/api-token/get-an-api-token

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/tokens/{tokenId}
Retrieve information about a specific API token.



# List API tokens
Source: https://astronomer.io/docs/astro/api/v-1/api-token/list-api-tokens

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/tokens
List information about all API tokens from an Organization. Filters on Workspace when Workspace ID is provided. When `includeOnlyOrganizationTokens` is `true`, only Organization API tokens are returned.



# Rotate API token
Source: https://astronomer.io/docs/astro/api/v-1/api-token/rotate-api-token

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/tokens/{tokenId}/rotate
Rotate an API token. Creates a new API token and invalidates the one you specify. Any workflows using the previous value stop working.



# Update an API token
Source: https://astronomer.io/docs/astro/api/v-1/api-token/update-an-api-token

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/tokens/{tokenId}
Update the name and description of an API token.



# Update API token roles
Source: https://astronomer.io/docs/astro/api/v-1/api-token/update-api-token-roles

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/tokens/{tokenId}/roles
Update Workspace and Organization roles for an API token.



# Get Organization Audit Logs
Source: https://astronomer.io/docs/astro/api/v-1/audit-logs

astro/api/v-1/openapi.yaml GET /organizations/{organizationId}/audit-logs
Retrieve logs across a specified time period for an Organization.

The response is returned as a gzipped NDJSON file.


# List authorization permission groups
Source: https://astronomer.io/docs/astro/api/v-1/authorization/list-authorization-permission-groups

/astro/api/v-1/openapi.yaml get /authorization/permission-groups
List the available permissions you can grant to a custom role.



# Changelog
Source: https://astronomer.io/docs/astro/api/v-1/changelog

Review changes to the Astro API v1.

<Update label="2026-08-20">
  ## Summary

  * Added an endpoint for cordoning and uncordoning a remote execution agent, along with the `Agent` schema that describes an agent's status, capabilities, queues, and task execution slots.

  ## Added

  * **Endpoints**
    * `POST /organizations/{organizationId}/deployments/{deploymentId}/agents/{agentId}/action` (`AgentAction`): Applies a `CORDON` or `UNCORDON` action to a remote agent. Returns the updated `Agent`.
  * **Schemas**
    * `Agent`: A remote execution agent. Reports `id`, `name`, `version`, `capabilities`, the `queues` it serves, its `slots`, `lastHeartbeatAt`, whether it `isCordoned` (with an optional `cordoningReason`), and a `status` of `HEALTHY`, `UNHEALTHY`, `CORDONED`, or `TERMINATED` with an optional `statusReason`.
    * `AgentSlots`: The agent's task execution slots, with required `available` and `total` counts.
    * `agentActionBody`: The `AgentAction` request body. Requires `action`, either `CORDON` to stop the agent accepting new work or `UNCORDON` to resume it.
</Update>

<Update label="2026-07-16">
  ## Summary

  * Added an optional `fieldErrors` array to the `Error` response schema. Request-validation failures (`400` responses) now include one entry per failed field, with a machine-readable code alongside the existing human-readable `message`.

  ## Added

  * **Properties**
    * `fieldErrors` (array of `FieldValidationError`): added to `Error`. Only present on `400` responses caused by request binding or validation. Each entry names the field, a validator code (for example `required`, `max`, `oneof`), and a friendly message.
  * **Schemas**
    * `FieldValidationError`: new schema with required `field`, `code`, and `message` string properties. Represents a single failed validation constraint.
</Update>

<Update label="2026-07-01">
  ## Summary

  * Added `setFields` to `EnvironmentObject` and `EnvironmentObjectLink` responses, and `unsetFields` to environment object link update overrides. Together they let clients distinguish a set-but-masked secret from an unset field, and clear a link override so it falls back to its parent value.
  * Changed the permission required to update Team roles from the org-wide `organization.teams.update` to `organization.teamRoles.access`. This fixes a regression where callers with only deployment-scoped team permissions received `403` errors when assigning a Team to a Deployment.

  ## Added

  * **Properties**
    * `setFields` (array of strings, required): added to `EnvironmentObject` and `EnvironmentObjectLink`. Names the value and override fields that currently hold a value, including masked secrets. Map members are reported as dotted paths (for example `extra.aws_secret`).
    * `unsetFields` (array of strings, maximum 100 items): added to `UpdateEnvironmentObjectOverridesRequest`. Names override fields to unset on a link so it inherits the parent value. A field can't be both set and listed in `unsetFields` in the same request.

  ## Changed

  * `POST /organizations/{organizationId}/teams/{teamId}/roles` (`UpdateTeamRoles`) now requires the `organization.teamRoles.access` permission instead of `organization.teams.update`. Individual role changes in the request are still authorized against their own scope (organization, workspace, or deployment).
</Update>

<Update label="2026-06-25">
  ## Summary

  * Added Private Network Egress mode for AWS clusters. When enabled, this mode disables public Internet connectivity from the cluster's Deployments and metrics exports. Cluster create, update, and read schemas gain an `isPrivateNetworkEgressEnabled` field.
  * Added a `description` field to environment objects. Environment object create, update, and read schemas now accept and return an optional description of up to 500 characters.

  ## Added

  * **Properties**
    * `isPrivateNetworkEgressEnabled` (boolean, AWS clusters only): added to `Cluster`, `CreateAwsClusterRequest`, and `UpdateDedicatedClusterRequest`. When true, disables public Internet connectivity from the cluster's Deployments and metrics exports.
    * `description` (string): added to `CreateEnvironmentObjectRequest` and `UpdateEnvironmentObjectRequest` (maximum 500 characters), and to `EnvironmentObject`.
</Update>

<Update label="2026-06-17">
  ## Summary

  * Added `hasAiFeaturesDisabled` and a required `isBlockedEnableAiFeatures` to the `Organization` schema. `hasAiFeaturesDisabled` indicates whether AI features are disabled for the Organization, and `isBlockedEnableAiFeatures` indicates whether the Organization is blocked from enabling AI features.
  * Added `hasAiFeaturesDisabled` to `UpdateOrganizationRequest` so callers can set whether AI features are disabled when updating an Organization.

  ## Added

  * **Properties**
    * `Organization`: `hasAiFeaturesDisabled` (boolean). Whether AI features are disabled for the Organization.
    * `Organization`: `isBlockedEnableAiFeatures` (boolean, required). Whether the Organization is blocked from enabling AI features.
    * `UpdateOrganizationRequest`: `hasAiFeaturesDisabled` (boolean). Whether AI features are disabled for the Organization.
</Update>

<Update label="2026-06-11">
  ## Summary

  * Added environment variable support to environment objects. A new `ENVIRONMENT_VARIABLE` object type lets you create, update, and read environment variables managed by the Astro Environment Manager, with an optional secret value.
  * Added SigV4 authentication for metrics export environment objects. A new `SIGV4` auth type accepts an `sigV4AssumeArn` IAM role ARN and an `sigV4StsRegion` AWS STS region.
  * Added Disaster Recovery (DR) support for GCP clusters. Cluster create, update, and read schemas gain `drPodSubnetRange`, `drServiceSubnetRange`, and `drServicePeeringRange`, and the existing DR fields are no longer scoped to AWS only.
  * Added a `location` field to `ProviderRegion` for multi-region DR compatibility.

  ## Added

  * **Enum values**
    * `ENVIRONMENT_VARIABLE`: added to the environment object `type` enum on `CreateEnvironmentObjectRequest` and `EnvironmentObject`, and to the `type` query parameter on `GET /organizations/{organizationId}/environment-objects`.
    * `SIGV4`: added to the `authType` enum on the six metrics export schemas (`CreateEnvironmentObjectMetricsExportRequest`, `CreateEnvironmentObjectMetricsExportOverridesRequest`, `UpdateEnvironmentObjectMetricsExportRequest`, `UpdateEnvironmentObjectMetricsExportOverridesRequest`, `EnvironmentObjectMetricsExport`, and `EnvironmentObjectMetricsExportOverrides`).
  * **Schemas**
    * `CreateEnvironmentObjectEnvironmentVariableRequest`: `isSecret` (boolean) and `value` (string).
    * `CreateEnvironmentObjectEnvironmentVariableOverridesRequest`: `value` (string).
    * `UpdateEnvironmentObjectEnvironmentVariableRequest`: `value` (string).
    * `UpdateEnvironmentObjectEnvironmentVariableOverridesRequest`: `value` (string).
    * `EnvironmentObjectEnvironmentVariable`: `isSecret` (boolean, required) and `value` (string, required). `value` is returned empty when the variable is a secret.
    * `EnvironmentObjectEnvironmentVariableOverrides`: `value` (string, required).
  * **Properties**
    * `environmentVariable`: added to `CreateEnvironmentObjectRequest`, `CreateEnvironmentObjectOverridesRequest`, `UpdateEnvironmentObjectRequest`, `UpdateEnvironmentObjectOverridesRequest`, and `EnvironmentObject`.
    * `environmentVariableOverrides`: added to `EnvironmentObjectLink`.
    * `sigV4AssumeArn` (string) and `sigV4StsRegion` (string): added to all six metrics export schemas listed under **Enum values**.
    * `drPodSubnetRange`, `drServiceSubnetRange`, and `drServicePeeringRange` (string, GCP clusters only): added to `Cluster`, `CreateGcpClusterRequest`, and `UpdateDedicatedClusterRequest`.
    * `enableReplicationTimeControl` (boolean): added to `CreateAzureClusterRequest`, `CreateGcpClusterRequest`, and `UpdateDedicatedClusterRequest`.
    * `drRegion` and `drVpcSubnetRange`: added to `UpdateDedicatedClusterRequest`.
    * `location` (string): added to `ProviderRegion`. The multi-region location code for DR compatibility.

  ## Changed

  * `enableReplicationTimeControl` description updated from "S3 Replication Time Control" to "Bucket Storage Replication Time Control" on `Cluster` and `CreateAwsClusterRequest`.
  * `drRegion` is no longer AWS-only: removed "For AWS clusters only" on `CreateAwsClusterRequest`, `CreateAzureClusterRequest`, and `CreateGcpClusterRequest`.
  * `drVpcSubnetRange` is no longer AWS-only: removed "For AWS clusters only" on `Cluster`, `CreateAwsClusterRequest`, `CreateAzureClusterRequest`, and `CreateGcpClusterRequest`.
  * `drSecondaryVpcCidr` description now notes it applies to AWS clusters only on `CreateAwsClusterRequest`.
  * `POST /organizations/{organizationId}/environment-objects` (CreateEnvironmentObject) description updated to include environment variables alongside connections, Airflow variables, and metrics export resources.
</Update>

<Update label="2026-06-10">
  ## Summary

  * Added two endpoints for managing allowed IP address ranges in bulk: `POST /organizations/{organizationId}/allowed-ip-address-ranges/bulk-create` and `POST /organizations/{organizationId}/allowed-ip-address-ranges/bulk-delete`. Each request accepts up to 1,000 items and processes the batch atomically.

  ## Added

  * **Endpoints**
    * `POST /organizations/{organizationId}/allowed-ip-address-ranges/bulk-create`: Create up to 1,000 allowed IP address ranges for an Organization in one request. The batch is created atomically: if any value fails validation or conflicts with an existing range, no ranges are created. The endpoint is not idempotent, so a retry after an unacknowledged 2xx response can return `409`. On success, the endpoint returns the created ranges as an `AllowedIpAddressRangesList`. Requires the `organization.allowedIpAddressRanges.create` permission.
    * `POST /organizations/{organizationId}/allowed-ip-address-ranges/bulk-delete`: Delete up to 1,000 allowed IP address ranges for an Organization in one request. The batch is deleted atomically. Unknown and duplicate IDs are accepted and ignored, and matching ranges for the Organization are deleted. The endpoint returns `204` with no response body. Requires the `organization.allowedIpAddressRanges.delete` permission.

  * **Schemas**
    * `BulkCreateAllowedIpAddressRangesRequest`: Request body for bulk create. Required: `allowedIpAddressRanges`, a non-empty array of up to 1,000 CIDR-format strings.
    * `BulkDeleteAllowedIpAddressRangesRequest`: Request body for bulk delete. Required: `allowedIpAddressRangeIds`, a non-empty array of up to 1,000 allowed IP address range IDs.
    * `AllowedIpAddressRangesList`: Response body for bulk create. Required: `allowedIpAddressRanges`, an array of `AllowedIpAddressRange`.
</Update>

<Update label="2026-06-04">
  ## Summary

  * Added `podEphemeralStorage` to the worker queue schemas so callers can set the ephemeral storage limit for each worker Pod.

  ## Added

  * **Properties**
    * `WorkerQueue`: `podEphemeralStorage` (string). The ephemeral storage limit for each worker Pod. Units are in Gibibytes or `Gi`. Example: `10Gi`.
    * `WorkerQueueRequest`: `podEphemeralStorage` (string). The ephemeral storage limit for each worker Pod. Must be a valid Kubernetes resource string, for example `10Gi`.
    * `UpdateWorkerQueueRequest`: `podEphemeralStorage` (string). The ephemeral storage limit for each worker Pod. Must be a valid Kubernetes resource string, for example `10Gi`.
</Update>

<Update label="2026-05-20">
  ## Summary

  * Added `lastRotatedAt` to `ApiToken` so callers can see when a token was last rotated.
  * Added a new `git` object (`CreateDeployGitRequest` on requests, `DeployGit` on responses) on `CreateDeployRequest` and `Deploy` for attaching git commit metadata to a deploy. A new `GENERIC` provider value (alongside `GITHUB`) supports non-GitHub remotes such as GitLab, Bitbucket, and self-hosted git via a `remoteUrl` field.
  * Added `IBM_ENTERPRISE` as a value on `OrganizationProductPlan.productPlanName`.
  * Narrowed `Workspace.defaultCloudProvider` from a free-form string to the enum `AWS`, `AZURE`, `GCP`.

  ## Added

  * **Schemas**
    * `CreateDeployGitRequest` and `DeployGit`: Git commit metadata associated with a deploy. Required: `commitSha`, `provider`. Optional: `account`, `authorName`, `authorUrl`, `authorUsername`, `beforeCommitSha`, `branch`, `commitUrl`, `path`, `remoteUrl`, `repo`. `provider` accepts `GITHUB` or `GENERIC`. For `GITHUB`, supply `account` and `repo` and leave `remoteUrl` empty. For `GENERIC`, supply `remoteUrl` and leave `account` and `repo` empty.

  * **Properties**
    * `ApiToken`: `lastRotatedAt` (string, date-time). The time when the API token was last rotated.
    * `CreateDeployRequest`, `Deploy`: `git` (object). Git commit metadata for the deploy. See `CreateDeployGitRequest` and `DeployGit`.

  * **Enum values**
    * `OrganizationProductPlan.productPlanName`: `IBM_ENTERPRISE`.

  ## Changed

  * **Schemas**
    * `Workspace.defaultCloudProvider`: Now restricted to the enum `AWS`, `AZURE`, `GCP` (previously a free-form string).
</Update>

<Update label="2026-04-22">
  ## Summary

  * Create Deployment requests now require only `name` and `workspaceId`. The server resolves infrastructure from Workspace defaults (or auto-selects when the Organization has a single non-shared cluster), infers the Deployment `type` from the resolved cluster, and applies defaults for runtime version, executor, scheduler size, and related fields. This reduces the create Deployment API from roughly 15 required fields down to 2.
  * Added `defaultClusterId`, `defaultCloudProvider`, and `defaultRegion` on `Workspace` so Workspace admins can pre-configure target infrastructure for new Deployments. `defaultClusterId` is mutually exclusive with `defaultCloudProvider` and `defaultRegion`.
  * Added `GET /organizations/{organizationId}/deployments/{deploymentId}/logs` and `GET /users/self` endpoints.
  * Added `hasAllowedIpAddressRanges` and `shouldEnforceDedicatedClusters` as required properties on the `Organization` schema.
  * Added `workspaceId` and `deploymentId` filters to `GET /organizations/{organizationId}/teams`.

  ## Added

  * **Endpoints**
    * `GET /organizations/{organizationId}/deployments/{deploymentId}/logs`: Get logs for an Astro Deployment. Supports filtering by log source (`scheduler`, `triggerer`, `worker`, `webserver`, `dag-processor`, `apiserver`), time range, text search, and pagination.
    * `GET /users/self`: Get the authenticated user's profile, roles, invites, and feature flags. Supports an optional `createIfNotExist` query parameter.

  * **Schemas**
    * `CreateDeploymentInstanceSpecRequest`
      * `au`: Integer. Astro unit allocation for the Deployment pod. Minimum `5`, maximum `24`. Optional.
      * `replicas`: Integer. Number of pod replicas. Minimum `1`, maximum `4`. Optional.
    * `UpdateDeploymentInstanceSpecRequest`
      * `au`: Integer. Astro unit allocation for the Deployment pod. Minimum `5`, maximum `24`. Required.
      * `replicas`: Integer. Number of pod replicas. Minimum `1`, maximum `4`. Required.
    * `DeploymentLog`
      * Required: `limit`, `maxNumResults`, `offset`, `resultCount`, `results`, `searchId`.
      * `results`: Array of `DeploymentLogEntry`.
    * `DeploymentLogEntry`
      * Required: `raw` (string), `source` (enum: `scheduler`, `webserver`, `triggerer`, `worker`, `dag-processor`, `apiserver`), `timestamp` (number).
    * `SelfUser`
      * Required: `avatarUrl`, `createdAt`, `fullName`, `id`, `status`, `updatedAt`, `username`.
      * Also includes `featureFlags`, `invites`, `isIdpManaged`, `organizationId`, and `roles`.
    * `SelfUserFeatureFlag`, `SelfUserInvite`, `SelfUserRole`, `SelfUserRoleScope`: Supporting schemas for `SelfUser`.
    * `UpdateWorkerQueueRequest`: Used by the `workerQueues` array on `UpdateDedicatedDeploymentRequest`, `UpdateHybridDeploymentRequest`, and `UpdateStandardDeploymentRequest`. Required fields: `isDefault`, `maxWorkerCount`, `minWorkerCount`, `name`, `workerConcurrency`.

  * **Properties**
    * `CreateWorkspaceRequest` and `UpdateWorkspaceRequest`: `defaultCloudProvider` (enum: `AWS`, `AZURE`, `GCP`), `defaultClusterId`, and `defaultRegion`. Workspace admins use these fields to pre-configure target infrastructure for new Deployments. When a create Deployment request omits `clusterId`, `cloudProvider`, and `region`, the server uses these Workspace defaults. `defaultClusterId` is mutually exclusive with `defaultCloudProvider` and `defaultRegion`.
    * `Organization`: `hasAllowedIpAddressRanges` (boolean, required) indicating whether the Organization has at least one allowed IP address range configured, and `shouldEnforceDedicatedClusters` (boolean, required).
    * `UpdateOrganizationRequest`: `shouldEnforceDedicatedClusters` (boolean).
    * `Workspace`: `defaultCloudProvider`, `defaultClusterId`, and `defaultRegion` (strings) reflecting the configured defaults.

  * **Query parameters**
    * `GET /organizations/{organizationId}/teams`: `workspaceId` and `deploymentId` filter the response to Teams with a role in the specified Workspace or Deployment.

  ## Changed

  * **Removed required fields on create Deployment requests.** When a field is omitted, the server applies a default. The only remaining required fields are `name` and `workspaceId`.
</Update>

<Update label="2026-04-02">
  ## Summary

  * Added new schemas for `UserTeamMembership` and `UserTeamsPaginated` to manage team memberships and pagination.
  * Enhanced `Cluster` schema with new properties for Disaster Recovery (DR) support, including `drRegion`, `isDrEnabled`, and others.
  * Introduced new permissions for managing environment objects and API tokens within organizations.
  * Removed `astroRuntimeVersion` property from deployment update requests.
  * Updated sorting options and descriptions for listing user teams within an organization.

  ## Added

  * **Schemas:**
    * `UserTeamMembership`
      * `dagRoles`: Array of DAG roles.
      * `deploymentRoles`: Array of deployment roles.
      * `description`: String, example: 'My Team description'.
      * `id`: String, example: 'clma5ftgk000008mhgev00k7d'.
      * `isIdpManaged`: Boolean, example: False.
      * `name`: String, example: 'My Team'.
      * `organizationRole`: Enum with values like '`ORGANIZATION_OWNER`', '`ORGANIZATION_MEMBER`', etc.
      * `rolesCount`: Integer, example: 1.
      * `workspaceRoles`: Array of workspace roles.
    * `UserTeamsPaginated`
      * `limit`: Integer, example: 10.
      * `offset`: Integer, example: 0.
      * `teams`: Array of `UserTeamMembership`.
      * `totalCount`: Integer, example: 100.
    * `Cluster`
      * `drRegion`: String, example: 'us-east-1'.
      * `drSecondaryVpcCidr`: String.
      * `drVpcSubnetRange`: String.
      * `enableReplicationTimeControl`: Boolean.
      * `failoverInProgress`: Boolean.
      * `isDrEnabled`: Boolean.
      * `isFailedOver`: Boolean.
    * `CreateAwsClusterRequest`, `CreateAzureClusterRequest`, `CreateGcpClusterRequest`
      * `drRegion`: String, example: 'us-west-2'.
      * `drSecondaryVpcCidr`: String, example: '100.64.0.0/19'.
      * `drVpcSubnetRange`: String, example: '172.20.0.0/22'.
      * `enableReplicationTimeControl`: Boolean.
    * `CreateDedicatedDeploymentRequest`, `CreateHybridDeploymentRequest`, `CreateStandardDeploymentRequest`
      * `drWorkloadIdentity`: String, example: 'arn:aws:iam::123456789:role/AirflowS3Logs-clmk2qqia000008mhff3ndjr0'.
    * `Deployment`
      * `drExternalIPs`: Array of strings.
      * `drOidcIssuerUrl`: String, example: 'https\://`westus2.oic.prod`-aks.azure.com/...'.
      * `effectiveDRWorkloadIdentity`: String.
    * `UpdateDedicatedClusterRequest`
      * `enableDr`: Boolean.
      * `isFailedOver`: Boolean.
    * `UpdateDedicatedDeploymentRequest`, `UpdateHybridDeploymentRequest`, `UpdateStandardDeploymentRequest`
      * `drWorkloadIdentity`: String.
      * `environmentVariables`: List of environment variables.

  * **Permissions:**
    * `/organizations/{organizationId}/environment-objects` and related endpoints: `organization.envObjects.access`.
    * `/organizations/{organizationId}/tokens` and related endpoints: `organization.apiTokens.access`.
    * `/organizations/{organizationId}/users/{userId}/roles`: `organization.userRoles.access`.

  ## Changed

  * **Schemas:**
    * `Cluster`
      * `secondaryVpcCidr` description updated to "The secondary VPC CIDR. For AWS clusters only."
      * Added `drRegion`, `isDrEnabled`, and `name` to required fields.
    * `OrganizationProductPlan`
      * Added `ENTERPRISE_BUSINESS_CRITICAL` to `productPlanName` enum.

  * **Endpoints:**
    * `/organizations/{organizationId}/users/{userId}/teams`
      * Response schema changed from `TeamsPaginated` to `UserTeamsPaginated`.
      * Permission action changed from `organization.users.get` to `organization.teams.get`.
      * Updated descriptions for parameters and summary.

  * **Removed:**
    * `astroRuntimeVersion` from `UpdateDedicatedDeploymentRequest`, `UpdateHybridDeploymentRequest`, and `UpdateStandardDeploymentRequest`.
</Update>

<Update label="2026-03-16">
  ## Summary

  * Added a new endpoint to list all Teams associated with a specific user.

  ## Added

  * **Endpoints**
    * `GET /organizations/{organizationId}/users/{userId}/teams`: List all Teams that a user belongs to within an Organization. Supports pagination with `offset` and `limit` query parameters, and sorting with the `sorts` query parameter.
</Update>

<Update label="2026-03-10">
  # v1 API Changelog

  ## Summary

  * Added discriminators to `CreateClusterRequest`, `CreateDeploymentRequest`, `UpdateClusterRequest`, and `UpdateDeploymentRequest` schemas. Discriminators remove ambiguity when creating or updating resources by mapping requests to the correct cloud provider or Deployment type.
  * Updated descriptions for `ApiTokenRole` and API token listing parameters.

  ## Added

  * **CreateClusterRequest Schema:**
    * Discriminator on `cloudProvider` with mappings for `AWS`, `AZURE`, and `GCP`.
  * **CreateDeploymentRequest Schema:**
    * Discriminator on `type` with mappings for `DEDICATED`, `HYBRID`, and `STANDARD`.
  * **UpdateClusterRequest Schema:**
    * Discriminator on `clusterType` with mappings for `DEDICATED` and `HYBRID`.
  * **UpdateDeploymentRequest Schema:**
    * Discriminator on `type` with mappings for `DEDICATED`, `HYBRID`, and `STANDARD`.

  ## Changed

  * **UpdateDedicatedDeploymentRequest, UpdateHybridDeploymentRequest, UpdateStandardDeploymentRequest Schemas:**
    * Removed description for `environmentVariables`.
  * **ApiTokenRole Schema:**
    * Updated description for `entityId`.
  * **API Token Listing Endpoint:**
    * Updated description for the parameter related to DAG tags.
</Update>

<Update label="2026-02-23">
  # v1 API Changelog

  ## Summary

  * Added new properties to the `DagRole`, `Deployment`, and `Team` schemas to enhance DAG access management.
  * Introduced new query parameters for pagination and sorting in the `/organizations/{organizationId}/tokens` and `/organizations/{organizationId}/users` endpoints.

  ## Added

  * **Schemas**
    * `DagRole`
      * `dagTag`: DAG tag, required if `DagId` is not specified.
    * `DeploymentEnvironmentVariable`
      * `updatedAt`: Format set to `date-time`.
    * `Invite`
      * `expiresAt`: Format set to `date-time`.
    * `Role`
      * `createdAt` and `updatedAt`: Examples and format set to `date-time`.
    * `RoleWithPermission`
      * `createdAt` and `updatedAt`: Examples and format set to `date-time`.
    * `Team`
      * `dagRoles`: Array of DAG roles.
  * **Endpoints**
    * `/organizations/{organizationId}/tokens`
      * `limit`: Limit for pagination.
      * `sorts`: Sorting criteria.
    * `/organizations/{organizationId}/users`
      * `limit`: Limit for pagination.
      * `sorts`: Sorting criteria.
  * **Enums**
    * `ApiTokenRole`
      * `entityType`: Added `DAG_TAG`.
</Update>

<Update label="2026-01-28">
  Initial release of the v1 Astro API.

  See [Migrate to v1](/docs/astro/api/v-1-beta-1/migrate-v1-api) for key changes and migration steps.
</Update>


# Create a cluster
Source: https://astronomer.io/docs/astro/api/v-1/cluster/create-a-cluster

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/clusters
Create a cluster in the Organization. An Astro cluster is a Kubernetes cluster that hosts the infrastructure required to run Deployments.



# Delete a cluster
Source: https://astronomer.io/docs/astro/api/v-1/cluster/delete-a-cluster

/astro/api/v-1/openapi.yaml delete /organizations/{organizationId}/clusters/{clusterId}
Delete a cluster.



# Get a cluster
Source: https://astronomer.io/docs/astro/api/v-1/cluster/get-a-cluster

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/clusters/{clusterId}
Retrieve details about a cluster.



# List clusters
Source: https://astronomer.io/docs/astro/api/v-1/cluster/list-clusters

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/clusters
List clusters in an Organization.



# Update a cluster
Source: https://astronomer.io/docs/astro/api/v-1/cluster/update-a-cluster

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/clusters/{clusterId}
Update a cluster in the Organization.



# Deploy to Astro with the API
Source: https://astronomer.io/docs/astro/api/v-1/deploy-with-api

Use the Astro API to deploy Apache Airflow code, dags, or images to Astro Deployments.

While you can deploy your Apache Airflow code to Astro using the Astro GitHub integration, the Astro CLI, or by configuring a CI/CD pipeline, your organization might prefer to use the Astro API to deploy code. This is because using the Astro API has very few dependencies, making it compatible with almost all CI/CD environments and security requirements.

If the Astro API has access to your Astro project files, you can use the `deploy` endpoints in the Astro API to complete either a complete project deploy, image-only deploy or dag-only deploy. You can then implement scripts to automate deploys as an alternative to using the Astro CLI or the Astro GitHub integration.

This best practice guide first walks through the steps that are necessary to deploy code to Astro using the Astro API for three different code deploy methods:

* A complete project deploy
* An image-only deploy
* A dags-only deploy

Then, the guide shows you the recommended way to combine these three automated processes to create a script with conditional logic that can automatically deploy code to Astro, depending on which types of files change in your Astro project. These examples are bash scripts that use Docker to build the image. You can also use a different scripting language, like Python, instead of bash.

## Feature overview

This guide highlights the following Astro features:

* The Astro API [`Deploy` endpoint](/docs/astro/api/v-1/deploy/list-deploys-for-a-deployment) to create and manage code deploys to an Astro Deployment.

## Prerequisites

This guide assumes that you have:

* An [API token](/docs/astro/deployment-api-tokens) with sufficient permissions to deploy code to Astro.
* At least one [Astro Deployment](/docs/astro/create-deployment).
* An [Astro project](/docs/cli/v1.43/develop-project) that's accessible from the machine that is making a request to the Astro API.
* [Docker](https://www.docker.com/), or an alternative container service like Podman.
* The following values:
  * Your Organization ID - See [List Organizations](/docs/astro/api/v-1/organization/list-organizations).
  * The Deployment ID - See [List Deployments](/docs/astro/api/v-1/deployments).
  * Your Astro API token - See [Create an API token](/docs/astro/automation-authentication#step-1-create-an-api-token).
  * The path where your Astro project exists.

## With dag-only deploys enabled

If you have dag-only deploys enabled, you can create scripts for complete project deploys, dag-only deploys, or image-only deploys. The following sections include a step by step description of the workflow followed by a bash script that executes the workflow.

See [Deploy code to Astro](/docs/astro/deploy-code) for more information about the different ways to update your dags and Airflow images.

### Complete project deploy

The following steps describe the different actions that the script performs to deploy a complete Astro project. Refer to [What happens during a project deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy) to learn the details about how a project deploy builds and deploys a new Astro image along with deploying dags.

1. Make a `POST` request to the `Deploy` endpoint to create a new `deploy` object. In your call, specify `type` as `IMAGE_AND_DAG`. Store the values for `id`, `imageRepository`, `imageTag`, and `dagsUploadURL` that are returned in the response to use in the following steps.

This action creates an object that represents the intent to deploy code to a Deployment. See the [Astro API documentation](/docs/astro/api/v-1/deploy/create-a-deploy) for request examples.

<Info>Replace the `<image-registry-hostname>` value in the following steps with the hostname returned by the API for `imageRepository`.</Info>

2. Authenticate to Astronomer's image registry using your Astro API token:

   ```bash wrap theme={null}

   docker login <image-registry-hostname> -u cli -p <your-api-token>

   ```

3. Build the image using the `imageRepository` and `imageTag` values that you retrieved in Step 1, as well as your Astro project path.

   ```bash wrap theme={null}

   docker build -t <imageRepository>:<imageTag> --platform=linux/amd64 <astro_project_path>

   ```

4. Push the image using the `imageRepository` and `imageTag` values that you retrieved in Step 1.

   ```bash wrap theme={null}

   docker push <imageRepository>:<imageTag>

   ```

5. Create a `tar.gz` file of your Astro project dags folder:

   ```bash wrap theme={null}

   tar -cvzf <path-to-create-tar-file>/dags.tar.gz "dags"

   ```

   <Info>Make sure to clean up the `dags.tar.gz` file after uploading.</Info>

6. Upload the `tar.gz` file by making a `PUT` call using the `dagsUploadURL` that you retrieved in Step 1. In this call, it is mandatory to pass the `x-ms-blob-type` as `BlockBlob`. Then, save the `x-ms-version-id` from the response header.

7. Using your deploy `id`, make a request to finalize the deploy. See [Astro API documentation](/docs/astro/api/v-1/deploy/finalize-a-deploy) for more information about formatting the API request.

   * On `Success`, your dags have successfully uploaded and a `x-ms-version-id` of the dags tarball is generated in the response headers. Pass this `x-ms-version-id` in the requested body to finish your updates.
   * It might take a few minutes for the changes to update in your Deployment.

<Accordion title="Complete project deploy script">
  ```bash expandable wrap theme={null}

  #!/bin/bash
  set -ex

  # Prerequisites: Set variables

  ORGANIZATION_ID=<set organization id>
  DEPLOYMENT_ID=<set deployment id>
  ASTRO_API_TOKEN=<set api token>
  ASTRO_PROJECT_PATH=<set path to your Astro project>

  # Step 1: Initialize deploy
  echo -e "Initiating Deploy Process for deployment $DEPLOYMENT_ID\n"
  CREATE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys" \
  --header "X-Astro-Client-Identifier: script" \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $ASTRO_API_TOKEN" \
  --data '{
  "type": "IMAGE_AND_DAG"
  }' | jq '.')

  DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id')
  REPOSITORY=$(echo $CREATE_DEPLOY | jq -r '.imageRepository')
  TAG=$(echo $CREATE_DEPLOY | jq -r '.imageTag')
  DAGS_UPLOAD_URL=$(echo $CREATE_DEPLOY | jq -r '.dagsUploadUrl')

  # Step 2: Log in to Docker
  docker login <image-registry-hostname> -u cli -p $ASTRO_API_TOKEN
  echo -e "\nBuilding Docker image $REPOSITORY:$TAG for $DEPLOYMENT_ID from $ASTRO_PROJECT_PATH"

  # Step 3: Build image
  docker build -t $REPOSITORY:$TAG --platform=linux/amd64 $ASTRO_PROJECT_PATH

  # Step 4: Push image
  echo -e "\nPushing Docker image $REPOSITORY:$TAG to $DEPLOYMENT_ID"
  docker push $REPOSITORY:$TAG

  # Step 5: Create tar file
  echo -e "\nCreating a dags tar file from $ASTRO_PROJECT_PATH/dags and stored in $ASTRO_PROJECT_PATH/dags.tar.gz\n"
  cd $ASTRO_PROJECT_PATH
  tar -cvzf "$ASTRO_PROJECT_PATH/dags.tar.gz" "dags"

  # Step 6: Upload dags tar file
  echo -e "\nUploading tar file $ASTRO_PROJECT_PATH/dags.tar.gz\n"
  VERSION_ID=$(curl -i --request PUT $DAGS_UPLOAD_URL \
  --header 'x-ms-blob-type: BlockBlob' \
  --header 'Content-Type: application/x-gtar' \
  --upload-file "$ASTRO_PROJECT_PATH/dags.tar.gz" | grep x-ms-version-id | awk -F': ' '{print $2}')

  VERSION_ID=$(echo $VERSION_ID | sed 's/\r//g') # Remove unexpected carriage return characters
  echo -e "\nTar file uploaded with version: $VERSION_ID\n"

  # Step 7: Finalizing Deploy
  FINALIZE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys/$DEPLOY_ID/finalize" \
  --header "X-Astro-Client-Identifier: script" \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $ASTRO_API_TOKEN" \
  --data '{"dagTarballVersion": "'$VERSION_ID'"}')

  ID=$(echo $FINALIZE_DEPLOY | jq -r '.id')
  if [[ "$ID" != null ]]; then
          echo -e "\nDeploy is Finalized. Image and dag changes for deployment $DEPLOYMENT_ID should be live in a few minutes"
          echo "Deployed Image tag: $TAG"
          echo "Deployed dag Tarball Version: $VERSION_ID"
  else
          MESSAGE=$(echo $FINALIZE_DEPLOY | jq -r '.message')
          if  [[ "$MESSAGE" != null ]]; then
                  echo $MESSAGE
          else
                  echo "Something went wrong. Reach out to astronomer support for assistance"
          fi
  fi

  # Cleanup
  echo -e "\nCleaning up the created tar file from $ASTRO_PROJECT_PATH/dags.tar.gz"
  rm -rf "$ASTRO_PROJECT_PATH/dags.tar.gz"

  ```
</Accordion>

### Dag-only deploy

The following script allows you to update only your Dag files. Refer to [Deploy dags](/docs/astro/deploy-dags) to learn the details about how Astro deploys dags.

1. Make a `POST` request to the `Deploy` endpoint to create a new `deploy` object. In your request, specify `type` as `DAG_ONLY`. Store the values for `id` and `dagsUploadURL` that are returned in the response to use in the following steps.

   This action creates an object that represents the intent to deploy code to a Deployment. See the [Astro API documentation](/docs/astro/api/v-1/deploy/create-a-deploy) for request usage and examples.

2. Create a `tar.gz` file of your Astro project dags folder:

   ```bash wrap theme={null}

   tar -cvzf <path to create the tar file>/dags.tar.gz "dags"

   ```

   <Note>Make sure to clean up the `dags.tar.gz` file after uploading.</Note>

3. Upload the tar file by making a `PUT` request using the `dagsUploadURL` that you retrieved in Step 2. In this request, it is mandatory to pass the `x-ms-blob-type` as `BlockBlob`. Then, save the `x-ms-version-id` from the response header.

4. Using your deploy `id`, make a request to finalize the deploy. See [Astro API documentation](/docs/astro/api/v-1/deploy/finalize-a-deploy) for more information about formatting the API request.

   * On `Success`, your dags have successfully uploaded and a `x-ms-version-id` of the dags tarball is generated. Pass this `x-ms-version-id` in the requested body to finish your updates.
   * It might take a few minutes for the changes to update in your Deployment.

<Accordion title="dag-only deploy script">
  ```bash expandable wrap theme={null}

  #!/bin/bash
  set -ex

  # Prerequisites: Set variables

  ORGANIZATION_ID=<set organization id>
  DEPLOYMENT_ID=<set deployment id>
  ASTRO_API_TOKEN=<set api token>
  ASTRO_PROJECT_PATH=<set path to your airflow project>

  # Step 1: Initialize deploy
  echo -e "Initiating Deploy Process for deployment $DEPLOYMENT_ID\n"
  CREATE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys" \
  --header "X-Astro-Client-Identifier: script" \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $ASTRO_API_TOKEN" \
  --data '{
  "type": "DAG_ONLY"
  }' | jq '.')

  DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id')
  DAGS_UPLOAD_URL=$(echo $CREATE_DEPLOY | jq -r '.dagsUploadUrl')

  # Step 2: Create a tar file of Astro project dags folder

  echo -e "\nCreating a dags tar file from $ASTRO_PROJECT_PATH/dags and stored in $ASTRO_PROJECT_PATH/dags.tar.gz\n"
  cd $ASTRO_PROJECT_PATH
  tar -cvzf "$ASTRO_PROJECT_PATH/dags.tar.gz" "dags"

  # Step 3: Upload tar file
  echo -e "\nUploading tar file $ASTRO_PROJECT_PATH/dags.tar.gz\n"
  VERSION_ID=$(curl -i --request PUT $DAGS_UPLOAD_URL \
  --header 'x-ms-blob-type: BlockBlob' \
  --header 'Content-Type: application/x-gtar' \
  --upload-file "$ASTRO_PROJECT_PATH/dags.tar.gz" | grep x-ms-version-id | awk -F': ' '{print $2}')

  VERSION_ID=$(echo $VERSION_ID | sed 's/\r//g') # Remove unexpected carriage return characters
  echo -e "\nTar file uploaded with version: $VERSION_ID\n"

  # Step 4: Finalizing Deploy
  FINALIZE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys/$DEPLOY_ID/finalize" \
  --header "X-Astro-Client-Identifier: script" \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $ASTRO_API_TOKEN" \
  --data '{"dagTarballVersion": "'$VERSION_ID'"}')

  ID=$(echo $FINALIZE_DEPLOY | jq -r '.id')
  if [[ "$ID" != null ]]; then
          echo -e "\nDeploy is Finalized. Dag changes for deployment $DEPLOYMENT_ID should be live in a few minutes"
          echo "Deployed dag Tarball Version: $VERSION_ID"
  else
          MESSAGE=$(echo $FINALIZE_DEPLOY | jq -r '.message')
          if  [[ "$MESSAGE" != null ]]; then
                  echo $MESSAGE
          else
                  echo "Something went wrong. Reach out to astronomer support for assistance"
          fi
  fi

  # Cleanup
  echo -e "\nCleaning up the created tar file from $ASTRO_PROJECT_PATH/dags.tar.gz"
  rm -rf "$ASTRO_PROJECT_PATH/dags.tar.gz"

  ```
</Accordion>

### Image-only deploy

The following script allows you to update only your Astro project by building and deploying a new Docker image. Refer to [What happens during a project deploy](/docs/astro/deploy-project-image#what-happens-during-a-project-deploy) to learn the details about how Astro deploys image updates.

1. Make a `POST` request to the `Deploy` endpoint to create a new `deploy` object. In your request, specify `type` as `IMAGE_ONLY`. Store the value for the `DeployID` that is returned. This action creates an object that represents the intent to deploy code to a Deployment.  See the [Astro API documentation](/docs/astro/api/v-1/deploy/create-a-deploy) for request usage and examples.

2. Log in to Docker with your Astro API token.

   ```bash wrap theme={null}

   docker login <image-registry-hostname> -u cli -p <your_astro_api_token>

   ```

3. Build the image using the `imageRepository` and `imageTag` values that you retrieved in Step 2, as well as your Astro project path.

   ```bash wrap theme={null}

   docker build -t imageRepository:imageTag --platform=linux/amd64 <astro_project_path>

   ```

4. Push the image using the `imageRepository` and `imageTag` values that you retrieved earlier.

   ```bash wrap theme={null}

   docker push imageRepository:imageTag

   ```

5. Using your deploy `id`, make a request to finalize the deploy. See [Astro API documentation](/docs/astro/api/v-1/deploy/finalize-a-deploy) for more information about formatting the API request.

   * On `Success`, your dags have successfully uploaded and a `x-ms-version-id` of the dags tarball is generated. Pass this `x-ms-version-id` in the requested body to finish your updates.
   * It might take a few minutes for the changes to update in your Deployment.

<Accordion title="Image-only deploy script">
  ```bash expandable wrap theme={null}
    #!/bin/bash
    set -ex

    # Prerequisites: Set variables

    ORGANIZATION_ID=<set organization id>
    DEPLOYMENT_ID=<set deployment id>
    ASTRO_API_TOKEN=<set api token>
    ASTRO_PROJECT_PATH=<set path to your airflow project>

    # Step 1: Initialize Deploy
    echo -e "Initiating Deploy Process for deployment $DEPLOYMENT_ID\n"
    CREATE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys" \
    --header "X-Astro-Client-Identifier: script" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $ASTRO_API_TOKEN" \
    --data '{
    "type": "IMAGE_ONLY"
    }' | jq '.')

    DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id')
    REPOSITORY=$(echo $CREATE_DEPLOY | jq -r '.imageRepository')
    TAG=$(echo $CREATE_DEPLOY | jq -r '.imageTag')

    # Step 2 Log in to Docker

    docker login <image-registry-hostname> -u cli -p $ASTRO_API_TOKEN

    # Step 3: Build Docker image
    echo -e "\nBuilding Docker image $REPOSITORY:$TAG for $DEPLOYMENT_ID from $ASTRO_PROJECT_PATH"
    docker build -t $REPOSITORY:$TAG --platform=linux/amd64 $ASTRO_PROJECT_PATH

    # Step 4: Push image
    echo -e "\nPushing Docker image $REPOSITORY:$TAG to $DEPLOYMENT_ID"
    docker push $REPOSITORY:$TAG

    # Step 5: Finalize Deploy
    FINALIZE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys/$DEPLOY_ID/finalize" \
    --header "X-Astro-Client-Identifier: script" \
    --header "Content-Type: application/json" \
    --header "Authorization: Bearer $ASTRO_API_TOKEN" \
    --data '{}')

    ID=$(echo $FINALIZE_DEPLOY | jq -r '.id')
    if [[ "$ID" != null ]]; then
            echo -e "\nDeploy is Finalized. Image changes for deployment $DEPLOYMENT_ID should be live in a few minutes"
            echo "Deployed Image tag: $TAG"
    else
            MESSAGE=$(echo $FINALIZE_DEPLOY | jq -r '.message')
            if  [[ "$MESSAGE" != null ]]; then
                    echo $MESSAGE
            else
                    echo "Something went wrong. Reach out to astronomer support for assistance"
            fi
    fi
  ```
</Accordion>

## With dag-only deploy disabled

You can only use complete project deploys if you have dag-only deploys disabled. Image-only and dag-only deploys do not work.

See [Deploy code to Astro](/docs/astro/deploy-code) for more information about the different ways to update your dags and Airflow images.

### Complete project deploy

1. Create the `Deploy` API call using the `IMAGE_AND_DAG` type. This action creates an object that represents the intent to deploy code to a Deployment.

2. After you create the `deploy`, retrieve the `id`, `imageRepository`, and `imageTag` values using the [`GET` deploy API call](/docs/astro/api/v-1/deploy/get-a-deploy-for-a-deployment), which you need in the following steps.

3. Log in to Docker with your Astro API token.

   ```bash wrap theme={null}

   docker login <image-registry-hostname> -u cli -p <your_astro_api_token>

   ```

4. Build the Docker image using the `imageRepository`, `imageTag`, and Astro project path values that you retrieved earlier.

   ```bash wrap theme={null}

   docker build -t imageRepository:imageTag --platform=linux/amd64 <astro_project_path>

   ```

5. Push the Docker image using the `imageRepository` and `imageTag` values that you retrieved earlier.

   ```bash wrap theme={null}

   docker push imageRepository:imageTag

   ```

6. Finalize the deploy. See [Finalize the deploy](/docs/astro/api/v-1/deploy/finalize-a-deploy) for more information about the API request.

   * On `Success`, the new image has successfully uploaded. Since you didn't update any dags in this deploy, pass the requested body as empty, `({})`.
   * It might take a few minutes for the changes to update in your Deployment.

<Accordion title="Complete project deploy script">
  ```bash expandable wrap theme={null}

  #!/bin/bash
  set -ex

  # Prerequisites: Set variables

  ORGANIZATION_ID=<set organization id>
  DEPLOYMENT_ID=<set deployment id>
  ASTRO_API_TOKEN=<set api token>
  ASTRO_PROJECT_PATH=<set path to your airflow project>

  # Step 1: Initializing Deploy
  echo -e "Initiating Deploy Process for deployment $DEPLOYMENT_ID\n"
  CREATE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys" \
  --header "X-Astro-Client-Identifier: script" \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $ASTRO_API_TOKEN" \
  --data '{
  "type": "IMAGE_AND_DAG"
  }' | jq '.')

  DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id')

  # Step 2-5: Build and Push Docker Image
  REPOSITORY=$(echo $CREATE_DEPLOY | jq -r '.imageRepository')
  TAG=$(echo $CREATE_DEPLOY | jq -r '.imageTag')
  docker login <image-registry-hostname> -u cli -p $ASTRO_API_TOKEN
  echo -e "\nBuilding Docker image $REPOSITORY:$TAG for $DEPLOYMENT_ID from $ASTRO_PROJECT_PATH"
  docker build -t $REPOSITORY:$TAG --platform=linux/amd64 $ASTRO_PROJECT_PATH
  echo -e "\nPushing Docker image $REPOSITORY:$TAG to $DEPLOYMENT_ID"
  docker push $REPOSITORY:$TAG

  # Step 6: Finalizing Deploy
  FINALIZE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys/$DEPLOY_ID/finalize" \
  --header "X-Astro-Client-Identifier: script" \
  --header "Content-Type: application/json" \
  --header "Authorization: Bearer $ASTRO_API_TOKEN" \
  --data '{}')

  ID=$(echo $FINALIZE_DEPLOY | jq -r '.id')
  if [[ "$ID" != null ]]; then
          echo -e "\nDeploy is Finalized. Image and dag changes for deployment $DEPLOYMENT_ID should be live in a few minutes"
          echo "Deployed Image tag: $TAG"
  else
          MESSAGE=$(echo $FINALIZE_DEPLOY | jq -r '.message')
          if  [[ "$MESSAGE" != null ]]; then
                  echo $MESSAGE
          else
                  echo "Something went wrong. Reach out to astronomer support for assistance"
          fi
  fi
  ```
</Accordion>

## (Recommended) Dynamic deploys based on files changed

In addition to manually triggering project, dag, and image deploys, you can create scripts that trigger specific code deploys depending on whether there have been changes to dags or files used to build the project image.

In this section, you can read a description of the process that the following code example shows the recommended way to combine the previous scripts to  trigger different code deploys depending on the files changed in your project. This example demonstrates how to combine all three types of deploys and the conditional logic for when to deploy each.

<Info>
  This recommended process requires that you enable [dag-only deploys](/docs/astro/deploy-dags).
</Info>

### Dynamic deploys description

In the following code example, the script completes the following processes:

1. Determine whether Dag files have been changed. If only dags have changed, then the script initiates a dag-only deploy.

2. Make a `POST` request to the `Deploy` endpoint to create a new `deploy` object.

3. If only dags have changed, the script initiates a dags-only deploy. If you changed more files, the script builds and deploys the project image, and then completes a dag-only deploy.

4. The script cleans up any tar files created during the build process.

To run the script, set the different environment variables to the values for your environment.

<Accordion title="Trigger deploys when files change code example">
  ```bash expandable wrap theme={null}
  #!/bin/bash
  set -ex

  # Prerequisites: Set variables

  ORGANIZATION_ID=<set organization id>
  DEPLOYMENT_ID=<set deployment id>
  ASTRO_API_TOKEN=<set api token>
  ASTRO_PROJECT_PATH=<set path to your airflow project>

  # Step 1: Determine if only dag files have changes
  files=$(git diff --name-only $(git rev-parse HEAD~1) -- .)
  dags_only=1
  for file in $files;do
  if [[ $file != "$ASTRO_PROJECT_PATH/dags"* ]];then
      echo "$file is not a dag, triggering a full image build"
      dags_only=0
      break
  fi
  done

  DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id')

  # Step 3: If only dags changed deploy only the dags in your 'dags' folder to your Deployment
  if [ $dags_only == 1 ]
  then
          echo -e "Initiating Deploy Process for deployment $DEPLOYMENT_ID\n"
          CREATE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys" \
          --header "X-Astro-Client-Identifier: script" \
          --header "Content-Type: application/json" \
          --header "Authorization: Bearer $ASTRO_API_TOKEN" \
          --data '{
          "type": "DAG_ONLY"
          }' | jq '.')

          DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id')

  	# Upload dags tar file
  	DAGS_UPLOAD_URL=$(echo $CREATE_DEPLOY | jq -r '.dagsUploadUrl')
  	echo -e "\nCreating a dags tar file from $ASTRO_PROJECT_PATH/dags and stored in $ASTRO_PROJECT_PATH/dags.tar.gz\n"
  	cd $ASTRO_PROJECT_PATH
  	tar -cvxf "$ASTRO_PROJECT_PATH/dags.tar.gz" "dags"
  	echo -e "\nUploading tar file $ASTRO_PROJECT_PATH/dags.tar.gz\n"
  	VERSION_ID=$(curl -i --request PUT $DAGS_UPLOAD_URL \
  	--header 'x-ms-blob-type: BlockBlob' \
  	--header 'Content-Type: application/x-gtar' \
  	--upload-file "$ASTRO_PROJECT_PATH/dags.tar.gz" | grep x-ms-version-id | awk -F': ' '{print $2}')

  	VERSION_ID=$(echo $VERSION_ID | sed 's/\r//g') # Remove unexpected carriage return characters
  	echo -e "\nTar file uploaded with version: $VERSION_ID\n"

  	# Finalizing Deploy
  	FINALIZE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys/$DEPLOY_ID/finalize" \
  	--header "X-Astro-Client-Identifier: script" \
  	--header "Content-Type: application/json" \
  	--header "Authorization: Bearer $ASTRO_API_TOKEN" \
  	--data '{"dagTarballVersion": "'$VERSION_ID'"}')

  	ID=$(echo $FINALIZE_DEPLOY | jq -r '.id')
  	if [[ "$ID" != null ]]; then
  	        echo -e "\nDeploy is Finalized. Dag changes for deployment $DEPLOYMENT_ID should be live in a few minutes"
  	        echo "Deployed dag Tarball Version: $VERSION_ID"
  	else
  	        MESSAGE=$(echo $FINALIZE_DEPLOY | jq -r '.message')
  	        if  [[ "$MESSAGE" != null ]]; then
  	                echo $MESSAGE
  	        else
  	                echo "Something went wrong. Reach out to astronomer support for assistance"
  	        fi
  	fi

  	# Step 4: Cleanup
  	echo -e "\nCleaning up the created tar file from $ASTRO_PROJECT_PATH/dags.tar.gz"
  	rm -rf "$ASTRO_PROJECT_PATH/dags.tar.gz"
  fi
  # Step 3: If any other files changed build your Astro project into a Docker image, push the image to your Deployment, and then push and dag changes
  if [ $dags_only == 0 ]
  then

          echo -e "Initiating Deploy Process for deployment $DEPLOYMENT_ID\n"
          CREATE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys" \
          --header "X-Astro-Client-Identifier: script" \
          --header "Content-Type: application/json" \
          --header "Authorization: Bearer $ASTRO_API_TOKEN" \
          --data '{
          "type": "IMAGE_AND_DAG"
          }' | jq '.')

          DEPLOY_ID=$(echo $CREATE_DEPLOY | jq -r '.id')

          # Build and Push Docker Image
  	REPOSITORY=$(echo $CREATE_DEPLOY | jq -r '.imageRepository')
  	TAG=$(echo $CREATE_DEPLOY | jq -r '.imageTag')
  	docker login <image-registry-hostname> -u cli -p $ASTRO_API_TOKEN
  	echo -e "\nBuilding Docker image $REPOSITORY:$TAG for $DEPLOYMENT_ID from $ASTRO_PROJECT_PATH"
  	docker build -t $REPOSITORY:$TAG --platform=linux/amd64 $ASTRO_PROJECT_PATH
  	echo -e "\nPushing Docker image $REPOSITORY:$TAG to $DEPLOYMENT_ID"
  	docker push $REPOSITORY:$TAG

  	# Upload dags tar file
  	DAGS_UPLOAD_URL=$(echo $CREATE_DEPLOY | jq -r '.dagsUploadUrl')
  	echo -e "\nCreating a dags tar file from $ASTRO_PROJECT_PATH/dags and stored in $ASTRO_PROJECT_PATH/dags.tar.gz\n"
  	cd $ASTRO_PROJECT_PATH
  	tar -cvxf "$ASTRO_PROJECT_PATH/dags.tar.gz" "dags"
  	echo -e "\nUploading tar file $ASTRO_PROJECT_PATH/dags.tar.gz\n"
  	VERSION_ID=$(curl -i --request PUT $DAGS_UPLOAD_URL \
  	--header 'x-ms-blob-type: BlockBlob' \
  	--header 'Content-Type: application/x-gtar' \
  	--upload-file "$ASTRO_PROJECT_PATH/dags.tar.gz" | grep x-ms-version-id | awk -F': ' '{print $2}')

  	VERSION_ID=$(echo $VERSION_ID | sed 's/\r//g') # Remove unexpected carriage return characters
  	echo -e "\nTar file uploaded with version: $VERSION_ID\n"

  	# Finalizing Deploy
  	FINALIZE_DEPLOY=$(curl --location --request POST "https://api.astronomer.io/v1/organizations/$ORGANIZATION_ID/deployments/$DEPLOYMENT_ID/deploys/$DEPLOY_ID/finalize" \
  	--header "X-Astro-Client-Identifier: script" \
  	--header "Content-Type: application/json" \
  	--header "Authorization: Bearer $ASTRO_API_TOKEN" \
  	--data '{"dagTarballVersion": "'$VERSION_ID'"}')

  	ID=$(echo $FINALIZE_DEPLOY | jq -r '.id')
  	if [[ "$ID" != null ]]; then
  	        echo -e "\nDeploy is Finalized. Image and dag changes for deployment $DEPLOYMENT_ID should be live in a few minutes"
  	        echo "Deployed Image tag: $TAG"
  	        echo "Deployed dag Tarball Version: $VERSION_ID"
  	else
  	        MESSAGE=$(echo $FINALIZE_DEPLOY | jq -r '.message')
  	        if  [[ "$MESSAGE" != null ]]; then
  	                echo $MESSAGE
  	        else
  	                echo "Something went wrong. Reach out to astronomer support for assistance"
  	        fi
  	fi

  	# Step 4: Cleanup
  	echo -e "\nCleaning up the created tar file from $ASTRO_PROJECT_PATH/dags.tar.gz"
  	rm -rf "$ASTRO_PROJECT_PATH/dags.tar.gz"
  fi
  ```
</Accordion>


# Create a Deploy
Source: https://astronomer.io/docs/astro/api/v-1/deploy/create-a-deploy

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/deployments/{deploymentId}/deploys
Create a new Deploy. A Deploy represents an intent to deploy new DAG code to an Astro Deployment.



# Deploy rollback of a deployment
Source: https://astronomer.io/docs/astro/api/v-1/deploy/deploy-rollback-of-a-deployment

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/deployments/{deploymentId}/deploys/{deployId}/rollback
Deploy rollback of a deployment



# Finalize a deploy
Source: https://astronomer.io/docs/astro/api/v-1/deploy/finalize-a-deploy

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/deployments/{deploymentId}/deploys/{deployId}/finalize
Finalize a deploy that you initialized using the `deploys` endpoint.



# Get a deploy for a deployment
Source: https://astronomer.io/docs/astro/api/v-1/deploy/get-a-deploy-for-a-deployment

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/deployments/{deploymentId}/deploys/{deployId}
Get a deploy for a deployment



# List deploys for a deployment
Source: https://astronomer.io/docs/astro/api/v-1/deploy/list-deploys-for-a-deployment

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/deployments/{deploymentId}/deploys
List deploys for a deployment



# Update a Deploy
Source: https://astronomer.io/docs/astro/api/v-1/deploy/update-a-deploy

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/deployments/{deploymentId}/deploys/{deployId}
Update an existing Deploy. A Deploy represents an intent to deploy new DAG code to an Astro Deployment.



# Configure a hibernation override for a deployment
Source: https://astronomer.io/docs/astro/api/v-1/deployment/configure-a-hibernation-override-for-a-deployment

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/deployments/{deploymentId}/hibernation-override



# Create a Deployment
Source: https://astronomer.io/docs/astro/api/v-1/deployment/create-a-deployment

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/deployments
Create a Deployment in the Organization. An Astro Deployment is an Airflow environment that is powered by all core Airflow components.



# Delete a Deployment
Source: https://astronomer.io/docs/astro/api/v-1/deployment/delete-a-deployment

/astro/api/v-1/openapi.yaml delete /organizations/{organizationId}/deployments/{deploymentId}
Delete a Deployment from an Organization.



# Delete a hibernation override for a deployment
Source: https://astronomer.io/docs/astro/api/v-1/deployment/delete-a-hibernation-override-for-a-deployment

/astro/api/v-1/openapi.yaml delete /organizations/{organizationId}/deployments/{deploymentId}/hibernation-override



# Get a Deployment
Source: https://astronomer.io/docs/astro/api/v-1/deployment/get-a-deployment

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/deployments/{deploymentId}
Retrieve details about a Deployment.



# Get Deployment logs
Source: https://astronomer.io/docs/astro/api/v-1/deployment/get-deployment-logs

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/deployments/{deploymentId}/logs
Get logs for an Astro Deployment. You must specify at least one log source.



# Trigger an action on a remote agent
Source: https://astronomer.io/docs/astro/api/v-1/deployment/trigger-an-action-on-a-remote-agent

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/deployments/{deploymentId}/agents/{agentId}/action



# Update a Deployment
Source: https://astronomer.io/docs/astro/api/v-1/deployment/update-a-deployment

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/deployments/{deploymentId}
Update a Deployment in the Organization.



# List Deployments
Source: https://astronomer.io/docs/astro/api/v-1/deployments

astro/api/v-1/openapi.yaml GET /organizations/{organizationId}/deployments
List Deployments in an Organization.

<Tip>
  To list multiple Deployments, string together the `deploymentIds` parameter. For example, `deployments?deploymentIds=deploymentId1&deploymentIds=deploymentId2`.
</Tip>


# Create a new environment object
Source: https://astronomer.io/docs/astro/api/v-1/environment/create-a-new-environment-object

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/environment-objects
Create an environment object for a Deployment or Workspace. An environment object represents a connection, environment variable, Airflow variable, or metrics export resource that is managed by the Astro Environment Manager.



# Delete an environment object
Source: https://astronomer.io/docs/astro/api/v-1/environment/delete-an-environment-object

/astro/api/v-1/openapi.yaml delete /organizations/{organizationId}/environment-objects/{environmentObjectId}
Delete an environment object from a Deployment or Workspace.



# Exclude linking an environment object
Source: https://astronomer.io/docs/astro/api/v-1/environment/exclude-linking-an-environment-object

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/environment-objects/{environmentObjectId}/exclude-linking
Exclude a specific Deployment from linking to an environment object created at the Workspace level.



# Get environment object
Source: https://astronomer.io/docs/astro/api/v-1/environment/get-environment-object

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/environment-objects/{environmentObjectId}
Retrieve details about an environment object.



# List environment objects
Source: https://astronomer.io/docs/astro/api/v-1/environment/list-environment-objects

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/environment-objects
List environment objects in a Workspace or Deployment.



# Update an environment object
Source: https://astronomer.io/docs/astro/api/v-1/environment/update-an-environment-object

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/environment-objects/{environmentObjectId}
Update an existing Deployment level or Workspace level environment object.



# Get started with the Astro API
Source: https://astronomer.io/docs/astro/api/v-1/get-started

Run a few simple commands to learn how to use the Astro API.

In this quick introduction to the Astro API, you'll make some simple requests to retrieve details about your Organization and create a Workspace API token.

## Prerequisites

* An [Astro account](/docs/astro/log-in-to-astro).
* An [Organization API token](/docs/astro/organization-api-tokens) with the Organization Owner role. Astronomer recommends that you create a new API token for this tutorial.
* An Astro [Workspace](/docs/astro/manage-workspaces).
* A method for making API requests. This tutorial assumes you're using curl, but you can also use tools such as Postman.

## Step 1: Make your first API request

To access most endpoints, you need to provide an Organization ID to the API as a path parameter. One of the few requests that doesn't require an Organization ID is the [List Organizations](https://www.astronomer.io/docs/astro/api/v-1/organization/list-organizations) request, which means that you can programmatically retrieve an Organization ID.

To retrieve the Organization ID through the API, run the following command:

```bash wrap theme={null}
curl --location 'https://api.astronomer.io/v1/organizations' \
--header 'Authorization: Bearer <your-organization-api-token>'
```

If the command was successful, then you receive a response that begins similarly to the following:

```json {14} wrap theme={null}
{
  "organizations": [
    {
      "billingEmail": "billing@example.com",
      "createdAt": "2022-11-22T04:37:12T",
      "createdBySubject": {
        "apiTokenName": "my-token",
        "avatarUrl": "https://avatar.url",
        "fullName": "Jane Doe",
        "id": "clm8qv74h000008mlf08scq7k",
        "subjectType": "USER",
        "username": "user1@example.com"
      },
      "id": "clmaxoarx000008l2c5ayb9pt",
      "isScimEnabled": false,
...
      ],
```

Copy the top-level `id` from this response. This is your Organization ID.

While you could have retrieved this value manually from the Astro UI, using the API lets you script this workflow and execute it on a regular basis.

## Step 2: Request Workspace details from the API

Using the Organization ID you copied, you can now find the ID for the Workspace where you want to create your API token.

Run the following command to list all Workspaces in your Organization:

```bash wrap theme={null}
curl --location 'https://api.astronomer.io/v1/organizations/<your-organization-id>/workspaces' \
--header 'Authorization: Bearer <your-api-token>'
```

If the command succeeds, the API returns a list of Workspaces similar to the following:

```json {15} wrap theme={null}
{
  "workspaces": [
    {
      "cicdEnforcedDefault": true,
      "createdAt": "2023-09-08T12:00:00Z",
      "createdBy": {
        "apiTokenName": "my-token",
        "avatarUrl": "https://avatar.url",
        "fullName": "Jane Doe",
        "id": "clm8qv74h000008mlf08scq7k",
        "subjectType": "USER",
        "username": "user1@example.com"
      },
      "description": "This is a test workspace",
      "id": "clm8t5u4q000008jq4qoc3036",
      "name": "My Workspace",
      "organizationId": "clm8t5u4q000008jq4qoc3036",
      "organizationName": "My Organization",
      "updatedAt": "2023-09-08T13:30:00Z",
      "updatedBy": {
        "apiTokenName": "my-token",
        "avatarUrl": "https://avatar.url",
        "fullName": "Jane Doe",
        "id": "clm8qv74h000008mlf08scq7k",
        "subjectType": "USER",
        "username": "user1@example.com"
      }
    }
  ]
}
```

In the response for your specific Workspace, copy the top-level `id`. This is your Workspace ID.

<Tip>
  If the API returns too many Workspaces, add some pagination parameters to your URL. For example, to limit your results to only the 20 most recently updated Workspaces, you can run:

  ```bash wrap theme={null}
  curl --location 'https://api.astronomer.io/v1/organizations/<your-organization-id>/workspaces?limit=20&sorts=updatedAt:asc' \
  --header 'Authorization: Bearer <your-organization-api-token>'
  ```
</Tip>

## Step 3: Update your token description using the API

Now that you have both an Organization ID and a Workspace ID, you can create a Workspace API token using the Astro API.

1. Run the following command to create a new Workspace API token:

   ```bash wrap theme={null}
   curl --location 'https://api.astronomer.io/v1/organizations/<your-organization-id>/tokens' \
   --header 'Content-Type: application/json' \
   --header 'Authorization: Bearer <your-organization-api-token>' \
   --data '{
       "description": "I wrote this description using the Astro API!",
       "entityId": "<your-workspace-id>",
       "name": "My new API token",
       "role": "WORKSPACE_MEMBER",
       "type": "WORKSPACE"
   }'
   ```

   If the request was successful, the API will return a response with your new token's details.

2. In the Astro UI, go to **Workspace Settings** > **Access Management** > **API Tokens** and find your Workspace API token. You should see your updated description under **Description**.


# Create a user invitation
Source: https://astronomer.io/docs/astro/api/v-1/invite/create-a-user-invitation

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/invites
Invite a user to an Organization.



# Delete a user invite
Source: https://astronomer.io/docs/astro/api/v-1/invite/delete-a-user-invite

/astro/api/v-1/openapi.yaml delete /organizations/{organizationId}/invites/{inviteId}
Delete an existing user invite.



# Migrate to v1
Source: https://astronomer.io/docs/astro/api/v-1/migrate-v1-api

Migrate your Astro API integrations from v1beta1 to the unified v1 endpoint.

This guide helps you migrate from the Astro API v1beta1 to v1. The v1 API consolidates the previously separate IAM and Platform APIs into a single endpoint with improved stability and support.

## Key changes

### API consolidation

The two separate v1beta1 APIs are now unified:

**Before (v1beta1):**

* IAM API: `https://api.astronomer.io/iam/v1beta1/`
* Platform API: `https://api.astronomer.io/platform/v1beta1/`

**After (v1):**

* Unified API: `https://api.astronomer.io/v1/`

### Unsupported endpoints

The following endpoints are **currently not supported** in v1:

* Alert endpoints:
  * `POST /organizations/{organizationId}/alerts`
  * `GET /organizations/{organizationId}/alerts`
  * `GET /organizations/{organizationId}/alerts/{alertId}`
  * `POST /organizations/{organizationId}/alerts/{alertId}`
  * `DELETE /organizations/{organizationId}/alerts/{alertId}`

* Notification channel endpoints:
  * `POST /organizations/{organizationId}/notification-channels`
  * `GET /organizations/{organizationId}/notification-channels`
  * `GET /organizations/{organizationId}/notification-channels/{notificationChannelId}`
  * `POST /organizations/{organizationId}/notification-channels/{notificationChannelId}`
  * `DELETE /organizations/{organizationId}/notification-channels/{notificationChannelId}`

If your integration relies on these endpoints, contact [Astronomer Support](/docs/astro/astro-support) for guidance on alternative approaches.

## Migration steps

### Step 1: Update base URLs

Update all API calls to use the new v1 base URL:

```diff wrap theme={null}
- const IAM_BASE_URL = 'https://api.astronomer.io/iam/v1beta1';
- const PLATFORM_BASE_URL = 'https://api.astronomer.io/platform/v1beta1';
+ const BASE_URL = 'https://api.astronomer.io/v1';
```

### Step 2: Update endpoint paths

All endpoints that previously used the IAM or Platform base URLs now use the unified v1 base URL:

**Example: List users**

```diff wrap theme={null}
- GET https://api.astronomer.io/iam/v1beta1/organizations/{organizationId}/users
+ GET https://api.astronomer.io/v1/organizations/{organizationId}/users
```

**Example: List deployments**

```diff wrap theme={null}
- GET https://api.astronomer.io/platform/v1beta1/organizations/{organizationId}/deployments
+ GET https://api.astronomer.io/v1/organizations/{organizationId}/deployments
```

### Step 3: Update alert and notification channel logic

Alert and notification channel APIs currently are not supported in v1. If your integration uses these endpoints, you'll need to update your workflows:

```diff wrap theme={null}
- // Create an alert
- const response = await fetch(
-   `${PLATFORM_BASE_URL}/organizations/${orgId}/alerts`,
-   {
-     method: 'POST',
-     headers: { 'Authorization': `Bearer ${token}` },
-     body: JSON.stringify(alertConfig)
-   }
- );
+ // Alert APIs are currently not supported in v1
+ // Contact Astronomer Support for alternative approaches
```

### Step 4: Test your integration

Before fully migrating to production:

1. **Update your development/staging environment** to use v1 endpoints
2. **Test all API calls** to ensure they work as expected
3. **Verify authentication** still works with the new base URL
4. **Check error handling** for any changes in response formats
5. **Monitor for any unexpected behavior**

### Step 5: Update production

After testing is complete, update your production environment to use the v1 API.

## Examples

### v1beta1

```python wrap theme={null}
# Before (v1beta1)
iam_base_url = "https://api.astronomer.io/iam/v1beta1"
platform_base_url = "https://api.astronomer.io/platform/v1beta1"

# List users
response = requests.get(
    f"{iam_base_url}/organizations/{org_id}/users",
    headers={"Authorization": f"Bearer {token}"}
)

# List deployments
response = requests.get(
    f"{platform_base_url}/organizations/{org_id}/deployments",
    headers={"Authorization": f"Bearer {token}"}
)
```

### v1

```python wrap theme={null}
# After (v1)
base_url = "https://api.astronomer.io/v1"

# List users
response = requests.get(
    f"{base_url}/organizations/{org_id}/users",
    headers={"Authorization": f"Bearer {token}"}
)

# List deployments
response = requests.get(
    f"{base_url}/organizations/{org_id}/deployments",
    headers={"Authorization": f"Bearer {token}"}
)
```

If you have questions or need assistance migrating to v1, reach out to [Astronomer Support](/docs/astro/astro-support).


# Get cluster options
Source: https://astronomer.io/docs/astro/api/v-1/options/get-cluster-options

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/cluster-options
Get all possible options for configuring a cluster.



# Get Deployment options
Source: https://astronomer.io/docs/astro/api/v-1/options/get-deployment-options

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/deployment-options
Get the options available for configuring a Deployment.



# Get an Organization
Source: https://astronomer.io/docs/astro/api/v-1/organization/get-an-organization

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}
Retrieve information about a specific Organization.



# List Organizations
Source: https://astronomer.io/docs/astro/api/v-1/organization/list-organizations

/astro/api/v-1/openapi.yaml get /organizations
List the details about all Organizations that you have access to. Requires using a personal access token (PAT) for authentication.



# Update an Organization
Source: https://astronomer.io/docs/astro/api/v-1/organization/update-an-organization

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}
Update an Organization's details.



# Astro API overview
Source: https://astronomer.io/docs/astro/api/v-1/overview

Use the Astro API REST endpoints to manage Organizations, Workspaces, and Deployments.

The Astro API is a standard REST API that you can use to develop applications and scripts for interacting with Astro components.

The v1 API provides a unified endpoint for managing your Astro infrastructure and resources, including:

* Organizations, Workspaces, and Deployments
* Clusters and deployment infrastructure
* Users, Teams, and role-based access control (RBAC)
* API tokens and authentication

To make your first request using the Astro API, see [Get started with the Astro API](/docs/astro/api/v-1/get-started).

<Note>
  **Astro API v1 is now generally available.** The previous v1beta1 API is deprecated and will reach end of support in January 2027. See the [v1beta1 deprecation notice](/docs/astro/api/v-1-beta-1/v1beta1-deprecation-notice) and [migration guide](/docs/astro/api/v-1-beta-1/migrate-v1-api) for details.
</Note>

<Tip>
  Looking to use the Airflow REST API instead? See the [Astro documentation](/docs/astro/airflow-api) to learn how to make requests to Airflow Deployments using the Airflow API.
</Tip>

## Authentication

All requests to the API must be authenticated. You can use [bearer authentication](https://swagger.io/docs/specification/authentication/bearer-authentication/) to authenticate with a [Workspace API token](/docs/astro/workspace-api-tokens), [Organization API token](/docs/astro/organization-api-tokens), or [Deployment API token](/docs/astro/deployment-api-tokens). The following example shows how you can add a token to a curl request:

```curl wrap theme={null}
curl --location 'https://api.astronomer.io/v1/organizations/<your-organization-id>/clusters' \
--header 'Authorization: Bearer <your-api-token>'
```

Endpoints can return subsets of specific attributes based on the permissions of your API token. If your token's role allows you to access something in the Astro UI or Astro CLI, it also allows you to access the same thing or action using the API. See [User permissions](/docs/astro/user-permissions) for a list of all possible permissions.

## Rate limiting

The maximum number of API requests you can make with the same API token depends on the type of request you're making:

* **`POST` requests**: You can make a make a maximum of 10 requests per second using the same API token.
* **`DELETE` requests**: You can make a make a maximum of 5 requests per second using the same API token.
* **`GET` requests**: You can make a make a maximum of 25 requests per second using the same API token.

## Idempotent requests

Astro supports different levels of [idempotency](https://en.wikipedia.org/wiki/Idempotence) for different request types.

* **`POST` requests (Create)**: Identical `POST` requests to create a new object will result in the creation of multiple objects. For example, if you make identical requests to create an Organization, Astro creates multiple Organizations with identical settings and unique IDs.
* **`POST` requests (Update)**: Idempotency is guaranteed for all `POST` requests to update an existing object.
* **`DELETE` requests**: Idempotency is guaranteed for all `DELETE` requests. Any successive identical `DELETE` requests return a 404 error.

## API status codes

If the API returns a `200` or `204` code, your API request was a success. If the API returns a `40x` or `500` code, your request resulted in one of the following errors:

* `400`: **Bad Request** - Your request was not successful because it was not formatted properly, possibly due to missing required parameters.
* `401`: **Unauthorized** - Your request did not include an API token.
* `404`: **Resource Not Found** - The resource you're trying to access does not exist.
* `403`: **Forbidden** - The API token you included did not have sufficient permissions to complete the request.
* `500`: **Internal server error** - The request could not be completed because of an error caused by Astro.

All error responses include a `requestId` that you can share with Astronomer support if you want to learn more about the error.

## Download OpenAPI specification

To export Astro API's OpenAPI specification for use on another platform, such as Postman or Swagger, download the YAML configuration from the following page and import it into the tool of your choice.

* [Astro API v1 specification](https://api.astronomer.io/spec/v1)


# Create a custom role
Source: https://astronomer.io/docs/astro/api/v-1/role/create-a-custom-role

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/roles
Create a custom role that you can assign to users, Teams, and API tokens.



# Delete a custom role
Source: https://astronomer.io/docs/astro/api/v-1/role/delete-a-custom-role

/astro/api/v-1/openapi.yaml delete /organizations/{organizationId}/roles/{customRoleId}
Delete a custom role.



# Get a custom role
Source: https://astronomer.io/docs/astro/api/v-1/role/get-a-custom-role

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/roles/{roleId}
Get details about a custom role.



# Get role templates
Source: https://astronomer.io/docs/astro/api/v-1/role/get-role-templates

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/role-templates
Get a list of available role templates in an Organization. A role template can be used as the basis for creating a new custom role.



# List roles
Source: https://astronomer.io/docs/astro/api/v-1/role/list-roles

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/roles
List available user roles in an Organization.



# Update custom role
Source: https://astronomer.io/docs/astro/api/v-1/role/update-custom-role

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/roles/{customRoleId}
Update the metadata or included permissions for a custom role.



# Add members to a team
Source: https://astronomer.io/docs/astro/api/v-1/team/add-members-to-a-team

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/teams/{teamId}/members
Add members to a team



# Create a Team
Source: https://astronomer.io/docs/astro/api/v-1/team/create-a-team

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/teams
Create a Team in an Organization. A Team is a group of users that have the same set of permissions in an Organization or Workspace.



# Delete a Team
Source: https://astronomer.io/docs/astro/api/v-1/team/delete-a-team

/astro/api/v-1/openapi.yaml delete /organizations/{organizationId}/teams/{teamId}
Delete a Team. Deleting a Team will remove all permissions associated with the Team. Users that previously belonged to the Team will no longer have these permissions.



# Get a Team
Source: https://astronomer.io/docs/astro/api/v-1/team/get-a-team

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/teams/{teamId}
Retrieve details about a specific Team.



# List Team members
Source: https://astronomer.io/docs/astro/api/v-1/team/list-team-members

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/teams/{teamId}/members
List the details about all users that belong to a specific Team.



# List Teams
Source: https://astronomer.io/docs/astro/api/v-1/team/list-teams

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/teams
List all Teams in an Organization. Optionally filter by Workspace or Deployment membership.



# Remove Team member
Source: https://astronomer.io/docs/astro/api/v-1/team/remove-team-member

/astro/api/v-1/openapi.yaml delete /organizations/{organizationId}/teams/{teamId}/members/{memberId}
Remove a user from a Team. The user loses all permissions associated with the Team.



# Update a Team
Source: https://astronomer.io/docs/astro/api/v-1/team/update-a-team

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/teams/{teamId}
Update a Team's details.



# Update Team roles
Source: https://astronomer.io/docs/astro/api/v-1/team/update-team-roles

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/teams/{teamId}/roles
Update Organization and Workspace roles for a Team.



# Get current user
Source: https://astronomer.io/docs/astro/api/v-1/user/get-current-user

/astro/api/v-1/openapi.yaml get /users/self
Get the authenticated user's profile and roles.



# Get user information
Source: https://astronomer.io/docs/astro/api/v-1/user/get-user-information

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/users/{userId}
Retrieve user information about a specific user account.



# List teams for a user
Source: https://astronomer.io/docs/astro/api/v-1/user/list-teams-for-a-user

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/users/{userId}/teams
List the teams a user belongs to within an Organization.



# Update a user's roles
Source: https://astronomer.io/docs/astro/api/v-1/user/update-a-users-roles

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/users/{userId}/roles
Update Organization and Workspace roles for a user.



# List Users
Source: https://astronomer.io/docs/astro/api/v-1/users

astro/api/v-1/openapi.yaml GET /organizations/{organizationId}/users
List users in an Organization or a specific Workspace within an Organization.

<Note>
  If you specify `workspaceId`, the response lists the user's Workspace and Organization roles. If you specify `deploymentId`, the response lists the user's Deployment and Organization roles. If you specify both `workspaceId` and `deploymentId`, the response lists Workspace, Deployment, and Organization roles for users that belong to both the specified Workspace and Deployment.
</Note>


# Astro API versioning, maintenance, and support
Source: https://astronomer.io/docs/astro/api/v-1/versioning-and-support

Learn how Astronomer versions and maintains the Astro API.

The Astro API is available in both Generally Available (GA) and beta versions. GA versions provide stable, production-ready contracts for long-term integrations.

## Available versions

| Version | Status     | Base URL                                                                                    | Release Date     | End of Support |
| ------- | ---------- | ------------------------------------------------------------------------------------------- | ---------------- | -------------- |
| v1      | GA         | `https://api.astronomer.io/v1/`                                                             | January 28, 2026 | Active         |
| v1beta1 | Deprecated | `https://api.astronomer.io/iam/v1beta1/`<br />`https://api.astronomer.io/platform/v1beta1/` | April 2024       | January 2027   |

<Note>
  **v1beta1 users** have a 12-month migration window to transition to v1. See the [v1 migration guide](/docs/astro/api/v-1-beta-1/migrate-v1-api) for key changes and migration steps.
</Note>

## Version upgrades and breaking changes

The Astro API uses a `major.minor` versioning strategy to provide stability while allowing for continuous improvement.

### Versioning approach

**Major version**

The API major version is indicated in the URI path. For example, v1 APIs use the base URL:

```text wrap theme={null}
https://api.astronomer.io/v1/
```

Breaking changes require a new major version with a dedicated migration guide and deprecation timeline.

**Minor version**

You can pin to a specific minor version using the `X-API-Version` header:

```bash wrap theme={null}
curl --location 'https://api.astronomer.io/v1/organizations/<org-id>/clusters' \
--header 'Authorization: Bearer <your-api-token>' \
--header 'X-API-Version: 1.0'
```

If you don't specify the `X-API-Version` header, your requests automatically use the latest minor version within the major version. This ensures you receive non-breaking improvements and new features automatically.

### OpenAPI specification

You can download the v1 OpenAPI specification for the Astro API from:

* **Latest minor version**: `https://api.astronomer.io/spec/v1`
* **Specific version**: `https://api.astronomer.io/spec/v1.0`

The `/spec/v1` endpoint always returns the latest minor version within the v1 major release, while specific version endpoints like `/spec/v1.0` return that exact specification.


# Create Workspace
Source: https://astronomer.io/docs/astro/api/v-1/workspace/create-workspace

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/workspaces
Create a Workspace.



# Delete Workspace
Source: https://astronomer.io/docs/astro/api/v-1/workspace/delete-workspace

/astro/api/v-1/openapi.yaml delete /organizations/{organizationId}/workspaces/{workspaceId}
Delete a Workspace.



# Get Workspace
Source: https://astronomer.io/docs/astro/api/v-1/workspace/get-workspace

/astro/api/v-1/openapi.yaml get /organizations/{organizationId}/workspaces/{workspaceId}
Get information about a Workspace.



# Update Workspace
Source: https://astronomer.io/docs/astro/api/v-1/workspace/update-workspace

/astro/api/v-1/openapi.yaml post /organizations/{organizationId}/workspaces/{workspaceId}
Update a Workspace.



# List Workspaces
Source: https://astronomer.io/docs/astro/api/v-1/workspaces

astro/api/v-1/openapi.yaml GET /organizations/{organizationId}/workspaces
List Workspaces in an Organization

<Tip>
  To list multiple Workspaces, string together the `workspaceIds` parameter. For example, `workspaces?workspaceIds=workspaceId1&workspaceIds=workspaceId2`.
</Tip>


# Astro CLI documentation archive
Source: https://astronomer.io/docs/cli/archive

Links to documentation for retired versions of the Astro CLI.

This document lists the Astro CLI documentation sets that Astronomer has retired from the documentation site.

While only maintained documentation sets appear under the Astro CLI documentation menu, Astronomer preserves all versions for historical reference. Documentation for Astro CLI versions v1.37 and earlier no longer receives updates and is not rendered on the Astronomer documentation site. You can still read these docs in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources).

To use the latest Astro CLI features and documentation, Astronomer recommends [upgrading to the latest version of the Astro CLI](/docs/cli/v1.45/install-cli).

## Retired versions

Documentation for the following Astro CLI versions is available in the [Astronomer docs resources GitHub repository](https://github.com/astronomer/astronomer-docs-resources):

* v1.37
* v1.36
* v1.35
* v1.34

If you notice an error or misleading information in any part of Astronomer's documentation, [create a GitHub issue](https://github.com/astronomer/docs/issues) or contact [Astronomer support](https://support.astronomer.io).


# Introduction to Apache Airflow®: A Technical Overview for Beginners
Source: https://astronomer.io/docs/learn/intro-to-airflow

Start learning Apache Airflow with this comprehensive guide. Covers Dags, tasks, scheduling, and all core concepts. Includes code examples and links to relevant tutorials.

[Apache Airflow®](https://airflow.apache.org/) is an open source tool for programmatically authoring, scheduling, and monitoring data pipelines. Every month, millions of new and returning users download Airflow and it has a large, active open source [community](https://airflow.apache.org/community/). The core principle of Airflow is to define data pipelines as code, allowing for dynamic and scalable workflows.

This guide offers an introduction to Apache Airflow and its core concepts. You'll learn about:

* Why you should use Airflow.
* Common use cases for Airflow.
* How to run Airflow.
* Important Airflow concepts.
* Where to find resources to learn more about Airflow.

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  Hands-on tutorial: [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Python. See the [Python Documentation](https://docs.python.org/3/tutorial/index.html).

## Why use Airflow

[Apache Airflow](https://www.astronomer.io/airflow) is a platform for programmatically authoring, scheduling, and monitoring workflows. It is especially useful for creating and orchestrating complex data pipelines.

Data orchestration sits at the heart of any modern data stack and provides elaborate automation of data pipelines. With orchestration, actions in your data pipeline become aware of each other and your data team has a central location to monitor, edit, and troubleshoot their workflows.

One of the core tenets of Airflow that sets it apart from other data orchestration tools is defining data pipelines as code. Modern data teams seek to define their workflows in code to:

* Manage workflows with their existing change-control and version-control tooling.
* Bulk-create/update workflows too numerous, large, or dynamic to manage from a GUI.
* Define re-usable, parameterizable workflow sections.

Airflow provides many additional benefits, including:

* **Tool agnosticism**: Airflow can connect to any application in your data ecosystem that allows connections through an API. Prebuilt [operators](/docs/learn/what-is-an-operator) exist to connect to many common data tools.
* **High extensibility**: Since Airflow pipelines are written in Python, you can build on top of the existing codebase and extend the functionality of Airflow to meet your needs. Anything you can do in Python, you can do in Airflow. Support to define tasks in languages other than Python is planned for 3.0+.
* **Infinite scalability**: Given enough computing power, you can orchestrate as many processes as you need, no matter the complexity of your pipelines.
* **Dynamic data pipelines**: Airflow offers the ability to create [dynamic tasks](/docs/learn/dynamic-tasks) to adjust your workflows based on the data you are processing at runtime.
* **Vibrant OSS community**: With millions of users and thousands of contributors, Airflow is here to stay and grow. Join the [Airflow Slack](https://apache-airflow-slack.herokuapp.com/) to become part of the community.
* **Observability**: The Airflow UI provides an immediate overview of all your data pipelines and can provide the source of truth for workflows in your whole data ecosystem.

<Frame>
  <img alt="Screenshot of the main view of the Airflow UI with seven enabled workflows." />
</Frame>

## Airflow use cases

Many data professionals at [companies of all sizes and types](https://github.com/apache/airflow/blob/main/INTHEWILD) use Airflow. Data engineers, data scientists, ML engineers, and data analysts all need to perform actions on data in a complex web of dependencies. With Airflow, you can orchestrate these actions and dependencies in a single platform, no matter which tools you are using and how complex your pipelines are.

<Frame>
  <img alt="Symbolic graph with Airflow shown as the center of the data ecosystem, with arrows pointing out from Airflow to logos of a variety of common data tools." />
</Frame>

Some common use cases of Airflow include:

* **ETL/ELT**: [86% of Airflow users](https://airflow.apache.org/blog/airflow-survey-2024/) use it for Extract-Transform-Load (ETL) and Extract-Load-Transfrom (ELT) patterns. Often, these pipelines support critical operational processes. See [Orchestrate dbt Core jobs with Airflow and Cosmos](/docs/learn/airflow-dbt) for an example use case.
* **Business operations**: 58% of Airflow users have used Airflow to orchestrate data supporting their business directly, creating data-powered applications and products, often in combination with MLOps pipelines. For an example use case, watch [The Laurel Algorithm: MLOps, AI, and Airflow for Perfect Timekeeping](https://www.astronomer.io/events/webinars/the-laurel-algorithm-mlops-ai-and-airflow-for-perfect-timekeeping-video/) webinar.
* **MLOps and GenAI**: 23% of Airflow users are already orchestrating Machine Learning Operations (MLOps) with Apache Airflow and 9% use Airflow specifically for GenAI use cases. An overview of best practices when using Airflow for MLOps can be found in [Best practices for orchestrating MLOps pipelines with Airflow](/docs/learn/airflow-mlops). See [Use Cohere and OpenSearch to analyze customer feedback in an MLOps pipeline](/docs/learn/2.x/use-case-llm-customer-feedback) for a complex use case involving advanced ML tools.
* **Managing infrastructure**: Airflow can be used to spin up and tear down infrastructure. For example, to create and delete temporary tables in a database or spin up and down a Spark cluster. This type of use case has been implemented by 18% of Airflow users. The [Airflow setup and teardown](/docs/learn/airflow-setup-teardown) guide shows how to use the most relevant feature for this use case.

Of course, these are just a few examples, you can orchestrate almost any kind of batch workflows with Airflow.

## Run Airflow

There are many ways to run Airflow, some of which are easier than others. Astronomer recommends:

* Using the open-source [**Astro CLI**](/docs/cli/v1.43/get-started-cli) to run Airflow locally. The Astro CLI is the easiest way to create a local Airflow instance running in containers and is free to use for everyone.
* Using [**Astro**](https://astronomer.io/try-astro) to run Airflow in production. Astro is a fully-managed SaaS application for data orchestration that helps teams write and run data pipelines with Apache Airflow at any level of scale. A [free trial](https://astronomer.io/try-astro) is available.

All Airflow installations include the mandatory Airflow components as part of their infrastructure: the api-server, scheduler, dag-processor, and metadata database. See [Airflow components](/docs/learn/airflow-components) for more information.

## Airflow concepts

To navigate Airflow resources, it is helpful to have a general understanding of the following Airflow concepts.

### Pipeline basics

* **DAG**: Directed Acyclic Graph. An Airflow DAG is a workflow defined as a graph, where all dependencies between nodes are directed and nodes don't self-reference, meaning there are no circular dependencies. For more information on Airflow DAGs, see [Introduction to Airflow DAGs](/docs/learn/dags).
* **DAG run**: The execution of a DAG at a specific point in time. A DAG run can be one of four different types: [`scheduled`](/docs/learn/scheduling-in-airflow), `manual`, [`dataset_triggered`](/docs/learn/airflow-datasets) or [`backfill`](/docs/learn/rerunning-dags).
* **Task**: A step in a DAG describing a single unit of work.
* **Task instance**: The execution of a task at a specific point in time.
* **Dynamic task**: An Airflow task that serves as a blueprint for a variable number of dynamically mapped tasks created at runtime. For more information, see [Dynamic tasks](/docs/learn/dynamic-tasks).
* **Asset**: An asset is a representation inside of Airflow of a real or abstract object that is created by a task. Assets can be files, tables, or not tied to a data object at all. For more information, see [Assets and data-aware scheduling in Airflow](/docs/learn/airflow-datasets).

The following screenshot shows a side by side comparison of the grid and graph of a simple DAG, called `example_astronauts`, with two tasks, `get_astronauts` and `print_astronaut_craft`. The `get_astronauts` task is a regular task, while the `print_astronaut_craft` task is a dynamic task. The grid view of the Airflow UI shows individual DAG runs and task instances, while the graph displays the structure of the DAG. You can learn more about the Airflow UI in [An introduction to the Airflow UI](/docs/learn/airflow-ui).

<Frame>
  <img alt="Screenshot of the Airflow UI Grid view with the Graph tab selected showing a DAG graph with a regular and a dynamic task as well as a DAG run and Task instance." />
</Frame>

<details>
  <summary>Click to view the full DAG code used for the screenshot</summary>

  ```python expandable wrap theme={null}
  """
  ## Astronaut ETL example DAG

  This DAG queries the list of astronauts currently in space from the
  Open Notify API and prints each astronaut's name and flying craft.
  There are two tasks, one to get the data from the API and save the results,
  and another to print the results. Both tasks are written in Python using
  Airflow's TaskFlow API, which allows you to easily turn Python functions into
  Airflow tasks, and automatically infer dependencies and pass data.

  The second task uses dynamic task mapping to create a copy of the task for
  each Astronaut in the list retrieved from the API. This list will change
  depending on how many Astronauts are in space, and the DAG will adjust
  accordingly each time it runs.

  For more explanation and getting started instructions, see our Write your
  first DAG tutorial: https://www.astronomer.io/docs/learn/get-started-with-airflow

  ![Picture of the ISS](https://www.esa.int/var/esa/storage/images/esa_multimedia/images/2010/02/space_station_over_earth/10293696-3-eng-GB/Space_Station_over_Earth_card_full.jpg)
  """

  from airflow import Dataset
  from airflow.decorators import dag, task
  from pendulum import datetime
  import requests


  # Define the basic parameters of the DAG, like schedule and start_date
  @dag(
      start_date=datetime(2024, 1, 1),
      schedule="@daily",
      catchup=False,
      doc_md=__doc__,
      default_args={"owner": "Astro", "retries": 3},
      tags=["example"],
  )
  def example_astronauts():
      # Define tasks
      @task(
          outlets=[Dataset("current_astronauts")]
      )  # Define that this task updates the `current_astronauts` Dataset
      def get_astronauts(**context) -> list[dict]:
          """
          This task uses the requests library to retrieve a list of Astronauts
          currently in space. The results are pushed to XCom with a specific key
          so they can be used in a downstream pipeline. The task returns a list
          of Astronauts to be used in the next task.
          """
          r = requests.get("http://api.open-notify.org/astros.json")
          number_of_people_in_space = r.json()["number"]
          list_of_people_in_space = r.json()["people"]

          context["ti"].xcom_push(
              key="number_of_people_in_space", value=number_of_people_in_space
          )
          return list_of_people_in_space

      @task
      def print_astronaut_craft(greeting: str, person_in_space: dict) -> None:
          """
          This task creates a print statement with the name of an
          Astronaut in space and the craft they are flying on from
          the API request results of the previous task, along with a
          greeting which is hard-coded in this example.
          """
          craft = person_in_space["craft"]
          name = person_in_space["name"]

          print(f"{name} is currently in space flying on the {craft}! {greeting}")

      # Use dynamic task mapping to run the print_astronaut_craft task for each
      # Astronaut in space
      print_astronaut_craft.partial(greeting="Hello! :)").expand(
          person_in_space=get_astronauts()  # Define dependencies using TaskFlow API syntax
      )


  # Instantiate the DAG
  example_astronauts()
  ```
</details>

It is a core best practice to keep tasks as atomic as possible, meaning that each task performs a single action. Additionally, tasks are idempotent, which means they produce the same output every time they are run with the same input. See [DAG writing best practices in Apache Airflow](/docs/learn/dag-best-practices).

### Write pipelines

There are two different ways to define pipelines in Airflow:

* **Task-oriented approach**: In the task-oriented approach, you define a DAG, fill it with tasks and define their dependencies. The mindset behind this approach is to think about the *actions* that need to be performed.
* **Asset-oriented approach**: Airflow 3.0 added the possibility to define DAGs in a more data-centric way. In the asset-oriented approach you define the asset (real or abstract) that you want to create directly and define dependencies between the [assets](/docs/learn/airflow-datasets). The mindset behind this approach is to think about the *assets* that need to be created.

#### Task-oriented approach

Airflow tasks are most commonly defined in Python code. You can define tasks using:

* **Decorators (`@task`)**: The [TaskFlow API](/docs/learn/airflow-decorators) allows you to define tasks by using a set of decorators that wrap Python functions. This is the easiest way to create tasks from existing Python scripts. Each call to a decorated function becomes one task in your DAG.
* **Operators (`XYZOperator`)**: [Operators](/docs/learn/what-is-an-operator) are classes abstracting over Python code designed to perform a specific action. You can instantiate an operator by providing the necessary parameters to the class. Each instantiated operator becomes one task in your DAG.

There are a couple of special types of operators that are worth mentioning:

* **Sensors**: [Sensors](/docs/learn/what-is-a-sensor) are Operators that keep running until a certain condition is fulfilled. For example, the [`HttpSensor`](https://airflow.apache.org/registry/providers/http#http-http-HttpSensor) waits for an HTTP request object to fulfill a user defined set of criteria.
* **Deferrable Operators**: [Deferrable Operators](/docs/learn/deferrable-operators) use the Python [asyncio](https://docs.python.org/3/library/asyncio.html) library to run tasks asynchronously. For example, the [DateTimeSensorAsync](https://airflow.apache.org/registry/providers/http#http-http-HttpOperator) waits asynchronously for a specific date and time to occur. Note that your Airflow environment needs to run a triggerer component to use deferrable operators.

Some commonly used building blocks, like the `BashOperator`, the `@task` decorator, or the `PythonOperator`, are part of core Airflow and automatically installed in all Airflow instances. Additionally, many operators are maintained separately to Airflow in **Airflow provider packages**, which group modules interacting with a specific service into a package.

You can browse all available operators and find detailed information about their parameters in the [Airflow Registry](https://airflow.apache.org/registry/). For many common data tools, there are [integration tutorials](/docs/learn/connections) available, showing a simple implementation of the provider package.

#### Asset-oriented approach

To define pipelines with using the asset-oriented approach, you can use the `@asset` decorator. See [Assets and data-aware scheduling in Airflow](/docs/learn/airflow-datasets) for more information.

### Additional concepts

While there is much more to Airflow than just DAGs and tasks, here are a few additional concepts and features that you are likely to encounter:

* **Airflow scheduling**: Airflow offers a variety of ways to schedule your DAGs. For more information, see [DAG scheduling and timetables in Airflow](/docs/learn/scheduling-in-airflow).
* **Airflow connections**: Airflow connections offer a way to store credentials and other connection information for external systems and reference them in your DAGs. For more information, see [Manage connections in Apache Airflow](/docs/learn/connections).
* **Airflow variables**: Airflow variables are key-value pairs that can be used to store information in your Airflow environment. For more information, see [Use Airflow variables](/docs/learn/airflow-variables).
* **XComs**: XCom is short for *cross-communication*, you can use XCom to pass information between your Airflow tasks. For more information, see [Passing data between tasks](/docs/learn/airflow-passing-data-between-tasks).
* **Airflow REST API**: The [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) allows you to interact with Airflow programmatically.

## Resources

* [Astronomer Webinars](https://www.astronomer.io/events/webinars/): Live deep dives into Airflow and Astronomer topics, all previous webinars are available to watch on-demand.
* [Astronomer Academy](https://academy.astronomer.io/): In depth video courses on Airflow and Astronomer topics.
* [Official Airflow Documentation](https://airflow.apache.org/docs/): The official documentation for Apache Airflow.
* [Airflow GitHub](https://github.com/apache/airflow): The official GitHub repository for Apache Airflow.
* [Airflow Slack](https://apache-airflow-slack.herokuapp.com/): The official Airflow Slack workspace, the best place to ask your Airflow questions!

## Next steps

Now that you have a basic understanding of Apache Airflow, you are ready to write your first DAG by following the [Get started with Apache Airflow](/docs/learn/get-started-with-airflow) tutorial.


# Introduction to data pipelines
Source: https://astronomer.io/docs/learn/intro-to-data-pipelines

Learn what data pipelines are, how they work, and why they matter. Covers pipeline components, types, and best practices for building reliable data workflows.

A *data pipeline* is a series of processes that move data from one or more sources to one or more destinations, applying transformations along the way. Pipelines range in complexity from a simple extract-and-load script to multi-stage workflows that clean, validate, enrich, and route data across systems.

Data pipelines exist because raw data rarely lives where it needs to be, in the format it needs to be in. Customer events arrive from APIs, sensor readings land in object storage, and transactional records sit in operational databases. Turning that raw data into something useful (a dashboard, a machine learning model, an analytics table) requires moving and transforming it through a defined sequence of steps.

## Components of a data pipeline

Most data pipelines share a common set of building blocks:

* **Sources**: Where data originates. Common sources include APIs, databases, message queues, flat files, and IoT sensors.
* **Destinations**: Where processed data lands. Destinations are typically data warehouses, data lakes, analytics platforms, or downstream applications.
* **Transformations**: The operations applied to data between source and destination. Transformations can include filtering, joining, aggregating, reformatting, or enriching data.
* **Orchestration**: The system that coordinates when and how each step runs, manages dependencies between steps, and handles failures. Without orchestration, pipeline steps run independently with no awareness of each other.
* **Monitoring and alerting**: The mechanisms that track pipeline health and notify teams when something fails or degrades. Monitoring is critical for catching issues before they propagate downstream.
* **Storage**: The intermediate and final systems where data is persisted at each stage. Storage choices affect pipeline performance, cost, and data retention.

## Common use cases

Data pipelines serve a wide range of use cases across data engineering, analytics, and AI:

* **ETL/ELT**: Extract data from source systems, transform it, and load it into a warehouse or data lake for analytics. This is the most common data pipeline pattern.
* **Data integration**: Consolidate data from multiple sources into a single system, resolving format differences and deduplicating records along the way.
* **MLOps**: Automate the end-to-end machine learning lifecycle, from data preparation and feature engineering through model training, validation, and deployment.
* **AI and LLM workflows**: Orchestrate retrieval-augmented generation (RAG) pipelines, embedding generation, fine-tuning jobs, and inference workflows that coordinate calls to AI services with data processing steps.
* **Reverse ETL**: Push processed data from a warehouse back into operational systems like CRMs, marketing platforms, or product databases.

## Types of data pipelines

Data pipelines are commonly categorized by how they process data. These categories aren't mutually exclusive; a single pipeline can combine batch and event-driven stages.

### Batch pipelines

Batch pipelines process data in discrete chunks. A batch pipeline might run every hour to pull new records from a database, transform them, and load them into a warehouse.

Batch pipelines don't have to run on fixed schedules. Modern orchestration tools support event-driven batch processing, where a pipeline runs in response to an external event such as a message in a queue, the arrival of a new file, or an update to an upstream dataset. The pipeline still processes data in a batch, but the trigger is an event rather than a clock.

Batch processing works well when:

* Data doesn't need to be available within milliseconds.
* The source system produces data in bulk or at known intervals.
* Transformations are compute-intensive and benefit from processing large volumes at once.
* External events can trigger pipeline runs when new data is available.

### Stream processing

Stream processing handles data continuously as individual records or small micro-batches arrive. Unlike batch processing, where data accumulates before being processed, stream processing systems ingest and act on each event as it occurs.

Stream processing works well when:

* Data must be available within seconds or less of being produced.
* Records arrive continuously and unpredictably from sources like event streams, message queues, or log systems.
* Downstream consumers depend on near-real-time updates, such as fraud detection, live dashboards, or alerting systems.

<Note>
  Batch and stream processing aren't an either-or choice. Many architectures use stream processing for low-latency needs and batch processing for heavier transformations, aggregations, or historical reprocessing, sometimes within the same pipeline.
</Note>

## Best practices

### Build incrementally

Start with a minimal pipeline that handles the core data flow, then add complexity as requirements become clearer. Trying to account for every edge case upfront leads to over-engineered pipelines that are harder to debug and maintain.

### Make pipelines modular

Break pipelines into discrete, reusable steps rather than writing monolithic scripts. Modular steps are easier to test individually, debug when they fail, and reuse across different pipelines.

### Define dependencies explicitly

Each step in a pipeline should declare what it depends on. Explicit dependencies ensure steps run in the correct order and that failures in upstream steps prevent downstream steps from running on bad data.

### Monitor and alert

Track pipeline runs, execution times, and data quality metrics. Set up alerts for failures, unexpected delays, and data anomalies. A pipeline that fails silently causes more damage than one that fails loudly.

### Use version control

Treat pipeline definitions as code. Store them in version control, review changes through pull requests, and maintain a history of what changed and why. This is especially important for pipelines defined programmatically rather than through a graphical interface.

### Use AI to accelerate development

AI-assisted tools can speed up pipeline development by generating boilerplate code, suggesting operators and connections, and helping debug failures. Tools like the [Astro IDE](/docs/astro/ide-overview), and Astronomer's open source [AI Agent tooling](https://github.com/astronomer/agents), which can be used with any AI you choose, provide context-aware code generation trained on Airflow best practices, so the generated code follows your project's patterns and is aware of your existing connections and configurations.

## Data pipelines and Apache Airflow

[Apache Airflow](/docs/learn/intro-to-airflow) is an open source platform for building and orchestrating data pipelines as code. In Airflow, a data pipeline is defined as a [Dag](/docs/learn/dags).

Airflow is well-suited for data pipeline orchestration because:

* **Pipelines as code**: You define pipelines in Python, which means you can use loops, conditionals, and variables to build dynamic workflows. Pipeline definitions live in version control alongside the rest of your codebase.
* **Dependency management**: Airflow enforces task execution order based on the dependencies you define. If an upstream task fails, downstream tasks don't run.
* **Scheduling**: Airflow includes a built-in scheduler that can run pipelines on cron-based schedules, event-driven triggers, or manual execution.
* **Visibility**: The Airflow UI provides a visual representation of your pipeline structure and execution history, making it straightforward to monitor runs and debug failures.
* **Extensibility**: Airflow integrates with a wide range of external systems through a library of pre-built operators and hooks for services like AWS, Google Cloud, Snowflake, Databricks, and more.

[Astro](https://www.astronomer.io/docs/astro) is a managed platform for running Airflow in production. Astro handles infrastructure management, provides [Astro Observe](/docs/astro/astro-observe) for monitoring pipeline health and observability, and includes the [Astro IDE](/docs/astro/ide-overview) for AI-assisted Dag development.

To get started with Airflow, see [Introduction to Apache Airflow](/docs/learn/intro-to-airflow). To learn how Airflow represents pipelines as Dags, see [Introduction to Dags](/docs/learn/dags).


# Learn Airflow 3
Source: https://astronomer.io/docs/learn/overview

Use tutorials and concepts to learn everything you need to know about Apache Airflow® 3 and Astro

<Info>
  [Apache Airflow®](https://airflow.apache.org/) 3 is here! Use our guides to learn everything you need to know about this new release.
</Info>

Airflow 3 brings highly requested new features that make it easier to use, more secure, and suitable for more use cases. Astronomer is here to help you get the most out of this next generation of Airflow, from upgrading, to implementing features like DAG versioning, backfills, and remote execution, to making sure you adhere to best practices. Read on for comprehensive guides on how to get started.

## New to Apache Airflow® and Astro?

<CardGroup>
  <Card title="Get started with Apache Airflow® 3 - Tutorial" icon="fan" href="/learn/get-started-with-airflow">
    Set up Airflow and run your first DAG in minutes with the Astro IDE.
  </Card>

  <Card title="Start your Astro trial" href="https://www.astronomer.io/dg/signup-airflow-3" icon="stars">
    Create a managed Airflow 3 Deployment with just a few clicks.
  </Card>
</CardGroup>

## Featured Airflow 3 concepts

<CardGroup>
  <Card title="Upgrade Airflow" href="/learn/airflow-upgrade-2-3">
    Upgrade from Airflow 2 to 3.
  </Card>

  <Card title="DAG versioning" href="/learn/airflow-dag-versioning">
    Track versions of your DAG code and maintain your development history in Airflow."
  </Card>

  <Card title="Backfills" href="/learn/rerunning-dags">
    Reprocess historical data from the UI, API, or CLI.
  </Card>

  <Card title="Assets" href="/learn/airflow-datasets">
    Use assets to define data-driven pipelines.
  </Card>

  <Card title="Event-driven scheduling" href="/learn/airflow-event-driven-scheduling">
    Run DAGs as soon as your data is updated.
  </Card>

  <Card title="Remote execution" href="/learn/airflow-executors-explained">
    Learn about remote execution capabilities in Airflow 3.
  </Card>
</CardGroup>


# AI context for data engineering
Source: https://astronomer.io/docs/learn/ai-context-for-data-engineering

Give AI tools the context they need to assist with Airflow data engineering.

An AI agent is only as useful as the context it has access to. Without any context engineering, your AI agent will only know what is in its training data and its Dag code tends to be generic, include bad practices, and likely use outdated versions of Airflow and Airflow providers.

This guide covers:

* Where local AI harnesses load context from automatically
* What you need to tell an agent about your Airflow environment before it starts writing Dags
* How to add the Airflow-specific skills that Astronomer maintains
* How to work with Otto, Astronomer's data engineering agent

<Tip>
  [Otto](/docs/astro/otto-overview), Astronomer's data engineering agent, has the best practices and Airflow-specific context described in this guide built-in, and additional advanced capabilities for local data engineering.

  To work with Otto locally, install the [Astro CLI](/docs/cli/v1.45/overview), sign in to your Astro account with `astro login` (a [free trial](https://www.astronomer.io/lp/signup/) is available), and then run `astro otto`.
</Tip>

<Info>
  This guide covers local AI agent harnesses. For context engineering for deployed AI agents, see the [AI Context Engineering with Apache Airflow®](https://www.astronomer.io/ebooks/ai-context-engineering-with-apache-airflow/) eBook.
</Info>

## Assumed knowledge

To get the most out of this guide, you should have:

* A local Airflow environment. See [Run Airflow locally](/docs/learn/run-airflow-locally).
* Basic familiarity with an AI coding agent harness, such as Claude Code or Cursor.

## Auto-loaded context

Most AI harnesses read certain files automatically when a session starts, before you type a prompt:

* **Global instructions**: A file at a fixed path that loads into every agent session on your computer, regardless of project. For Claude Code, this is `~/.claude/CLAUDE.md` by default.
* **Repository instructions**: A file at the root of your project, read as soon as an agent starts there. `AGENTS.md` is a cross-harness standard for this. See [agents.md](https://agents.md) for examples, and [`ruler`](https://github.com/intellectronica/ruler) to generate harness-specific config files from a single `AGENTS.md`.
* **Tool and MCP descriptions**: Available tool and MCP server descriptions load as soon as the harness starts, whether or not you use them. Each connected MCP server costs tokens for this reason, regardless of whether the agent calls any of its tools.

## Context you provide

Auto-loaded files contain global or repository-level information. Everything specific to the Dag(s) you're about to write you supply yourself, before instructing the agent to start writing the spec.

* **Your environment**: Your Airflow version, provider versions, executor, worker queue setup, and similar information. Frontier models train on years of publicly available Dag code, and their training data contains far more Airflow 2 than Airflow 3 code. A generic coding agent without explicit instructions can write code that is incompatible with your version.

* **Documentation for anything recent**: If your planned pipeline should use a feature added in a recent release, paste relevant sections of the Astro, Airflow, Airflow provider, and other package documentation into the prompt. Many websites have `llms.txt` or "View as Markdown" options to make it easier for you to copy the information. For Airflow providers it often makes sense to let your agent query the [Airflow Registry](https://airflow.apache.org/registry/api-explorer/) for details about provider packages. With the Astro CLI that's `astro api airflow`, which you can add to your agent's allowlist. See [Restrict agent commands](/docs/learn/run-airflow-locally#restrict-agent-commands).

  <Frame>
    <img alt="The Copy page menu on an Astronomer documentation page, with options to copy the page as Markdown for LLMs, view it as plain text, or open it in Claude or ChatGPT." />
  </Frame>

* **A reference Dag**: Keep a folder of *golden records*, reference Dags organized by use case or departments, depending on your organizational structure. If one or more of your planned Dags resembles a golden Dag, instruct your agent to inspect the reference Dag before writing the spec. Make sure to review your reference Dags periodically and update them with the latest Airflow features. Otto, Astronomer's data engineering agent, already has context on the Dags in your environment and uses them for reference patterns.

## Airflow-specific skills

Astronomer maintains the open-source [`astronomer/agents`](https://github.com/astronomer/agents) skills, covering Airflow operations, Dag authoring and testing, dbt, lineage, and data discovery.

For Claude Code:

```bash theme={null}
claude plugin marketplace add astronomer/agents
claude plugin install astronomer-data@astronomer
```

For Cursor:

```bash theme={null}
npx skills add astronomer/agents --skill '*' -a cursor
```

For any other AI coding agent:

```bash theme={null}
npx skills add astronomer/agents --skill '*'
```

## Otto

<Info>
  **Labs**

  Otto is in [Labs](/docs/astro/feature-previews).
</Info>

Otto is Astronomer's data engineering agent. It is a harness that already contains the best practices defined in this chapter, including the open-source `astronomer/agents` skills, and adds specialized capabilities such as a proprietary compatibility knowledge base drawn from Astronomer's experience running Airflow at scale. Otto can be used with the latest Anthropic, OpenAI, and Google Gemini models.

Use Otto to:

* Explore your Dags, Deployments, and data warehouse.
* [Author and debug Dags](/docs/learn/develop-dags-with-ai): describe what you need, and let Otto write the spec and the Dags, while iteratively testing them in your local Airflow environment.
* Investigate production failures using task logs, run history, Astro Deployment configurations, and Dag source.
* Review pull requests, with inline comments and commit suggestions.
* Plan and execute Airflow upgrades against Astronomer's compatibility knowledge base.
* Migrate workflows from legacy orchestration systems to Airflow on Astro.

To use Otto, sign in to your Astro account and run `astro otto` from your project folder:

```bash theme={null}
astro login
astro otto
```

See [Otto overview](/docs/astro/otto-overview) for more information.

### Delegate to Otto

Instead of running Otto interactively yourself, your AI coding harness can delegate Airflow-specific work directly to Otto, invoking `astro otto` in headless mode as a sub-agent and continuing after Otto returns a result. Ask your harness to `use Otto`, `ask Otto`, or `delegate to Otto`, and it hands off the task for you if it has access to the [`delegating-to-otto`](https://github.com/astronomer/agents/blob/main/skills/delegating-to-otto/SKILL.md) skill.

Headless invocation supports session continuity, permission modes, tool allowlists, model selection, structured output, and MCP configuration, so a harness can resume or reference a specific Otto session instead of starting fresh each time. See the [`astro otto`](/docs/cli/v1.44/astro-otto) reference for the full set of options.


# Advanced asset-based scheduling in Apache Airflow®
Source: https://astronomer.io/docs/learn/airflow-advanced-asset-scheduling

Using assets to implement DAG dependencies and scheduling in Airflow.

With Assets, Dags that access the same data can have explicit, visible relationships, and Dags can be scheduled based on updates to these assets. This feature helps make Airflow data-aware and expands Airflow scheduling capabilities beyond time-based methods such as cron.

The basics of asset-based scheduling, including fundamental concepts and terminology, are covered in [Basic asset-based scheduling in Apache Airflow®](/docs/learn/airflow-datasets). This guide covers advanced asset-based scheduling concepts.

In this guide, you'll learn:

* How to use conditional asset scheduling to schedule a Dag based on an asset expression.
* How to use combined asset and time-based scheduling to schedule a Dag based on both a time-based schedule (cron or any other [Timetable](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timetable.html)) plus whenever an asset expression is fulfilled.
* How to use partitioned asset schedules.
* How to attach extra information to, and retrieve extra information from, asset events.
* How to use asset aliases to create dynamic asset schedules.
* How to use asset listeners to run code when certain asset events occur anywhere in your Airflow instance.

<Tip>
  Assets can be used to schedule a Dag based on messages in a message queue. This sub-type of data-aware scheduling is called event-driven scheduling. See [Schedule Dags based on Events in a Message Queue](/docs/learn/airflow-event-driven-scheduling) for more information.
</Tip>

<Tip>
  Assets are a fundamental scheduling paradigm in Airflow. To learn more about when to use assets vs other scheduling paradigms, check out the free [Apache Airflow® orchestration paradigms eBook](https://www.astronomer.io/ebooks/apache-airflow-orchestration-paradigms).
</Tip>

<Note>
  This guide covers data-aware schedules with Assets, for more information on the `@asset` decorator shorthand see [@asset syntax in Apache Airflow®](/docs/learn/airflow-asset-decorator).
</Note>

## Assumed knowledge

To get the most out of this guide, you should have an existing knowledge of:

* Airflow scheduling concepts. See [Schedule Dags in Airflow](/docs/learn/scheduling-in-airflow).
* Basic asset-based scheduling. See [Basic asset-based scheduling in Apache Airflow®](/docs/learn/airflow-datasets).

## Advanced asset concepts

When using asset-based scheduling, updates are produced to assets, creating **asset events**. These updates can be created by different [methods](/docs/learn/airflow-datasets#updating-an-asset), the most common one being to set the `outlets` parameter of a task to a list of assets to update upon successful completion. Dags can be scheduled based on asset events created for one or more assets, and tasks can be given access to all events attached to an asset by defining the asset as one of their `inlets`.

In addition to these fundamental concepts covered in the [Basic asset-based scheduling in Apache Airflow®](/docs/learn/airflow-datasets) guide, when using advanced techniques for asset-based scheduling, you should understand the following terms:

* **Asset expression**: a logical expression using AND (`&`) and OR (`|`) operators to define the schedule of a Dag scheduled on updates to several assets.
* **AssetAlias**: an object that can be associated with one or more assets and used to create schedules based on assets created at runtime, see [Asset aliases](#asset-aliases).
* **Metadata**: a class to attach `extra` information to an asset event from within the producer task. This functionality can be used to pass asset event-related metadata between tasks, see [Attach extra information](#attach-extra-information) and [Retrieve extra information](#retrieve-extra-information).
* **AssetWatcher**: a class that is used in [event-driven scheduling](/docs/learn/airflow-event-driven-scheduling) to watch for a `TriggerEvent` caused by a message in a message queue.
* **Queued asset event**: It is common to have Dags scheduled to run as soon as a set of assets have received at least one update each. While there are still asset events missing to trigger the Dag, all asset events for other assets the Dag is scheduled on are queued asset events. A queued asset event is defined by its asset, timestamp and the Dag it is queuing for. One asset event can create a queued asset event for several Dags. You can access queued asset events for a specific Dag or a specific asset programmatically, using the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_asset_queued_events).

The **Assets** tab in the [Airflow UI](/docs/learn/airflow-ui) provides a list of active assets in your Airflow environment with an asset graph for each asset showing its dependencies to Dags and other assets, see [Asset graph](/docs/learn/airflow-datasets#asset-graph) for more information.

<Note>
  Airflow is only aware of updates to assets that occur by tasks, API calls, or in the Airflow UI, see [Methods to update an asset](/docs/learn/airflow-datasets#updating-an-asset). It doesn't monitor updates to assets that occur outside of Airflow. For example, Airflow won't notice if you manually add a file to an S3 bucket referenced by an asset. See [When not to use Airflow assets](/docs/learn/airflow-datasets#when-not-to-use-airflow-assets) for more information.
</Note>

<Note>
  Assets events are only registered by Dags or listeners in the same Airflow environment. If you want to create cross-Deployment dependencies with Assets you will need to use the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) to create an asset event in the Airflow environment where your downstream Dag is located. See the [Cross-deployment dependencies](/docs/astro/best-practices/cross-deployment-dependencies#assets-example) for an example implementation on Astro.
</Note>

## Conditional asset scheduling

When using [basic asset-based scheduling](/docs/learn/airflow-datasets), you can schedule a Dag based on one or more assets by providing a list of assets to the `schedule` parameter. If you provide more than one asset, the Dag will run when all of the assets have received at least one update each.

For more complex scheduling needs, you can use conditional asset scheduling to schedule a Dag based on an asset expression.

An asset expression is a logical expression using AND (`&`) and OR (`|`) operators to define the schedule of a Dag scheduled on updates to several assets. The asset expression is given to the `schedule` parameter, wrapped in `()`.

For example, to schedule a Dag on an update to *either* `asset1`, `asset2`, `asset3`, or `asset4`, you can use the following syntax.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import Asset, dag

  @dag(
      schedule=(
          Asset("asset1")
          | Asset("asset2")
          | Asset("asset3")
          | Asset("asset4")
      ),  # Use () instead of [] to be able to use conditional asset scheduling!
  )
  def downstream1_on_any():

      # your tasks here

  downstream1_on_any()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.sdk import Asset, DAG

  with DAG(
      dag_id="downstream1_on_any",
      schedule=(
          Asset("asset1")
          | Asset("asset2")
          | Asset("asset3")
          | Asset("asset4")
      ),  # Use () instead of [] to be able to use conditional asset scheduling!
  ):

      # your tasks here
  ```
</details>

The `downstream1_on_any` Dag is triggered whenever *any* of the assets `asset1`, `asset2`, `asset3`, or `asset4` are updated. The schedule of the Dag is listed as **x of 4 Assets updated** in the Airflow UI, you can see the asset expression that defines the schedule as a pop up when clicking on the schedule.

<Frame>
  <img alt="Screenshot of the Airflow UI with a pop up showing the asset expression for the downstream1_on_any Dag listing the 4 assets under &#x22;any&#x22;" />
</Frame>

You can also combine the logical operators to create more complex expressions. For example, to schedule a Dag on an update to either `asset1` or `asset2` and either `asset3` or `asset4`, you can use the following syntax:

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import Asset, dag

  @dag(
      schedule=(
          (Asset("asset1") | Asset("asset2"))
          & (Asset("asset3") | Asset("asset4"))
      ),  # Use () instead of [] to be able to use conditional asset scheduling!
  )
  def downstream2_one_in_each_group():

      # your tasks here

  downstream2_one_in_each_group()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.sdk import Asset, DAG

  with DAG(
      dag_id="downstream2_one_in_each_group",
      schedule=(
          (Asset("asset1") | Asset("asset2"))
          & (Asset("asset3") | Asset("asset4"))
      ),  # Use () instead of [] to be able to use conditional asset scheduling!
  ):

      # your tasks here
  ```
</details>

## Combined asset and time-based scheduling

You can combine asset-based scheduling with time-based scheduling with the `AssetOrTimeSchedule` timetable. A Dag scheduled with this timetable will run either when its `timetable` condition is met or when its `asset` condition is met. You can use any [Timetable](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timetable.html) in the `timetable` parameter.

The following Dag runs on a time-based schedule defined by the `0 0 * * *` cron expression, which is every day at midnight. The Dag also runs when either `asset3` or `asset4` is updated.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import Asset, dag, task
  from pendulum import datetime
  from airflow.timetables.assets import AssetOrTimeSchedule
  from airflow.timetables.trigger import CronTriggerTimetable

  @dag(
      start_date=datetime(2025, 3, 1),
      schedule=AssetOrTimeSchedule(
          timetable=CronTriggerTimetable("0 0 * * *", timezone="UTC"),
          assets=(Asset("asset3") | Asset("asset4")),
          # Use () instead of [] to be able to use conditional asset scheduling!
      )
  )
  def toy_downstream3_asset_and_time_schedule():

      # your tasks here

  toy_downstream3_asset_and_time_schedule()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.sdk import Asset, DAG
  from pendulum import datetime
  from airflow.timetables.assets import AssetOrTimeSchedule
  from airflow.timetables.trigger import CronTriggerTimetable

  with DAG(
      dag_id="toy_downstream3_asset_and_time_schedule",
      start_date=datetime(2024, 3, 1),
      schedule=AssetOrTimeSchedule(
          timetable=CronTriggerTimetable("0 0 * * *", timezone="UTC"),
          assets=(Asset("asset3") | Asset("asset4")),
          # Use () instead of [] to be able to use conditional asset scheduling!
      )
  ):
      # your tasks here

  ```
</details>

## Partitioned asset schedules

Airflow 3.2 introduced the concept of [partitioned Dag runs and partitioned asset events](/docs/learn/airflow-partitioned-runs), which have a `partition_key` attached to them. The partition key can be used in tasks in a partitioned Dag run to partition data, for example in a SQL statement.

To schedule a Dag based on *partitioned* asset events, you set its `schedule` parameter to an instance of `PartitionedAssetTimetable`.

```python wrap theme={null}
from airflow.sdk import dag, PartitionedAssetTimetable, Asset

@dag(schedule=PartitionedAssetTimetable(assets=Asset("my_partitioned_asset")))
```

This Dag will only be triggered when the `my_partitioned_asset` is updated by a *partitioned* asset event, not by regular asset events.
You can modify the partition key by providing a `partition_key_mapper` to the `PartitionedAssetTimetable` instance, for example to change the time grain of the partition key to daily or weekly.

There are three ways to create a partitioned asset event:

* By updating an asset manually in the Airflow UI and providing a `partition_key` in the asset event creation dialog.
* By updating an asset using the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/create_asset_event) and providing a `partition_key` in the request body.
* By updating an asset using the `outlets` parameter of a task in a Dag that is scheduled using a `CronPartitionTimetable` timetable.

For more information on partitioned Dag runs and asset events, see the [Partitioned Dag runs and asset events in Apache Airflow®](/docs/learn/airflow-partitioned-runs) guide.

<Note>
  Partitioned asset events created by a task in a Dag using the `CronPartitionTimetable` timetable are intended for partition-aware downstream scheduling, and don't trigger non-partition-aware Dags.
</Note>

## Asset event extras

You can attach extra information to an asset event and retrieve it in downstream tasks. This is useful for passing metadata between tasks, including tasks located in different Dags, for example information about the asset event you are working with.

Asset event extras are attached to individual asset events. If you want to attach information to the asset itself, see [asset state store](#asset-state-store).

### Attach extra information

When updating an asset in the Airflow UI or making a [`POST` request](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/create_asset_event) to the Airflow REST API, you can attach extra information to the asset event by providing an `extra` JSON payload.

You can add extra information from within the producing task using either the `Metadata` class or accessing `outlet_events` from the Airflow context. The information passed needs to be JSON serializable.

To use the `Metadata` class to attach information to an asset, follow the example in the code snippet below. Make sure that the asset used in the metadata class is also defined as an outlet in the producer task.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # from airflow.sdk import Asset, Metadata, task

  my_asset_1 = Asset("x-asset1")

  @task(outlets=[my_asset_1])
  def attach_extra_using_metadata():
      num = 23
      yield Metadata(my_asset_1, {"myNum": num})

      return "hello :)"

  attach_extra_using_metadata()

  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # from airflow.providers.standard.operators.python import PythonOperator
  # from airflow.sdk import Asset, Metadata

  my_asset_1 = Asset("x-asset1")

  def attach_extra_using_metadata_func():
      num = 23
      yield Metadata(my_asset_1, {"myNum": num})

      return "hello :)"

  attach_extra_using_metadata = PythonOperator(
      task_id="attach_extra_using_metadata",
      python_callable=my_function,
      outlets=[my_asset_1]
  )

  ```
</details>

You can also access the `outlet_events` from the Airflow context directly to add an extra dictionary to an asset event.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import Asset, Metadata, task

  my_asset_2 = Asset("x-asset2")

  @task(outlets=[my_asset_2])
  def use_outlet_events(outlet_events): # outlet_events is pulled out of the Context
      num = 19
      outlet_events[my_asset_2].extra = {"my_num": num}

      return "hello :)"

  use_outlet_events()

  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # from airflow.providers.standard.operators.python import PythonOperator
  # from airflow.sdk import Asset, Metadata

  my_asset_2 = Asset("x-asset2")

  def attach_extra_using_metadata_func():
      num = 19
      context["outlet_events"][my_asset_2].extra = {"my_num": num}

      return "hello :)"

  attach_extra_using_metadata = PythonOperator(
      task_id="attach_extra_using_metadata",
      python_callable=my_function,
      outlets=[my_asset_2]
  )

  ```
</details>

Asset extras can be viewed in the Airflow UI in the asset graph of an asset.

<Frame>
  <img alt="Screenshot of the asset extra information." />
</Frame>

### Retrieve extra information

Extras attached to asset events can be programmatically retrieved from within Airflow tasks. Any Airflow task instance in a Dag run has access to the list of assets that were involved in triggering that specific Dag run. Additionally, you can give any Airflow task access to all asset events of a specific asset by providing the asset to the task's `inlets` parameter. Defining inlets doesn't affect the schedule of the Dag.

To access all asset events that were involved in triggering a Dag run within a TaskFlow API task, you can pull `triggering_asset_events` from the [Airflow context](/docs/learn/airflow-context). In a traditional operator, you can use [Jinja templating](/docs/learn/templating) in any templateable field of the operator to access information in the Airflow context.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # from airflow.sdk import task

  @task
  def get_extra_triggering_run(triggering_asset_events):
      # triggering_asset_events - all events that triggered this specific Dag run, and is pulled from the Context
      # the loop below wont run if the Dag is manually triggered
      for asset, asset_list in triggering_asset_events.items():
          print(asset, asset_list)
          print(asset_list[0].extra)
          # you can also fetch the run_id and other information about the upstream Dags
          print(asset_list[0].source_run_id)
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # from airflow.operators.bash import BashOperator

  get_extra_triggering_run_bash = BashOperator(
      task_id="get_extra_triggering_run_bash",
      # This statement errors when there are no triggering events, for example in a manual Dag run!
      bash_command="echo {{ (triggering_asset_events.values() | first | first).extra }} ", 
  )
  ```
</details>

If you want to access asset extras independently from which asset events triggered a Dag run, you have the option to directly provide an asset to a task as an inlet. In a TaskFlow API task you can fetch the `inlet_events` from the [Airflow context](/docs/learn/airflow-context).

```python wrap theme={null}
# from airflow.sdk import Asset, task

my_asset_2 = Asset("x-asset2")

# note that my_asset_2 does not need to be part of the Dags schedule
# you can provide as many inlets as you wish
@task(inlets=[my_asset_2])
def get_extra_inlet(inlet_events):  # inlet_events is pulled out of the Context
    # inlet_events are listed earliest to latest by timestamp
    asset_events = inlet_events[my_asset_2]
    # protect against the asset not existing
    if len(asset_events) == 0:
        print(f"No asset_events for {my_asset_2.uri}")
    else:
        # accessing the latest asset event for this asset
        # if the extra does not exist, return None
        my_extra = asset_events[-1].extra
        print(my_extra)

get_extra_inlet()

```

## Asset state store

The asset state store, added in Airflow 3.3, lets you store key-value pairs attached to assets. Stored values must be JSON-serializable.

Tasks in any Dag can access information in the asset state store of any asset given to the task in its `inlets` or `outlets`.

Use the asset state store when you want to persist a small piece of information for an asset across Dag runs, such as a watermark, a cursor, or the ID of the last processed record.

Asset state stores are attached to assets. If you want to attach information to individual asset events, see [asset event extras](#asset-event-extras).

To add an asset state store entry from within a Dag, access the `asset_state_store` from the Airflow context and `.set` a new entry:

```python wrap theme={null}
from airflow.sdk import task, Asset

my_asset = Asset("my_asset")


@task(outlets=[my_asset])
def write_state(asset_state_store=None):  # retrieves context["asset_state_store"] directly
    asset_state_store[my_asset].set("watermark", "2026-06-22T00:00:00")

write_state()
```

Similarly, you can use `.get` to read information from the `asset_state_store`:

```python wrap theme={null}
from airflow.sdk import task, Asset

my_asset = Asset("my_asset")


@task(inlets=[my_asset])
def read_state(asset_state_store=None):
    watermark = asset_state_store[my_asset].get("watermark", default=None)

read_state()
```

You can delete information from the asset state store by using the `.delete(key)` and `.clear()` methods.

```python wrap theme={null}
from airflow.sdk import task, Asset

my_asset = Asset("my_asset")


@task(inlets=[my_asset])
def read_state(asset_state_store=None):
    asset_state_store[my_asset].delete("watermark")
    asset_state_store[my_asset].clear()  # remove all key-value pairs from the asset

read_state()
```

For a task with exactly one concrete inlet or outlet, you can call the methods directly on `asset_state_store` without needing to provide the specific asset object.

```python wrap theme={null}
@task(inlets=[my_asset])
def read_state(asset_state_store=None):
    watermark = asset_state_store.get("watermark")
```

You can also view and modify asset state store entries in the Airflow UI. Go to **Assets**, select an asset, then open the **Asset State Store** tab.

<Frame>
  <img alt="Airflow UI Asset page showing the Asset State Store tab for an asset with a my_num key, its value, and controls to add, clear, edit, and delete entries" />
</Frame>

Endpoints to get, set, delete, and clear entries in the asset state store are available in the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html).

In contrast to [task state store](/docs/learn/airflow-task-state-store) entries, asset state store entries don't have a retention limit.

## Asset aliases

You have the option to create asset aliases to schedule Dags based on assets with names generated at runtime. An asset alias is defined by a unique `name` string and can be used in place of a regular asset in `outlets` and `schedules`. Any number of asset events updating different assets can be attached to an asset alias.

There are two ways to add an asset event to an asset alias:

* Using the `Metadata` class.
* Using `outlet_events` pulled from the Airflow context.

See the code below for examples, note how the name of the asset is determined at runtime inside the producing task.

<details>
  <summary>Metadata</summary>

  ```python wrap theme={null}
  # from airflow.sdk import Asset, AssetAlias, Metadata, task

  my_alias_name = "my_alias"

  @task(outlets=[AssetAlias(my_alias_name)])
  def attach_event_to_alias_metadata():
      bucket_name = "my-bucket"  # determined at runtime, for example based on upstream input
      yield Metadata(
          asset=Asset(f"updated_{bucket_name}"),
          extra={"k": "v"},  # extra has to be provided, can be {}
          alias=AssetAlias(my_alias_name),
      )

  attach_event_to_alias_metadata()

  ```
</details>

<details>
  <summary>Context</summary>

  ```python wrap theme={null}
  # from airflow.sdk import Asset, AssetAlias, Metadata, task

  my_alias_name = "my_alias"

  @task(outlets=[AssetAlias(my_alias_name)])
  def attach_event_to_alias_context(outlet_events): # outlet_events is pulled out of the Context
      bucket_name = "my-other-bucket"   # determined at runtime, for example based on upstream input
      outlet_events[AssetAlias(my_alias_name)].add(
          Asset(f"updated_{bucket_name}"), extra={"k": "v"}
      )  # extra is optional

  attach_event_to_alias_context()

  ```
</details>

In the consuming Dag you can use an asset alias in place of a regular asset.

```python {6} wrap theme={null}
from airflow.sdk import AssetAlias, dag
from airflow.providers.standard.operators.empty import EmptyOperator

my_alias_name = "my_alias"

@dag(schedule=[AssetAlias(my_alias_name)])
def my_consumer_dag():

    EmptyOperator(task_id="empty_task")

my_consumer_dag()

```

Once the `my_producer_dag` containing the `attach_event_to_alias_metadata` task completes successfully, reparsing of all Dags scheduled on the asset alias `my_alias` is automatically triggered. This reparsing step attaches the `updated_{bucket_name}` asset to the `my_alias` asset alias and the schedule resolves, triggering one run of the `my_consumer_dag`.

Any further asset event for the `updated_{bucket_name}` asset will now trigger the `my_consumer_dag`. If you attach asset events for several assets to the same asset alias, a Dag scheduled on that asset alias will run as soon as any of the assets that were ever attached to the asset alias receive an update.

See [Scheduling based on asset aliases](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/asset-scheduling.html#scheduling-based-on-asset-aliases) for more information and examples of using asset aliases.

To use asset aliases with traditional operators, you need to attach the asset event to the alias inside the operator logic. If you are using operators besides the `PythonOperator`, you can either do so in a custom operator's `.execute` method or by passing a `post_execute` callable to existing operators ([experimental](https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features)). Use `outlet_events` when attaching asset events to aliases in traditional or custom operators. Note that for [deferrable operators](/docs/learn/deferrable-operators), attaching an asset event to an alias is only supported in the `execute_complete` or `post_execute` method.

```python wrap theme={null}
def _attach_event_to_alias(context, result):  # result = the return value of the execute method
    # use any logic to determine the URI
    uri = "s3://my-bucket/my_file.txt"
    context["outlet_events"][AssetAlias(my_alias_name)].add(Asset(uri))  

BashOperator(
    task_id="t2",
    bash_command="echo hi",
    outlets=[AssetAlias(my_alias_name)],
    post_execute=_attach_event_to_alias,  # using the post_execute parameter is experimental
)
```

<details>
  <summary>Click to view an example of a custom operator attaching an asset event to an asset alias.</summary>

  ```python expandable wrap theme={null}
  """
  ### AssetAlias in a custom operator
  """

  from airflow.sdk import Asset, AssetAlias, dag
  from airflow.sdk.bases.operator import BaseOperator
  import logging

  t_log = logging.getLogger("airflow.task")

  my_alias_name = "my-alias"


  # custom operator producing to an asset alias
  class MyOperator(BaseOperator):
      """
      Simple example operator that attaches an asset event to an asset alias.
      :param my_bucket_name: (str) The name of the bucket to use in the asset name.
      """

      # define the .__init__() method that runs when the DAG is parsed
      def __init__(self, my_bucket_name, my_alias_name, *args, **kwargs):
          # initialize the parent operator
          super().__init__(*args, **kwargs)
          # assign class variables
          self.my_bucket_name = my_bucket_name
          self.my_alias_name = my_alias_name

      def execute(self, context):

          # add your custom operator logic here

          # use any logic to derive the dataset URI
          my_asset_name = f"updated_{self.my_bucket_name}"
          context["outlet_events"][AssetAlias(self.my_alias_name)].add(
              Asset(my_asset_name)
          )

          return "hi :)"

      # define the .post_execute() method that runs after the execute method (optional)
      # result is the return value of the execute method
      def post_execute(self, context, result=None):
          # write to Airflow task logs
          self.log.info("Post-execution step")

          # It is also possible to add events to the alias in the post_execute method


  @dag
  def asset_alias_custom_operator():

      MyOperator(
          task_id="t1",
          my_bucket_name="my-bucket",
          my_alias_name=my_alias_name,
          outlets=[AssetAlias(my_alias_name)],
      )


  asset_alias_custom_operator()
  ```
</details>

## Asset listeners

A listener is a type of [Airflow plugin](/docs/learn/using-airflow-plugins) that can be used to run custom code when certain events occur anywhere in your Airflow instance. There are four listener hooks relating to asset events:

* `on_asset_created`: runs when a new asset is created.
* `on_asset_alias_created`: runs when a new asset alias is created.
* `on_asset_changed`: runs when any asset change occurs.
* `on_asset_event_emitted`: runs when an asset event is emitted, generally called together with `on_asset_changed`.

To implement a listener, you need to create a `@hookimpl`-decorated function for your listener hook of choice and then register them in an Airflow plugin.

```python expandable wrap theme={null}

from airflow.plugins_manager import AirflowPlugin
from airflow.listeners.types import AssetEvent
from airflow.serialization.definitions.assets import SerializedAsset, SerializedAssetAlias
from airflow.listeners import hookimpl


@hookimpl
def on_asset_created(asset: SerializedAsset):
    """Execute when a new asset is created."""


@hookimpl
def on_asset_alias_created(asset_alias: SerializedAssetAlias):
    """Execute when a new asset alias is created."""


@hookimpl
def on_asset_changed(asset: SerializedAsset):
    """Execute when asset change is registered."""


@hookimpl
def on_asset_event_emitted(asset_event: AssetEvent):
    """
    Execute when an asset event is emitted.

    This is generally called together with ``on_asset_changed``, but with
    information on the emitted event instead.
    """


class MyListenerPlugin(AirflowPlugin):
    name = "my_listener_plugin"

    listeners = [
        on_asset_created,
        on_asset_alias_created,
        on_asset_changed,
        on_asset_event_emitted,
    ]
```

See the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/listeners.html) for more information on listeners.


# Airflow cluster policies
Source: https://astronomer.io/docs/learn/airflow-advanced-cluster-policies

Learn about everything you need to use the Apache Airflow cluster policies.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

Cluster policies are sets of rules that Airflow administrators can define to mutate or perform custom logic on a few important Airflow objects:

* DAG
* Task
* Task Instance
* Pod
* Airflow Context variables

These policies allow administrators to centrally manage how Airflow users or DAG writers are interacting with the Airflow Cluster. Cluster policies can modify or restrict a user's capability based on organization policy or ensure that users conform to common standards. For example, you might want to ensure all DAGs have `tags` in your production Airflow environment.

<Info>
  Unlike Airflow Plugins, Cluster Policies aren't visible in the Airflow UI. Since end users lack visibility into the installed Cluster Policies, Astronomer recommends implementing logging every time a policy modifies an Airflow object to inform users of the change.
</Info>

Here are some common use cases for cluster policies:

* Enforce Task or DAG-level retries
* Verify a DAG's `catchup` parameter based on production or development environment
* Limit the resources requested by a `KubernetesPodOperator`
* Routing critical jobs to a specific Celery `queue` or Airflow `pool`
* Add missing `tags` or `owner` emails

In this guide, you'll learn about the types of cluster policies, how the policies work, and how to implement them in Airflow.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* The Astro CLI. See [Get started with Astro CLI](/docs/cli/v1.43/get-started-cli)

## Types of cluster policies

You can use four types of cluster policies in Airflow:

* DAG policy: This policy is applicable to a DAG object, and takes a DAG object `dag` as a parameter.
* Task policy: This policy is applicable to a Task object.
* Task Instance policy: This policy is applicable to a Task Instance, which is an instance of a Task object and is created at run time.
* Pod policy: This policy is applicable to a Kubernetes Pod launched by `KubernetesPodOperator` or `KubernetesExecutor` at runtime.

## How cluster policies work

<Frame>
  <img alt="Cluster policies" />
</Frame>

Any attributes defined using cluster policies take precedence over the attributes defined in your DAG or Task. Once implemented, if a DAG or task isn't compliant with your set policies, the policy will raise the `AirflowClusterPolicyViolation` exception and the DAG won't be loaded. The Airflow web UI displays this exception as an `import error`.

You can also use the `AirflowClusterPolicySkipDag` exception to skip a DAG. For example, you may want to skip month-end DAGs from daily processing or skip any DAGs with the wrong environment tag. Another possible use case could be when you are migrating from a deprecated source system to a new source system. You might want to skip the old DAGs to avoid any failures and alerts. Note that this exception won't be displayed on the Airflow web UI.

### DAG policy

The DAG policy allows you to overwrite or reconfigure a DAG’s parameters based on the criteria you set. You can implement this using `dag_policy` function. It runs at the time the DAG is loaded from the `DagBag`. It allows you to:

* Mutate a DAG object after it is loaded in the `DagBag`.
* Run code after your DAG has been fully generated.

This means that the DAG processor still parses all DAG files even if you skip one using a DAG policy.

Some example implementations include:

* Enforcing a default owner for your DAGs.
* Enforcing certain tags for DAGs, either default or based on conditions.
* Ensuring development DAGs don't run in production.
* Stopping a DAG from being executed by raising an `AirflowClusterPolicyViolation` exception.

Note that the `dag_policy` is applied before the `task_policy` and after the DAG has been completely loaded. Hence, overriding the `default_args` parameter has no effect using `dag_policy`. If you want to override the default operator settings, use task policies instead.

#### Example

```python wrap theme={null}
@hookimpl
def ensure_dags_are_tagged(dag: "DAG") -> None:
    tag_labels = [tag.split(":")[0]
    for tag in dag.tags]
        if not "Owner" in tag_labels:
            raise AirflowClusterPolicyViolation(f"{dag.dag_id} does not have a 'Owner' tag defined.")
```

### Task policy

A task policy allows you to overwrite or reconfigure a task’s parameters. You can implement this using `task_policy` function. It gets executed when the task is created during parsing of the task from `DagBag` at load time and mutates tasks after they have been added to a DAG. This means that the whole task definition can be altered in the task policy. It doesn't relate to a specific task running in a `DagRun`. The `task_policy` defined is applied to all the task instances that will be executed in the future. It expects a `BaseOperator` as a parameter.

Some example implementations include:

* Enforcing a task timeout policy.
* Using a different environment for different operators.
* Overriding a `on_success_callback` or `on_failure_callback` for a task.

#### Example

```python wrap theme={null}
@hookimpl
def task_policy(task: "BaseOperator") -> None:
    min_timeout = datetime.timedelta(hours=24)
    if not task.execution_timeout or task.execution_timeout > min_timeout:
        raise AirflowClusterPolicyViolation(f"{task.dag.dag_id}:{task.task_id} time out is greater than {min_timeout}")
```

### Task Instance policy

<Info>
  If you are on Airflow version `2.9.1` or lower, you might see some inconsistencies in the application of `task_instance_mutation_hook`. This was fixed in Airflow `2.9.2`.
</Info>

A Task Instance policy allows you to alter task instances before the Airflow scheduler queues them. You can implement a Task Instance policy using the function `task_instance_mutation_hook`. This is different from the `task_policy` function, which inspects and mutates tasks “as defined”. By contrast, task instance policies inspect and mutate task instances before execution. It takes a TaskInstance object, `task_instance`, as a parameter. This policy applies not to a task but to the instance of a task that relates to a particular `DagRun`. It is only applied to the currently executed run (in other words, instance) of that task. The policy is applied to a task instance in an Airflow worker before the task instance is executed, not in the DAG file processor.

Some example implementations include:

* Enforcing a specific queue for certain Airflow Operators.
* Modifying a task instance between retries.

#### Example

```python wrap theme={null}
@hookimpl
def task_instance_mutation_hook(task_instance:TaskInstance):
    if task_instance.try_number >= 3:
        task_instance.queue = "big-machine"
```

<Tip>
  Note that since Airflow determines priority weight dynamically using weight rules, you can't alter the `priority_weight` of a task instance within the Task Instance mutation hook.
</Tip>

### Pod policy

This policy is applicable to Kubernetes Pod created at runtime when using the `KubernetesPodOperator` or `KubernetesExecutor`. You can implement this policy using `pod_mutation_hook` function. This is a policy function that allows altering a `kubernetes.client.models.V1Pod` object before Airflow passes it to the Kubernetes client for scheduling. It takes a Pod object `pod` as a parameter. Note that this cluster policy is available only from Airflow version `2.6`.

For instance, one could use this to alter the resources for a Pod or to add sidecar or init containers to every worker pod launched. Astro, however, doesn't allow adding init or sidecar containers. [Astro](/docs/astro/deployment-metrics) provides advanced logging, metrics collection, and multiple ways to manage your environment without the need to run separate containers to collect stats or apply environment settings.

Some example implementations include:

* Setting resource requests and limits.
* Increasing resources assigned to a Pod.

#### Example

```python wrap theme={null}
from kubernetes.client import models as k8s
from airflow.policies import hookimpl

@hookimpl
def pod_mutation_hook(pod) -> None:
    print("hello from pod_mutation_hook ",type(pod))

    resources = k8s.V1ResourceRequirements(
        requests={
            "cpu": "100m",
            "memory": "256Mi",
        },
        limits={
            "cpu": "1000m",
            "memory": "1Gi",
        },
    )
    pod.spec.containers[0].resources = resources
```

## Implementation

In this section, we describe how to use `pluggy` to implement cluster policies for an Airflow project using Astro CLI. `pluggy` is useful for plugin management, allowing you to have multiple implementations of the policy functions.

Note that the `pluggy` method is available only in Airflow version 2.6 and above. For versions lower than 2.6, a similar implementation is possible using the `config/airflow_local_settings.py` file in your `$AIRFLOW_HOME`. You can define your policies within this file. There is no need to build or install any package when you use the `airflow_local_settings.py` file. However, on Astro, you can only implement policies using the `pluggy` interface.

### Step 1: Create a package for your policies

The simplest way to implement cluster policies is to build a package for them that you apply to your Airflow environment. You can add this package to the `plugins` folder of your Astro project and install it by [customizing your `Dockerfile`](/docs/cli/v1.43/customize-dockerfile). This method uses `setuptools` entrypoint for your project. You can read more about Python packaging [here](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/).

For example, you can create a package `plugins` with the following structure:

```bash wrap theme={null}
plugins/
│
├── src/
│   └── policy_plugin/
│       ├── __init__.py
│       └── policy.py
│
├── pyproject.toml
└── README.md

```

1. In the `pyproject.toml` file, add the following:

   ```bash wrap theme={null}
   [build-system]
   requires = ["setuptools >= 61.0"]
   build-backend = "setuptools.build_meta"

   [project]
   name = "policy_plugin"
   version = "0.3.0"
   dependencies = ["apache-airflow>=2.6"]
   requires-python = ">=3.8"
   description = "Airflow cluster policy"

   [project.entry-points.'airflow.policy']
   _ = 'policy_plugin.policy'

   ```

2. Define the policies in `policy.py`:

   ```python wrap theme={null}
   from airflow.policies import hookimpl
   from airflow.exceptions import AirflowClusterPolicyViolation

   @hookimpl
   def task_policy(task):
       print("Hello from task_policy")
       doc_str = "This is a test doc string"
       task.doc = doc_str

   @hookimpl
   def dag_policy(dag):
       """Ensure that DAG has at least one tag and skip the DAG with `only_for_beta` tag."""
       print("Hello from DAG policy")
       if not dag.tags:
           raise AirflowClusterPolicyViolation(
               f"DAG {dag.dag_id} has no tags. At least one tag required. File path: {dag.fileloc}"
           )
   ```

   <Tip>
     When using Airflow version lower than 2.6 or when you don't want to package your policies, you can define these policies in `config/airflow_local_settings.py` and rebuild your local Astro project.
   </Tip>

3. (Optional) Build the Python package:

   ```bash wrap theme={null}
   python -m build
   ```

### Step 2: Setup your Astro project

1. Initialize your Astro project using the [Astro CLI](/docs/cli/v1.43/get-started-cli) or reopen your Astro project.

2. Copy over your plugin package to the `plugins` directory of your Astro project.

3. Add the following line to your `Dockerfile`:

   ```docker wrap theme={null}
   COPY plugins plugins
   RUN pip install ./plugins
   ```

   <Tip>
     **Alternate setup**

     To avoid copying over the source code or to reuse the package across multiple projects, it's recommended to `build` the plugin package. You can then choose to distribute the `wheel` file or upload the package to a private Python repository for easy management and version control.

     You can copy over the `wheel` file to the `plugins` directory and `pip install` in your `Dockerfile`:

     ```docker wrap theme={null}
     RUN pip install .plugins/plugin_package-*.whl
     ```

     You can read more about Python packaging [here](https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#packaging-your-project).
   </Tip>

4. Run `astro dev restart` to refresh your local Airflow instance. Run `astro deploy` to build and deploy to your Astro Deployment.

## See also

* [Airflow docs](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/cluster-policies.html#how-do-define-a-policy-function) on Cluster policies
* [Airflow summit session](https://airflowsummit.org/sessions/2023/an-introduction-to-airflow-cluster-policies/) on Cluster policies


# Orchestrate Ray jobs on Anyscale with Apache Airflow®
Source: https://astronomer.io/docs/learn/airflow-anyscale

Learn how to use the Anyscale provider package to orchestrate Ray jobs on Anyscale with Apache Airflow®.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Anyscale](https://www.anyscale.com/) is a compute platform for AI/ML workloads built on the open-source [Ray](https://www.ray.io/) framework, providing the layer for parallel processing and distributed computing. The [Anyscale provider package](https://github.com/astronomer/astro-provider-anyscale) for [Apache Airflow®](https://airflow.apache.org/) allows you to interact with Anyscale from your Airflow DAGs. This tutorial shows a simple example of how to use the Anyscale provider package to orchestrate Ray jobs on Anyscale with Airflow. For more in-depth information, see the [Anyscale provider documentation](https://astronomer.github.io/astro-provider-anyscale/).

For instructions on how to run open-source Ray jobs with Airflow, see the [Orchestrate Ray jobs with Apache Airflow®](/docs/learn/airflow-ray) tutorial.

<Tip>
  This tutorial shows a simple implementation of the Anyscale provider package. For a more complex example, see the [Processing User Feedback: an LLM-fine-tuning reference architecture with Ray on Anyscale](/docs/learn/reference-architecture-fine-tuning-anyscale) reference architecture.
</Tip>

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* Ray basics. See the [Getting Started section of the Ray documentation](https://docs.ray.io/en/latest/ray-overview/getting-started.html).
* Anyscale basics. See [Get started section of the Anyscale documentation](https://docs.anyscale.com/get-started).
* Airflow operators. See [Airflow operators](/docs/learn/what-is-an-operator).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).
* An [Anyscale](https://www.anyscale.com/) account with AI platform features enabled. You also need to have at least one [suitable image](https://docs.anyscale.com/reference/anyscale-base-images/) and [compute config](https://docs.anyscale.com/configuration/compute-configuration/#create-a-compute-config) available in your Anyscale account.

## Step 1: Configure your Astro project

Use the Astro CLI to create and run an Airflow project on your local machine.

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-anyscale-tutorial && cd astro-anyscale-tutorial
   $ astro dev init
   ```

2. In the `requirements.txt` file, add the [Anyscale provider](https://github.com/astronomer/astro-provider-anyscale).

   ```text wrap theme={null}
   astro-provider-anyscale==1.0.1
   ```

3. Run the following command to start your Airflow project:

   ```sh wrap theme={null}
   astro dev start
   ```

## Step 2: Configure a Ray connection

<Info>
  For Astro customers, Astronomer recommends taking advantage of the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager) to store connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See [Manage Astro connections in branch-based deploy workflows](/docs/astro/best-practices/connections-branch-deploys).
</Info>

1. In the Airflow UI, go to **Admin** -> **Connections** and click **+**.

2. Create a new connection and choose the **Anyscale** connection type. Enter the following information:

   * **Connection ID**: `anyscale_conn`
   * **API Key**: Your [Anyscale API key](https://docs.anyscale.com/endpoints/text-generation/authenticate)

3. Click **Save**.

## Step 3: Write a DAG to orchestrate Anyscale jobs

1. Create a new file in your `dags` directory called `anyscale_script.py` and add the following code:

   ```python wrap theme={null}
   # anyscale_script.py
   import numpy as np
   import ray
   import argparse

   @ray.remote
   def square(x):
       return x**2

   def main(data):
       ray.init()
       data = np.array(data)
       futures = [square.remote(x) for x in data]
       results = ray.get(futures)
       mean = np.mean(results)
       print(f"Mean squared value: {mean}")
       return mean

   if __name__ == "__main__":
       parser = argparse.ArgumentParser(description="Process some integers.")
       parser.add_argument(
           "data", nargs="+", type=float, help="List of numbers to process"
       )
       args = parser.parse_args()
       data = args.data
       main(data)

   ```

2. Create a new file in your `dags` directory called `anyscale_tutorial.py`.

3. Copy and paste the code below into the file:

   ```python expandable wrap theme={null}
   """
   ## Anyscale Tutorial

   This tutorial demonstrates how to use the Anyscale provider in Airflow to
   parallelize a task using Ray on Anyscale.
   """

   from airflow.decorators import dag
   from airflow.operators.python import PythonOperator
   from anyscale_provider.operators.anyscale import SubmitAnyscaleJob
   from airflow.models.baseoperator import chain
   from pathlib import Path

   CONN_ID = "anyscale_conn"
   FOLDER_PATH = Path(__file__).parent


   def _generate_data() -> list:
       """
       Generate sample data
       Returns:
           list: List of integers
       """
       import random

       return [random.randint(1, 100) for _ in range(10)]


   @dag(
       start_date=None,
       schedule=None,
       catchup=False,
       tags=["ray", "example"],
       doc_md=__doc__,
   )
   def anyscale_tutorial():

       data = PythonOperator(
           task_id="generate_data",
           python_callable=_generate_data,
       )

       get_mean_squared_value = SubmitAnyscaleJob(
           task_id="SubmitRayJob",
           conn_id=CONN_ID,
           name="AstroJob",
           image_uri="< your image uri >",  # e.g. "anyscale/ray:2.35.0-slim-py312-cpu"
           compute_config="< your compute config >",  # e.g. airflow-integration-testing:1
           entrypoint="python anyscale_script.py {{ ti.xcom_pull(task_ids='generate_data') | join(' ') }}",
           working_dir=str(FOLDER_PATH),  # the folder containing the script
           requirements=["requests", "pandas", "numpy", "torch"],
           max_retries=1,
           job_timeout_seconds=3000,
           poll_interval=30,
       )

       chain(data, get_mean_squared_value)


   anyscale_tutorial()
   ```

   * The `generate_data` task randomly generates a list of 10 integers.
   * The `get_mean_squared_value` task submits a Ray job on Anyscale to calculate the mean squared value of the list of integers.

## Step 4: Run the DAG

1. In the Airflow UI, click the play button to manually run your DAG.

2. After the DAG runs successfully, check your Anyscale account to see the job submitted by Airflow.

   <Frame>
     <img alt="Anyscale showing a Job completed successfully." />
   </Frame>

## Conclusion

Congratulations! You've run a Ray job on Anyscale using Apache Airflow. You can now use the Anyscale provider package to orchestrate more complex jobs, see [Processing User Feedback: an LLM-fine-tuning reference architecture with Ray on Anyscale](/docs/learn/reference-architecture-fine-tuning-anyscale) for an example.


# @asset syntax in Apache Airflow®
Source: https://astronomer.io/docs/learn/airflow-asset-decorator

Using the @asset decorator to create a Dag with one task that updates an asset.

The `@asset` decorator is a shorthand to create one Dag with one task that updates an [asset](/docs/learn/airflow-datasets). This decorator is used in the asset-oriented approach to writing Dags which constitutes a mindset shift to put the data asset front and center. Whether you use the asset-oriented or task-oriented approach to writing Dags is a matter of preference. Dags created using the asset-oriented approach are shown like any other Dag in the Airflow UI.

In this guide, you'll learn:

* How to use `@asset` to create a Dag with one task that updates an asset.
* How to use `@asset.multi` to create a Dag with one task that updates multiple assets.

<Tip>
  If you are looking for instructions on how to use asset-based scheduling in Airflow with the `Asset` object, see [Basic asset-based scheduling in Apache Airflow®](/docs/learn/airflow-datasets), as well as [Advanced asset-based scheduling](/docs/learn/airflow-advanced-asset-scheduling).
</Tip>

<Tip>
  The `@asset` decorator is an example of a Dag authoring paradigm (asset-oriented) that is different from the task-oriented approach. To learn more about different Dag authoring paradigms, check out the free [Apache Airflow® orchestration paradigms eBook](https://www.astronomer.io/ebooks/apache-airflow-orchestration-paradigms).
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an existing knowledge of:

* Airflow basic asset-based scheduling. See [Basic asset-based scheduling in Apache Airflow®](/docs/learn/airflow-datasets).
* Airflow decorators. See [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).

## Use @asset

The following code snippet defines a Dag with the Dag ID `my_asset` that runs on a `@daily` schedule. It contains one task with the task ID `my_asset` that, upon successful completion updates an asset with the name `my_asset`.

```python wrap theme={null}
from airflow.sdk import asset 

@asset(schedule="@daily")
def my_asset():
    # your task logic here
    pass
```

You can schedule assets based on other assets to create data-centric pipelines. Since each `@asset` decorator creates one Dag, data needs to be passed between tasks using cross-Dag XComs. The following shows the same simple ETL pipeline accomplished using the asset-oriented and the task-oriented approach.

<details>
  <summary>Asset</summary>

  ```python expandable wrap theme={null}
  from airflow.sdk import asset


  @asset(schedule="@daily")
  def extracted_data():
      return {"a": 1, "b": 2}


  @asset(schedule=extracted_data)
  def transformed_data(context):

      data = context["ti"].xcom_pull(
          dag_id="extracted_data",
          task_ids="extracted_data",
          key="return_value",
          include_prior_dates=True,
      )
      return {k: v * 2 for k, v in data.items()}


  @asset(schedule=transformed_data)
  def loaded_data(context):

      data = context["task_instance"].xcom_pull(
          dag_id="transformed_data",
          task_ids="transformed_data",
          key="return_value",
          include_prior_dates=True,
      )
      summed_data = sum(data.values())
      print(f"Summed data: {summed_data}")
  ```
</details>

<details>
  <summary>Task</summary>

  ```python expandable wrap theme={null}
  from airflow.sdk import Asset, dag, task


  @dag(schedule="@daily")
  def extract_dag():

      @task(outlets=[Asset("extracted_data")])
      def extract_task():
          return {"a": 1, "b": 2}

      extract_task()


  extract_dag()


  @dag(schedule=[Asset("extracted_data")])
  def transform_dag():

      @task(outlets=[Asset("transformed_data")])
      def transform_task(**context):
          data = context["ti"].xcom_pull(
              dag_id="extract_dag",
              task_ids="extract_task",
              key="return_value",
              include_prior_dates=True,
          )
          return {k: v * 2 for k, v in data.items()}

      transform_task()


  transform_dag()


  @dag(schedule=[Asset("transformed_data")])
  def load_dag():

      @task
      def load_task(**context):
          data = context["ti"].xcom_pull(
              dag_id="transform_dag",
              task_ids="transform_task",
              key="return_value",
              include_prior_dates=True,
          )
          summed_data = sum(data.values())
          print(f"Summed data: {summed_data}")

      load_task()


  load_dag()
  ```
</details>

The code above creates three Dags that depend on each other, each containing one task that updates one asset:

<Frame>
  <img alt="DAGs view showing 3 DAGs." />
</Frame>

## @`asset.multi`

To update several assets from the same Dag written with the asset-oriented approach, you can use `@asset.multi`. The code example below will create one Dag with the Dag ID `my_multi_asset` that contains one task called `my_multi_asset` that, upon successful completion, updates two assets with the names `asset_a` and `asset_b`.

```python wrap theme={null}
from airflow.sdk import Asset, asset

@asset.multi(schedule="@daily", outlets=[Asset("asset_a"), Asset("asset_b")])
def my_multi_asset():
    pass
```


# Run a task in Azure Container Instances with Airflow
Source: https://astronomer.io/docs/learn/airflow-azure-container-instances

Learn how to orchestrate containers with Azure Container Instances from your Airflow DAGs.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Azure Container Instances](https://azure.microsoft.com/en-us/services/container-instances/) (ACI) is one service that Azure users can leverage for working with containers. In this tutorial, you'll learn how to orchestrate ACI using Airflow and create a DAG that runs a task in an ACI container.

<Info>
  All code in this tutorial can be found on [the Astronomer Registry](https://registry.astronomer.io/dags/azure-container-instance).
</Info>

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of ACI. See [Getting started with Azure Container Instances](https://azure.microsoft.com/en-us/products/container-instances/#getting-started).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow connections. See [Managing your Connections in Apache Airflow](/docs/learn/connections).

## Prerequisites

To complete this tutorial, you need:

* Access to ACI. See [Quickstart: Deploy a container instance in Azure using the Azure portal](https://learn.microsoft.com/en-us/azure/container-instances/container-instances-quickstart-portal) for instructions. If you don't already have ACI access, Azure offers a \$200 credit when you sign up for a free Azure account.
* The [Astro CLI](/docs/cli/v1.43/get-started-cli).

## Step 1: Create an Azure service principal

An Azure service principal is required for external tools like Airflow to connect to your Azure resources. Identify the Azure resource group you want to create your ACI in (or create a new one), then create service principal with write access over that resource group. For more information, see [Use the portal to create a Microsoft Entra ID application and service principal that can access resources](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal).

## Step 2: Configure your Astro project

Now that you have your Azure resources configured, you can move on to setting up Airflow.

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-aci-tutorial && cd astro-aci-tutorial
   $ astro dev init
   ```

2. Add the following line to the `requirements.txt` file of your Astro project:

   ```text wrap theme={null}
   apache-airflow-providers-microsoft-azure
   ```

   This installs the Azure provider package that contains all of the relevant ACI modules.

3. Run the following command to start your project in a local environment:

   ```sh wrap theme={null}
   astro dev start
   ```

## Step 3: Add an Airflow connection to ACI

Add a connection that Airflow will use to connect to ACI. In the Airflow UI, go to **Admin** -> **Connections**.

Create a new connection named `azure_container_conn_id` and choose the **Azure Container Instance** connection type.

Specify your Client ID in the **Login** field, Client Secret in the **Password** field, and Tenant and Subscription IDs in the **Extras** field as JSON. It should look something like this:

<Frame>
  <img alt="ACI Connection" />
</Frame>

## Step 4: Choose a Docker image

Choose a Docker image that you want to run. The `AzureContainerInstancesOperator` will run any Docker image in a container with your specifications. If you don't have an image, you can use a pre-built one such as Docker's `hello-world:latest` image. You can search for other available images in [Docker's container image repository](https://hub.docker.com/search?q=).

## Step 5: Create your DAG

In your Astro project `dags/` folder, create a new file called `aci-pipeline.py`. Paste the following code into the file:

```python wrap theme={null}
from airflow.models.dag import DAG
from airflow.providers.microsoft.azure.operators.container_instances import AzureContainerInstancesOperator
from datetime import datetime, timedelta

with DAG('azure_container_instances',
         start_date=datetime(2020, 12, 1),
         max_active_runs=1,
         schedule='@daily',
         default_args = {
            'retries': 1,
            'retry_delay': timedelta(minutes=1)
        },
         catchup=False
         ) as dag:

    opr_run_container = AzureContainerInstancesOperator(
        task_id='run_container',
        ci_conn_id='azure_container_conn_id',
        registry_conn_id=None,
        resource_group='<your-resource-group>',
        name='azure-tutorial-container',
        image='hello-world:latest',
        region='East US',
        cpu=1,
        memory_in_gb=1.5,
        fail_if_exists=False

    )
```

Update the `resource_group` parameter to the name of the resource group you created in Step 1. You may wish to update some of the other parameters in your operator, particularly the `image` and `registry_conn_id` if you chose a different Docker image. The following parameters are defined in this example:

* **`ci_conn_id`**: The connection ID for the Airflow connection you created in Step 3.
* **`registry_conn_id`**: The connection ID to connect to a registry. In this tutorial we use DockerHub, which is public and doesn't require credentials, so we pass in `None`.
* **`resource_group`**: The Azure resource group you created in Step 1.
* **`name`**: The name you want to give your ACI. Note that this must be unique within the resource group.
* **`image`**: The Docker image you chose in Step 4. In this case we use a simple Hello World example from Docker.
* **`region`**: The Azure region we want our ACI deployed to
* **`cpu`**: The number of CPUs to allocate to your container. In this example we use the default minimum. For more information on allocating CPUs and memory, refer to the [Azure documentation](https://docs.microsoft.com/en-us/azure/container-instances/container-instances-faq).
* **`memory_in_gb`**: The amount of memory to allocate to the container. In example we use the default minimum.
* **`fail_if_exists`**: Whether you want the operator to raise an exception if the container group already exists (default value is `True`). If it's set to False and the container group name already exists within the given resource group, the operator will attempt to update the container group based on the other parameters before running and terminating upon completion.

You can also provide the operator with other parameters such as environment variables, volumes, and a command as needed to run the container. For more information on the `AzureContainerInstancesOperator`, check out the [Airflow Registry](https://airflow.apache.org/registry/providers/microsoft-azure#microsoft-azure-container_instances-AzureContainerInstancesOperator).

<Info>
  This operator can also be used to run existing container instances and make certain updates, including the docker image, environment variables, or commands. Some updates to existing container groups aren't possible with the operator, including CPU, memory, and GPU; those updates require deleting the existing container group and recreating it, which can be accomplished using the [`AzureContainerInstanceHook`](https://airflow.apache.org/registry/providers/microsoft-azure#microsoft-azure-container_instance-AzureContainerInstanceHook).
</Info>

## Step 6: Run the DAG and review the task logs

Go to the Airflow UI, unpause your `azure_container_instances` DAG, and trigger it to run the image in your ACI. An ACI will spin up, run the container with the Hello World image, and spin down. Go to the Airflow task log, and you should see the printout from the container has propagated to the logs:

<Frame>
  <img alt="ACI Task Log" />
</Frame>

## Additional considerations

There are multiple ways to manage containers with Airflow on Azure. The most flexible and scalable method is to use the [`KubernetesPodOperator`](/docs/learn/kubepod-operator). This lets you run any container as a Kubernetes pod, which means you can pass in resource requests and other native Kubernetes parameters. Using this operator requires an [AKS](https://azure.microsoft.com/en-us/services/kubernetes-service/) cluster (or a hand-rolled Kubernetes cluster).

If you aren't running on [AKS](https://azure.microsoft.com/en-us/services/kubernetes-service/), ACI can be a great choice:

* It's easy to use and requires little setup
* You can run containers in different regions
* It's typically the cheapest; since no virtual machines or higher-level services are required, **you only pay for the memory and CPU used by your container group while it is active**
* Unlike the [`DockerOperator`](https://airflow.apache.org/registry/providers/docker#docker-docker-DockerOperator), it doesn't require running a container on the host machine

With these points in mind, Astronomer recommends using ACI with the `AzureContainerInstancesOperator` for testing or lightweight tasks that don't require scaling. For heavy production workloads, you should use AKS and the `KubernetesPodOperator`.


# Run Azure Data Factory pipelines with Airflow
Source: https://astronomer.io/docs/learn/airflow-azure-data-factory-integration

Learn how to orchestrate remote jobs in Azure Data Factory with your Apache Airflow DAGs.

<head>
  <meta name="robots" />
</head>

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

Azure Data Factory (ADF) is a commonly used service for constructing data pipelines and jobs. With a little preparation, you can combine it with Airflow to use the best of both tools. In this tutorial, you'll learn why you might want to use these two tools together and how to run your ADF pipeline from your Airflow DAG.

<Info>
  All code in this tutorial can be found on [the Astronomer Registry](https://registry.astronomer.io/dags/azure-data-factory-dag).
</Info>

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of Azure Data Factory. See [Introduction to Data Factory](https://learn.microsoft.com/en-us/azure/data-factory/introduction).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow connections. See [Managing your Connections in Apache Airflow](/docs/learn/connections).

## Prerequisites

To complete this tutorial, you need:

* Two ADF pipelines. If you don't currently have an ADF pipeline in your Azure account and are new to ADF, check out the [ADF quick start docs](https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-portal) for help getting started.
* The [Astro CLI](/docs/cli/v1.43/get-started-cli).

## Step 1: Make your ADF pipelines runnable

Before you can orchestrate your ADF pipelines with Airflow, you have to make the pipelines runnable by an external service. There are multiple ways to do this depending on how you manage authentication within your Azure account. This tutorial shows how to register an App with Microsoft Entra to get a **Client ID** and **Client Secret** (API Key) for your Data Factory.

1. Go to Microsoft Entra ID and click **App registrations** to see a list of registered apps. If you created a Resource group, you should already have an app registered with the same name. Otherwise you can create a new one.

   <Frame>
     <img alt="ADF App Registration" />
   </Frame>

   Click the app associated with your resource group, and note the **Application (client) Id**. You'll need this to connect Airflow to ADF.

2. Go to **Certificates & Secrets** -> **New client secret** and create a **Client Secret** to connect Data Factory to Airflow.

3. Connect your **Client Secret** API key to your Data Factory instance. Go back to the overview of your Data Factory and click **Access Control** -> **Add role assignments** and add your **Application** as a contributor to the Data Factory.

   <Frame>
     <img alt="ADF Access Control" />
   </Frame>

4. Add a role assignment with the following settings:

   * Role: Data Factory Contributor
   * Assign access to: User, group, or service principal

   Search for your app, add it to **Selected members**, and click **Save**.

<Info>
  Find additional detail on requirements for interacting with Azure Data Factory using the REST API in the [Microsoft documentation](https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-rest-api). You can also reference [the documentation](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#register-an-application-with-azure-ad-and-create-a-service-principal) for more information on creating a registered application in Microsoft Entra ID.
</Info>

## Step 2: Configure your Astro project

Now that you have your Azure resources configured, you can move on to setting up Airflow.

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-adf-tutorial && cd astro-adf-tutorial
   $ astro dev init
   ```

2. Add the following line to the `requirements.txt` file of your Astro project:

   ```text wrap theme={null}
   apache-airflow-providers-microsoft-azure
   ```

   This installs the Azure provider package that contains all of the relevant ADF modules.

3. Run the following command to start your project in a local environment:

   ```sh wrap theme={null}
   astro dev start
   ```

## Step 3: Create an Airflow connection to ADF

Add a connection that Airflow will use to connect to ADF. In the Airflow UI, go to **Admin** -> **Connections**.

Create a new connection named `azure_data_factory` and choose the `Azure Data Factory` connection type. Enter the following information:

* **Login**: Your Azure **Client ID** from Step 1
* **Password**: Your Azure **Client secret** from Step 1
* **Extras**: `{"tenantId":"Your Tenant ID", "subscriptionId":"Your Subscription ID"}`

## Step 4: Create your DAG

In your Astro project `dags/` folder, create a new file called `adf-pipeline.py`. Paste the following code into the file:

```python expandable wrap theme={null}
from datetime import datetime, timedelta

from airflow.models import DAG, BaseOperator
from airflow.operators.empty import EmptyOperator
from airflow.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperator
from airflow.providers.microsoft.azure.sensors.data_factory import AzureDataFactoryPipelineRunStatusSensor
from airflow.utils.edgemodifier import Label

with DAG(
    dag_id="example_adf_run_pipeline",
    start_date=datetime(2021, 8, 13),
    schedule="@daily",
    catchup=False,
    default_args={
        "retries": 1,
        "retry_delay": timedelta(minutes=3),
        "azure_data_factory_conn_id": "azure_data_factory",
        "factory_name": "my-data-factory",  # This can also be specified in the ADF connection.
        "resource_group_name": "my-resource-group",  # This can also be specified in the ADF connection.
    },
    default_view="graph",
) as dag:
    begin = EmptyOperator(task_id="begin")
    end = EmptyOperator(task_id="end")

    # [START howto_operator_adf_run_pipeline]
    run_pipeline1: BaseOperator = AzureDataFactoryRunPipelineOperator(
        task_id="run_pipeline1",
        pipeline_name="pipeline1",
        parameters={"myParam": "value"},
    )
    # [END howto_operator_adf_run_pipeline]

    # [START howto_operator_adf_run_pipeline_async]
    run_pipeline2: BaseOperator = AzureDataFactoryRunPipelineOperator(
        task_id="run_pipeline2",
        pipeline_name="pipeline2",
        wait_for_termination=False,
    )

    pipeline_run_sensor: BaseOperator = AzureDataFactoryPipelineRunStatusSensor(
        task_id="pipeline_run_sensor",
        run_id=run_pipeline2.output["run_id"],
    )
    # [END howto_operator_adf_run_pipeline_async]

    begin >> Label("No async wait") >> run_pipeline1
    begin >> Label("Do async wait with sensor") >> run_pipeline2
    [run_pipeline1, pipeline_run_sensor] >> end
```

Update the following parameters in the DAG code:

* `pipeline_name` in the `run_pipeline1` and `run_pipeline2` tasks to the names of your two ADF pipelines.
* `factory_name` in the `default_args` to your factory name.
* `resource_group_name` in the `default_args` to your resource group name from Step 1.

The DAG graph should look similar to this:

<Frame>
  <img alt="Graph View" />
</Frame>

## Step 5: Run your DAG to execute your ADF pipelines

Go to the Airflow UI, unpause your `example_adf_run_pipeline` DAG, and trigger it to run your ADF pipelines.
The DAG will execute both ADF pipelines in parallel (tasks `run_pipeline1` and `run_pipeline2`), and then will use an `AzureDataFactoryPipelineRunStatusSensor` to wait until `pipeline2` has completed before finishing the DAG.

To learn more about all of the ADF modules in the Microsoft Azure provider, check out the [Airflow Registry](https://airflow.apache.org/registry/providers/microsoft-azure).

## Why use Airflow with ADF

ADF is an easy to learn tool that allows you to quickly create jobs without writing code. It integrates seamlessly with on-premises data sources and other Azure services. However, it has some disadvantages when used alone - namely:

* Building and integrating custom tools can be difficult
* Integrations with services outside of Azure are limited
* Orchestration capabilities are limited
* Custom packages and dependencies can be complex to manage

That's where Airflow comes in. ADF jobs can be run using an Airflow DAG, giving the full capabilities of Airflow orchestration beyond using ADF alone. This allows users that are comfortable with ADF to write their job there, while Airflow acts as the control plane for orchestration.

For a more complex example of orchestrating dependent ADF pipelines with Airflow, see [Orchestrating Multiple Azure Data Factory Pipelines in Airflow](https://registry.astronomer.io/dags/airflow-azure-data-factory).


# Branching in Airflow
Source: https://astronomer.io/docs/learn/airflow-branch-operator

Learn about Airflow's multiple options for building conditional logic and branching within DAGs, including the BranchPythonOperator and ShortCircuitOperator.

When designing your data pipelines, you may encounter use cases that require more complex task flows than "Task A > Task B > Task C." For example, you may have a use case where you need to decide between multiple tasks to execute based on the results of an upstream task. Or you may have a case where part of your pipeline should only run under certain external conditions. Fortunately, Airflow has multiple options for building conditional logic and/or branching into your DAGs.

In this guide, you'll learn how you can use `@task.branch` (`BranchPythonOperator`) and `@task.short_circuit` (`ShortCircuitOperator`), other available branching operators, and additional resources to implement conditional logic in your Airflow DAGs.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow operators. See [Airflow operators](/docs/learn/what-is-an-operator).
* Dependencies in Airflow. See [Managing Dependencies in Apache Airflow](/docs/learn/managing-dependencies).
* Using Airflow decorators. See [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).

## `@task.branch` (`BranchPythonOperator`)

One of the simplest ways to implement branching in Airflow is to use the `@task.branch` decorator, which is a decorated version of the [`BranchPythonOperator`](https://airflow.apache.org/registry/providers/standard#standard-python-BranchPythonOperator). `@task.branch` accepts any Python function as an input as long as the function returns a list of valid IDs for Airflow tasks that the DAG should run after the function completes.

In the following example we use a `choose_branch` function that returns one set of task IDs if the result is greater than 0.5 and a different set if the result is less than or equal to 0.5:

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # from airflow.sdk import task 

  result = 1

  @task.branch
  def choose_branch(result):
      if result > 0.5:
          return ['task_a', 'task_b']
      return ['task_c']

  choose_branch(result)
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # from airflow.providers.standard.operators.python import BranchPythonOperator

  result = 1

  def choose_branch(result):
      if result > 0.5:
          return ['task_a', 'task_b']
      return ['task_c']

  branching = BranchPythonOperator(
      task_id='branching',
      python_callable=choose_branch,
      op_args=[result]
  )
  ```
</details>

In general, the `@task.branch` decorator is a good choice if your branching logic can be easily implemented in a simple Python function. Whether you want to use the decorated version or the traditional operator is a question of personal preference.

The code below shows a full example of how to use `@task.branch` in a DAG:

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  """Example DAG demonstrating the usage of the `@task.branch`
  TaskFlow API decorator."""

  from airflow.sdk import dag, Label, task
  from airflow.providers.standard.operators.empty import EmptyOperator

  import random

  @dag
  def branch_python_operator_decorator_example():

      run_this_first = EmptyOperator(task_id="run_this_first")

      options = ["branch_a", "branch_b", "branch_c", "branch_d"]

      @task.branch(task_id="branching")
      def random_choice(choices):
          return random.choice(choices)

      random_choice_instance = random_choice(choices=options)

      run_this_first >> random_choice_instance

      join = EmptyOperator(
          task_id="join",
          trigger_rule="none_failed_min_one_success"
      )

      for option in options:

          t = EmptyOperator(
              task_id=option
          )

          empty_follow = EmptyOperator(
              task_id="follow_" + option
          )

          # Label is optional here, but it can help identify more complex branches
          random_choice_instance >> Label(option) >> t >> empty_follow >> join

  branch_python_operator_decorator_example()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  """Example DAG demonstrating the usage of the BranchPythonOperator."""

  from airflow.sdk import DAG, Label
  from airflow.providers.standard.operators.empty import EmptyOperator
  from airflow.providers.standard.operators.python import BranchPythonOperator
  import random

  with DAG(
      dag_id='branch_python_operator_example'
  ) as dag:

      run_this_first = EmptyOperator(
          task_id='run_this_first',
      )

      options = ['branch_a', 'branch_b', 'branch_c', 'branch_d']

      branching = BranchPythonOperator(
          task_id='branching',
          python_callable=lambda: random.choice(options),
      )

      run_this_first >> branching

      join = EmptyOperator(
          task_id='join',
          trigger_rule="none_failed_min_one_success",
      )

      for option in options:

          t = EmptyOperator(
              task_id=option,
          )

          empty_follow = EmptyOperator(
              task_id='follow_' + option,
          )

          # Label is optional here, but it can help identify more complex branches
          branching >> Label(option) >> t >> empty_follow >> join
  ```
</details>

In this DAG, `random.choice()` returns one random option out of a list of four branches. In the following screenshot, where `branch_b` was randomly chosen, the two tasks in `branch_b` were successfully run while the others were skipped.

<Frame>
  <img alt="Branching Graph View" />
</Frame>

If you have downstream tasks that need to run regardless of which branch is taken, like the `join` task in the previous example, you need to update the [trigger rule](/docs/learn/airflow-trigger-rules). The default trigger rule in Airflow is `all_success`, which means that if upstream tasks are skipped, then the downstream task won't run. In the previous example, `none_failed_min_one_success` is specified to indicate that the task should run as long as one upstream task succeeded and no tasks failed.

You can also set a [task group](/docs/learn/task-groups) as the direct downstream element of a branching task by returning its `task_group_id` in your decorated function or `python_callable` instead of a `task_id`. All root tasks of the task group run if the branching tasks return the `task_group_id`.

<details>
  <summary>Click to view sample DAG code and a corresponding task graph.</summary>

  ```python expandable wrap theme={null}
  from airflow.decorators import dag, task_group, task
  from airflow.models.baseoperator import chain
  from pendulum import datetime


  @dag(
      dag_display_name="Task Group Branching",
      start_date=datetime(2024, 8, 1),
      schedule=None,
      catchup=False,
      tags=["Branching"],
  )
  def task_group_branching():

      @task.branch
      def upstream_task():
          return "my_task_group"

      @task_group
      def my_task_group():

          @task
          def t1():
              return "hi"

          t1()

          @task
          def t2():
              return "hi"

          t2()

      @task
      def outside_task():
          return "hi"

      chain(upstream_task(), [my_task_group(), outside_task()])


  task_group_branching()
  ```

  <Frame>
    <img alt="Screenshot of graph in UI of DAG using task grouping." />
  </Frame>
</details>

Finally, note that with the `@task.branch` decorator your Python function *must* return at least one task ID for whichever branch is chosen (in other words, it can't return nothing). If one of the paths in your branching should do nothing, you can use an `EmptyOperator` in that branch.

## `@task.short_circuit` (`ShortCircuitOperator`)

Another option for implementing conditional logic in your DAGs is the `@task.short_circuit` decorator, which is a decorated version of the [`ShortCircuitOperator`](https://airflow.apache.org/registry/providers/standard#standard-python-ShortCircuitOperator). This operator takes a Python function that returns `True` or `False` based on logic implemented for your use case. If `True` is returned, the DAG continues, and if `False` is returned, all downstream tasks are skipped.

`@task.short_circuit` is useful when you know that some tasks in your DAG should run only occasionally. For example, maybe your DAG runs daily, but some tasks should only run on Sundays. Or maybe your DAG orchestrates a machine learning model, and tasks that publish the model should only be run if a certain accuracy is reached after training. This type of logic can also be implemented with `@task.branch`, but that operator requires a task ID to be returned. Using the `@task.short_circuit` decorator can be cleaner in cases where the conditional logic equates to "run or not" as opposed to "run this or that."

The following DAG shows an example of how to implement `@task.short_circuit`:

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  """Example DAG demonstrating the usage of the @task.short_circuit decorator."""

  from airflow.sdk import dag, task, chain
  from airflow.providers.standard.operators.empty import EmptyOperator

  @dag
  def short_circuit_operator_decorator_example():

      @task.short_circuit
      def condition_is_True():
          return True

      @task.short_circuit
      def condition_is_False():
          return False

      ds_true = [EmptyOperator(task_id='true_' + str(i)) for i in [1, 2]]
      ds_false = [EmptyOperator(task_id='false_' + str(i)) for i in [1, 2]]

      chain(condition_is_True(), *ds_true)
      chain(condition_is_False(), *ds_false)

  short_circuit_operator_decorator_example()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  """Example DAG demonstrating the usage of the ShortCircuitOperator."""

  from airflow.sdk import DAG, chain
  from airflow.providers.standard.operators.empty import EmptyOperator
  from airflow.providers.standard.operators.python import ShortCircuitOperator

  with DAG(
      dag_id='short_circuit_operator_example'
  ) as dag:

      cond_true = ShortCircuitOperator(
          task_id='condition_is_True',
          python_callable=lambda: True,
      )

      cond_false = ShortCircuitOperator(
          task_id='condition_is_False',
          python_callable=lambda: False,
      )

      ds_true = [EmptyOperator(task_id='true_' + str(i)) for i in [1, 2]]
      ds_false = [EmptyOperator(task_id='false_' + str(i)) for i in [1, 2]]

      chain(cond_true, *ds_true)
      chain(cond_false, *ds_false)
  ```
</details>

In this DAG there are two short circuits, one which always returns `True` and one which always returns `False`. When you run the DAG, you can see that tasks downstream of the `True` condition operator ran, while tasks downstream of the `False` condition operator were skipped.

<Frame>
  <img alt="Short Circuit" />
</Frame>

## Other branch operators

Airflow offers a few other branching operators that work similarly to the `BranchPythonOperator` but for more specific contexts:

* [`BranchSQLOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-BranchSQLOperator): Branches based on whether a given SQL query returns `true` or `false`.
* [`BranchDayOfWeekOperator`](https://airflow.apache.org/registry/providers/standard#standard-weekday-BranchDayOfWeekOperator): Branches based on whether the current day of week is equal to a given `week_day` parameter.
* [`BranchDateTimeOperator`](https://airflow.apache.org/registry/providers/standard#standard-datetime-BranchDateTimeOperator): Branches based on whether the current time is between `target_lower` and `target_upper` times.
* [`BranchExternalPythonOperator`](https://airflow.apache.org/registry/providers/standard#standard-python-BranchExternalPythonOperator): Branches based on a Python function like the [`BranchPythonOperator`](#@task-branch-branchpythonoperator), but runs in a preexisting virtual environment like the [`ExternalPythonOperator`](/docs/learn/airflow-isolated-environments).
* [`BranchPythonVirtualenvOperator`](https://airflow.apache.org/registry/providers/standard#standard-python-BranchPythonVirtualenvOperator): Branches based on a Python function like the [`BranchPythonOperator`](#@task-branch-branchpythonoperator), but runs in newly created virtual environment like the [`PythonVirtualenvOperator`](https://airflow.apache.org/registry/providers/standard#standard-python-PythonVirtualenvOperator). The environment can be cached by providing a `venv_cache_path`.

All of these operators take `follow_task_ids_if_true` and `follow_task_ids_if_false` parameters which provide the list of task(s) to include in the branch based on the logic returned by the operator.


# Orchestrate Cohere LLMs with Apache Airflow
Source: https://astronomer.io/docs/learn/airflow-cohere

Learn how to integrate Cohere and Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Cohere](https://cohere.com/) is a natural language processing (NLP) platform that provides an API to access large language models (LLMs). The [Cohere Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-cohere/stable/index.html) offers modules to easily integrate Cohere with Airflow.

In this tutorial, you use Airflow and the Cohere Airflow provider to generate recipe suggestions based on a list of ingredients and countries of recipe origin. Additionally, you create embeddings of the recipes and perform dimensionality reduction using principal component analysis (PCA) to plot recipe similarity in two dimensions.

## Why use Airflow with Cohere?

Cohere provides highly specialized out-of-the box and custom LLMs. Countless applications use these models for both user-facing needs, such as to moderate user-generated content, and internal purposes, like providing insight into customer support tickets.

Integrating Cohere with Airflow into one end-to-end machine learning pipeline allows you to:

* Use Airflow's [data-driven scheduling](/docs/learn/airflow-datasets) to run operations with Cohere LLM endpoints based on upstream events in your data ecosystem, like when new user input is ingested or a new dataset is available.
* Send several requests to a model endpoint in parallel based on upstream events in your data ecosystem or user input with [Airflow params](/docs/learn/airflow-params).
* Add Airflow features like [retries](/docs/learn/rerunning-dags#automatically-retry-tasks) and [alerts](/docs/learn/error-notifications-in-airflow) to your Cohere operations. This is critical for day 2 MLOps operations, for example, to handle model service outages.
* Use Airflow to orchestrate the creation of vector embeddings with Cohere models, which is especially useful for very large datasets that can't be processed automatically by vector databases.

## Time to complete

This tutorial takes approximately 15 minutes to complete (cooking your recommended recipe not included).

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of the Cohere API. See [Cohere Documentation](https://docs.cohere.com/reference/about).
* The basics of vector embeddings. See the [Cohere Embeddings guide](https://docs.cohere.com/docs/embeddings).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow hooks. See [Hooks 101](/docs/learn/what-is-a-hook).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).
* A Cohere API key. You can generate an API key in the [Cohere dashboard](https://dashboard.cohere.com/api-keys), accessible with a Cohere account. A free tier API key is sufficient for this tutorial.

## Step 1: Configure your Astro project

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-cohere-tutorial && cd astro-cohere-tutorial
   $ astro dev init
   ```

2. Add the following lines to your `requirements.txt` file to install the Cohere Airflow provider and other supporting packages:

   ```text wrap theme={null}
   apache-airflow-providers-cohere==1.0.0
   matplotlib==3.8.1
   seaborn==0.13.0
   scikit-learn==1.3.2
   pandas==1.5.3
   numpy==1.26.2
   adjustText==0.8
   ```

3. To create an [Airflow connection](/docs/learn/connections) to Cohere, add the following environment variables to your `.env` file. Make sure to provide `<your-cohere-api-key>`.

   ```text wrap theme={null}
   AIRFLOW_CONN_COHERE_DEFAULT='{
       "conn_type": "cohere",
       "password": "<your-cohere-api-key>"
   }'
   ```

## Step 2: Create your DAG

1. In your `dags` folder, create a file called `recipe_suggestions.py`.

2. Copy the following code into the file.

   ```python expandable wrap theme={null}
   """
   ## Get recipe suggestions using Cohere's LLMs, embed and visualize the results

   This DAG shows how to use the Cohere Airflow provider to interact with the Cohere API.
   The DAG generates recipes based on user input via Airflow params, embeds the
   responses using Cohere embeddings, and visualizes them in 2 dimensions using PCA,
   matplotlib and seaborn.
   """

   from airflow.decorators import dag, task
   from airflow.models.param import Param
   from airflow.models.baseoperator import chain
   from airflow.providers.cohere.hooks.cohere import CohereHook
   from airflow.providers.cohere.operators.embedding import CohereEmbeddingOperator
   from sklearn.metrics.pairwise import euclidean_distances
   from sklearn.decomposition import PCA
   from adjustText import adjust_text
   from pendulum import datetime
   import matplotlib.pyplot as plt
   import seaborn as sns
   import pandas as pd
   import numpy as np


   COHERE_CONN_ID = "cohere_default"
   IMAGE_PATH = "include/recipe_plot.png"


   @dag(
       start_date=datetime(2023, 11, 1),
       schedule=None,
       catchup=False,
       params={
           "countries": Param(
               ["Switzerland", "Norway", "New Zealand", "Cameroon", "Bhutan", "Chile"],
               type="array",
               title="Countries of recipe origin",
               description="Enter from which countries you would like to get recipes."
               + "List at least two countries.",
           ),
           "pantry_ingredients": Param(
               ["gruyere", "olives", "potatoes", "onions", "pineapple"],
               type="array",
               description="List the ingredients you have in your pantry, you'd like to use",
           ),
           "type": Param(
               "vegetarian",
               type="string",
               enum=["vegan", "vegetarian", "omnivore"],
               description="Select the type of recipe you'd like to get.",
           ),
           "max_tokens_recipe": Param(
               500,
               type="integer",
               description="Enter the max number of tokens the model should generate.",
           ),
           "randomness_of_recipe": Param(
               25,
               type="integer",
               description=(
                   "Enter the desired randomness of the recipe on a scale"
                   + "from 0 (no randomness) to 50 (full randomness). "
                   + "This setting corresponds to 10x the temperature setting in the Cohere API."
               ),
           ),
       },
   )
   def recipe_suggestions():
       @task
       def get_countries_list(**context):
           "Pull the list of countries from the context."
           countries = context["params"]["countries"]
           return countries

       @task
       def get_ingredients_list(**context):
           "Pull the list of ingredients from the context."
           ingredients = context["params"]["pantry_ingredients"]
           return ingredients

       @task
       def get_a_recipe(
           cohere_conn_id: str, country: str, ingredients_list: list, **context
       ):
           "Get recipes from the Cohere API for your pantry ingredients for a given country."
           type = context["params"]["type"]
           max_tokens_answer = context["params"]["max_tokens_recipe"]
           randomness_of_answer = context["params"]["randomness_of_recipe"]
           co = CohereHook(conn_id=cohere_conn_id).get_conn

           response = co.generate(
               model="command",
               prompt=f"Please provide a delicious {type} recipe from {country} "
               + f"that uses as many of these ingredients: {', '.join(ingredients_list)} as possible, "
               + "if you can't find a recipe that uses all of them, suggest an additional desert."
               + "Bonus points if it's a traditional recipe from that country, "
               + "you can name the city or region it's from and you can provide "
               + "vegan alternatives for the ingredients."
               + "Provide the full recipe with all steps and ingredients.",
               max_tokens=max_tokens_answer,
               temperature=randomness_of_answer / 10,
           )

           recipe = response.generations[0].text

           print(f"Your recipe from {country}")
           print(f"for the ingredients {', '.join(ingredients_list)} is:")
           print(recipe)

           with open(f"include/{country}_recipe.txt", "w") as f:
               f.write(recipe)

           return recipe

       countries_list = get_countries_list()
       ingredients_list = get_ingredients_list()
       recipes_list = get_a_recipe.partial(
           cohere_conn_id=COHERE_CONN_ID, ingredients_list=ingredients_list
       ).expand(country=countries_list)

       get_embeddings = CohereEmbeddingOperator.partial(
           task_id="get_embeddings",
           conn_id=COHERE_CONN_ID,
       ).expand(input_text=recipes_list)

       @task
       def plot_embeddings(embeddings, text_labels, file_name="embeddings_plot.png"):
           "Plot the embeddings of the recipes."

           embeddings = [x[0] for x in embeddings]
           print(text_labels)

           pca = PCA(n_components=2)
           reduced_embeddings = pca.fit_transform(embeddings)

           plt.figure(figsize=(10, 8))
           df_embeddings = pd.DataFrame(reduced_embeddings, columns=["PC1", "PC2"])
           sns.scatterplot(
               df_embeddings, x="PC1", y="PC2", s=100, color="gold", edgecolor="black"
           )

           font_style = {"color": "black"}
           texts = []
           for i, label in enumerate(text_labels):
               texts.append(
                   plt.text(
                       reduced_embeddings[i, 0],
                       reduced_embeddings[i, 1],
                       label,
                       fontdict=font_style,
                       fontsize=15,
                   )
               )

           # prevent overlapping labels
           adjust_text(texts, arrowprops=dict(arrowstyle="->", color="red"))

           distances = euclidean_distances(reduced_embeddings)
           np.fill_diagonal(distances, np.inf)  # exclude cases where the distance is 0

           n = distances.shape[0]
           distances_list = [
               (distances[i, j], (i, j)) for i in range(n) for j in range(i + 1, n)
           ]

           distances_list.sort(reverse=True)

           legend_handles = []
           for dist, (i, j) in distances_list:
               (line,) = plt.plot(
                   [reduced_embeddings[i, 0], reduced_embeddings[j, 0]],
                   [reduced_embeddings[i, 1], reduced_embeddings[j, 1]],
                   "gray",
                   linestyle="--",
                   alpha=0.3,
               )
               legend_handles.append(line)

           legend_labels = [
               f"{text_labels[i]} - {text_labels[j]}: {dist:.2f}"
               for dist, (i, j) in distances_list
           ]

           for i in range(len(reduced_embeddings)):
               for j in range(i + 1, len(reduced_embeddings)):
                   plt.plot(
                       [reduced_embeddings[i, 0], reduced_embeddings[j, 0]],
                       [reduced_embeddings[i, 1], reduced_embeddings[j, 1]],
                       "gray",
                       linestyle="--",
                       alpha=0.5,
                   )

           plt.legend(
               legend_handles,
               legend_labels,
               title="Distances",
               loc="center left",
               bbox_to_anchor=(1, 0.5),
           )

           plt.tight_layout()
           plt.title(
               "2D Visualization of recipe similarities", fontsize=16, fontweight="bold"
           )
           plt.xlabel("PCA Component 1", fontdict=font_style)
           plt.ylabel("PCA Component 2", fontdict=font_style)

           plt.savefig(file_name, bbox_inches="tight")
           plt.close()

       chain(
           get_embeddings,
           plot_embeddings(
               get_embeddings.output,
               text_labels=countries_list,
               file_name=IMAGE_PATH,
           ),
       )


   recipe_suggestions()
   ```

   This DAG consists of five tasks to make a simple MLOps pipeline.

   * The `get_ingredients` task fetches the list of ingredients that the user found in their pantry and wants to use in their recipe. The input `pantry_ingredients` param is provided by [Airflow params](/docs/learn/airflow-params) when you run the DAG.
   * The `get_countries` task uses [Airflow params](/docs/learn/airflow-params) to retrieve the list of user-provided countries to get recipes from.
   * The `get_a_recipe` task uses the [`CohereHook`](https://airflow.apache.org/docs/apache-airflow-providers-cohere/stable/_api/airflow/providers/cohere/hooks/cohere/index.html) to connect to the Cohere API and use the [`/generate` endpoint](https://docs.cohere.com/reference/generate) to get a tasty recipe suggestion based on the user's pantry ingredients and one of the countries they provided. This task is [dynamically mapped](/docs/learn/dynamic-tasks) over the list of countries to generate one task instance per country. The recipes are saved as `.txt` files in the `include` folder.
   * The `get_embeddings` task uses the [`CohereEmbeddingOperator`](https://airflow.apache.org/docs/apache-airflow-providers-cohere/stable/operators/embedding.html) to generate vector embeddings of the recipes generated by the upstream `get_a_recipe` task. This task is dynamically mapped over the list of recipes to retrieve one set of embeddings per recipe. This pattern allows for efficient parallelization of the vector embedding generation.
   * The `plot_embeddings` task takes the embeddings created by the upstream task and performs dimensionality reduction using [PCA](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) to plot the embeddings in two dimensions.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the successful completion of the recipe_suggestions DAG in the Grid view with the Graph tab selected. 6 countries were provided to get recipes suggestions from, which led to 8 mapped task instances of both the get_a_recipe and get_embeddings task." />
   </Frame>

## Step 3: Run your DAG

1. Run `astro dev start` in your Astro project to start Airflow and open the Airflow UI at `localhost:8080`.

2. In the Airflow UI, run the `recipe_suggestions` DAG by clicking the play button. Then, provide [Airflow params](/docs/learn/airflow-params) for:

   * `Countries of recipe origin`: A list of the countries you want to get recipe suggestions from. Make sure to create one line per country and to provide at least two countries.
   * `pantry_ingredients`: A list of the ingredients you have in your pantry and want to use in the recipe. Make sure to create one line per ingredient.
   * `type`: Select your preferred recipe type.
   * `max_tokens_recipe`: The maximum number of tokens available for the recipe.
   * `randomness_of_recipe`: The randomness of the recipe. The value provided is divided by 10 and given to the [`temperature` parameter](https://docs.cohere.com/docs/temperature) of the Cohere API. The scale for the param ranges from 0 to 50, with 0 being the most deterministic and 50 being the most random.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the params available for the recipe_suggestions DAG with the default choices." />
   </Frame>

3. Go to the `include` folder to view the image file created by the `plot_embeddings` task. The image should look similar to the one below.

   <Frame>
     <img alt="Screenshot of the image created by the plot_embeddings task showing the two dimensional representation of the closeness of recipes associated with different countries." />
   </Frame>

## Step 4: (Optional) Cook your recipe

1. Choose one of the recipes in the `include` folder.
2. Navigate to your kitchen and cook the recipe you generated using Cohere with Airflow.
3. Enjoy!

## Conclusion

Congratulations! You used Airflow and Cohere to get recipe suggestions based on your pantry items. You can now use Airflow to orchestrate Cohere operations in your own machine learning pipelines.


# Orchestrate AI tasks with Apache Airflow® and the Common AI provider
Source: https://astronomer.io/docs/learn/airflow-common-ai-provider

Use the Common AI provider to add AI-based tasks to your Dags, including AI agents with access to Airflow-based tools.

The [Airflow Common AI provider](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/index.html) is an [Airflow provider package](https://airflow.apache.org/docs/apache-airflow-providers/index.html) that contains several operators and other modules to add AI-based tasks to your Dags, from simple LLM calls to AI agents with access to Airflow-based tools. It is built on top of [PydanticAI](https://pydantic.dev/docs/ai/overview/) and can be used with any [compatible model provider](https://pydantic.dev/docs/ai/models/overview/), including OpenAI, Anthropic, Gemini, AWS Bedrock, HuggingFace, and more.

In this guide you'll learn:

* Basic AI concepts to understand how to use the Common AI provider.
* How to install the Common AI provider and connect Airflow to your model provider.
* How to use the Common AI decorators and operators.
* How to add toolsets, durable execution, and human-in-the-loop review to your agentic tasks.

## Assumed knowledge

To get the most out of this guide, you should have an existing knowledge of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Airflow operators and decorators. See [Airflow operators](/docs/learn/what-is-an-operator) and [Airflow decorators](/docs/learn/airflow-decorators).
* Airflow hooks. See [Airflow hooks](/docs/learn/what-is-a-hook).

## Concepts

The Common AI provider abstracts calls to LLMs (large language models) and LMMs (large multimodal models), often as an AI agent with tool access. The calls are made through PydanticAI, which provides a consistent interface for [all compatible model providers](https://pydantic.dev/docs/ai/models/overview/).

| Concept                    | Description                                                                                                                                                                                                                                                   |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| LLM                        | Large language model. A model that takes text input and generates text output.                                                                                                                                                                                |
| LMM                        | Large multimodal model. A model that takes input from multiple modalities (text, images, audio, video, and other inputs) and generates output in one or more modalities.                                                                                      |
| [AI agent](#@task-agent)   | An LLM or LMM that has access to a set of tools. Typically, agents perform multiple steps of output generation and tool calls to achieve a goal. The goal of an agent can range from providing output to performing complex tasks involving multiple systems. |
| [Tools](#toolsets)         | Functions that an AI agent can call. Agents typically use tools to interact with an MCP or API to perform an action in another system. For example, retrieving data from a database or writing a file to object storage.                                      |
| [MCP](#pre-built-toolsets) | [Model Context Protocol](https://modelcontextprotocol.io/docs/getting-started/intro), a standard way for agents to connect to tools. You can think of an MCP server as an agent-callable interface for a tool or data source, often wrapping an API.          |

<Warning>
  As soon as [tools](#toolsets) are added to an [AI agent](#@task-agent), the agent can independently call that tool and perform any action that is possible through the tool. If you are adding tools, especially custom tooling with access to production systems, make sure to scope the access AI agents have to prevent destructive actions like dropping a table or deleting a file. **Never rely on instructions in a prompt** to restrict your agent from performing actions, always enforce access control programmatically.
</Warning>

<Note>
  Keep in mind that LLMs and LMMs aren't deterministic and therefore tasks using these models aren't **idempotent**. This means you might get different results when rerunning or backfilling Dags that contain AI-based tasks, even if all inputs are the same.
</Note>

## Install the Common AI provider

To use the Common AI provider, you need to be at least on Airflow 3.0 and install it by adding it to your `requirements.txt` file. Make sure to pin the latest [version](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/changelog.html).

```text wrap theme={null}
apache-airflow-providers-common-ai==<version>
```

Most operators in the Common AI provider depend on modules from the [Airflow standard provider](https://airflow.apache.org/docs/apache-airflow-providers-standard/stable/index.html), which is pre-installed when using the Astro CLI. Additionally, you'll need to install the [Airflow Common SQL provider](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/stable/changelog.html) when using [`SQLToolset`](#pre-built-toolsets), [`@task.llm_sql`](#@task-llm_sql), [`@task.llm_schema_compare`](#@task-llm_schema_compare), or other features that read database metadata through a `DbApiHook`.

```text wrap theme={null}
apache-airflow-providers-standard==<version>
apache-airflow-providers-common-sql==<version>
```

## Set the PydanticAI connection

The Common AI provider uses the same operators and decorators across many model providers by interacting with them through PydanticAI. You can you switch providers by updating the [Airflow connection](/docs/learn/connections).

The connection has the following format:

```text wrap theme={null}
AIRFLOW_CONN_PYDANTICAI_DEFAULT='{
    "conn_type": "pydanticai",
    "host": "<your_host>",
    "password": "<your_api_key>",
    "extra": {
        "model": "<your_provider>:<your_model>"
    }
}'
```

You can set a default model for the connection by specifying `model` in the connection `extra` field in the format `<your_provider>:<your_model>` (for example `anthropic:claude-opus-4-7`). Models can be overridden at the task level, as long as the model is accessible through the provided Airflow connection.

## Decide which decorator to use

The following table lists each decorator and operator in the Common AI provider and describes typical use cases.

| Decorator or operator                                                                | Description                                                                                                                                                                                                                                                                                                                                          |
| ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`@task.agent`](#@task-agent) (`AgentOperator`)                                      | Multi-step agent with toolsets, optional durable caching, or iterative human-in-the-loop review. Use it when a model needs access to external systems to perform a task. For example, an agent could assemble a feature proposal based on retrieving recent chat messages, the current product roadmap, and a support ticket with a feature request. |
| [`@task.llm`](#@task-llm) (`LLMOperator`)                                            | Single-turn model call with optional approval and optional direct edits to the output in the Airflow UI. Use this decorator when you only need a single model call.                                                                                                                                                                                  |
| [`@task.llm_sql`](#@task-llm_sql) (`LLMSQLQueryOperator`)                            | Natural language to SQL with optional access to the table schema, and SQL validation. This decorator works well to create Dags that can run ad-hoc queries [on demand](/docs/learn/airflow-event-driven-scheduling).                                                                                                                                      |
| [`@task.llm_branch`](#@task-llm_branch) (`LLMBranchOperator`)                        | Let the model pick one or more downstream tasks from [branches](/docs/learn/airflow-branch-operator). Use it when you want to pick different workflow paths based on unstructured input, for example routing support tickets to different teams.                                                                                                          |
| [`@task.llm_file_analysis`](#@task-llm_file_analysis) (`LLMFileAnalysisOperator`)    | Analyze files from object or local storage. This decorator is useful for document summarization across a set of files.                                                                                                                                                                                                                               |
| [`@task.llm_schema_compare`](#@task-llm_schema_compare) (`LLMSchemaCompareOperator`) | Compare table schemas across databases or mixed sources and summarize differences between them. Use this decorator when you have frequent schema changes that you need to detect to adjust downstream tasks.                                                                                                                                         |

## `@task.agent`

The `@task.agent` decorator and `AgentOperator` let you run a PydanticAI agent as a task in your Airflow Dag.

```python expandable wrap theme={null}
from datetime import timedelta
from airflow.sdk import task
from pydantic import BaseModel
from pydantic_ai import FunctionToolset


class MyToolset(FunctionToolset): ... # your custom toolset

class MyOutputClass(BaseModel): ... # your custom output class


@task.agent(
    llm_conn_id="pydanticai_default",  # required: your Airflow connection ID
    model_id="<your_provider>:<your_model>",  # default: None (uses the model set in the connection)
    # Optional parameters:
    system_prompt="<your_system_prompt>",  # default: ""
    output_type=MyOutputClass,  # can be a primitive type or a subclass of pydantic.BaseModel, default: str
    toolsets=[MyToolset()],  # can be set to a list of pydantic_ai.toolset instances and/or Airflow hooks, default: None
    enable_tool_logging=True,  # default: True
    agent_params={"tool_timeout": 60.0},  # default: None
    durable=False,  # default: False
    enable_hitl_review=True,  # default: False
    max_hitl_iterations=5,  # default: 5
    hitl_timeout=timedelta(minutes=5),  # default: None
    hitl_poll_interval=10.0,  # default: 10.0
)
def my_agentic_task(my_input: str) -> str:
    return f"This is the user prompt! {my_input}"


my_agentic_task(my_input="Say hello!")
```

The string returned by the `@task.agent` decorated function is the *user prompt input* to the agent. When using the `AgentOperator` directly, providing a user prompt with the `prompt` parameter is required.

The only other mandatory parameter is `llm_conn_id`, the [Airflow connection ID](#set-the-pydanticai-connection) to use for the LLM.

The following optional parameters are available for the `@task.agent` decorator:

* `model_id`: The model to use for the agent. If not given, the operator uses the model set in the [Airflow connection](#set-the-pydanticai-connection). You need to set the model in the format `<your_provider>:<your_model>`, for example `anthropic:claude-opus-4-7`.
* `system_prompt`: The system prompt to use for the agent, which is loaded into context before the user prompt. It is common to give general instructions to the agent in the system prompt about their role and task, as well as information about available tools. Note that instructions given in the prompt aren't guaranteed to be followed.
* [`output_type`](#output-type): The format for the output from the agent. You can use primitive types like `str`, `int`, `float`, `bool`, or a subclass of [Pydantic BaseModel](https://pydantic.dev/docs/validation/latest/api/pydantic/base_model/) for more complex structured outputs. Defaults to `str`. The `output_type` format is enforced, in contrast to instructions in the prompt.
* [`toolsets`](#toolsets): The toolsets the agent has access to as a list of [`pydantic_ai`.toolset](https://pydantic.dev/docs/ai/api/pydantic-ai/toolsets/) instances. Defaults to `None`.
* `enable_tool_logging`: If `True`, every toolset is wrapped in a `LoggingToolset` that logs tool calls with timing at INFO level and arguments at DEBUG level. Defaults to `True`.
* `agent_params`: Additional keyword arguments passed to the agent constructor. See the [PydanticAI agent documentation](https://pydantic.dev/docs/ai/api/pydantic-ai/agent/#parameters) for a list of available parameters.
* `durable`: Whether to enable step-level caching of model responses and tool results. Note that in order to use durable execution you need to set `AIRFLOW__COMMON_AI__DURABLE_CACHE_PATH`. See [durable execution](#durable-execution). Defaults to `False`. Can't be used with human-in-the-loop review.
* [`enable_hitl_review`](#human-in-the-loop-review): Whether to enable human-in-the-loop review through an Airflow plugin. Defaults to `False`. Needs Airflow 3.1+ and can't be used with durable execution.
* `max_hitl_iterations`: The maximum number of iterations of human-in-the-loop review. Defaults to `5`.
* `hitl_timeout`: The timeout for the human-in-the-loop review as a timedelta object. Defaults to `None`.
* `hitl_poll_interval`: The interval between polling for human-in-the-loop review. Defaults to `10.0`.

### Toolsets

AI agents can use tools to perform actions in another system. Tools are typically implemented as functions that can be called by the agent. A toolset is a collection of such functions in a class.

When using the Common AI provider, your agents can use three types of toolsets:

* Pre-built toolsets included in the Common AI provider: `SQLToolset` and `MCPToolset`.
* Use any [Airflow hook](/docs/learn/what-is-a-hook) as a toolset by wrapping it in the `HookToolset` class, the hook methods are the tools the agent can call.
* Custom toolsets you can implement yourself by subclassing a [`pydantic_ai`.toolset](https://pydantic.dev/docs/ai/api/pydantic-ai/toolsets/#functiontoolset) class.

#### Pre-built toolsets

The [SQLToolset](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/_api/airflow/providers/common/ai/toolsets/sql/index.html) is a curated toolset that gives your agent access to a SQL database, only allowing the following methods: `list_tables`, `get_schema`, `query` (`SELECT` only by default), and `check_query`. It can be used with any SQL database that is supported by the `DbApiHook`.

```python wrap theme={null}
from airflow.providers.common.ai.toolsets.sql import SQLToolset

toolsets=[
    SQLToolset(
        db_conn_id="my_sql_conn",
        allowed_tables=["my_table"],  # restrict which tables the agent can access through list_tables and get_schema
        schema="my_schema",  # Database schema/namespace for table listing and introspection.
        allow_writes=False,  # If True, modify operations are allowed (insert, update, delete). Default: False.
        max_rows=1000  # Maximum number of rows to return from a query. Default: 50.
    )
],
```

<Warning>
  The `allowed_tables` parameter does *not* parse or validate table references in SQL queries. An LLM can still query tables outside this list if it guesses the name. For query-level restrictions, use database-level permissions (for example a read-only role with grants limited to specific tables).

  Setting `allow_writes=True` allows the agent to perform modify operations (insert, update, delete) on the database. Use at your own risk.
</Warning>

The [MCPToolset](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/_api/airflow/providers/common/ai/toolsets/mcp/index.html) is a toolset that gives your agent access to an MCP server. It uses the `MCPHook` to retrieve credentials from an [Airflow connection](/docs/learn/connections) and creates a [PydanticAI MCP server instance](https://pydantic.dev/docs/ai/api/pydantic-ai/mcp/).

```python wrap theme={null}
from airflow.providers.common.ai.toolsets.mcp import MCPToolset

toolsets=[MCPToolset(mcp_conn_id="mcp_default")],
```

Set the [MCP server connection](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/connections/mcp.html) as an Airflow connection. There are two main types of transports for MCP servers:

* **Streamable HTTP**: Uses HTTP to stream responses, recommended for remote servers. If your `host` doesn't require authentication, you can omit the `password` field.

  ```text wrap theme={null}
  AIRFLOW_CONN_MCP_DEFAULT='{
      "conn_type": "mcp",
      "host": "http://localhost:3001/mcp",
      "password": "<your_auth_token>",  
      "extra": {
          "transport": "http"
      }
  }'
  ```

* **stdio**: Runs the MCP server as a subprocess communicating over stdin/stdout.

  ```text wrap theme={null}
  AIRFLOW_CONN_MCP_DEFAULT='{
      "conn_type": "mcp",
      "extra": {
          "transport": "stdio", 
          "command": "uvx", 
          "args": ["-m", "mcp_server"] 
      }
  }'
  ```

#### Use Airflow hooks as toolsets

You can use any [Airflow hook](/docs/learn/what-is-a-hook) as a toolset by wrapping it in the `HookToolset` class. The hook methods are the tools the agent can call. The `allowed_methods` parameter is required to make methods available to the agent, auto-discovery is intentionally disabled for safety purposes.

```python wrap theme={null}
from airflow.providers.common.ai.toolsets.hook import HookToolset

toolsets=[
    HookToolset(
        hook=MyHook(conn_id="my_conn"),
        allowed_methods=["method1", "method2"],
        tool_name_prefix="my_prefix_", # optional, default: ""
    )
],
```

You can explore available hooks in the [Airflow registry](https://airflow.apache.org/registry/).

### Human-in-the-loop review

Human-in-the-loop review lets you iterate on the output of an agentic task by reviewing the agent output and providing feedback. An Airflow plugin adds a **HITL Review** tab to the task instance page in the Airflow UI. This is different from [human-in-the-loop operators](/docs/learn/airflow-human-in-the-loop), which surface decisions under **Required Actions**.

```python wrap theme={null}
from airflow.sdk import task
from datetime import timedelta

@task.agent(
    enable_hitl_review=True,
    max_hitl_iterations=5,
    hitl_timeout=timedelta(minutes=5),
    hitl_poll_interval=10.0,
)
def my_agentic_task(my_input: str) -> str:
    return f"This is the user prompt! {my_input}"


my_agentic_task(my_input="Say hello!")
```

On each iteration you have three options:

* **Approve**: The task succeeds with the current assistant output and the HITL review ends.
* Provide feedback to the agent and click **Send**: The agent will take the feedback as user prompt and regenerate the output. The maximum number of iterations is controlled by the `max_hitl_iterations` parameter.
* **Reject**: The task fails and the next iteration isn't started.

<Frame>
  <img alt="Airflow UI for an agent task instance with the HITL Review tab selected: multi-turn thread with user feedback and assistant replies by iteration, a feedback field, Send, and Approve and Reject actions." />
</Frame>

The `hitl_timeout` parameter provides an optional time limit for the entire HITL review phase, that is, all review rounds combined. When that limit is exceeded, the task fails. The `hitl_poll_interval` parameter is the number of seconds to wait between polls for the reviewer action on the human-action XCom key.

<Tip>
  You can have human-in-the-loop steps in Airflow Dags outside of agentic tasks by using standalone [HITL operators](/docs/learn/airflow-human-in-the-loop).
</Tip>

### Durable execution

The `durable` parameter allows you to enable step-level caching of model responses and tool results. When a task is retried, steps that have already finished don't run again; the task reads their results from the cache instead.

To enable durable mode you need to set the `AIRFLOW__COMMON_AI__DURABLE_CACHE_PATH` [environment variable](/docs/astro/manage-env-vars) to the path where the cache will be stored. The destination can be a local temporary directory on the worker or remote object storage like S3 or GCS. After successful task execution the cache is deleted.

```text wrap theme={null}
AIRFLOW__COMMON_AI__DURABLE_CACHE_PATH=/path/to/cache
```

To use remote object storage, add your connection ID to the path and provide the credentials in the connection, similar to the configuration of an [Object Storage XCom Backend](/docs/learn/custom-xcom-backends-tutorial). To use S3 as the durable cache destination, set the following:

```text wrap theme={null}
AIRFLOW__COMMON_AI__DURABLE_CACHE_PATH=s3://<your_conn_id>@my-bucket/some/prefix/

AIRFLOW_CONN_<your_conn_id>='{
    "conn_type": "aws",
    "login": "<your-aws-access-key>",
    "password": "<your-aws-secret-key>",
    "extra": {
        "region_name": "<your-region>"
    }
}'
```

To run an agent in durable mode set `durable=True`.

```python wrap theme={null}
from airflow.sdk import task

@task.agent(
    durable=True,
)
```

When a durable agent retries and uses previously cached results you'll see a log message like this:

```text wrap theme={null}
[2026-04-26 20:37:33] INFO - Durable: replayed 3 cached steps (2 model, 1 tool), executed 4 new steps (2 model, 2 tool) 
```

<Note>
  You can only use `durable=True` if `enable_hitl_review=False`.
</Note>

### Output type

Pass `output_type` to set the format of the agent output. It can be a primitive type like `str`, `int`, `float`, `bool`, or a subclass of [Pydantic BaseModel](https://pydantic.dev/docs/validation/latest/api/pydantic/base_model/) for more complex structured outputs. Use the `Field` class to add descriptions to the fields for the agent to use.

```python wrap theme={null}
from typing import Literal

from pydantic import BaseModel, Field

class MyOutputClass(BaseModel):
    my_field_one: Literal["A", "B", "C", "D", "F"] = Field(
        description=(
            "Letter grade for xyz..."
        )
    )
    my_field_two: str = Field(
        description="String that describes xyz..."
    )
    my_field_three: list[str] = Field(
        description=(
            "List of strings that describe xyz..."
        )
    )
    my_field_four: int = Field(
        description=(
            "Number that describes xyz..."
        )
    )
```

Using the above `MyOutputClass` as the `output_type` parameter the agentic task will always produce a JSON output with the fields and their values.

```python wrap theme={null}
{
    "my_field_one": "A",
    "my_field_two": "In summary...",
    "my_field_three": ["In detail...", "In detail..."],
    "my_field_four": 10
}
```

## `@task.llm`

The [`@task.llm` decorator and `LLMOperator`](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/operators/llm.html) make a single turn LLM call and return the model output.

```python wrap theme={null}
from datetime import timedelta
from airflow.sdk import task

@task.llm(
    llm_conn_id="pydanticai_default",  # required: your Airflow connection ID
    model_id="<your_provider>:<your_model>",  # default: None (uses the model set in the connection)
    # Optional parameters:
    system_prompt="",  # default: ""
    output_type=str,  # default: str; use a BaseModel subclass for structured output
    agent_params=None,  # default: None; passed to the PydanticAI Agent constructor
    require_approval=False,  # default: False; defer for human approve or reject in the UI
    approval_timeout=timedelta(minutes=10),  # default: None; max wait for approval
    allow_modifications=False,  # default: False; reviewer may edit text before approve
)
def my_llm_task(user_context: str) -> str:
    return user_context


my_llm_task()
```

The string your `@task.llm` callable returns is the user prompt sent to the model. When you using `LLMOperator` directly, the `prompt` argument is required.

Optional parameters:

* `model_id`: Overrides the model in the connection `extra` (format: `<provider>:<model>`).
* `system_prompt`: System instructions loaded before the user prompt.
* `output_type`: Return type for the run. Defaults to `str`. For structured JSON, set a subclass of [Pydantic `BaseModel`](https://pydantic.dev/docs/validation/latest/api/pydantic/base_model/), see [Output type](#output-type).
* `agent_params`: Extra keyword arguments for the PydanticAI `Agent` constructor (for example `model_settings`). See the [PydanticAI Agent parameters](https://pydantic.dev/docs/ai/api/pydantic-ai/agent/#parameters). Despite performing a single turn LLM call, you can still pass arguments to the agent constructor when using `@task.llm`.
* `require_approval`: When `True`, the task defers after generation until a human approves or rejects through the approval UI. Defaults to `False`.
* `approval_timeout`: Time limit to wait for a review when `require_approval` is `True`. When it is exceeded, the task fails. Defaults to `None`.
* `allow_modifications`: When `True` with approval enabled, the reviewer can edit the generated text before approval; that edited value becomes the task result. Defaults to `False`.

When `require_approval` is `True`, the task defers after the model returns its output. In the Airflow UI, open the task instance, select the **Required Action** tab, read the generated text, edit the output if needed, and then click **Approve** or **Reject**.

<Frame>
  <img alt="Airflow UI for a deferred llm task instance on the Required Action tab: Markdown model output (events, pilot notes, cargo), an editable output field with optional edits before approval, and Approve and Reject controls." />
</Frame>

<Note>
  Human-in-the-loop review behaves differently for `@task.llm` and `@task.agent`. With `@task.llm`, the task [defers](/docs/learn/deferrable-operators) after generation until a human approves or rejects through the approval UI. There is only one approval step, and the reviewer can edit the model output before approval when `allow_modifications` is `True`. With `@task.agent` and [`enable_hitl_review=True`](#human-in-the-loop-review), the task doesn't defer; it keeps running after output generation and waits on the **HITL Review** tab for approval, rejection, or feedback that triggers another iteration. Use `max_hitl_iterations` to cap how many review rounds run.
</Note>

<Tip>
  You can have human-in-the-loop steps in Airflow Dags outside of agentic tasks by using standalone [HITL operators](/docs/learn/airflow-human-in-the-loop).
</Tip>

## `@task.llm_sql`

The [`@task.llm_sql` decorator and `LLMSQLQueryOperator`](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/operators/llm_sql.html) turn natural language into SQL. The operator can pull table metadata through a `DbApiHook` from `db_conn_id`, or you can supply a full schema string yourself. It generates SQL only; it doesn't run queries. Downstream tasks (for example [`SQLExecuteQueryOperator`](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/stable/_api/airflow/providers/common/sql/operators/sql/index.html#airflow.providers.common.sql.operators.sql.SQLExecuteQueryOperator)) can execute the string returned in XCom.

```python wrap theme={null}
from datetime import timedelta

from airflow.sdk import task


@task.llm_sql(
    llm_conn_id="pydanticai_default",  # required: your Airflow connection ID
    db_conn_id="postgres_default",  # optional: connection that resolves to DbApiHook for accessing table schema
    table_names=["orders", "customers"],  # optional: tables to describe when using db_conn_id
    schema_context=None,  # optional: manual schema text; when set, skips db_conn_id access
    validate_sql=True,  # default True: validate generated SQL with sqlglot
    dialect=None,  # optional: for example "postgres"; inferred from the hook when None
    model_id="<your_provider>:<your_model>",  # optional, default None
    system_prompt="",  # optional: appended to the built-in SQL safety instructions
    agent_params=None,  # optional: passed to the PydanticAI Agent constructor
    require_approval=False,
    approval_timeout=timedelta(minutes=10),
    allow_modifications=False,
)
def my_nl_sql_task(question: str) -> str:
    return question


my_nl_sql_task()
```

The string your callable returns is the natural language `prompt` that describes the query you want. When you use `LLMSQLQueryOperator` directly, pass that text with the `prompt` argument. You must set `llm_conn_id` to the [PydanticAI connection](#set-the-pydanticai-connection).

Additional parameters:

* `db_conn_id`: Airflow connection used to access metadata about your database. The hook must be a `DbApiHook`. Omit when you fully describe the schema with `schema_context`.
* `table_names`: List of table names to include when accessing the database with `db_conn_id`.
* `schema_context`: Free-form schema description. When you set this, the operator doesn't access the database for metadata.
* `validate_sql`: When `True` (default), generated SQL is checked with [sqlglot](https://sqlglot.com/sqlglot.html) before the task finishes.
* `allowed_sql_types`: Tuple of allowed statement roots (defaults to read-only shapes such as `Select`, `Union`, `Intersect`, and `Except` in sqlglot). Be careful when adding writing statements like `Insert`, `Update`, or `Delete`.
* `dialect`: sqlglot dialect name (for example `postgres`, `mysql`). When `None`, the operator tries to infer it from the database hook.
* `datasource_config`: Optional extra configuration for the data source side of generation (see the provider source for structure when you need it).

Parameters inherited from `LLMOperator` (`model_id`, `system_prompt`, `output_type`, `agent_params`, `require_approval`, `approval_timeout`, `allow_modifications`) behave like they do for `@task.llm`. When `require_approval` is `True` and `allow_modifications` is `True`, a reviewer can edit the generated SQL; the provider re-validates edited SQL against the `allowed_sql_types` rules before returning it.

<Frame>
  <img alt="Airflow UI for a deferred llm_sql task instance on the Required Action tab: natural language prompt about available spacecraft, generated SELECT on the spacecraft table, an editable output field for the SQL, and Approve and Reject controls." />
</Frame>

## `@task.llm_branch`

The [`@task.llm_branch` decorator and `LLMBranchOperator`](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/operators/llm_branch.html) extend the `LLMOperator` with [branching](/docs/learn/airflow-branch-operator). At run time the operator reads downstream task IDs from the Dag, exposes them to the model as a constrained Enum through PydanticAI structured output, and skips tasks the model doesn't select. Note that you need to create a [dependency](/docs/learn/managing-dependencies) between the branch task and its downstream candidates.

```python expandable wrap theme={null}
from airflow.sdk import dag, task


@dag
def example_llm_branch():
    @task.llm_branch(
        llm_conn_id="pydanticai_default",
        model_id="<your_provider>:<your_model>",  # optional, default None
        allow_multiple_branches=False,  # default False
        system_prompt="Route support tickets to the right team.",
        agent_params=None,  # optional: passed to the PydanticAI Agent constructor
    )
    def route_ticket(message: str) -> str:
        return f"Route this support ticket: {message}"

    @task
    def handle_billing():
        return "Handling billing issue"

    @task
    def handle_auth():
        return "Handling auth issue"

    @task
    def handle_general():
        return "Handling general issue"

    chain(
        route_ticket("I was charged twice for my subscription."), 
        [
            handle_billing(),
            handle_auth(),
            handle_general(),
        ]
    )


example_llm_branch()
```

The string your callable returns is the user `prompt`. When you use `LLMBranchOperator` directly, pass that text with the `prompt` argument. Set `llm_conn_id` to the [PydanticAI connection](#set-the-pydanticai-connection). Additionally, you can set `allow_multiple_branches` to `True` to allow the model to return multiple downstream task IDs.

Parameters inherited from `LLMOperator` (`model_id`, `system_prompt`, `agent_params`, and the same optional human-in-the-loop approval fields as [`@task.llm`](#@task-llm)) behave the same way as for a plain LLM task.

## `@task.llm_file_analysis`

The [`@task.llm_file_analysis` decorator and `LLMFileAnalysisOperator`](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/operators/llm_file_analysis.html) analyze one file, a prefix, or a small set of files through a single LLM call. The operator resolves `file_path` with [`ObjectStoragePath`](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html), normalizes supported formats into text context, and optionally attaches PNG, JPG, or PDF inputs as multimodal payloads when `multi_modal=True`. For object storage URIs you can embed the connection ID in the path (for example `s3://<conn_id>@my-bucket/prefix/`) or set `file_conn_id` separately.

```python wrap theme={null}
from datetime import timedelta
from airflow.sdk import , task

@task.llm_file_analysis(
    llm_conn_id="pydanticai_default",
    file_path="s3://aws_default@my-bucket/reports/quarterly.pdf",
    file_conn_id=None,  # optional: overrides connection embedded in file_path
    multi_modal=True,  # default False; set True for vision-capable models on images/PDF
    max_files=20,  # default 20; caps files resolved from a prefix
    max_file_size_bytes=5 * 1024 * 1024,  # default 5 MiB per file
    max_total_size_bytes=20 * 1024 * 1024,  # default 20 MiB total
    max_text_chars=100_000,  # default 100000; how much text to read from the files
    sample_rows=10,  # default 10; preview rows for CSV, Parquet, Avro
    model_id="<your_provider>:<your_model>",
    system_prompt="",
    output_type=str,  # default str
    agent_params=None,
    require_approval=False,
    approval_timeout=timedelta(minutes=10),
    allow_modifications=False,
)
def review_quarterly_report() -> str:
    return "Extract the key revenue, risk, and compliance findings from this report."

review_quarterly_report()

```

The string your callable returns is the analysis `prompt`. When you use `LLMFileAnalysisOperator` directly, pass that text with the `prompt` argument.

Additional parameters:

* `file_path`: File or prefix to analyze (local paths or object storage paths supported by Airflow object storage).
* `file_conn_id`: Optional Airflow connection for the storage backend when it isn't embedded in `file_path`.
* `multi_modal`: When `True`, PNG, JPG, JPEG, and PDF inputs can be sent as binary attachments; requires a multimodal-capable model.
* `max_files`, `max_file_size_bytes`, `max_total_size_bytes`, `max_text_chars`, `sample_rows`: Guardrails for listing, reading, and how much normalized text or row samples reach the model. Extra files under a large prefix are omitted and the operator notes that in the prompt context.

Optional dependencies: Parquet and Avro handling need the provider extras described in the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/operators/llm_file_analysis.html).

## `@task.llm_schema_compare`

The [`@task.llm_schema_compare` decorator and `LLMSchemaCompareOperator`](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/operators/llm_schema_compare.html) read schema metadata from two or more systems and ask an LLM to flag differences. The task result is a dict `SchemaCompareResult` with fields such as `compatible`, `mismatches`, and `summary`. Each entry in `mismatches` is a `SchemaMismatch` with severity, column, types, and suggested actions.

You can supply databases in two ways:

* `db_conn_ids` together with `table_names`: shorthand to compare the same logical table across connections. Each connection must resolve to a `DbApiHook`.
* `data_sources`: a list of [`DataSourceConfig`](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/stable/_api/airflow/providers/common/sql/config/index.html#airflow.providers.common.sql.config.DataSourceConfig) objects for more complex setups (for example object storage or catalog-backed sources combined with `db_conn_ids`).

```python wrap theme={null}
from datetime import timedelta
from airflow.sdk import task

@task.llm_schema_compare(
    llm_conn_id="pydanticai_default",
    db_conn_ids=["postgres_source", "snowflake_target"],
    table_names=["customers"],
    context_strategy="full",  # "full" (default) or "basic"; full adds keys and indexes
    model_id="<your_provider>:<your_model>",
    agent_params=None,
    require_approval=False,
    approval_timeout=timedelta(minutes=10),
    allow_modifications=False,
)
def check_migration_readiness() -> str:
    return (
        "Compare schemas and flag breaking changes for nightly ETL. "
        "Suggest migration actions where they help."
    )

check_migration_readiness()

```

The string your callable returns is the comparison `prompt`. When you use `LLMSchemaCompareOperator` directly, pass that text with the `prompt` argument.

Additional parameters:

* `db_conn_ids` and `table_names`: Use together for the same table name across multiple database connections.
* `data_sources`: Optional list of `DataSourceConfig` for mixed database and object storage comparisons.
* `context_strategy`: `"basic"` sends column names and types only; `"full"` (default) adds primary keys, foreign keys, and indexes to the context sent to the model.
* `system_prompt`: The operator ships a default prompt that encodes cross-system type rules and severity levels. If you set `system_prompt` to any string, it replaces that default; import [`DEFAULT_SYSTEM_PROMPT`](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/_api/airflow/providers/common/ai/operators/llm_schema_compare/index.html#airflow.providers.common.ai.operators.llm_schema_compare.DEFAULT_SYSTEM_PROMPT) and concatenate when you want to extend rather than replace.

Parameters inherited from `LLMOperator` (`model_id`, `agent_params`, and optional approval fields) match [`@task.llm`](#@task-llm). Downstream tasks can branch on `comparison_result["compatible"]` or inspect `mismatches` as shown in the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/operators/llm_schema_compare.html).


# Apache Airflow® components
Source: https://astronomer.io/docs/learn/airflow-components

Understand the core components of Apache Airflow®. Review their functions and find out which components to run for specific use cases.

When working with [Apache Airflow®](https://airflow.apache.org/), understanding the underlying infrastructure components and how they function can help you develop and run your DAGs, troubleshoot issues, and successfully run Airflow.

In this guide, you'll learn about the core components of Airflow.

<Note>
  There were significant changes to the Airflow architecture between Airflow 2 and Airflow 3, greatly improving Airflow's security posture and enabling new features such as remote execution. The most important impact of those changes for DAG authors is that directly accessing the metadata database from within Airflow tasks isn't possible anymore. See the [Upgrade from Apache Airflow® 2 to 3](/docs/learn/airflow-upgrade-2-3) guide and the [Airflow release notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html) for more information.
</Note>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).

## Core components

The following are the core components of Airflow:

* **Scheduler:** The scheduler is the heart of Airflow. It monitors all tasks and DAGs and schedules task instances to run as soon as their dependencies are fulfilled. When creating a new DAG run, the scheduler always picks the latest [version of that DAG](/docs/learn/airflow-dag-versioning). When a task is ready to run, the scheduler uses its configured [executor](/docs/learn/airflow-executors-explained) to run the task on a worker.
* **API server:** A FastAPI server that serves the Airflow UI, as well as three APIs:
  * An API for workers to interact with when running task instances.
  * An internal API for the Airflow UI that provides updates on dynamic UI components such as the state of task instances and DAG runs.
  * The public [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) that users can interact with.
* **DAG processor:** The DAG processor is responsible for retrieving and parsing the files from the location determined by the configured [DAG bundle(s)](/docs/learn/airflow-dag-versioning).
* **Metadata database:** The Airflow metadata database stores information vital to Airflow’s functioning, such as Airflow connections, serialized DAGs, and XCom information. It also contains the history of previous DAG runs and task instances alongside metadata about their states. The most common backend used for the Airflow metadata database is PostgreSQL. See the Airflow documentation for [supported versions](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html).
* **Triggerer:** A separate process which supports running asynchronous Python functions as part of trigger classes. The triggerer is needed to use [deferrable operators](/docs/learn/deferrable-operators) and [event-driven scheduling](/docs/learn/airflow-event-driven-scheduling).

If you run Airflow locally using the [Astro CLI](/docs/cli/v1.43/install-cli), you'll notice that when you start Airflow using `astro dev start`, it will spin up five containers, one for each of the core components.

```text wrap theme={null}
CONTAINER ID    IMAGE                [...]      PORTS       NAMES
f565...         [...]/airflow:latest [...]      >8080/tcp   [...]apiserver1
77e6…           [...]/airflow:latest [...]                  [...]dagprocessor1
88dc...         [...]/airflow:latest [...]                  [...]triggerer1
aa5a...         [...]/airflow:latest [...]                  [...]scheduler1
5f0e...         postgres:12.6        [...]      >5432/tcp   [...]postgres-1
```

In addition to these core components, one or more workers can be spun up. The type of workers used depends on the configured [executor](/docs/learn/airflow-executors-explained) of the scheduler.

## Component interaction

The following diagram shows how the core components interact with each other:

<Frame>
  <img alt="Architecture" />
</Frame>

On a high level, this is what happens when you add a simple new DAG to your Airflow environment:

1. The DAG is parsed by the DAG processor, which stores a serialized version of it in the Airflow metadata database.
2. The scheduler checks the serialized DAGs to determine whether any DAG is eligible for execution based on its defined [schedule](/docs/learn/scheduling-in-airflow). This process includes checking the schedules against the current time and checking for information such as updates to [assets](/docs/learn/airflow-datasets) or events fired by [AssetWatchers](/docs/learn/airflow-event-driven-scheduling).
3. When the scheduler determines that a DAG is ready for its next run, its configured [executor](/docs/learn/airflow-executors-explained) decides how and where to run the first task instance(s) of the DAG run.
4. Next, the task instance(s) are scheduled and subsequently queued. The workers poll the queue for any queued task instances they can run.
5. The worker who picked up the task instance runs it, and metadata such as the task instance status or [XCom](/docs/learn/airflow-passing-data-between-tasks) is sent from the worker via the API server to be stored in the Airflow metadata database. If the task needs any information, such as an [Airflow connection](/docs/learn/connections), the worker sends a request to the API server for this information. The API server retrieves the details from the Airflow metadata database and hands them back to the worker. As the task instance is running, the worker writes task instance logs directly to the defined log storage location.
6. Some of this information, such as the task instance status, is in turn important for the scheduler. It monitors all DAGs and, as soon as their dependencies are fulfilled, schedules task instances to run.
7. The scheduler needs the status of the task instances in the DAG to determine which other task instances now fulfill their dependencies and can be scheduled to run. While this process is going on in the background, the Airflow UI, served by the API server, displays information about the current DAG and task statuses that it retrieves from the Airflow metadata database.

## Manage Airflow infrastructure

All Airflow components should be run on an infrastructure that is appropriate for the requirements of your organization. For example, using the [Astro CLI](/docs/cli/v1.43/install-cli) to run Airflow on a local computer can be helpful when testing and for DAG development, but it is insufficient to support running DAGs in production.

The following resources can help you manage Airflow components:

* OSS [Production Docker Images](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html#using-production-docker-images)
* OSS [Official Helm Chart](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html#using-official-airflow-helm-chart)
* Managed Airflow on [Astro](https://www.astronomer.io/product/). A [free trial](https://www.astronomer.io/lp/signup) of Astro is available.

Scalability is also an important consideration when setting up your production Airflow environment. See [Scaling out Airflow](/docs/learn/airflow-scaling-workers).

## High availability

Airflow can be made highly available, which makes it suitable for large organizations with critical production workloads. Running multiple Scheduler replicas in an active-active model makes Airflow more performant and resilient, eliminating a single point of failure within your Airflow environment.

In Astro, you can configure high availability when creating a new deployment by enabling the **High Availability** toggle switch in the UI or by setting `isHighAvailability` to `true` using the [API](/docs/astro/api/v-1/overview) or [Terraform](https://registry.terraform.io/providers/astronomer/astro/latest/docs).


# Access the Apache Airflow context
Source: https://astronomer.io/docs/learn/airflow-context

Access the Airflow context in your tasks.

The Airflow context is a dictionary containing information about a running DAG and its Airflow environment that can be accessed from a task. One of the most common values to retrieve from the Airflow context is the [`ti` / `task_instance` keyword](#ti-/-task_instance), which allows you to access attributes and methods of the [`taskinstance` object](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/taskinstance/index.html).

Other common reasons to access the Airflow context are:

* You want to use [DAG-level parameters](/docs/learn/airflow-params) in your Airflow tasks.
* You want to use the DAG run's [logical date](/docs/learn/scheduling-in-airflow#dag-run-timestamps) in an Airflow task, for example as part of a file name.
* You want to explicitly push and pull values to [XCom](/docs/learn/airflow-passing-data-between-tasks#xcom) with a custom key.

Use this document to learn about the data stored in the Airflow context and how to access it.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Basic Python. See the [Python Documentation](https://docs.python.org/3/tutorial/index.html).
* Airflow operators. See [Airflow operators](/docs/learn/what-is-an-operator).

## Access the Airflow context

The Airflow context is available in all Airflow tasks. You can access information from the context using the following methods:

* Pass the `**context` argument to the function used in a [`@task` decorated task](/docs/learn/airflow-decorators) or [`PythonOperator`](https://airflow.apache.org/registry/providers/standard#standard-python-PythonOperator).
* Pass the `context` argument to a `@asset` decorated function. See [Assets and data-aware scheduling](/docs/learn/airflow-datasets) for more information.
* Use [Jinja templating](/docs/learn/templating) in traditional Airflow operators.
* Access the context `kwarg` in the `.execute` method of any traditional or custom operator.

You can't access the Airflow context dictionary outside of an Airflow task.

### Retrieve the Airflow context using the `@task` decorator or `PythonOperator`

To access the Airflow context in a `@task` decorated task or `PythonOperator` task, you need to add a `**context` argument to your task function. This will make the context available as a dictionary in your task.

The following code snippets show how to print out the full context dictionary from a task:

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # from airflow.sdk import task
  from pprint import pprint

  @task
  def print_context(**context):
      pprint(context)
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # from airflow.providers.standard.operators.python import PythonOperator
  from pprint import pprint

  def print_context_func(**context):
      pprint(context)

  print_context = PythonOperator(
      task_id="print_context",
      python_callable=print_context_func,
  )
  ```
</details>

### Retrieve the Airflow context using Jinja templating

Many elements of the Airflow context can be accessed by using [Jinja templating](/docs/learn/templating). You can get the list of all parameters that allow templates for any operator by printing out its `.template_fields` attribute.

For example, you can access a DAG run's logical date in the format `YYYY-MM-DD` by using the template `{{ ds }}` in the `bash_command` parameter of the `BashOperator`.

```python wrap theme={null}
# from airflow.providers.standard.operators.bash import BashOperator

print_logical_date = BashOperator(
    task_id="print_logical_date",
    bash_command="echo {{ ds }}",
)
```

It is also common to use Jinja templating to access [XCom](/docs/learn/airflow-passing-data-between-tasks#xcom) values in the parameter of a traditional task. In the code snippet below, the first task `return_greeting` will push the string "Hello" to XCom, and the second task `greet_friend` will use a Jinja template to pull that value from the `ti` (task instance) object of the Airflow context and print `Hello friend! :)` into the logs.

```python wrap theme={null}
# from airflow.providers.standard.operators.bash import BashOperator
# from airflow.sdk import task

@task 
def return_greeting():
    return "Hello"

greet_friend = BashOperator(
    task_id="greet_friend",
    bash_command="echo '{{ ti.xcom_pull(task_ids='return_greeting') }} friend! :)'",
)

return_greeting() >> greet_friend
```

Find an up to date list of all available templates in the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html). Learn more about using XComs to pass data between Airflow tasks in [Pass data between tasks](/docs/learn/airflow-passing-data-between-tasks).

### Retrieve the Airflow context using custom operators

In a traditional operator, the Airflow context is always passed to the `.execute` method using the `context` keyword argument. If you write a [custom operator](/docs/learn/airflow-importing-custom-hooks-operators), you have to include a `context` kwarg in the `execute` method as shown in the following custom operator example.

```python wrap theme={null}
from airflow.sdk.bases.operator import BaseOperator

class PrintDAGIDOperator(BaseOperator):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def execute(self, context):
        print(context["dag"].dag_id)
```

## Common Airflow context values

This section gives an overview of the most commonly used keys in the Airflow context dictionary. To see an up-to-date list of all keys and their types, view the [Airflow source code](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/context.py).

### ti / `task_instance`

The `ti` or `task_instance` key contains the [TaskInstance object](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/taskinstance/index.html). The most commonly used attributes are `.xcom_pull` and `.xcom_push`, which allow you to push and pull [XComs](/docs/learn/airflow-passing-data-between-tasks).

The following DAG shows an example of using `context["ti"].xcom_push(...)` and `context["ti"].xcom_pull(...)` to explicitly pass data between tasks.

```python expandable wrap theme={null}
from pendulum import datetime
from airflow.decorators import dag, task


@dag(
    start_date=datetime(2023, 6, 1),
    schedule=None,
    catchup=False,
)
def context_and_xcom():
    @task
    def upstream_task(**context):
        context["ti"].xcom_push(key="my_explicitly_pushed_xcom", value=23)
        return 19

    @task
    def downstream_task(passed_num, **context):
        returned_num = context["ti"].xcom_pull(
            task_ids="upstream_task", key="return_value"
        )
        explicit_num = context["ti"].xcom_pull(
            task_ids="upstream_task", key="my_explicitly_pushed_xcom"
        )

        print("Returned Num: ", returned_num)
        print("Passed Num: ", passed_num)
        print("Explicit Num: ", explicit_num)

    downstream_task(upstream_task())


context_and_xcom()
```

The `downstream_task` will print the following information to the logs:

```text wrap theme={null}
[2023-06-16, 13:14:11 UTC] {logging_mixin.py:149} INFO - Returned Num:  19
[2023-06-16, 13:14:11 UTC] {logging_mixin.py:149} INFO - Passed Num:  19
[2023-06-16, 13:14:11 UTC] {logging_mixin.py:149} INFO - Explicit Num:  23
```

### Scheduling keys

One of the most common reasons to access the Airflow context in your tasks is to retrieve information about the scheduling of their DAG. A common pattern is to use the timestamp of the logical date in names of files written from a DAG to create a unique file for each DAG run.

The task below creates a new text file in the `include` folder for each DAG run with the timestamp in the filename in the format `YYYY-MM-DDTHH:MM:SS+00:00`. Refer to [Templates reference](https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html) for an up to date list of time related keys in the context, and [Jinja templating](/docs/learn/templating) for more information on how to pass these values to templateable parameters of traditional operators.

```python wrap theme={null}
# from airflow.sdk import task

@task
def write_file_with_ts(**context):
    ts = context["ts"]
    with open(f"include/{ts}_hello.txt", "a") as f:
        f.write("Hello, World!")
```

### `dag_run`

The `dag_run` key contains the [DAG run object](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/models/dagrun.py). A commonly used attribute of the DAG run object is `run_type`, which indicates how the DAG was triggered.

```python wrap theme={null}
# from airflow.sdk import task

@task
def print_dagrun_info(**context):
    print(context["dag_run"].run_type)
```

### params

The `params` key contains a dictionary of all DAG- and task-level params that were passed to a specific task instance. Individual params can be accessed using their respective key.

```python wrap theme={null}
# from airflow.sdk import task

@task
def print_param(**context):
    print(context["params"]["my_favorite_param"])
```

Learn more about params in the [Airflow params guide](/docs/learn/airflow-params).

### var

The `var` key contains all Airflow variables of your Airflow instance. [Airflow variables](/docs/learn/airflow-variables) are key-value pairs that are commonly used to store instance-level information that rarely changes.

```python wrap theme={null}
# from airflow.sdk import task

@task
def get_var_from_context(**context):
    print(context["var"]["value"].get("my_regular_var"))
    print(context["var"]["json"].get("my_json_var")["num2"])
```

## Context parameters relating to timestamps

Your dag run type, that is scheduled vs asset-triggered, can determine which timestamp keys are available in the context. The following code snippet contains a task that prints out the full list of context keys available, as well as all keys relating to scheduling timestamps.

```python expandable wrap theme={null}
from airflow.sdk import dag, task, task_group, Asset

@dag
def my_context_dag():
    @task 
    def print_context_keys(**context):
        print("All context keys: ", context.keys())
        print("--------------")
        print("DAG run details relating to timestamps:")
        print("run_id from the dag_run key: ", context["dag_run"].run_id)
        print("logical_date from the dag_run key: ", context["dag_run"].logical_date)
        print("data_interval_start from the dag_run key: ", context["dag_run"].data_interval_start)
        print("data_interval_end from the dag_run key: ", context["dag_run"].data_interval_end)
        print("run_after from the dag_run key: ", context["dag_run"].run_after)
        print("start_date from the dag_run key: ", context["dag_run"].start_date) 
        print("end_date from the dag_run key: ", context["dag_run"].end_date)
        print("--------------")
        print("Top-level context keys relating to timestamps:")
        print("prev_start_date_success: ", context["prev_start_date_success"])
        print("prev_end_date_success: ", context["prev_end_date_success"])

        # The keys below are only available in a scheduled run, or manual run with a logical date provided
        # in an asset triggered run or a manual run with no logical date provided, these keys will be MISSING
        # causing a key error if used!
        print("logical_date: ", context["logical_date"])
        print("ds: ", context["ds"])
        print("ds_nodash: ", context["ds_nodash"])
        print("ts: ", context["ts"])
        print("ts_nodash: ", context["ts_nodash"])
        print("data_interval_start: ", context["data_interval_start"])        
        print("data_interval_end: ", context["data_interval_end"])
        print("previous_data_interval_start_success: ", context["prev_data_interval_start_success"])
        print("previous_data_interval_end_success: ", context["prev_data_interval_end_success"])

    print_context_keys()

my_context_dag()
```

Note that if your DAG is triggered by an asset or if you created a manual / API triggered run and set the logical date explicitly to `None`, the following keys will be missing from the context dictionary and trying to access them will raise a `KeyError`:

* `logical_date`
* `ds`
* `ds_nodash`
* `ts`
* `ts_nodash`
* `data_interval_start`
* `data_interval_end`
* `previous_data_interval_start_success`
* `previous_data_interval_end_success`


# DAG-level parameters in Airflow
Source: https://astronomer.io/docs/learn/airflow-dag-parameters

Learn about all important DAG-level parameters in Airflow.

In Airflow, you can configure when and how your DAG runs by setting parameters in the DAG object. DAG-level parameters affect how the entire DAG behaves, as opposed to task-level parameters which only affect a single task or [Airflow configs](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html) which affect the entire Airflow instance.

This guide covers all user-relevant DAG-level parameters in Airflow.

## Basic DAG-level parameters

There are four basic DAG-level parameters. It is best practice to always set these parameters in any DAG:

| Parameter    | Description                                                                                                                                                                                                                                                                                       |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dag_id`     | The name of the DAG. This must be unique for each DAG in the Airflow environment. When using the `@dag` decorator and not providing the `dag_id` parameter name, the function name is used as the `dag_id`. When using the `DAG` class, this parameter is required.                               |
| `start_date` | The date and time after which the DAG starts being scheduled. Note that the first actual run of the DAG may be later than this date depending on how you define the schedule. See [DAG scheduling and timetables in Airflow](/docs/learn/scheduling-in-airflow) for more information. Default: `None`. |
| `schedule`   | The schedule for the DAG. There are many different ways to define a schedule, see [Scheduling in Airflow](/docs/learn/scheduling-in-airflow) for more information. Defaults to `None`.                                                                                                                 |
| `catchup`    | Whether the scheduler should backfill all missed DAG runs between the current date and the start date when the DAG is unpaused. This parameter defaults to `False`. See [Catchup](/docs/learn/rerunning-dags#catchup) for more information.                                                            |

## UI parameters

Some parameters [add documentation](/docs/learn/custom-airflow-ui-docs-tutorial) to a DAG or change its appearance in the Airflow UI:

| Parameter     | Description                                                                                                                                                                                                                                          |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `description` | A short string that is displayed in the Airflow UI next to the DAG name.                                                                                                                                                                             |
| `doc_md`      | A string that is rendered as [DAG documentation](/docs/learn/custom-airflow-ui-docs-tutorial) in the Airflow UI. Tip: use `__doc__` to use the docstring of the Python file. It is a best practice to give all your DAGs a descriptive DAG documentation. |
| `tags`        | A list of tags shown in the Airflow UI to help with filtering DAGs.                                                                                                                                                                                  |

## Jinja templating parameters

There are parameters that relate to [Jinja templating](/docs/learn/templating), such as:

| Parameter                       | Description                                                                                                                                                                                                                                                                                                                       |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `template_searchpath`           | A list of folders where Jinja looks for templates. The path of the DAG file is included by default.                                                                                                                                                                                                                               |
| `template_undefined`            | The behavior of Jinja when a variable is undefined. Defaults to [StrictUndefined](https://jinja.palletsprojects.com/en/3.0.x/api/#jinja2.StrictUndefined).                                                                                                                                                                        |
| `render_template_as_native_obj` | Whether to render Jinja templates as native Python objects instead of strings. Defaults to `False`.                                                                                                                                                                                                                               |
| `user_defined_macros`           | A dictionary of macros that are available in the DAG's Jinja templates. Use `user_defined_filters` to add filters and `jinja_environment_kwargs` for additional Jinja configuration. See [Macros: using custom functions and variables in templates](/docs/learn/templating#macros-using-custom-functions-and-variables-in-templates). |

## Scaling

Some parameters can be used to scale your DAG's resource usage in Airflow. See [Scaling Airflow to optimize performance](/docs/learn/airflow-scaling-workers) for more information.

| Parameter                         | Description                                                                                                        |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `max_active_tasks`                | The number of task instances allowed to run concurrently one run of this DAG.                                      |
| `max_active_runs`                 | The number of active DAG runs allowed to run concurrently for this DAG.                                            |
| `max_consecutive_failed_dag_runs` | (experimental) The maximum number of consecutive failed DAG runs, after which the scheduler will disable this DAG. |

## Callback parameters

These parameters help you configure the behavior of [Airflow callbacks](/docs/learn/error-notifications-in-airflow#airflow-callbacks).

| Parameter             | Description                                                         |
| --------------------- | ------------------------------------------------------------------- |
| `on_success_callback` | A function to be executed after completion of a successful DAG run. |
| `on_failure_callback` | A function to be executed after a failed DAG run.                   |

<Tip>
  On Astro, you can use Astro alerts instead of or in addition to Airflow callbacks. See [When to use Airflow or Astro alerts for your pipelines on Astro](/docs/astro/best-practices/airflow-vs-astro-alerts) for more information.
</Tip>

## Other parameters

Other DAG parameters include:

| Parameter                 | Description                                                                                                                                                                                                                                                                                                                                                        |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `end_date`                | The date beyond which no further DAG runs will be scheduled. Defaults to `None`.                                                                                                                                                                                                                                                                                   |
| `default_args`            | A dictionary of parameters that are applied to all tasks in the DAG. These parameters are passed directly to each operator, so they must be parameters that are part of the `BaseOperator`. You can override default arguments at the task level.                                                                                                                  |
| `params`                  | A dictionary of DAG-level Airflow params. See [Airflow params](/docs/learn/airflow-params) for more information.                                                                                                                                                                                                                                                        |
| `dagrun_timeout`          | The time it takes for a DAG run of this DAG to time out and be marked as `failed`.                                                                                                                                                                                                                                                                                 |
| `access_control`          | Specify optional permissions for roles specific to an individual DAG. See [DAG-level permissions](https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html#dag-level-permissions). This can't be implemented on Astro. Astronomer recommends customers to use [Astro's RBAC features](/docs/astro/user-permissions) instead.                  |
| `is_paused_upon_creation` | Whether the DAG is paused when it is created. When not set, the Airflow config `core.dags_are_paused_at_creation` is used, which defaults to `True`.                                                                                                                                                                                                               |
| `auto_register`           | Defaults to `True` and can be set to `False` to prevent DAGs using a `with` context from being automatically registered which can be relevant in some advanced dynamic DAG generation use cases. See [Registering dynamic DAGs](https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html#registering-dynamic-dags).                 |
| `fail_fast`               | You can set this parameter to `True` to stop DAG execution as soon as one task in this DAG fails. Any tasks that are still running are marked as `failed`, and any tasks that haven't run yet are marked as `skipped`. Note that you can't have any [trigger rule](/docs/learn/airflow-trigger-rules) other than `all_success` in a DAG with `fail_fast` set to `True`. |
| `dag_display_name`        | Overrides the `dag_id` to display a different DAG name in the Airflow UI. This parameter allows special characters.                                                                                                                                                                                                                                                |
| `allowed_run_types`       | (Airflow 3.2+) A list of run types that are allowed for this Dag. Available run types are `scheduled`, `backfill`, `manual`, `asset_triggered`, and `asset_materialization`. If `manual` isn't in the list of allowed run types, a schedule has to be defined. By default all run types are allowed for a Dag.                                                     |


# Dag Versioning and Dag Bundles
Source: https://astronomer.io/docs/learn/airflow-dag-versioning

Learn how the Dag versioning feature functions and how to configure a versioned Dag bundle.

Dag versioning, the most frequently requested feature by the Airflow community, is available in Airflow 3.0! This feature allows you to track changes to your Dags over time in the Airflow UI, allowing you to see the complete history of your Dag runs. Dag versioning is automatic and doesn't require any setup. Additionally, versioned Dag bundles allow you to prevent version collisions during code pushes and rerun historical Dags using their original code.

This guide gives an introduction to Dag versioning and Dag bundles, including how to set up a versioned `GitDagBundle`.

## Assumed knowledge

To get the most out of this guide, you should have an existing knowledge of:

* Airflow Dags. See [Introduction to Apache Airflow® Dags](/docs/learn/dags).
* Basic GitHub concepts. See the [GitHub documentation](https://docs.github.com/en).

## Importance of Dag versioning

In Airflow 2, both the Airflow UI and Dag execution always used the latest Dag code. This led to two major constraints:

* **No observability of previous Dag versions**: If you changed a Dag and then, for example, removed a task, all history for previous runs of that task disappeared in the grid and graph view of the Airflow UI.
* **Version collisions during code pushes**: If the code of a Dag changed while a Dag was still running, some tasks of the same run might have been executed using the older version while others used the newer version. This situation carried a significant risk of unintended consequences. For example:
  * The older Dag version might use task A to retrieve the name of table X in a relational database for data insertion and task B to insert data into that table.
  * If the Dag was updated to change the table from X to Y in the middle of a run, task A (from the old version) might pass the table name X while the updated task B inserted data intended for table Y into table X.

Dag bundles and Dag versioning were introduced in Airflow 3 to address these issues.

## Dag versioning vs Dag bundles

Airflow 3 introduces two new concepts:

* **Dag versioning**: Airflow now keeps track of changes to your Dags. This is automatic and happens no matter which Dag bundle is used.
  * A new Dag version is created every time a Dag run is created for a Dag that has undergone a structural change since the last run. A structural change is any change that affects `serdag`. This includes changes to Dag or task parameters, task dependencies, task IDs or adding or removing tasks.
  * Each Dag run is associated with a Dag version that is visible in the Airflow UI.
  * Whenever a new Dag run is initiated, the scheduler uses the latest version of the Dag to create a run.
* **Dag bundle**: A collection of files containing Dag code and supporting files. Dag bundles are named after the backend they use to store the Dag code. For example, the `LocalDagBundle` uses the local file system to store Dag code, while the `GitDagBundle` uses a Git repository.
  * Some Dag bundles are versioned, such as the `GitDagBundle`. A version of a Dag bundle is created by versioning the underlying backend. For example, a new version of the `GitDagBundle` is created by every [Git](https://git-scm.com/doc) commit, whether or not any Dags change.
  * The default `LocalDagBundle` isn't versioned.

Dag versioning is automatic in Airflow 3 and doesn't require any setup. Using a Dag bundle other than `LocalDagBundle` requires changes to your Airflow configuration.

## Dag versioning

You can view Dag versions in several places in the Airflow UI. In the **Options** menu of the Dag graph, you can select which version of the Dag graph you want to display. The Dag details page also shows the latest available version of the Dag, which is used to create new Dag runs.

<Frame>
  <img alt="Dag versioning in the Airflow UI graph." />
</Frame>

The Dag grid now retains the history for all tasks, even if they were removed in the latest version of the Dag. You can also select which version of the Dag code you want to display in the **Code** tab.

<Frame>
  <img alt="Dag versioning in the Airflow UI grid and code tab." />
</Frame>

## Dag bundles

<Note>For Dags running on Astro using Hosted [execution mode](/docs/astro/execution-mode), a specialized versioned Dag bundle is configured automatically, without any need for additional setup. Configuring custom Dag bundles and using several Dag bundles in the same Astro Deployment is only supported when using [Remote Execution mode](/docs/astro/execution-mode#remote-execution). For more information, see [Configure Dag bundles for Remote Execution](/docs/astro/remote-execution-configure-dag-sources).</Note>

Dag bundles contain Dag code and supporting files. There are versioned and unversioned Dag bundles; the default Dag bundle (`LocalDagBundle`) isn't versioned, while the `GitDagBundle` is versioned. Support for other Dag bundle backends is planned for future releases.

Versioned and unversioned Dag bundles behave differently in the following situations:

* **Clearing/rerunning a previous Dag run**:

  * Unversioned Dag bundle: Airflow uses the current Dag code, that is, the latest version of the Dag.
  * Versioned Dag bundle: By default, the scheduler uses the Dag version that existed at the time of the Dag run to determine which task instances to create and the workers use the code contained in the Dag bundle version that existed at the time of the original Dag run to execute their tasks. You can configure the rerun behavior, for example by checking the **Run with latest bundle version** box on the clearing form or setting `run_on_latest_version=True` in an API call. For more configuration options see [Run on latest version](#run-on-latest-version).

  <Frame>
    <img alt="Clearing form of a Dag in the Airflow UI showing the Run with latest bundle version checkbox." />
  </Frame>

* **Clearing/rerunning individual tasks of a previous Dag run**:

  * Unversioned Dag bundle: Airflow uses the latest version of the Dag for tasks that are rerun.
  * Versioned Dag bundle: By default, when the run's Dag version differs from the latest and a task instance is cleared in the Airflow UI, Airflow reruns the task using the code in the latest Dag bundle version. To rerun using the version the task originally used, uncheck the **Run with latest bundle version** box. If the task instance is cleared through the API or CLI, the default is to use the Dag bundle version of the original run; set `run_on_latest_version=True` to use the latest version instead. For more configuration options see [Run on latest version](#run-on-latest-version).

  <Frame>
    <img alt="Clear Task Instance form in the Airflow UI showing the Run with latest bundle version checkbox." />
  </Frame>

* **Backfilling a Dag run**:

  * Unversioned Dag bundle: Airflow uses the current Dag code, that is, the latest version of the Dag.
  * Versioned Dag bundle: By default, the scheduler uses the latest version of the Dag for *all* Dag runs created in a [backfill](/docs/learn/rerunning-dags#backfill), including re-runs of existing runs when selecting **Missing and Errored Runs** or **All Runs**. You can configure the rerun behavior for *existing* Dag runs within a backfill, for example by unchecking the **Run with latest bundle version** box on the backfill form or setting `run_on_latest_version` in an API call to `False`, to use the original version of the code. Missing runs created in a backfill always use the latest Dag version. For more configuration options see [Run on latest version](#run-on-latest-version).

  <Frame>
    <img alt="Backfill form of a Dag in the Airflow UI showing the Run with latest bundle version checkbox." />
  </Frame>

* **Changing code while a Dag is running**:
  * Unversioned Dag bundle: The Dag always uses the current Dag code at the time it starts a task, as in Airflow 2.
  * Versioned Dag bundle: The Dag run finishes using the bundle version it started with.

* **Making code changes**:
  * Unversioned Dag bundle: Every structural change to the Dag creates a new Dag version.
  * Versioned Dag bundle: Every committed or saved structural change to a Dag creates a new version of that Dag. This means with every new bundle version, all Dags that have had structural changes will also have a new Dag version.

See the Airflow documentation on [Dag bundles](https://airflow.apache.org/docs/apache-airflow-providers-git/stable/bundles/index.html) for more information, including how to create a custom Dag bundle.

### Run on latest version

In Airflow 3.3+ you can define defaults for the rerun behavior of Dags stored in versioned Dag bundles at the Dag and configuration level. This determines whether the **Run with latest bundle version** check boxes in the Airflow UI are checked or unchecked by default.

If you set `rerun_with_latest_version=True` any clearing of a Dag run or task instance, as well as any backfill run of the Dag will use the code version from the most recent Dag bundle, unless overridden in the Airflow UI or API call that initiates the run. Conversely, setting this parameter to `False`, will default any cleared run, or run re-created in a backfill to use the original version of its code.

```python wrap theme={null}
@dag(rerun_with_latest_version=True)
```

You can set a default value for your Airflow instance at the configuration level with [`AIRFLOW__CORE__RERUN_WITH_LATEST_VERSION`](http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/configurations-ref.html#rerun-with-latest-version).

### Set up a GitDagBundle

To directly fetch your Dag code from a GitHub repository, you can use the `GitDagBundle`. This bundle is versioned. To configure a `GitDagBundle` for an Astro CLI project, follow these steps:

1. Push your Dag code to a [GitHub repository](https://docs.github.com/en).

2. Install the `git` package in your Astro project by adding it to your `packages.txt` file.

3. Install the [Airflow Git provider](https://airflow.apache.org/docs/apache-airflow-providers-git/stable/index.html) by adding the following to your `requirements.txt` file. Replace `<version>` with the latest version of the provider package.

   ```text wrap theme={null}
   apache-airflow-providers-git==<version>
   ```

4. Define a Git connection using an environment variable in your `.env` file. Replace `<account>` and `<repo>` with the name of your GitHub account and repository, respectively. Replace `github_pat_<your-token>` with your GitHub personal access token. The token only requires read permissions to the content of the repository.

   ```text wrap theme={null}
   AIRFLOW_CONN_MY_GIT_CONN='{
       "conn_type": "git",
       "host": "https://github.com/<account>/<repo>.git",
       "password": "github_pat_<your-token>"
   }'
   ```

5. Change the `[dag_processor].dag_bundle_config_list` configuration to use a `GitDagBundle` by setting the associated environment variable in your `.env` file. Replace `your-bundle-name` with the name you want to give to your Dag bundle. The `subdir` should point to the directory in your GitHub repository where your Dag code is stored. The `tracking_ref` should point to the branch you want to use.

   ```text wrap theme={null}
   AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST='[
       {
           "name": "your-bundle-name",
           "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
           "kwargs": {
               "git_conn_id": "my_git_conn",
               "subdir": "dags",  
               "tracking_ref": "main"  
           }
       }
   ]'
   ```

6. Restart your project using `astro dev restart` to apply the changes.

## Programmatic Dags and Dag bundles

If you are creating your Dags programmatically, that is, you are using Python code to generate your Dag code and want to use a versioned Dag bundle, you need to ensure that there are no Dag structure changes without a Dag bundle change.

The reason is, that when clearing a Dag run, the scheduler uses the Dag bundle version that existed at the time of the Dag run to determine which task instances to create. The workers use the code contained in the Dag bundle version that existed at the time of the original Dag run to execute their tasks. In the rare case where programmatic Dag creation leads to a Dag structure, and therefore Dag version change without a Dag bundle change, the scheduler and workers will use different Dag versions to create and execute the tasks. This can lead to unexpected behavior.

An example for programmatic Dag creation that is safe to use with a versioned Dag bundle is usage of the [dag-factory](https://pypi.org/project/dag-factory/) or to create tasks in a loop that only changes when the code changes:

```python wrap theme={null}

# this list only changes when the code changes
my_tables = ["TABLE_A", "TABLE_B", "TABLE_C"]

for my_table in my_tables:
    @task(
        task_id=f"modify_{i}",
    )
    def modify_table(my_table):
        # do something with the table
        pass

    modify_table(my_table=my_table)
```

If you are using top-level code that connects to an external system (a practice that we caution against, see [Avoid top-level code in your Dag file](/docs/learn/dag-best-practices#avoid-top-level-code-in-your-dag-file)), you might have a change in Dag structure without a change in the Dag bundle. An example would be if the list `my_tables` from the above example is created by querying a database.


# Understanding the Airflow metadata database
Source: https://astronomer.io/docs/learn/airflow-database

Learn about everything you need to use the Apache Airflow metadata database.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

The metadata database is a core component of Airflow. It stores crucial information such as the configuration of your Airflow environment's roles and permissions, as well as all metadata for past and present DAG and task runs.

A healthy metadata database is critical for your Airflow environment. Losing data stored in the metadata database can both interfere with running DAGs and prevent you from accessing data for past DAG runs. As with any core Airflow component, having a backup and disaster recovery plan in place for the metadata database is essential.

In this guide, you'll learn everything you need to know about the Airflow metadata database to ensure a healthy Airflow environment, including:

* Database specifications.
* Important content stored in the database.
* Best practices for using the metadata database.
* How to use the Airflow REST API to access the metadata database.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Airflow core components. See [Airflow's components](/docs/learn/airflow-components).

## Database specifications

Airflow uses SQLAlchemy and Object Relational Mapping (ORM) in Python to connect with the metadata database from the application layer. Any database supported by [SQLAlchemy](https://www.sqlalchemy.org/) can theoretically be configured to host Airflow's metadata. The most common databases used are:

* Postgres
* MySQL
* SQLite

While SQLite is the default on Apache Airflow, Postgres is by far the most common choice and is recommended for most use cases by the Airflow community. Astronomer uses Postgres for all of its Airflow environments, including local environments running with the Astro CLI and deployed environments on the cloud.

You should also consider the size of your metadata database when setting up your Airflow environment. Production environments typically use a managed database service, which includes features like autoscaling and automatic backups. The size you need will depend heavily on the workloads running in your Airflow instance. For reference, Apache Airflow uses a 2 GB SQLite database by default, but this is intended for development purposes only. The Astro CLI starts Airflow environments with a 1 GB Postgres database.

Changes to the Airflow metadata database configuration and its schema are very common and happen with almost every minor update. To downgrade your Airflow environment, use the [`db downgrade`](https://airflow.apache.org/docs/apache-airflow/stable/howto/usage-cli.html#downgrading-airflow) command.

## Metadata database content

There are several types of metadata stored in the metadata database.

* User  login information and permissions.
* Information used in DAGs, like variables, connections and XComs.
* Data about DAG and task runs which are generated by the scheduler.
* Other minor tables, such as tables which store DAG code in different formats or information about import errors.

For many use cases you can access content from the metadata database in the Airflow UI or the [stable REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html). These points of access always beat querying the metadata database directly!

### User information (security)

A set of tables store information about Airflow users, including their [permissions](https://airflow.apache.org/docs/apache-airflow/stable/security/index.html) to various Airflow features. As an admin user, you can access some of the content of these tables in the Airflow UI under the **Security** tab.

### DAG configurations and variables (Admin)

DAGs can retrieve and use a variety of information from the metadata database such as:

* [Variables](https://airflow.apache.org/docs/apache-airflow/stable/howto/variable.html).
* [Connections](/docs/learn/connections).
* [XComs](/docs/learn/airflow-passing-data-between-tasks).
* [Pools](/docs/learn/airflow-pools).

The information in these tables can be viewed and modified under the **Admin** tab in the Airflow UI.

### DAG and task runs (browse)

The scheduler depends on the Airflow metadata database to keep track of past and current events. The majority of this data can be found under the **Browse** tab in the Airflow UI.

* **DAG Runs** stores information on all past and current DAG runs including whether they were successful, whether they were scheduled or manually triggered, and detailed timing information.
* **Jobs** contains data used by the scheduler to store information about past and current jobs of different types (`SchedulerJob`, `TriggererJob`, `LocalTaskJob`).
* **Audit logs** shows events of various types that were logged to the metadata database (for example, DAGs being paused or tasks being run).
* **Task Instances** contains a record of every task run with a variety of attributes such as the priority weight, duration, or the URL to the task log.
* **Task Reschedule** lists tasks that have been rescheduled.
* **Triggers** shows all currently running [triggers](/docs/learn/deferrable-operators).
* **SLA Misses** keeps track of tasks that missed their SLA.

### Other tables

There are additional tables in the metadata database storing data ranging from DAG tags over serialized DAG code, import errors to current states of sensors. Some of the information in these tables will be visible in the Airflow UI in various places:

* The source code of DAGs can be found by clicking on a DAG name from the main view and then going to the **Code** view.
* Import errors appear at the top of the **DAGs** view in the UI.
* DAG tags will appear underneath their respective DAG with a cyan background.

## Airflow metadata database best practices

* When upgrading or downgrading Airflow, always follow the [recommended steps for changing Airflow versions](https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading.html?highlight=upgrade): back up the metadata database, check for deprecated features, pause all DAGs, and make sure no tasks are running.

* Use caution when [pruning old records](https://airflow.apache.org/docs/apache-airflow/stable/usage-cli.html#purge-history-from-metadata-database) from your database with `db clean`. For example, pruning records could affect future runs for tasks that use the `depends_on_past` argument. The `db clean` command allows you to delete records older than `--clean-before-timestamp` from all metadata database tables or a list of tables specified.

* Memory in the Airflow metadata database can be limited depending on your setup, and running low on memory in your metadata database can cause performance issues in Airflow. This is one of the many reasons why Astronomer advises against moving large amounts of data with XCom, and recommends using a cleanup and archiving mechanism in any production deployments.

* Since the metadata database is critical for the scalability and resiliency of your Airflow deployment, it is best practice to use a managed database service for production environments, for example [AWS RDS](https://aws.amazon.com/rds/) or [Google Cloud SQL](https://cloud.google.com/sql). Alternatively, you can use a managed Airflow service like [Astro](https://www.astronomer.io/lp/signup/) with a built-in scalable and resilient metadata database.

* When configuring a database backend, make sure your version is fully supported by checking the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html#choosing-database-backend).

## Use the Airflow REST API to access the metadata database

The best method for retrieving data from the metadata database is using the Airflow UI or making a GET request to the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html).

Between the UI and API, much of the metadata database can be viewed without the risk inherent in direct querying. In rare cases where neither the Airflow UI nor the REST API can provide sufficient data, it is possible to use SQLAlchemy with Airflow models to access data from the metadata database. Direct querying of the metadata database isn't recommended since direct manipulation can result in corruption of your Airflow instance.

This section shows three examples of how to use the Airflow REST API to interact with the Airflow metadata database.

### Retrieve the number of successfully completed tasks

A common reason users may want to access the metadata database is to get metrics like the total count of successfully completed tasks.

Using the [stable REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#section/Overview) to query the metadata database is the recommended way to programmatically retrieve this information. Make sure you have [correctly authorized API use](https://airflow.apache.org/docs/apache-airflow/stable/security/api.html) in your Airflow instance and set the `ENDPOINT_URL` to the correct location (for local development: `http://localhost:8080/`).

The Python script below uses the `requests` library to make a GET request to the Airflow API for all successful (`state=success`) Task Instances of all (shorthand: `~`) DAG runs of all (`~`) DAGs in the Airflow instance. A user name and password stored as environment variables are used for authentication. By printing the `total_entries` property of the API response JSON, one can get a count of all successfully completed tasks.

```python wrap theme={null}
# import the request library
import requests
import os

# provide the location of your airflow instance
ENDPOINT_URL = "http://localhost:8080/"

# in this example env variables were used to store login information
# you will need to provide your own credentials
user_name = os.environ["USERNAME_AIRFLOW_INSTANCE"]
password = os.environ["PASSWORD_AIRFLOW_INSTANCE"]

# query the API for successful task instances from all dags and all dag runs (~)
req = requests.get(
    f"{ENDPOINT_URL}/api/v1/dags/~/dagRuns/~/taskInstances?state=success",
    auth=(user_name, password),
)

# from the API response print the value for "total entries"
print(req.json()["total_entries"])
```

It is also possible to navigate to **Browse** -> **Task Instances** in the Airflow UI and filter the task instances for all with a state of `success`. The `Record Count` will be on the right side of your screen.

<Frame>
  <img alt="Count successful tasks Airflow UI" />
</Frame>

### Pause and unpause a DAG

Pausing and unpausing DAGs is a common action when running Airflow and while you can achieve this by manually toggling DAGs in the Airflow UI, depending on your use case and the number of DAGs you want to toggle this might be tedious. The Airflow REST API offers a simple way to pause and unpause DAGs by sending a PATCH request.

The Python script below sends a PATCH request to the Airflow API to update the entry for the DAG with a specific ID (here `example_dag_basic`), which is paused (`update_mask=is_paused`) with a JSON that will set the `is_paused` property to `True` therefore unpausing the DAG.

```python wrap theme={null}
# import the request library
import requests
import os

# provide the location of your airflow instance
ENDPOINT_URL = "http://localhost:8080/"

# in this example env variables were used to store login information
# you will need to provide your own credentials
user_name = os.environ["USERNAME_AIRFLOW_INSTANCE"]
password = os.environ["PASSWORD_AIRFLOW_INSTANCE"]

# data to update, for unpausing, simply set this to False
update = {"is_paused": True}
# specify the dag to pause/unpause
dag_id = "example_dag_basic"

# query the API to patch all tasks as paused
req = requests.patch(
    f"{ENDPOINT_URL}/api/v1/dags/{dag_id}?update_mask=is_paused",
    json=update,
    auth=(user_name, password),
)

# print the API response
print(req.text)
```

### Delete a DAG

Deleting the metadata of a DAG can be accomplished either by clicking the trashcan icon in the Airflow UI or sending a `DELETE` request with the Airflow REST API. This isn't possible while the DAG is still running, and won't delete the Python file in which the DAG is defined, meaning the DAG will appear again in your UI with no history at the next parsing of the `/dags` folder from the scheduler.

The Python script below sends a DELETE request to a DAG with a specific ID (here: `dag_to_delete`).

```python wrap theme={null}
# import the request library
import requests
import os

# provide the location of your airflow instance
ENDPOINT_URL = "http://localhost:8080/"

# in this example env variables were used to store login information
# you will need to provide your own credentials
user_name = os.environ["USERNAME_AIRFLOW_INSTANCE"]
password = os.environ["PASSWORD_AIRFLOW_INSTANCE"]

# specify which dag to delete
dag_id = "dag_to_delete"

# send the deletion request
req = requests.delete(
    f"{ENDPOINT_URL}/api/v1/dags/{dag_id}", auth=(user_name, password)
)

# print the API response
print(req.text)
```


# Tutorial: How to Orchestrate Databricks Jobs with Airflow
Source: https://astronomer.io/docs/learn/airflow-databricks

Step-by-step guide to orchestrating Databricks with Airflow. Learn to trigger notebooks, run jobs, and build data pipelines. Includes example Dag code.

[Databricks](https://databricks.com/) is a popular unified data and analytics platform built around [Apache Spark](https://spark.apache.org/) that provides users with fully managed Apache Spark clusters and interactive workspaces.

The open source [Airflow Databricks provider](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/index.html) provides full observability and control from Airflow so you can manage Databricks from one place, including enabling you to orchestrate your Databricks notebooks from Airflow and execute them as [Databricks jobs](https://docs.databricks.com/en/workflows/index.html).

## Why use Airflow with Databricks

Many data teams use Databricks' optimized Spark engine to run heavy workloads like machine learning models, data transformations, and data analysis. While Databricks offers some orchestration with Databricks Workflows, they are limited in functionality and don't integrate with the rest of your data stack. Using a tool-agnostic orchestrator like Airflow gives you several advantages, like the ability to:

* Use CI/CD to manage your workflow deployment. Airflow Dags are Python code, and can be [integrated with a variety of CI/CD tools](/docs/astro/ci-cd-templates/template-overview) and [tested](/docs/learn/testing-airflow).
* Use [task groups](/docs/learn/task-groups) within Databricks jobs, enabling you to collapse and expand parts of larger Databricks jobs visually.
* Use [Airflow assets](/docs/learn/airflow-datasets) to trigger Databricks jobs from tasks in other Dags in your Airflow environment or using the Airflow REST API [Create asset event endpoint](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/create_asset_event), allowing for a data-driven architecture.
* Use familiar Airflow code as your interface to orchestrate Databricks notebooks as jobs.
* [Inject parameters](#parameters) into your Databricks job at the job-level. These parameters can be dynamic and retrieved at runtime from other Airflow tasks.
* Directly jump from a task in your Airflow Dag to the corresponding Databricks job in the Databricks UI using an operator extra link.

## Time to complete

This step-by-step tutorial takes approximately 30 minutes to complete. After completing this tutorial, you will have a working Airflow Dag that orchestrates Databricks notebooks as a Databricks Workflow.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of Databricks. See [Getting started with Databricks](https://www.databricks.com/learn).
* Airflow fundamentals, such as writing Dags and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow connections. See [Managing your Connections in Apache Airflow](/docs/learn/connections).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* Access to a Databricks workspace. See [Databricks' documentation](https://docs.databricks.com/getting-started/index.html) for instructions. You can use any workspace that has access to the [Databricks Workflows](https://docs.databricks.com/workflows/index.html) feature. You need a user account with permissions to create notebooks and Databricks jobs. You can use any underlying cloud service, and a [14-day free trial](https://www.databricks.com/try-databricks) is available.

## Step 1: Configure your Astro project

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-databricks-tutorial && cd astro-databricks-tutorial
   $ astro dev init
   ```

2. Add the [Airflow Databricks provider package](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/index.html) to your `requirements.txt` file.

   ```text wrap theme={null}
   apache-airflow-providers-databricks==7.8.0
   ```

## Step 2: Create Databricks notebooks

You can orchestrate any Databricks notebooks in a Databricks job using the Airflow Databricks provider. If you don't have Databricks notebooks ready, follow these steps to create two notebooks:

1. [Create an empty notebook](https://docs.databricks.com/notebooks/notebooks-manage.html) in your Databricks workspace called `notebook1`.

2. Copy and paste the following code into the first cell of the `notebook1` notebook.

   ```python wrap theme={null}
   print("Hello")
   ```

3. Create a second empty notebook in your Databricks workspace called `notebook2`.

4. Copy and paste the following code into the first cell of the `notebook2` notebook.

   ```python wrap theme={null}
   print("World")
   ```

## Step 3: Configure the Databricks connection

1. Start Airflow by running `astro dev start`.

2. In the Airflow UI, go to **Admin** > **Connections** and click **+**.

3. Create a new connection named `databricks_conn`. Select the connection type **Databricks** and enter the following information:

   * **Connection ID**: `databricks_conn`.
   * **Connection Type**: `Databricks`.
   * **Host**: Your Databricks host address (format: `https://dbc-1234cb56-d7c8.cloud.databricks.com/`).
   * **Password**: Your [Databricks personal access token](https://docs.databricks.com/dev-tools/auth.html#databricks-personal-access-tokens).

   Alternatively, you can create an [OAuth connection](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m) to your Databricks workspace by providing the **Host**, **Service Principal Client ID** as **Login**, **Service Principal Client Secret** as **Password** and set `service_principal_oauth` to `True` in the **Extra** field.

<Note>
  Astro customers can use the [Astro Environment Manager](/docs/astro/manage-connections-variables) to create a connection to Databricks, stored in the Astro-managed secrets backend. This connection can be shared across multiple Deployments in a Workspace.
</Note>

## Step 4: Create your Dag

1. In your `dags` folder, create a file called `my_simple_databricks_dag.py`.

2. Copy and paste the following Dag code into the file. Replace `<your-databricks-login-email>` variable with your Databricks login email. If you already had Databricks notebooks and didn't create new ones in Step 2, adjust the `notebook_path` parameters in the two DatabricksNotebookOperators to point to the existing notebooks. Adjust the `job_cluster_spec` to match your available cloud resources.

   ```python expandable wrap theme={null}
   """
   ### Run notebooks in databricks as a Databricks Workflow using the Airflow Databricks provider

   This Dag runs two Databricks notebooks as a Databricks workflow.
   """

   from airflow.sdk import dag, chain
   from airflow.providers.databricks.operators.databricks import DatabricksNotebookOperator
   from airflow.providers.databricks.operators.databricks_workflow import (
       DatabricksWorkflowTaskGroup,
   )
   from pendulum import datetime

   DATABRICKS_LOGIN_EMAIL = "<your-databricks-login-email>"
   DATABRICKS_NOTEBOOK_NAME_1 = "notebook1"
   DATABRICKS_NOTEBOOK_NAME_2 = "notebook2"
   DATABRICKS_NOTEBOOK_PATH_1 = (
       f"/Users/{DATABRICKS_LOGIN_EMAIL}/{DATABRICKS_NOTEBOOK_NAME_1}"
   )
   DATABRICKS_NOTEBOOK_PATH_2 = (
       f"/Users/{DATABRICKS_LOGIN_EMAIL}/{DATABRICKS_NOTEBOOK_NAME_2}"
   )
   DATABRICKS_JOB_CLUSTER_KEY = "tutorial-cluster"
   DATABRICKS_CONN_ID = "databricks_conn"

   # adjust if necessary for example to align the spark version with your Notebooks
   job_cluster_spec = [
       {
           "job_cluster_key": DATABRICKS_JOB_CLUSTER_KEY,
           "new_cluster": {
               "cluster_name": "",
               "spark_version": "15.4.x-scala2.12",
               "azure_attributes": {
                   "first_on_demand": 1,
                   "availability": "SPOT_WITH_FALLBACK_AZURE",
                   "spot_bid_max_price": -1,
               },
               "node_type_id": "Standard_DS3_v2",
               "spark_env_vars": {"PYSPARK_PYTHON": "/databricks/python3/bin/python3"},
               "enable_elastic_disk": False,
               "data_security_mode": "LEGACY_SINGLE_USER_STANDARD",
               "runtime_engine": "STANDARD",
               "num_workers": 1,
           },
       }
   ]


   @dag
   def my_simple_databricks_dag():
       task_group = DatabricksWorkflowTaskGroup(
           group_id="databricks_workflow",
           databricks_conn_id=DATABRICKS_CONN_ID,
           job_clusters=job_cluster_spec,
       )

       with task_group:
           notebook_1 = DatabricksNotebookOperator(
               task_id="notebook1",
               databricks_conn_id=DATABRICKS_CONN_ID,
               notebook_path=DATABRICKS_NOTEBOOK_PATH_1,
               source="WORKSPACE",
               job_cluster_key=DATABRICKS_JOB_CLUSTER_KEY,
           )
           notebook_2 = DatabricksNotebookOperator(
               task_id="notebook2",
               databricks_conn_id=DATABRICKS_CONN_ID,
               notebook_path=DATABRICKS_NOTEBOOK_PATH_2,
               source="WORKSPACE",
               job_cluster_key=DATABRICKS_JOB_CLUSTER_KEY,
           )
           chain(notebook_1, notebook_2)


   my_simple_databricks_dag()
   ```

   This Dag uses the Airflow Databricks provider to create a Databricks job that runs two notebooks. The `databricks_workflow` task group, created using the `DatabricksWorkflowTaskGroup` class, automatically creates a Databricks job that executes the Databricks notebooks you specified in the individual DatabricksNotebookOperators. One of the biggest benefits of this setup is the use of a Databricks job cluster, allowing you to [significantly reduce your Databricks cost](https://www.databricks.com/product/pricing). The task group contains three tasks:

   * The `launch` task, which the task group automatically generates, provisions a Databricks `job_cluster` with the spec defined as `job_cluster_spec` and creates the Databricks job from the tasks within the task group.
   * The `notebook1` task runs the `notebook1` notebook in this cluster as the first part of the Databricks job.
   * The `notebook2` task runs the `notebook2` notebook as the second part of the Databricks job.

3. Run the Dag manually by clicking the play button and view the Dag in the graph tab. In case the task group appears collapsed, click it in order to expand and see all tasks.

   <Frame>
     <img alt="Airflow Databricks Dag graph tab showing a successful run of the Dag with one task group containing three tasks: launch, notebook1 and notebook2." />
   </Frame>

4. View the completed Databricks job in the Databricks UI.

   <Frame>
     <img alt="Successful run of a Databricks job in the Databricks UI." />
   </Frame>

## Step 5: (optional) Add a task to run SQL

You can run any SQL query in Databricks using the `DatabricksSqlOperator` from the [Airflow Databricks provider](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/index.html). In your Dag, outside of the `databricks_workflow` task group, add the following task. Replace the placeholder values with your own values.

```python wrap theme={null}
from airflow.providers.databricks.operators.databricks_sql import DatabricksSqlOperator

run_sql = DatabricksSqlOperator(
    task_id="run_sql",
    databricks_conn_id=DBX_CONN_ID,
    http_path=f"/sql/1.0/warehouses/{DATABRICKS_SQL_WAREHOUSE_ID}",
    catalog="<your-catalog-name>",
    schema="<your-schema-name>",
    sql="<your-sql-query>",
    parameters={"<your-parameter-name>": "<your-parameter-value>"}
)
```

Alternatively, you can also use the `DatabricksHook` directly in any `@task` decorated function or `PythonOperator` in your Dag.

```python wrap theme={null}
    @task
    def run_sql():
        from airflow.providers.databricks.hooks.databricks import DatabricksHook

        hook = DatabricksHook(DBX_CONN_ID)

        re = hook.post_sql_statement(
            json={
                "warehouse_id": DATABRICKS_SQL_WAREHOUSE_ID,
                "catalog": "<your-catalog-name>",
                "schema": "<your-schema-name>",
                "statement": "<your-sql-query>",
                "parameters": [
                    {"name": "<your-parameter-name>", "value": "<your-parameter-value>", "type": "<your-parameter-type>"},
                ],
            }
        )
```

## How it works

This section explains Airflow Databricks provider functionality in more depth. You can learn more about the Airflow Databricks provider, including more information about other available operators, in the [provider documentation](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/index.html).

### Parameters

The DatabricksWorkflowTaskGroup provides configuration options via several parameters:

* `job_clusters`: the job clusters parameters for this job to use. You can provide the full `job_cluster_spec` as shown in the tutorial Dag.

* `notebook_params`: a dictionary of parameters to make available to all notebook tasks in a job. This operator is templatable, see below for a code example:

  ```python wrap theme={null}
  dbx_workflow_task_group = DatabricksWorkflowTaskGroup(
      group_id="databricks_workflow",
      databricks_conn_id=_DBX_CONN_ID,
      job_clusters=job_cluster_spec,
      notebook_params={
          "my_date": "{{ ds }}"
      },
  )
  ```

  To retrieve this parameter inside your Databricks notebook add the following code to a Databricks notebook cell:

  ```python wrap theme={null}
  dbutils.widgets.text("my_date", "my_default_value", "Description")
  my_date = dbutils.widgets.get("my_date")
  ```

* `notebook_packages`: a list of dictionaries defining Python packages to install in all notebook tasks in a job.

* `extra_job_params`: a dictionary with properties to override the default Databricks job definitions.

You also have the ability to specify parameters at the task level in the `DatabricksNotebookOperator`:

* `notebook_params`: a dictionary of parameters to make available to the notebook.
* `notebook_packages`: a list of dictionaries defining Python packages to install in the notebook.

Note that you can't specify the same packages in both the `notebook_packages` parameter of a DatabricksWorkflowTaskGroup and the `notebook_packages` parameter of a task using the `DatabricksNotebookOperator` in that same task group. Duplicate entries in this parameter cause an error in Databricks.


# Basic asset-based scheduling in Apache Airflow®
Source: https://astronomer.io/docs/learn/airflow-datasets

Using assets to schedule Dags based on successful completion of tasks.

With Assets, Dags that access the same data can have explicit, visible relationships, and Dags can be scheduled based on updates to these assets. This feature helps make Airflow data-aware and expands Airflow scheduling capabilities beyond time-based methods such as cron.

Assets can help resolve common issues. For example, consider a data engineering team with a Dag that creates a table with cleaned data and a machine learning team with a Dag that trains a model on that data. Using assets, the machine learning team's Dag runs only when the data engineering team's Dag has produced an update to the asset. An asset can represent anything, from a table in a database, to a file in object storage, to a fine-tuned LLM, to an abstract entity like a certain business process having completed.

In this guide, you'll learn:

* When to use assets in Airflow.
* Basic asset concepts and terminology.
* How to schedule Dags based on basic asset schedules.
* How to update assets in Airflow.
* How to view asset dependencies in the Airflow UI.
* Which options exist for [advanced asset-based scheduling](#options-for-advanced-asset-based-scheduling).

<Info>
  Assets are a separate feature from object storage, which allows you to interact with files in cloud and local object storage systems. To learn more about using Airflow to interact with files, see [Use Airflow object storage to interact with cloud storage in an ML pipeline](/docs/learn/airflow-object-storage-tutorial).
</Info>

## Assumed knowledge

To get the most out of this guide, you should have an existing knowledge of:

* Airflow scheduling concepts. See [Schedule Dags in Airflow](/docs/learn/scheduling-in-airflow).

## When to use Airflow assets

Assets allow you to define explicit dependencies between Dags and updates to your data.

Basic asset-based scheduling helps you to:

* Standardize communication between teams. Assets can function like an API to communicate when data in a specific location has been updated and is ready for use.
* Reduce the amount of code necessary to implement [cross-Dag dependencies](/docs/learn/cross-dag-dependencies). Even if your Dags don't depend on data updates, you can create a dependency that triggers a Dag after a task in another Dag updates an asset.
* Get better visibility into how your Dags are connected and how they depend on data. The **Assets** graphs in the Airflow UI display how assets and Dags depend on each other and can be used to navigate between them.
* Reduce costs, because assets don't use a worker slot in contrast to sensors or [other implementations of cross-Dag dependencies](/docs/learn/cross-dag-dependencies).
* Create cross-deployment dependencies using the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/create_dataset_event). [Astro](https://www.astronomer.io/lp/signup/?referral=docs-what-astro-banner\&utm_medium=docs\&utm_content=astro-trial\&utm_source=body) customers can use the [Cross-deployment dependencies](/docs/astro/best-practices/cross-deployment-dependencies) best practices documentation for guidance.

See [Advanced asset-based scheduling](/docs/learn/airflow-advanced-asset-scheduling) for more information on the capabilities of advanced asset-based scheduling.

<Tip>
  Assets are a fundamental scheduling paradigm in Airflow. To learn more about when to use assets vs other scheduling paradigms, check out the free [Apache Airflow® orchestration paradigms ebook](https://www.astronomer.io/ebooks/apache-airflow-orchestration-paradigms).
</Tip>

### When not to use Airflow assets

Airflow is only aware of updates to assets that occur by tasks, API calls, or in the Airflow UI. It doesn't monitor updates to assets that occur outside of Airflow. For example, Airflow won't notice if you manually add a file to an S3 bucket referenced by an asset.

To create Airflow dependencies based on outside events, you can use:

* [Airflow sensors](/docs/learn/what-is-a-sensor): Synchronously check for a condition to be met. A lot of sensors have a [deferrable mode](/docs/learn/deferrable-operators).
* [Async functions in @task decorators](/docs/learn/deferrable-operators): Asynchronously check for a condition to be met.
* [Deferrable operators](/docs/learn/deferrable-operators): Use the Airflow triggerer component to asynchronously check for a condition to be met, releasing the worker slot during long-running tasks.

[Event-driven scheduling](/docs/learn/airflow-event-driven-scheduling) based on messages in a message queue is a type of [advanced asset-based scheduling](/docs/learn/airflow-advanced-asset-scheduling).

## Basic asset concepts

You can define assets in your Dag code and use them to create cross-Dag dependencies. Airflow uses the following terms related to asset-based scheduling:

* **Asset**: an object in Airflow that represents a concrete or abstract data entity and is defined by a unique name. Optionally, a URI can be attached to the asset, when it represents a concrete data entity, like a file in object storage or a table in a relational database.
* **Asset schedule**: the schedule of a Dag that is triggered as soon as asset events for one or more assets are created. All assets a Dag is scheduled on are shown in the Dag graph in the Airflow UI, as well as reflected in the dependency graph of the **Assets** tab.
* **Producer task**: a task that produces updates to one or more assets provided to its `outlets` parameter, creating asset events when it completes successfully.
* **Asset event**: an event that is attached to an asset and created whenever a producer task updates that particular asset. An asset event is defined by being attached to a specific asset plus the timestamp of when a producer task updated the asset. Optionally, an asset event can contain an `extra` dictionary with additional information about the asset or asset event.

Two parameters relating to Airflow assets exist in all Airflow operators and decorators:

* `outlets`: a task parameter that contains the list of assets a specific task produces updates to, as soon as it completes successfully. All outlets of a task are shown in the Dag graph in the Airflow UI, as well as reflected in the dependency graph of the **Assets** tab as soon as the Dag code is parsed, independently of whether or not any asset events have occurred. Note that Airflow is **not** yet aware of the underlying data. It is up to you to determine which tasks should be considered producer tasks for an asset. As long as a task has an outlet asset, Airflow considers it a producer task even if that task doesn't operate on the referenced asset.
* `inlets`: a task parameter that contains the list of assets a specific task has access to, typically to access `extra` information from related asset events. Defining inlets for a task does **not** affect the schedule of the Dag containing the task.

To summarize, tasks produce updates to assets given to their `outlets` parameter, and this action creates asset events. Dags can be scheduled based on asset events created for one or more assets, and tasks can be given access to all events attached to an asset by defining the asset as one of their `inlets`. An asset is defined as an object in the Airflow metadata database as soon as it is referenced in either the `outlets` parameter of a task or the `schedule` of a Dag.

Using advanced asset-based scheduling introduces additional concepts, see [Advanced asset-based scheduling](/docs/learn/airflow-advanced-asset-scheduling#advanced-asset-concepts) for more information.

## Asset definition

An asset is defined as an object in the Airflow metadata database as soon as it is referenced in either the `outlets` parameter of a task, the `inlets` parameter of a task, or the `schedule` of a Dag.

The code snippet below shows how you can define an asset using the `outlets` parameter in both a [`@task` decorator](/docs/learn/airflow-decorators) and a traditional operator (`BashOperator`).

```python wrap theme={null}
from airflow.sdk import Asset, dag, task
from airflow.providers.standard.operators.bash import BashOperator

@dag 
def dag_a():

    @task(outlets=[Asset("asset_a")])
    def task_a():
        pass

    task_a()

    BashOperator(
        task_id="task_bash",
        bash_command="echo 'Hello, World!'",
        outlets=[Asset("asset_a_bash")]
    )

dag_a()
```

Defining an asset in the schedule of a Dag is done by providing the asset to the `schedule` parameter. This creates a schedule for the Dag to run as soon as the asset is updated (an asset event is created for the asset).

```python wrap theme={null}
from airflow.sdk import Asset, dag

@dag(schedule=[Asset("asset_b")]) 
def dag_b():
```

Lastly, you can define an asset in the `inlets` parameter of any task. Note that `inlets` don't affect the schedule of the Dag containing the task.

```python wrap theme={null}
from airflow.sdk import Asset, dag, task
from airflow.providers.standard.operators.bash import BashOperator

@dag 
def dag_c():

    @task(inlets=[Asset("asset_c")])
    def task_c():
        pass

    task_c()

    BashOperator(
        task_id="task_bash",
        bash_command="echo 'Hello, World!'",
        inlets=[Asset("asset_c_bash")]
    )

dag_c()
```

<Tip>
  The same task can have inlets and outlets defined and information about the asset event can be accessed using the Airflow context inside the task. See [Asset event extras](/docs/learn/airflow-advanced-asset-scheduling#asset-event-extras) in the [Advanced asset-based scheduling](/docs/learn/airflow-advanced-asset-scheduling) guide for more information.
</Tip>

All registered assets appear in the **Assets** tab of the Airflow UI, alongside any Dags scheduled on the asset, as well as any producing tasks.

<Frame>
  <img alt="Screenshot of the Assets tab in the Airflow UI showing the my_asset_one, my_asset_two and my_asset_three assets." />
</Frame>

Clicking on any asset opens the [asset graph](#asset-graph) for that asset.

## Updating an asset

There are five ways to update an asset by creating an asset event.

* A task with an `outlets` parameter that references the asset completes successfully, in the example above the `task_a` task produces an update to the `asset_a` asset and the `task_bash` task produces an update to the `asset_a_bash` asset. You can provide several assets in the list of assets, for example `outlets=[Asset("asset_a"), Asset("asset_b")]`, then successful task completion will produce an asset event for each of the assets in the list.

The **Asset Events** tab of the task instance details page lists all asset events that one task instance task produced.

<Frame>
  <img alt="Screenshot of the Asset Events tab of the task instance details page showing the asset events that the task instance produced." />
</Frame>

* A `POST` request to the [assets endpoint of the Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Dataset).

* A manual update in the Airflow UI by using the **Create Asset Event** button on the [asset graph](#asset-graph). There are two options when creating an asset event in the UI:

  * **Materialize**: This option runs the full Dag which contains the task that produces the asset event.
  * **Manual**: This option directly creates a new asset event without running any task that would normally produce the asset event. This option is useful for testing or when you want to create an asset event for an asset that isn't updated from within a Dag in this Airflow instance.

  <Frame>
    <img alt="Screenshot of the Airflow UI showing manual updates to an asset." />
  </Frame>

* A Dag defined using `@asset` completes successfully. Under the hood, `@asset` creates a Dag with one task which produces the asset, see [asset decorator syntax](/docs/learn/airflow-asset-decorator) for more information.

* An `AssetWatcher` that listens for a `TriggerEvent` caused by a message in a message queue. See [event-driven scheduling](/docs/learn/airflow-event-driven-scheduling) for more information.

## Asset schedule

Once a Dag is scheduled on one (or more) assets and unpaused in the Airflow UI, it will run as soon as an asset event is created for each of the assets it is scheduled on, regardless of the [method that created the asset event](#updating-an-asset).

In the **Dags** view you can see which asset a Dag is scheduled on in the **Schedule** column.

<Frame>
  <img alt="Screenshot of the Dags view in the Airflow UI showing the schedule column with the asset name." />
</Frame>

Any asset-based runs of this Dag have a Dag ID starting with `asset_triggered_`, the Run Type **Asset Triggered** and a database icon on the Dag run duration bar.

<Frame>
  <img alt="Screenshot of the Dags view in the Airflow UI showing an asset-based run of the my_consumer_dag Dag." />
</Frame>

The **Asset Events** tab of the Dag run details page lists all asset events that triggered a particular Dag run (Source Asset Events)

<Frame>
  <img alt="Screenshot of the Asset Events tab of the Dag run details page showing the asset events that triggered the Dag run." />
</Frame>

There are some important rules to note about the asset schedule:

* Asset events only count towards the schedule of a Dag while the Dag is unpaused. If the Dag is paused it will ignore all updates to assets and start with a blank slate upon being unpaused.
* Dags that are scheduled on an asset are triggered every time a task that updates that asset completes successfully. For example, if `task1` and `task2` both produce `asset_a`, a consumer Dag of `asset_a` runs twice - first when `task1` completes, and again when `task2` completes.
* Dags scheduled on an asset are triggered as soon as the first task with that asset as an outlet finishes, even if there are downstream producer tasks that also operate on the asset.
* If you provide several assets in the `schedule` parameter of a Dag, the Dag will run as soon as an asset event is created for each of the assets it is scheduled on. After a Dag run the schedule is reset and the Dag will again wait for an asset event to be created for each of the assets it is scheduled on *after* the last Dag run. See [Multiple Assets in the Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/asset-scheduling.html#multiple-assets) for more information. For more complex multi-asset scheduling scenarios, see [Options for advanced asset-based scheduling](#options-for-advanced-asset-based-scheduling).
* Dags that are triggered by assets don't have the concept of a data interval. If you need to pass time-based information to a downstream Dag, use a [partitioned asset schedule](/docs/learn/airflow-advanced-asset-scheduling#partitioned-asset-schedules).

## Asset graph

Clicking on any asset opens the asset graph for this asset. Each asset graph has 2 different views:

* **Scheduling**: This view connects each Dag with all assets that any tasks in said Dag produce updates to, as well as any Dags scheduled on an asset.

For example the Scheduling view for the asset graph for `asset_a` shows the relationship between the `dag_a` Dag and the `asset_a` and `asset_a_bash` assets (even though `asset_a` and `asset_a_bash` aren't directly connected to each other).

<Frame>
  <img alt="Screenshot of the Scheduling view of the asset graph for asset_a showing the relationship between the dag_a Dag and the asset_a and asset_a_bash assets." />
</Frame>

* **Task Dependencies**: This view connects each task with all asset that the task updates through its `outlets` parameter, as well as any tasks that have the asset as one of their `inlets` parameter.

For example the Task Dependencies view for the asset graph for `asset_a` shows the relationship between the `asset_a` and `task_a` which has `asset_a` defined in its `outlets` parameter.

<Frame>
  <img alt="Screenshot of the Task Dependencies view of the asset graph showing the relationship between the asset_a and task_a which has asset_a defined in its outlets parameter." />
</Frame>

Similarly, the Task Dependencies view for `asset_c` shows the relationship between the `asset_c` and `task_c` which has `asset_c` defined in its `inlets` parameter. Note that the Scheduling view for the asset graph of `asset_c` is empty because `inlets` don't affect any Dag scheduling.

<Frame>
  <img alt="Screenshot showing the Task Dependencies view of the asset graph for asset_c showing the relationship between the asset_c and task_c which has asset_c defined in its inlets parameter." />
</Frame>

The asset graph allows you to track asset-based schedules across many Dags and tasks. The screenshot below shows a more complex example of the asset graph for `asset_4` which contains seven assets and six Dags.

<Frame>
  <img alt="Screenshot of complex asset dependencies in the Airflow UI." />
</Frame>

<details>
  <summary>Click to view the code for the example above.</summary>

  ```python expandable wrap theme={null}
  from airflow.sdk import Asset, dag, task

  @dag 
  def dag_1():

      @task(outlets=[Asset("asset_1")])
      def task_1():
          pass

      task_1()

      @task(outlets=[Asset("asset_2")])
      def task_2():
          pass

      task_2()

  dag_1()

  @dag(schedule=[Asset("asset_0")]) 
  def dag_2():

      @task(outlets=[Asset("asset_3")])
      def task_3():
          pass

      task_3()


  dag_2()

  @dag
  def dag_3():

      @task(outlets=[Asset("asset_3"), Asset("asset_4")])
      def task_5():
          pass

      task_5()


  dag_3()


  @dag(schedule=[Asset("asset_3")])
  def dag_4():

      @task 
      def task_7():
          pass

      task_7()

  dag_4()

  @dag(schedule=[Asset("asset_4"), Asset("asset_1")])
  def dag_5():
      
      @task(outlets=[Asset("asset_5")]) 
      def task_8():
          pass

      task_8()

  dag_5()

  @dag(schedule=[Asset("asset_5")])
  def dag_7():
      
      @task(outlets=[Asset("asset_7")])
      def task_10():
          pass

      task_10()

  dag_7()
  ```
</details>

## Options for advanced asset-based scheduling

The [advanced asset-based scheduling](/docs/learn/airflow-advanced-asset-scheduling) guide covers more complex asset-based scheduling scenarios, such as:

* [Conditional asset scheduling](/docs/learn/airflow-advanced-asset-scheduling#conditional-asset-scheduling): Schedule a Dag based on an asset expression, which can include any combination of assets, and `|` (OR) and `&` (AND) logical operators.
* [Combined asset and time-based scheduling](/docs/learn/airflow-advanced-asset-scheduling#combined-asset-and-time-based-scheduling): Schedule a Dag to run on both a time-based schedule (cron or any other [Timetable](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timetable.html)) plus whenever an asset expression is fulfilled.
* [Asset event extras](/docs/learn/airflow-advanced-asset-scheduling#asset-event-extras): Attach extra information to an asset event and retrieve it in downstream tasks.
* [Partitioned asset schedules](/docs/learn/airflow-advanced-asset-scheduling#partitioned-asset-schedules): Attach partition keys to asset events to create partitioned Dag runs.
* [Asset aliases](/docs/learn/airflow-advanced-asset-scheduling#asset-aliases): Create named aliases for assets to reference in your Dag code and attach asset events to the alias at runtime. This is especially useful when using assets in dynamic task mapping.
* [Cross-deployment dependencies](/docs/astro/best-practices/cross-deployment-dependencies#assets-example): You can use assets to trigger a Dag in one Airflow environment from within a Dag in another Airflow environment.
* [Asset listeners](/docs/learn/airflow-advanced-asset-scheduling#asset-listeners): Use listeners to run code when certain asset events occur anywhere in your Airflow instance.

<Note>
  For information on the `@asset` decorator, which is a more concise way to create one Dag containing one task that produces an asset, see [asset decorator syntax](/docs/learn/airflow-asset-decorator).
</Note>

## Example: Basic asset-based scheduling

The simplest asset schedule is one Dag scheduled based on updates to one asset which is produced to by one task. In this example, we define that the `my_producer_task` task in the `my_producer_dag` Dag produces updates to the `my_asset` asset, creating attached asset events, and schedule the `my_consumer_dag` Dag to run once for every asset event created.

First, provide the asset to the outlets parameter of the producer task.

<details>
  <summary>Taskflow</summary>

  ```python {1,6} wrap theme={null}
  from airflow.sdk import Asset, dag, task

  @dag
  def my_producer_dag():

      @task(outlets=[Asset("my_asset")])
      def my_producer_task():
          pass

      my_producer_task()

  my_producer_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python {1,12} wrap theme={null}
  from airflow.sdk import Asset, DAG
  from airflow.providers.standard.operators.python import PythonOperator

  with DAG(dag_id="my_producer_dag"):

      def my_function():
          pass

      my_task = PythonOperator(
          task_id="my_producer_task",
          python_callable=my_function,
          outlets=[Asset("my_asset")]
      )

  ```
</details>

You can see the relationship between the Dag containing the producing task (`my_producer_dag`) and the asset in the **Asset Graph** located in the **Assets** tab of the Airflow UI.

<Frame>
  <img alt="Screenshot of the Dependency Graph of the Assets tab showing my_producer_dag connected to the my_asset asset." />
</Frame>

The graph view of the `my_producer_dag` shows the asset as well, if **external conditions** or **all Dag dependencies** are selected in the **Options** menu.

<Frame>
  <img alt="Screenshot of a Dag Graph showing my_producer_task connected to the my_asset asset." />
</Frame>

Next, schedule the `my_consumer_dag` to run as soon as a new asset event is produced to the `my_asset` asset.

<details>
  <summary>Taskflow</summary>

  ```python {1,6} wrap theme={null}
  from airflow.sdk import Asset, dag
  from airflow.providers.standard.operators.empty import EmptyOperator

  @dag(
      schedule=[Asset("my_asset")],
  )
  def my_consumer_dag():

      EmptyOperator(task_id="empty_task")

  my_consumer_dag()

  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python {1,7} wrap theme={null}
  from airflow.sdk import Asset, DAG
  from airflow.operators.empty import EmptyOperator

  with DAG(
      dag_id="my_consumer_dag",
      schedule=[Asset("my_asset")]
  ):

      EmptyOperator(task_id="empty_task")

  ```
</details>

You can see the relationship between the Dag containing the producing task (`my_producer_dag`), the consuming Dag `my_consumer_dag`, and the asset in the **asset graph** located in the **Assets** tab of the Airflow UI.

<Frame>
  <img alt="Screenshot of the Dependency Graph of the Assets tab showing my_producer_dag connected to the my_asset asset which is connected to my_consumer_dag" />
</Frame>

When **external conditions** or **all Dag dependencies** are selected, the `my_consumer_dag` graph shows the asset as well.

<Frame>
  <img alt="Screenshot of a Dag Graph showing my_producer_task connected to the my_asset asset." />
</Frame>

After unpausing the `my_consumer_dag`, every successful completion of the `my_producer_task` task triggers a run of the `my_consumer_dag`.

<Frame>
  <img alt="Screenshot Dags page with one run each of the my_producer_dag and my_consumer_dag as well as the asset schedule displayed" />
</Frame>

The producing task lists the **Asset Events** it caused in its details page, including a link to the **Triggered Dag Run**.

<Frame>
  <img alt="Screenshot of the Details tab of the my_producer_task showing one Asset event of the my_asset with one Triggered Dag Run" />
</Frame>

The triggered Dag run of the `my_consumer_dag` also lists the asset event, including a link to the source Dag from within which the asset event was created.

<Frame>
  <img alt="Screenshot of the Details tab of the Dag run of the my_consumer_dag showing one Asset event of the my_asset" />
</Frame>


# Orchestrate dbt Core projects with Airflow and Cosmos
Source: https://astronomer.io/docs/learn/airflow-dbt

Learn how to use Cosmos to orchestrate dbt Core projects with Airflow.

<img />[dbt Core](https://docs.getdbt.com/) is an open-source library for analytics engineering that helps users build interdependent SQL models for in-warehouse data transformation, using ephemeral compute of data warehouses. [Cosmos](https://astronomer.github.io/astronomer-cosmos/) is an open-source package developed by Astronomer to run dbt models that are part of a dbt Core project within Airflow.

### dbt on Airflow with Cosmos and the Astro CLI

The open-source provider package [Cosmos](https://astronomer.github.io/astronomer-cosmos/) allows you to integrate dbt jobs into Airflow by automatically creating Airflow tasks from dbt models. You can turn your dbt Core projects into an Airflow Dag or task group with just a few lines of code.

<Tip>You can find comprehensive instructions on how to set up Cosmos for different data warehouses, Cosmos configuration options, and how to optimize Cosmos performance in the [Orchestrating dbt with Apache Airflow® using Cosmos eBook](https://www.astronomer.io/ebooks/orchestrating-dbt-with-airflow-using-cosmos/?utm_source=website\&utm_medium=learn-guides\&utm_campaign=learn-dbt-tutorial-11-25) and a shorter summary of the most important concepts in the [Quick Notes: Airflow + dbt with Cosmos](https://www.astronomer.io/ebooks/quick-notes-airflow-dbt-with-cosmos/?utm_source=website\&utm_medium=learn-guides\&utm_campaign=learn-dbt-tutorial-11-25).</Tip>

## Why use Airflow with dbt Core?

dbt Core offers the possibility to build modular, reusable SQL components with built-in dependency management and [incremental builds](https://docs.getdbt.com/docs/build/incremental-models).

With [Cosmos](https://astronomer.github.io/astronomer-cosmos/), you can integrate dbt jobs into your open-source Airflow orchestration environment as standalone Dags or as task groups within Dags.

The benefits of using Airflow with dbt Core include:

* Use Airflow's [data-aware scheduling](/docs/learn/airflow-datasets) and [Airflow sensors](/docs/learn/what-is-a-sensor) to run models depending on other events in your data ecosystem.
* Turn each dbt model into a task, complete with Airflow features like [retries](/docs/learn/rerunning-dags#automatically-retry-tasks) and [error notifications](/docs/learn/error-notifications-in-airflow), as well as full observability into past runs directly in the Airflow UI.
* Run `dbt test` on tables created by individual models immediately after a model has completed. Catch issues before moving downstream and integrate additional [data quality checks](/docs/learn/data-quality) with your preferred tool to run alongside dbt tests.
* Run dbt projects using [Airflow connections](/docs/learn/connections) instead of dbt profiles. You can store all your connections in one place, directly within Airflow or by using a [secrets backend](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html).
* Use native support for installing and running dbt in a virtual environment to avoid dependency conflicts with Airflow.
* [Generate](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html) and [host](https://astronomer.github.io/astronomer-cosmos/configuration/hosting-docs.html) dbt docs with Airflow.

With Astro, you get all the above benefits and you can deploy your dbt project to your Astro Deployment independently of your Airflow project using the Astro CLI. For more information, see [Deploy dbt projects to Astro](/docs/astro/deploy-dbt-project).

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of dbt Core. See [What is dbt?](https://docs.getdbt.com/docs/introduction).
* Airflow fundamentals, such as writing Dags and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* How Airflow and dbt concepts relate to each other. See [Similar dbt and Airflow concepts](https://astronomer.github.io/astronomer-cosmos/getting_started/dbt-airflow-concepts.html).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow task groups. See [Airflow task groups](/docs/learn/task-groups).
* Airflow connections. See [Manage connections in Apache Airflow](/docs/learn/connections).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* Access to a data warehouse supported by dbt Core. See [dbt documentation](https://docs.getdbt.com/docs/supported-data-platforms) for all supported warehouses. This tutorial uses a Postgres database.

You don't need to have dbt Core installed locally in order to complete this tutorial.

## Step 1: Configure your Astro project

To use dbt Core with Airflow install dbt Core in a virtual environment and Cosmos in a new Astro project.

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-dbt-core-tutorial && cd astro-dbt-core-tutorial
   $ astro dev init
   ```

2. Add [Cosmos](https://github.com/astronomer/astronomer-cosmos), the [Airflow Postgres provider](https://airflow.apache.org/registry/providers/postgres/) and the [dbt Postgres adapter](https://github.com/dbt-labs/dbt-adapters) to your Astro project `requirements.txt` file. If you are using a different data warehouse, replace `apache-airflow-providers-postgres` and `dbt-postgres` with the provider package for your data warehouse. You can find information on all provider packages on the [Airflow Registry](https://airflow.apache.org/registry/).

   ```text wrap theme={null}
   astronomer-cosmos==1
   apache-airflow-providers-postgres==6
   apache-airflow-providers-common-sql==1
   dbt-postgres==1
   ```

3. (Alternative) If you can't install your dbt adapter in the same environment as Airflow due to package conflicts you can create a dbt executable in a virtual environment. In your `Dockerfile` add the following lines to the end of the file:

   ```text wrap theme={null}
   # replace dbt-postgres with another supported adapter if you're using a different warehouse type
   RUN python -m venv dbt_venv && source dbt_venv/bin/activate && \
       pip install --no-cache-dir dbt-postgres && deactivate
   ```

   This code runs a bash command when the Docker image is built that creates a virtual environment called `dbt_venv` inside of the Astro CLI scheduler container. The `dbt-postgres` package, which also contains `dbt-core`, is installed in the virtual environment. If you are using a different data warehouse, replace `dbt-postgres` with the adapter package for your data warehouse.

<Tip>
  There are other options to run Cosmos even if you can't install your dbt adapter in the `requirements.txt` file or create a virtual environment in your Docker image. See the [Cosmos documentation on execution modes](https://astronomer.github.io/astronomer-cosmos/getting_started/execution-modes.html) for more information.
</Tip>

## Step 2: Prepare your dbt project

To integrate your dbt project with Airflow, you need to add the project folder to your Airflow environment. For this step you can either add your own project or follow the steps below to create a simple project using two models.

1. Create a folder called `dbt` in your `include` folder.

2. In the `dbt` folder, create a folder called `my_simple_dbt_project`.

3. In the `my_simple_dbt_project` folder add your `dbt_project.yml`. This configuration file needs to contain at least the name of the project. This tutorial additionally shows how to inject a variable called `my_name` from Airflow into your dbt project.

   ```yaml wrap theme={null}
   version: '0.1'
   name: 'my_simple_dbt_project'
   vars:
       my_name: "No entry"
   ```

4. Add your dbt models in a subfolder called `models` in the `my_simple_dbt_project` folder. You can add as many models as you want to run. This tutorial uses the following two models:

   `model1.sql`:

   ```sql wrap theme={null}
   SELECT '{{ var("my_name") }}' as name
   ```

   `model2.sql`:

   ```sql wrap theme={null}
   SELECT * FROM {{ ref('model1') }}
   ```

   `model1.sql` selects the variable `my_name`. `model2.sql` depends on `model1.sql` and selects everything from the upstream model.

You should now have the following structure within your Airflow environment:

```text wrap theme={null}
.
└── dags
└── include
    └── dbt
        └── my_simple_dbt_project
           ├── dbt_project.yml
           └── models
               ├── model1.sql
               └── model2.sql
```

<Note>
  If storing your dbt project alongside your Airflow project isn't feasible, there are other ways to use Cosmos, even if the dbt project is hosted in a different location, for example by using a manifest file to parse the project and a containerized execution mode. See the [Cosmos documentation](https://astronomer.github.io/astronomer-cosmos/configuration/index.html) for more information.
</Note>

## Step 3: Create an Airflow connection to your data warehouse

Cosmos allows you to apply Airflow connections to your dbt project.

1. Start Airflow by running `astro dev start`.

2. In the Airflow UI, go to **Admin** -> **Connections** and click **+**.

3. Create a new connection named `db_conn`. Select the connection type and supplied parameters based on the data warehouse you are using. For a Postgres connection, enter the following information:

   * **Connection ID**: `db_conn`.
   * **Connection Type**: `Postgres`.
   * **Host**: Your Postgres host address.
   * **Schema**: Your Postgres database.
   * **Login**: Your Postgres login username.
   * **Password**: Your Postgres password.
   * **Port**: Your Postgres port.

<Note>
  If a connection type for your database isn't available, you might need to make it available by adding the [relevant provider package](https://airflow.apache.org/registry/) to `requirements.txt` and running `astro dev restart`.
</Note>

## Step 4: Write your Airflow Dag

The Dag you'll write uses Cosmos to create tasks from existing dbt models and the [`SQLExecuteQueryOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLExecuteQueryOperator) to query a table that was created. You can add more upstream and downstream tasks to embed the dbt project within other actions in your data ecosystem.

1. In your `dags` folder, create a file called `my_simple_dbt_dag.py`.

2. Copy and paste the following Dag code into the file:

   ```python expandable wrap theme={null}
   """
   ### Run a dbt Core project as a task group with Cosmos

   Simple DAG showing how to run a dbt project as a task group, using
   an Airflow connection and injecting a variable into the dbt project.
   """

   from airflow.sdk import dag, chain
   from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
   from cosmos import DbtTaskGroup, ProjectConfig, ProfileConfig, ExecutionConfig

   # adjust for other database types
   from cosmos.profiles.postgres import PostgresUserPasswordProfileMapping
   import os

   YOUR_NAME = "YOUR_NAME"
   CONNECTION_ID = "db_conn"
   DB_NAME = "YOUR_DB_NAME"
   SCHEMA_NAME = "YOUR_SCHEMA_NAME"
   MODEL_TO_QUERY = "model2"
   # The path to the dbt project
   DBT_PROJECT_PATH = f"{os.environ['AIRFLOW_HOME']}/include/dbt/my_simple_dbt_project"

   # OPTIONAL: The path where Cosmos will find the dbt executable
   # in the virtual environment created in the Dockerfile if you cannot 
   # install your dbt adapter in requirements.txt due to package conflicts.
   # DBT_EXECUTABLE_PATH = f"{os.environ['AIRFLOW_HOME']}/dbt_venv/bin/dbt"

   profile_config = ProfileConfig(
       profile_name="default",
       target_name="dev",
       profile_mapping=PostgresUserPasswordProfileMapping(
           conn_id=CONNECTION_ID,
           profile_args={"schema": SCHEMA_NAME},
       ),
   )

   # OPTIONAL: The path where Cosmos will find the dbt executable
   # execution_config = ExecutionConfig(
   #     dbt_executable_path=DBT_EXECUTABLE_PATH,
   # )


   @dag(
       params={"my_name": YOUR_NAME},
   )
   def my_simple_dbt_dag():
       transform_data = DbtTaskGroup(
           group_id="transform_data",
           project_config=ProjectConfig(DBT_PROJECT_PATH),
           profile_config=profile_config,
           # OPTIONAL: your execution config if you are using a virtual environment
           # execution_config=execution_config,
           operator_args={
               "vars": '{"my_name": {{ params.my_name }} }',
           },
           default_args={"retries": 2},
       )

       query_table = SQLExecuteQueryOperator(
           task_id="query_table",
           conn_id=CONNECTION_ID,
           sql=f"SELECT * FROM {DB_NAME}.{SCHEMA_NAME}.{MODEL_TO_QUERY}",
       )

       chain(transform_data, query_table)


   my_simple_dbt_dag()
   ```

   This Dag uses the `DbtTaskGroup` class from the Cosmos package to create a task group from the models in your dbt project. Dependencies between your dbt models are automatically turned into dependencies between Airflow tasks. Make sure to add your own values for `YOUR_NAME`, `YOUR_DB_NAME`, and `YOUR_SCHEMA_NAME`.

   Using the `vars` keyword in the dictionary provided to the `operator_args` parameter, you can inject variables into the dbt project. This DAG injects `YOUR_NAME` for the `my_name` variable. If your dbt project contains dbt tests, they will be run directly after a model has completed. Note that it is a best practice to set `retries` to at least 2 for all tasks that run dbt models.

<Tip> In some cases, especially in larger dbt projects, you might run into a `DagBag import timeout` error. This error can be resolved by increasing the value of the Airflow configuration [core.`dagbag_import_timeout`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dagbag-import-timeout).</Tip>

3. Run the Dag manually by clicking the play button and view the Dag in the graph view. Expand the [task groups](/docs/learn/task-groups) to see all tasks.

   <Frame>
     <img alt="Cosmos Dag graph view" />
   </Frame>

4. Check the [XCom](/docs/learn/airflow-passing-data-between-tasks) returned by the `query_table` task to see your name in the `model2` table.

<Note> The DbtTaskGroup class populates an Airflow task group with Airflow tasks created from dbt models inside of a normal Dag. To directly define a full Dag containing only dbt models use the `DbtDag` class, as shown in the [Cosmos documentation](https://astronomer.github.io/astronomer-cosmos/getting_started/astro.html).</Note>

Congratulations! You've run a Dag using Cosmos to automatically create tasks from dbt models. You can learn more about how to configure Cosmos in the [Cosmos documentation](https://astronomer.github.io/astronomer-cosmos/index.html).

<Tip> If you are running large dbt projects and want to increase performance, there are several options available to you. A recent feature is the experimental watcher execution mode that can reduce Dag execution time by up to 80% and reaches speeds on par with running `dbt build` with the dbt CLI. See the [Cosmos documentation](https://astronomer.github.io/astronomer-cosmos/getting_started/watcher-execution-mode.html) for more information.</Tip>

## Alternative ways to run dbt Core with Airflow

While using Cosmos is recommended, there are several other ways to run dbt Core with Airflow.

### Use the `BashOperator`

You can use the [`BashOperator`](https://airflow.apache.org/registry/providers/standard#standard-bash-BashOperator) to execute specific dbt commands. It's recommended to run `dbt-core` and the dbt adapter for your database in a virtual environment because there often are dependency conflicts between dbt and other packages.

The Dag below uses the `BashOperator` to activate the virtual environment and execute `dbt_run` for a dbt project.

```python wrap theme={null}
from airflow.sdk import dag
from airflow.providers.standard.operators.bash import BashOperator

PATH_TO_DBT_PROJECT = "<path to your dbt project>"
PATH_TO_DBT_VENV = "<path to your venv activate binary>"


@dag
def simple_dbt_dag():
    dbt_run = BashOperator(
        task_id="dbt_run",
        bash_command="source $PATH_TO_DBT_VENV && dbt run --models .",
        env={"PATH_TO_DBT_VENV": PATH_TO_DBT_VENV},
        cwd=PATH_TO_DBT_PROJECT,
    )


simple_dbt_dag()
```

Using the `BashOperator` to run `dbt run` and other dbt commands can be useful during development. However, running dbt at the project level has a couple of issues:

* There is low observability into what execution state the project is in.
* Failures are absolute and require all models in a project to be run again, which can be costly.

### Use a manifest file

Using a dbt-generated `manifest.json` file gives you more visibility into the steps dbt is running in each task. This file is generated in the target directory of your `dbt` project and contains its full representation. For more information on this file, see the [dbt documentation](https://docs.getdbt.com/reference/dbt-artifacts/).

Cosmos can parse manifest files, see the [Cosmos documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html) for more information.


# Orchestrate dbt Cloud jobs with Airflow
Source: https://astronomer.io/docs/learn/airflow-dbt-cloud

Learn how to use the dbt Cloud Provider to orchestrate dbt Cloud jobs with Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[dbt Cloud](https://getdbt.com/) is a managed service that provides a hosted architecture to run dbt, a tool that helps you build interdependent SQL models for in-warehouse data transformation.

The [dbt Cloud Airflow provider](https://airflow.apache.org/registry/providers/dbt-cloud) allows users to orchestrate and execute actions in dbt Cloud as DAGs. Running dbt with Airflow ensures a reliable, scalable environment for models, as well as the ability to trigger models based on upstream dependencies in your data ecosystem.

<Info>
  For a tutorial on how to use the open-source dbt Core package with Airflow see [Orchestrate dbt Core with Cosmos](/docs/learn/2.x/airflow-dbt).
</Info>

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of dbt. See [Getting started with dbt Cloud](https://docs.getdbt.com/guides/getting-started).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* A dbt Cloud account. A [14-day free trial](https://www.getdbt.com/signup/) is available.
* Access to a data warehouse supported by dbt Cloud. View the [dbt documentation](https://docs.getdbt.com/docs/supported-data-platforms) for an up-to-date list of adapters.

## Step 1: Configure your Astro project

An Astro project contains all of the files you need to run Airflow locally.

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-dbt-cloud-tutorial && cd astro-dbt-cloud-tutorial
   $ astro dev init
   ```

2. Add the [dbt Cloud provider](https://airflow.apache.org/registry/providers/dbt-cloud) to your `requirements.txt` file.

   ```text wrap theme={null}
   apache-airflow-providers-dbt-cloud
   ```

3. Run the following command to start your Astro project:

   ```sh wrap theme={null}
   $ astro dev start
   ```

## Step 2: Configure a dbt connection

1. In the Airflow UI, go to **Admin** -> **Connections** and click **+**.

2. Create a new connection named `dbt_conn` and choose the `dbt Cloud` connection type. Configure the following values for the connection:

   * **Tenant**: The URL under which your API cloud is hosted. The default value is `cloud.getdbt.com`.
   * **Account ID**: (Optional) The default dbt account to use with this connection.
   * **API Token**: A dbt [user token](https://docs.getdbt.com/docs/dbt-cloud-apis/user-tokens).

## Step 3: Configure a dbt Cloud job

In the dbt Cloud UI, create one dbt Cloud job. The contents of this job don't matter for this tutorial. Optionally, you can use the jaffle shop example from dbt's [Quickstart documentation](https://docs.getdbt.com/docs/quickstarts/overview). Copy the dbt Cloud `job_id` for use in the next step.

## Step 4: Write a dbt Cloud DAG

1. In your `dags` folder, create a file called `check_before_running_dbt_cloud_job.py`.

2. Copy the following code into the file, making sure to replace `<your dbt Cloud job id>` with the `job_id` you copied.

   ```python expandable wrap theme={null}
   from airflow.decorators import dag, task
   from airflow.providers.dbt.cloud.hooks.dbt import DbtCloudHook, DbtCloudJobRunStatus
   from airflow.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperator
   from pendulum import datetime

   DBT_CLOUD_CONN_ID = "dbt_conn"
   JOB_ID = "<your dbt Cloud job id>"


   @dag(
       start_date=datetime(2022, 2, 10),
       schedule="@daily",
       catchup=False,
   )
   def check_before_running_dbt_cloud_job():
       @task.short_circuit
       def check_job(job_id):
           """
           Retrieves the last run for a given dbt Cloud job and checks
           to see if the job is not currently running.
           """
           hook = DbtCloudHook(DBT_CLOUD_CONN_ID)
           runs = hook.list_job_runs(job_definition_id=job_id, order_by="-id")
           if not runs[0].json().get("data"):
               return True
           else:
               latest_run = runs[0].json()["data"][0]
               return DbtCloudJobRunStatus.is_terminal(latest_run["status"])

       trigger_job = DbtCloudRunJobOperator(
           task_id="trigger_dbt_cloud_job",
           dbt_cloud_conn_id=DBT_CLOUD_CONN_ID,
           job_id=JOB_ID,
           check_interval=600,
           timeout=3600,
       )

       check_job(job_id=JOB_ID) >> trigger_job


   check_before_running_dbt_cloud_job()
   ```

   This DAG shows a simple implementation of using the [`DbtCloudRunJobOperator`](https://airflow.apache.org/registry/providers/dbt-cloud#dbt-cloud-dbt-DbtCloudRunJobOperator) and [`DbtCloudHook`](https://airflow.apache.org/registry/providers/dbt-cloud#dbt-cloud-dbt-DbtCloudHook). The DAG consists of two tasks:

   * `check_job_is_not_running`: Uses the [`ShortCircuitOperator`](https://airflow.apache.org/registry/providers/standard#standard-python-ShortCircuitOperator) to ensure that the dbt Cloud job with the specified `JOB_ID` isn't currently running. The list of currently running dbt Cloud jobs is retrieved using the `list_job_runs()` method of the `DbtCloudHook`. Next, the `latest_run` is selected and its `status` parameter will be evaluated for being a terminal status or not. If the status of the latest run is terminal, this means the job isn't currently running and the pipeline should go ahead triggering another run of this job. If the status of the latest run isn't terminal, this means that a job with the given `JOB_ID` is still running in the dbt Cloud. The function used in the `ShortCircuitOperator` will return `False`, therefore causing the DAG to short circuit and skip any downstream tasks.
   * `trigger_dbt_cloud_job`: Uses the `DbtCloudRunJobOperator` to trigger a run of the dbt Cloud job with the correct `JOB_ID`.

3. Run the DAG and verify that the dbt Cloud job ran in the dbt Cloud UI.

   The full code for this example, along with other DAGs that implement the dbt Cloud provider, can be found on the [Astronomer Registry](https://registry.astronomer.io/dags/dbt_cloud_operational_check/versions/3.0.0).

Congratulations! You've run a DAG which uses the dbt Cloud provider to orchestrate a job run in dbt Cloud.

<Info>
  You can find more examples of how to use dbt Cloud with Airflow in [dbt's documentation](https://docs.getdbt.com/guides/orchestration/airflow-and-dbt-cloud/1-airflow-and-dbt-cloud).
</Info>

## Deferrable dbt Cloud operators

If you are orchestrating long-running dbt Cloud jobs using Airflow, you may benefit from leveraging [deferrable operators](/docs/learn/deferrable-operators) for cost savings and scalability. The Astronomer providers package contains deferrable versions of several dbt modules:

* [DbtCloudHookAsync](https://airflow.apache.org/registry/providers/dbt-cloud#dbt-cloud-dbt-DbtCloudHook): Asynchronous version of the `DbtCloudHook`.
* [`DbtCloudRunJobTrigger`](https://airflow.apache.org/registry/providers/dbt-cloud#dbt-cloud-dbt-DbtCloudRunJobTrigger): Trigger class used in deferrable dbt Cloud operators.
* [DbtCloudJobRunSensorAsync](https://registry.astronomer.io/providers/astronomer-providers/modules/dbtcloudjobrunsensorasync): Asynchronously checks the status of dbt Cloud job runs.
* [DbtCloudRunJobOperatorAsync](https://airflow.apache.org/registry/providers/dbt-cloud#dbt-cloud-dbt-DbtCloudRunJobOperator): Executes a dbt Cloud job asynchronously and waits for the job to reach a terminal status before completing successfully.

## See also

* Webinar: [Introducing Cosmos: The Easy Way to Run dbt Models in Airflow](https://www.astronomer.io/events/webinars/introducing-cosmos-the-east-way-to-run-dbt-models-in-airflow/).
* Demo: [See how to deploy your dbt projects to Astro](https://www.astronomer.io/dbt-demo/)


# Introduction to the TaskFlow API and Airflow decorators
Source: https://astronomer.io/docs/learn/airflow-decorators

An overview of Airflow decorators and how they can improve the DAG authoring experience.

The *TaskFlow API* is a functional API for using decorators to define DAGs and tasks, which simplifies the process for passing data between tasks and defining dependencies. You can use TaskFlow decorator functions (for example, `@task`) to pass data between tasks by providing the output of one task as an argument to another task. Decorators are a simpler, cleaner way to define your tasks and DAGs and can be used in combination with traditional operators.

In this guide, you'll learn about the benefits of decorators and the decorators available in Airflow. You'll also review an example DAG and learn when you should use decorators and how you can combine them with traditional operators in a DAG.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Python. See the [Python Documentation](https://docs.python.org/3/tutorial/index.html).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).

## What is a decorator?

In Python, [decorators](https://realpython.com/primer-on-python-decorators/) are functions that take another function as an argument and extend the behavior of that function. For example, the `@multiply_by_100_decorator` takes any function as the `decorated_function` argument and returns the result of that function multiplied by 100.

```python wrap theme={null}
# definition of the decorator function
def multiply_by_100_decorator(decorated_function):
    def wrapper(num1, num2):
        result = decorated_function(num1, num2) * 100
        return result

    return wrapper

# definition of the `add` function decorated with the `multiply_by_100_decorator`
@multiply_by_100_decorator
def add(num1, num2):
    return num1 + num2

# definition of the `subtract` function decorated with the `multiply_by_100_decorator`
@multiply_by_100_decorator
def subtract(num1, num2):
    return num1 - num2

# calling the decorated functions
print(add(1, 9))  # prints 1000
print(subtract(4, 2))  # prints 200
```

In the context of Airflow, decorators contain more functionality than this simple example, but the basic idea is the same: the Airflow decorator function extends the behavior of a normal Python function to turn it into an Airflow task, task group or DAG.

## When to use the TaskFlow API

The purpose of the TaskFlow API in Airflow is to simplify the DAG authoring experience by eliminating the boilerplate code required by traditional operators. The result can be cleaner DAG files that are more concise and easier to read.

In general, whether you use the TaskFlow API is a matter of your own preference and style. In most cases, a TaskFlow decorator and the corresponding traditional operator will have the same functionality. You can also [mix decorators and traditional operators](#mixing-taskflow-decorators-with-traditional-operators) within a single DAG.

## How to use the TaskFlow API

The TaskFlow API allows you to write your Python tasks with decorators. It handles passing data between tasks using XCom and infers task dependencies automatically.

Using decorators to define your Python functions as tasks is easy. Let's take a before and after example. Under the **Traditional syntax** tab below, there is a basic ETL DAG with tasks to get data from an API, process the data, and store it. Click the **Decorators** tab to see the same DAG written using Airflow decorators.

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  import logging
  from datetime import datetime

  import requests
  from airflow import DAG
  from airflow.operators.python import PythonOperator

  API = "https://api.coingecko.com/api/v3/simple/price?ids=bitcoin&vs_currencies=usd&include_market_cap=true&include_24hr_vol=true&include_24hr_change=true&include_last_updated_at=true"


  def _extract_bitcoin_price():
      return requests.get(API).json()["bitcoin"]


  def _process_data(ti):
      response = ti.xcom_pull(task_ids="extract_bitcoin_price")
      logging.info(response)
      processed_data = {"usd": response["usd"], "change": response["usd_24h_change"]}
      ti.xcom_push(key="processed_data", value=processed_data)


  def _store_data(ti):
      data = ti.xcom_pull(task_ids="process_data", key="processed_data")
      logging.info(f"Store: {data['usd']} with change {data['change']}")


  with DAG(
      "classic_dag", schedule="@daily", start_date=datetime(2021, 12, 1), catchup=False
  ):
      extract_bitcoin_price = PythonOperator(
          task_id="extract_bitcoin_price", python_callable=_extract_bitcoin_price
      )

      process_data = PythonOperator(task_id="process_data", python_callable=_process_data)

      store_data = PythonOperator(task_id="store_data", python_callable=_store_data)

      extract_bitcoin_price >> process_data >> store_data
  ```
</details>

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  import logging
  from datetime import datetime
  from typing import Dict

  import requests
  from airflow.decorators import dag, task

  API = "https://api.coingecko.com/api/v3/simple/price?ids=bitcoin&vs_currencies=usd&include_market_cap=true&include_24hr_vol=true&include_24hr_change=true&include_last_updated_at=true"


  @dag(schedule="@daily", start_date=datetime(2021, 12, 1), catchup=False)
  def taskflow():
      @task(task_id="extract", retries=2)
      def extract_bitcoin_price() -> Dict[str, float]:
          return requests.get(API).json()["bitcoin"]

      @task(multiple_outputs=True)
      def process_data(response: Dict[str, float]) -> Dict[str, float]:
          logging.info(response)
          return {"usd": response["usd"], "change": response["usd_24h_change"]}

      @task
      def store_data(data: Dict[str, float]):
          logging.info(f"Store: {data['usd']} with change {data['change']}")

      store_data(process_data(extract_bitcoin_price()))


  taskflow()
  ```
</details>

The decorated version of the DAG eliminates the need to explicitly instantiate the `PythonOperator`, has much less code and is easier to read. Notice that it also doesn't require using `ti.xcom_pull` and `ti.xcom_push` to [pass data between tasks](/docs/learn/airflow-passing-data-between-tasks). This is all handled by the TaskFlow API when you define your task dependencies with `store_data(process_data(extract_bitcoin_price()))`.

Here are some other things to keep in mind when using decorators:

* You must call all decorated functions in your DAG file so that Airflow can register the task or DAG. For example, `taskflow()` is called at the end of the previous example to call the DAG function.

* When you define a task, the `task_id` defaults to the name of the function you decorated. If you want to change this behavior, you can pass a `task_id` to the decorator as shown in the `extract` task example. Similarly, other `BaseOperator` task-level parameters, such as `retries` or `pool`, can be defined within the decorator:

  ```python wrap theme={null}
  from airflow.sdk import task

  @task(
      task_id="say_hello_world"
      retries=3,
      pool="my_pool",
  )
  def taskflow_func():
      return "Hello World"

  taskflow_func()  # this creates a task with the task_id `say_hello_world`
  ```

* Override task-level parameters when you call the task by using the `.override()` method. The `()` at the end of the line calls the task with the overridden parameters applied.

  ```python wrap theme={null}
  # this creates a task with the task_id `greeting`
  taskflow_func.override(retries=5, pool="my_other_pool", task_id="greeting")()
  ```

* If you call the same task multiple times and don't override the `task_id`, Airflow creates multiple unique task IDs by appending a number to the end of the original task ID (for example, `say_hello`, `say_hello__1`, `say_hello__2`, etc). You can see the result of this in the following example:

  ```python wrap theme={null}
  from airflow.sdk import task

  # task definition
  @task
  def say_hello(dog):
      return f"Hello {dog}!"

  ### calling the task 4 times, creating 4 tasks in the DAG
  # this task will have the id `say_hello` and print "Hello Avery!"
  say_hello("Avery")
  # this task will have the id `greet_dog` and print "Hello Piglet!"
  say_hello.override(task_id="greet_dog")("Piglet")
  # this task will have the id `say_hello__1` and print "Hello Peanut!"
  say_hello("Peanut")
  # this task will have the id `say_hello__2` and print "Hello Butter!"
  say_hello("Butter")
  ```

* You can decorate a function that is imported from another file as shown in the following code snippet:

  ```python wrap theme={null}
  from airflow.sdk import task
  from include.my_file import my_function

  @task
  def taskflow_func():
      my_function()
  ```

  This is recommended in cases where you have lengthy Python functions since it will make your DAG file easier to read.

* You can assign the output of a called decorated task to a Python object to be passed as an argument into another decorated task. This is helpful when the output of one decorated task is needed in several downstream functions.

  ```python wrap theme={null}
  from airflow.sdk import task

  @task 
  def get_fruit_options():
      return ["peach", "raspberry", "pineapple"]

  @task
  def eat_a_fruit(list):
      index = random.randint(0, len(list) - 1)
      print(f"I'm eating a {list[index]}!")

  @task 
  def gift_a_fruit(list):
      index = random.randint(0, len(list) - 1)
      print(f"I'm giving you a {list[index]}!")

  # you can assign the output of a decorated task to a Python object
  my_fruits = get_fruit_options()
  eat_a_fruit(my_fruits)
  gift_a_fruit(my_fruits)
  ```

View more examples on how to use Airflow task decorators in the [Astronomer webinars](https://www.astronomer.io/events/webinars/writing-functional-dags-with-decorators/) and the Apache Airflow [TaskFlow API tutorial](https://airflow.apache.org/docs/apache-airflow/stable/tutorial/taskflow.html).

## Mixing TaskFlow decorators with traditional operators

If you have a DAG that uses `PythonOperator` and other operators that don't have decorators, you can easily combine decorated functions and traditional operators in the same DAG. For example, you can add a `BashOperator` to the previous example by updating your code to the following:

```python expandable wrap theme={null}
import logging
from datetime import datetime
from typing import Dict

import requests
from airflow.decorators import dag, task
from airflow.operators.email import EmailOperator

API = "https://api.coingecko.com/api/v3/simple/price?ids=bitcoin&vs_currencies=usd&include_market_cap=true&include_24hr_vol=true&include_24hr_change=true&include_last_updated_at=true"


@dag(schedule="@daily", start_date=datetime(2021, 12, 1), catchup=False)
def taskflow():
    @task(task_id="extract", retries=2)
    def extract_bitcoin_price() -> Dict[str, float]:
        return requests.get(API).json()["bitcoin"]

    @task(multiple_outputs=True)
    def process_data(response: Dict[str, float]) -> Dict[str, float]:
        logging.info(response)
        return {"usd": response["usd"], "change": response["usd_24h_change"]}

    @task
    def store_data(data: Dict[str, float]):
        logging.info(f"Store: {data['usd']} with change {data['change']}")

    email_notification = EmailOperator(
        task_id="email_notification",
        to="noreply@astronomer.io",
        subject="dag completed",
        html_content="the dag has finished",
    )

    store_data(process_data(extract_bitcoin_price())) >> email_notification


taskflow()
```

Note that when adding traditional operators, dependencies are still defined using the `chain` function or bit-shift operators (`>>`). See [Manage task and task group dependencies in Airflow](/docs/learn/managing-dependencies) for more information on how to explicitly define dependencies in Airflow.

You can pass information between decorated tasks and traditional operators using [XCom](/docs/learn/airflow-passing-data-between-tasks). See the following sections for examples.

### TaskFlow to TaskFlow

If both tasks are defined using the TaskFlow API, you can pass information directly between them by providing the called task function as a positional argument to the downstream task. Airflow will infer the dependency between the two tasks.

```python wrap theme={null}
from airflow.sdk import task

@task
def get_23_TF():
    return 23

@task
def plus_10_TF(x):
    return x + 10

plus_10_TF(get_23_TF())  # plus_10_TF will return 33
# or `plus_10_TF(x=get_23_TF())` if you want to use kwargs
```

### Traditional operator to TaskFlow

Pass the `.output` values of traditional Operator-based tasks to the callable of TaskFlow tasks to automatically create a relationship between the two tasks. For example:

```python wrap theme={null}
from airflow.sdk import task
from airflow.providers.standard.operators.python import PythonOperator

def first_task_callable():
    return "hello world"

first_task = PythonOperator(
    task_id="first_task",
    python_callable=first_task_callable
)

first_task_result = first_task.output

@task
def second_task(first_task_result_value):
    return f"{first_task_result_value} and hello again"

# Providing first_task_result (traditional operator result) as an argument to a second_task
# (a TaskFlow function) automatically registers the dependency between the tasks.
second_task(first_task_result)
```

When no outputs are passed as arguments to automatically register dependencies, but you still want one task to complete before the second starts, use the `chain` function (or `>>`) to create the relationship between the first task and the second task:

```python wrap theme={null}
from airflow.sdk import task, chain
from airflow.providers.standard.operators.empty import EmptyOperator

@task
def first_task():
    return "hello world"

_second_task = EmptyOperator(
    task_id="second_task",
)

_first_task = first_task()

chain(_first_task, _second_task)
```

The data a task returns varies based on its operator. A searchable registry of operators at the [Airflow Registry](https://airflow.apache.org/registry/) describes what operators return and documents any operator parameters that can be used to control the return format.

### TaskFlow to traditional operator

TaskFlow tasks, when called, return a reference (referred to internally as an XComArg) that can be passed into [templateable fields](/docs/learn/templating#templateable-fields-and-scripts) of traditional operators to automatically create a relationship between the two tasks.

The list of templateable fields varies by operator. A searchable registry of operators at the [Airflow Registry](https://airflow.apache.org/registry/) details which fields are templateable by default and users can [control which fields are templateable](/docs/learn/templating#templating-additional-fields).

Here's how you could pass the result of a TaskFlow function to a traditional `PythonOperator`'s callable using an argument:

```python wrap theme={null}
from airflow.sdk import task, chain
from airflow.providers.standard.operators.python import PythonOperator

@task
def first_task():
    return "hello"

_first_task = first_task()

def second_task_callable(x):
    uppercase_text = x.upper()
    return f"Upper cased version of prior task result: {uppercase_text}"

# when first_task_result (an XComArg) is provided as an argument to a templated function (op_args)
# Airflow automatically registers that second_task depends on first_task
_second_task = PythonOperator(
    task_id="second_task",
    python_callable=second_task_callable,
    op_args=[ _first_task ] # note op_args requires a LIST of XComArgs
)

```

When only the order of task execution is important, don't pass the return value of the first task as a parameter to the second task - instead use `chain` to explicitly create the relationship. For example:

```python wrap theme={null}
from airflow.sdk import task, chain
from airflow.providers.standard.operators.empty import EmptyOperator

@task
def first_task():
    return "hello world"

_first_task = first_task()
_second_task = EmptyOperator(task_id="second_task")

chain(_first_task, _second_task)
```

### Traditional operator to traditional operator

For the sake of completeness the below example shows how to use the output of one traditional operator in another traditional operator by accessing the `.output` attribute of the upstream task. The dependency has to be defined explicitly using the `chain` function.

```python wrap theme={null}
from airflow.sdk import chain
from airflow.providers.standard.operators.python import PythonOperator

def get_23_traditional():
    return 23

def plus_10_traditional(x):
    return x + 10

get_23_task = PythonOperator(
    task_id="get_23_task",
    python_callable=get_23_traditional
)

plus_10_task = PythonOperator(
    task_id="plus_10_task",
    python_callable=plus_10_traditional,
    op_args=[get_23_task.output]
)

# plus_10_task will return 33

# when only using traditional operators, define dependencies explicitly
chain(get_23_task, plus_10_task)
```

<Info>
  If you want to access any XCom that isn't the returned value from an operator, you can use the `xcom_pull` method inside a function, see [how to access ti / `task_instance` in the Airflow context](/docs/learn/airflow-context#ti-/-task_instance) for an example. Traditional operators can also pull from XCom using [Jinja templates](/docs/learn/templating) in templateable parameters.
</Info>

## Available Airflow decorators

There are several decorators available to use with Airflow. This list provides a reference of currently available decorators:

* DAG decorator (`@dag()`), which creates a DAG.
* TaskGroup decorator (`@task_group()`), which creates a [TaskGroup](/docs/learn/task-groups).
* Task decorator (`@task()`), which creates a Python task.
* Bash decorator (`@task.bash()`) which creates a [`BashOperator`](/docs/learn/bashoperator#when-to-use-the-bashoperator) task.
* Python Virtual Env decorator (`@task.virtualenv()`), which runs your Python task in a [virtual environment](https://www.astronomer.io/events/webinars/running-airflow-tasks-in-isolated-environments/).
* Docker decorator (`@task.docker()`), which creates a [`DockerOperator`](https://airflow.apache.org/registry/providers/docker#docker-docker-DockerOperator) task.
* [Short circuit decorator](/docs/learn/airflow-branch-operator#@task-short_circuit-shortcircuitoperator) (`@task.short_circuit()`), which evaluates a condition and skips downstream tasks if the condition is False.
* [Branch decorator](/docs/learn/airflow-branch-operator#@task-branch-branchpythonoperator) (`@task.branch()`), which creates a branch in your DAG based on an evaluated condition.
* [BranchExternalPython decorator](/docs/learn/airflow-branch-operator#other-branch-operators) (`@task.branch_external_python`), which creates a branch in your DAG running Python code in a pre-existing virtual environment.
* [`BranchPythonVirtualenvOperator`](/docs/learn/airflow-branch-operator#other-branch-operators) (`@task.branch_virtualenv`), which creates a branch in your DAG running Python code in a newly created virtual environment. The environment can be cached by providing a `venv_cache_path`.
* Kubernetes pod decorator (`@task.kubernetes()`), which runs a [`KubernetesPodOperator`](/docs/learn/kubepod-operator) task.
* [Sensor decorator](/docs/learn/what-is-a-sensor#sensor-decorator-/-pythonsensor) (`@task.sensor()`), which turns a Python function into a sensor.
* [PySpark decorator](https://airflow.apache.org/docs/apache-airflow-providers-apache-spark/stable/decorators/pyspark.html) (`@task.pyspark()`), which is injected with a SparkSession and SparkContext object if available.

You can also [create your own custom task decorator](https://airflow.apache.org/docs/apache-airflow/stable/howto/create-custom-decorator.html).


# Use DuckDB with Apache Airflow
Source: https://astronomer.io/docs/learn/airflow-duckdb

Learn how to use DuckDB with Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[DuckDB](https://duckdb.org/) is an open-source in-process SQL OLAP database management system. It allows you to run complex queries on relational datasets using either local, file-based DuckDB instances, or the cloud service [MotherDuck](https://motherduck.com/). The ability to create a local DuckDB instance is useful for testing complex Airflow pipelines without the need to connect to a remote database.

Airflow can interact with DuckDB in two key ways:

* Use the DuckDB Python package directly in [@task decorated tasks](/docs/learn/airflow-decorators). This method is useful if you want to do ad-hoc analysis in-memory or combine information stored in various DuckDB files.
* Connect to DuckDB using the [DuckDB Airflow provider](https://airflow.apache.org/registry/providers/airflow-provider-duckdb/0.1.0/). The DuckDB Airflow provider is ideal if you access the same DuckDB database from many tasks in your Airflow environment and want to standardize this connection in a central place. You can also use the `DuckDBHook` to create custom operators to modularize your DuckDB interactions from within Airflow.

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Webinar: [How to use DuckDB with Airflow](https://www.astronomer.io/events/webinars/how-to-use-duckdb-with-airflow/).
  * Example repository: [Astronomer's DuckDB example repository](https://github.com/astronomer/airflow-duckdb-examples).
</Tip>

## Time to complete

This tutorial takes approximately 15 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of DuckDB. See [the DuckDB documentation](https://duckdb.org/docs/guides/index).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow decorators. See [Introduction to Airflow decorators](/docs/learn/airflow-decorators).
* Airflow connections. See [Manage connections in Apache Airflow](/docs/learn/connections).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).

## Step 1: Configure your Astro project

To use DuckDB with Airflow, install the [DuckDB Airflow provider](https://github.com/astronomer/airflow-provider-duckdb) in your Astro project. This will also install the newest version of the [DuckDB Python package](https://pypi.org/project/duckdb).

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-duckdb-tutorial && cd astro-duckdb-tutorial
   $ astro dev init
   ```

2. Add the DuckDB Airflow provider to your Astro project `requirements.txt` file.

   ```text wrap theme={null}
   airflow-provider-duckdb==0.2.0
   ```

3. If you are connecting to MotherDuck, the DuckDB cloud service, you need to use the amd64 version of Astro Runtime to prevent package conflicts. In this case, replace the `FROM` statement in your Dockerfile with the following line:

   ```Dockerfile wrap theme={null}
   FROM --platform=linux/amd64 quay.io/astronomer/astro-runtime:8.6.0
   ```

   If you are only using DuckDB locally, you don't need to modify your Dockerfile.

## Step 2: Create a DAG using the DuckDB Python package

You can use the [DuckDB Python package](https://pypi.org/project/duckdb/) directly in your `@task` decorated tasks. This method doesn't require you to configure an Airflow connection.

1. Start Airflow by running `astro dev start`.

2. Create a new file in your `dags` folder called `duckdb_tutorial_dag_1.py`.

3. Copy and paste the following DAG code into the file:

   ```python expandable wrap theme={null}
   """
   ### DuckDB Tutorial DAG 1

   This DAG shows how to use the DuckDB package directly in a @task decorated task.
   """

   from airflow.decorators import dag, task
   from pendulum import datetime
   import duckdb
   import pandas as pd


   @dag(start_date=datetime(2023, 6, 1), schedule=None, catchup=False)
   def duckdb_tutorial_dag_1():
       @task
       def create_pandas_df():
           "Create a pandas DataFrame with toy data and return it."
           ducks_in_my_garden_df = pd.DataFrame(
               {"colors": ["blue", "red", "yellow"], "numbers": [2, 3, 4]}
           )

           return ducks_in_my_garden_df

       @task
       def create_duckdb_table_from_pandas_df(ducks_in_my_garden_df):
           "Create a table in DuckDB based on a pandas DataFrame and query it"

           # change the path to connect to a different database
           conn = duckdb.connect("include/my_garden_ducks.db")
           conn.sql(
               f"""CREATE TABLE IF NOT EXISTS ducks_garden AS 
               SELECT * FROM ducks_in_my_garden_df;"""
           )

           sets_of_ducks = conn.sql("SELECT numbers FROM ducks_garden;").fetchall()
           for ducks in sets_of_ducks:
               print("quack " * ducks[0])

       create_duckdb_table_from_pandas_df(ducks_in_my_garden_df=create_pandas_df())


   duckdb_tutorial_dag_1()
   ```

   This simple DAG passes a pandas DataFrame from an upstream task to a downstream task. The downstream task uses the DuckDB Python package to create and query a table in DuckDB. You can control the database you connect to by changing the string in the `duckdb.connect()` function:

   * Use an empty string to utilize an in-memory database (For example, `duckdb.connect("")`).
   * Specify a local file path to create/connect to a local DuckDB database in which your table will persist (For example, `duckdb.connect("include/my_garden_ducks.db")`)
   * Specify a MotherDuck connection string without a database to connect to your default MotherDuck database (For example, `duckdb.connect(f"motherduck:?token={YOUR_MOTHERDUCK_TOKEN}")`).
   * Specify a MotherDuck connection string with a database to connect to a specific MotherDuck database (For example, `duckdb.connect(f"motherduck:{YOUR_DB}?token={YOUR_MOTHERDUCK_TOKEN}")`)

4. Open Airflow at `http://localhost:8080/`. Run the DAG manually by clicking the play button, then click the DAG name to view the DAG in the **Grid** view. In the logs for `create_duckdb_table_from_pandas_df`, you will find a quack for each duck in your garden.

   <Frame>
     <img alt="DuckDB tutorial DAG 1 Grid view" />
   </Frame>

## Step 3: Create a DuckDB Airflow connection

Next, you will create a DAG that instead uses the DuckDB Airflow provider. To use the provider, you will need to define an Airflow connection to your DuckDB database.

1. In the Airflow UI, go to **Admin** -> **Connections** and click **+**.

2. Create a new connection named `my_local_duckdb_conn` using the following information:

   * **Connection ID**: `my_local_duckdb_conn`.
   * **Connection Type**: `DuckDB`.
   * **Path to local database file**: `include/my_garden_ducks.db`.

   <Frame>
     <img alt="DuckDB tutorial DAG 1 Grid view" />
   </Frame>

3. Click **Save**. Note that you can't currently test a connection to DuckDB from the Airflow UI.

<Info>
  If you are connecting to MotherDuck, you will need to add your [MotherDuck Service token](https://motherduck.com/docs/authenticating-to-motherduck/) in the **MotherDuck Service token** field and leave the **Path to local database file** field empty. Optionally, you can add a MotherDuck database name in the **MotherDuck database name** field. The default name is the default MotherDuck database (`my_db`).
</Info>

## Step 4: Create a DAG using the Airflow DuckDB provider

1. Create a new file in your `dags` folder called `duckdb_tutorial_dag_2.py`.

2. Copy and paste the following DAG code into the file:

   ```python wrap theme={null}
   """
   ### DuckDB tutorial DAG 2

   This DAG shows how to use the DuckDBHook in an Airflow task.
   """

   from airflow.decorators import dag, task
   from pendulum import datetime
   from duckdb_provider.hooks.duckdb_hook import DuckDBHook

   DUCKDB_CONN_ID = "my_local_duckdb_conn"
   DUCKDB_TABLE_NAME = "ducks_garden"


   @dag(start_date=datetime(2023, 6, 1), schedule=None, catchup=False)
   def duckdb_tutorial_dag_2():
       @task
       def query_duckdb(my_table, conn_id):
           my_duck_hook = DuckDBHook.get_hook(conn_id)
           conn = my_duck_hook.get_conn()

           r = conn.execute(f"SELECT * FROM {my_table};").fetchall()
           print(r)

           return r

       query_duckdb(my_table=DUCKDB_TABLE_NAME, conn_id=DUCKDB_CONN_ID)


   duckdb_tutorial_dag_2()
   ```

   This simple DAG will query all information from a table in a DuckDB instance. Make sure the table you are querying exists in the DuckDB instance you specified in your DuckDB connection.

3. Open Airflow at `http://localhost:8080/`. Run the DAG manually by clicking the play button.

<Info>
  You can use the `DuckDBHook` to create [custom operators](/docs/learn/airflow-importing-custom-hooks-operators) to modularize your interactions with DuckDB. You can find an example of a [custom DuckDB operator for ingesting Excel files](https://github.com/astronomer/airflow-duckdb-examples/blob/main/include/custom_operators/duckdb_operator.py).
</Info>

## Conclusion

Congratulations! You successfully used DuckDB with Airflow. Quack!


# Event-driven scheduling
Source: https://astronomer.io/docs/learn/airflow-event-driven-scheduling

Learn how schedule DAGs based on messages in a message queue.

Event-driven scheduling is a sub-type of data-aware scheduling where a DAG is triggered when messages are posted to a message queue. This is useful for scenarios where you want to trigger a DAG based on events that occur outside of Airflow, such as data delivery to an external system or IoT sensor events, and is key to inference execution pipelines.

In this guide you will learn about the concepts used in event-driven scheduling, common usage patterns, and how to implement an example using Amazon SQS and an example using Apache Kafka.

## Assumed knowledge

To get the most out of this guide, you should have an existing knowledge of:

* Airflow assets. See [Assets and data-aware scheduling in Airflow](/docs/learn/airflow-datasets).

## Concepts

There are a number of concepts that are important to understand when using event-driven scheduling.

* **Data-aware scheduling**: Data-aware or data-driven scheduling refers to all ways you can schedule a DAG based on updates to assets. Updates to assets outside of event-driven scheduling occur by tasks in the same Airflow instance completing successfully, manually through the Airflow UI, or through a call to the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html). See [Assets and data-aware scheduling in Airflow](/docs/learn/airflow-datasets).
* **Event-driven scheduling**: Event-driven scheduling is a sub-type of data-aware scheduling where a DAG is run based on messages posted to a message queue. This message in the queue is triggered by an event that occurs outside of Airflow.
* **Message queue**: A message queue is a service that allows you to send and receive messages between different systems. Examples of message queues include Amazon SQS, RabbitMQ, and Apache Kafka. Event-driven scheduling in Airflow 3.0 is supported for Amazon SQS with support for other message queues planned for future releases.
* **Trigger**: A trigger is an asynchronous Python function running in the Airflow triggerer [component](/docs/learn/airflow-components). Triggers that inherit from `BaseEventTrigger` can be used in AssetWatchers for event-driven scheduling. The trigger is responsible for polling the message queue for new messages, when a new message is found, a `TriggerEvent` is created. The message is deleted from the queue.
* **AssetWatcher**: An AssetWatcher is a class in Airflow that watches one or more triggers for events. When a trigger fires a `TriggerEvent`, the AssetWatcher updates the asset it is associated with, creating an `AssetEvent`. The payload of the trigger is attached to the `AssetEvent` in its `extra` dictionary.
* **AssetEvent**: An `Asset` is an object in Airflow that represents a concrete or abstract data entity.
  For example, an asset can be a file, table in a database, or not tied to any specific data. An `AssetEvent` represents one update to an asset. In the context of event-driven scheduling, the `AssetEvent` represents one message having been detected in the message queue.

## When to use event-driven scheduling

Basic and advanced [data-aware scheduling](/docs/learn/airflow-datasets) are great for use cases where updates to assets occur within Airflow or can be accomplished through a call to the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html). However, there are scenarios where you need a DAG to run based on events in external systems. Two common patterns exist for event-driven scheduling:

1. **Data delivery to an external system**: Data is delivered to an external system, such as manually by a domain expert, and a data-ready event is sent to a message queue. The DAG in Airflow is scheduled based on this message event and runs an extract-transform-load (ETL) pipeline that processes the data in the external system.

2. **IoT sensor events**: An Internet of Things (IoT) device sends a sensor event to a message queue. A DAG in Airflow is scheduled based on this message event and consumes the message to evaluate the sensor value. If the evaluation determines that an alert is warranted, an alert event is published to another message queue.

<Frame>
  <img alt="Two patterns using event-driven scheduling." />
</Frame>

One common use case for event-driven scheduling is inference execution, where the DAG that is triggered involves a call to a machine learning model. Airflow can be used to orchestrate inference execution pipelines of all types, including in Generative AI applications.
A key change enabling inference execution in Airflow 3.0 is that a DAG can be triggered with `None` provided as the `logical_date`, meaning simultaneous triggering of multiple DAG runs is possible.

<Info>
  Currently, the `MessageQueueTrigger` works with Amazon SQS and Apache Kafka out-of-the-box with support for other message queues planned for future releases. You can create your own triggers to use with AssetWatchers by inheriting from the `BaseEventTrigger` class. See the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/event-scheduling.html) for more information on supported triggers for event-driven scheduling.
</Info>

## Example: Amazon SQS

This example shows how to configure a DAG to run as soon as a message is posted to an Amazon SQS queue.

1. Create an Amazon SQS queue. See [Amazon Simple Queue Service Documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_CreateQueue.html) for instructions.

2. Add the [Airflow Common Messaging provider](https://airflow.apache.org/docs/apache-airflow-providers-common-messaging/stable/triggers.html) and [Airflow Amazon provider](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/index.html) to your Airflow instance. When using the Astro CLI, you can add the providers to your `requirements.txt` file:

   ```text wrap theme={null}
   apache-airflow-providers-amazon>=9.7.0
   apache-airflow-providers-common-messaging>=1.0.2
   aiobotocore
   ```

3. Set the connection to your Amazon SQS queue in your Airflow instance. Note that the connection needs to include the `region_name` in the `extra` field. Replace `<your access key>`, `<your secret key>`, and `<your region>` with your AWS credentials and region. For other authentication options see the [Airflow Amazon provider documentation](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html).

   ```text wrap theme={null}
   AIRFLOW_CONN_AWS_DEFAULT='{
       "conn_type":"aws",
       "login":"<your access key>",
       "password":"<you secret key>",
       "extra": {
           "region_name":"<your region>",
       }
   }'
   ```

   <Note>
     Your AWS user requires at least `sqs:ReceiveMessage` and `sqs:DeleteMessage` permissions.
   </Note>

4. Create a new file in the `dags` folder of your Airflow project and add the following code. Replace the `SQS_QUEUE` with the URL to your message queue.

   ```python expandable wrap theme={null}
   from airflow.providers.common.messaging.triggers.msg_queue import MessageQueueTrigger
   from airflow.sdk import Asset, AssetWatcher, dag, task
   import os

   # Define the SQS queue URL
   SQS_QUEUE = "https://sqs.<region>.amazonaws.com/<acct id>/<queue name>"

   # Define a trigger that listens to an external message queue (AWS SQS in this case)
   trigger = MessageQueueTrigger(
       aws_conn_id="aws_default",
       queue=SQS_QUEUE,
       waiter_delay=30,  # delay in seconds between polls
   )

   # Define an asset that watches for messages on the queue
   sqs_queue_asset = Asset(
       "sqs_queue_asset", watchers=[AssetWatcher(name="sqs_watcher", trigger=trigger)]
   )


   # Schedule the DAG to run when the asset is triggered
   @dag(schedule=[sqs_queue_asset])
   def event_driven_dag():
       @task
       def process_message(**context):
           # Extract the triggering asset events from the context
           triggering_asset_events = context["triggering_asset_events"]
           for event in triggering_asset_events[sqs_queue_asset]:
               # Get the message from the TriggerEvent payload
               print(
                   f'Processing message: {event.extra["payload"]["message_batch"][0]["Body"]}'
               )

       process_message()


   event_driven_dag()
   ```

   This DAG is scheduled to run as soon as the `sqs_queue_asset` asset is updated. This asset uses one `AssetWatcher` with the name `sqs_watcher` that watches one `MessageQueueTrigger`. This trigger is polling for new messages in the provided SQS queue.

   The `process_message` task gets the triggering asset events from the [Airflow context](/docs/learn/airflow-context) and prints the message body from the triggering message. The `process_message` task is a placeholder for your own task that processes the message.

5. Create a new message in the SQS queue. It will trigger the DAG to run and the `process_message` task will print the message body.

## Example: Apache Kafka

To use Apache Kafka as the message queue for event-driven scheduling, follow these steps:

1. Create an Apache Kafka topic. See the [Apache Kafka documentation](https://kafka.apache.org/documentation/#quickstart) for instructions.

2. Add the [Airflow Common Messaging provider](https://airflow.apache.org/docs/apache-airflow-providers-common-messaging/stable/triggers.html) and [Airflow Apache Kafka provider](https://airflow.apache.org/docs/apache-airflow-providers-apache-kafka/stable/index.html) to your Airflow instance. When using the Astro CLI, you can add the providers to your `requirements.txt` file:

   ```text wrap theme={null}
   apache-airflow-providers-apache-kafka>=1.9.0
   apache-airflow-providers-common-messaging>=1.0.2
   ```

3. Set the connection to your Apache Kafka topic in your Airflow instance. Below is an example connection JSON, depending on your Kafka setup you may need to provide additional values in the `extra` dictionary.

   ```text wrap theme={null}
   AIRFLOW_CONN_KAFKA_DEFAULT='{
       "conn_type": "general",
       "extra": {
           "bootstrap.servers": "<your host>:<your port>",
           "group.id": "<your group id>",
           "security.protocol": "<your security protocol>",
           "enable.auto.commit": false,
           "auto.offset.reset": "beginning"
       }
   }'
   ```

4. Create a new file in the `include` folder of your Airflow project called `kafka_trigger.py` and add the following code. This function will be applied to the Kafka message as soon as it is received and return the value of the message to be processed by the Dag.

   ```python wrap theme={null}
   import json

   # Define a function to apply when a message is received
   # This function will be called with the message as an argument
   def apply_function(*args, **kwargs):
       message = args[-1]
       val = json.loads(message.value())
       print(f"Value in message is {val}")
       return val
   ```

5. Create a new file in the `dags` folder of your Airflow project and add the following code. Replace the placeholders in the `KAFKA_QUEUE` with your information.

   ```python expandable wrap theme={null}
   import json

   from airflow.providers.common.messaging.triggers.msg_queue import MessageQueueTrigger
   from airflow.providers.standard.operators.empty import EmptyOperator
   from airflow.sdk import dag, Asset, AssetWatcher, task

   # Define the Kafka queue URL
   # Replace <your_kafka_host>, <port>, and <your_topic> with your Kafka
   KAFKA_QUEUE = "kafka://<your_kafka_host>:<port>/<your_topic>"

   # Define a trigger that listens to an external message queue (Kafka in this case)
   # Note that the argument given to `apply_function` is the path to the function in the `include` folder.
   trigger = MessageQueueTrigger(
       queue=KAFKA_QUEUE,
       apply_function="include.kafka_trigger.apply_function",
   )

   # Define an asset that watches for messages on the Kafka topic
   kafka_topic_asset = Asset(
       "kafka_topic_asset", watchers=[AssetWatcher(name="kafka_watcher", trigger=trigger)]
   )


   @dag(schedule=[kafka_topic_asset])
   def event_driven_dag():
       @task
       def process_message(**context):
           # Extract the triggering asset events from the context
           triggering_asset_events = context["triggering_asset_events"]
           for event in triggering_asset_events[kafka_topic_asset]:
               # Get the message from the TriggerEvent
               print(f"Processing message: {event}")

       process_message()


   event_driven_dag()
   ```

   This DAG is scheduled to run as soon as the `kafka_topic_asset` asset is updated. This asset uses one `AssetWatcher` with the name `kafka_watcher` that watches one `MessageQueueTrigger`. This trigger is polling for new messages in the provided Kafka topic.

   The `process_message` task gets the triggering asset events from the [Airflow context](/docs/learn/airflow-context) and prints the event information from the trigger event that includes the message. The `process_message` task is a placeholder for your own task that processes the message.

6. Create a new message in the Kafka topic. It will trigger the DAG to run and the `process_message` task will print the event information.


# Apache Airflow® Executors
Source: https://astronomer.io/docs/learn/airflow-executors-explained

An introduction to Apache Airflow® Executors

Executors are a configuration property of the [Airflow scheduler component](/docs/learn/airflow-components). The executor you choose for a task determines *where* and *how* a task is run. You can choose from several pre-configured executors that are designed for different use cases, or define your own [custom executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html).

In this guide you'll learn about the executors available in Airflow 3 and how to choose the right one for your use case.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Airflow components. See [Airflow components](/docs/learn/airflow-components).

## How to choose your executor

For production Airflow deployments, there are three main recommended executors you can choose from:

<CardGroup>
  <Card title="AstroExecutor" icon="rocket" href="#astroexecutor">
    The `AstroExecutor` is a proprietary executor that is exclusively available for Astro users in Airflow 3 Deployments. It uses agents that are managed by the API server component and can be used both for hosted and remote execution mode Deployments.
  </Card>

  <Card title="KubernetesExecutor" icon="box" href="#kubernetesexecutor">
    The `KubernetesExecutor` is a containerized executor that runs each task instance in an individual Kubernetes Pod. On Astro, it can be used for hosted execution mode Deployments in Airflow 2 and Airflow 3.
  </Card>

  <Card title="CeleryExecutor" icon="server" href="#celeryexecutor">
    The `CeleryExecutor` is a queued executor that sends tasks to a Celery broker to be picked up by Celery workers. On Astro, it can be used for hosted execution mode Deployments in Airflow 2 and Airflow 3.
  </Card>
</CardGroup>

### AstroExecutor

The [AstroExecutor](/docs/astro/astro-executor) is the default for all Airflow 3.x Deployments. It uses agents (workers) that pull their work from the API server component and run them in subprocesses. The API server manages the agent lifecycle and controls task assignment logic, which is more reliable than the `CeleryExecutor`, and starts tasks faster than the `KubernetesExecutor`.

This executor can be used for both [hosted](/docs/astro/execution-mode#hosted-execution) and [remote](/docs/astro/execution-mode#remote-execution) execution mode Deployments on Astro. It is the only executor enabling remote execution on Astro; for remote execution capabilities in other Airflow environments, you can use the [EdgeExecutor](https://airflow.apache.org/docs/apache-airflow-providers-edge3/stable/edge_executor.html).

Choose the `AstroExecutor` for:

* All Airflow 3 Deployments on Astro, unless you have specific requirements to use another executor.
* All remote execution mode Deployments on Astro.

You can use [worker queues](/docs/astro/configure-worker-queues) with the `AstroExecutor` to run tasks on different worker types with varying resource configurations.

### KubernetesExecutor

The `KubernetesExecutor` is a containerized executor, which means every task instance is run in its own Kubernetes Pod. As such, access to a Kubernetes cluster is required to use this executor. The `KubernetesExecutor` allows for full task isolation and fine-grained control over the resources allocated to each task, with the trade-off of a slower task startup time.

Choose the `KubernetesExecutor` for:

* Deployments that require a high degree of task isolation.
* Deployments in which many tasks need to have a specific resource configuration.

There are some additional requirements for using the `KubernetesExecutor`, which are automatically fulfilled in Astro Deployments configured to run with this executor.

* The [Airflow Kubernetes provider](https://airflow.apache.org/registry/providers/cncf-kubernetes/) needs to be installed.
* The Airflow metadata database can't be a SQLite database.

You can customize your Kubernetes Pods by setting a base configuration and overriding it at the individual task level using the `pod_override` parameter. See [Configure tasks to run with the KubernetesExecutor](/docs/astro/kubernetes-executor) for more information on how to configure the `KubernetesExecutor` on Astro and [Kubernetes Executor - Configuration](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/kubernetes_executor.html#configuration) for more information on how to configure the `KubernetesExecutor` when running it in a self-hosted Airflow environment.

### CeleryExecutor

The `CeleryExecutor` allows you to scale your workload horizontally by running tasks on multiple [Celery](https://docs.celeryq.dev/en/latest/getting-started/) workers that pick up their tasks from a queue. It can start tasks quickly, since no additional infrastructure needs to be provisioned after the initial setup, and scale horizontally to run many tasks concurrently. Due to these characteristics, the `CeleryExecutor` is a common default choice for Airflow 2 Deployments on Astro and self-managed Airflow 2 and 3 environments.

Choose the `CeleryExecutor` for:

* All Airflow 2 Deployments on Astro, unless you have specific requirements to use the `KubernetesExecutor`.
* Deployments that run consistent workloads with tasks starting frequently.
* Deployments that have high needs for horizontal scaling to run many tasks concurrently.

There are additional requirements for running the `CeleryExecutor`, which are automatically fulfilled in Astro Deployments configured to run with this executor.

* The [Airflow Celery provider](https://airflow.apache.org/registry/providers/celery/) needs to be installed.
* A Celery backend like Redis, RabbitMQ, or Redis Sentinel needs to be installed and configured. Astro Deployments use Redis.

On Astro, you can use [worker queues](/docs/astro/configure-worker-queues) with the `CeleryExecutor` to run tasks on different worker types with varying resource configurations.

To learn more about different configuration options for the `CeleryExecutor`, see [Configure the CeleryExecutor](/docs/astro/celery-executor) for Astro and [Celery Executor](https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/celery_executor.html) for self-managed Airflow environments.

### LocalExecutor

The `LocalExecutor` runs inside the scheduler process and is the simplest option for task execution in Airflow 3, where the `SequentialExecutor` and `DebugExecutor` have been removed. Since the `LocalExecutor` runs in the same process as the scheduler, it doesn't require any additional infrastructure, but it can have an impact on the scheduler's performance when running many tasks.

Choose the `LocalExecutor` for:

* Local development and testing. The [Astro CLI](/docs/cli/v1.43/overview) uses the LocalExecutor.
* Very light production environments for self-managed Airflow environments

The Airflow configuration variable [`[core].parallelism`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#parallelism) determines the maximum number of tasks that can run concurrently with the `LocalExecutor` per scheduler. The value needs to be at least 1 (default: 32).

### Other executors

There are a couple of other executors that are available in self-managed Airflow environments.

* **[EdgeExecutor](https://airflow.apache.org/docs/apache-airflow-providers-edge3/stable/edge_executor.html)**: This executor allows you to distribute your tasks to workers in different remote locations through HTTP(s) connections. It is available as part of the [Edge3 provider](https://airflow.apache.org/docs/apache-airflow-providers-edge3/stable/index.html) package for Airflow deployments on version 2.10 or later.
* **[AWS ECS Executor](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/executors/ecs-executor.html):** The `AwsEcsExecutor` is a containerized executor that runs each task instance in an individual ECS task. It is available as part of the [Amazon provider](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/index.html) package.
* **[AWS Batch Executor](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/executors/batch-executor.html):** The `AwsBatchExecutor` runs tasks in separate containers scheduled by AWS Batch. It is available as part of the Amazon provider package.
* **[AWS Lambda Executor](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/executors/lambda-executor.html)** ([experimental](https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features)): The `AwsLambdaExecutor` submits tasks to AWS Lambda to run asynchronously. It is available as part of the Amazon provider package.

The hybrid executors `CeleryKubernetesExecutor` and `LocalKubernetesExecutor` have been removed in Airflow 3. In a self-managed Airflow environment, you can configure multiple executors instead, see [Configure an executor in self-hosted Airflow](#configure-an-executor-in-self-hosted-airflow).

## Configure an executor on Astro

On Astro you can set your executor during the Deployment creation process. The default executor is the `AstroExecutor`.

If you are creating your Deployment in the Astro UI, you can further configure the `AstroExecutor` (for example to configure worker queues) or choose a different executor.

1. Click the **Switch to custom configuration** button.

<Frame>
  <img alt="Switch to custom configuration button" />
</Frame>

2. Select a different executor.

<Frame>
  <img alt="Select executor on Astro" />
</Frame>

On Astro you can also create Deployments programmatically and set your executor as a configuration option:

* [Astro CLI](/docs/cli/v1.43/astro-deployment-create): set the executor using the `--executor` flag. The options are `AstroExecutor`, `CeleryExecutor`, or `KubernetesExecutor`.
* [Astro API](/docs/astro/api/v-1/deployment/create-a-deployment): set the executor using the `executor` parameter in the `POST` request. The options are `ASTRO`, `CELERY`, or `KUBERNETES`.
* [Astro Terraform Provider](https://registry.terraform.io/providers/astronomer/astro/latest/docs/resources/deployment): set the executor using the `executor` parameter in the `astro_deployment` resource. The options are `ASTRO`, `CELERY`, or `KUBERNETES`.

## Configure an executor in self-hosted Airflow

Open-source Airflow allows you to configure [multiple executors](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html#using-multiple-executors-concurrently) and assign each task to a different executor using the `executor` parameter.

In your Airflow configuration file, set the `[core].executor` variable to the executor(s) you want to use, separated by a comma.

```text wrap theme={null}
[core]
executor = KubernetesExecutor,CeleryExecutor,MyExecutor:my.custom.module.ExecutorClass
```

<Note>
  You can't use two instances of the same executor in the same Airflow environment.
</Note>

In your task code, assign the executor to the task using the `executor` parameter. The first executor in the list is the default executor for all tasks where no executor is specified.

```python wrap theme={null}
# from airflow.sdk import task

@task(executor="CeleryExecutor")
def my_task():
    pass

# from airflow.providers.standard.operators.bash import BashOperator

BashOperator(
    task_id="my_task",
    executor="MyExecutor",
    bash_command="echo 'Hello World!'",
)
```


# Use Fivetran with Apache Airflow
Source: https://astronomer.io/docs/learn/airflow-fivetran

Learn how to orchestrate Fivetran syncs using Airflow

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Fivetran](https://www.fivetran.com/) is a popular ELT platform that automates ingesting data from a variety of sources into a database, offering pre-built integrations for many common data tools.
Using Airflow with Fivetran allows you to schedule your Fivetran syncs based on events in your larger data ecosystem, as well as trigger downstream actions after a sync has finished.

In this tutorial, you'll learn how to install and use the Airflow Fivetran provider to submit and monitor Fivetran syncs.

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Webinar: [Hands-on Workshop: automate your data ingestion with Fivetran and Astronomer](https://www.astronomer.io/events/webinars/workshop-automate-data-ingestion-fivetran-astronomer/).
</Tip>

## Time to complete

This tutorial takes approximately 1 hour to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of Fivetran. See Fivetran's [Getting started](https://fivetran.com/docs/getting-started) documentation.
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).

## Prerequisites

* A Fivetran account. Fivetran offers a [14-day free trial](https://fivetran.com/signup) for new customers.
* The [Astro CLI](/docs/cli/v1.43/overview).
* A [GitHub](https://github.com/) account with the permissions for the following GitHub scopes: `repo`, `read:org`, `admin:org_hook`, `admin:repo_hook`.

## Step 1: Configure your Astro project

An Astro project contains all of the files you need to run Airflow locally.

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-fivetran-project && cd astro-fivetran-project
   $ astro dev init
   ```

2. Add the [Fivetran provider package](https://github.com/astronomer/airflow-provider-fivetran-async) to your `requirements.txt` file:

   ```text wrap theme={null}
   airflow-provider-fivetran-async
   ```

3. Run the following command to start your project in a local environment:

   ```sh wrap theme={null}
   astro dev start
   ```

## Step 2: Create a new private GitHub repository

For this tutorial you will use metadata from a GitHub repository as your data source. We recommend using a new repository to prevent unintentionally ingesting large amounts of data.

1. [Create a new private GitHub repository](https://docs.github.com/en/get-started/quickstart/create-a-repo) called `airflow-fivetran-tutorial`.
2. Commit at least one change to the repository. The content of your commit doesn't matter for this tutorial.

## Step 3: Connect Fivetran to a destination

Fivetran needs at least one destination to be configured in order to create syncs. A destination is a relational database where your ingested data will be loaded into.

1. Log in to your [Fivetran account](https://www.fivetran.com/).

2. Click **Destinations** in the left menu, then click **ADD DESTINATION** in the upper right corner.

3. Choose either **Connect your destination** or **I don't have one**. For this tutorial you can either connect to an existing data warehouse by following the [relevant Fivetran documentation](https://fivetran.com/docs/databases), or use a [Fivetran-managed BigQuery service](https://fivetran.com/docs/destinations/bigquery/managed-bigquery). In this tutorial we will use Fivetran's managed BigQuery service by selecting **I don't have one** and then clicking **CONTINUE SETUP**.

4. Configure your destination connection. You can choose any configuration. Astronomer recommends using UTC as your timezone in all data tools as a best practice. Click **SAVE & TEST** to save your destination.

5. Click **Continue** to get back to the list of your destinations.

## Step 4: Configure a Fivetran connector

Fivetran needs at least one [connector](https://fivetran.com/docs/getting-started/fivetran-dashboard/connectors) to be configured in order to create syncs. A Fivetran connector reaches out to a specific data source, receives data from it and writes it to your destination. View the Fivetran website for an [up-to-date list of connectors](https://www.fivetran.com/connectors).

1. In Fivetran, click **Connectors** in the left menu and then click **ADD CONNECTOR** in the upper right corner.

2. Select the GitHub Connector and click **CONTINUE SETUP**.

3. Configure the GitHub Connector:

   * **Destination schema**: `in_github`
   * **Authentication mode**: Either [OAuth or a Personal Access Token](https://fivetran.com/docs/applications/github/setup-guide), then click **AUTHORIZE**. In this tutorial, we use OAuth authentication.

4. Authenticate Fivetran to your GitHub account.

5. In **Sync Mode** select **Sync Specific Repositories**, then in **Repositories** select the `airflow-fivetran-tutorial` repository.

6. Click **SAVE & TEST**. After your connection has been tested click **CONTINUE**.

7. Click **Start Sync** to start your initial sync. This initial synchronization will load all historic metadata from your GitHub repository and has to be completed in Fivetran for the sync to become active. Once the Fivetran sync is active, you can set the sync frequency under the **Setup** tab or run the sync using the Fivetran API.

## Step 5: Generate a Fivetran API key

You have now created a Fivetran [sync](https://fivetran.com/docs/getting-started/syncoverview) that will extract new metadata from a GitHub repository and load it into a relational database on a time-based schedule. But to run the sync every time a specific event happens in your data ecosystem, you need to use Airflow for orchestration.

To connect Airflow to Fivetran, create a Fivetran API key.

1. In Fivetran, open your user account and click **API Key**.

2. Click **Generate API Key**. Copy the API key information to a secure place for later.

## Step 6: Create an Airflow connection to Fivetran

1. In a web browser, go to `localhost:8080` to access the Airflow UI.

2. Click **Admin** -> **Connections** -> **+** to create a new connection.

3. Name your connection `fivetran_conn` and select the **Fivetran** connection type. Provide your Fivetran API key and Fivetran API secret. If the Fivetran connection type isn't available, try restarting your Airflow instance with `astro dev restart` to ensure the contents of `requirements.txt` have been installed.

4. Click **Save**.

## Step 7: Create your Airflow DAG

For this tutorial you will create a DAG that triggers your Fivetran sync to ingest the GitHub repository metadata to your destination.

1. Open your `astro-fivetran-project` in a code-editor.

2. In your `dags` folder add a new Python file called `my_fivetran_dag.py`.

3. Copy and paste the following DAG code into the file:

   ```python wrap theme={null}
   from airflow.decorators import dag, task
   from pendulum import datetime
   from fivetran_provider_async.operators import FivetranOperatorAsync

   FIVETRAN_CONNECTOR_ID = "<your Fivetran connector ID>"
   GITHUB_REPOSITORY = "<your GitHub handle>/airflow-fivetran-tutorial"
   TAG_NAME = "sync-metadata"


   @dag(start_date=datetime(2023, 1, 1), schedule="@daily", catchup=False)
   def my_fivetran_dag():
       @task
       def upstream():
           return "Hello"

       run_fivetran_sync = FivetranOperatorAsync(
           task_id="run_fivetran_sync",
           fivetran_conn_id="fivetran_conn",
           connector_id=FIVETRAN_CONNECTOR_ID,
       )

       @task
       def downstream():
           return "Goodbye"

       upstream() >> run_fivetran_sync >> downstream()


   my_fivetran_dag()
   ```

   This DAG contains three tasks:

   * The `upstream` task runs before the Fivetran sync job is run. This task could contain a sensor or deferrable operator waiting for an action to be completed in another data tool.
   * The `run_fivetran_sync` task uses the FivetranOperatorAsync to trigger the Fivetran connector specified as `FIVETRAN_CONNECTOR_ID` as soon as the `upstream` task has completed successfully.
   * The `downstream` task runs after the Fivetran sync job has finished. Commonly, tasks containing data transformations on the data loaded into your Fivetran destination will be set as downstream tasks.

4. Update the `FIVETRAN_CONNECTOR_ID` variable with the ID of your connector. You can find the ID of your connector in the Fivetran UI under the **Setup** tab:

   <Frame>
     <img alt="Fivetran connector ID" />
   </Frame>

5. Add your GitHub username to the `GITHUB_REPOSITORY` variable.

6. Save your DAG file with the changed variable names.

The FivetranOperatorAsync is one of many [deferrable operators](/docs/learn/deferrable-operators). Instead of taking up a worker slot, these operators will hand their task to the Airflow Triggerer component while waiting for a condition to be fulfilled. For longer running tasks, this can result in cost savings and greater scalability as more worker slots are available.

## Step 8: Run your DAG

1. In the Airflow UI, unpause your DAG to start its last scheduled DAG run with a logical date of yesterday.

2. Click on the `run_fivetran_sync` task in the Airflow **Grid View**. This task will be in a deferred state (violet square) until the Fivetran sync is completed.

   <Frame>
     <img alt="Fivetran Deferred" />
   </Frame>

3. After the DAG run has finished successfully, verify in your Fivetran UI that the sync shows an additional `Manual Update Triggered` entry in its `User Actions`.

   <Frame>
     <img alt="Fivetran additional sync" />
   </Frame>

## Step 9: (optional) Visualize your commits with Tableau

As a bonus, you can implement data observability into your data pipeline. If you are using a custom Fivetran destination, you can simply view your data directly and connect your favorite BI tool to it. If you followed this tutorial using Fivetran's managed BigQuery service, you can add a BI tool like Tableau to view your data.

1. In the Fivetran UI go to **Destinations** and select your warehouse, click the **BI tools** tab, and click **+ BI tool**.

2. Authenticate to Tableau. See [Fivetran documentation](https://fivetran.com/docs/destinations/bigquery/managed-bigquery/tableau-setup-guide). Note that you have to authorize the same Google user to Tableau as you authorized to the Fivetran managed BigQuery service.

3. [Create a new Tableau worksheet](https://help.tableau.com/current/pro/desktop/en-us/environment_workspace.htm) either in Tableau desktop or in Tableau Cloud.

4. Connect your Tableau worksheet to the BigQuery project created by the Fivetran managed service. Make sure you are using the same Google user you authorized in Fivetran. See [Tableau documentation](https://help.tableau.com/current/pro/desktop/en-us/examples_googlebigquery.htm).

5. You can now visualize all elements of the GitHub repository metadata that has been loaded into BigQuery. Using the **commit** table, drag and drop the **commit(Count)** item from the Tables section to the `Rows` list and the **Committer Date** to the `Columns` list. Change the grain of **Committer Date** displayed to **Minute** by clicking on the right side of the element in the **Columns** list. This creates a line graph showing how many commits were made each minute in the `airflow-fivetran-tutorial` repository. The following screenshot shows that two commits were made in separate minutes:

   <Frame>
     <img alt="Tableau commits 1" />
   </Frame>

6. Add a few commits to your `airflow-fivetran-tutorial` repository.

7. Rerun `my_fivetran_dag` manually by clicking on the play button in the Airflow UI.

8. View your updated dashboard with commits per minute for the aggregated hour.

   <Frame>
     <img alt="Tableau commits 1" />
   </Frame>

## Conclusion

Using Airflow with Fivetran allows you to embed your Fivetran syncs in your larger data ecosystem, making the scheduling of Fivetran syncs event- and data-driven. The [`airflow-provider-fivetran-async`](https://airflow.apache.org/registry/providers/fivetran) offers asynchronous capabilities to make your architecture more scalable and efficient.


# Orchestrate Great Expectations with Airflow
Source: https://astronomer.io/docs/learn/airflow-great-expectations

Orchestrate Great Expectations data quality checks with your Airflow Dags.

[Great Expectations](https://greatexpectations.io) (GX) is an open source Python-based data validation framework. You can test your data by expressing what you "expect" from it as simple declarative statements in JSON or YAML, then run validations using those [Expectation Suites](https://docs.greatexpectations.io/docs/core/define_expectations/organize_expectation_suites) against data [SQL data](https://docs.greatexpectations.io/docs/core/connect_to_data/sql_data), [Filesystem Data](https://docs.greatexpectations.io/docs/core/connect_to_data/filesystem_data) or a [pandas DataFrame](https://docs.greatexpectations.io/docs/core/connect_to_data/dataframes). The [airflow-provider-great-expectations](https://great-expectations.github.io/airflow-provider-great-expectations/latest/getting-started/) package provides operators for running Great Expectations validations directly in your Dags.

<Warning>
  Version 1.0.0 of the provider introduces three specialized operators that replace the legacy `GreatExpectationsOperator`:
</Warning>

| Operator                       | Use case                                                             |
| ------------------------------ | -------------------------------------------------------------------- |
| `GXValidateDataFrameOperator`  | Validate in-memory Spark or Pandas DataFrames                        |
| `GXValidateBatchOperator`      | Validate data not in memory using a BatchDefinition                  |
| `GXValidateCheckpointOperator` | Most feature-rich: supports triggering actions on validation results |

**Requirements**: Python 3.10+, Great Expectations 1.7.0+, Apache Airflow 2.1+.

Install with:

```text wrap theme={null}
pip install airflow-provider-great-expectations
```

Each operator has its own import path:

```python wrap theme={null}
from great_expectations_provider.operators.validate_dataframe import GXValidateDataFrameOperator
from great_expectations_provider.operators.validate_batch import GXValidateBatchOperator
from great_expectations_provider.operators.validate_checkpoint import GXValidateCheckpointOperator
```

#### Choose the right operator

When deciding which operator fits your use case, consider:

1. **Where is your data?** In memory as a DataFrame, or in an external data source?
2. **Do you need to trigger actions?** Such as sending notifications or updating external systems based on validation results.
3. **What Data Context do you need?** Ephemeral for stateless validations, or persistent to track results over time.

| Scenario                                                      | Recommended operator           |
| ------------------------------------------------------------- | ------------------------------ |
| Data already in memory as Pandas or Spark DataFrame           | `GXValidateDataFrameOperator`  |
| Data in a database, warehouse, or file system                 | `GXValidateBatchOperator`      |
| Need to trigger Slack notifications, emails, or other actions | `GXValidateCheckpointOperator` |
| Want full GX Core features with ValidationDefinitions         | `GXValidateCheckpointOperator` |

#### `GXValidateDataFrameOperator`

Use this operator when your data is already in memory as a Pandas or Spark DataFrame. This is the simplest option, you only need a DataFrame and your expectations.

<Accordion title="GXValidateDataFrameOperator example">
  ```python expandable wrap theme={null}
  from great_expectations_provider.operators.validate_dataframe import GXValidateDataFrameOperator

  def configure_dataframe():
      """Load data into a DataFrame using an Airflow hook."""
      from airflow.providers.common.sql.hooks.sql import DbApiHook

      hook = DbApiHook.get_hook("my_db_conn")
      return hook.get_pandas_df("SELECT * FROM daily_planet_report")

  def configure_expectations(dataframe):
      """Define expectations for the DataFrame."""
      import great_expectations as gx

      suite = gx.ExpectationSuite(name="daily_report_suite")
      suite.add_expectation(
          gx.expectations.ExpectColumnValuesToNotBeNull(column="planet_name")
      )
      suite.add_expectation(
          gx.expectations.ExpectColumnValuesToNotBeNull(column="total_passengers")
      )
      suite.add_expectation(
          gx.expectations.ExpectColumnValuesToBeBetween(
              column="total_net_fare_usd", min_value=0
          )
      )
      suite.add_expectation(
          gx.expectations.ExpectColumnValuesToBeBetween(
              column="total_discounts_usd", min_value=0
          )
      )
      return suite

  _validate_dataframe = GXValidateDataFrameOperator(
      task_id="validate_with_gx_dataframe",
      configure_dataframe=configure_dataframe,
      configure_expectations=configure_expectations,
      result_format="SUMMARY",
      context_type="ephemeral",
  )
  ```
</Accordion>

<Info>
  **configure\_expectations signature**

  The `configure_expectations` callable receives the DataFrame as its first argument and must return an `Expectation` or `ExpectationSuite`.
</Info>

#### `GXValidateBatchOperator`

Use this operator when your data isn't in memory. You configure GX to connect directly to your data source by defining a `BatchDefinition`. This approach works with databases, warehouses, and file systems.

<Tip>
  The provider includes helper functions to build connection strings from Airflow connections, so you don't need to duplicate credentials.
</Tip>

<Accordion title="GXValidateBatchOperator example (SQL)">
  ```python expandable wrap theme={null}
  from great_expectations_provider.operators.validate_batch import GXValidateBatchOperator
  from great_expectations_provider.common.external_connections import build_snowflake_connection_string

  SNOWFLAKE_CONN_ID = "snowflake_default"


  def configure_batch_definition(context):
      """Configure a batch definition for a SQL table."""
      connection_string = build_snowflake_connection_string(
          conn_id=SNOWFLAKE_CONN_ID,
          schema="my_schema"
      )

      data_source = context.data_sources.add_sql(
          name="snowflake_ds",
          connection_string=connection_string,
      )

      table_asset = data_source.add_table_asset(
          name="daily_report_table",
          table_name="daily_planet_report",
      )

      return table_asset.add_batch_definition_whole_table(name="full_table")


  def configure_expectations(context):
      """Define expectations and add them to the context."""
      import great_expectations.expectations as gxe
      from great_expectations import ExpectationSuite

      return context.suites.add_or_update(
          ExpectationSuite(
              name="daily_report_batch_suite",
              expectations=[
                  gxe.ExpectColumnValuesToNotBeNull(column="planet_name"),
                  gxe.ExpectColumnValuesToNotBeNull(column="total_passengers"),
                  gxe.ExpectColumnValuesToBeBetween(
                      column="total_net_fare_usd", min_value=0
                  ),
                  gxe.ExpectColumnValuesToBeBetween(
                      column="total_discounts_usd", min_value=0
                  ),
              ],
          )
      )


  _validate_batch = GXValidateBatchOperator(
      task_id="validate_with_gx_batch",
      configure_batch_definition=configure_batch_definition,
      configure_expectations=configure_expectations,
      result_format="SUMMARY",
      context_type="ephemeral",
  )
  ```
</Accordion>

<Accordion title="GXValidateBatchOperator example (file-based)">
  For file-based data (CSV, Parquet), use `add_pandas_filesystem` instead:

  ```python wrap theme={null}
  def configure_batch_definition(context):
      """Configure a batch definition to read from a CSV file."""
      from pathlib import Path

      data_source = context.data_sources.add_pandas_filesystem(
          name="my_datasource",
          base_directory=Path("/path/to/data"),
      )
      csv_asset = data_source.add_csv_asset(name="daily_report_csv")
      return csv_asset.add_batch_definition_path(
          name="daily_report_batch",
          path="daily_report.csv",
      )
  ```
</Accordion>

<Info>
  **configure\_expectations signature**

  For `GXValidateBatchOperator`, the `configure_expectations` callable receives the GX `context` as its first argument (not the DataFrame). Use `context.suites.add_or_update()` to register your suite.
</Info>

#### `GXValidateCheckpointOperator`

Use this operator when you need the full power of GX Core, including the ability to trigger actions based on validation results. This requires the most configuration. You define a `Checkpoint` with a `BatchDefinition`, `ExpectationSuite`, and `ValidationDefinition`.

<Accordion title="GXValidateCheckpointOperator example">
  ```python expandable wrap theme={null}
  from great_expectations_provider.operators.validate_checkpoint import GXValidateCheckpointOperator
  from great_expectations_provider.common.external_connections import build_snowflake_connection_string

  SNOWFLAKE_CONN_ID = "snowflake_default"

  def configure_checkpoint(context):
      """Configure a full GX checkpoint with validation and actions."""
      import great_expectations.expectations as gxe
      from great_expectations import Checkpoint, ExpectationSuite, ValidationDefinition

      connection_string = build_snowflake_connection_string(
          conn_id=SNOWFLAKE_CONN_ID,
          schema="my_schema"
      )

      batch_definition = (
          context.data_sources.add_sql(
              name="snowflake_ds",
              connection_string=connection_string,
          )
          .add_table_asset(
              name="daily_report_table",
              table_name="daily_planet_report",
          )
          .add_batch_definition_whole_table(name="full_table")
      )

      expectation_suite = context.suites.add(
          ExpectationSuite(
              name="daily_report_checkpoint_suite",
              expectations=[
                  gxe.ExpectColumnValuesToNotBeNull(column="planet_name"),
                  gxe.ExpectColumnValuesToNotBeNull(column="total_passengers"),
                  gxe.ExpectColumnValuesToBeBetween(
                      column="total_net_fare_usd", min_value=0
                  ),
                  gxe.ExpectColumnValuesToBeBetween(
                      column="total_discounts_usd", min_value=0
                  ),
              ],
          )
      )

      validation_definition = context.validation_definitions.add(
          ValidationDefinition(
              name="daily_report_validation",
              data=batch_definition,
              suite=expectation_suite,
          )
      )

      return context.checkpoints.add(
          Checkpoint(
              name="daily_report_checkpoint",
              validation_definitions=[validation_definition],
              actions=[],  # Add SlackNotificationAction, EmailAction, etc.
          )
      )

  _validate_checkpoint = GXValidateCheckpointOperator(
      task_id="validate_with_gx_checkpoint",
      configure_checkpoint=configure_checkpoint,
      context_type="ephemeral",
  )
  ```
</Accordion>


# Human-in-the-loop workflows with Airflow
Source: https://astronomer.io/docs/learn/airflow-human-in-the-loop

Learn how include humans in your Dags with the human-in-the-loop feature.

Human-in-the-loop (HITL) workflows are processes that require human intervention, for example, to approve or reject an AI generated output, or choose a [branch](/docs/learn/airflow-branch-operator) in a Dag depending on the result of an upstream task. The [Airflow standard provider](https://airflow.apache.org/docs/apache-airflow-providers-standard/stable/index.html) contains a set of operators to create tasks that will wait for human input, either in the Airflow UI or through the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html). Note that you need to be on Airflow 3.1+ to use the human-in-the-loop operators.

This guide covers the available HITL operators, as well as how to interact with them in the UI and through the API.

<Frame>
  <img alt="Gif showing a human-in-the-loop workflow in the Airflow UI." />
</Frame>

## Assumed knowledge

To get the most out of this guide, you should have an existing knowledge of:

* Airflow basics. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Deferrable operators. See [Deferrable operators](/docs/learn/deferrable-operators).

## When to use human-in-the-loop workflows

Human-in-the-loop workflows are useful whenever you need the input from a human (or another entity outside of Airflow) within a Dag. For example:

* Your Dag uses AI to create responses to a support ticket, and you want to ask a human to review the response and approve it or request changes.
* Your Dag generates a compliance report and you need a human to verify and acknowledge the results.
* You'd like to use AI to route product feature requests to the appropriate team and need a human to decide where to send edge cases.
* You have a Dag that requires input from a domain expert, for example to input feedback gathered in user research interviews.

Human-in-the-loop workflows are very common, especially with increased usage of AI to generate assets that are in need of human review. The Airflow HITL features allows you to build these workflows and have non-technical team members provide their input either in the Airflow UI or through an implementation built around the relevant [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) endpoints.

## Required Actions

Each task instance created by a human-in-the-loop operator creates a **Required Action** object. You can view a list of all required actions (pending and resolved) for your whole Airflow instance under **Browse > Required Actions** in the Airflow UI.

<Frame>
  <img alt="Screenshot of the Airflow UI showing the list of Required Actions under Browse -> Required Actions." />
</Frame>

To respond to a required action you can either:

* Navigate to the task instance page's **Required Actions** tab and respond directly in the UI.
* Make a call to the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html)'s update HITL detail endpoint.

In Airflow 3.3+ you can reply to required actions directly from the **Required Actions** tab after clicking on the sidebar icon.

<Frame>
  <img alt="The Airflow UI Required Actions list with a slide-out panel showing a required action's details and response form." />
</Frame>

## Human-in-the-loop operators

There are 4 human-in-the-loop operators available in the Airflow standard provider:

* [`HITLOperator`](https://airflow.apache.org/registry/providers/standard#standard-hitl-HITLOperator): The base class for all human-in-the-loop operators.
* [`ApprovalOperator`](https://airflow.apache.org/registry/providers/standard#standard-hitl-ApprovalOperator): A specialised form of the `HITLOperator` where the two options are `Approve` and `Reject`.
* [`HITLBranchOperator`](https://airflow.apache.org/registry/providers/standard#standard-hitl-HITLBranchOperator): A specialised form of the `HITLOperator` where the user input provides a [branching](/docs/learn/airflow-branch-operator) decision, choosing the (set of) task(s) to run next.
* [`HITLEntryOperator`](https://airflow.apache.org/registry/providers/standard#standard-hitl-HITLEntryOperator): A specialised form of the `HITLOperator` where the user provides input to a form.

While waiting for human input, human-in-the-loop operators don't take up a worker slot. In Airflow 3.1 and 3.2, human-in-the-loop operators are implemented as [deferrable operators](/docs/learn/deferrable-operators), meaning that they run an asynchronous trigger process in the [Triggerer](/docs/learn/airflow-components) component while waiting for the human input. In Airflow 3.3+, human-in-the-loop operators enter the `awaiting_input` task state while waiting for a required action to receive a response. The `awaiting_input` state doesn't require a running triggerer or worker.

When choosing which human-in-the-loop operator to use, consider the following:

* If the human-in-the-loop action centers around a branching decision (choosing the next task(s) to run), use the `HITLBranchOperator`.
* If you are looking to make a binary decision (approval or rejection of information displayed at runtime), use the `ApprovalOperator`. If users need to give additional input add a form field using `params`.
* If you are mostly interested in letting users provide input to a form, use the `HITLEntryOperator`.
* For all other use cases, you can use the `HITLOperator` directly. It allows you to display custom options for your user to choose from, as well as additional input fields with `params`.

In a lot of cases you'll likely be using a combination of these operators to build your human-in-the-loop workflow, for example using the `HITLBranchOperator` upstream to choose whether to accept an AI generated ticket response or escalate the ticket to a human. Then, downstream, the human's answer can be provided in a `HITLEntryOperator`. Both the user decision (`chosen_options`) and any input to parameters (`params_input`) are available for downstream tasks to use by pulling the information from [XComs](/docs/learn/airflow-passing-data-between-tasks).

### `HITLOperator`

The `HITLOperator` is the base class for all human-in-the-loop operators. It is the most versatile operator in this operator family. With it you can display information, let the user choose one or more from a list of options and accept additional input using a form based on [Airflow params](/docs/learn/airflow-params).

Two parameters are mandatory when instantiating the `HITLOperator`:

* `subject` (required): The subject of the templated action which is displayed as the title in the **Required Actions** tab. This field is templatable, which means you can use [Jinja templates](/docs/learn/templating) to render information at runtime, including information computed by an upstream task.
* `options` (required): A list of strings that are rendered as response options at the bottom of the required action form. Note that the list can't be empty. The chosen options can be retrieved in downstream tasks by pulling the information from [XComs](/docs/learn/airflow-passing-data-between-tasks), which is stored as a list under the `chosen_options` key.

There are also several optional parameters that you can use to further configure the behavior of the `HITLOperator`:

* `body`: The main text body. This field is templatable as well and supports markdown formatting.
* `defaults`: Optionally, you can provide a list of one or more options that are selected by default if the task times out before a human responds. All default options need to be in the `options` list.
* `multiple`: If set to `True`, the user can select multiple options. Default is `False`.
* `params`: With this parameter you can create form fields for any user input to the required action using [Airflow params](/docs/learn/airflow-params). Note that not all param functionality is supported for human-in-the-loop operators. These params can be retrieved in downstream tasks by pulling the information from [XComs](/docs/learn/airflow-passing-data-between-tasks), which is stored as a nested dictionary under the `params_input` key.
* `execution_timeout`: This is a [`BaseOperator`](https://registry.astronomer.io/providers/apache-airflow/versions/latest/modules/BaseOperator) parameter that times out the task after a specified duration provided as a [`datetime.timedelta`](https://docs.python.org/3/library/datetime.html#datetime.timedelta) or [pendulum duration](https://pendulum.eustace.io/docs/#duration) object. Default is `None`. After the timeout has been reached the behavior depends on whether you provided a `defaults` list or not:
  * If you provided a `defaults` list, the default(s) is/are chosen as the response and the task succeeds.
  * If you didn't provide a `defaults` list, the task fails.
* `assigned_users`: A list of all users who are allowed to respond to the required action. Users are provided as `HITLUser` objects (`from airflow.sdk.execution_time.hitl import HITLUser`) with an `id` and `name` field.
  * If you are running Airflow on [Astro](https://www.astronomer.io/lp/signup/), the id of each user is their Astro ID in the format `cl1a2b3cd456789ef1gh2ijkl3`. You can find each user's Astro ID under **Organization** -> **Access Management**.
  * If you are using the [SimpleAuthManager](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/auth-manager.html#simpleauthmanager) the id of each user is their username.
  * If you are using the [FabAuthManager](https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/auth-manager/index.html) the id of each user is their email.
* `notifiers`: A list of notifiers of which to execute the `.notify()` method when the task starts running. See [Use notifiers with HITL operators](#use-notifiers-with-hitl-operators) for an example.

The following example shows a simple use of the `HITLOperator` where the output of an upstream task is fetched from [XComs](/docs/learn/airflow-passing-data-between-tasks) and the user is given 3 response options to choose from. One additional input field for the `expense_amount` is rendered in the Airflow UI using an [Airflow param](/docs/learn/airflow-params). After 5 minutes (`execution_timeout`) the task times out and chooses the default option: `ACH Transfer` as the response and `10000` as the expense amount.

The downstream `print_result` task prints out the information pushed to XComs by the `HITLOperator` task: the `chosen_options` and the `params_input` values.

```python expandable wrap theme={null}
from airflow.providers.standard.operators.hitl import HITLOperator
from airflow.sdk import dag, task, chain, Param


@dag
def HITLOperator_syntax_example():

    @task
    def upstream_task():
        return "Review expense report and approve vendor payment method."

    _upstream_task = upstream_task()

    _hitl_task = HITLOperator(
        task_id="hitl_task",
        subject="Expense Approval Required",  # templatable
        body="{{ ti.xcom_pull(task_ids='upstream_task') }}", # templatable
        options=["ACH Transfer", "Wire Transfer", "Corporate Check"],  # cannot be empty!
        defaults=["ACH Transfer"],
        multiple=False,  # default: False
        params={
            "expense_amount": Param(
                10000,
                type="number",
            )
        },
        execution_timeout=timedelta(minutes=5),  # default: None
    )

    @task
    def print_result(hitl_output):
        print(f"Expense amount: ${hitl_output['params_input']['expense_amount']}")
        print(f"Payment method: {hitl_output['chosen_options']}")

    _print_result = print_result(_hitl_task.output)

    chain(_upstream_task, _hitl_task)


HITLOperator_syntax_example()
```

The open required action form is displayed on the task instance page's **Required Actions** tab.

<Frame>
  <img alt="Screenshot of the Airflow UI showing the HITLOperator task instance page with the Required Actions tab open." />
</Frame>

### `ApprovalOperator`

The `ApprovalOperator` is a specialised form of the `HITLOperator` where the only two possible response options are `Approve` and `Reject`. Additionally, you can provide param form fields for any user input to the required action using [Airflow params](/docs/learn/airflow-params).

If the human chooses to `Approve` the task succeeds. If the human chooses to `Reject` the task succeeds but all downstream tasks are skipped.

```python expandable wrap theme={null}
from airflow.providers.standard.operators.hitl import ApprovalOperator
from airflow.sdk import dag, task, chain, Param


@dag
def ApprovalOperator_syntax_example():

    @task
    def upstream_task():
        return "Pineapple on pizza?"

    _upstream_task = upstream_task()

    _hitl_task = ApprovalOperator(
        task_id="approval_task",
        subject="Your task:",
        body="{{ ti.xcom_pull(task_ids='upstream_task') }}",
        defaults="Approve", # other option: "Reject"
        params={
            "second_topping": Param(
                "olives",
                type="string",
            )
        },
    )

    @task
    def print_result(hitl_output):
        print(f"Params input: {hitl_output['params_input']}")
        print(f"Chosen options: {hitl_output['chosen_options']}")

    _print_result = print_result(_hitl_task.output)

    chain(_upstream_task, _hitl_task)


ApprovalOperator_syntax_example()
```

The action form shows the two options `Approve` and `Reject` alongside any param form fields.

<Frame>
  <img alt="Screenshot of the Airflow UI showing the ApprovalOperator task instance page with the Required Actions tab open." />
</Frame>

### `HITLBranchOperator`

If you want to [branch](/docs/learn/airflow-branch-operator) your Dag based on the human input, you can use the `HITLBranchOperator`. This operator allows the user to choose one or more tasks that are directly downstream of the `HITLBranchOperator` task to run next. All tasks that aren't chosen will be skipped.

You can use the `options_mapping` parameter to map the human facing options to the task IDs of the tasks that are downstream of the `HITLBranchOperator` task.

```python expandable wrap theme={null}
from airflow.providers.standard.operators.hitl import HITLBranchOperator
from airflow.sdk import dag, task, chain


_budget_categories = ["marketing", "research_development", "facilities", "training", "technology"]


@dag
def HITLBranchOperator_syntax_example():

    @task
    def upstream_task():
        return {
            "total_budget": "$4B",
        }

    _upstream_task = upstream_task()

    _hitl_branch_task = HITLBranchOperator(
        task_id="hitl_branch_task",
        subject="Budget Category Approval",
        body="""**Total Budget Available:** {{ ti.xcom_pull(task_ids='upstream_task')['total_budget'] }}

Select the funding proposals to approve for this quarter.""",
        options=_budget_categories,
        defaults=["marketing", "research_development"],
        multiple=True,
    )

    for _category in _budget_categories:

        @task(
            task_id=f"{_category}",  # needs to match options in HITLBranchOperator
        )
        def category_task():
            print(f"Processing budget approval for {_category}")

        _category_task = category_task()
        chain(_hitl_branch_task, _category_task)

    chain(_upstream_task, _hitl_branch_task)


HITLBranchOperator_syntax_example()
```

The screenshot below shows the graph view created by the code snippet above with 5 tasks downstream of the `HITLBranchOperator` task and the **Required Actions** tab showing the form input.

<Frame>
  <img alt="Screenshot of the Airflow UI showing the HITLBranchOperator task instance page with the Required Actions tab open." />
</Frame>

After approving the budget for 3 of the categories the Dag completes with 3 downstream tasks being run and 2 being skipped.

<Frame>
  <img alt="Screenshot of the Airflow UI showing the HITLBranchOperator task instance page with the Required Actions tab open." />
</Frame>

### `HITLEntryOperator`

The `HITLEntryOperator` is a specialised form of the `HITLOperator` where the user provides input to a form and then submits the input without choosing from a list of options.

```python expandable wrap theme={null}
from airflow.providers.standard.operators.hitl import HITLEntryOperator
from airflow.sdk import dag, task, chain, Param


@dag
def HITLEntryOperator_syntax_example():

    @task
    def upstream_task():
        return "How can I auto-pause a dag if it fails?"

    _upstream_task = upstream_task()

    _hitl_task = HITLEntryOperator(
        task_id="hitl_task",
        subject="Please respond to this ticket!",
        body="{{ ti.xcom_pull(task_ids='upstream_task') }}",
        params={
            "response": Param(
                "You can use the max_consecutive_failed_dag_runs parameter! :)",
                type="string",
            ),
            "urgency": Param(
                "p3",
                type="string",
            ),
        },
    )

    @task
    def print_result(hitl_output):
        print(f"Params input: {hitl_output['params_input']}")
        print(f"Chosen options: {hitl_output['chosen_options']}")

    _print_result = print_result(_hitl_task.output)

    chain(_upstream_task, _hitl_task)


HITLEntryOperator_syntax_example()
```

<Frame>
  <img alt="Screenshot of the Airflow UI showing the HITLEntryOperator task instance page with the Required Actions tab open." />
</Frame>

## Use notifiers with HITL operators

You can use an [Airflow notifier](/docs/learn/error-notifications-in-airflow) to send information from the human-in-the-loop operator to another system, such as Slack or email. The `.notify()` method of the notifier is executed when the task starts running. A simple implementation is to use the `HITLOperator.generate_link_to_ui_from_context` method to return a link to the required action in the Airflow UI for users to click to respond.

The code snippet below shows a sample notifier `MyNotifier` that prints the required action information and the link to the required action to the Airflow logs.

```python expandable wrap theme={null}
from airflow.sdk import BaseNotifier, Context, dag, task, Param
from airflow.providers.standard.operators.hitl import HITLOperator
from datetime import timedelta

_BASE_URL = "http://localhost:28080"

class MyNotifier(BaseNotifier):
    template_fields = ("message",)

    def __init__(self, message: str = "") -> None:
        self.message = message

    def notify(self, context: Context) -> None:
        
        task_state = context['ti'].state
        if task_state == "running":

            # this method generates a direct link to the UI page where the user can respond
            url = HITLOperator.generate_link_to_ui_from_context(
                context=context,
                base_url=_BASE_URL,
            )

            # placeholder code, you can send the URL to any service you want
            self.log.info(self.message)
            self.log.info("Url to respond %s", url)
        else:
            self.log.info("Task state: %s", task_state)
            self.log.info("No response needed!")


notifier_class = MyNotifier(
    message="""
Subject: {{ task.subject }}
Body: {{ task.body }}
Options: {{ task.options }}
"""
)


@dag
def notifier_example():
    HITLOperator(
        task_id="hitl_task",
        subject="Choose a number: ",
        options=["23", "19", "42"],
        notifiers=[notifier_class],
    )


notifier_example()
```

Of course you can also add all regular callback functions such as `on_failure_callback`, `on_success_callback`, etc. to the human-in-the-loop operators.

## Human-in-the-loop API endpoints

If your human (or other entity) doesn't have access to the Airflow UI, you can use the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) to poll for required actions and respond to them.

The relevant endpoints are:

* `GET api/v2/hitlDetails/` to get a list of required actions in an Airflow instance filtered by flags like `state`, `dag_id` and more.
* `GET api/v2/hitlDetails/{dag_id}/{dag_run_id}/{task_id}` to get the details of a specific required action.
* `PATCH api/v2/hitlDetails/{dag_id}/{dag_run_id}/{task_id}` to respond to a specific required action.

These API calls can be combined with others to create scripts like the one below that allows you to respond to all pending required actions in a specific Dag from the command line.

```python expandable wrap theme={null}
import requests
from typing import Any

_USERNAME = "admin"
_PASSWORD = "admin"
_HOST = "http://localhost:28080/"  # To learn how to send API requests to Airflow running on Astro see: https://www.astronomer.io/docs/astro/airflow-api/

_DAG_ID = "HITLOperator_syntax_example"
_TASK_ID = "hitl_task"


def _pick_option(options: list[str]):
    print("Available options: ", options)
    chosen_option = input("Enter the option you want to select: ")
    return chosen_option


def _pick_params(param: dict[str, Any]):
    print("Input for param:", param)
    param_input = input("Enter your value for the param: ")
    return param_input


def _get_jwt_token():
    token_url = f"{_HOST}/auth/token"
    payload = {"username": _USERNAME, "password": _PASSWORD}
    headers = {"Content-Type": "application/json"}
    response = requests.post(token_url, json=payload, headers=headers)

    token = response.json().get("access_token")
    return token


def _get_running_dagruns_for_dag(dag_id: str):
    url = f"{_HOST}/api/v2/dags/{dag_id}/dagRuns?state=running"
    headers = {"Authorization": f"Bearer {_get_jwt_token()}"}
    response = requests.get(url, headers=headers)
    return response.json()


def _get_hitl_details(dag_id: str, dag_run_id: str, task_id: str):
    url = f"{_HOST}/api/v2/hitlDetails/{dag_id}/{dag_run_id}/{task_id}"
    headers = {"Authorization": f"Bearer {_get_jwt_token()}"}
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        subject = response.json()["subject"]
        body = response.json()["body"]
        options = response.json()["options"]
        params = response.json()["params"]
        print("--------------------------------")
        print("Required Action found for: ", dag_id, "DAG Run: ", dag_run_id, "Task: ", task_id)
        print("Subject: ", subject)
        print("Body: ", body)
        print("Options: ", options)
        print("Params: ", params)
        print("--------------------------------")
        return {
            "subject": subject,
            "body": body,
            "options": options,
            "params": params,
        }
    elif response.status_code == 404:
        print("--------------------------------")
        print("404 - No required action found for: ", dag_id, "DAG Run: ", dag_run_id, "Task: ", task_id)
        print("Response: ", response.json())
        print("--------------------------------")
        return None
    else:
        print("--------------------------------")
        print("Error: ", response.status_code)
        print("Response: ", response.json())
        print("--------------------------------")
        return None


def _add_hitl_response(
    dag_id: str, dag_run_id: str, task_id: str, options: list[str], params: dict[str, Any]
):
    url = f"{_HOST}/api/v2/hitlDetails/{dag_id}/{dag_run_id}/{task_id}"
    headers = {"Authorization": f"Bearer {_get_jwt_token()}"}
    chosen_options = [_pick_option(options)]
    if params:
        params_input = {f"{param}": _pick_params(param) for param in params}
    else:
        params_input = {}
    response = requests.patch(
        url, headers=headers, json={"chosen_options": chosen_options, "params_input": params_input}
    )
    if response.status_code == 200:
        print("--------------------------------")
        print("Hitl response added for DAG: ", dag_id, "DAG Run: ", dag_run_id, "Task: ", task_id)
        print("Chosen options: ", chosen_options)
        print("Params input: ", params_input)
        print("Response status code: ", response.status_code)
        print("Response: ", response.json())
        print("--------------------------------")
    elif response.status_code == 409:
        print("--------------------------------")
        print("409 - Already updated action for: ", dag_id, "DAG Run: ", dag_run_id, "Task: ", task_id)
        print("Response: ", response.json())
        print("--------------------------------")
    else:
        print("--------------------------------")
        print("Error: ", response.status_code)
        print("Response: ", response.json())
        print("--------------------------------")


def main():
    dag_runs = _get_running_dagruns_for_dag(_DAG_ID)["dag_runs"]
    if not dag_runs:
        print("No running dag runs found for DAG: ", _DAG_ID)
    else:
        for dag_run in dag_runs:
            hitl_details = _get_hitl_details(_DAG_ID, dag_run["dag_run_id"], _TASK_ID)
            if hitl_details:
                _add_hitl_response(
                    _DAG_ID, dag_run["dag_run_id"], _TASK_ID, hitl_details["options"], hitl_details["params"]
                )


if __name__ == "__main__":
    main()
```


# Custom hooks and operators
Source: https://astronomer.io/docs/learn/airflow-importing-custom-hooks-operators

How to correctly import custom hooks and operators.

One of the great benefits of Airflow is its vast network of provider packages that provide hooks, operators, and sensors for many common use cases. Another great benefit of Airflow is that it is highly customizable because everything is defined in Python code. If a hook, operator, or sensor you need doesn't exist in the open source, you can easily define your own.

In this guide, you'll learn how to define your own custom Airflow operators and hooks to use in your DAGs. To explore existing hooks, operators, and sensors, visit the [Airflow Registry](https://airflow.apache.org/registry/).

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow hooks. See [Hooks 101](/docs/learn/what-is-a-hook).
* Managing Airflow project structure. See [Managing Airflow code](/docs/learn/managing-airflow-code).

## Create a custom operator

A custom operator is a Python class which can be imported into your DAG file. Like regular operators, instantiating a custom operator will create an Airflow task.

At a minimum, a custom operator must:

* Inherit from the `BaseOperator` or any other existing operator.
* Define an `.__init__()` method which runs when the DAG is parsed.
* Define an `.execute()` method which runs when a task uses this operator.

Optionally you can:

* Define a `.pre_execute()` method which runs before the `.execute()` method. This is particularly useful for adding functionality to an existing operator without the need to override the `.execute()` method.
* Define a `.post_execute()` method which runs after the `.execute()` method. `post_execute()` is useful for logging or cleanup tasks that should run after the main task logic, or to push additional information to XCom. The return value of `.execute()` is passed to `.post_execute()` as the `result` argument.

The following is an example of a custom operator called `MyOperator`:

```python expandable wrap theme={null}
# import the operator to inherit from
from airflow.sdk.bases.operator import BaseOperator

# define the class inheriting from an existing operator class
class MyOperator(BaseOperator):
    """
    Simple example operator that logs one parameter and returns a string saying hi.
    :param my_parameter: (required) parameter taking any input.
    """

    # define the .__init__() method that runs when the DAG is parsed
    def __init__(self, my_parameter, *args, **kwargs):
        # initialize the parent operator
        super().__init__(*args, **kwargs)
        # assign class variables
        self.my_parameter = my_parameter

    # define the .pre_execute() method that runs before the execute method (optional)
    def pre_execute(self, context):
        # write to Airflow task logs
        self.log.info("Pre-execution step")

    # define the .execute() method that runs when a task uses this operator.
    # The Airflow context must always be passed to '.execute()', so make
    # sure to include the 'context' kwarg.
    def execute(self, context):
        # write to Airflow task logs
        self.log.info(self.my_parameter)
        # the return value of '.execute()' will be pushed to XCom by default
        return "hi :)"

    # define the .post_execute() method that runs after the execute method (optional)
    # result is the return value of the execute method
    def post_execute(self, context, result=None):
        # write to Airflow task logs
        self.log.info("Post-execution step")

```

If your custom operator is modifying functionality of an existing operator, your class can inherit from the operator you are building on instead of the `BaseOperator`. For more detailed instructions see [Creating a custom Operator](https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html).

<Info>
  It is possible to pass a callable to any operator's `pre_execute` or `post_execute` parameter to inject custom logic into it without needing to define a custom operator. Note that this feature is considered [experimental](https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features).
</Info>

## Create a custom hook

A custom hook is a Python class which can be imported into your DAG file. Like regular hooks, custom hooks can be used to create connections to external tools from within your task code. Custom hooks often contain methods that interact with an external API, which makes them better to use in custom operators than direct API calls.

At a minimum, a custom hook must:

* Inherit from the `BaseHook` or any other existing hook.
* Define an `.__init__()` method which runs when the DAG is parsed.

Many hooks include a `.get_conn()` method wrapping around a call to the `BaseHook` method `.get_connection()` to retrieve information from an Airflow connection. It is common to call the `.get_conn()` method within the `.__init__()` method. The following is the minimum recommended code to start with for most custom hooks:

```python expandable wrap theme={null}
# import the hook to inherit from
from airflow.hooks.base import BaseHook

# define the class inheriting from an existing hook class
class MyHook(BaseHook):
    """
    Interact with <external tool>.
    :param my_conn_id: ID of the connection to <external tool>
    """

    # provide the name of the parameter which receives the connection id
    conn_name_attr = "my_conn_id"
    # provide a default connection id
    default_conn_name = "my_conn_default"
    # provide the connection type
    conn_type = "general"
    # provide the name of the hook
    hook_name = "MyHook"

    # define the .__init__() method that runs when the DAG is parsed
    def __init__(
        self, my_conn_id: str = default_conn_name, *args, **kwargs
    ) -> None:
        # initialize the parent hook
        super().__init__(*args, **kwargs)
        # assign class variables
        self.my_conn_id = my_conn_id
        # (optional) call the '.get_conn()' method upon initialization
        self.get_conn()

    def get_conn(self):
        """Function that initiates a new connection to your external tool."""
        # retrieve the passed connection id
        conn_id = getattr(self, self.conn_name_attr)
        # get the connection object from the Airflow connection
        conn = self.get_connection(conn_id)

        return conn

    # add additional methods to define interactions with your external tool
```

## Import custom hooks and operators

After you've defined a custom hook or operator, you need to make it available to your DAGs. Some legacy Airflow documentation or forums may reference registering your custom operator as an Airflow plugin, but this isn't necessary. To import a custom operator or hook to your DAGs, the operator or hook file needs to be in a directory that is present in your `PYTHONPATH`. See the Apache Airflow [module management documentation](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/modules_management.html) for more info.

When using the [Astro CLI](/docs/cli/v1.43/install-cli) you can add your custom operator file to the `include` directory of your Astro project. Consider adding sub-folders to make your `include` directory easier to navigate.

<Note>On Astro with Airflow 3, the `dags/` folder isn't on the `PYTHONPATH` and you can't customize the `PYTHONPATH` to include it. Place custom hooks, operators, and other shared modules in the `include/` folder so that your Dags can import them.</Note>

```text wrap theme={null}
.
├── .astro/
├── dags/
│   └── example_dag.py
├── include/
│   └── custom_operators/
│       └── my_operator.py
│   └── custom_hooks/
│       └── my_hook.py
├── plugins/
├── tests/
├── .dockerignore
├── .env
├── .gitignore
├── .airflow_settings.yaml
├── Dockerfile
├── packages.txt
├── README.md
└── requirements.txt
```

For more details on why Astronomer recommends this project structure, see the [Managing Airflow Code guide](/docs/learn/managing-airflow-code).

Using the project structure shown above, you can import the `MyOperator` class from the `my_operator.py` file and the `MyHook` class from the `my_hook.py` file in your DAGs with the following import statements:

```text wrap theme={null}
from include.custom_operators.my_operator import MyOperator
from include.custom_hooks.my_hook import MyHook
```

## Example implementation

The following code defines the `MyBasicMathOperator` class. This operator inherits from the `BaseOperator` and can perform arithmetic when you provide it two numbers and an operation. This code is saved in the `include` folder in a file called `basic_math_operator.py`.

```python expandable wrap theme={null}
from airflow.sdk.bases.operator import BaseOperator

class MyBasicMathOperator(BaseOperator):
    """
    Example Operator that does basic arithmetic.
    :param first_number: first number to put into an equation
    :param second_number: second number to put into an equation
    :param operation: mathematical operation to perform
    """

    # provide a list of valid operations
    valid_operations = ("+", "-", "*", "/")
    # define which fields can use Jinja templating
    template_fields = ("first_number", "second_number")

    def __init__(
        self,
        first_number: float,
        second_number: float,
        operation: str,
        *args,
        **kwargs,
    ):
        super().__init__(*args, **kwargs)
        self.first_number = first_number
        self.second_number = second_number
        self.operation = operation

        # raise an import error if the operation provided is not valid
        if self.operation not in self.valid_operations:
            raise ValueError(
                f"{self.operation} is not a valid operation. Choose one of {self.valid_operations}"
            )

    def execute(self, context):
        self.log.info(
            f"Equation: {self.first_number} {self.operation} {self.second_number}"
        )
        if self.operation == "+":
            res = self.first_number + self.second_number
            self.log.info(f"Result: {res}")
            return res
        if self.operation == "-":
            res = self.first_number - self.second_number
            self.log.info(f"Result: {res}")
            return res
        if self.operation == "*":
            res = self.first_number * self.second_number
            self.log.info(f"Result: {res}")
            return res
        if self.operation == "/":
            try:
                res = self.first_number / self.second_number
            except ZeroDivisionError as err:
                self.log.critical(
                    "If you have set up an equation where you are trying to divide by zero, you have done something WRONG. - Randall Munroe, 2006"
                )
                raise ZeroDivisionError

            self.log.info(f"Result: {res}")
            return res
```

In addition to the custom operator, the example DAG uses a custom hook to connect to the CatFactAPI. This hook abstracts retrieving the API URL from an [Airflow connection](/docs/learn/connections) and makes several calls to the API in a loop. This code should also be placed in the `include` directory in a file called `cat_fact_hook.py`.

```python expandable wrap theme={null}
"""This module allows you to connect to the CatFactAPI."""

from airflow.hooks.base import BaseHook
import requests as re

class CatFactHook(BaseHook):
    """
    Interact with the CatFactAPI.

    Performs a connection to the CatFactAPI and retrieves a cat fact client.

    :cat_fact_conn_id: Connection ID to retrieve the CatFactAPI url.
    """

    conn_name_attr = "cat_conn_id"
    default_conn_name = "cat_conn_default"
    conn_type = "http"
    hook_name = "CatFact"

    def __init__(
        self, cat_fact_conn_id: str = default_conn_name, *args, **kwargs
    ) -> None:
        super().__init__(*args, **kwargs)
        self.cat_fact_conn_id = cat_fact_conn_id
        self.get_conn()

    def get_conn(self):
        """Function that initiates a new connection to the CatFactAPI."""

        # get the connection object from the Airflow connection
        conn = self.get_connection(self.cat_fact_conn_id)

        # return the host URL
        return conn.host

    def log_cat_facts(self, number_of_cat_facts_needed: int = 1):
        """Function that logs between 1 to 10 catfacts depending on its input."""
        if number_of_cat_facts_needed < 1:
            self.log.info(
                "You will need at least one catfact! Setting request number to 1."
            )
            number_of_cat_facts_needed = 1
        if number_of_cat_facts_needed > 10:
            self.log.info(
                f"{number_of_cat_facts_needed} are a bit many. Setting request number to 10."
            )
            number_of_cat_facts_needed = 10

        cat_fact_connection = self.get_conn()

        # log several cat facts using the connection retrieved
        for i in range(number_of_cat_facts_needed):
            cat_fact = re.get(cat_fact_connection).json()
            self.log.info(cat_fact["fact"])
        return f"{i} catfacts written to the logs!"

```

To use this custom hook, you need to create an Airflow connection with the connection ID `cat_fact_conn`, the connection type **HTTP**, and the Host `http://catfact.ninja/fact`.

<Frame>
  <img alt="Cat fact connection" />
</Frame>

You can then import the custom operator and custom hook into your DAG. Because the custom operator has defined `first_value` and `second_value` as `template_fields`, you can pass values from other tasks to these parameters using Jinja templating.

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  from pendulum import datetime
  from airflow.decorators import dag, task
  from include.basic_math_operator import MyBasicMathOperator
  from include.cat_fact_hook import CatFactHook


  @dag(
      schedule_interval="@daily",
      start_date=datetime(2021, 1, 1),
      # render Jinja template as native Python object
      render_template_as_native_obj=True,
      catchup=False,
  )
  def my_math_cat_dag():
      add = MyBasicMathOperator(
          task_id="add",
          first_number=23,
          second_number=19,
          operation="+",
          # any BaseOperator arguments can be used with the custom operator too
          doc_md="Addition Task.",
      )

      multiply = MyBasicMathOperator(
          task_id="multiply",
          # use the return value from the add task as the first_number, pulling from XCom
          first_number="{{ ti.xcom_pull(task_ids='add', key='return_value') }}",
          second_number=35,
          operation="-",
      )

      @task
      def use_cat_fact_hook(number):
          num_catfacts_needed = round(number)
          # instatiating a CatFactHook at runtime of this task
          hook = CatFactHook("cat_fact_conn")
          hook.log_cat_facts(num_catfacts_needed)

      add >> multiply >> use_cat_fact_hook(multiply.output)


  my_math_cat_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  from pendulum import datetime
  from airflow import DAG
  from airflow.operators.python import PythonOperator
  from include.basic_math_operator import MyBasicMathOperator
  from include.cat_fact_hook import CatFactHook


  def use_cat_fact_hook(number):
      num_catfacts_needed = round(number)
      # instatiating a CatFactHook at runtime of this task
      hook = CatFactHook("cat_fact_conn")
      hook.log_cat_facts(num_catfacts_needed)


  with DAG(
      dag_id="my_math_cat_dag",
      schedule_interval="@daily",
      start_date=datetime(2021, 1, 1),
      # render Jinja template as native Python object
      render_template_as_native_obj=True,
      catchup=False,
  ):
      add = MyBasicMathOperator(
          task_id="add",
          first_number=23,
          second_number=19,
          operation="+",
          # any BaseOperator arguments can be used with the custom operator too
          doc_md="Addition Task.",
      )

      multiply = MyBasicMathOperator(
          task_id="multiply",
          # use the return value from the add task as the first_number, pulling from XCom
          first_number="{{ ti.xcom_pull(task_ids='add', key='return_value') }}",
          second_number=35,
          operation="-",
      )

      use_cat_fact_hook_task = PythonOperator(
          task_id="use_cat_fact_hook",
          python_callable=use_cat_fact_hook,
          op_args=[multiply.output],
      )

      add >> multiply >> use_cat_fact_hook_task
  ```
</details>


# Run tasks in an isolated environment in Apache Airflow
Source: https://astronomer.io/docs/learn/airflow-isolated-environments

Learn how to run an Airflow task in an isolated environment.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

It is very common to run a task with different dependencies than your Airflow environment. Your task might need a different Python version than core Airflow, or it has packages that conflict with your other tasks. In these cases, running tasks in an isolated environment can help manage dependency conflicts and enable compatibility with your execution environments.

In Airflow, you have several options for running custom Python code in isolated environments. This guide teaches you how to choose the right isolated environment option for your use case, implement different virtual environment operators and decorators, and access Airflow context and variables in isolated environments.

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Astronomer Academy: [Airflow: The `ExternalPythonOperator`](https://academy.astronomer.io/astro-runtime-the-externalpythonoperator).
  * Astronomer Academy: [Airflow: The `KubernetesPodOperator`](https://academy.astronomer.io/astro-runtime-the-kubernetespodoperator-1).
  * Webinar: [Running Airflow Tasks in Isolated Environments](https://www.astronomer.io/events/webinars/running-airflow-tasks-in-isolated-environments-video/).
  * Learn from code: [Isolated environments example DAGs repository](https://github.com/astronomer/learn-demos/tree/airflow-isolated-environments).
</Tip>

<Info>
  This guide covers options to isolate individual tasks in Airflow. If you want to run all of your Airflow tasks in dedicated Kubernetes pods, consider using the [Kubernetes Executor](/docs/learn/airflow-executors-explained#kubernetesexecutor). Astronomer customers can set their Deployments to use the KubernetesExecutor in the Astro UI, see [Manage Airflow executors on Astro](/docs/astro/executors-overview).
</Info>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow decorators. See [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).
* Airflow operators. See [Airflow operators](/docs/learn/what-is-an-operator).
* Python Virtual Environments. See [Python Virtual Environments: A Primer](https://realpython.com/python-virtual-environments-a-primer/).
* Kubernetes basics. See the [Kubernetes Documentation](https://kubernetes.io/docs/home/).

## When to use isolated environments

There are two situations when you might want to run a task in an isolated environment:

* Your task requires a **different version of Python** than your Airflow environment. Apache Airflow is compatible with and available in Python 3.8, 3.9, 3.10, 3.11, and 3.12. The Astro Runtime has [images](https://quay.io/repository/astronomer/astro-runtime?tab=tags) available for all supported Python versions, so you can run Airflow inside Docker in a reproducible environment. See [Prerequisites](https://airflow.apache.org/docs/apache-airflow/stable/installation/prerequisites.html) for more information.
* Your task requires **different versions of Python packages** that conflict with the package versions installed in your Airflow environment. To know which Python packages are pinned to which versions within Airflow, you can retrieve the full list of constraints for each Airflow version by going to:

  ```text wrap theme={null}
  https://raw.githubusercontent.com/apache/airflow/constraints-<AIRFLOW VERSION>/constraints-<PYTHON VERSION>.txt
  ```

<Tip>
  **Airflow Best Practice**

  Make sure to pin all package versions, both in your core Airflow environment (`requirements.txt`) and in your isolated environments. This helps you avoid unexpected behavior due to package updates that might create version conflicts.
</Tip>

### Limitations

When creating isolated environments in Airflow, you might not be able to use common Airflow features or connect to your Airflow environment in the same way you would in a regular Airflow task.

Common limitations include:

* You [can't pass all Airflow context variables](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#id1) to a virtual decorator, since Airflow doesn't support serializing `var`, `ti`, and `task_instance` objects. See [Use Airflow context variables in isolated environments](#use-airflow-context-variables-in-isolated-environments).
* You don't have access to your [secrets backend](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html) from within the isolated environment. To access your secrets, consider passing them in through [Jinja templating](/docs/learn/templating). See [Use Airflow variables in isolated environments](#use-airflow-variables-in-isolated-environments).
* Installing Airflow itself, or Airflow provider packages in the environment provided to the `@task.external_python` decorator or the `ExternalPythonOperator`, can lead to unexpected behavior. If you need to use Airflow or an Airflow provider module inside your virtual environment, Astronomer recommends using the `@task.virtualenv` decorator or the `PythonVirtualenvOperator` instead. See [Use Airflow packages in isolated environments](#use-airflow-packages-in-isolated-environments).

## Choose an isolated environment option

Airflow provides several options for running tasks in isolated environments.

To run tasks in a dedicated Kubernetes Pod you can use:

* [`@task.kubernetes`](#kubernetes-pod-operator) decorator
* [`KubernetesPodOperator`](#kubernetes-pod-operator) (KPO)

To run tasks in a Python virtual environment you can use:

* [`@task.external_python`](#external-python-operator) decorator / `ExternalPythonOperator` (EPO)
* [`@task.virtualenv`](#virtualenv-operator) decorator / `PythonVirtualenvOperator` (PVO)
* [`@task.branch_external_python`](#virtual-branching-operators) decorator / `BranchExternalPythonOperator` (BEPO)
* [`@task.branch_virtualenv`](#virtual-branching-operators) decorator / `BranchPythonVirtualenvOperator` (BPVO)

The virtual environment decorators have operator equivalents with the same functionality. Astronomer recommends using decorators where possible because they simplify the handling of [XCom](/docs/learn/airflow-passing-data-between-tasks).

<Frame>
  <img alt="Graph of options for isolated environments in Airflow." />
</Frame>

Which option you choose depends on your use case and the requirements of your task. The table below shows which decorators and operators are best for particular use cases.

| Use Case                                                         | Implementation Options                                                                                                         |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| Run a Python task in a K8s Pod                                   | [`@task.kubernetes`](#kubernetes-pod-operator),<br /> [`KubernetesPodOperator`](#kubernetes-pod-operator)                      |
| Run a Docker image without additional Python code in a K8s Pod   | [`KubernetesPodOperator`](#kubernetes-pod-operator)                                                                            |
| Run a Python task in an existing (reusable) virtual environment  | [`@task.external_python`](#external-python-operator),<br /> [`ExternalPythonOperator`](#external-python-operator)              |
| Run a Python task in a new virtual environment                   | [`@task.virtualenv`](#virtualenv-operator),<br /> [`PythonVirtualenvOperator`](#virtualenv-operator)                           |
| Run branching code in an existing (reusable) virtual environment | [`@task.branch_external_python`](#virtual-branching-operators), [`BranchExternalPythonOperator`](#virtual-branching-operators) |
| Run branching code in a new virtual environment                  | [`@task.branch_virtualenv`](#virtual-branching-operators), [`BranchPythonVirtualenvOperator`](#virtual-branching-operators)    |
| Install different packages for each run of a task                | [`PythonVirtualenvOperator`](#virtualenv-operator),<br />[`BranchPythonVirtualenvOperator`](#virtual-branching-operators)      |

Another consideration when choosing an operator is the infrastructure you have available. Operators that run tasks in Kubernetes pods allow you to have full control over the environment and resources used, but they require a Kubernetes cluster. Operators that run tasks in Python virtual environments are easier to set up, but don't provide the same level of control over the environment and resources used.

| Requirements         | Decorators                                                                                                                                                                                                                                              | Operators                                                                                                                                                                                                                                                               |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A Kubernetes cluster | [`@task.kubernetes`](#kubernetes-pod-operator)                                                                                                                                                                                                          | [`KubernetesPodOperator`](#kubernetes-pod-operator)                                                                                                                                                                                                                     |
| A Docker image       | [`@task.kubernetes`](#kubernetes-pod-operator) (with Python installed)                                                                                                                                                                                  | [`KubernetesPodOperator`](#kubernetes-pod-operator) (with or without Python installed)                                                                                                                                                                                  |
| A Python binary      | [`@task.external_python`](#external-python-operator),<br /> [`@task.branch_external_python`](#virtual-branching-operators),<br /> [`@task.virtualenv`](#virtualenv-operator) (\*),<br /> [`@task.branch_virtualenv`](#virtual-branching-operators) (\*) | [`ExternalPythonOperator`](#external-python-operator),<br /> [`BranchExternalPythonOperator`](#virtual-branching-operators),<br /> [`PythonVirtualenvOperator`](#virtualenv-operator) (\*),<br /> [`BranchPythonVirtualenvOperator`](#virtual-branching-operators) (\*) |

\*Only required if you need to use a different Python version than your Airflow environment.

## External Python operator

The ExternalPython operator, `@task.external_python` decorator or `ExternalPythonOperator`, runs a Python function in an existing virtual Python environment, isolated from your Airflow environment. To use the `@task.external_python` decorator or the `ExternalPythonOperator`, you need to create a separate Python environment to reference. You can use any Python binary created by any means.

The easiest way to create a Python environment when using the Astro CLI is with the [Astronomer PYENV BuildKit](https://github.com/astronomer/astro-provider-venv). The BuildKit can be used by adding a comment on the first line of the Dockerfile as shown in the following example. Adding this comment enables you to create virtual environments with the `PYENV` keyword.

```dockerfile wrap theme={null}
# syntax=quay.io/astronomer/airflow-extensions:v1

FROM quay.io/astronomer/astro-runtime:10.3.0-python-3.11

# create a virtual environment for the ExternalPythonOperator and @task.external_python decorator
# using Python 3.9 and install the packages from epo_requirements.txt
PYENV 3.9 epo_pyenv epo_requirements.txt
```

<Note>
  To use the BuildKit, the [Docker BuildKit Backend](https://docs.docker.com/build/buildkit/) needs to be enabled. This is the default as of Docker Desktop version 23.0, but might need to be enabled manually in older versions of Docker.
</Note>

You can add any Python packages to the virtual environment by putting them into a separate requirements file. In this example, by using the name `epo_requirements.txt`. Make sure to pin all package versions.

```text wrap theme={null}
pandas==1.4.4
```

<Warning>
  Installing Airflow itself and Airflow provider packages in isolated environments can lead to unexpected behavior and isn't recommended. If you need to use Airflow or Airflow provider modules inside your virtual environment, Astronomer recommends choosing the `@task.virtualenv` decorator or the `PythonVirtualenvOperator`. See [Use Airflow packages in isolated environments](#use-airflow-packages-in-isolated-environments).
</Warning>

After restarting your Airflow environment, you can use this Python binary by referencing the environment variable `ASTRO_PYENV_<my-pyenv-name>`. If you choose an alternative method to create you Python binary, you need to set the `python` parameter of the decorator or operator to the location of your Python binary.

<details>
  <summary>TaskFlow</summary>

  To run any Python function in your virtual environment, use the `@task.external_python` decorator on it and set the `python` parameter to the location of your Python binary.

  ```python wrap theme={null}
  # from airflow.decorators import task
  # import os

  @task.external_python(python=os.environ["ASTRO_PYENV_epo_pyenv"])
  def my_isolated_task():
      import pandas as pd
      import sys
      print(f"The python version in the virtual env is: {sys.version}")
      print(f"The pandas version in the virtual env is: {pd.__version__}")
      # your code to run in the isolated environment
  ```
</details>

<details>
  <summary>Traditional</summary>

  To run any Python function in your virtual environment, define the `python_callable` parameter of the `ExternalPythonOperator` with your Python function, and set the `python` parameter to the location of your Python binary.

  ```python wrap theme={null}
  # from airflow.operators.python import ExternalPythonOperator
  # import os

  def my_isolated_function():
      import pandas as pd
      import sys
      print(f"The python version in the virtual env is: {sys.version}")
      print(f"The pandas version in the virtual env is: {pd.__version__}")

  my_isolated_task = ExternalPythonOperator(
      task_id="my_isolated_task",
      python_callable=my_isolated_function,
      python=os.environ["ASTRO_PYENV_epo_pyenv"]
  )
  ```
</details>

<details>
  <summary>TaskFlow XCom</summary>

  You can pass information into and out of the `@task.external_python` decorated task the same way as you would when interacting with a `@task` decorated task, see also [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).

  ```python expandable wrap theme={null}
  """
  ## Toy example of using the @task.external_python decorator

  The @task.external_python decorator is used to run any Python code in an existing isolated Python environment.
  """

  from airflow.decorators import dag, task
  import pandas as pd
  import sys
  import os


  @dag(
      start_date=None,
      schedule=None,
      doc_md=__doc__,
      description="@task.external_python",
      default_args={
          "owner": "airflow",
          "retries": 0,
      },
      tags=["@task.external_python"],
  )
  def external_python_decorator_dag():

      @task
      def upstream_task():
          print(f"The python version in the upstream task is: {sys.version}")
          print(f"The pandas version in the upstream task is: {pd.__version__}")
          return {"num": 1, "word": "hello"}

      @task.external_python(python=os.environ["ASTRO_PYENV_epo_pyenv"])
      def my_isolated_task(upstream_task_output: dict):
          """
          This function runs in an isolated environment.
          Args:
              upstream_task_output (dict): contains a number and a word.
          Returns:
              pd.DataFrame: A dictionary containing the transformed inputs.
          """
          import pandas as pd
          import sys

          print(f"The python version in the virtual env is: {sys.version}")
          print(f"The pandas version in the virtual env is: {pd.__version__}")

          num = upstream_task_output["num"]
          word = upstream_task_output["word"]

          num_plus_one = num + 1
          word_plus_exclamation = word + "!"

          df = pd.DataFrame(
              {
                  "num_plus_one": [num_plus_one],
                  "word_plus_exclamation": [word_plus_exclamation],
              },
          )

          return df

      @task
      def downstream_task(arg):
          print(f"The python version in the downstream task is: {sys.version}")
          print(f"The pandas version in the downstream task is: {pd.__version__}")
          return arg

      downstream_task(my_isolated_task(upstream_task()))


  external_python_decorator_dag()
  ```
</details>

<details>
  <summary>Traditional XCom</summary>

  You can pass information into the `ExternalPythonOperator` by using a [Jinja template](/docs/learn/templating) retrieving [XCom](/docs/learn/airflow-passing-data-between-tasks) values from the [Airflow context](/docs/learn/airflow-context). To pass information out of the `ExternalPythonOperator`, return it from the `python_callable`.
  Note that Jinja templates are rendered as strings unless you set `render_template_as_native_obj=True` in the Dag or task definition.

  ```python expandable wrap theme={null}
  """
  ## Toy example of using the ExternalPythonOperator

  The ExternalPythonOperator is used to run any Python code in an existing isolated Python environment.
  """

  from airflow.decorators import dag, task
  from airflow.models.baseoperator import chain
  from airflow.operators.python import ExternalPythonOperator
  import pandas as pd
  import sys
  import os


  def my_isolated_function(num: int, word: str) -> dict:
      """
      This function will be passed to the ExternalPythonOperator to
      run in an isolated environment.
      Args:
          num (int): An integer to be incremented by 1.
          word (str): A string to have an exclamation mark added to it.
      Returns:
          pd.DataFrame: A dictionary containing the transformed inputs.
      """
      import pandas as pd
      import sys

      print(f"The python version in the virtual env is: {sys.version}")
      print(f"The pandas version in the virtual env is: {pd.__version__}")

      num_plus_one = num + 1
      word_plus_exclamation = word + "!"

      df = pd.DataFrame(
          {
              "num_plus_one": [num_plus_one],
              "word_plus_exclamation": [word_plus_exclamation],
          },
      )

      return df


  @dag(
      start_date=None,
      schedule=None,
      doc_md=__doc__,
      description="ExternalPythonOperator",
      render_template_as_native_obj=True,
      default_args={
          "owner": "airflow",
          "retries": 0,
      },
      tags=["ExternalPythonOperator"],
  )
  def external_python_operator_dag():

      @task
      def upstream_task():
          print(f"The python version in the upstream task is: {sys.version}")
          print(f"The pandas version in the upstream task is: {pd.__version__}")
          return {"num": 1, "word": "hello"}

      my_isolated_task = ExternalPythonOperator(
          task_id="my_isolated_task",
          python_callable=my_isolated_function,
          python=os.environ["ASTRO_PYENV_epo_pyenv"],
          op_kwargs={
              # note that render_template_as_native_obj=True in the DAG definition
              # to render num as an integer
              "num": "{{ ti.xcom_pull(task_ids='upstream_task')['num']}}",
              "word": "{{ ti.xcom_pull(task_ids='upstream_task')['word']}}",
          },
      )

      @task
      def downstream_task(arg):
          print(f"The python version in the downstream task is: {sys.version}")
          print(f"The pandas version in the downstream task is: {pd.__version__}")
          return arg

      chain(upstream_task(), my_isolated_task, downstream_task(my_isolated_task.output))


  external_python_operator_dag()
  ```
</details>

To get a list of all parameters of the `@task.external_python` decorator /  `ExternalPythonOperator`, see the [Airflow Registry](https://airflow.apache.org/registry/providers/standard#standard-python-ExternalPythonOperator).

## Virtualenv operator

The Virtualenv operator (`@task.virtualenv` or `PythonVirtualenvOperator`) creates a new virtual environment each time the task runs. If you only specify different package versions and use the same Python version as your Airflow environment, you don't need to create or specify a Python binary.

<Warning>
  Installing Airflow itself and Airflow provider packages in isolated environments can lead to unexpected behavior and is generally not recommended. See [Use Airflow packages in isolated environments](#use-airflow-packages-in-isolated-environments).
</Warning>

<details>
  <summary>TaskFlow</summary>

  Add the pinned versions of the packages to the `requirements` parameter of the `@task.virtualenv` decorator. The decorator creates a new virtual environment at runtime.

  ```python wrap theme={null}
  # from airflow.decorators import task

  @task.virtualenv(requirements=["pandas==1.5.1"])  # add your requirements to the list
  def my_isolated_task(): 
      import pandas as pd
      print(f"The pandas version in the virtual env is: {pd.__version__}")"
      # your code to run in the isolated environment
  ```
</details>

<details>
  <summary>Traditional</summary>

  Add the pinned versions of the packages you need to the `requirements` parameter of the `PythonVirtualenvOperator`. The operator creates a new virtual environment at runtime.

  ```python wrap theme={null}
  # from airflow.operators.python import PythonVirtualenvOperator

  def my_isolated_function():
      import pandas as pd
      print(f"The pandas version in the virtual env is: {pd.__version__}")
      # your code to run in the isolated environment

  my_isolated_task = PythonVirtualenvOperator(
      task_id="my_isolated_task",
      python_callable=my_isolated_function,
      requirements=[
          "pandas==1.5.1",
      ]  # add your requirements to the list
  )
  ```
</details>

<details>
  <summary>TaskFlow XCom</summary>

  You can pass information into and out of the `@task.virtualenv` decorated task using the same process as you would when interacting with a `@task` decorated task. See [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators) for more detailed information.

  ```python expandable wrap theme={null}
  """
  ## Toy example of using the @task.virtualenv decorator

  The @task.virtualenv decorator is used to run any Python code in a new isolated Python environment.
  """

  from airflow.decorators import dag, task
  import pandas as pd


  @dag(
      start_date=None,
      schedule=None,
      doc_md=__doc__,
      description="@task.virtualenv",
      default_args={
          "owner": "airflow",
          "retries": 0,
      },
      tags=["@task.virtualenv"],
  )
  def virtualenv_decorator_dag():

      @task
      def upstream_task():
          print(f"The pandas version in the upstream task is: {pd.__version__}")
          return {"num": 1, "word": "hello"}

      @task.virtualenv(requirements=["pandas==1.5.1"])
      def my_isolated_task(upstream_task_output: dict):
          """
          This function runs in an isolated environment.
          Args:
              upstream_task_output (dict): contains a number and a word.
          Returns:
              pd.DataFrame: A dictionary containing the transformed inputs.
          """
          import pandas as pd

          print(f"The pandas version in the virtual env is: {pd.__version__}")

          num = upstream_task_output["num"]
          word = upstream_task_output["word"]

          num_plus_one = num + 1
          word_plus_exclamation = word + "!"

          df = pd.DataFrame(
              {
                  "num_plus_one": [num_plus_one],
                  "word_plus_exclamation": [word_plus_exclamation],
              },
          )

          return df

      @task
      def downstream_task(arg):
          print(f"The pandas version in the downstream task is: {pd.__version__}")
          return arg

      downstream_task(my_isolated_task(upstream_task_output=upstream_task()))


  virtualenv_decorator_dag()
  ```
</details>

<details>
  <summary>Traditional XCom</summary>

  You can pass information into the `PythonVirtualenvOperator` by using a [Jinja template](/docs/learn/templating) to retrieve [XCom](/docs/learn/airflow-passing-data-between-tasks) values from the [Airflow context](/docs/learn/airflow-context). To pass information out of the `PythonVirtualenvOperator`, return it from the `python_callable`.
  Note that Jinja templates are rendered as strings unless you set `render_template_as_native_obj=True` in the Dag or task definition.

  ```python expandable wrap theme={null}
  """
  ## Toy example of using the PythonVirtualenvOperator

  The PythonVirtualenvOperator is used to run any Python code in a new isolated Python environment.
  """

  from airflow.decorators import dag, task
  from airflow.models.baseoperator import chain
  from airflow.operators.python import PythonVirtualenvOperator
  import pandas as pd
  import sys


  def my_isolated_function(num: int, word: str) -> dict:
      """
      This function will be passed to the PythonVirtualenvOperator to
      run in an isolated environment.
      Args:
          num (int): An integer to be incremented by 1.
          word (str): A string to have an exclamation mark added to it.
      Returns:
          pd.DataFrame: A dictionary containing the transformed inputs.
      """
      import pandas as pd

      print(f"The pandas version in the virtual env is: {pd.__version__}")

      num_plus_one = num + 1
      word_plus_exclamation = word + "!"

      df = pd.DataFrame(
          {
              "num_plus_one": [num_plus_one],
              "word_plus_exclamation": [word_plus_exclamation],
          },
      )

      return df


  @dag(
      start_date=None,
      schedule=None,
      doc_md=__doc__,
      description="PythonVirtualenvOperator",
      render_template_as_native_obj=True,
      default_args={
          "owner": "airflow",
          "retries": 0,
      },
      tags=["PythonVirtualenvOperator"],
  )
  def python_virtualenv_operator_dag():

      @task
      def upstream_task():
          print(f"The python version in the upstream task is: {sys.version}")
          print(f"The pandas version in the upstream task is: {pd.__version__}")
          return {"num": 1, "word": "hello"}

      my_isolated_task = PythonVirtualenvOperator(
          task_id="my_isolated_task",
          python_callable=my_isolated_function,
          requirements=["pandas==1.5.1"],
          op_kwargs={
              # note that render_template_as_native_obj=True in the DAG definition
              # to render num as an integer
              "num": "{{ ti.xcom_pull(task_ids='upstream_task')['num']}}",
              "word": "{{ ti.xcom_pull(task_ids='upstream_task')['word']}}",
          },
      )

      @task
      def downstream_task(arg):
          print(f"The python version in the downstream task is: {sys.version}")
          print(f"The pandas version in the downstream task is: {pd.__version__}")
          return arg

      chain(upstream_task(), my_isolated_task, downstream_task(my_isolated_task.output))


  python_virtualenv_operator_dag()
  ```
</details>

Since the `requirements` parameter of the `PythonVirtualenvOperator` is [templatable](/docs/learn/templating), you can use [Jinja templating](/docs/learn/templating) to pass information at runtime. For example, you can use a Jinja template to install a different version of pandas for each run of the task.

```python wrap theme={null}
# from airflow.decorators import task
# from airflow.models.baseoperator import chain
# from airflow.operators.python import PythonVirtualenvOperator

@task
def get_pandas_version():
    pandas_version = "1.5.1"  # retrieve the pandas version according to your logic
    return pandas_version

my_isolated_task = PythonVirtualenvOperator(
    task_id="my_isolated_task",
    python_callable=my_isolated_function,
    requirements=[
        "pandas=={{ ti.xcom_pull(task_ids='get_pandas_version') }}",
    ],
)

chain(get_pandas_version(), my_isolated_task)
```

If your task requires a different Python version than your Airflow environment, you need to install the Python version your task requires in your Airflow environment so the Virtualenv task can use it. Use the [Astronomer PYENV BuildKit](https://github.com/astronomer/astro-provider-venv) to install a different Python version in your Dockerfile.

```dockerfile wrap theme={null}
# syntax=quay.io/astronomer/airflow-extensions:v1

FROM quay.io/astronomer/astro-runtime:10.3.0-python-3.11

PYENV 3.10 pyenv_3_10
```

<Note>
  To use the BuildKit, the [Docker BuildKit Backend](https://docs.docker.com/build/buildkit/) needs to be enabled. This is the default starting in Docker Desktop version 23.0, but might need to be enabled manually in older versions of Docker.
</Note>

The Python version can be referenced directly using the `python` parameter of the decorator/operator.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # from airflow.decorators import task

  @task.virtualenv(
      requirements=["pandas==1.5.1"], 
      python_version="3.10",  # specify the Python version
  )
  def my_isolated_task():
      import pandas as pd
      import sys
      print(f"The python version in the virtual env is: {sys.version}")
      print(f"The pandas version in the virtual env is: {pd.__version__}")
      # your code to run in the isolated environment
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  #from airflow.operators.python import PythonVirtualenvOperator

  def my_isolated_function():
      import pandas as pd
      import sys
      print(f"The python version in the virtual env is: {sys.version}")
      print(f"The pandas version in the virtual env is: {pd.__version__}")
      # your code to run in the isolated environment

  my_isolated_task = PythonVirtualenvOperator(
      task_id="my_isolated_task",
      python_callable=my_isolated_function,
      requirements=["pandas==1.5.1"],
      python_version="3.10",  # specify the Python version
  )
  ```
</details>

To get a list of all parameters of the `@task.virtualenv` decorator or `PythonVirtualenvOperator`, see the [Airflow Registry](https://airflow.apache.org/registry/providers/standard#standard-python-PythonVirtualenvOperator).

## Kubernetes pod operator

The Kubernetes operator, `@task.kubernetes` decorator or `KubernetesPodOperator`, runs an Airflow task in a dedicated Kubernetes pod. You can use the `@task.kubernetes` to run any custom Python code in a separate Kubernetes pod on a Docker image with Python installed, while the `KubernetesPodOperator` runs any existing Docker image.

To use the `@task.kubernetes` decorator or the `KubernetesPodOperator`, you need to provide a Docker image and have access to a Kubernetes cluster. The following example shows how to use the modules to run a task in a separate Kubernetes pod in the same namespace and Kubernetes cluster as your Airflow environment. For more information on how to use the `KubernetesPodOperator`, see [Use the `KubernetesPodOperator`](/docs/learn/kubepod-operator) and [Run the `KubernetesPodOperator` on Astro](/docs/astro/kubernetespodoperator).

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # from airflow.decorators import task
  # from airflow.configuration import conf

  # if you are running Airflow on Kubernetes, you can get 
  # the current namespace from the Airflow conf
  namespace = conf.get("kubernetes", "NAMESPACE")

  @task.kubernetes(
      image="<YOUR IMAGE>",
      in_cluster=True,
      namespace=namespace,
      name="<YOUR POD NAME>",
      get_logs=True,
      log_events_on_failure=True,
      do_xcom_push=True,
  )
  def my_isolated_task(num: int):
      return num + 1
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
  # from airflow.configuration import conf

  # if you are running Airflow on Kubernetes, you can get 
  # the current namespace from the Airflow conf
  namespace = conf.get("kubernetes", "NAMESPACE")

  my_isolated_task = KubernetesPodOperator(
      task_id="my_isolated_task",
      namespace=namespace,
      # your Docker image contains the scripts to run in the isolated environment  
      image="<YOUR IMAGE>",
      name="<YOUR POD NAME>",
      in_cluster=True,
      is_delete_operator_pod=True,
      get_logs=True,
  )
  ```
</details>

## Virtual branching operators

Virtual branching operators allow you to run conditional task logic in an isolated Python environment.

* `@task.branch_external_python` decorator or `BranchExternalPythonOperator`: Run conditional task logic in an existing virtual Python environment.
* `@task.branch_virtualenv` decorator or `BranchPythonVirtualenvOperator`: Run conditional task logic in a newly created virtual Python environment.

To run conditional task logic in an isolated environment, use the branching versions of the virtual environment decorators and operators. You can learn more about branching in Airflow in the [Branching in Airflow](/docs/learn/airflow-branch-operator) guide.

<details>
  <summary>TaskFlow Epo</summary>

  ```python wrap theme={null}
  # from airflow.decorators import task
  # import os

  @task.branch_external_python(python=os.environ["ASTRO_PYENV_epo_pyenv"])
  def my_isolated_task():
      import pandas as pd
      import random
      print(f"The pandas version in the virtual env is: {pd.__version__}")

      num = random.randint(0, 100)

      if num > 50:
          # return the task_id of the downstream task that should be executed
          return "downstream_task_a"  
      else:
          return "downstream_task_b"
  ```
</details>

<details>
  <summary>Traditional Epo</summary>

  ```python wrap theme={null}
  # from airflow.operators.python import BranchExternalPythonOperator
  # import os

  def my_isolated_function():
      import pandas as pd
      import random
      print(f"The pandas version in the virtual env is: {pd.__version__}")

      num = random.randint(0, 100)

      if num > 50:
          # return the task_id of the downstream task that should be executed
          return "downstream_task_a"  
      else:
          return "downstream_task_b"

  my_isolated_task = BranchExternalPythonOperator(
      task_id="my_isolated_task",
      python_callable=my_isolated_function,
      python=os.environ["ASTRO_PYENV_epo_pyenv"]
  )
  ```
</details>

<details>
  <summary>TaskFlow Venv</summary>

  ```python wrap theme={null}
  # from airflow.decorators import task

  @task.branch_virtualenv(requirements=["pandas==1.5.3"])
  def my_isolated_task():
      import pandas as pd
      import random
      print(f"The pandas version in the virtual env is: {pd.__version__}")

      num = random.randint(0, 100)

      if num > 50:
          # return the task_id of the downstream task that should be executed
          return "downstream_task_a"
      else:
          return "downstream_task_b"
  ```
</details>

<details>
  <summary>Traditional Venv</summary>

  ```python wrap theme={null}
  # from airflow.operators.python import BranchPythonVirtualenvOperator

  def my_isolated_function():
      import pandas as pd
      import random
      print(f"The pandas version in the virtual env is: {pd.__version__}")

      num = random.randint(0, 100)

      if num > 50:
          # return the task_id of the downstream task that should be executed
          return "downstream_task_a"  
      else:
          return "downstream_task_b"

  my_isolated_task = BranchPythonVirtualenvOperator(
      task_id="my_isolated_task",
      python_callable=my_isolated_function,
      requirements=["pandas==1.5.1"],
  )
  ```
</details>

## Use Airflow context variables in isolated environments

Some variables from the [Airflow context](/docs/learn/airflow-context) can be passed to isolated environments, for example the `logical_date` of the DAG run. Due to compatibility issues, other objects from the context such as `ti` can't be passed to isolated environments. For more information, see the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#id1).

<details>
  <summary>TaskFlow Epo</summary>

  ```python wrap theme={null}
  # from airflow.decorators import task
  # import os

  # note that to be able to use the logical date, pendulum needs to be installed in the epo_pyenv
  @task.external_python(python=os.environ["ASTRO_PYENV_epo_pyenv"])
  def my_isolated_task(logical_date): 
      print(f"The logical date is: {logical_date}")
      # your code to run in the isolated environment 

  my_isolated_task()
  ```
</details>

<details>
  <summary>Traditional Epo</summary>

  ```python wrap theme={null}
  # from airflow.operators.python import ExternalPythonOperator
  # import os

  def my_isolated_function(logical_date_from_op_kwargs):
      print(f"The logical date is: {logical_date_from_op_kwargs}")
      # your code to run in the isolated environment 

  my_isolated_task = ExternalPythonOperator(
      task_id="my_isolated_task",
      python_callable=my_isolated_function,
      # note that to be able to use the logical date, pendulum needs to be installed in the epo_pyenv
      python=os.environ["ASTRO_PYENV_epo_pyenv"],
      op_kwargs={
          "logical_date_from_op_kwargs": "{{ logical_date }}",
      },
  )
  ```
</details>

<details>
  <summary>TaskFlow Venv</summary>

  ```python wrap theme={null}
  # from airflow.decorators import task

  @task.virtualenv(
      requirements=[
          "pandas==1.5.1",
          "pendulum==3.0.0",
      ],  # pendulum is needed to use the logical date
  )
  def my_isolated_task(logical_date):
      print(f"The logical date is: {logical_date}")
      # your code to run in the isolated environment
  ```
</details>

<details>
  <summary>Traditional Venv</summary>

  ```python wrap theme={null}
  # from airflow.operators.python import PythonVirtualenvOperator

  def my_isolated_function(logical_date_from_op_kwargs):
      print(f"The logical date is: {logical_date_from_op_kwargs}")
      # your code to run in the isolated environment 

  my_isolated_task = PythonVirtualenvOperator(
      task_id="my_isolated_task",
      python_callable=my_isolated_function,
      requirements=[
          "pandas==1.5.1",
          "pendulum==3.0.0",
      ],  # pendulum is needed to use the logical date
      op_kwargs={
          "logical_date_from_op_kwargs": "{{ logical_date }}"
      },
  )
  ```
</details>

## Use Airflow variables in isolated environments

You can inject Airflow variables into isolated environments by using [Jinja templating](/docs/learn/templating) in the `op_kwargs` argument of the `PythonVirtualenvOperator` or `ExternalPythonOperator`. This strategy lets you pass secrets into your isolated environment, which are masked in the logs according to rules described in [Hide sensitive information in Airflow variables](/docs/learn/airflow-variables#hide-sensitive-information-in-airflow-variables).

<details>
  <summary>Traditional Venv</summary>

  ```python wrap theme={null}
  # from airflow.operators.python import PythonVirtualenvOperator

  def my_isolated_function(password_from_op_kwargs):
      print(f"The password is: {password_from_op_kwargs}")

  my_isolated_task = PythonVirtualenvOperator(
      task_id="my_isolated_task",
      python_callable=my_isolated_function,
      requirements=["pandas==1.5.1"],
      python_version="3.10",
      op_kwargs={
          "password_from_op_kwargs": "{{ var.value.my_secret }}",
      },
  )
  ```
</details>

<details>
  <summary>Traditional Epo</summary>

  ```python wrap theme={null}
  # from airflow.operators.python import ExternalPythonOperator
  # import os

  def my_isolated_function(password_from_op_kwargs):
      print(f"The password is: {password_from_op_kwargs}")

  my_isolated_task = ExternalPythonOperator(
      task_id="my_isolated_task",
      python_callable=my_isolated_function,
      python=os.environ["ASTRO_PYENV_epo_pyenv"],
      op_kwargs={
          "password_from_op_kwargs": "{{ var.value.my_secret }}",
      },
  )
  ```
</details>

## Use Airflow packages in isolated environments

<Warning>
  Using Airflow packages inside of isolated environments can lead to unexpected behavior and isn't recommended.
</Warning>

If you need to use Airflow or an Airflow provider module inside your virtual environment, use the `@task.virtualenv` decorator or the `PythonVirtualenvOperator` instead of the `@task.external_python` decorator or the `ExternalPythonOperator`.
As of Airflow 2.8, you can cache the virtual environment for reuse by providing a `venv_cache_path` to the `@task.virtualenv` decorator or `PythonVirtualenvOperator`, to speed up subsequent runs of your task.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # from airflow.decorators import task

  @task.virtualenv(
      requirements=[
          "apache-airflow-providers-snowflake==5.3.0",
          "apache-airflow==2.8.1",
          "pandas==1.5.3",
      ],
      venv_cache_path="/tmp/venv_cache",  # optional caching of the virtual environment
  )
  def my_isolated_task():
      from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
      import pandas as pd

      hook = SnowflakeHook(snowflake_conn_id="MY_SNOWFLAKE_CONN_ID")
      result = hook.get_first("SELECT * FROM MY_TABLE LIMIT 1")
      print(f"The pandas version in the virtual env is: {pd.__version__}")

      return result

  my_isolated_task()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # from airflow.operators.python import PythonVirtualenvOperator

  def my_isolated_function():
      from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
      import pandas as pd

      hook = SnowflakeHook(snowflake_conn_id="MY_SNOWFLAKE_CONN_ID")
      result = hook.get_first("SELECT * FROM MY_TABLE LIMIT 1")
      print(f"The pandas version in the virtual env is: {pd.__version__}")

      return result

  my_isolated_task = PythonVirtualenvOperator(
      task_id="my_isolated_task",
      python_callable=my_isolated_function,
      requirements=[
          "pandas==1.5.3",
          "apache-airflow==2.8.1",
          "apache-airflow-providers-snowflake==5.3.0",
      ],
      venv_cache_path="/tmp/venv_cache",  # optional caching of the virtual environment
  )
  ```
</details>


# Use Apache Kafka with Apache Airflow
Source: https://astronomer.io/docs/learn/airflow-kafka

How to produce to and consume from Kafka topics using the Kafka Airflow provider

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Apache Kafka](https://kafka.apache.org/documentation/) is an open source tool for handling event streaming. Combining Kafka and Airflow allows you to build powerful pipelines that integrate streaming data with batch processing.
In this tutorial, you'll learn how to install and use the [Kafka Airflow provider](https://airflow.apache.org/registry/providers/apache-kafka/) to interact directly with Kafka topics.

<Warning>
  While it is possible to manage a Kafka cluster with Airflow, be aware that Airflow itself shouldn't be used for streaming or low-latency processes. See the [Best practices](#best-practices) section for more information.
</Warning>

## Time to complete

This tutorial takes approximately 1 hour to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of Apache Kafka. See the official [Introduction to Kafka](https://kafka.apache.org/intro).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).

## Quickstart

If you have a GitHub account, you can use the [quickstart repository](https://github.com/astronomer/airflow-kafka-quickstart) for this tutorial, which automatically starts up Airflow, initiates a local Kafka cluster, and configures all necessary connections. Clone the quickstart repository and then skip to [Step 6: Run the DAGs](#step-6-run-the-dags).

## Prerequisites

* A Kafka cluster with a topic. This tutorial uses a cluster hosted by [Confluent Cloud](https://www.confluent.io/), which has a free trial option. See the [Confluent documentation](https://developer.confluent.io/quickstart/kafka-on-confluent-cloud/) for how to create a Kafka cluster and topic in Confluent Cloud.
* The [Astro CLI](/docs/cli/v1.43/get-started-cli).

<Info>
  To connect a [local Kafka cluster](https://kafka.apache.org/documentation/#quickstart) to an Airflow instance running in Docker, set the following properties in your Kafka cluster's `server.properties` file before starting your Kafka cluster:

  ```text wrap theme={null}
  listeners=PLAINTEXT://:9092,DOCKER_HACK://:19092
  advertised.listeners=PLAINTEXT://localhost:9092,DOCKER_HACK://host.docker.internal:19092
  listener.security.protocol.map=PLAINTEXT:PLAINTEXT,DOCKER_HACK:PLAINTEXT
  ```

  You can learn more about connecting to local Kafka from within a Docker container in [Confluent's Documentation](https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/#scenario-5).
</Info>

## Step 1: Configure your Astro project

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-kafka-tutorial && cd astro-kafka-tutorial
   $ astro dev init
   ```

2. Add the following packages to your `packages.txt` file:

   ```text wrap theme={null}
   build-essential
   librdkafka-dev
   ```

3. Add the following packages to your `requirements.txt` file:

   ```text wrap theme={null}
   confluent-kafka==2.1.1
   apache-airflow-providers-apache-kafka==1.0.0
   ```

4. Run the following command to start your project in a local environment:

   ```sh wrap theme={null}
   astro dev start
   ```

## Step 2: Create two Kafka connections

The Kafka Airflow provider uses a Kafka connection assigned to the `kafka_conn_id` parameter of each operator to interact with a Kafka cluster. For this tutorial you define two Kafka connections, because two different consumers will be created.

1. In your web browser, go to `localhost:8080` to access the Airflow UI.

2. Click **Admin** > **Connections** > **+** to create a new connection.

3. Name your connection `kafka_default` and select the **Apache Kafka** connection type. Provide the details for the connection to your Kafka cluster as JSON in the **Extra** field.

   If you connect to a local Kafka cluster created with the `server.properties` in the info box from the [Prerequisites](#prerequisites) section, use the following configuration:

   ```json wrap theme={null}
   {
       "bootstrap.servers": "kafka:19092",
       "group.id": "group_1",
       "security.protocol": "PLAINTEXT",
       "auto.offset.reset": "beginning"
   }
   ```

   The key-value pairs for your connection depend on what kind of Kafka cluster you are connecting to. Most operators in the Kafka Airflow provider mandate that you define the `bootstrap.servers` key. You can find a full list of optional connection parameters in the [librdkafka documentation](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION).

4. Click **Save**.

5. Create a second new connection.

6. Name your second connection `kafka_listener` and select the `Apache Kafka` connection type. Provide the same details as you did in Step 2, but set the `group.id` to `group_2`. You must have a second connection with a different `group.id` because the DAGs in this tutorial have two consuming tasks that consume messages from the same Kafka topic. Learn more in [Kafka's Consumer Configs documentation](https://kafka.apache.org/documentation/#consumerconfigs).

7. Click **Save**.

## Step 3: Create a DAG with a producer and a consumer task

The [Kafka Airflow provider package](https://airflow.apache.org/registry/providers/apache-kafka/) contains a `ProduceToTopicOperator`, which you can use to produce messages directly to a Kafka topic, and a `ConsumeFromTopicOperator`, which you can use to directly consume messages from a topic.

1. Create a new file in your `dags` folder called `produce_consume_treats.py`.

2. Copy and paste the following code into the `produce_consume_treats.py` file:

   ```python expandable wrap theme={null}
   """
   ### DAG which produces to and consumes from a Kafka cluster

   This DAG will produce messages consisting of several elements to a Kafka cluster and consume
   them.
   """

   from airflow.decorators import dag, task
   from pendulum import datetime
   from airflow.providers.apache.kafka.operators.produce import ProduceToTopicOperator
   from airflow.providers.apache.kafka.operators.consume import ConsumeFromTopicOperator
   import json
   import random

   YOUR_NAME = "<your name>"
   YOUR_PET_NAME = "<your (imaginary) pet name>"
   NUMBER_OF_TREATS = 5
   KAFKA_TOPIC = "my_topic"


   def prod_function(num_treats, pet_name):
       """Produces `num_treats` messages containing the pet's name, a randomly picked
       pet mood post treat and whether or not it was the last treat in a series."""

       for i in range(num_treats):
           final_treat = False
           pet_mood_post_treat = random.choices(
               ["content", "happy", "zoomy", "bouncy"], weights=[2, 2, 1, 1], k=1
           )[0]
           if i + 1 == num_treats:
               final_treat = True
           yield (
               json.dumps(i),
               json.dumps(
                   {
                       "pet_name": pet_name,
                       "pet_mood_post_treat": pet_mood_post_treat,
                       "final_treat": final_treat,
                   }
               ),
           )


   def consume_function(message, name):
       "Takes in consumed messages and prints its contents to the logs."

       key = json.loads(message.key())
       message_content = json.loads(message.value())
       pet_name = message_content["pet_name"]
       pet_mood_post_treat = message_content["pet_mood_post_treat"]
       print(
           f"Message #{key}: Hello {name}, your pet {pet_name} has consumed another treat and is now {pet_mood_post_treat}!"
       )


   @dag(
       start_date=datetime(2023, 4, 1),
       schedule=None,
       catchup=False,
       render_template_as_native_obj=True,
   )
   def produce_consume_treats():
       @task
       def get_your_pet_name(pet_name=None):
           return pet_name

       @task
       def get_number_of_treats(num_treats=None):
           return num_treats

       @task
       def get_pet_owner_name(your_name=None):
           return your_name

       produce_treats = ProduceToTopicOperator(
           task_id="produce_treats",
           kafka_config_id="kafka_default",
           topic=KAFKA_TOPIC,
           producer_function=prod_function,
           producer_function_args=["{{ ti.xcom_pull(task_ids='get_number_of_treats')}}"],
           producer_function_kwargs={
               "pet_name": "{{ ti.xcom_pull(task_ids='get_your_pet_name')}}"
           },
           poll_timeout=10,
       )

       consume_treats = ConsumeFromTopicOperator(
           task_id="consume_treats",
           kafka_config_id="kafka_default",
           topics=[KAFKA_TOPIC],
           apply_function=consume_function,
           apply_function_kwargs={
               "name": "{{ ti.xcom_pull(task_ids='get_pet_owner_name')}}"
           },
           poll_timeout=20,
           max_messages=20,
           max_batch_size=20,
       )

       [
           get_your_pet_name(YOUR_PET_NAME),
           get_number_of_treats(NUMBER_OF_TREATS),
       ] >> produce_treats
       get_pet_owner_name(YOUR_NAME) >> consume_treats

       produce_treats >> consume_treats


   produce_consume_treats()
   ```

   This DAG produces messages to a Kafka topic (`KAFKA_TOPIC`) and consumes them.

   * The `produce_treats` task retrieves the number of treats (`num_treats`) to give to your pet from the upstream `get_number_of_treats` task. Then, the task supplies the number of treats to the `producer_function` as a positional argument with the `producer_function_args` parameter.
     In a similar process, the task also retrieves the name of your pet from the upstream `get_your_pet_name` task and provides it as a kwarg to `producer_function_kwargs`.
   * Next, the `produce_treats` task writes one message for every treat to a Kafka topic. Each message contains the pet name, a randomly picked pet mood after the treat has been given, and whether or not a treat was the last one in a series. The `ProduceToTopicOperator` accomplishes this by using a function passed to its `producer_function` parameter, which returns a generator containing key-value pairs.
   * The `consume_treats` task consumes messages from the same Kafka topic and modifies them to print a string to the logs using the callable provided to the `apply_function` parameter. This task also retrieves a value from an upstream task and supplies it as a kwarg to the `apply_function` with the `apply_function_kwargs` parameter.

3. Navigate to the Airflow UI (`localhost:8080` if you are running Airflow locally) and manually run your DAG.

4. View the produced events in your Kafka cluster. The following example screenshot shows four messages that have been produced to a topic called `test_topic_1` in Confluent Cloud.

   <Frame>
     <img alt="Producer logs" />
   </Frame>

5. View the logs of your `consume_treats` task, which shows a list of the consumed events.

   <Frame>
     <img alt="Consumer logs" />
   </Frame>

<Info>
  If you defined a schema for your Kafka topic, the generator needs to return compatible objects. In this example, the generator produces a JSON value.
</Info>

<Tip>
  The `ConsumeFromTopicOperator` can replace classical sinks by containing the logic to write messages to a storage destination in its `apply_function`. This gives you the advantage of being able to use Airflow to schedule message consumption from a Kafka topic based on complex logic embedded in your wider data ecosystem. For example, you can write messages to S3 using the [`S3CreateObjectOperator`](https://airflow.apache.org/registry/providers/amazon#amazon-s3-S3CreateObjectOperator), which depends on other upstream task having completed successfully, such as the creation of a specific S3 bucket.
</Tip>

## Step 4: Create a listener DAG

Airflow can run a function when a specific message appears in your Kafka topic. The `AwaitMessageTriggerFunctionSensor` is a [deferrable operator](/docs/learn/deferrable-operators) that listens to your Kafka topic for a message that fulfills specific criteria, which, when met, runs the callable provided to `event_triggered_function`. The `TriggerDagRunOperator` can be used within the `event_triggered_function` to initiate a run of a downstream DAG.

1. Create a new file in your `dags` folder called `listen_to_the_stream.py`.

2. Copy and paste the following code into the file:

   ```python expandable wrap theme={null}
   """
   ### DAG continuously listening to a Kafka topic for a specific message

   This DAG will always run and asynchronously monitor a Kafka topic for a message
   which causes the funtion supplied to the `apply_function` parameter to return a value.
   If a value is returned by the `apply_function`, the `event_triggered_function` is
   executed. Afterwards the task will go into a deferred state again.
   """

   from airflow.decorators import dag
   from pendulum import datetime
   from airflow.providers.apache.kafka.sensors.kafka import (
       AwaitMessageTriggerFunctionSensor,
   )
   from airflow.operators.trigger_dagrun import TriggerDagRunOperator
   import json
   import uuid

   PET_MOODS_NEEDING_A_WALK = ["zoomy", "bouncy"]
   KAFKA_TOPIC = "my_topic"


   def listen_function(message, pet_moods_needing_a_walk=[]):
       """Checks if the message received indicates a pet is in
       a mood listed in `pet_moods_needing_a_walk` when they received the last
       treat of a treat-series."""

       message_content = json.loads(message.value())
       print(f"Full message: {message_content}")
       pet_name = message_content["pet_name"]
       pet_mood_post_treat = message_content["pet_mood_post_treat"]
       final_treat = message_content["final_treat"]
       if final_treat:
           if pet_mood_post_treat in pet_moods_needing_a_walk:
               return pet_name, pet_mood_post_treat


   def event_triggered_function(event, **context):
       "Kicks off a downstream DAG with conf and waits for its completion."

       pet_name = event[0]
       pet_mood_post_treat = event[1]
       print(
           f"Due to {pet_name} being in a {pet_mood_post_treat} mood, a walk is being initiated..."
       )
       # use the TriggerDagRunOperator (TDRO) to kick off a downstream DAG
       TriggerDagRunOperator(
           trigger_dag_id="walking_my_pet",
           task_id=f"triggered_downstream_dag_{uuid.uuid4()}",
           wait_for_completion=True,  # wait for downstream DAG completion
           conf={"pet_name": pet_name},
           poke_interval=5,
       ).execute(context)

       print(f"The walk has concluded and {pet_name} is now happily taking a nap!")


   @dag(
       start_date=datetime(2023, 4, 1),
       schedule="@continuous",
       max_active_runs=1,
       catchup=False,
       render_template_as_native_obj=True,
   )
   def listen_to_the_stream():
       listen_for_mood = AwaitMessageTriggerFunctionSensor(
           task_id="listen_for_mood",
           kafka_config_id="kafka_listener",
           topics=[KAFKA_TOPIC],
           # the apply function will be used from within the triggerer, this is
           # why it needs to be a dot notation string
           apply_function="listen_to_the_stream.listen_function",
           poll_interval=5,
           poll_timeout=1,
           apply_function_kwargs={"pet_moods_needing_a_walk": PET_MOODS_NEEDING_A_WALK},
           event_triggered_function=event_triggered_function,
       )


   listen_to_the_stream()
   ```

   This DAG has one task called `listen_for_mood` which uses the `AwaitMessageTriggerFunctionSensor` to listen to messages in all topics supplied to its `topics` parameters. For each message that is consumed, the following actions are performed:

   * The `listen_function` supplied to the `apply_function` parameter of the `AwaitMessageTriggerFunctionSensor` consumes and processes the message. The `listen_function` is provided as a dot notation string, which is necessary because the Airflow triggerer component needs to access this function.
   * If the message consumed causes the `listen_function` to return a value, a [TriggerEvent](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html) fires.
   * After a TriggerEvent fires, the `AwaitMessageTriggerFunctionSensor` executes the function provided to the `event_triggered_function` parameter.
     In this example, the `event_triggered_function` starts a downstream DAG using the `.execute()` method of the [`TriggerDagRunOperator`](/docs/learn/cross-dag-dependencies#triggerdagrunoperator).
   * After the `event_triggered_function` completes, the `AwaitMessageTriggerFunctionSensor` returns to a deferred state.

   The `AwaitMessageTriggerFunctionSensor` always runs and listens. If the task fails, like if a malformed message is consumed, the DAG completes as `failed` and automatically starts its next DAG run because of the [`@continuous` schedule](/docs/learn/scheduling-in-airflow#continuous-timetable).

<Info>
  When working locally, you need to restart your Airflow instance to apply changes to the `apply_function` of the `AwaitMessageTriggerFunctionSensor` because the function is imported into the [Triggerer](/docs/learn/deferrable-operators#terms-and-concepts), which doesn't periodically restart. To restart Airflow, run `astro dev restart` in your terminal. Changes to the `event_triggered_function` of the `AwaitMessageTriggerFunctionSensor` don't require a restart of your Airflow instance.

  On Astro, the Triggerer is restarted automatically when a new image is deployed, but not on dag-only deploys, see [Deploy DAGs to Astro](/docs/astro/deploy-dags).
</Info>

## Step 5: Create a downstream DAG

The `event_triggered_function` of the `AwaitMessageTriggerFunctionSensor` operator starts a downstream DAG. This example shows how to implement a dependency based on messages that appear in your Kafka topic.

1. Create a new file in your `dags` folder called `walking_my_pet.py`.

2. Copy and paste the following code into the file:

   ```python wrap theme={null}
   """
   ### Simple DAG that runs one task with params

   This DAG uses one string type param and uses it in a python decorated task.
   """

   from airflow.decorators import dag, task
   from pendulum import datetime
   from airflow.models.param import Param
   import random


   @dag(
       start_date=datetime(2023, 4, 1),
       schedule=None,
       catchup=False,
       render_template_as_native_obj=True,
       params={"pet_name": Param("Undefined!", type="string")},
   )
   def walking_my_pet():
       @task
       def walking_your_pet(**context):
           pet_name = context["params"]["pet_name"]
           minutes = random.randint(2, 10)
           print(f"{pet_name} has been on a {minutes} minute walk!")

       walking_your_pet()


   walking_my_pet()
   ```

   This DAG acts as a downstream dependency to the `listen_to_the_stream` DAG. You can add any tasks to this DAG.

## Step 6: Run the DAGs

Now that all three DAGs are ready, run them to see how they work together.

1. Make sure you unpause all DAGs in the Airflow UI and that your Kafka cluster is running.

2. The `listen_to_the_stream` DAG immediately starts running after it unpauses and the `listen_for_mood` task goes into a **Deferred** state, which is indicated with a purple square in the Airflow UI.

   <Frame>
     <img alt="Kafka deferred state" />
   </Frame>

3. Manually run the `produce_consume_treats` DAG to give your pet some treats and produce a few messages to the Kafka cluster.

4. Check the logs of the `listen_for_mood` task in the `listen_to_the_stream` DAG to see if a message fitting the criteria defined by the `listen_function` has been detected. You might need to run the `produce_consume_treats` DAG a couple of times for a message to appear.

   If the TriggerEvent of the `listen_for_mood` task fires, the `listen_for_mood` task logs show the `walking_my_pet` DAG initiating.

   <Frame>
     <img alt="Kafka logs TDRO" />
   </Frame>

5. Finally, check the logs of the `walking_my_pet` task to see how long your pet enjoyed their walk!

## Best practices

Apache Kafka is a tool optimized for streaming messages at high frequencies, for example in an IoT application. Airflow is designed to handle orchestration of data pipelines in batches.

Astronomer recommends to combine these two open source tools by handling low-latency processes with Kafka and data orchestration with Airflow.

Common patterns include:

* Configuring a Kafka cluster with a blob storage like S3 as a sink. Batch process data from S3 at regular intervals.
* Using the `ProduceToTopicOperator` in Airflow to produce messages to a Kafka cluster as one of several producers.
* Consuming data from a Kafka cluster through the `ConsumeFromTopicOperator` in batches using the apply function to extract and load information to a blob storage or data warehouse.
* Listening for specific messages in a data stream running through a Kafka cluster using the `AwaitMessageTriggerFunctionSensor` to trigger downstream tasks after the message appears.

## Conclusion

Congratulations! You used the Kafka Airflow provider to directly interact with a Kafka topic from within Apache Airflow.


# Use a listener to send a Slack notification when a dataset is updated
Source: https://astronomer.io/docs/learn/airflow-listeners

Learn how to use Airflow listeners.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Airflow listeners](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/listeners.html#listeners) allow you to execute custom code when certain events occur anywhere in your Airflow instance, for example when any DAG run fails or any dataset is updated.

Listeners are implemented as an [Airflow plugin](/docs/learn/using-airflow-plugins) and can contain any code. In this tutorial, you'll use a listener to send a Slack notification whenever any dataset is updated.

<Info>
  If you only need to implement notifications for specific DAGs and tasks, consider using [Airflow callbacks](/docs/learn/error-notifications-in-airflow#airflow-callbacks) instead.
</Info>

<Warning>
  The `on_dataset_created` and `on_dataset_changed` listeners are currently considered experimental and might be subject to breaking changes in future releases.
</Warning>

## Time to complete

This tutorial takes approximately 15 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow plugins. See [Airflow plugins](/docs/learn/using-airflow-plugins).
* Airflow datasets. See [Datasets and data-aware scheduling in Airflow](/docs/learn/airflow-datasets).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli) using [Astro Runtime](/docs/runtime/runtime-release-notes) 10+ (Airflow 2.8+).
* A Slack workspace with an [Incoming Webhook](https://api.slack.com/messaging/webhooks) configured.

## Step 1: Configure your Astro project

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-listener-tutorial && cd astro-listener-tutorial
   $ astro dev init
   ```

2. Add the following line to your Astro project `requirements.txt` file to install the Slack Airflow provider.

   ```text wrap theme={null}
   apache-airflow-providers-slack==8.4.0
   ```

3. Add the following environment variable to your Astro project `.env` file to create an [Airflow connection](/docs/learn/connections) to Slack. Make sure to replace `<your-slack-webhook-token>` with your own Slack webhook token in the format of `T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX`.

   ```text wrap theme={null}
   AIRFLOW_CONN_SLACK_WEBHOOK_CONN='{
       "conn_type": "slackwebhook",
       "host": "https://hooks.slack.com/services/",
       "password": "<your-slack-webhook-token>"
   }'
   ```

## Step 2: Create your listener

To define an Airflow listener, you add the code you want to execute to a relevant `@hookimpl`-decorated [listener function](https://github.com/apache/airflow/tree/main/airflow-core/src/airflow/listeners/spec). In this example, you define your code in the `on_dataset_changed` function to run whenever any dataset is updated.

1. Create a new file called `listeners_code.py` in your `plugins` folder.
2. Copy the following code into the file:

```python wrap theme={null}
from airflow.datasets import Dataset
from airflow.listeners import hookimpl
from airflow.models.taskinstance import TaskInstance
from airflow.utils.state import TaskInstanceState
from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
from sqlalchemy.orm.session import Session
from datetime import datetime

SLACK_CONN_ID = "slack_webhook_conn"

@hookimpl
def on_dataset_changed(dataset: Dataset):
    """Execute if a dataset is updated."""
    print("I am always listening for any Dataset changes and I heard that!")
    print("Posting to Slack...")
    hook = SlackWebhookHook(slack_webhook_conn_id=SLACK_CONN_ID)
    hook.send(text=f"A dataset was changed!")
    print("Done!")
    if dataset.uri == "file://include/bears":
        print("Oh! This is the bears dataset!")
        print("Bears are great :)")
        start_date = datetime.now().date()
        end_date = datetime(2024, 10, 4).date()
        days_until = (end_date - start_date).days
        print(f"Only approximately {days_until} days until fat bear week!")
```

This listener is defined using the [`on_dataset_changed` hookspec](https://github.com/apache/airflow/blob/v2-10-stable/airflow/listeners/spec/dataset.py). It posts a message to Slack whenever any dataset is updated and executes an additional print statement if the dataset that is being updated has the URI `file://include/bears`.

## Step 3: Create the listener plugin

For Airflow to recognize your listener, you need to create a [plugin](/docs/learn/using-airflow-plugins) that registers it.

1. Create a new file called `listener_plugin.py` in your `plugins` folder.

2. Copy the following code into the file:

   ```python wrap theme={null}
   from airflow.plugins_manager import AirflowPlugin
   from plugins import listeners_code

   class MyListenerPlugin(AirflowPlugin):
       name = "my_listener_plugin"
       listeners = [listeners_code]
   ```

3. If your local Airflow environment is already running, restart it to apply the changes to your plugins.

## Step 4: Create your DAG

1. In your `dags` folder, create a file called `producer_dag.py`.

2. Copy the following code into the file.

   ```python expandable wrap theme={null}
   """
   ## DAG to produce to a Dataset showcasing the on_dataset_changed listener

   This DAG will produce to a Dataset, updating it which triggers the
   on_dataset_changed listener define as an Airflow Plugin.

   The DAG also shows the difference between a Dataset and ObjectStoragePath.
   """

   from airflow.datasets import Dataset
   from airflow.decorators import dag, task
   from airflow.io.path import ObjectStoragePath
   from pendulum import datetime
   import requests


   URI = "file://include/bears"
   MY_DATASET = Dataset(URI)
   base_local = ObjectStoragePath(URI)


   @dag(
       start_date=datetime(2023, 12, 1),
       schedule="0 0 * * 0",
       catchup=False,
       doc_md=__doc__,
       tags=["on_dataset_changed listener", "2-8"],
   )
   def producer_dag():
       @task(
           outlets=[MY_DATASET],
       )
       def get_bear(base):
           r = requests.get("https://placebear.com/200/300")
           file_path = base / "bear.jpg"

           if r.status_code == 200:
               base.mkdir(parents=True, exist_ok=True)
               file_path.write_bytes(r.content)
               file_path.replace("bear.jpg")
           else:
               print(f"Failed to retrieve image. Status code: {r.status_code}")

       get_bear(base=base_local)


   producer_dag()
   ```

   This simple DAG contains one task that queries the [placebear](https://placebear.com/) API and writes the image retrieved to a local `.png` file in the `include` folder using the [Airflow object storage](/docs/learn/airflow-object-storage-tutorial) feature.
   The task produces an update to the `file://include/bears` dataset, which triggers the listener you created in [Step 2](#step-2-create-your-listener).

## Step 5: Run your DAG

1. Run `astro dev start` in your Astro project to start Airflow, then open the Airflow UI at `localhost:8080`.

2. In the Airflow UI, run the `producer_dag` DAG by clicking the play button.

3. After the DAG run completed, go to the task logs of the `get_bear` task to see print statements from your listener plugin.

   ```text wrap theme={null}
   [2023-12-17, 14:46:51 UTC] {logging_mixin.py:188} INFO - I am always listening for any dataset changes and I heard that!
   [2023-12-17, 14:46:51 UTC] {logging_mixin.py:188} INFO - Posting to Slack...
   [2023-12-17, 14:46:51 UTC] {base.py:83} INFO - Using connection ID 'slack_webhook_conn' for task execution.
   [2023-12-17, 14:46:51 UTC] {logging_mixin.py:188} INFO - Done!
   [2023-12-17, 14:46:51 UTC] {logging_mixin.py:188} INFO - Oh! This is the bears dataset!
   [2023-12-17, 14:46:51 UTC] {logging_mixin.py:188} INFO - Bears are great :)
   [2023-12-17, 14:46:51 UTC] {logging_mixin.py:188} INFO - Only approximately 292 days until fat bear week
   ```

4. Open your Slack workspace to see a new message from your webhook.

   <Frame>
     <img alt="Screenshot of a Slack message sent by the webhook, saying &#x22;A dataset was changed!&#x22;" />
   </Frame>

5. (Optional) View your complimentary bear picture at `include/bears/bear.png`.

## Conclusion

Congratulations! You now know how to create an Airflow listener to run custom code whenever any dataset is updated in your whole Airflow environment. Following the same pattern you can implement listeners for [other events](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/listeners.html#listeners), such as when any task has failed, any DAG starts running or a lifecycle event occurs.


# Best practices for orchestrating MLOps pipelines with Airflow
Source: https://astronomer.io/docs/learn/airflow-mlops

Learn how to use Airflow to run machine learning in production.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

**Machine Learning Operations** (MLOps) is a broad term encompassing everything needed to run machine learning models in production. MLOps is a rapidly evolving field with many different best practices and behavioral patterns, with Apache Airflow providing tool agnostic orchestration capabilities for all steps.

In this guide you learn:

* How Airflow fits into the MLOps landscape.
* How Airflow can be used for large language model operations (LLMOps).
* How Airflow can help you implement best practices for different MLOps components.
* Which Airflow features and integrations are especially useful for MLOps.
* Where to find additional resources and reference implementations on using Airflow for MLOps.

<Tip>
  Ready to get started? Check out the recommended resources that showcase ML and AI implementations with Airflow:

  * [Orchestrate LLMs and Agents with Apache Airflow®](https://www.astronomer.io/ebooks/orchestrate-llms-and-agents-with-airflow/) eBook.
  * [Context Graphs with Apache Airflow®](https://www.astronomer.io/ebooks/quick-notes-context-graphs-with-apache-airflow) quick notes.
  * [AI Context Engineering with Apache Airflow®](https://www.astronomer.io/events/webinars/ai-context-engineering-with-airflow-video) webinar.
  * Our [GenAI](/docs/learn/reference-architecture-context-graph) and [MLOps](/docs/learn/reference-architecture-snowpatrol) reference architectures.
</Tip>

## Assumed knowledge

To get the most benefits from this guide, you need an understanding of:

* The basics of [Machine Learning](https://www.coursera.org/specializations/machine-learning-introduction).
* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).

## Why use Airflow for MLOps?

**Machine learning operations** (MLOps) encompasses all patterns, tools, and best practices related to running machine learning models in production.

Apache Airflow sits at the heart of the modern MLOps stack. Because it is tool agnostic, Airflow can orchestrate all actions in any MLOps tool that has an API. Combined with already being the de-facto standard for orchestrating data pipelines, Airflow is the perfect tool for data engineers and machine learning engineers to standardize their workflows and collaborate on pipelines.

The benefits of using Airflow for MLOps are:

* **Python native**: You use Python code to define Airflow pipelines, which makes it easy to integrate the most popular machine learning tools and embed your ML operations in a best practice CI/CD workflow. By using the [decorators](/docs/learn/airflow-decorators) of the TaskFlow API you can turn existing scripts into Airflow tasks.
* **Extensible**: Airflow itself is written in Python, which makes it extensible with [custom modules](/docs/learn/airflow-importing-custom-hooks-operators) and [Airflow plugins](/docs/learn/using-airflow-plugins).
* **Monitoring and alerting**: Airflow comes with production-ready monitoring and alerting modules like [Airflow notifiers](/docs/learn/error-notifications-in-airflow#custom-notifiers), [extensive logging features](/docs/learn/logging), and [Airflow listeners](/docs/learn/airflow-listeners). They enable you to have fine-grained control over how you monitor your ML operations and how Airflow alerts you if something goes wrong.
* **Pluggable compute**: When using Airflow you can [pick and choose](#modelops) the compute you want to use for each task. This allows you to use the perfect environment and resources for every single action in your ML pipeline. For example, you can run your data engineering tasks on a Spark cluster and your model training tasks on a GPU instance.
* **Data agnostic**: Airflow is data agnostic, which means it can be used to orchestrate any data pipeline, regardless of the data format or storage solution. You can plug in any new data storage, such as the latest vector database or your favorite RDBMS, with minimal effort.
* **Incremental and idempotent pipelines**: Airflow allows you to define pipelines that operate on data collected in a specified timeframe and to perform [backfills and reruns](/docs/learn/rerunning-dags#backfill) of a set of idempotent tasks. This lends itself well to creating feature stores, especially for time-dimensioned features, which form the basis of advanced model training and selection.
* **Ready for day 2 Ops**: Airflow is a mature orchestrator, coming with built-in functionality such as [automatic retries](/docs/learn/rerunning-dags#automatically-retry-tasks), complex [dependencies](/docs/learn/managing-dependencies) and [branching](/docs/learn/airflow-branch-operator) logic, as well as the option to make pipelines [dynamic](/docs/learn/dynamic-tasks).
* **Integrations**: Airflow has a large ecosystem of [integrations](https://airflow.apache.org/registry/providers/), including many popular [MLOps tools](#airflow-integrations-for-mlops).
* **Shared platform**: Both data engineers and ML engineers use Airflow, which allows teams to create direct dependencies between their pipelines, such as using [Airflow Datasets](/docs/learn/airflow-datasets).
* **Use existing expertise**: Many organizations are already using Apache Airflow for their data engineering workflows and have developed best practices and custom tooling around it. This means that data engineers and ML engineers alike can build upon existing processes and tools to orchestrate and monitor ML pipelines.

## Why use Airflow for LLMOps?

**Large Language Model Operations** (LLMOps) is a subset of MLOps that describes interactions with large language models (LLMs). In contrast to traditional ML models, LLMs are often too large to be trained from scratch and LLMOps techniques instead revolve around adapting existing LLMs to new use cases.

The three main techniques for LLMOps are:

* **Prompt engineering**: This is the simplest technique to influence the output of an LLM. You can use Airflow to create a pipeline that ingests user prompts and modifies them according to your needs, before sending them to the LLM inference endpoint.
* **Retrieval augmented generation** (RAG): RAG pipelines retrieve relevant context from domain-specific and often proprietary data to improve the output of an LLM.
* **Fine-tuning**: Fine-tuning LLMs typically involves retraining the final layers of an LLM on a specific dataset. This often requires more complex pipelines and a larger amount of [compute](#modelops) that can be orchestrated with Airflow.

## Components of MLOps

MLOps describes different patterns in how organizations can productionize machine learning models. MLOps consists of four main components:

* **BusinessOps**: the processes and activities in an organization that are needed to deliver any outcome, including successful MLOps workflows.
* **DevOps**: the software development (dev) and IT operations (ops) practices that are needed for the delivery of any high quality software, including machine learning based applications.
* **DataOps**: the practices and tools surrounding data engineering and data analytics to build the foundation for machine learning implementations.
* **ModelOps**: automated governance, management and monitoring of machine learning models in production.

Organizations are often at different stages of maturity for each of these components when starting their MLOps journey. Using Apache Airflow to orchestrate MLOps pipelines can help you progress in all of them.

### BusinessOps

The first component of MLOps is to make sure there is strategic alignment with all stakeholders. This component varies widely depending on your organization and use case and can include:

* **Business strategy**: Defining what ML is used for in an organization and what trade-offs are acceptable. Often, models can be optimized for different metrics, for example high recall or high precision, and domain experts are needed to determine the right metrics and model strategy.
* **Model governance**: Creating and following regulations for how your organization uses machine learning. This often depends on relevant regulations, like GDPR or HIPAA. Airflow has a built-in integration option with [Open Lineage](/docs/learn/airflow-openlineage), the open-source standard for tracking data lineage, which is a key component of model governance.

### DevOps

Since you define Airflow pipelines in Python code, you can apply DevOps best practices when using Airflow. This includes:

* **Version control**. All code and configuration should be stored in a version control system like [Git](https://git-scm.com/). Version control allows you to track all changes of your pipeline, ML model, and environment over time and roll back to previous versions if needed. Astro customers can take advantage of [Deployment rollbacks](/docs/astro/deploy-history).
* **Continuous integration/ continuous delivery** ([CI/CD](https://resources.github.com/ci-cd/)). It is a standard software best practice for all code to undergo automatic testing, linting, and deployment. This ensures that your code is always in a working state and that any changes are automatically deployed to production. Airflow integrates with all major CI/CD tools, see [CI/CD templates](/docs/astro/ci-cd-templates/template-overview) for popular templates.

<Info>
  Astronomer customers can use the Astro GitHub integration, which allows you to automatically deploy code from a GitHub repository to an Astro deployment, viewing Git metadata in the Astro UI. See [Deploy code with the Astro GitHub integration](/docs/astro/deploy-github-integration).
</Info>

* **Infrastructure as code** ([IaC](https://en.wikipedia.org/wiki/Infrastructure_as_code)). Ideally, all infrastructure is defined as code and follows the same CI/CD process as your pipeline and model code. This allows you to control and, if necessary, roll back environment changes, or quickly deploy new instances of your model.

In practice, following modern DevOps patterns when using Airflow for MLOps means:

* Storing all Airflow code and configuration in a version control system like Git.
* Setting up development, staging, and production branches in your version control system and also connecting them to different Airflow environments. For Astro customers, see [Manage Astro connections in branch-based deploy workflows](/docs/astro/best-practices/connections-branch-deploys).
* Use automatic testing and linting for all Airflow code before deployment.
* Define all infrastructure as code and use the same CI/CD process for infrastructure as for your Airflow code.
* Store model artifacts in a versioned system. This can be a dedicated tool like MLFlow or an object storage solution.

<Frame>
  <img alt="Diagram showing how Airflow code and configuration is stored in a version control system and deployed to different Airflow environments." />
</Frame>

### DataOps

There is no MLOps without data. You need to have robust data engineering workflows in place in order to confidently train, test, and deploy ML models in production. Apache Airflow has been used by millions of data engineers to create reliable best practice data pipelines, providing a strong foundation for your MLOps workflows.

Give special considerations to the following:

* [**Data quality**](/docs/learn/data-quality) and data cleaning. If your data is of bad quality, your model predictions will be too. Astronomer recommends incorporating data quality checks and data cleaning steps into your data pipelines to define and monitor the requirements your data has to fulfill in order for downstream ML operations to be successful. Airflow supports integration with any data quality tool that has an API, and has pre-built integrations for tools such as [Great Expectations](/docs/learn/airflow-great-expectations) and [Soda Core](/docs/learn/soda-data-quality).
* **Data preprocessing and feature engineering**. It is common for data to undergo several transformation steps before it is ready to be used as input for an ML model. These steps can include simple [preprocessing steps](https://scikit-learn.org/stable/modules/preprocessing.html) like scaling, one-hot-encoding, or imputation of missing values. It can also include more complex steps like [feature selection](https://scikit-learn.org/stable/modules/feature_selection.html#feature-selection), [dimensionality reduction](https://en.wikipedia.org/wiki/Dimensionality_reduction), or [feature extraction](https://scikit-learn.org/stable/modules/feature_extraction.html). Airflow allows you to run preprocessing and feature engineering steps in a pythonic way using [Airflow decorators](/docs/learn/airflow-decorators).
* **Data storage**.
  * Training and testing data. The best way to store your data highly depends on your data and type of ML. Data engineering includes ingesting data and moving it to the ideal platform for your ML model to access. This can, for example, be an object storage solution, a relational database management system (RDBMS), or a vector database. Airflow integrates with all these options, with tools such as [Airflow object storage](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html) simplifying common operations.
  * Model artifacts. Model artifacts include model parameters, hyperparameters, and other metadata. Airflow integrates with specialized version control systems such as [MLFlow](https://mlflow.org/) or [Weights & Biases](/docs/learn/airflow-weights-and-biases).

Apart from the foundations described earlier, second day data quality operations in MLOps often include advanced topics like data governance, [data lineage](/docs/learn/airflow-openlineage), and data cataloging as well as monitoring of data drift.

In practice, following modern data engineering patterns when using Airflow for MLOps means:

* Following general [Airflow best practices](/docs/learn/dag-best-practices) for DAG writing, such as keeping tasks atomic and idempotent.
* Using Airflow to orchestrate data ingestion from sources such as APIs, source databases, and object storage into a working location for your ML model. This might be a vector database, a relational database, or an object storage solution depending on your use case.
* Incorporating data quality checks into your Airflow pipelines, with critical data quality checks halting the pipeline or alerting you if they fail.
* Using Airflow to orchestrate data preprocessing and feature engineering steps.
* Moving data to permanent cold storage after it has been used for training and testing.

<Frame>
  <img alt="Diagram showing an example ETL/ELT pipeline for Machine Learning." />
</Frame>

### ModelOps

After you establish strong DevOps and data engineering foundations, you can start to implement model operations.

With Airflow you can use the ML tools and compute locations of your choice. Some organizations choose to use external compute for all of their heavy workloads, for example:

* **External Kubernetes clusters**: with the [`KubernetesPodOperator`](/docs/learn/kubepod-operator) (and its decorator version `@task.kubernetes`).
* **Databricks**: with the [Astro Databricks provider](https://github.com/astronomer/astro-provider-databricks) and [Databricks Airflow provider](https://airflow.apache.org/registry/providers/databricks/).
* **Spark**: with modules from the [Spark Airflow provider](https://airflow.apache.org/registry/providers/apache-spark/).
* **External compute**: in [AWS](https://airflow.apache.org/registry/providers/amazon/), [Azure](https://airflow.apache.org/registry/providers/microsoft-azure/) and [Google Cloud](https://airflow.apache.org/registry/providers/google/) using the respective Airflow providers.

Other Airflow users decide to [scale up](/docs/learn/airflow-scaling-workers) their Airflow infrastructure with larger worker nodes for compute intensive tasks. Astro customers can use the [worker queues](/docs/astro/configure-worker-queues) feature, which lets them decide the exact specifications for the workers each task can run on. This means that Airflow only uses large workers for the biggest workloads, saving compute and cost.

In practice, following modern model operations patterns when using Airflow for MLOps means:

* Performing exploratory data analysis and test potential ML applications on small subsets of data in a notebook environment before moving to production. Tools like [Jupyter notebooks](https://jupyter.org/) are widely used and run Python code you can later convert to tasks in Airflow DAGs.
* Using Airflow to orchestrate model training, fine-tuning, testing, and deployment.
* Having Airflow tasks monitor the performance of your model and perform automated actions such as re-training, re-deploying, or alerting if the performance drops below a certain threshold.

<Frame>
  <img alt="Diagram showing different Airflow DAGs relating to model operations in an MLOps pipeline" />
</Frame>

## How Airflow addresses MLOps challenges

When using Apache Airflow for MLOps, there are three main patterns you can follow:

* Using Apache Airflow to orchestrate actions in other MLOps tools. Airflow is a tool-agnostic orchestrator, which means it can also orchestrate all actions in ML specific tools like [MLFlow](https://mlflow.org/) or [AWS SageMaker](/docs/learn/airflow-sagemaker).
* Combine orchestration of actions in other tools with ML operations running within Airflow. For example, you can create vector embeddings in a Python function in Airflow and then use these embeddings to train a model in [Google Datalab](https://cloud.google.com/monitoring/datalab/set-up-datalab). Modules like the [`@task.kubernetes`](/docs/learn/kubepod-operator#use-the-@task-kubernetes-decorator) or [`@task.external_python_operator`](/docs/learn/airflow-isolated-environments) make it easy to run any Python code in isolation with optimized environments and resources.
* Run all your MLOps using Python modules inside Airflow tasks. Since Airflow can run any Python code and [scale indefinitely](/docs/learn/airflow-scaling-workers), you can use it as an all-purpose MLOps tool.

### Airflow features for MLOps

A specific set of Airflow features can help you implement MLOps best practices:

* [Data driven scheduling](/docs/learn/airflow-datasets): With Airflow datasets, you can schedule DAGs to run after a specific dataset is updated by any task in the DAG. For example, you can schedule your model training DAG to run after the training dataset is updated by the data engineering DAG. See [Orchestrate machine learning pipelines with Airflow datasets](/docs/learn/2.x/use-case-airflow-datasets-multi-team-ml).

  <Frame>
    <img alt="Screenshot of the Datasets view showing the dataset_de_dag as the producing DAG to the postgres://prodserver:2319/trainset dataset. The dataset_ml_dag is the consuming DAG." />
  </Frame>

* [Dynamic task mapping](/docs/learn/dynamic-tasks): In Airflow, you can map tasks and task groups dynamically at runtime. This allows you to run similar operations in parallel without knowing in advance how many operations run in any given DAGrun. For example, you can dynamically run a set of model training tasks in parallel, each with different hyperparameters.

  <Frame>
    <img alt="DAG with a dynamically mapped task group training and evaluating a model with different dynamically changing sets of hyperparameters, then selecting and deploying the best model." />
  </Frame>

* [Setup and teardown](/docs/learn/airflow-setup-teardown): Airflow allows you to define setup and teardown tasks which create and remove resources used for machine learning. This extends the concept of infrastructure as code to your ML environment, by making the exact state of your environment for a specific ML operation reproducible.

  <Frame>
    <img alt="DAG with setup/ teardown tasks creating and tearing down an ML cluster." />
  </Frame>

* [Branching](/docs/learn/airflow-branch-operator): Airflow allows you to branch your DAG based on the outcome of a task. You can use this to create different paths in your DAG based on the outcome of a task. For example, you can branch your DAG based on the performance of a model on a test set and only deploy the model if it performs above a certain threshold.

  <Frame>
    <img alt="DAG with a branching task deciding whether or not a model is retrained and redeployed." />
  </Frame>

* [Alerts and Notifications](/docs/learn/error-notifications-in-airflow): Airflow has a wide variety of options to alert you of events in your pipelines, such as DAG or task failures. It is a best practice to set up alerts for critical events in your ML pipelines, such as a drop in model performance or a data quality check failure. Astronomer customers can use [Astro Alerts](/docs/astro/alerts).

  <Frame>
    <img alt="Screenshot of an Astro alert in a Slack channel." />
  </Frame>

* [Automatic retries](/docs/learn/rerunning-dags#automatically-retry-tasks): Airflow allows you to configure tasks to automatically retry if they fail according to custom set delays. This feature is critical to protect your pipeline against outages of external tools or rate limits and can be configured at the global, DAG, or the individual task level.

* [Backfills and Reruns](/docs/learn/rerunning-dags#backfill): In Airflow, you can rerun previous DAG runs and create backfill DAG runs for any historical period. If your DAGs run on increments of time-dimensioned data and are idempotent, you can retroactively change features and create new ones based on historical data. This is a key pattern for creating feature stores containing time-dimensioned features to train and test your models.

### Airflow integrations for MLOps

With Airflow, you can orchestrate actions in any MLOps tool that has an API. Many MLOps tools have integrations available with pre-defined modules like operators, decorators, and hooks to interact with the tool. For example, there are integrations for:

* [AWS SageMaker](/docs/learn/airflow-sagemaker). A tool to train and deploy machine learning models on AWS.
* [Databricks](/docs/learn/airflow-databricks). A tool to run Apache Spark workloads.
* [Cohere](/docs/learn/airflow-cohere). A tool to train and deploy LLMs.
* [OpenAI](/docs/learn/airflow-openai). A tool to train and deploy large models, including `GPT-4` and `DALL·E 3`.
* [Weights & Biases](/docs/learn/airflow-weights-and-biases). A tool to track and visualize machine learning experiments.
* [Weaviate](/docs/learn/airflow-weaviate). An open source vector database.
* [OpenSearch](/docs/learn/airflow-opensearch). An open source search engine with advanced ML features.
* [Pgvector](/docs/learn/airflow-pgvector). An extension enabling vector operations in PostgreSQL.
* [Pinecone](/docs/learn/airflow-pinecone). A proprietary vector database.
* (Beta) [Snowpark](https://airflow.apache.org/registry/providers/astro-provider-snowflake/). An interface to run non-SQL code in Snowflake, includes the machine learning library [Snowpark ML](https://docs.snowflake.com/developer-guide/snowpark-ml/index).
* [Azure ML](https://azure.microsoft.com/en-us/free/machine-learning). A tool to train and deploy machine learning models on Azure.

Additionally, the provider packages for the main cloud providers include modules to interact with their ML tools and compute options:

* [AWS](https://airflow.apache.org/registry/providers/amazon/)
* [Azure](https://airflow.apache.org/registry/providers/microsoft-azure/)
* [Google Cloud](https://airflow.apache.org/registry/providers/google/)

## Resources

To learn more about using Airflow for MLOps, check out the following resources:

* Reference architectures:
  * [GenAI](/docs/learn/reference-architecture-context-graph)
  * [MLOps](/docs/learn/reference-architecture-snowpatrol)
* Webinars:
  * [Modern Infrastructure for World Class AI Applications](https://www.astronomer.io/events/webinars/modern-infrastructure-for-world-class-ai-applications-video/) - a joint webinar with Astronomer and Weaviate.
  * [Driving Next-Gen AI Applications with AWS and Astronomer](https://www.astronomer.io/events/webinars/driving-next-gen-ai-applications-with-aws-and-astronomer-video/)
  * [Optimizing ML/AI Workflows with Essential Airflow Features](https://www.astronomer.io/events/webinars/optimizing-ml-ai-workflows-with-essential-airflow-features-video/)
  * [Airflow at Faire: Democratizing Machine Learning at Scale](https://www.astronomer.io/events/webinars/airflow-at-faire-democratizing-machine-learning-at-scale/)
  * [How to Orchestrate Machine Learning Workflows with Airflow](https://www.astronomer.io/events/webinars/how-to-orchestrate-machine-learning-workflows-with-airflow/)
  * [Batch Inference with Airflow and SageMaker](https://www.astronomer.io/events/webinars/batch-inference-with-airflow-and-sagemaker/)
  * [Using Airflow with Tensorflow and MLFlow](https://www.astronomer.io/events/webinars/using-airflow-with-tensorflow-mlflow/)
* eBooks and white papers:
  * [GenAI Cookbook](https://www.astronomer.io/ebooks/gen-ai-airflow-cookbook/)
  * [Guide to Data Orchestration for Generative AI](https://www.astronomer.io/white-papers/gen-ai-data-orchestration/)
* Podcast episodes:
  * [Using Airflow To Power Machine Learning Pipelines at Optimove with Vasyl Vasyuta](https://www.astronomer.io/podcast/using-airflow-to-power-machine-learning-pipelines-at-optimove-with-vasyl-vasyuta/)
  * [The Intersection of AI and Data Management at Dosu with Devin Stein](https://www.astronomer.io/podcast/the-intersection-of-ai-and-data-management-at-dosu-with-devin-stein/)
  * [How Laurel Uses Airflow To Enhance Machine Learning Pipelines with Vincent La and Jim Howard](https://www.astronomer.io/podcast/laurel-uses-airflow-enhance-machine-learning-pipelines-vincent-la-jim-howard/)
  * [AI-Powered Vehicle Automation at Ford Motor Company with Serjesh Sharma](https://www.astronomer.io/podcast/ai-powered-vehicle-automation-at-ford-motor-company-with-serjesh-sharma/)

Astronomer wants to help you succeed with Airflow by continuously creating resources on how to use Airflow for MLOps. If you have any questions or suggestions for additional topics to cover, contact us in the `#airflow-astronomer` channel in the [Apache Airflow Slack](https://apache-airflow-slack.herokuapp.com/).


# Orchestrate MongoDB operations with Apache Airflow
Source: https://astronomer.io/docs/learn/airflow-mongodb

Learn how to load vector embeddings into MongoDB with Apache Airflow.

[MongoDB](https://www.mongodb.com/) is a general-purpose document database that supports high-dimensional embeddings for text data and complex search queries. With the [Mongo DB Airflow provider](https://airflow.apache.org/registry/providers/mongo/), you can orchestrate interactions with MongoDB from your Airflow DAGs.

In this tutorial, you'll use Apache Airflow, MongoDB, and OpenAI to create a pipeline to ingest, embed, store, and query video game descriptions in MongoDB.

## Why use Airflow with MongoDB?

MongoDB can store and query high-dimensional embeddings for text data, which is useful for applications like recommendation systems. You can orchestrate all steps of the process with Airflow, from ingesting data to querying it.

By integrating MongoDB with Airflow, you can:

* Use Airflow's [assets and data-aware scheduling](/docs/learn/airflow-datasets) capabilities to trigger operations in MongoDB based on other events in your data ecosystem, like the training of a new model having completed or the successful ingestion of new text data.
* Create pipelines interacting with MongoDB that adapt to changes in your data at runtime using [dynamic](/docs/learn/dynamic-tasks) Airflow tasks.
* Add Airflow features such as retries, branching, and alerts to MongoDB operations for handling complex task dependencies or task failures.

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of MongoDB. See [Getting started](https://www.mongodb.com/docs/manual/tutorial/getting-started/).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow decorators. See [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).
* Airflow connections. See [Managing your connections in Apache Airflow](/docs/learn/connections).

## Prerequisites

* A MongoDB cluster. Astronomer recommends using [MongoDB Atlas](https://www.mongodb.com/cloud/atlas/register), a hosted MongoDB cluster with integrated data services that offers a free trial. See [Getting started with MongoDB Atlas](https://www.mongodb.com/docs/atlas/getting-started/).
* The [Astro CLI](/docs/cli/v1.43/overview).
* An OpenAI API key of at least [tier 1](https://platform.openai.com/docs/guides/rate-limits/usage-tiers). If you don't want to use OpenAI, you have to adjust the code in the embedding functions to use a different embedding service.

## Step 1: Configure your MongoDB Atlas cluster

First, you need to configure your MongoDB Atlas cluster so Airflow can connect to it.

1. In your MongoDB Atlas account under **Security**, go to **Database Access** and create a database user with a password. Make sure the user has privileges to write data to the database and to save the password in a secure location, as you need it later.

   <Frame>
     <img alt="Screenshot of the MongoDB Atlas UI showing how to add a new user." />
   </Frame>

2. If you haven't already done so during your MongoDB setup, go to **Security** > **Network Access** and add your public IP address to the IP access list. You can find your public IP address on Mac and Linux by running `curl ifconfig.co/`, or on Windows by running `ipconfig /all`.

## Step 2: Configure your Astro project

Use the Astro CLI to create and run an Airflow project locally.

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-mongodb-tutorial && cd astro-mongodb-tutorial
   $ astro dev init
   ```

2. Add the following lines to the `requirements.txt` file of your Astro project:

   ```text wrap theme={null}
   apache-airflow-providers-mongo==5.0.3
   openai==1.78.0
   ```

   This installs the [Mongo provider](https://airflow.apache.org/registry/providers/mongo) package that contains all of the relevant MongoDB modules for Airflow, as well as the OpenAI package for embedding text data.

3. In the `.env` file add the following environment variables:

   ```text wrap theme={null}
   AIRFLOW_CONN_MONGODB_DEFAULT='{
       "conn_type": "mongo",
       "host": "<your_cluster>.<identifier>.mongodb.net",
       "login": "<your_user>",
       "password": "<your_password>",
       "extra": {
           "srv": "true",
           "ssl": "true"
       }
   }'

   OPENAI_API_KEY="sk-<your_openai_api_key>"
   ```

   Replace `<your openai api key>` with your OpenAI API key and `<your_cluster>`, `<your_user>`, and `<your_password>` with your MongoDB Atlas cluster name, database user, and database user password created in [Step 1](#step-1-configure-your-mongodb-atlas-cluster). You can find your MongoDB Atlas cluster name in the Atlas UI by clicking on **Connect** in your cluster overview.

<Info>
  The `AIRFLOW_CONN_MONGODB_DEFAULT` environment variable is used to create a connection to your MongoDB cluster in Airflow with the connection ID `mongodb_default`. In the MongoDB Airflow provider version 4.2.2 and later, it is also possible to [set the connection in the Airflow UI](/docs/learn/connections#defining-connections-in-the-airflow-ui). To do so, provide the following connection details:

  * Connection Id: `mongodb_default`
  * Connection Type: `MongoDB`
  * Host: `<your_cluster>.<identifier>.mongodb.net` (for example: `mycluster.abcde.mongodb.net`)
  * Login: `<your_user>`
  * Password: `<your_password>`
  * Extra Fields JSON: `{"srv": "true", "ssl": "true"}`

  While leaving all other fields blank.
</Info>

## Step 3: Add your data

The DAG in this tutorial runs a query on vectorized game descriptions.

Create a new file called `games.txt` in the `include` directory, then copy and paste the following information:

```text wrap theme={null}
1 ::: Minecraft (2009) ::: sandbox ::: In a blocky, procedurally-generated world, players explore, gather resources, craft tools, and build structures, with the option to fight off monsters and explore vast environments.
2 ::: The Sims 2 (2004) ::: life simulation ::: Players create and control characters, managing their lives, relationships, and homes in a virtual world, while guiding them through everyday tasks and fulfilling personal ambitions.
3 ::: Call of Duty: Modern Warfare 2 (2009) ::: shooter ::: In this intense military first-person shooter, players join elite military operations across the globe, fighting in a fast-paced war against a dangerous enemy.
4 ::: Halo 3 (2007) ::: sci-fi shooter ::: Master Chief returns to finish the fight against the Covenant and the Flood, battling across futuristic environments in a war to save humanity.
5 ::: Star Wars: Battlefront 2 (2005) ::: action shooter ::: Players fight in large-scale battles across iconic Star Wars locations, engaging in both ground and space combat to claim victory.
6 ::: Age of Mythology (2002) ::: real-time strategy ::: Players command armies of mythological creatures and heroes, leveraging the powers of gods to wage war in an ancient, myth-inspired world.
7 ::: Stronghold (2001) ::: real-time strategy ::: In a medieval world, players build and manage castles, control armies, and lay siege to enemies, balancing economic management with military strategy.
8 ::: Command & Conquer: Tiberium Wars (2007) ::: real-time strategy ::: In a futuristic setting, players lead global military factions battling over the alien resource Tiberium, engaging in fast-paced tactical warfare.
9 ::: Minesweeper (1990) ::: puzzle ::: In this classic puzzle game, players use logic to uncover hidden mines on a grid without triggering any explosions.
10 ::: Addy Junior (2000) ::: educational ::: An educational game designed to help children improve reading, math, and problem-solving skills through engaging, playful activities.
11 ::: Impossible Creatures (2003) ::: real-time strategy ::: Players design and combine creatures using DNA from various animals, creating unique hybrids to lead in battle across an alternate 1930s world.
12 ::: World of Warcraft (2004) ::: MMORPG ::: Players explore a vast fantasy world, completing quests, battling enemies, and forming alliances in an ever-changing landscape of adventure.
```

## Step 4: Create your DAG

In your Astro project `dags` folder, create a new file called `query_game_vectors.py`. Paste the following code into the file:

```python expandable wrap theme={null}
"""
## Tutorial DAG: Load and query video game descriptions with MongoDB and OpenAI
"""

import logging
import os

from airflow.decorators import dag, task
from airflow.models.baseoperator import chain
from airflow.models.param import Param
from airflow.operators.empty import EmptyOperator
from airflow.providers.mongo.hooks.mongo import MongoHook
from pendulum import datetime

t_log = logging.getLogger("airflow.task")

_MONGO_DB_CONN = os.getenv("MONGO_DB_CONN", "mongodb_default")
_MONGO_DB_DATABASE_NAME = os.getenv("MONGO_DB_DATABASE_NAME", "games")
_MONGO_DB_COLLECTION_NAME = os.getenv("MONGO_DB_COLLECTION_NAME", "games_nostalgia")
_MONGO_DB_SEARCH_INDEX_NAME = os.getenv("MONGO_DB_SEARCH_INDEX_NAME", "find_me_a_game")
_MONGO_DB_VECTOR_COLUMN_NAME = os.getenv("MONGO_DB_VECTOR_COLUMN_NAME", "vector")

_OPENAI_EMBEDDING_MODEL = os.getenv("OPENAI_EMBEDDING_MODEL", "text-embedding-3-small")
_OPENAI_EMBEDDING_MODEL_DIMENSIONS = os.getenv(
    "OPENAI_EMBEDDING_MODEL_DIMENSIONS", 1536
)

_DATA_TEXT_FILE_PATH = os.getenv("DATA_TEXT_FILE_PATH", "include/games.txt")

_COLLECTION_EXISTS_TASK_ID = "collection_already_exists"
_CREATE_COLLECTION_TASK_ID = "create_collection"
_CREATE_INDEX_TASK_ID = "create_search_index"
_INDEX_EXISTS_TASK_ID = "search_index_already_exists"


def _get_mongodb_database(
    mongo_db_conn_id: str = _MONGO_DB_CONN,
    mongo_db_database_name: str = _MONGO_DB_DATABASE_NAME,
):
    """
    Get the MongoDB database.
    Args:
        mongo_db_conn_id (str): The connection ID for the MongoDB connection.
        mongo_db_database_name (str): The name of the database.
    Returns:
        The MongoDB database.
    """
    hook = MongoHook(mongo_conn_id=mongo_db_conn_id)
    client = hook.get_conn()
    return client[mongo_db_database_name]


def _create_openai_embeddings(text: str, model: str):
    """
    Create embeddings for a text with the OpenAI API.
    Args:
        text (str): The text to create embeddings for.
        model (str): The OpenAI model to use.
    Returns:
        The embeddings for the text.
    """
    from openai import OpenAI

    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    response = client.embeddings.create(input=text, model=model)
    embeddings = response.data[0].embedding

    return embeddings


@dag(
    start_date=datetime(2024, 10, 1),
    schedule=None,
    catchup=False,
    max_consecutive_failed_dag_runs=5,
    tags=["mongodb"],
    doc_md=__doc__,
    params={
        "game_concepts": Param(
            ["fantasy", "quests"],
            type="array",
            description=(
                "What kind of game do you want to play today?"
                + " Add one concept per line."
            ),
        ),
    },
)
def query_game_vectors():

    @task.branch
    def check_for_collection() -> str:
        "Check if the provided collection already exists and decide on the next step."
        database = _get_mongodb_database()
        collection_list = database.list_collection_names()
        if _MONGO_DB_COLLECTION_NAME in collection_list:
            return _COLLECTION_EXISTS_TASK_ID
        else:
            return _CREATE_COLLECTION_TASK_ID

    @task(task_id=_CREATE_COLLECTION_TASK_ID)
    def create_collection():
        "Create a new collection in the database."
        database = _get_mongodb_database()
        database.create_collection(_MONGO_DB_COLLECTION_NAME)

    collection_already_exists = EmptyOperator(task_id=_COLLECTION_EXISTS_TASK_ID)
    collection_ready = EmptyOperator(
        task_id="collection_ready", trigger_rule="none_failed"
    )

    @task
    def extract() -> list:
        """
        Extract the games from the text file.
        Returns:
            list: A list with the games.
        """
        import re

        with open(_DATA_TEXT_FILE_PATH, "r") as f:
            games = f.readlines()

        games_list = []

        for game in games:

            parts = game.split(":::")
            title_year = parts[1].strip()
            match = re.match(r"(.+) \((\d{4})\)", title_year)

            title, year = match.groups()
            year = int(year)

            genre = parts[2].strip()
            description = parts[3].strip()

            game_data = {
                "title": title,
                "year": year,
                "genre": genre,
                "description": description,
            }

            games_list.append(game_data)

        return games_list

    @task(map_index_template="{{ game_str }}")
    def transform_create_embeddings(game: dict) -> dict:
        """
        Create embeddings for the game description.
        Args:
            game (dict): A dictionary with the game's data.
        Returns:
            dict: The game's data with the embeddings.
        """
        embeddings = _create_openai_embeddings(
            text=game.get("description"), model=_OPENAI_EMBEDDING_MODEL
        )
        game[_MONGO_DB_VECTOR_COLUMN_NAME] = embeddings

        # optional: setting the custom map index
        from airflow.operators.python import get_current_context

        context = get_current_context()
        context["game_str"] = f"{game['title']} ({game['year']}) - {game['genre']}"

        return game

    @task(trigger_rule="none_failed", map_index_template="{{ game_str }}")
    def load_data_to_mongo_db(game_data: dict) -> None:
        """
        Load the game data to the MongoDB collection.
        Args:
            game_data (dict): A dictionary with the game's data.
        """

        database = _get_mongodb_database()
        collection = database[_MONGO_DB_COLLECTION_NAME]

        filter_query = {
            "title": game_data["title"],
            "year": game_data["year"],
            "genre": game_data["genre"],
        }

        game_str = f"{game_data['title']} ({game_data['year']}) - {game_data['genre']}"

        existing_document = collection.find_one(filter_query)

        if existing_document:
            if existing_document.get("description") != game_data["description"]:
                collection.update_one(
                    filter_query, {"$set": {"description": game_data["description"]}}
                )
                t_log.info(f"Updated description for record: {game_str}")
            else:
                t_log.info(f"Skipped duplicate record: {game_str}")
        else:
            collection.update_one(
                filter_query, {"$setOnInsert": game_data}, upsert=True
            )
            t_log.info(f"Inserted record: {game_str}")

        # optional: setting the custom map index
        from airflow.operators.python import get_current_context

        context = get_current_context()
        context["game_str"] = game_str

    @task.branch
    def check_for_search_index() -> str:
        "Check if the provided index already exists and decide on the next step."
        database = _get_mongodb_database()
        collection = database[_MONGO_DB_COLLECTION_NAME]
        index_list = collection.list_search_indexes().to_list()
        index_name_list = [index.get("name") for index in index_list]
        if _MONGO_DB_SEARCH_INDEX_NAME in index_name_list:
            return _INDEX_EXISTS_TASK_ID
        else:
            return _CREATE_INDEX_TASK_ID

    @task(task_id=_CREATE_INDEX_TASK_ID)
    def create_search_index():
        """
        Create a search index model for the MongoDB collection.
        """
        from pymongo.operations import SearchIndexModel

        database = _get_mongodb_database()
        collection = database[_MONGO_DB_COLLECTION_NAME]

        search_index_model = SearchIndexModel(
            definition={
                "mappings": {
                    "dynamic": True,
                    "fields": {
                        _MONGO_DB_VECTOR_COLUMN_NAME: {
                            "type": "knnVector",
                            "dimensions": _OPENAI_EMBEDDING_MODEL_DIMENSIONS,
                            "similarity": "cosine",
                        }
                    },
                },
            },
            name=_MONGO_DB_SEARCH_INDEX_NAME,
        )

        collection.create_search_index(model=search_index_model)

    search_index_already_exists = EmptyOperator(task_id=_INDEX_EXISTS_TASK_ID)

    @task.sensor(
        poke_interval=10, timeout=3600, mode="poke", trigger_rule="none_failed"
    )
    def wait_for_full_indexing():
        """
        Wait for the search index to be fully built.
        """
        from airflow.sensors.base import PokeReturnValue

        database = _get_mongodb_database()
        collection = database[_MONGO_DB_COLLECTION_NAME]

        index_list = collection.list_search_indexes().to_list()
        index = next(
            (
                index
                for index in index_list
                if index.get("name") == _MONGO_DB_SEARCH_INDEX_NAME
            ),
            None,
        )

        if index:
            status = index.get("status")
            if status == "READY":
                t_log.info(f"Search index is {status}. Ready to query.")
                condition_met = True
            elif status == "FAILED":
                raise ValueError("Search index failed to build.")
            else:
                t_log.info(
                    f"Search index is {status}. Waiting for indexing to complete."
                )
                condition_met = False
        else:
            raise ValueError("Search index not found.")

        return PokeReturnValue(is_done=condition_met)

    @task
    def embed_concepts(**context):
        """
        Create embeddings for the provided concepts.
        """
        from openai import OpenAI

        client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
        game_concepts = context["params"]["game_concepts"]
        game_concepts_str = " ".join(game_concepts)

        embeddings = client.embeddings.create(
            input=game_concepts_str, model=_OPENAI_EMBEDDING_MODEL
        )

        return embeddings.to_dict()

    @task
    def query(query_vector: list):
        """
        Query the MongoDB collection for games based on the provided concepts.
        """

        db = _get_mongodb_database()
        collection = db[_MONGO_DB_COLLECTION_NAME]

        results = collection.aggregate(
            [
                {
                    "$vectorSearch": {
                        "exact": True,
                        "index": _MONGO_DB_SEARCH_INDEX_NAME,
                        "limit": 1,
                        "path": _MONGO_DB_VECTOR_COLUMN_NAME,
                        "queryVector": query_vector["data"][0]["embedding"],
                    }
                }
            ]
        )

        results_list = []

        for result in results:

            game_id = str(result["_id"])
            title = result["title"]
            year = result["year"]
            genre = result["genre"]
            description = result["description"]

            t_log.info(f"You should play {title}!")
            t_log.info(f"It was released in {year} and belongs to the {genre} genre.")
            t_log.info(f"Description: {description}")

            results_list.append(
                {
                    "game_id": game_id,
                    "title": title,
                    "year": year,
                    "genre": genre,
                    "description": description,
                }
            )

        return results_list

    _extract = extract()
    _transform_create_embeddings = transform_create_embeddings.expand(game=_extract)
    _load_data_to_mongo_db = load_data_to_mongo_db.expand(
        game_data=_transform_create_embeddings
    )

    _query = query(embed_concepts())

    chain(
        check_for_collection(),
        [create_collection(), collection_already_exists],
        collection_ready,
    )

    chain(
        collection_ready,
        check_for_search_index(),
        [create_search_index(), search_index_already_exists],
        wait_for_full_indexing(),
        _query,
    )

    chain(collection_ready, _load_data_to_mongo_db, _query)


query_game_vectors()
```

This DAG consists of thirteen tasks to make a simple ML orchestration pipeline.

* First, the `check_for_collection` task checks if the `games_nostalgia` collection already exists in the `games` database. If it does, the collection creation is skipped, if not, the collection is created by the `create_collection` task.
* Once the collection is ready, a similar pattern is used to create a search index `find_me_a_game` if it doesn't already exist.
* Simultaneously, the game descriptions are being ingested in an ETL pipeline where the transformation includes the creating of vector embeddings using OpenAI's `text-embedding-3-small` model. The embeddings are then stored in the `games_nostalgia` collection alongside the game data.
* After the search index is ready and the data is ingested, the custom [sensor](/docs/learn/what-is-a-sensor) `wait_for_full_indexing` makes sure the search index is fully built before the `query` task is triggered.
* Finally, the `query` task queries the `games_nostalgia` collection for the game with the most similar description to the concepts provided in the [Airflow params](/docs/learn/airflow-params) dictionary.

<Frame>
  <img alt="Screenshot of the Airflow UI showing the query_game_vectors DAG graph." />
</Frame>

## Step 5: Run the DAG and review the data

Now you can run the DAG manually to find a game to play!

1. Run `astro dev start` in your Astro project to start Airflow and open the Airflow UI at `localhost:8080`. Sign in with `admin` as the username and password.

2. In the Airflow UI, run the `query_game_vectors` DAG by clicking the play button. Then, provide [Airflow params](/docs/learn/airflow-params) for `game_concepts`.

   <Frame>
     <img alt="Screenshot of the Airflow UI Trigger DAG view showing the concepts fantasy and quests selected as query params." />
   </Frame>

3. After the DAG completes successfully, go to the task logs of the `query` task to see the game with the most similar description to the concepts you provided.

   ```text wrap theme={null}
   [2025-05-09, 13:28:38] INFO - You should play World of Warcraft!
   [2025-05-09, 13:28:38] INFO - It was released in 2004 and belongs to the MMORPG genre.
   [2025-05-09, 13:28:38] INFO - Description: Players explore a vast fantasy world, completing quests, battling enemies, and forming alliances in an ever-changing landscape of adventure.: source="airflow.task"
   ```

## Conclusion

Congratulations! You used Airflow and MongoDB to get a game suggestion! You can now use Airflow to orchestrate MongoDB operations in your own pipelines.


# Run Airflow tasks in other languages
Source: https://astronomer.io/docs/learn/airflow-multilanguage

Learn how to run Airflow tasks in languages other than Python.

Airflow 3 enables users to write SDKs allowing definition of Airflow tasks in languages other than Python. Experimental SDKs for Golang and Java are available as of the Task SDK 1.3 release.

Support for other languages:

* Makes it easier for users to migrate workflows from legacy tools written in languages other than Python to Airflow.
* Makes Airflow more accessible to developers who prefer to code in another language.
* Gives users access to features unique to a supported language.

<Tip>
  Multilanguage support is currently experimental and under development. This guide is subject to change and will be expanded over time. If you want to contribute to support writing Airflow tasks in the language of your choice, contact the Airflow developers in the [Airflow Slack](https://apache-airflow-slack.herokuapp.com/) or the [Airflow Dev list](https://airflow.apache.org/community/).
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have existing knowledge of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Depending on your target language:
  * Basic Golang concepts. See [Golang documentation](https://go.dev/doc/).
  * Basic Java concepts. See [Learn Java](https://dev.java/learn/).

## How it works

Two environment variables configure which SDK coordinators are available, in addition to the default Python Task SDK:

* `AIRFLOW__SDK__COORDINATORS`: a JSON containing all available coordinators.
* `AIRFLOW__SDK__QUEUE_TO_COORDINATOR`: a JSON that maps `queue` names to coordinator names.

When adding tasks in other languages, you write the task logic in your target language in a separate module and if needed, compile it. Two elements need to be present: the `dag_id`, matching the id of the Dag in which the task is used, and a `task_id` that matches the id of the function decorated with `@task.stub` in the Dag. The Dag itself is written in Python.

```python wrap theme={null}
from airflow.sdk import task

@task.stub(queue="my-language-queue")
def my_task(): ...

my_task()
```

When the task runs, Airflow uses the queue, `dag_id`, and `task_id` to find the matching function in the target language and executes it. How you declare the ids differs by language, as shown in the examples below.

Additionally, the SDKs contain a client for reading Variables, Connections, and XCom, as well as a logger. A value returned from the task becomes its XCom.

The compiled bundle and the coordinator configuration must be available on the Airflow component that runs your tasks.

<Tip>
  When using task SDKs for other languages on Astro, you need to create matching [worker queues](/docs/astro/configure-worker-queues) in addition to setting `AIRFLOW__SDK__COORDINATORS` and `AIRFLOW__SDK__QUEUE_TO_COORDINATOR` as [environment variables](/docs/astro/manage-env-vars). For example, if you are using the Golang SDK, and set `AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"golang": "go"}'`, you need to create a worker queue with the name `golang`.

  Additionally, make sure any compiled binaries (for example, Go executables) are compatible with `linux/amd64`, the architecture of workers on Astro Hosted.
</Tip>

## Golang SDK example

<Warning>
  The Golang SDK is experimental and still under development. You can track its status [here](https://pkg.go.dev/github.com/apache/airflow/go-sdk).
</Warning>

Make sure your Airflow project is at least on version 3.3 and using the Task SDK version 1.3+.

### Step 1: Configure the executable coordinator

Add two environment variables to your `.env` file. The first maps the `golang` queue to a coordinator named `go`. The second defines that coordinator, which scans the `executables_root` location for compiled bundles and runs them. The executables location needs to be accessible to your Airflow worker.

```text wrap theme={null}
AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"golang": "go"}'

AIRFLOW__SDK__COORDINATORS='{
    "go": {
        "classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator",
        "kwargs": {
            "executables_root": ["/usr/local/airflow/include/go_bundle/bin"]
        }
    }
}'
```

### Step 2: Write a Go task bundle

1. Create the directory for the Go module, the parent directory of the `executables_root`:

   ```sh wrap theme={null}
   $ mkdir -p include/go_bundle && cd include/go_bundle
   ```

2. Create a `go.mod` file. The `require` line pins the SDK version, and the `tool` directive makes the bundle packer available through `go tool`. Replace the placeholders with your versions.

   ```text wrap theme={null}
   module example.com/go-bundle

   go <your-go-version>

   require github.com/apache/airflow/go-sdk <your-go-sdk-version>

   tool github.com/apache/airflow/go-sdk/cmd/airflow-go-pack
   ```

3. Create a `main.go` file with the task logic.

   ```go expandable wrap theme={null}
   package main

   import (
       "encoding/json"
       "fmt"
       "log"
       "log/slog"
       "runtime"

       v1 "github.com/apache/airflow/go-sdk/bundle/bundlev1"
       "github.com/apache/airflow/go-sdk/bundle/bundlev1/bundlev1server"
       "github.com/apache/airflow/go-sdk/sdk"
   )

   var (
       bundleName    = "go_task_syntax_example"
       bundleVersion = "1.0.0"
   )

   type bundle struct{}

   var _ v1.BundleProvider = (*bundle)(nil)

   func (b *bundle) GetBundleVersion() v1.BundleInfo {
       return v1.BundleInfo{Name: bundleName, Version: &bundleVersion}
   }

   func (b *bundle) RegisterDags(dagbag v1.Registry) error {
       d := dagbag.AddDag("go_task_syntax_example")
       d.AddTask(transform)
       return nil
   }

   func main() {
       if err := bundlev1server.Serve(&bundle{}); err != nil {
           log.Fatal(err)
       }
   }

   func transform(ctx sdk.TIRunContext, client sdk.Client, logger *slog.Logger) (any, error) {
       ti := ctx.TaskInstance()

       raw, err := client.GetXCom(ctx, ti.DagID, ti.RunID, "extract", nil, "return_value", nil)
       if err != nil {
           return nil, fmt.Errorf("reading extract XCom: %w", err)
       }

       var payload struct {
           Numbers []float64 `json:"numbers"`
       }
       encoded, err := json.Marshal(raw)
       if err != nil {
           return nil, fmt.Errorf("re-encoding extract payload: %w", err)
       }
       if err := json.Unmarshal(encoded, &payload); err != nil {
           return nil, fmt.Errorf("decoding extract payload: %w", err)
       }

       var sum float64
       for _, n := range payload.Numbers {
           sum += n
       }

       logger.Info("summed numbers in Go", "sum", sum, "count", len(payload.Numbers))
       return map[string]any{
           "sum":         sum,
           "count":       len(payload.Numbers),
           "computed_by": "Go " + runtime.Version(),
       }, nil
   }
   ```

   The `RegisterDags` method binds Go functions to the Python Dag. The `dag_id` you pass to `AddDag` must match the Python `@dag` id, and each function name passed to `AddTask` must match a Python stub task name. The `transform` function reads the `extract` task's XCom, sums the numbers, and pushes the result to XCom.

### Step 3: Build the bundle

From the `include/go_bundle` directory, compile the bundle for the architecture of your Airflow containers (`--goos linux` for Linux containers). Use `arm64` on Apple Silicon or `amd64` on Intel and AMD machines. Note that you need [Go](https://go.dev/dl/) 1.24 or later to compile the task bundle.

```sh wrap theme={null}
$ go mod tidy
$ go tool airflow-go-pack --goos linux --goarch arm64 --output ./bin/go_task_syntax_example .
```

This writes a single executable to `include/go_bundle/bin`, which is the `executables_root` you set in Step 1.

<Note>
  Astro Hosted workers use `linux/amd64`, which means you'll need to compile with `--goos linux --goarch amd64` before deploying your project to Astro.
</Note>

### Step 4: Create the Dag

In your `dags` folder, create a file called `go_task_syntax_example.py` with the following code:

```python wrap theme={null}
import random

from airflow.sdk import dag, task, chain


@dag(tags=["go sdk"])
def go_task_syntax_example():
    @task
    def extract():
        return {"numbers": [random.randint(1, 100) for _ in range(random.randint(3, 6))]}

    @task.stub(queue="golang")
    def transform(): ...

    @task
    def load(result):
        print(f"Go returned {result}")
        return result

    extracted = extract()
    transformed = transform()
    chain(extracted, transformed)
    load(transformed)


go_task_syntax_example()
```

The Python `extract` task pushes a list of numbers to XCom, the Go `transform` task reads that list and sums it, and the Python `load` task reads the result back from Go.

The `transform` task uses `@task.stub(queue="golang")` and has no Python body. The stub tells Airflow the task's name and its place in the Dag, and the `queue` value routes it to the Go coordinator. The `dag_id` and the stub task name must match the values registered in the Go bundle.

<Note>
  When using the Golang SDK on Astro, you need to create a matching [worker queue](/docs/astro/configure-worker-queues) in addition to setting `AIRFLOW__SDK__COORDINATORS` and `AIRFLOW__SDK__QUEUE_TO_COORDINATOR` as [environment variables](/docs/astro/manage-env-vars). For example, for `AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"golang": "go"}'`, you need to create a worker queue with the name `golang`.
</Note>

## Java SDK example

<Warning>
  The Java SDK is experimental and still under development. You can track its status in the [java-sdk directory](https://github.com/apache/airflow/tree/main/java-sdk) of the Airflow repository.
</Warning>

Make sure your Airflow project is at least on version 3.3 and using the Task SDK version 1.3+. The Java task runs as a compiled jar, so the Airflow component that runs your tasks also needs a Java runtime, for example `openjdk-21-jre-headless`. You'll need to add the runtime to your `packages.txt` file to install it in your image.

### Step 1: Configure the Java coordinator

Add two environment variables to your `.env` file. The first maps the `java` queue to a coordinator named `java`. The second defines that coordinator, which scans the `jars_root` location for compiled bundle jars and runs them.

```text wrap theme={null}
AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "java"}'

AIRFLOW__SDK__COORDINATORS='{
    "java": {
        "classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
        "kwargs": {
            "jars_root": ["/usr/local/airflow/include/java_bundle"]
        }
    }
}'
```

### Step 2: Write a Java task bundle

1. Create the module directory, including the `com/example/bundle` package path where the source files live. You run Gradle from the module root, so change into `include/java_sdk`:

   ```sh wrap theme={null}
   $ mkdir -p include/java_sdk/src/java/com/example/bundle
   $ cd include/java_sdk
   ```

2. Create the Gradle project files. `gradle.properties` sets the SDK version in one place, `settings.gradle` names the project and points Gradle at the Apache snapshot repository, and `build.gradle` applies the Airflow SDK plugin, pulls in the SDK and its annotation processor, and points `airflowBundle` at the bundle's main class. Set `projectVersion` to the SDK version you are targeting; current builds are published as snapshots.

   `gradle.properties`:

   ```text wrap theme={null}
   org.gradle.configuration-cache=true
   projectVersion=1.0.0-SNAPSHOT
   ```

   `settings.gradle`:

   ```text wrap theme={null}
   pluginManagement {
       repositories {
           maven {
               url "https://repository.apache.org/content/repositories/snapshots/"
               mavenContent { snapshotsOnly() }
           }
           gradlePluginPortal()
           mavenCentral()
       }
   }

   rootProject.name = "airflow-java-sdk-etl-example"
   ```

   `build.gradle`:

   ```text expandable wrap theme={null}
   plugins {
       id("org.apache.airflow.sdk") version "${projectVersion}"
   }

   repositories {
       maven {
           url "https://repository.apache.org/content/repositories/snapshots/"
           mavenContent { snapshotsOnly() }
       }
       mavenCentral()
   }

   dependencies {
       annotationProcessor("org.apache.airflow:airflow-sdk-processor:${projectVersion}")
       implementation("org.apache.airflow:airflow-sdk:${projectVersion}")
       implementation("org.slf4j:slf4j-simple:2.0.17")
   }

   java {
       sourceCompatibility = JavaVersion.VERSION_17
       targetCompatibility = JavaVersion.VERSION_17
   }

   sourceSets {
       main {
           java.srcDir("src/java")
           resources.srcDir("src/resources")
       }
   }

   airflowBundle {
       mainClass = "com.example.bundle.EtlBundleBuilder"
   }
   ```

3. Create the task class at `src/java/com/example/bundle/JavaEtlExample.java`. The `@Builder.Dag` and `@Builder.Task` annotations set the ids, and `@Builder.XCom(task = "extract")` adds the upstream Python task's XCom as a method parameter. The returned `Map` becomes the task's XCom.

   ```java expandable wrap theme={null}
   package com.example.bundle;

   import java.util.LinkedHashMap;
   import java.util.List;
   import java.util.Map;
   import org.apache.airflow.sdk.*;
   import org.slf4j.Logger;
   import org.slf4j.LoggerFactory;

   @Builder.Dag(id = "java_task_syntax_example")
   public class JavaEtlExample {
     private static final Logger logger = LoggerFactory.getLogger(JavaEtlExample.class);

     @Builder.Task(id = "transform")
     public Map<String, Object> transform(
         Client client, @Builder.XCom(task = "extract") Map<String, Object> payload) {
       logger.info("[transform/java] received payload from python 'extract' task: {}", payload);

       List<?> numbers = (List<?>) payload.get("numbers");
       long sum = 0;
       for (Object n : numbers) {
         sum += ((Number) n).longValue();
       }

       Map<String, Object> result = new LinkedHashMap<>();
       result.put("sum", sum);
       result.put("count", numbers.size());
       result.put("computed_by", "Java " + System.getProperty("java.version"));

       logger.info("[transform/java] summed {} numbers to {}", numbers.size(), sum);
       return result;
     }
   }
   ```

4. Create the bundle entry point at `src/java/com/example/bundle/EtlBundleBuilder.java`. It implements `BundleBuilder`, registers the Dag classes, and serves the bundle from `main`.

   ```java wrap theme={null}
   package com.example.bundle;

   import java.util.List;
   import org.apache.airflow.sdk.*;

   public class EtlBundleBuilder implements BundleBuilder {
     @Override
     public Iterable<Dag> getDags() {
       return List.of(JavaEtlExampleBuilder.build());
     }

     public static void main(String[] args) {
       var bundle = new EtlBundleBuilder().build();
       Server.create(args).serve(bundle);
     }
   }
   ```

   `JavaEtlExampleBuilder` is generated at compile time by the annotation processor from the `@Builder` annotations on `JavaEtlExample`.

### Step 3: Build the bundle

From the `include/java_sdk` directory, build the bundle jar with Gradle, then copy the jar into the `jars_root` you set in Step 1. Building needs a JDK (this example builds with Java 21).

```sh wrap theme={null}
$ gradle bundle
$ cp build/bundle/*.jar ../java_bundle/
```

The coordinator loads the jar from `include/java_bundle`.

### Step 4: Create the Dag

In your `dags` folder, create a file called `java_task_syntax_example.py` with the following code:

```python wrap theme={null}
import random

from airflow.sdk import dag, task, chain


@dag(tags=["Java SDK"])
def java_task_syntax_example():
    @task
    def extract():
        return {"numbers": [random.randint(1, 100) for _ in range(random.randint(3, 6))]}

    @task.stub(queue="java")
    def transform(): ...

    @task
    def load(result):
        print(f"Java returned {result}")
        return result

    extracted = extract()
    transformed = transform()
    chain(extracted, transformed)
    load(transformed)


java_task_syntax_example()
```

The `transform` task uses `@task.stub(queue="java")` and has no Python body. The `queue` value routes it to the Java coordinator, and the `dag_id` and stub task name must match the values registered in the Java bundle.

<Note>
  When using the Java SDK on Astro, you need to create a matching [worker queue](/docs/astro/configure-worker-queues) in addition to setting `AIRFLOW__SDK__COORDINATORS` and `AIRFLOW__SDK__QUEUE_TO_COORDINATOR` as [environment variables](/docs/astro/manage-env-vars). For example, for `AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "java"}'`, you need to create a worker queue with the name `java`.
</Note>

## Other ways to run tasks in other languages

You can also run tasks in other languages using the following methods:

* Use the `BashOperator` to run a script in another language. For example, you can use the `BashOperator` to run a JavaScript or R script. See [Run a script in another programming language](/docs/learn/bashoperator#example-run-a-script-in-another-programming-language) for more information.
* Use the `KubernetesPodOperator` to run any Docker image, which can include code in any language. See [Use the `KubernetesPodOperator` to run a script in another language](/docs/learn/kubepod-operator#example-use-the-kubernetespodoperator-to-run-a-script-in-another-language) for more information.


# Use Airflow object storage to interact with cloud storage in an ML pipeline
Source: https://astronomer.io/docs/learn/airflow-object-storage-tutorial

Learn how to use Airflow object storage.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

Airflow 2.8 introduced the [Airflow object storage](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html) feature to simplify how you interact with remote and local object storage systems.

This tutorial demonstrates the object storage feature using a simple machine learning pipeline. The pipeline trains a classifier to predict whether a sentence is more likely to have been said by Star Trek's Captain Kirk or Captain Picard.

## Why use Airflow object storage?

Object stores are ubiquitous in modern data pipelines. They are used to store raw data, model-artifacts, image, video, text and audio files, and more. Because each object storage system has different file naming and path conventions, it can be challenging to work with data across many different object stores.

Airflow's object storage feature allows you to:

* Abstract your interactions with object stores using a [Path API](https://docs.python.org/3/library/pathlib.html). Note that some limitations apply due to the nature of different remote object storage systems. See [Cloud Object Stores are not real file systems](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html#cloud-object-stores-are-not-real-file-systems).
* Switch between different object storage systems without having to change your DAG code.
* Transfer files between different object storage systems without needing to use `XToYTransferOperator` operators.
* Transfer large files efficiently. For object storage, Airflow uses [`shutil.copyfileobj`()](https://docs.python.org/3/library/shutil.html#shutil.copyfileobj) to stream files in chunks instead of loading them into memory in their entirety.

## Time to complete

This tutorial takes approximately 20 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* TaskFlow API. See [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).
* The basics of [pathlib](https://docs.python.org/3/library/pathlib.html).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).
* An object storage system to interact with. This tutorial uses [Amazon S3](https://aws.amazon.com/s3/), but you can use [Google Cloud Storage](https://cloud.google.com/storage), [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/) or local file storage as well.

## Step 1: Configure your Astro project

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-object-storage-tutorial && cd astro-object-storage-tutorial
   $ astro dev init
   ```

2. Add the following lines to your Astro project `requirements.txt` file to install the Amazon provider with the `s3fs` extra, as well as the [scikit-learn](https://scikit-learn.org/stable/) package. If you are using Google Cloud Storage or Azure Blob Storage, install the [Google provider](https://airflow.apache.org/registry/providers/google/) or [Azure provider](https://airflow.apache.org/registry/providers/microsoft-azure/) instead.

   ```text wrap theme={null}
   apache-airflow-providers-amazon[s3fs]==8.13.0
   scikit-learn==1.3.2
   ```

3. To create an [Airflow connection](/docs/learn/connections) to AWS S3, add the following environment variable to your `.env` file. Make sure to replace `<your-aws-access-key-id>` and `<your-aws-secret-access-key>` with your own AWS credentials. Adjust the connection type and parameters if you are using a different object storage system.

   ```text wrap theme={null}
   AIRFLOW_CONN_MY_AWS_CONN='{
       "conn_type": "aws",
       "login": "<your-aws-access-key-id>",
       "password": "<your-aws-secret-access-key>",
   }'
   ```

## Step 2: Prepare your data

In this example pipeline you will train a classifier to predict whether a sentence is more likely to have been said by Captain Kirk or Captain Picard. The training set consists of 3 quotes from each captain stored in `.txt` files.

1. Create a new bucket in your S3 account called `astro-object-storage-tutorial`.
2. In the bucket, create a folder called `ingest` with two subfolders `kirk_quotes` and `picard_quotes`.
3. Upload the files from Astronomer's [GitHub repository](https://github.com/astronomer/2-8-example-dags/tree/main/include/ingestion_data_object_store_use_case) into the respective folders.

## Step 3: Create your DAG

1. In your `dags` folder, create a file called `object_storage_use_case.py`.

2. Copy the following code into the file.

   ```python expandable wrap theme={null}
   """
   ## Move files between object storage system locations in an MLOps pipeline

   This DAG shows the basic use of the Airflow 2.8 Object Storage feature to
   copy files between object storage system locations in an MLOps pipeline training
   a Naive Bayes Classifier to distinguish between quotes from Captain Kirk and
   Captain Picard and provide a prediction for a user-supplied quote.

   To be able to run this DAG you will need to add the contents of
   `include/ingestion_data_object_store_use_case` to your object storage system,
   install the relevant provider package for your object storage and define an
   Airflow connection to it.
   If you do not want to use remote storage you can use `file://` for local object
   storage and adjust the paths accordingly.
   """

   from airflow.decorators import dag, task
   from pendulum import datetime
   from airflow.io.path import ObjectStoragePath
   from airflow.models.baseoperator import chain
   from airflow.models.param import Param
   import joblib
   import base64
   import io

   OBJECT_STORAGE_INGEST = "s3"
   CONN_ID_INGEST = "my_aws_conn"
   PATH_INGEST = "astro-object-storage-tutorial/ingest/"

   OBJECT_STORAGE_TRAIN = "s3"
   CONN_ID_TRAIN = "my_aws_conn"
   PATH_TRAIN = "astro-object-storage-tutorial/train/"

   OBJECT_STORAGE_ARCHIVE = "file"
   CONN_ID_ARCHIVE = None
   PATH_ARCHIVE = "include/archive/"


   base_path_ingest = ObjectStoragePath(
       f"{OBJECT_STORAGE_INGEST}://{PATH_INGEST}", conn_id=CONN_ID_INGEST
   )

   base_path_train = ObjectStoragePath(
       f"{OBJECT_STORAGE_TRAIN}://{PATH_TRAIN}", conn_id=CONN_ID_TRAIN
   )

   base_path_archive = ObjectStoragePath(
       f"{OBJECT_STORAGE_ARCHIVE}://{PATH_ARCHIVE}", conn_id=CONN_ID_ARCHIVE
   )


   @dag(
       start_date=datetime(2023, 12, 1),
       schedule=None,
       catchup=False,
       tags=["ObjectStorage"],
       doc_md=__doc__,
       params={
           "my_quote": Param(
               "Time and space are creations of the human mind.",
               type="string",
               description="Enter a quote to be classified as Kirk-y or Picard-y.",
           )
       },
   )
   def object_storage_use_case():
       @task
       def list_files_ingest(base: ObjectStoragePath) -> list[ObjectStoragePath]:
           """List files in remote object storage including subdirectories."""

           labels = [obj for obj in base.iterdir() if obj.is_dir()]
           files = [f for label in labels for f in label.iterdir() if f.is_file()]
           return files

       @task
       def copy_files_ingest_to_train(src: ObjectStoragePath, dst: ObjectStoragePath):
           """Copy a file from one remote system to another.
           The file is streamed in chunks using shutil.copyobj"""

           src.copy(dst=dst)

       @task
       def list_files_train(base: ObjectStoragePath) -> list[ObjectStoragePath]:
           """List files in remote object storage."""

           files = [f for f in base.iterdir() if f.is_file()]
           return files

       @task
       def get_text_from_file(file: ObjectStoragePath) -> dict:
           """Read files in remote object storage."""

           bytes = file.read_block(offset=0, length=None)
           text = bytes.decode("utf-8")

           key = file.key
           filename = key.split("/")[-1]
           label = filename.split("_")[-2]
           return {"label": label, "text": text}

       @task
       def train_model(train_data: list[dict]):
           """Train a Naive Bayes Classifier using the files in the train folder."""

           from sklearn.feature_extraction.text import CountVectorizer
           from sklearn.naive_bayes import MultinomialNB
           from sklearn.pipeline import make_pipeline
           from sklearn.model_selection import train_test_split

           text_data = [d["text"] for d in train_data]
           labels = [d["label"] for d in train_data]

           X_train, X_test, y_train, y_test = train_test_split(
               text_data, labels, test_size=0.2, random_state=42
           )

           model = make_pipeline(CountVectorizer(), MultinomialNB())

           model.fit(X_train, y_train)

           buffer = io.BytesIO()
           joblib.dump(model, buffer)
           buffer.seek(0)

           encoded_model = base64.b64encode(buffer.getvalue()).decode("utf-8")

           return encoded_model

       @task
       def use_model(encoded_model: str, **context):
           """Load the model and use it for prediction."""
           my_quote = context["params"]["my_quote"]

           model_binary = base64.b64decode(encoded_model)

           buffer = io.BytesIO(model_binary)
           model = joblib.load(buffer)

           predictions = model.predict([my_quote])

           print(f"The quote: '{my_quote}'")
           print(f"sounds like it could have been said by {predictions[0].capitalize()}")

       @task
       def copy_files_train_to_archive(src: ObjectStoragePath, dst: ObjectStoragePath):
           """Copy a file from a remote system to local storage."""

           src.copy(dst=dst)

       @task
       def empty_train(base: ObjectStoragePath):
           """Empty the train folder."""

           for file in base.iterdir():
               file.unlink()

       files_ingest = list_files_ingest(base=base_path_ingest)
       files_copied = copy_files_ingest_to_train.partial(dst=base_path_train).expand(
           src=files_ingest
       )
       files_train = list_files_train(base=base_path_train)
       chain(files_copied, files_train)
       train_data = get_text_from_file.expand(file=files_train)
       encoded_model = train_model(train_data=train_data)
       use_model(encoded_model=encoded_model)
       chain(
           encoded_model,
           copy_files_train_to_archive.partial(dst=base_path_archive).expand(
               src=files_train
           ),
           empty_train(base=base_path_train),
       )


   object_storage_use_case()
   ```

   This DAG uses three different object storage locations, which can be aimed at different object storage systems by changing the `OBJECT_STORAGE_X`, `PATH_X` and `CONN_ID_X` for each location.

   * `base_path_ingest`: The base path for the ingestion data. This is the path to the training quotes you uploaded in [Step 2](#step-2-prepare-your-data).
   * `base_path_train`: The base path for the training data, this is the location from which data for training the model will be read.
   * `base_path_archive`: The base path for the archive location where data that has previously been used for training will be moved to.

   The DAG consists of eight tasks to make a simple MLOps pipeline.

   * The `list_files_ingest` task takes the `base_path_ingest` as an input and iterates through the subfolders `kirk_quotes` and `picard_quotes` to return all files in the folders as individual `ObjectStoragePath` objects. Using the object storage feature enables you to use the `.iterdir()`, `.is_dir()` and `.is_file()` methods to list and evaluate object storage contents no matter which object storage system they are stored in.
   * The `copy_files_ingest_to_train` task is [dynamically mapped](/docs/learn/dynamic-tasks) over the list of files returned by the `list_files_ingest` task. It takes the `base_path_train` as an input and copies the files from the `base_path_ingest` to the `base_path_train` location, providing an example of transferring files between different object storage systems using the `.copy()` method of the `ObjectStoragePath` object. Under the hood, this method uses `shutil.copyfileobj()` to stream files in chunks instead of loading them into memory in their entirety.
   * The `list_files_train` task lists all files in the `base_path_train` location.
   * The `get_text_from_file` task is dynamically mapped over the list of files returned by the `list_files_train` task to read the text from each file using the `.read_blocks()` method of the `ObjectStoragePath` object. Using the object storage feature enables you to switch the object storage system, for example to Azure Blob storage, without needing to change the code. The file name provides the label for the text and both, label and full quote are returned as a dictionary to be passed via [XCom](/docs/learn/airflow-passing-data-between-tasks) to the next task.
   * The `train_model` task trains a [Naive Bayes classifier](https://scikit-learn.org/stable/modules/naive_bayes.html) on the data returned by the `get_text_from_file` task. The fitted model is serialized as a base64 encoded string and passed via XCom to the next task.
   * The `use_model` task deserializes the trained model to run a prediction on a user-provided quote, determining whether the quote is more likely to have been said by Captain Kirk or Captain Picard. The prediction is printed to the logs.
   * The `copy_files_train_to_archive` task copies the files from the `base_path_train` to the `base_path_archive` location analogous to the `copy_files_ingest_to_train` task.
   * The `empty_train` task deletes all files from the `base_path_train` location.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the successful completion of the object_storage_use_case DAG in the Grid view with the Graph tab selected." />
   </Frame>

## Step 4: Run your DAG

1. Run `astro dev start` in your Astro project to start Airflow, then open the Airflow UI at `localhost:8080`.

2. In the Airflow UI, run the `object_storage_use_case` DAG by clicking the play button. Provide any quote you like to the `my_quote` [Airflow param](/docs/learn/airflow-params).

3. After the DAG run completes, go to the task logs of the `use_model` task to see the prediction made by the model.

   ```text wrap theme={null}
   [2023-12-11, 00:19:22 UTC] {logging_mixin.py:188} INFO - The quote: 'Time and space are creations of the human mind.'
   [2023-12-11, 00:19:22 UTC] {logging_mixin.py:188} INFO - sounds like it could have been said by Picard
   ```

## Conclusion

Congratulations! You just used Airflow's object storage feature to interact with files in different locations. To learn more about other methods and capabilities of this feature, see the [OSS Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html).


# Orchestrate OpenAI operations with Apache Airflow
Source: https://astronomer.io/docs/learn/airflow-openai

Learn how to integrate OpenAI and Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[OpenAI](https://openai.com/) is an AI research and deployment company that provides an API for accessing state of the art models like [GPT-4](https://openai.com/gpt-4) and [DALL·E 3](https://openai.com/dall-e-3). The [OpenAI Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-openai/stable/index.html) offers modules to easily integrate OpenAI with Airflow.

In this tutorial you'll use Airflow and the OpenAI Airflow provider to ask a question to Star Trek captains, create embeddings of the answers from each captain, and plot them in two dimensions.

## Why use Airflow with OpenAI?

OpenAI offers a variety of powerful model endpoints for different tasks like text generation, vector embedding, and translation tasks. These models are used in both user-facing applications, such as chatbots, and internal applications, such as a smart search for internal knowledge base content.

Integrating OpenAI with Airflow into an end-to-end machine learning pipeline allows you to:

* Use Airflow's [data-driven scheduling](/docs/learn/airflow-datasets) to run operations using OpenAI model endpoints based on upstream events in your data ecosystem, such as when new user input is ingested or a new dataset is available.
* Send several requests to a model endpoint in parallel based on upstream events in your data ecosystem or user input via [Airflow params](/docs/learn/airflow-params).
* Monitor the OpenAI service using Airflow [alerts](/docs/learn/error-notifications-in-airflow) and protect against API rate limits and outages with [Airflow retries](/docs/learn/rerunning-dags#automatically-retry-tasks).
* Use Airflow to orchestrate the creation of vector embeddings using OpenAI models, which is especially useful for large datasets that can't be processed automatically by vector databases.

## Time to complete

This tutorial takes approximately 15 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of the OpenAI API. See [OpenAI Introduction](https://platform.openai.com/docs/introduction).
* The basics of vector embeddings. See the [OpenAI Embeddings guide](https://platform.openai.com/docs/guides/embeddings).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow hooks. See [Hooks 101](/docs/learn/what-is-a-hook).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).
* An OpenAI API key with at least [tier 1 usage limits](https://platform.openai.com/docs/guides/rate-limits/usage-tiers).

## Step 1: Configure your Astro project

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-openai-tutorial && cd astro-openai-tutorial
   $ astro dev init
   ```

2. Add the following lines to your `requirements.txt` file to install the OpenAI Airflow provider and other supporting packages:

   ```text wrap theme={null}
   apache-airflow-providers-openai==1.0.0
   openai==0.28.1
   matplotlib==3.8.1
   seaborn==0.13.0
   scikit-learn==1.3.2
   pandas==1.5.3
   numpy==1.26.2
   adjustText==0.8
   ```

3. To create an [Airflow connection](/docs/learn/connections) to OpenAI, add the following environment variables to your `.env` file. Make sure to replace `<your-openai-api-key>` with your own OpenAI API key.

   ```text wrap theme={null}
   AIRFLOW_CONN_OPENAI_DEFAULT='{
       "conn_type": "openai",
       "password": "<your-openai-api-key>"
   }'
   ```

## Step 2: Create your DAG

1. In your `dags` folder, create a file called `captains_dag.py`.

2. Copy the following code into the file.

   ```python expandable wrap theme={null}
   """
   ## Ask questions to Star Trek captains using OpenAI's LLMs, embed and visualize the results

   This DAG shows how to use the OpenAI Airflow provider to interact with the OpenAI API.
   The DAG asks a question to a list of Star Trek captains based on values you provide via
   Airflow params, embeds the responses using the OpenAI text-embedding-ada-002 model,
   and visualizes the embeddings in 2 dimensions using PCA, matplotlib and seaborn.
   """

   from airflow.decorators import dag, task
   from airflow.models.param import Param
   from airflow.models.baseoperator import chain
   from airflow.providers.openai.hooks.openai import OpenAIHook
   from airflow.providers.openai.operators.openai import OpenAIEmbeddingOperator
   from sklearn.metrics.pairwise import euclidean_distances
   from sklearn.decomposition import PCA
   from adjustText import adjust_text
   from pendulum import datetime
   import matplotlib.pyplot as plt
   import seaborn as sns
   import pandas as pd
   import numpy as np
   import openai


   OPENAI_CONN_ID = "openai_default"
   IMAGE_PATH = "include/captains_plot.png"

   star_trek_captains_list = [
       "James T. Kirk",
       "Jean-Luc Picard",
       "Benjamin Sisko",
       "Kathryn Janeway",
       "Jonathan Archer",
       "Christopher Pike",
       "Michael Burnham",
       "Saru",
   ]


   @dag(
       start_date=datetime(2023, 11, 1),
       schedule=None,
       catchup=False,
       params={
           "question": Param(
               "Which is your favorite ship?",
               type="string",
               title="Question to ask the captains",
               description="Enter what you would like to ask the captains.",
               min_length=1,
               max_length=500,
           ),
           "captains_to_ask": Param(
               star_trek_captains_list,
               type="array",
               description="List the captains whose answers you would like to compare. "
               + "Suggestions: "
               + ", ".join(star_trek_captains_list),
           ),
           "max_tokens_answer": Param(
               100,
               type="integer",
               description="Maximum number of tokens to generate for the answer.",
           ),
           "randomness_of_answer": Param(
               10,
               type="integer",
               description=(
                   "Enter the desired randomness of the answer on a scale"
                   + "from 0 (no randomness) to 20 (full randomness). "
                   + "This setting corresponds to 10x the temperature setting in the OpenAI API."
               ),
               min=0,
               max=20,
           ),
       },
   )
   def captains_dag():
       @task
       def get_captains_list(**context):
           "Pull the list of captains to ask from the context."
           captains_list = context["params"]["captains_to_ask"]
           return captains_list

       @task
       def ask_a_captain(open_ai_conn_id: str, captain_to_ask, **context):
           "Ask a captain a question using gpt-3.5-turbo."
           question = context["params"]["question"]
           max_tokens_answer = context["params"]["max_tokens_answer"]
           randomness_of_answer = context["params"]["randomness_of_answer"]
           hook = OpenAIHook(conn_id=open_ai_conn_id)
           openai.api_key = hook._get_api_key()

           response = openai.ChatCompletion.create(
               model="gpt-3.5-turbo",
               messages=[
                   {"role": "system", "content": f"You are captain {captain_to_ask}."},
                   {"role": "user", "content": question},
               ],
               temperature=randomness_of_answer / 10,
               max_tokens=max_tokens_answer,
           )

           response = response.choices[0]["message"]["content"]

           print(f"Your Question: {question}")
           print(f"Captain {captain_to_ask} said: {response}")

           return response

       captains_list = get_captains_list()
       captain_responses = ask_a_captain.partial(open_ai_conn_id=OPENAI_CONN_ID).expand(
           captain_to_ask=captains_list
       )

       get_embeddings = OpenAIEmbeddingOperator.partial(
           task_id="get_embeddings",
           conn_id=OPENAI_CONN_ID,
           model="text-embedding-ada-002",
       ).expand(input_text=captain_responses)

       @task
       def plot_embeddings(embeddings, text_labels, file_name="embeddings_plot.png"):
           "Plot the embeddings of the captain responses."
           pca = PCA(n_components=2)
           reduced_embeddings = pca.fit_transform(embeddings)

           plt.figure(figsize=(10, 8))
           df_embeddings = pd.DataFrame(reduced_embeddings, columns=["PC1", "PC2"])
           sns.scatterplot(
               df_embeddings, x="PC1", y="PC2", s=100, color="gold", edgecolor="black"
           )

           font_style = {"color": "black"}
           texts = []
           for i, label in enumerate(text_labels):
               texts.append(
                   plt.text(
                       reduced_embeddings[i, 0],
                       reduced_embeddings[i, 1],
                       label,
                       fontdict=font_style,
                       fontsize=15,
                   )
               )

           # prevent overlapping labels
           adjust_text(texts, arrowprops=dict(arrowstyle="->", color="red"))

           distances = euclidean_distances(reduced_embeddings)
           np.fill_diagonal(distances, np.inf)  # exclude cases where the distance is 0

           n = distances.shape[0]
           distances_list = [
               (distances[i, j], (i, j)) for i in range(n) for j in range(i + 1, n)
           ]

           distances_list.sort(reverse=True)

           legend_handles = []
           for dist, (i, j) in distances_list:
               (line,) = plt.plot(
                   [reduced_embeddings[i, 0], reduced_embeddings[j, 0]],
                   [reduced_embeddings[i, 1], reduced_embeddings[j, 1]],
                   "gray",
                   linestyle="--",
                   alpha=0.3,
               )
               legend_handles.append(line)

           legend_labels = [
               f"{text_labels[i]} - {text_labels[j]}: {dist:.2f}"
               for dist, (i, j) in distances_list
           ]

           for i in range(len(reduced_embeddings)):
               for j in range(i + 1, len(reduced_embeddings)):
                   plt.plot(
                       [reduced_embeddings[i, 0], reduced_embeddings[j, 0]],
                       [reduced_embeddings[i, 1], reduced_embeddings[j, 1]],
                       "gray",
                       linestyle="--",
                       alpha=0.5,
                   )

           plt.legend(
               legend_handles,
               legend_labels,
               title="Distances",
               loc="center left",
               bbox_to_anchor=(1, 0.5),
           )

           plt.tight_layout()
           plt.title(
               "2D Visualization of captain responses", fontsize=16, fontweight="bold"
           )
           plt.xlabel("PCA Component 1", fontdict=font_style)
           plt.ylabel("PCA Component 2", fontdict=font_style)

           plt.savefig(file_name, bbox_inches="tight")
           plt.close()

       chain(
           get_embeddings,
           plot_embeddings(
               get_embeddings.output,
               text_labels=captains_list,
               file_name=IMAGE_PATH,
           ),
       )


   captains_dag()
   ```

   This DAG consists of four tasks to make a simple MLOps pipeline.

   * The `get_captains_list` task fetches the list of Star Trek captains you want to ask your question to. You'll provide the list of captains when you run the DAG with [Airflow params](/docs/learn/airflow-params).
   * The `ask_a_captain` task uses the [`OpenAIHook`](https://airflow.apache.org/docs/apache-airflow-providers-openai/stable/_api/airflow/providers/openai/hooks/openai/index.html) to connect to the OpenAI API. It then uses the [chat completion endpoint](https://platform.openai.com/docs/guides/text-generation/chat-completions-api) to generate answers to the question you provide. This task is [dynamically mapped](/docs/learn/dynamic-tasks) over the list of captains to generate one dynamically mapped task instance per captain.
   * The `get_embeddings` task is defined using the [`OpenAIEmbeddingOperator`](https://airflow.apache.org/docs/apache-airflow-providers-openai/stable/operators/openai.html) to generate vector embeddings of the answers generated by the upstream `ask_a_captain` task. This task is dynamically mapped over the list of answers to retrieve one set of embeddings per answer. This pattern allows for efficient parallelization of the vector embedding generation.
   * The `plot_embeddings` task takes the embeddings created by the upstream task and performs dimensionality reduction using [PCA](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) to plot the embeddings in two dimensions.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the successful completion of the captains_dag DAG in the Grid view with the Graph tab selected. All 8 captains available were selected to be asked the question, which led to 8 mapped task instances of both the ask_a_captain and get_embeddings task." />
   </Frame>

## Step 3: Run your DAG

1. Run `astro dev start` in your Astro project to start Airflow, then open the Airflow UI at `localhost:8080`.

2. In the Airflow UI, run the `captains_dag` DAG by clicking the **Play** button. Then, provide [Airflow params](/docs/learn/airflow-params) for:

   * `Question to ask the captain`: The question you want to ask the captains.
   * `captains_to_ask`: A list of Star Trek captains you want to ask the question to. Make sure to create one line per captain and to provide at least two names.
   * `max_tokens_answer`: The maximum number of tokens available for the answer.
   * `randomness_of_answer`: The randomness of the answer. The value provided is divided by 10 and given to the `temperature` parameter of the [chat completion endpoint](https://platform.openai.com/docs/guides/text-generation/reproducible-outputs). The scale for the param ranges from 0 to 20, with 0 being the most deterministic and 20 being the most random.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the params available for the captains_dag DAG with the default choices." />
   </Frame>

3. After the DAG run completed, go to the `include` folder to view the image file created by the `plot_embeddings` task. The image should look similar to the one below.

   <Frame>
     <img alt="Screenshot of the image created by the plot_embeddings task showing the two dimensional representation of the closeness of answers associated with different Star Trek captains." />
   </Frame>

## Conclusion

Congratulations! You used Airflow and OpenAI to get answers from your favorite Star Trek captains and compare them visually. You can now use Airflow to orchestrate OpenAI operations in your own machine learning pipelines. 🖖


# How to use Otto to automatically investigate Dag failures and PR a fix
Source: https://astronomer.io/docs/learn/airflow-otto-rca-auto-fix



<Info> This feature is in [Labs](/docs/astro/feature-previews). </Info>

[Otto](/docs/astro/otto-overview) is Astronomer's AI agent, specialized in data engineering. Otto helps you write, test, upgrade, and investigate your Airflow Dags running on Astro, and you can interact with Otto from the Astro UI, Astro CLI, and the Astro API.

<img alt="Architecture diagram showing a Dag failure triggering an Astro alert, which starts a second Dag that calls Otto to diagnose the failure and open a GitHub pull request with a fix." />

This diagram shows the architecture you'll implement in this tutorial.

* Setting up an Astro alert that gets triggered on the failure of a Dag and responds by starting a run of the auto fix Dag.
* The auto-fix Dag which uses Otto to investigate the failure having all necessary context about your environment, uses `@task.agent` to create a fix based on Otto's suggestions, and submits a PR to GitHub for your review.

## Assumed knowledge

To get the most out of this tutorial, you should have an understanding of:

* Basic [Airflow concepts](/docs/learn/intro-to-airflow).
* How to use the [GitHub REST API](https://docs.github.com/en/rest).

## Prerequisites

* An API key of an LLM provider that is [compatible with Pydantic AI](https://pydantic.dev/docs/ai/models/overview/). This tutorial uses an OpenAI API key. This API key is needed for the agentic tasks that create the GitHub PR. Otto interacts with models through Astronomer's LLM gateway to run the root-cause-analysis, and does not need an API key.

## Step 0: Sign up for Astro and install the Astro CLI

If you already have an Astro account and the latest version of the [Astro CLI](https://www.astronomer.io/docs/astro/cli) installed, continue with [Step 1](#step-1-create-a-new-astro-deployment).

1. If you don't have an Astro account yet, sign up for a [free trial of Astro](https://www.astronomer.io/lp/signup/), which gives access to Otto.

You do not need to select a template in the onboarding flow. Click **Or skip this and go to your workspace** when asked to select a template.

<img alt="Astro onboarding flow with the option to skip choosing a template and go to your workspace." />

2. Use the following command to install the latest version of the [Astro CLI](https://www.astronomer.io/docs/astro/cli).

```sh theme={null}
curl -sSL install.astronomer.io | sudo bash -s
```

<Note>
  If you cannot install the Astro CLI locally, you can still complete this tutorial by deploying the tutorial repository directly from GitHub using the [GitHub integration](/docs/astro/deploy-github-integration).
</Note>

## Step 1: Create a new Astro Deployment

[Create a new Deployment](/docs/astro/create-deployment). You can choose any settings. Astronomer recommends the `Development` template for testing.

<img alt="Create a new Deployment in the Astro UI." />

## Step 2: Fork and clone the tutorial repository

[Fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) and [clone](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) this [GitHub repository](https://github.com/astronomer/otto-rca-auto-fix-tutorial).

It contains a fully functional Airflow project with two Dags:

* `create_tracking_labels`: This is a test Dag designed to fail with an error for Otto to investigate automatically.
* `otto_rca_to_gh_pr`: This is the Dag that runs the investigation.

<img alt="Graph view of the otto_rca_to_gh_pr Dag in the Airflow UI." />

The `otto_rca_to_gh_pr` Dag consists of six tasks:

* `parse_alert`: Extracts the source Dag ID and run ID from the triggering alert. If the alert message doesn't include a recognizable Dag ID or is missing a run ID, the task raises an error.
* `dedup_check`: Checks GitHub for an existing open pull request labeled with the source Dag ID. If one already exists, the task skips the rest of the Dag run to avoid opening a duplicate fix. This is a simple example of deduplication logic. You might need more complex logic in a production setup specifying the issue to be fixed by task ID and deployment ID in addition to the Dag ID.
* `get_diagnosis`: Requests a root cause diagnosis for the failed Dag run from Otto, using the Astro Organization ID, Deployment ID, and an investigation token.
* `fetch_source_file`: Retrieves the current content of the failing Dag's source file from the base branch of the GitHub repository.
* `propose_fix`: An agent task that sends the diagnosis and current file content to an LLM, which proposes the smallest set of exact search-and-replace edits that resolve the diagnosed root cause.
* `open_pr`: Applies the proposed edits to the source file, formats the result with `ruff`, then creates a branch, commits the change, and opens a labeled GitHub pull request.

You can review the [full Dag code](https://github.com/astronomer/otto-rca-auto-fix-tutorial/blob/main/dags/otto_rca_to_gh_pr.py). The code is modularized, and imports helper functions from the `include` folder. The `request_diagnosis` function in [`include/airflow_rca`](https://github.com/astronomer/otto-rca-auto-fix-tutorial/blob/main/include/airflow_rca.py) shows the interaction with Otto.

<Note>
  The failing Dag and the auto-fix Dag are running in the same Astro Deployment for convenience in the context of the tutorial. You can use the `otto_rca_to_gh_pr` Dag to investigate any Dag failures in any of your Astro Deployments.
</Note>

## Step 3: Deploy the tutorial project

Run the following commands in your repository's root to sign in to Astro and push code to your Astro Deployment, created in [Step 1](#step-1-create-a-new-astro-deployment).

```bash theme={null}
astro login
astro deploy
```

<Note>
  If you cannot install the Astro CLI locally, you can still complete this tutorial by deploying the tutorial repository directly from GitHub using the [GitHub integration](/docs/astro/deploy-github-integration).
</Note>

<Note>
  You can also interact with Otto from the Astro CLI. Just run `astro otto` and ask anything about Airflow, Astro, and your projects.
</Note>

## Step 4: Create a Deployment token

In order for an Astro alert to start a Dag in one of your Deployments, you need to give it a Deployment token.

1. On your Deployment, click **Access**, then **API Tokens**, and **Add API Token**.

<img alt="Access tab of an Astro Deployment showing the API Tokens sub-tab and the Add API Token menu with options to create a new Deployment API token." />

2. Create a new token with the name `AlertToken` of the `Standard` Kind with the `Deployment Admin` Role.

3. Click `Create API Token` and store the token value in a safe location.

## Step 5: Create an Astro alert

In order for any Dag failure to start the investigation Dag, you need to create an [Astro alert](/docs/astro/alerts) with a Dag trigger.

1. In the Astro UI, open the **Alerting** dropdown and click **Alerts** -> **+ New Alert**.

<img alt="Alerts page in the Astro UI showing the navigation to Alerting > Alerts and the New Alert button." />

2. The first part of the alert form describes *when* the alert should fire. Select the `DAG Failure` type, your workspace, and Deployment name. For this tutorial we'll scope the alert to only trigger when the `create_tracking_labels` Dag fails.

<img alt="Astro alert form set to the DAG Failure type with Critical severity, scoped to the Sandbox workspace, Otto RCA tutorial Deployment, and the create_tracking_labels Dag ID." />

<Note>
  Note that if you are running the `otto_rca_to_gh_pr` Dag in the same Deployment as Dags that need to be fixed, be careful about selecting `All DAGs` for the alert. You want to avoid a situation where any failure in the `otto_rca_to_gh_pr` triggers another run of the same, creating an infinite loop.
</Note>

3. Next you need to add the `DAG trigger` notification channel. Click `+ New Notification Channel` and create a channel with the name `otto_rca_to_gh_pr Dag`, the type `DAG trigger`, and aim it at your Deployment and the `otto_rca_to_gh_pr` Dag. Copy your Deployment API token that you saved at the end of [Step 4](#step-4-create-a-deployment-token) into the `DEPLOYMENT API TOKEN` field. Lastly make the channel available to your entire organization, workspace, or just to this tutorial Deployment.

<img alt="New Notification Channel dialog with the DAG Trigger channel type selected, pointed at the Otto RCA tutorial Deployment and the otto_rca_to_gh_pr Dag, with a Deployment API token field and channel availability set to the entire organization." />

4. Click `Create Notification Channel` to create the channel and `Create Alerts` to save the alert.

## Step 6: Create a GitHub token

In order to be able to raise a PR on GitHub, the AI agent needs access to your GitHub repository. For a production use case, you'll likely want to setup a scoped [GitHub App](https://docs.github.com/en/apps/creating-github-apps/about-creating-github-apps/about-creating-github-apps). For this tutorial, you can use a scoped personal access token.

1. Go to your [GitHub account's Developer Settings](https://github.com/settings/personal-access-tokens) and click **Generate new token** to create a new fine-grained token.

<img alt="GitHub Developer Settings showing the Personal access tokens > Fine-grained tokens page and the Generate new token button." />

2. Give your GitHub any name and scope it to your fork of the tutorial repository.

3. Add four permissions: Contents (Read and write), Metadata (Read-only), Pull requests (Read and write), and Issues (Read and write).

<img alt="New fine-grained personal access token scoped to the forked tutorial repository, with Contents (Read and write), Issues (Read-only), Metadata (Read-only), and Pull requests (Read and write) permissions." />

4. Create the token and copy it to a safe location.

## Step 7: Add environment variables and a connection

The last set up step is to add the necessary environment variables and the Airflow connection to your model provider to your Deployment.

1. Go to your Astro Deployment and click **Environment**, then **Environment Variables** and **Edit Deployment Variables**.

<img alt="Environment tab of an Astro Deployment showing the Environment Variables sub-tab and the Edit Deployment Variables button." />

2. Add the following three environment variables, marking all tokens and keys as `SECRET` with the toggle.

   * `ASTRO_API_TOKEN`: You can reuse the same Astro Deployment token you created in step [Step 4](#step-4-create-a-deployment-token). This is the credential the Airflow task uses to authenticate to Astro in order to be able to run the Otto investigation.
   * `GITHUB_REPO`: Your fork of the tutorial repository in the format `<account>/<repo>`.
   * `GITHUB_TOKEN`: Your GitHub token, retrieved in [Step 6](#step-6-create-a-github-token).

3. Click **Update Environment Variables** to save your changes.

4. Still on your Deployment's environment tab, click **Connections** to add your Pydantic AI connection for the `@task.agent` task that drafts your PR.

<img alt="Environment tab of an Astro Deployment showing the Connections sub-tab and the New Connection button." />

5. Select the `Generic` connection form and create a connection to your LLM provider. If you are using OpenAI, add the following values.

   * **Connection ID**: `pydanticai_default`
   * **Connection Type**: `pydanticai`
   * **Password**: Your OpenAI API Key.
   * **Extra**: `{"model":"openai:gpt-5"}`. You can use any valid model for your model provider.

<Note>
  Alternatively, you can use another LLM provider, as long as it is [compatible with Pydantic AI](https://pydantic.dev/docs/ai/models/overview/) and you adjust the extra provided to the `pydantic-ai-slim[<your LLM provider>]` in the `requirements.txt` file for your Astro project. For more options on how to configure the Pydantic AI connection used with the Common AI provider's `@task.agent` decorator, see the [Common AI provider connection documentation](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/connections/pydantic_ai.html).
</Note>

## Step 8: Test the Dag

Now it is time to test this setup!

1. Go to the Airflow UI and make sure both Dags are unpaused.

<img alt="Airflow UI Dags list showing the create_tracking_labels and otto_rca_to_gh_pr Dags with their unpause toggles highlighted." />

2. Run the `create_tracking_labels` Dag manually. It should fail its last task with a `KeyError`.

3. The failing Dag causes the Astro alert to run, and automatically triggers the `otto_rca_to_gh_pr`, which investigates the failure and proposes a fix.

4. Wait for the Dag to finish, then check your GitHub repository for a new PR.

<img alt="Automated GitHub pull request opened by the otto_rca_to_gh_pr Dag, showing Otto's diagnosis of a KeyError in print_tracking_labels caused by a list-mutation-during-iteration bug in drop_cancelled." />

## Conclusion

Congratulations! You've implemented a self-healing Dag using Otto to investigate a failure and AI orchestration with the Common AI provider to create a PR fixing the issue. This is just one of many possible ways you can include Otto in your pipelines. Another example is [to post Otto's diagnosis to Slack](/docs/astro/otto-investigate#example-post-the-diagnosis-to-slack).


# Create and use params in Airflow
Source: https://astronomer.io/docs/learn/airflow-params

Create and use DAG and task-level params in Airflow.

Params are arguments which you can pass to an Airflow DAG or task at runtime and are stored in the [Airflow context dictionary](/docs/learn/airflow-context) for each DAG run. You can pass DAG and task-level params by using the `params` parameter.

Params are ideal to store information that is specific to individual DAG runs like changing dates, file paths or ML model configurations. Params aren't encrypted and therefore not suitable to pass secrets. See also [Best practices for storing information in Airflow](/docs/learn/airflow-variables#best-practices-for-storing-information-in-airflow).

This guide covers:

* How to pass params to a DAG at runtime.
* How to define DAG-level param defaults which are rendered in the **Trigger DAG** UI.
* How to access params in an Airflow task.
* The hierarchy of params in Airflow.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow DAGs. See [Introduction to Airflow DAGs](/docs/learn/dags).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow context. See [Access the Apache Airflow context](/docs/learn/airflow-context).

## Pass params to a DAG run at runtime

Params can be passed to a DAG at runtime in four different ways:

* In the Airflow UI by using the **Trigger DAG** form. This form appears when you click the **Trigger DAG** (**Play**) button in the Airflow UI.
* Running a DAG with the `--conf` flag using the Airflow CLI ([`airflow dags trigger`](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#trigger)).
* Using the `TriggerDagRunOperator` with the `conf` parameter.
* Making a `POST` request to the Airflow REST APIs [Trigger Dag Run](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/trigger_dag_run) endpoint and using the `conf` parameter.

Param values passed to a DAG by any of these methods will override existing default values for the same key as long as the [Airflow core config `dag_run_conf_overrides_params`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-run-conf-overrides-params) is set to `True` (which is the default setting).

<Info>
  You can only pass params to a DAG that are JSON-serializable. Any params that aren't JSON-serializable will cause a DAG Import Error (`ParamValidationError`) upon DAG parsing.
</Info>

### Trigger DAG form

You can pass params to DAGs from the Airflow UI by clicking the **Play** button in the DAGs overview or the blue **Trigger** button on an individual DAG page.

<Frame>
  <img alt="Play button" />
</Frame>

This button opens a form in which you can specify details for the DAG run. If the DAG has any params with defaults values defined, the form will render a field for each those params under **Run Parameters**. Additional params can be added under **Advanced Options** -> **Configuration JSON**.

<Frame>
  <img alt="Trigger DAG form" />
</Frame>

Under **Advanced Options**, you can also set the **Logical Date**, define a **Run ID**, and add a **Dag Run Note** for the manual DAG run. Note that any params already defined under **Run Parameters** will be included in the **Configuration JSON** by default.

<Frame>
  <img alt="Advanced options trigger form" />
</Frame>

In the **Trigger DAG** form:

* You can change any default values for params that have been defined in the DAG file under **Run Parameters** or in the **Configuration JSON**.
* You can add new params under **Advanced Options** -> **Configuration JSON**.
* You can set the **Logical Date** of the DAG run to any date. Note that you can also provide no logical date, by clearing it in the calendar picker.
* You can set the **Run ID** to any string. If no Run ID is specified, Airflow generates one based on the run after date.
* You can add a **Dag Run Note** to the DAG run.

After setting the configuration, you can start the DAG run with the **Trigger** button.

### CLI

When you run an [Airflow DAG from the CLI](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#dags), you can pass params to the DAG run by providing a JSON string to the `--conf` flag. For example, to trigger the `params_default_example` DAG with the value of `Hello from the CLI` for `param1`, run:

<details>
  <summary>Astro</summary>

  Run Airflow commands from the Astro CLI using `astro dev run`:

  ```sh wrap theme={null}
  astro dev run dags trigger params_defaults_example --conf '{"param1" : "Hello from the CLI"}'
  ```
</details>

<details>
  <summary>Airflow</summary>

  ```sh wrap theme={null}
  airflow dags trigger params_defaults_example --conf '{"param1" : "Hello from the CLI"}'
  ```
</details>

The CLI prints the configuration for the triggered run to the command line:

<Frame>
  <img alt="CLI output" />
</Frame>

You can use a `--conf` flag with the following Airflow CLI sub-commands:

* `airflow dags backfill`
* `airflow dags test`
* `airflow dags trigger`

### `TriggerDagRunOperator`

The [`TriggerDagRunOperator`](/docs/learn/cross-dag-dependencies#triggerdagrunoperator) is a core Airflow operator that allows you to start a DAG run from within another DAG. You can use the `TriggerDAGRunOperator` `conf` param to trigger the dependent DAG with a specific configuration.

The DAG below uses the `TriggerDagRunOperator` to trigger the `tdro_example_downstream` DAG while passing a dynamic value for the `upstream_color` param using the `conf` parameter. The value for `upstream_color` is passed using a [Jinja template](/docs/learn/templating) pulling the return value of an upstream task using [XCom](/docs/learn/airflow-passing-data-between-tasks#xcom).

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from pendulum import datetime
  from airflow.decorators import dag, task
  from airflow.operators.trigger_dagrun import TriggerDagRunOperator
  import random


  @dag(
      start_date=datetime(2023, 6, 1),
      schedule="@daily",
      catchup=False,
  )
  def tdro_example_upstream():
      @task
      def choose_color():
          color = random.choice(["blue", "red", "green", "yellow"])
          return color

      tdro = TriggerDagRunOperator(
          task_id="tdro",
          trigger_dag_id="tdro_example_downstream",
          conf={"upstream_color": "{{ ti.xcom_pull(task_ids='choose_color')}}"},
      )

      choose_color() >> tdro


  tdro_example_upstream()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  from pendulum import datetime
  from airflow.decorators import dag, task
  from airflow.operators.trigger_dagrun import TriggerDagRunOperator
  from airflow.operators.python import PythonOperator
  import random


  def choose_color_func():
      color = random.choice(["blue", "red", "green", "yellow"])
      return color


  @dag(
      start_date=datetime(2023, 6, 1),
      schedule="@daily",
      catchup=False,
  )
  def tdro_example_upstream_traditional():
      choose_color = PythonOperator(
          task_id="choose_color",
          python_callable=choose_color_func,
      )

      tdro = TriggerDagRunOperator(
          task_id="tdro",
          trigger_dag_id="tdro_example_downstream",
          conf={"upstream_color": "{{ ti.xcom_pull(task_ids='choose_color')}}"},
      )

      choose_color >> tdro


  tdro_example_upstream_traditional()
  ```
</details>

Runs of the `tdro_example_downstream` DAG that are triggered by this upstream DAG will override the default value of the `upstream_color` param with the value passed using the `conf` parameter, which leads to the `print_color` task to print either `red`, `green`, `blue` or `yellow`.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from pendulum import datetime
  from airflow.decorators import dag, task


  @dag(
      start_date=datetime(2023, 6, 1),
      schedule=None,
      catchup=False,
      params={"upstream_color": "Manual run, no upstream color available."},
  )
  def tdro_example_downstream():
      @task
      def print_color(**context):
          print(context["params"]["upstream_color"])

      print_color()


  tdro_example_downstream()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from pendulum import datetime
  from airflow.decorators import dag
  from airflow.operators.python import PythonOperator


  def print_color_func(**context):
      print(context["params"]["upstream_color"])


  @dag(
      start_date=datetime(2023, 6, 1),
      schedule=None,
      catchup=False,
      params={"upstream_color": "Manual run, no upstream color available."},
  )
  def tdro_example_downstream_traditional():
      PythonOperator(
          task_id="print_color",
          python_callable=print_color_func,
      )


  tdro_example_downstream_traditional()
  ```
</details>

## Define DAG-level param defaults

To specify params for all runs of a given DAG, pass default values to the `param` parameter of the `@dag` decorator or the `DAG` class in your DAG file. You can directly specify a default value or use the `Param` class to define a default value with additional attributes.

The DAG below has two DAG-level params with defaults: `param1` and `param2`, the latter only accepting integers.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from pendulum import datetime
  from airflow.decorators import dag, task
  from airflow.models.param import Param


  @dag(
      start_date=datetime(2023, 6, 1),
      schedule=None,
      catchup=False,
      params={
          "param1": "Hello!",
          "param2": Param(
              23,
              type="integer",
          ),
      },
  )
  def simple_param_dag():
      @task
      def print_all_params(**context):
          print(context["params"]["param1"] * 3)
          print(context["params"]["param2"])

      print_all_params()


  simple_param_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from pendulum import datetime
  from airflow import DAG
  from airflow.operators.python import PythonOperator
  from airflow.models.param import Param


  def print_all_params_func(**context):
      print(context["params"]["param1"] * 3)
      print(context["params"]["param2"])


  with DAG(
      dag_id="simple_param_dag",
      start_date=datetime(2023, 6, 1),
      schedule=None,
      catchup=False,
      params={
          "param1": "Hello!",
          "param2": Param(
              23,
              type="integer",
          ),
      },
  ):
      PythonOperator(
          task_id="print_all_params",
          python_callable=print_all_params_func,
      )
  ```
</details>

If you define DAG-level param defaults, the **Trigger DAG** form renders a field for each param. From this UI, you can then override your defaults for individual DAG runs. A param with a red asterisk is a required param.

<Frame>
  <img alt="Trigger DAG with simple defaults" />
</Frame>

<Info>
  When you specify a required `type` for a param, the field will be a required input by default because of [JSON validation](https://json-schema.org/draft/2020-12/json-schema-validation.html#name-dates-times-and-duration). To make a field optional but still require a specific input type, allow NULL values by setting the type to `["null", "<my_type>"]`.
</Info>

<Info>
  If you don't specify a type for your param, Airflow will infer it based on the default value you provide.
</Info>

### Param types

The following param types are supported:

* `string`: A string. This is the default type.
* `null`: Allows the param to be None by being left empty.
* `integer`: An integer.
* `number`: A float (or integer).
* `boolean`: `True` or `False`.
* `array`: An HTML multi line text field, every line edited will be made into a string array as the value.
* `object`: A JSON entry field.

### Param attributes

Aside from the `type` attribute, the `Param` class has several other attributes that you can use to define how users interact with the param:

* `title`: The title of the param that appears in the **Trigger DAG** UI.
* `description`: A description of the param.
* `section`: Creates a section under which the param will appear in the **Trigger DAG** UI. All params with no specified section will appear under the default section **DAG conf Parameters**.
* `format`: A [JSON format](https://json-schema.org/draft/2020-12/json-schema-validation.html#name-dates-times-and-duration) that Airflow will validate a user's input against. Airflow 3.3+ allows you to specify `format="duration"` to pass an ISO 8601 duration.
* `enum`: A list of valid values for a param. Setting this attribute creates a dropdown menu in the UI.
* `const`: Defines a permanent default value and hides the param from the **Trigger DAG** UI. Note that you still need to provide a `default` value for the param.

All `Param` attributes are optional to set. For string type params, you can additionally set `minLength` and `maxLength` to define the minimum and maximum length of the input. Similarly, integer and number type params can have a `minimum` and `maximum` value.

### Param examples in the Airflow UI

This section presents a few examples of params and how they are rendered in the **Trigger DAG** UI.

The code snippet below defines a mandatory string param with a few UI elements to help users input a value.

```python wrap theme={null}
from airflow.sdk import Param

"my_string_param": Param(
    "Airflow is awesome!",
    type="string",
    title="Favorite orchestrator:",
    description="Enter your favorite data orchestration tool.",
    section="Important params",
    minLength=1,
    maxLength=200,
)
```

<Frame>
  <img alt="String param example" />
</Frame>

When you define [date, datetime, or time param](https://datatracker.ietf.org/doc/html/rfc3339#section-5.6), a calendar picker appears in the **Trigger DAG** UI.

```python wrap theme={null}
from airflow.sdk import Param

"my_datetime_param": Param(
    "2016-10-18T14:00:00+00:00",
    type="string",
    format="date-time",
),
```

<Frame>
  <img alt="Datetime param example" />
</Frame>

Providing a list of values to the `enum` attribute will create a dropdown menu in the **Trigger DAG** UI. Note that the default value must also be in the list of valid values provided to `enum`. Due to JSON validation rules, a value has to be selected.

```python wrap theme={null}
from airflow.sdk import Param

"my_enum_param": Param(
    "Hi :)", type="string", enum=["Hola :)", "Hei :)", "Bonjour :)", "Hi :)"]
),
```

<Frame>
  <img alt="Enum param example" />
</Frame>

A boolean type param will create a toggle in the **Trigger DAG** UI.

```python wrap theme={null}
from airflow.sdk import Param

"my_bool_param": Param(True, type="boolean"),
```

<Frame>
  <img alt="Bool param example" />
</Frame>

An array type param will create a multi-line text field in the **Trigger DAG** UI. Each line will be converted to an item in the array.

```python wrap theme={null}
from airflow.sdk import Param

"my_array_param": Param(["Hello Airflow", ":)"], type="array"),
```

<Frame>
  <img alt="Array param example" />
</Frame>

An object type param will create a JSON entry field in the **Trigger DAG** UI. The value of the param must be a valid JSON object.

```python wrap theme={null}
from airflow.sdk import Param

"my_object_param": Param({"a": 1, "b": 2}, type="object"),
```

<Frame>
  <img alt="Object param example" />
</Frame>

## Define task-level param defaults

You can set task-level param defaults in the same way as for DAG-level params. If a param of the same key is specified at both the DAG and task level, the DAG-level param will take precedence.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  @task(params={"param1": "Hello World!"})
  def t1(**context):
      print(context["params"]["param1"])
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  t1 = BashOperator(
      task_id="t1",
      bash_command="echo {{ params.param1 }}",
      params={"param1": "Hello World!"},
  )
  ```
</details>

## Access params in a task

You can access params in an Airflow task like you can with other elements in the [Airflow context](/docs/learn/airflow-context).

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import task

  @task
  def t1(**context):
      print(context["params"]["my_param"])
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.providers.standard.operators.python import PythonOperator

  def t1_func(**context):
      print(context["params"]["my_param"])

  t1 = PythonOperator(
      task_id="t1",
      python_callable=t1_func,
  )
  ```
</details>

Params are also accessible as a [Jinja template](/docs/learn/templating) using the `{{ params.my_param }}` syntax.

If you try to access a param that hasn't been specified for a specific DAG run, the task will fail with an exception.

## Param precedence

The order of precedence for params, with the first item taking most precedence, is as follows:

* Params that have been provided for a specific DAG run by a method detailed in [pass params to a DAG run at runtime](#pass-params-to-a-dag-run-at-runtime) as long as the [Airflow config core.`dag_run_conf_overrides_params`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-run-conf-overrides-params) is set to `True`.
* Param defaults that have been defined at the DAG level.
* Param defaults that have been defined at the task level.


# Partitioned Dag runs and asset events in Apache Airflow®
Source: https://astronomer.io/docs/learn/airflow-partitioned-runs

Partition Dag runs and asset events in Airflow.

Airflow 3.2 introduced the concept of partitioned Dag runs and partitioned asset events. A partitioned Dag run is a Dag run with a `partition_key` attached to it, and a partitioned asset event is an asset event that has a `partition_key` attached to it. Any string can be a partition key, with time-based partition keys being the most common. Partition keys can be used in tasks in a partitioned Dag run to partition data, for example in a SQL statement.

In this guide, you'll learn:

* When to use partitioned Dag runs and asset events.
* How to create a partitioned Dag run.
* How to create a partitioned asset event.
* How to schedule a Dag based on partitioned asset events.

## Assumed knowledge

To get the most out of this guide, you should have an existing knowledge of:

* Airflow scheduling concepts. See [Schedule Dags in Airflow](/docs/learn/scheduling-in-airflow).
* Basic asset-based scheduling. See [Basic asset-based scheduling in Apache Airflow®](/docs/learn/airflow-datasets).

## When to use partitions

Situations in which you should consider using partitioned Dag runs and asset events are:

* When you want to process data from a specific time period in every Dag run. For example, if you have a Dag that runs once a day and should always process the data from the previous day.
* When you have a Dag that is scheduled based on asset events, within which you want to process data from a specific time period. You can use the `partition_key` to partition the data inside of the downstream Dag run. The `partition_key` propagates from the upstream Dag run to the downstream Dag run, and you can adjust its grain with [partition key mappers](#partition-key-mappers).
* When you want to process data for a specific segment in manual or API-triggered Dag runs. For example, if your Dag generates a report for a specific department, you can set a different `partition_key` for each Dag run in the Trigger Dag run config or the API request body.

## Create a partitioned Dag run

A partitioned Dag run is a Dag run with a `partition_key` attached to it. There are four ways to create a partitioned Dag run:

* By running a Dag manually in the Airflow UI and attaching a `partition_key` in the Trigger Dag run config.

  <Frame>
    <img alt="Screenshot of the Airflow UI showing the Trigger Dag run config with the partition key input field." />
  </Frame>

* By running a Dag using the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/trigger_dag_run) and attaching a `partition_key` in the request body.

* By fulfilling the asset schedule condition of a Dag that uses the `PartitionedAssetTimetable` timetable.

* By using the `CronPartitionTimetable` timetable. Note that only Dag runs of the type `scheduled` and [`backfill`](/docs/learn/rerunning-dags#backfill) are partitioned, manual runs aren't, unless you are providing a `partition_key` in the Trigger Dag run config.

<Tip>
  You can access partition keys from within any task in a partitioned Dag run, see [Access partition keys](#access-partition-keys) for more information.
</Tip>

### CronPartitionTimetable

The `CronPartitionTimetable` is a timetable that creates partitioned Dag runs with an automatic `partition_key` attached that is based on the `run_after` timestamp of each scheduled and backfilled Dag run.

```python wrap theme={null}
from airflow.sdk import CronPartitionTimetable

@dag(
    schedule=CronPartitionTimetable("0 0 * * *", timezone="UTC"),
)
```

You can offset the partition key by providing the `run_offset` parameter to the `CronPartitionTimetable` instance. The offset is relative to the cron expression, for example if your Dag runs once every hour, a `run_offset` of `-12` will offset the partition key by 12 hours. The Dag run with the run id `2026-03-16T09:00:00` will have a partition key of `2026-03-15T21:00:00`.

```python wrap theme={null}
@dag(
    schedule=CronPartitionTimetable("0 * * * *", timezone="UTC", run_offset=-12),
)
```

## Create a partitioned asset event

A partitioned asset event is an asset event that has a `partition_key` attached to it. There are four ways to create a partitioned asset event:

* By updating an asset manually in the Airflow UI and providing a `partition_key` in the Asset Event creation dialog.

  <Frame>
    <img alt="Screenshot of the Airflow UI showing the Asset Event creation dialog with the partition key input field." />
  </Frame>

* By updating an asset using the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/create_asset_event) and providing a `partition_key` in the request body.

* By updating an asset using the `outlets` parameter of a task in a Dag that is scheduled using a `CronPartitionTimetable` timetable.

* (Airflow 3.3+) By updating an asset using the `outlets` parameter of a task and using the `.add_partitions` method on the asset event object inside the task to add one or more `partition_key` values to the asset event. See [Create partitions in a task](#create-partitions-in-a-task).

<Note>
  Partitioned asset events created by a task in a Dag scheduled with a `CronPartitionTimetable` or by using the `add_partitions` method are intended for partition-aware downstream scheduling, and don't trigger non-partition-aware Dags.
</Note>

The example below shows a Dag that is scheduled using a `CronPartitionTimetable` timetable. In every *scheduled* or *backfilled* run of this Dag, successful completion of the `my_task_partitioned_upstream` task will create a partitioned asset event for the `my_partitioned_asset` asset. The `partition_key` will be the `run_after` timestamp of the Dag run.

```python wrap theme={null}
from airflow.sdk import dag, task, Asset, CronPartitionTimetable


my_asset = Asset("my_partitioned_asset")


@dag(
    schedule=CronPartitionTimetable("0 0 * * *", timezone="UTC"),
)
def my_dag_partitioned_upstream():
    @task(outlets=[my_asset])
    def my_task_partitioned_upstream(**context):
        pass

    my_task_partitioned_upstream()


my_dag_partitioned_upstream()
```

### Create partitions in a task

In Airflow 3.3+ you can assign `partition_key` values at runtime from within your task code, using the `.add_partitions` method of the asset event fetched from the Airflow context.

The example Dag below contains one task (`my_task`) that updates one asset `my_partition_at_runtime_asset`, creating 3 asset events at runtime, each with a different partition key. In this example the task chooses 3 random values from the list of departments.

```python expandable wrap theme={null}
import random

from airflow.sdk import (
    Asset,
    dag,
    task
)

data_ready = Asset("my_partition_at_runtime_asset")

DEPARTMENTS = [
    "Engineering",
    "Product",
    "Legal",
    "Sales",
    "Marketing",
    "Finance",
    "HR",
    "Operations",
]


@dag
def my_partition_at_runtime_dag():

    @task(outlets=[data_ready])
    def my_task(**context):
        context["outlet_events"][data_ready].add_partitions(
            random.sample(DEPARTMENTS, 3)
        )

    my_task()

my_partition_at_runtime_dag()
```

In the **Asset Events** tab of the task instance you can see 3 asset events for the `my_partition_at_runtime_asset` were created, each with a different partition key value.

<Frame>
  <img alt="Screenshot of the Airflow UI showing the Asset Events tab of a task instance with three created asset events, each with a partition key." />
</Frame>

A Dag scheduled to run based on partitioned asset events of the `my_partition_at_runtime_asset` asset will start 3 runs, one for each partition key.

<Frame>
  <img alt="Screenshot of the Airflow UI showing an asset-triggered downstream Dag run with the mapped partition key Product and its source asset event." />
</Frame>

## Schedule on a partitioned asset

To schedule a Dag on a partitioned asset, you set its `schedule` parameter to an instance of `PartitionedAssetTimetable`.

```python wrap theme={null}
from airflow.sdk import dag, PartitionedAssetTimetable, Asset

@dag(schedule=PartitionedAssetTimetable(assets=Asset("my_partitioned_asset")))
```

This Dag will run whenever the `my_partitioned_asset` is updated by a *partitioned* asset event. It won't run based on regular asset events produced for the `my_partitioned_asset` asset.

You can modify the grain of the partition key by providing a partition key mapper to the `PartitionedAssetTimetable` instance.
For example, to partition the data by day, you can use the `StartOfDayMapper` to normalize the partition key to the day in the format `YYYY-MM-DD`. See [Partition key mappers](#partition-key-mappers) for more information on the available partition key mappers.

```python wrap theme={null}
from airflow.sdk import dag, PartitionedAssetTimetable, Asset, StartOfDayMapper

my_partitioned_asset = Asset("my_partitioned_asset")

@dag(
    schedule=PartitionedAssetTimetable(
        assets=my_partitioned_asset,
        partition_mapper_config={my_partitioned_asset: StartOfDayMapper()}
    )
)
def my_dag():

    @task 
    def my_task(**context):
        print(context["dag_run"].partition_key) # will print the partition key in the format `YYYY-MM-DD`

    my_task()

my_dag()
```

### Combined partitioned asset schedules

You can combine multiple assets in a single `PartitionedAssetTimetable` instance to create a composite asset schedule using the same logical expressions (AND (`&`) plus OR (`|`)) as when creating a [conditional asset schedule](/docs/learn/airflow-advanced-asset-scheduling#conditional-asset-scheduling) with regular assets.

```python wrap theme={null}
from airflow.sdk import dag, PartitionedAssetTimetable, Asset

my_combined_asset_one = Asset("my_combined_asset_one")
my_combined_asset_two = Asset("my_combined_asset_two")


@dag(
    schedule=(
        PartitionedAssetTimetable(assets=(my_combined_asset_one & my_combined_asset_two))
    )
)
def my_combined_partitioned_asset_dag():
    @task
    def my_task(**context):
        partition_key = context["dag_run"].partition_key
        print(partition_key)

    my_task()


my_combined_partitioned_asset_dag()
```

If one of the assets is updated with a partitioned asset event, a pending run of this Dag will be created. Pending runs are visible in the Airflow UI by clicking on its schedule.

<Frame>
  <img alt="Screenshot of the Airflow UI showing a pending run of a Dag with a composite asset schedule." />
</Frame>

If several assets have been updated with a partitioned asset event of different partition keys, several pending runs are created, each with a different partition key. Each pending run has a visualization of which assets have been updated for that specific partition key and which assets are still pending.

<Frame>
  <img alt="Screenshot of the Airflow UI showing multiple pending runs of a Dag with a composite asset schedule." />
</Frame>

A run of this Dag is only triggered when both `my_combined_asset_one` and `my_combined_asset_two` are updated with a partitioned asset event sharing the *same* partition key. Note that queued partitioned asset events for other partition keys are *not* reset by this.

You can use different partition key mappers for each asset in the `PartitionedAssetTimetable`, see [Partition key mappers](#partition-key-mappers).

## Partition keys

Partition keys are strings attached to partitioned Dag runs and asset events. You can use them in tasks in a partitioned Dag run to partition data, for example in a SQL statement.

### Access partition keys

The `partition_key` can be accessed inside any task in a partitioned Dag run from within the [Airflow context](/docs/learn/airflow-context) or by using [Jinja templating](/docs/learn/templating).

```python wrap theme={null}
# from airflow.sdk import task
# from airflow.providers.standard.operators.bash import BashOperator

@task
def print_partition_key(**context):
    print(context["partition_key"])


BashOperator(
    task_id="print_partition_key",
    bash_command="echo {{ partition_key }}",
)
```

One of the most common use cases is to use the partition key in a SQL statement to partition the data used in a specific Dag run. For example, in a Dag that runs once a day, you can use the partition key to select the data for the previous day.

```python wrap theme={null}
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator

SQLExecuteQueryOperator(
    task_id="execute_query",
    conn_id="my_snowflake_conn",
    sql="""
    SELECT * FROM my_table 
    WHERE 
        my_timestamp >= DATEADD(day, -1, '{{ partition_key }}'::DATE) 
        AND my_timestamp < '{{ partition_key }}'::DATE
    ;""",
)
```

In Airflow 3.3+ you can also directly access `partition_date` in partitioned Dag runs to format the date directly, for example with `ds` to convert the full key into `yyyy-mm-dd` format.

```python wrap theme={null}
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator

SQLExecuteQueryOperator(
    task_id="execute_query",
    conn_id="my_snowflake_conn",
    sql="""
    SELECT * FROM my_table 
    WHERE 
        my_timestamp >= DATEADD(day, -1, '{{ partition_date | ds }}') 
        AND my_timestamp < '{{ partition_date | ds }}'
    ;""",
)
```

## Partition key mappers

Partition key mappers are used to modify the partition key of a Dag run. You can use them to change the grain of the partition key, to map composite keys segment by segment, to create one-to-many or many-to-one patterns, or to validate that keys are in a fixed allow-list.

### One-to-one mappers

The following one-to-one partition key mappers are available:

* `IdentityMapper`: keeps keys unchanged. Default mapper.
* Temporal mappers change the grain of the partition key:
  * `StartOfHourMapper`: normalizes time keys to the hour in the format `YYYY-MM-DDTHH`, for example the partition key `2026-03-16T09:37:51` is mapped to `2026-03-16T09`.
  * `StartOfDayMapper`: normalizes time keys to the day in the format `YYYY-MM-DD` (`2026-03-16T09:37:51` -> `2026-03-16`)
  * `StartOfWeekMapper`: normalizes time keys to the week in the format `YYYY-MM-DD (W%V)` (`2026-03-16T09:37:51` -> `2026-03-16 (W12)`)
  * `StartOfMonthMapper`: normalizes time keys to the month in the format `YYYY-MM` (`2026-03-16T09:37:51` -> `2026-03`)
  * `StartOfQuarterMapper`: normalizes time keys to the quarter in the format `YYYY-Q<n>` (`2026-03-16T09:37:51` -> `2026-Q1`). The quarters are based on the calendar year, Q1 starts in January, Q2 in April, Q3 in July, and Q4 in October.
  * `StartOfYearMapper`: normalizes time keys to the year in the format `YYYY` (`2026-03-16T09:37:51` -> `2026`).
* `ProductMapper`: maps composite keys segment by segment, applying one mapper per segment and then rejoining the mapped segments. For example, with the key `Finance|2026-03-16T09:00:00`, `ProductMapper(IdentityMapper(), StartOfDayMapper())` produces `Finance|2026-03-16`. See [Composite partition keys](#composite-partition-keys).
* `AllowedKeyMapper`: validates that keys are in a fixed allow-list and passes the key through unchanged if valid. For example, `AllowedKeyMapper(["Marketing", "Finance", "Sales"])` accepts only those department keys and rejects all others.

You can also change the default partition key mapper for all assets in the `PartitionedAssetTimetable` by providing a `default_partition_mapper` parameter.

```python wrap theme={null}
from airflow.sdk import dag, PartitionedAssetTimetable, Asset, StartOfDayMapper

@dag(
    schedule=PartitionedAssetTimetable(
        assets=Asset("my_partitioned_asset"),
        default_partition_mapper=StartOfDayMapper(),
    )
) 
```

To override the default partition key mapper for a specific asset, you can set the `partition_mapper_config` parameter of the `PartitionedAssetTimetable` instance to a dictionary of asset instances and partition key mappers.

```python wrap theme={null}
from airflow.sdk import dag, PartitionedAssetTimetable, Asset, StartOfDayMapper, StartOfWeekMapper

@dag(
    schedule=PartitionedAssetTimetable(
        assets=Asset("my_partitioned_asset"),
        default_partition_mapper=StartOfDayMapper(),
        partition_mapper_config={
            Asset("my_partitioned_asset"): StartOfWeekMapper(),
        },
    )
) 
```

You can use different partition key mappers for each asset in the `PartitionedAssetTimetable`.

```python wrap theme={null}
from airflow.sdk import dag, PartitionedAssetTimetable, Asset, StartOfQuarterMapper, StartOfWeekMapper

@dag(
    schedule=PartitionedAssetTimetable(
        assets=(my_combined_asset_one & my_combined_asset_two),
        partition_mapper_config={
            my_combined_asset_one: StartOfQuarterMapper(),
            my_combined_asset_two: StartOfWeekMapper(),
        },
    )
) 
```

<Note>
  When chaining several Dags with a partitioned asset schedule, the partition key mappers need to be identical for all Dags after the first one in the chain. For example a Dag which uses a `StartOfDayMapper` will fail the task producing to the next asset in the chain if the next Dag in the chain uses a `StartOfWeekMapper`.
</Note>

### Composite partition keys

Composite partition keys are partition keys that are composed of multiple segments, separated by `|` delimiters. For example, the partition key `Finance|2026-03-16T09:00:00|Revenue` is a composite partition key with three segments: `Finance`, `2026-03-16T09:00:00`, and `Revenue`.

You can use the `ProductMapper` partition key mapper to map composite keys segment by segment, applying one mapper per segment and then rejoining the mapped segments. For example, with the key `Finance|2026-03-16T09:00:00`, `ProductMapper(IdentityMapper(), StartOfDayMapper())` produces `Finance|2026-03-16`.

```python wrap theme={null}
from airflow.sdk import dag, task, PartitionedAssetTimetable, Asset, ProductMapper, IdentityMapper, StartOfDayMapper, AllowedKeyMapper

@dag(
    schedule=PartitionedAssetTimetable(
        assets=Asset("my_partitioned_asset"),
        partition_mapper_config={
            Asset("my_partitioned_asset"): ProductMapper(IdentityMapper(), StartOfDayMapper(), AllowedKeyMapper(["Revenue", "ARR"])),
        },
    )
) 
def my_composite_dag():

    @task
    def my_task(**context):
        partition_key = context["dag_run"].partition_key
        print(partition_key) # prints the partition key in the format `Finance|2026-03-16|Revenue`

    my_task()


my_composite_dag()
```

<Note>
  The given composite partition key needs to match the number of segments in the `ProductMapper` instance and needs to be valid for all mappers in the `ProductMapper` instance in order to trigger a Dag run. Invalid composite partition keys cause an error.
</Note>

### One-to-many: `FanOutMapper`

The `FanOutMapper`, introduced in Airflow 3.3, creates multiple downstream Dag runs based on one partitioned asset event.

The `FanOutMapper` uses an `upstream_mapper` to normalize the upstream key to its period start, a `window` instance that enumerates the elements in that period, and an optional `downstream_mapper` to format each element.

The available windows are `HourWindow`, `DayWindow`, `WeekWindow`, `MonthWindow`, `QuarterWindow`, and `YearWindow`. Each window provides a default `downstream_mapper`, except `HourWindow` and custom windows, for which you need to set a `downstream_mapper` explicitly.

For example, if your asset `my_asset` gets updated once a day and you'd like to create 24 Dag runs, each Dag run processing the data for one hour in the day, you can use the `FanOutMapper` with the `upstream_mapper=StartOfDayMapper()`, which normalizes the often irregular partition key (for example: `2026-06-22T03:19:23`) to the start of the day (`2026-06-22`) and then use the `DayWindow` to fan out to create a set of 24 Dag runs, with hourly partition keys (`2026-06-22T00`, `2026-06-22T01`, ..., `2026-06-22T23`).

```python wrap theme={null}
from airflow.sdk import dag, PartitionedAssetTimetable, Asset, FanOutMapper, StartOfDayMapper, DayWindow

@dag(
    schedule=PartitionedAssetTimetable(
        assets=Asset("my_asset"),
        default_partition_mapper=FanOutMapper(
            upstream_mapper=StartOfDayMapper(),
            window=DayWindow(),
        ),
    )
)
```

<Note>
  By default the maximum number of partitions a `FanOutMapper` can create is `1000`. You can adjust this setting for your entire Airflow instance with the [`AIRFLOW__SCHEDULER__PARTITION_MAPPER_MAX_DOWNSTREAM_KEYS`](http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/configurations-ref.html#partition-mapper-max-downstream-keys) configuration variable, and override it for individual `FanOutMapper` instances with the `max_downstream_keys` parameter. If the number of partitioned Dag runs that would be created by a partitioned asset event exceed `max_downstream_keys`, *no* Dag runs are created for that partitioned asset event and the audit logs of the *upstream* Dag log a `partition fan-out exceeded` event.
</Note>

### Many-to-one: `RollupMapper`

The `RollupMapper`, introduced in Airflow 3.3, is the inverse of the `FanOutMapper`. It maps multiple upstream partition keys to one downstream partition key, creating a single downstream Dag run once all expected upstream keys have arrived. It takes an `upstream_mapper` to normalize each upstream key to the downstream grain and a `window` instance that declares the full set of upstream keys required for a given downstream key.

For example, if `my_asset` gets updated every hour and you want a downstream Dag to run as soon as all 24 partitions for a day have been updated, you can use the `RollupMapper` with the `StartOfDayMapper` as `upstream_mapper` and the `DayWindow` as the `window`.

```python wrap theme={null}
from airflow.sdk import dag, PartitionedAssetTimetable, Asset, RollupMapper, StartOfDayMapper, DayWindow

@dag(
    schedule=PartitionedAssetTimetable(
        assets=Asset("my_partitioned_asset"),
        default_partition_mapper=RollupMapper(
            upstream_mapper=StartOfDayMapper(),
            window=DayWindow(),
        ),
    )
)
```

The Airflow UI displays how many and which of the partitions have arrived.

<Frame>
  <img alt="Screenshot of the Airflow UI showing the many_to_one_downstream Dag scheduled on the many_to_one_asset with 18 of 24 required partition keys arrived." />
</Frame>

By default the downstream Dag waits for all keys in the window to arrive. This is the `wait_policy=WaitForAll()`. You can also set `wait_policy=MinimumCount(n=5)`, which means the downstream Dag runs as soon as 5 partition keys for the window have arrived.

#### `SegmentWindow`

The `RollupMapper` also supports categorical rollups, where a fixed set of string segment keys roll up into one downstream partition key. Pair a `FixedKeyMapper`, which collapses every upstream key onto one fixed downstream key, with a `SegmentWindow`, which declares the set of segment keys that make up the downstream partition. The downstream Dag run is triggered once every segment key in the window has arrived. For example, in the code sample below, the `Legal` and `Sales` segment keys roll up into one `legal_and_sales` downstream Dag run.

```python wrap theme={null}
from airflow.sdk import dag, PartitionedAssetTimetable, Asset, RollupMapper, FixedKeyMapper, SegmentWindow

@dag(
    schedule=PartitionedAssetTimetable(
        assets=Asset("my_partitioned_asset"),
        default_partition_mapper=RollupMapper(
            upstream_mapper=FixedKeyMapper("legal_and_sales"),
            window=SegmentWindow(["Legal", "Sales"]),
        ),
    )
)
```


# Pass data between tasks
Source: https://astronomer.io/docs/learn/airflow-passing-data-between-tasks

Learn more about the most common methods to implement data sharing between your Airflow tasks, including an in-depth explanation of XCom.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

Sharing data between tasks is a very common use case in Airflow. If you've been writing DAGs, you probably know that breaking them up into smaller tasks is a best practice for debugging and recovering quickly from failures. What do you do when one of your downstream tasks requires metadata about an upstream task, or processes the results of the task immediately before it?

There are a few methods you can use to implement data sharing between your Airflow tasks. In this guide, you'll walk through the two most commonly used methods, learn when to use them, and use some example DAGs to understand how they can be implemented.

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Astronomer Academy: [Airflow: XComs 101](https://academy.astronomer.io/path/airflow-101/astro-runtime-xcoms-101) module.
  * Webinar: [How to pass data between your Airflow tasks](https://www.astronomer.io/events/webinars/how-to-pass-data-between-your-airflow-tasks/).
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* DAG writing best practices. See [DAG writing best practices in Apache Airflow](/docs/learn/dag-best-practices).

## Best practices

Before you dive into the specifics, there are a couple of important concepts to understand before you write DAGs that pass data between tasks.

### Ensure idempotency

An important concept for any data pipeline, including an Airflow DAG, is [idempotency](https://en.wikipedia.org/wiki/Idempotence). This is the property whereby an operation can be applied multiple times without changing the result. This concept is often associated with your entire DAG. If you execute the same DAGRun multiple times, you will get the same result. However, this concept also applies to tasks within your DAG. If every task in your DAG is idempotent, your full DAG is idempotent as well.

When designing a DAG that passes data between tasks, it's important that you ensure that each task is idempotent. This helps with recovery and ensures no data is lost if a failure occurs.

### Consider the size of your data

Knowing the size of the data you are passing between Airflow tasks is important when deciding which implementation method to use. As you'll learn, XComs are one method of passing data between tasks, but they are only appropriate for small amounts of data. Large data sets require a method making use of intermediate storage and possibly utilizing an external processing framework.

## XCom

The first method for passing data between Airflow tasks is to use XCom, which is a key Airflow feature for sharing task data.

### What is XCom

[XCom](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html) is a built-in Airflow feature. XComs allow tasks to exchange task metadata or small amounts of data. They are defined by a key, value, and timestamp.

XComs can be "pushed," meaning sent by a task, or "pulled," meaning received by a task. When an XCom is pushed, it is stored in the Airflow metadata database and made available to all other tasks. Any time a task returns a value (for example, when your Python callable for your [`PythonOperator`](https://airflow.apache.org/registry/providers/standard#standard-python-PythonOperator) has a return), that value is automatically pushed to XCom. Tasks can also be configured to push XComs by calling the `xcom_push()` method. Similarly, `xcom_pull()` can be used in a task to receive an XCom.

You can view your XComs in the Airflow UI by going to **Admin** > **XComs**. You should see something like this:

<Frame>
  <img alt="XCom UI" />
</Frame>

### When to use XComs

XComs should be used to pass small amounts of data between tasks. For example, task metadata, dates, model accuracy, or single value query results are all ideal data to use with XCom.

While you can technically pass large amounts of data with XCom, be very careful when doing so and consider using [a custom XCom backend](/docs/learn/custom-xcom-backend-strategies) and [scaling your Airflow resources](/docs/learn/airflow-scaling-workers).

When you use the standard XCom backend, the size-limit for an XCom is determined by your metadata database. Common sizes are:

* Postgres: 1 GB
* SQLite: 2 GB
* MySQL: 64 KB

You can see that these limits aren't very big. If you think your data passed through XCom might exceed the size of your metadata database, either use a custom XCom backend or [intermediary data storage](#intermediary-data-storage).

The second limitation in using the standard XCom backend is that only certain types of data can be serialized.

By default, Airflow supports serializations for:

* [JSON](https://www.json.org/json-en.html).
* [pandas DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) (Airflow version 2.6+).
* [Delta Lake tables](https://delta.io/) (Airflow version 2.8+).
* [Apache Iceberg tables](https://iceberg.apache.org/) (Airflow version 2.8+).

If you need to serialize other data types you can do so using a [custom XCom backend](/docs/learn/custom-xcom-backend-strategies).

### Custom XCom backends

Using a [custom XCom backend](/docs/learn/custom-xcom-backend-strategies) means you can push and pull XComs to and from an external system such as S3, GCS, or HDFS rather than the default of Airflow's metadata database. You can also implement your own serialization and deserialization methods to define how XComs are handled. To learn how to implement a custom XCom backend using Amazon S3, Google Cloud Storage or Azure blob Storage, follow this [step-by-step tutorial](/docs/learn/custom-xcom-backends-tutorial).

### Example DAG using XComs

In this section, you'll review a DAG that uses XCom to pass data between tasks. The DAG uses XComs to analyze cat facts that are retrieved from an API. To implement this use case, the first task makes a request to the [cat facts API](http://catfact.ninja/fact) and pulls the `fact` parameter from the results. The second task takes the results from the first task and performs an analysis. This is a valid use case for XCom, because the data being passed between the tasks is a short string.

<details>
  <summary>TaskFlow</summary>

  You can use the [TaskFlow API](https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html) to push and pull values to and from XCom. To push a value to XCom return it at the end of your task as with traditional operators. To retrieve a value from XCom provide the object created by the upstream task as an input to your downstream task.

  Using the TaskFlow API usually requires less code to pass data between tasks than working with the traditional syntax.

  ```python expandable wrap theme={null}
  from airflow.decorators import dag, task
  from pendulum import datetime

  import requests
  import json

  url = "http://catfact.ninja/fact"

  default_args = {"start_date": datetime(2021, 1, 1)}


  @dag(schedule="@daily", default_args=default_args, catchup=False)
  def xcom_taskflow_dag():
      @task
      def get_a_cat_fact():
          """
          Gets a cat fact from the CatFacts API
          """
          res = requests.get(url)
          return {"cat_fact": json.loads(res.text)["fact"]}

      @task
      def print_the_cat_fact(cat_fact: str):
          """
          Prints the cat fact
          """
          print("Cat fact for today:", cat_fact)
          # run some further cat analysis here

      # Invoke functions to create tasks and define dependencies
      print_the_cat_fact(get_a_cat_fact())


  xcom_taskflow_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  In this DAG using traditional syntax, there are two `PythonOperator` tasks which share data using the `xcom_push` and `xcom_pull` functions. In the `get_a_cat_fact` function, the `xcom_push` method was used to allow the `key` name to be specified. Alternatively, the function could be configured to return the `cat_fact` value, because any value returned by an operator in Airflow is automatically pushed to XCom.

  For the `xcom_pull` call in the `analyze_cat_facts` function, you specify the `key` and `task_ids` associated with the XCom you want to retrieve. This allows you to pull any XCom value (or multiple values) at any time into a task. It doesn't need to be from the task immediately prior as shown in this example.

  ```python expandable wrap theme={null}
  import json
  from pendulum import datetime, duration
  import requests
  from airflow import DAG
  from airflow.operators.python import PythonOperator


  def get_a_cat_fact(ti):
      """
      Gets a cat fact from the CatFacts API
      """
      url = "http://catfact.ninja/fact"
      res = requests.get(url)
      ti.xcom_push(key="cat_fact", value=json.loads(res.text)["fact"])


  def analyze_cat_facts(ti):
      """
      Prints the cat fact
      """
      cat_fact = ti.xcom_pull(key="cat_fact", task_ids="get_a_cat_fact")
      print("Cat fact for today:", cat_fact)
      # run some analysis here


  with DAG(
      "xcom_dag",
      start_date=datetime(2021, 1, 1),
      max_active_runs=2,
      schedule=duration(minutes=30),
      default_args={"retries": 1, "retry_delay": duration(minutes=5)},
      catchup=False,
  ) as dag:
      get_cat_data = PythonOperator(
          task_id="get_a_cat_fact", python_callable=get_a_cat_fact
      )

      analyze_cat_data = PythonOperator(
          task_id="analyze_data", python_callable=analyze_cat_facts
      )

      get_cat_data >> analyze_cat_data
  ```
</details>

If you run this DAG and then go to the XComs page in the Airflow UI, you'll see that a new row has been added for your `get_a_cat_fact` task with the key `cat_fact` and Value returned from the API.

<Frame>
  <img alt="Example XCom" />
</Frame>

In the logs for the `analyze_data` task, you can see the value from the prior task was printed, meaning the value was successfully retrieved from XCom.

<Frame>
  <img alt="Example XCom Log" />
</Frame>

## Intermediary data storage

As mentioned previously, XCom is a great option for sharing data between tasks because it doesn't rely on any tools external to Airflow itself. However, it is only designed to be used for very small amounts of data. What if the data you need to pass is a little bit larger, for example a small dataframe?

The best way to manage this use case is to use intermediary data storage. This means saving your data to some system external to Airflow at the end of one task, then reading it in from that system in the next task. This is commonly done using cloud file storage such as S3, GCS, or Azure Blob Storage, but it could also be done by loading the data in either a temporary or persistent table in a database.

While this is a great way to pass data that is too large to be managed with XCom, you should still exercise caution. Airflow is meant to be an orchestrator, not an execution framework. If your data is very large, it is probably a good idea to complete any processing using a framework like Spark or compute-optimized data warehouses like Snowflake or dbt.

### Example DAG

Building on the previous cat fact example, you are now interested in getting more cat facts and processing them. This case would not be ideal for XCom, but since the data returned is a small dataframe, it can be processed with Airflow.

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  from pendulum import datetime, duration
  from io import StringIO

  import pandas as pd
  import requests
  from airflow.decorators import dag, task
  from airflow.providers.amazon.aws.hooks.s3 import S3Hook

  S3_CONN_ID = "aws_conn"
  BUCKET = "myexamplebucketone"


  @task
  def upload_to_s3(cat_fact_number):
      # Instantiate
      s3_hook = S3Hook(aws_conn_id=S3_CONN_ID)

      # Base URL
      url = "http://catfact.ninja/fact"

      # Grab data
      res = requests.get(url).json()

      # Convert JSON to csv
      res_df = pd.DataFrame.from_dict([res])
      res_csv = res_df.to_csv()

      # Take string, upload to S3 using predefined method
      s3_hook.load_string(
          res_csv,
          "cat_fact_{0}.csv".format(cat_fact_number),
          bucket_name=BUCKET,
          replace=True,
      )


  @task
  def process_data(cat_fact_number):
      """Reads data from S3, processes, and saves to new S3 file"""
      # Connect to S3
      s3_hook = S3Hook(aws_conn_id=S3_CONN_ID)

      # Read data
      data = StringIO(
          s3_hook.read_key(
              key="cat_fact_{0}.csv".format(cat_fact_number), bucket_name=BUCKET
          )
      )
      df = pd.read_csv(data, sep=",")

      # Process data
      processed_data = df[["fact"]]
      print(processed_data)

      # Save processed data to CSV on S3
      s3_hook.load_string(
          processed_data.to_csv(),
          "cat_fact_{0}_processed.csv".format(cat_fact_number),
          bucket_name=BUCKET,
          replace=True,
      )


  @dag(
      start_date=datetime(2021, 1, 1),
      max_active_runs=1,
      schedule="@daily",
      default_args={"retries": 1, "retry_delay": duration(minutes=1)},
      catchup=False,
  )
  def intermediary_data_storage_dag():
      upload_to_s3(cat_fact_number=1) >> process_data(cat_fact_number=1)


  intermediary_data_storage_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  from pendulum import datetime, duration
  from io import StringIO

  import pandas as pd
  import requests
  from airflow import DAG
  from airflow.operators.python import PythonOperator
  from airflow.providers.amazon.aws.hooks.s3 import S3Hook

  S3_CONN_ID = "aws_conn"
  BUCKET = "myexamplebucketone"


  def upload_to_s3(cat_fact_number):
      # Instantiate
      s3_hook = S3Hook(aws_conn_id=S3_CONN_ID)

      # Base URL
      url = "http://catfact.ninja/fact"

      # Grab data
      res = requests.get(url).json()

      # Convert JSON to csv
      res_df = pd.DataFrame.from_dict([res])
      res_csv = res_df.to_csv()

      # Take string, upload to S3 using predefined method
      s3_hook.load_string(
          res_csv,
          "cat_fact_{0}.csv".format(cat_fact_number),
          bucket_name=BUCKET,
          replace=True,
      )


  def process_data(cat_fact_number):
      """Reads data from S3, processes, and saves to new S3 file"""
      # Connect to S3
      s3_hook = S3Hook(aws_conn_id=S3_CONN_ID)

      # Read data
      data = StringIO(
          s3_hook.read_key(
              key="cat_fact_{0}.csv".format(cat_fact_number), bucket_name=BUCKET
          )
      )
      df = pd.read_csv(data, sep=",")

      # Process data
      processed_data = df[["fact"]]
      print(processed_data)

      # Save processed data to CSV on S3
      s3_hook.load_string(
          processed_data.to_csv(),
          "cat_fact_{0}_processed.csv".format(cat_fact_number),
          bucket_name=BUCKET,
          replace=True,
      )


  with DAG(
      "intermediary_data_storage_dag",
      start_date=datetime(2021, 1, 1),
      max_active_runs=1,
      schedule="@daily",
      default_args={"retries": 1, "retry_delay": duration(minutes=1)},
      catchup=False,
  ) as dag:
      generate_file_task = PythonOperator(
          task_id="generate_file",
          python_callable=upload_to_s3,
          op_kwargs={"cat_fact_number": 1},
      )

      process_data_task = PythonOperator(
          task_id="process_data",
          python_callable=process_data,
          op_kwargs={"cat_fact_number": 1},
      )

      generate_file_task >> process_data_task
  ```
</details>

In this DAG you used the [`S3Hook`](https://airflow.apache.org/registry/providers/amazon#amazon-s3-S3Hook) to save data retrieved from the API to a CSV on S3 in the `generate_file` task. The `process_data` task then takes the data from S3, converts it to a dataframe for processing, and then saves the processed data back to a new CSV on S3.


# Airflow pools
Source: https://astronomer.io/docs/learn/airflow-pools

Use pools to control Airflow task parallelism.

One of the benefits of Apache Airflow is that it is built to scale. With the right supporting infrastructure, you can run many tasks in parallel seamlessly. Unfortunately, horizontal scalability also necessitates some guardrails. For example, you might have many tasks that interact with the same source system, such as an API or database, that you don't want to overwhelm with requests. Airflow [pools](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/pools.html) are designed for exactly this use case.

Pools allow you to limit parallelism for an arbitrary set of tasks, allowing you to control when your tasks are run. They are often used in cases where you want to limit the number of parallel tasks that do a certain thing. For example, tasks that make requests to the same API or database, or tasks that run on a GPU node of a Kubernetes cluster.

In this guide, you'll learn basic Airflow pool concepts, how to create and assign pools, and what you can and can't do with pools. You'll also implement some sample DAGs that use pools to fulfill simple requirements.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* The basics of scaling Airflow. See [Scaling out Airflow](/docs/learn/airflow-scaling-workers).

## Create a pool

There are three ways you can create and manage pools in Airflow:

* The Airflow UI: Go to **Admin** > **Pools** and add a new record. You can define a name, the number of slots, and a description.

  <Frame>
    <img alt="Pools UI" />
  </Frame>

* The Airflow CLI: Run the `airflow pools` command with the `set` subcommand to create a new pool. See the [Airflow CLI documentation](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#pools) for the full list of pool commands. With the Airflow CLI, you can also import pools from a JSON file with the `import` subcommand. This can be useful if you have a large number of pools to define and doing so programmatically would be more efficient.

* The Airflow REST API: To create a pool, submit a POST request with the name and number of slots as the payload. For more information on working with pools from the API, see the [API documentation](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/post_pool).

## Assign tasks to a pool

By default, all tasks in Airflow get assigned to the `default_pool` which has 128 slots. You can modify the number of slots, but you can't remove the default pool. Tasks can be assigned to other pools by updating the `pool` parameter. This parameter is part of the `BaseOperator`, so it can be used with any operator.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  @task(task_id="task_a", pool="my_pool")
  def sleep_function():
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  task_a = PythonOperator(
      task_id="task_a", python_callable=sleep_function, pool="my_pool"
  )
  ```
</details>

When tasks are assigned to a pool, they are scheduled as normal until all of the pool's slots are filled. As slots become available, the remaining queued tasks start running.

If you assign a task to a pool that doesn't exist, then the task isn't scheduled when the DAG runs. There are currently no errors or checks for this in the Airflow UI, so be sure to double check the name of the pool that you're assigning a task to.

You can control which tasks in the pool run first by assigning [priority weights](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/priority-weight.html). These are assigned at the pool level with the `priority_weights` parameter. Higher values get higher priority in the executor queue. You can define your own custom `weight_rule`, see [Custom Weight Rule](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/priority-weight.html#custom-weight-rule).

For example, in the DAG snippet below `task_a` and `task_b` are both assigned to the `single_task_pool` which has one slot. `task_b` has a priority weight of 2, while `task_a` has the default priority weight of 1. Therefore, `task_b` is executed first.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  @task
  def sleep_function(x):
      time.sleep(x)

  @dag
  def pool_dag():
      sleep_function.override(task_id="task_a", pool="single_task_pool")(5)
      sleep_function.override(
          task_id="task_b", pool="single_task_pool", priority_weight=2
      )(10)

  pool_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  def sleep_function(x):
      time.sleep(x)

  with DAG(
      dag_id="pool_dag",
  ) as dag:

      task_a = PythonOperator(
          task_id="task_a",
          python_callable=sleep_function,
          pool="single_task_pool",
          op_args=[5],
      )

      task_b = PythonOperator(
          task_id="task_b",
          python_callable=sleep_function,
          pool="single_task_pool",
          priority_weight=2,
          op_args=[10],
      )

  ```
</details>

Additionally, you can configure the number of slots occupied by a task by updating the `pool_slots` parameter (the default is 1). Modifying this value could be useful in cases where you are using pools to manage resource utilization.

## Pool limitations

When working with pools, keep in mind the following limitations:

* Each task can only be assigned to a single pool.
* Pools are meant to control parallelism for task Instances. If you need to place limits on the number of concurrent DagRuns for a single DAG or all DAGs, use the `max_active_runs` or `core.max_active_runs_per_dag` parameters.

## Example: Limit tasks hitting an API endpoint

This example shows how to implement a pool to control the number of tasks hitting an API endpoint.

In this scenario, five tasks across two different DAGs hit the API and may run concurrently based on the DAG schedules. However, to limit the tasks hitting the API at a time to three, you'll create a pool named `api_pool` with three slots. You'll also prioritize the tasks in the `pool_priority_dag` when the pool is full.

In the `pool_priority_dag` below, all three of the tasks hit the API endpoint and should all be assigned to the pool, so you define the `pool` argument in the DAG `default_args` to apply to all tasks. You also want all three of these tasks to have the same priority weight and for them to be prioritized over tasks in the second DAG, so you assign a `priority_weight` of three as a default argument. This value is arbitrary. To prioritize these tasks, you can assign any integer that is higher than the priority weights defined in the second DAG.

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  from pendulum import datetime, duration

  import requests
  from airflow.decorators import dag, task


  @task
  def api_function(**kwargs):
      url = "http://catfact.ninja/fact"
      res = requests.get(url)
      return res.json()


  @dag(
      start_date=datetime(2021, 8, 1),
      schedule="*/30 * * * *",
      catchup=False,
      default_args={
          "pool": "api_pool",
          "retries": 1,
          "retry_delay": duration(minutes=5),
          "priority_weight": 3,
      },
  )
  def pool_priority_dag():
      api_function.override(task_id="task_a")()

      api_function.override(task_id="task_b")()

      api_function.override(task_id="task_c")()


  pool_priority_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from pendulum import datetime, duration

  import requests
  from airflow import DAG
  from airflow.operators.python import PythonOperator


  def api_function(**kwargs):
      url = "http://catfact.ninja/fact"
      res = requests.get(url)
      return res.json()


  with DAG(
      "pool_priority_dag",
      start_date=datetime(2021, 8, 1),
      schedule="*/30 * * * *",
      catchup=False,
      default_args={
          "pool": "api_pool",
          "retries": 1,
          "retry_delay": duration(minutes=5),
          "priority_weight": 3,
      },
  ) as dag:
      task_a = PythonOperator(task_id="task_a", python_callable=api_function)

      task_b = PythonOperator(task_id="task_b", python_callable=api_function)

      task_c = PythonOperator(task_id="task_c", python_callable=api_function)
  ```
</details>

In the `pool_chill_dag` DAG, there are two tasks that hit the API endpoint that should be assigned to the pool, but there are also two other tasks that don't hit the API. Therefore, you assign the pool and priority weights in the `PythonOperator` instantiations.

To prioritize `task_x` over `task_y` while keeping both at a lower priority than the tasks in the first DAG, you assign `task_x` a priority weight of 2 and leave `task_y` with the default priority weight of 1.

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  from pendulum import datetime, duration
  import requests
  from airflow.decorators import dag, task
  from airflow.operators.empty import EmptyOperator


  @task
  def api_function(**kwargs):
      url = "http://catfact.ninja/fact"
      res = requests.get(url)
      return res.json()


  @dag(
      start_date=datetime(2023, 1, 1),
      schedule="*/30 * * * *",
      catchup=False,
      default_args={"retries": 1, "retry_delay": duration(minutes=5)},
  )
  def pool_unimportant_dag():
      task_w = EmptyOperator(task_id="start")

      task_x = api_function.override(
          task_id="task_x",
          pool="api_pool",
          priority_weight=2,
      )()

      task_y = api_function.override(task_id="task_y", pool="api_pool")()

      task_z = EmptyOperator(task_id="end")

      task_w >> [task_x, task_y] >> task_z


  pool_unimportant_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  from pendulum import datetime, duration
  import requests
  from airflow import DAG
  from airflow.operators.empty import EmptyOperator
  from airflow.operators.python import PythonOperator


  def api_function(**kwargs):
      url = "http://catfact.ninja/fact"
      res = requests.get(url)
      return res.json()


  with DAG(
      "pool_unimportant_dag",
      start_date=datetime(2023, 1, 1),
      schedule="*/30 * * * *",
      catchup=False,
      default_args={"retries": 1, "retry_delay": duration(minutes=5)},
  ) as dag:
      task_w = EmptyOperator(task_id="start")

      task_x = PythonOperator(
          task_id="task_x",
          python_callable=api_function,
          pool="api_pool",
          priority_weight=2,
      )

      task_y = PythonOperator(
          task_id="task_y", python_callable=api_function, pool="api_pool"
      )

      task_z = EmptyOperator(task_id="end")

      task_w >> [task_x, task_y] >> task_z
  ```
</details>


# Apache Airflow® ETL Quickstart
Source: https://astronomer.io/docs/learn/airflow-quickstart-etl

Build and run an ETL pipeline with Apache Airflow® and the Astro CLI in 15 minutes.

Welcome to Astronomer's [Apache Airflow®](https://airflow.apache.org/) ETL Quickstart! 🚀

You will set up and run a fully functional Airflow project for a TaaS (Trees-as-a-Service) business that provides personalized, hyperlocal recommendations for which trees to plant in an area. This quickstart focuses on an ETL (extract-transform-load) pattern to generate tree planting recommendations for anyone from individuals to large corporations 🌲🌳🌴!

<Frame>
  <img alt="Screenshot of the Airflow UI showing the ETL DAG contained in this Quickstart" />
</Frame>

<Tip>
  **Other ways to learn**

  Optionally, you can choose to use the [workshop version](https://github.com/astronomer/devrel-public-workshops/tree/airflow-quickstart-workshop) of this quickstart with hands-on exercises in the README to practice using key Airflow features.
</Tip>

## Time to complete

This quickstart takes approximately 15 minutes to complete.

## Assumed knowledge

To get the most out of this quickstart, make sure you have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Basic Python. See the [Python Documentation](https://docs.python.org/3/tutorial/index.html).

## Prerequisites

* [Homebrew](https://brew.sh/) installed on your local machine.
* An integrated development environment (IDE) for Python development. This quickstart uses [Cursor](https://cursor.com/) which is very similar to [Visual Studio Code](https://code.visualstudio.com/).
* (Optional) A local installation of [Python 3](https://www.python.org/downloads/) to improve your Python developer experience.

## Step 1: Install the Astro CLI

The free Astro CLI is the easiest way to run Airflow locally in a containerized environment. Follow the instructions in this step to install the Astro CLI on a Mac using [Homebrew](https://brew.sh/), for other installation options and operating systems see [Install the Astro CLI](/docs/cli/v1.43/install-cli).

1. Run the following command in your terminal to install the Astro CLI.

   ```sh wrap theme={null}
   brew install astro
   ```

2. Verify the installation and check your Astro CLI version. You need to be on at least version **1.34.0** to run the quickstart.

   ```sh wrap theme={null}
   astro version
   ```

3. (Optional). Upgrade the Astro CLI to the latest version.

   ```sh wrap theme={null}
   brew upgrade astro
   ```

<Note>
  If you can't install the Astro CLI locally, skip to [Run the quickstart without the Astro CLI](#run-the-quickstart-without-the-astro-cli) to deploy and run the project with a [free trial of Astro](https://www.astronomer.io/lp/signup/?utm_source=website\&utm_medium=learn-guides\&utm_campaign=quickstart-etl-7-25).
</Note>

## Step 2: Clone and open the project

1. [Clone](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) the quickstart code from its [branch on GitHub](https://github.com/astronomer/devrel-public-workshops). This command will create a folder called `devrel-public-workshops` on your computer.

   ```sh wrap theme={null}
   git clone -b airflow-quickstart-complete --single-branch https://github.com/astronomer/devrel-public-workshops.git
   ```

2. Open the project folder in your IDE of choice. This quickstart uses [Cursor](https://cursor.com/) which is very similar to [Visual Studio Code](https://code.visualstudio.com/).

<Tip>
  If you quickly need a new Airflow project in the future you can always create one in any empty directory by running `astro dev init`.
</Tip>

## Step 3: Start the project

The code you cloned from GitHub already contains a fully functional Airflow project. Let's start it!

1. Run the following command in the root of the cloned folder to start the quickstart:

   ```sh wrap theme={null}
   astro dev start
   ```

<Info>
  If port 8080 or 5432 are in use on your machine, Airflow won't be able to start. To run Airflow on alternative ports, run:

  ```sh wrap theme={null}
  astro config set webserver.port <available-port>
  astro config set postgres.port <available-port>
  ```
</Info>

2. As soon as the project has started, the Airflow UI opens in your default browser. When running the start command for the first time this might take a couple of minutes. Note that as long as the Airflow project is running, you can always access the UI in another browser or additional tab by going to `localhost:8080`.

3. Click the **DAGs** button (1) in the Airflow UI to get to the DAGs overview page to see all the DAGs contained in this project.

   <Frame>
     <img alt="Airflow UI home screen" />
   </Frame>

## Step 4: Run the setup DAG

You should now see 3 DAGs. They will be paused by default, with no runs yet. In this step, you'll run the `trees_database_setup` DAG to set up the database used in the ETL DAG in [Step 5](#step-5-run-the-etl-dags).

1. Click the play button (2) of the `trees_database_setup` DAG (1) to open its DAG Trigger form.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the DAGs overview with 3 paused DAGs" />
   </Frame>

   Note that there is an Import Error (3) for a fourth DAG. This DAG performs a call to a Large Language Model (LLM) to generate personalized messages based on a tree recommendation and needs a little bit more setup to use. See the [Airflow Quickstart - GenAI](/docs/learn/airflow-quickstart-genai) for instructions. For this ETL quickstart you don't have to worry about this import error; it doesn't affect the DAGs in the ETL pipeline.

2. On the DAG Trigger form, make sure that **Single Run** (1) is selected and the Checkbox for **Unpause `trees_database_setup` on trigger** is checked (2). Then click the blue **Trigger** button (3) to create a DAG run.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the DAG Trigger form" />
   </Frame>

3. After a few seconds the DAG run should be complete and you'll see a dark green bar in the Airflow UI (1).

   <Frame>
     <img alt="Screenshot of the Airflow UI showing DAGs overview with 1 successful run of the trees_database_setup DAG" />
   </Frame>

   In your IDE you can open the include folder (1) to see that a new file called `trees.db` (2) was created. This is the [DuckDB](https://duckdb.org/) database the ETL DAG interacts with.

   <Frame>
     <img alt="Screenshot of Cursor showing where to see the DuckDB file." />
   </Frame>

## Step 5: Run the ETL DAGs

The `trees_database_setup` DAG created the `trees.db` and filled it with some sample data. Now it is time for you to run the ETL pipeline consisting of two DAGs: `etl_trees`, which loads an additional record to the database with tree recommendations for you, and the `trees_analytics` DAG which summarizes the database contents.

These two DAGs depend on each other using an [Airflow Asset](/docs/learn/airflow-datasets), which means that as soon as a specific task in the first DAG (the `summarize_onboarding` task in the `etl_trees` DAG) completes successfully, the second DAG (`trees_analytics`) will run automatically.

1. Unpause both DAGs by clicking their pause toggle to turn them blue (1 and 2). Once unpaused, DAGs run on their defined [schedule](/docs/learn/scheduling-in-airflow), which can be time-based (for example run once per day at midnight UTC) or data-aware like in this example. Next, open the DAG trigger form for the `etl_trees` DAG by clicking on its play button (3).

   <Frame>
     <img alt="Screenshot of the Airflow UI showing DAGs overview with all DAGs unpaused" />
   </Frame>

2. The `etl_trees` DAG runs with [params](/docs/learn/airflow-params). If the DAG runs based on its schedule the given defaults are used, but on a manual run, like right now, you can provide your own values. Enter your name (1) and your location (2), then trigger (3) a DAG run.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the DAG Trigger form" />
   </Frame>

3. After 10-20 seconds both the `etl_trees` and the `trees_analytics` DAG have completed successfully (1 and 2).

   <Frame>
     <img alt="Screenshot of the Airflow UI with successful DAG runs" />
   </Frame>

## Step 6: Explore the ETL DAG

Let's explore the ETL DAG in more detail.

1. Click the DAG name to get to the DAG overview. From here you can access a lot of detailed information about this specific DAG. To navigate to the logs of individual task instances, click the squares in the grid view. Open the logs for the `summarize_onboarding` task (1) to see your tree recommendations!

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the Grid view" />
   </Frame>

2. Next, to view the dependencies between your DAGs you can toggle between the **Grid** and **Graph** view in the top left corner of the DAG overview (2). Each node in the graph corresponds to one task. The edges between the nodes denote how the tasks depend on each other, and by default the DAG graph is read from left to right. The code of the DAG can be viewed by clicking on the Code tab (1).

   <Frame>
     <img alt="Screenshot of the Graph view of a DAG." />
   </Frame>

   In the screenshot above you can see the graph of the ETL DAG, it consists of 6 tasks:

   * `extract_user_data`: This task extracts user data. In this quickstart example the DAG uses the data entered in the DAG Trigger form, the name and location of one user. In a real-world use case, it would likely extract several users' records at the same time through calling an API or reading from a database.
   * `transform_user_data`: This task uses the data the first task extracted and creates a comprehensive user record from it using modularized functions stored in the `include` folder.
   * `generate_tree_recommendations`: The third task generates the tree recommendations based on the transformed data about the user.
   * `check_tables_exist`: Before loading data into the database, this task makes sure that the needed tables exist in the database.
   * `load_user_data`: This task loads data into the database, it can only run after both `generate_tree_recommendations` and `check_tables_exist` have completed successfully.
   * `summarize_onboarding`: The final task summarizes the data that was just loaded to the database and prints it to the logs.

3. Let's check out the code that defines this DAG by clicking on the Code tab. Note that while you can view the DAG code in the Airflow UI, you can only make changes to it in your IDE, not directly in the UI. You can see how each task in your DAG corresponds to one function that has been turned into an Airflow task using the [`@task` decorator](/docs/learn/airflow-decorators).

<Note>
  The [`@task` decorator](/docs/learn/airflow-decorators) is one of several options for defining your DAGs. The two other options are using [traditional operators](/docs/learn/what-is-an-operator) or the [`@asset` decorator](/docs/learn/airflow-datasets#asset-definition).
</Note>

5. The second DAG in this ETL pipeline, `trees_analytics`, is defined with the [`@asset` decorator](/docs/learn/airflow-datasets#asset-definition), a shorthand to create one DAG with one task updating one [Asset](/docs/learn/airflow-datasets).

   <Frame>
     <img alt="Screenshot of the task logs of the trees_analytics task showing 501 users in the database." />
   </Frame>

   Click the DAG name in the DAG overview page, and then on the square for the `trees_analytics` task (1) to view the logs containing summary analytics about your tree recommendations. Since the setup DAG adds 500 users to the database, and you just added yourself with the ETL DAG, you should see `501` users in the log output (2). Viewing the DAG code (3) you can see how the `@asset` decorator quickly turns a Python function into an Airflow DAG with one task.

6. (Optional). Make a small change to your DAG code, for example by adding a print statement in one of the `@task` decorated functions. After the change has taken effect, run your DAG again and see the added print statement in the task logs.

<Tip>
  When running Airflow with default settings, it can take up to 30 seconds for DAG changes to be visible in the UI and up to 5 minutes for a new DAG (with a new DAG ID) to show up in the UI. If you don't want to wait you can run the following command to parse all existing and new DAG files in your `dags` folder.

  ```sh wrap theme={null}
  astro dev run dags reserialize
  ```
</Tip>

## Step 7: Deploy your project

It is time to move the project to production!

1. If you don't have access to Astro already, sign up for a free [Astro trial](https://www.astronomer.io/lp/signup/?utm_source=website\&utm_medium=learn-guides\&utm_campaign=quickstart-etl-7-25).

2. [Create a new Deployment](/docs/astro/create-deployment) in your Astro workspace.

3. Run the following command in your local CLI to authenticate your computer to Astro. Follow the sign-in instructions in the window that opens in your browser.

   ```sh wrap theme={null}
   astro login
   ```

4. Run the command to the deploy to Astro.

   ```sh wrap theme={null}
   astro deploy -f
   ```

5. Once the deploy has completed, click the blue **Open Airflow** button on your Deployment to see the DAGs running in the cloud. Unpause all 3 DAGs in your Astro environment and run the `trees_database_setup` DAG manually. This will trigger a run of all 3 DAGs since they depend on each other using an `Asset` schedule.

6. Check the logs of the `summarize_onboarding` task in the `etl_trees` DAG to see the tree recommendations generated for the default user set in the [Airflow params](/docs/learn/airflow-params).

7. (Optional) Change the default user and default location by adding two environment variables to your Astro Deployment. In the Astro UI click the **Environment** tab (1) for your Deployment and add your own values for `USER_NAME` and `USER_LOCATION`.

   <Frame>
     <img alt="Astro UI showing the Environment tab." />
   </Frame>

8. (Optional) After the new environment variables have synced to your Astro Deployment (this can take a few minutes), run the `etl_trees` manually. The default suggestions in the DAG Trigger form are now based on your own values and you'll get another set of personal tree recommendations, this time on Astro!

## Next steps

Awesome! You ran an ETL pipeline locally and in the cloud. To continue your learning we recommend the following resources:

* If you are curious about the DAG currently showing as an import error and want to learn how to use Airflow for GenAI workflows, check out the [Airflow GenAI quickstart](/docs/learn/airflow-quickstart-genai).
* To get a structured video-based introduction to Apache Airflow and its concepts, sign up for the [Airflow 101 (Airflow 3) Learning Path](https://academy.astronomer.io/path/airflow-101) in the Astronomer Academy.
* For a short step-by-step walkthrough of the most important Airflow features, complete our [Airflow tutorial](/docs/learn/get-started-with-airflow).

## Run the quickstart without the Astro CLI

If you can't install the Astro CLI on your local computer, you can still run the pipeline in this example.

1. Sign up for a free trial of [Astro](https://www.astronomer.io/lp/signup/?utm_source=website\&utm_medium=learn-guides\&utm_campaign=quickstart-etl-7-25).

2. [Create a new Deployment](/docs/astro/create-deployment) in your Astro workspace.

3. [Fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) the [Airflow quickstart](https://github.com/astronomer/devrel-public-workshops) repository to your GitHub account. Make sure to **uncheck** the **Copy the main branch only** box!

   <Frame>
     <img alt="Screenshot of GitHub showing how to fork the repository and the box to uncheck" />
   </Frame>

4. [Set up the Astro GitHub integration](/docs/astro/deploy-github-integration) to map your Deployment to the `airflow-quickstart-complete` branch of your forked repository.

5. Select **Trigger Git Deploy** from the **More actions** menu in your Deployment settings to trigger the first deploy for the mapped repository.

   <Frame>
     <img alt="Screenshot of the Astro UI showing how to Trigger a Git Deploy" />
   </Frame>

6. Once the deploy has completed, click the blue **Open Airflow** button on your Deployment to see the DAGs running in the cloud. From here you can jump back to [Step 4](#step-4-run-the-setup-dag) and complete the quickstart, skipping over the deploy instructions in [Step 7](#step-7-deploy-your-project), since you already deployed your project!


# Apache Airflow® GenAI Quickstart
Source: https://astronomer.io/docs/learn/airflow-quickstart-genai

Build and run a GenAI pipeline with Apache Airflow® and the Airflow AI SDK in 15 minutes.

Welcome to Astronomer's [Apache Airflow®](https://airflow.apache.org/) GenAI Quickstart! 🚀

You will set up and run a fully functional Airflow project for a TaaS (Trees-as-a-Service) business that provides personalized, hyperlocal recommendations for which trees to plant in an area. This quickstart focuses on a GenAI DAG that uses the [Airflow AI SDK](https://github.com/astronomer/airflow-ai-sdk) to create a personalized description of your future garden 🌲🌳🌴!

<Frame>
  <img alt="Screenshot of the Airflow UI showing the GenAI DAG contained in this Quickstart" />
</Frame>

<Tip>
  **Other ways to learn**

  You can watch the recording of our recent [Orchestrating LLM workflows with the Airflow AI SDK](https://www.astronomer.io/events/webinars/from-zero-to-production-orchestrating-llm-workflows-with-the-airflow-ai-sdk-video/) webinar to learn more about all decorators in the [Airflow AI SDK](https://github.com/astronomer/airflow-ai-sdk).

  For a hands-on version of this project, check out the [Airflow quickstart workshop](https://github.com/astronomer/devrel-public-workshops/tree/airflow-quickstart-workshop) containing exercises to practice how to use key Airflow features.
</Tip>

## Time to complete

This quickstart takes approximately 15 minutes to complete.

## Assumed knowledge

To get the most out of this quickstart, make sure you have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Basic Python. See the [Python Documentation](https://docs.python.org/3/tutorial/index.html).

## Prerequisites

* [Homebrew](https://brew.sh/) installed on your local machine.
* An integrated development environment (IDE) for Python development. This quickstart uses [Cursor](https://cursor.com/) which is very similar to [Visual Studio Code](https://code.visualstudio.com/).
* An [OpenAI API Key](https://platform.openai.com/docs/api-reference/authentication).
* (Optional) A local installation of [Python 3](https://www.python.org/downloads/) to improve your Python developer experience.

## Step 1: Install the Astro CLI

The free Astro CLI is the easiest way to run Airflow locally in a containerized environment. Follow the instructions in this step to install the Astro CLI on a Mac using [Homebrew](https://brew.sh/), for other installation options and operating systems see [Install the Astro CLI](/docs/cli/v1.43/install-cli).

1. Run the following command in your terminal to install the Astro CLI.

   ```sh wrap theme={null}
   brew install astro
   ```

2. Verify the installation and check your Astro CLI version. You need to be on at least version **1.34.0** to run the quickstart.

   ```sh wrap theme={null}
   astro version
   ```

3. (Optional). Upgrade the Astro CLI to the latest version.

   ```sh wrap theme={null}
   brew upgrade astro
   ```

<Note>
  If you can't install the Astro CLI locally, skip to [Run the quickstart without the Astro CLI](#run-the-quickstart-without-the-astro-cli) to deploy and run the project with a [free trial of Astro](https://www.astronomer.io/lp/signup/?utm_source=website\&utm_medium=learn-guides\&utm_campaign=quickstart-etl-7-25).
</Note>

## Step 2: Clone and open the project

1. [Clone](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) the quickstart code from its [branch on GitHub](https://github.com/astronomer/devrel-public-workshops). This command will create a folder called `devrel-public-workshops` on your computer.

   ```sh wrap theme={null}
   git clone -b airflow-quickstart-complete --single-branch https://github.com/astronomer/devrel-public-workshops.git
   ```

2. Open the project folder in your IDE of choice.

<Tip>
  If you quickly need a new Airflow project in the future you can always create one in any empty directory by running `astro dev init`.
</Tip>

## Step 3: Add your OpenAI API key

In order to be able to send requests to a Large Language Model (LLM) via the Airflow AI SDK you need to have an API Key and make it available to the Airflow AI SDK as an environment variable. This quickstart uses [OpenAI](https://platform.openai.com/docs/overview); you can learn about other model providers that are compatible with the Airflow AI SDK in the [Airflow AI SDK's `pyproject.toml` file](https://github.com/astronomer/airflow-ai-sdk/blob/main/pyproject.toml#L30).

1. In the root of your repository, create a new file called `.env` (1). This file is ignored by git and a good place to define your (secret) environment variables

   <Frame>
     <img alt="Screenshot of Cursor showing an added .env file for the OpenAI API Key." />
   </Frame>

2. In the `.env` file, define a variable called `OPENAI_API_KEY` and set it to your [OpenAI API Key](https://platform.openai.com/docs/api-reference/authentication) (2).

3. To use the AI SDK you also need to install the [Airflow AI SDK Python package](https://github.com/astronomer/airflow-ai-sdk) with the extra relevant to your model provider by adding it to the `requirements.txt` file. This has already been done for you in the quickstart repository.

<Note>
  Whenever you make changes to your `.env` or `requirements.txt` file you need to restart your Airflow environment using `astro dev restart` for the changes to take effect.
</Note>

## Step 4: Start the project

The code you cloned from GitHub already contains a fully functional Airflow project. Let's start it!

1. Run the following command in the root of the cloned folder to start the quickstart:

   ```sh wrap theme={null}
   astro dev start
   ```

<Info>
  If port 8080 or 5432 are in use on your machine, Airflow won't be able to start. To run Airflow on alternative ports, run:

  ```sh wrap theme={null}
  astro config set webserver.port <available-port>
  astro config set postgres.port <available-port>
  ```
</Info>

2. As soon as the project has started, the Airflow UI opens in your default browser. When running the start command for the first time this might take a couple of minutes. Note that as long as the Airflow project is running, you can always access the UI in another browser or additional tab by going to `localhost:8080`.

3. Click the **Dags** button (1) in the Airflow UI to get to the DAG overview page to see the DAGs contained in this project.

   <Frame>
     <img alt="Airflow UI home screen" />
   </Frame>

## Step 5: Run the GenAI DAG

You should now see 4 DAGs. They will be paused by default, with no runs yet. For this quickstart, only the `genai_trees` DAG will be relevant. The other DAGs belong to the [Airflow ETL Quickstart](/docs/learn/airflow-quickstart-etl).

1. Open the Trigger DAG form of the `genai_trees` DAG by clicking on its play button (1).

   <Frame>
     <img alt="DAGs overview showing the play button for the genai_trees DAG." />
   </Frame>

2. The `genai_trees` DAG runs with [params](/docs/learn/airflow-params). If the DAG were to run based on a schedule the given defaults are used, but on a manual run, like right now, you can provide your own values. In the Trigger DAG form, enter your name (1) and your location (2), then trigger (3) a DAG run. Make sure the **Unpause `genai_trees` on trigger** checkbox (4) is selected.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the DAG Trigger form" />
   </Frame>

3. After 20-30 seconds the `genai_trees` DAG has completed successfully (1).

   <Frame>
     <img alt="DAGs overview showing a successful run of the genai_trees DAG." />
   </Frame>

## Step 6: Explore the GenAI DAG

Let's explore the ETL DAG in more detail.

1. Click the DAG name to get to the DAG overview. From here you can access a lot of detailed information about this specific DAG.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the Grid view" />
   </Frame>

2. To navigate to the logs of individual task instances, click the squares in the grid view. Open the logs for the `print_llm_output` task (1) to see your garden description!

3. Next, to view the dependencies between your DAGs you can toggle between the **Grid** and **Graph** view in the top left corner of the DAG overview (2). Each node in the graph corresponds to one task. The edges between the nodes denote how the tasks depend on each other, and by default the DAG graph is read from left to right. The code of the DAG can be viewed by clicking on the Code tab (1).

   <Frame>
     <img alt="Screenshot of the Graph view of a DAG." />
   </Frame>

   In the screenshot above you can see the graph of the ETL DAG, it consists of 6 tasks:

   * `extract_user_data`: This task extracts user data. In this quickstart example the DAG uses the data entered in the DAG Trigger form, the name and location of one user. In a real-world use case, it would likely extract several users' records at the same time through calling an API or reading from a database.
   * `transform_user_data`: This task uses the data the first task extracted and creates a comprehensive user record from it using modularized functions stored in the `include` folder.
   * `generate_tree_recommendations`: The third task generates the tree recommendations based on the transformed data about the user.
   * `generate_garden_description`: This task uses the `@task.llm` decorator of the Airflow AI SDK to make a call to the OpenAI API to generate a description of your future garden, if you were to follow the tree recommendations.
   * `print_llm_output`: The final task prints the output of the LLM to the task logs.

4. Lastly, let's check out the code that defines this DAG by clicking on the Code tab. Note that while you can view the DAG code in the Airflow UI, you can only make changes to it in your IDE, not directly in the UI. You can see how each task in your DAG corresponds to one function that has been turned into an Airflow task using the [`@task` decorator](/docs/learn/airflow-decorators).

<Note>
  The [`@task` decorator](/docs/learn/airflow-decorators) is one of several options for defining your DAGs. The two other options are using [traditional operators](/docs/learn/what-is-an-operator) or the [`@asset` decorator](/docs/learn/airflow-datasets#asset-definition).
</Note>

5. (Optional). Make a small change to your DAG code, for example by adding a print statement in one of the `@task` decorated functions. After the change has taken effect, run your DAG again and see the added print statement in the task logs.

<Tip>
  When running Airflow with default settings, it can take up to 30 seconds for DAG changes to be visible in the UI and up to 5 minutes for a new DAG (with a new DAG ID) to show up in the UI. If you don't want to wait, you can run the following command to parse all existing and new DAG files in your `dags` folder.

  ```sh wrap theme={null}
  astro dev run dags reserialize
  ```
</Tip>

## Step 7: Deploy your project

It is time to move the project to production!

1. If you don't have access to Astro already, sign up for a free [Astro trial](https://www.astronomer.io/lp/signup/?utm_source=website\&utm_medium=learn-guides\&utm_campaign=quickstart-etl-7-25).

2. [Create a new Deployment](/docs/astro/create-deployment) in your Astro workspace.

3. Run the following command in your local CLI to authenticate your computer to Astro. Follow the sign-in instructions in the window that opens in your browser.

   ```sh wrap theme={null}
   astro login
   ```

4. Run the command to the deploy to Astro.

   ```sh wrap theme={null}
   astro deploy -f
   ```

5. Since environment variables often contain secrets, they aren't deployed from `.env` with the rest of your project code. To add environment variables like your `OPENAI_API_KEY` to an Astro Deployment, use the **Environment** tab (1) on your Deployment. Make sure to mark the environment variables as secret!

   <Frame>
     <img alt="Astro UI showing the Environment tab." />
   </Frame>

6. Once the deploy has completed, click the blue **Open Airflow** button on your Deployment to see the DAGs running in the cloud. Here, you can run the `genai_trees` DAG again to get another garden description!

## Next steps

Awesome! You ran a GenAI DAG locally and in the cloud. To continue your learning we recommend the following resources:

* If you are curious about the other DAGs in this Quickstart which form an ETL pipeline, check out the [Airflow ETL quickstart](/docs/learn/airflow-quickstart-etl).
* To get a structured video-based introduction to Apache Airflow and its concepts, sign up for the [Airflow 101 (Airflow 3) Learning Path](https://academy.astronomer.io/path/airflow-101) in the Astronomer Academy.
* For a short step-by-step walkthrough of the most important Airflow features, complete our [Airflow tutorial](/docs/learn/get-started-with-airflow).

## Run the quickstart without the Astro CLI

If you can't install the Astro CLI on your local computer, you can still run the pipeline in this example.

1. Sign up for a free trial of [Astro](https://www.astronomer.io/lp/signup/?utm_source=website\&utm_medium=learn-guides\&utm_campaign=quickstart-etl-7-25).

2. [Create a new Deployment](/docs/astro/create-deployment) in your Astro workspace.

3. [Fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) the [Airflow quickstart](https://github.com/astronomer/devrel-public-workshops) repository to your GitHub account. Make sure to **uncheck** the **Copy the main branch only** box!

   <Frame>
     <img alt="Screenshot of GitHub showing how to fork the repository and the box to uncheck" />
   </Frame>

4. [Set up the Astro GitHub integration](/docs/astro/deploy-github-integration) to map your Deployment to the `airflow-quickstart-complete` branch of your forked repository.

5. Select **Trigger Git Deploy** from the **More actions** menu in your Deployment settings to trigger the first deploy for the mapped repository.

   <Frame>
     <img alt="Screenshot of the Astro UI showing how to Trigger a Git Deploy" />
   </Frame>

6. Add an environment variable called `OPENAI_API_KEY` with the value of your [OpenAI API Key](https://platform.openai.com/docs/api-reference/authentication) to your Astro Deployment by clicking on the **Environment** tab (1) on your Deployment. Make sure to mark the environment variables as secret! This key will be used by the Airflow AI SDK to make a call to OpenAI generating your future garden description.

   <Frame>
     <img alt="Astro UI showing the Environment tab." />
   </Frame>

7. Once the deploy has completed, click the blue **Open Airflow** button on your Deployment to see the DAGs running in the cloud. From here you can complete [Step 5](#step-5-run-the-genai-dag) and complete the quickstart, skipping over the deploy instructions in [Step 7](#step-7-deploy-your-project), since you already deployed your project!


# Scaling Airflow to optimize performance
Source: https://astronomer.io/docs/learn/airflow-scaling-workers

See which parameters to modify when scaling up data pipelines to make the most of Airflow. Learn about the environment, DAG, and task-level settings.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

One of the biggest strengths of Apache Airflow is its ability to scale to meet the changing demands of your organization. To make the most of Airflow, there are a few key settings that you should consider modifying as you scale up your data pipelines.

Airflow exposes a number of parameters that are closely related to DAG and task-level performance. These include:

* Environment-level settings.
* DAG-level settings.
* Task-level settings.

In this guide, you'll learn about the key parameters that you can use to modify Airflow performance. you'll also learn how your choice of executor can impact scaling and how best to respond to common scaling issues.

This guide references the parameters available in Airflow version 2.0 and later. If you're using an earlier version of Airflow, some of the parameter names might be different.

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Webinar: [Scaling Out Airflow](https://www.astronomer.io/events/webinars/scaling-out-airflow/).
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow core components. See [Airflow's components](/docs/learn/airflow-components).
* Airflow executors. See [Airflow executors explained](/docs/learn/airflow-executors-explained).

## Parameter tuning

Airflow has many parameters that impact its performance. Tuning these settings can impact DAG parsing and task scheduling performance, parallelism in your Airflow environment, and more.

The reason Airflow allows so many adjustments is that, as an agnostic orchestrator, Airflow is used for a wide variety of use cases. Airflow admins or DevOps engineers might tune scaling parameters at the environment level to ensure that their supporting infrastructure isn't overstressed, while DAG authors might tune scaling parameters at the DAG or task level to ensure that their pipelines don't overwhelm external systems. Knowing the requirements of your use case before scaling Airflow will help you choose which parameters to modify.

### Environment-level settings

Environment-level settings are those that impact your entire Airflow environment (all DAGs). They all have default values that can be overridden by setting the appropriate environment variable or modifying your `airflow.cfg` file. Generally, all default values can be found in the [Airflow Configuration Reference](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html). To check current values for an existing Airflow environment, go to **Admin** > **Configurations** in the Airflow UI. For more information, see [Setting Configuration Options](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html) in the Apache Airflow documentation.

If you're running Airflow on Astronomer, you should modify these parameters with Astronomer environment variables. For more information, see [Environment Variables on Astronomer](/docs/astro/environment-variables).

You should modify environment-level settings if you want to tune performance across all of the DAGs in your Airflow environment. This is particularly relevant if you want your DAGs to run well on your support infrastructure.

#### Core settings

Core settings control the number of processes running concurrently and how long processes run across an entire Airflow environment. The associated environment variables for all parameters in this section are formatted as `AIRFLOW__CORE__PARAMETER_NAME`.

* `parallelism`: The maximum number of tasks that can run concurrently on each scheduler within a single Airflow environment. For example, if this setting is set to 32, and there are two schedulers, then no more than 64 tasks can be in a running or queued state at once across all DAGs. If your tasks remain in a scheduled state for an extended period, you might want to increase this value. The default value is 32.

  On Astro, this value is [set automatically](/docs/astro/configure-worker-queues#worker-queue-settings) based on your maximum worker count, meaning that you don't have to configure it.

* `max_active_tasks_per_dag` (formerly `dag_concurrency`): The maximum number of tasks that can be scheduled at the same time across all runs of a DAG. Use this setting to prevent any one DAG from taking up too many of the available slots from parallelism or your pools. The default value is 16.

  If you increase the amount of resources available to Airflow (such as Celery workers or Kubernetes resources) and notice that tasks are still not running as expected, you might have to increase the values of both `parallelism` and `max_active_tasks_per_dag`.

* `max_active_runs_per_dag`: Determines the maximum number of active DAG runs (per DAG) that the Airflow scheduler can create at a time. In Airflow, a [DAG run](https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html) represents an instantiation of a DAG in time, much like a task instance represents an instantiation of a task. This parameter is most relevant if Airflow needs to backfill missed DAG runs. Consider how you want to handle these scenarios when setting this parameter. The default value is 16.

* `dag_file_processor_timeout`: How long a `DagFileProcessor`, which processes a DAG file, can run before timing out. The default value is 50 seconds.

* `dagbag_import_timeout`: How long the `dagbag` can import DAG objects before timing out in seconds, which must be lower than the value set for `dag_file_processor_timeout`. If your DAG processing logs show timeouts, or if your DAG isn't showing in the DAGs list or the import errors, try increasing this value. You can also try increasing this value if your tasks aren't executing, since workers need to fill up the `dagbag` when tasks execute. The default value is 30 seconds.

#### Scheduler settings

[Scheduler config settings](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#scheduler) control how the scheduler parses DAG files and creates DAG runs. The associated environment variables for all parameters in this section are formatted as `AIRFLOW__SCHEDULER__PARAMETER_NAME`.

* `min_file_process_interval`: The frequency that each DAG file is parsed, in seconds. Updates to DAGs are reflected after this interval. A low number increases scheduler CPU usage. If you have dynamic DAGs created by complex code, you can increase this value to improve scheduler performance. The default value is 30 seconds.

* `dag_dir_list_interval`: The frequency that the DAGs directory is scanned for new files, in seconds. The lower the value, the faster new DAGs are processed and the higher the CPU usage. The default value is 300 seconds (5 minutes).

  It's helpful to know how long it takes to parse your DAGs (`dag_processing.total_parse_time`) to know what values to choose for `min_file_process_interval` and `dag_dir_list_interval`. If your `dag_dir_list_interval` is less than the amount of time it takes to parse each DAG, performance issues can occur.

  <Tip>
    If you have less than 200 DAGs in a Deployment on Astro, it's safe to set `AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL=30` (30 seconds) as a Deployment-level [environment variable](/docs/astro/environment-variables).
  </Tip>

* `parsing_processes` (formerly `max_threads`): How many processes the scheduler can run in parallel to parse DAGs. Astronomer recommends setting a value that is twice your available vCPUs. Increasing this value can help serialize a large number of DAGs more efficiently. If you are running multiple schedulers, this value applies to each of them. The table below lists the default number of parsing processes for each Astro Hosted deployment size.

| Small | Medium | Large | Extra Large                    |
| ----- | ------ | ----- | ------------------------------ |
| 2     | 2      | 6     | 12 (across two DAG Processors) |

* `file_parsing_sort_mode`: Determines how the scheduler lists and sorts DAG files to determine the parsing order. Set to one of: `modified_time`, `random_seeded_by_host` and `alphabetical`. The default value is `modified_time`.

* `scheduler_heartbeat_sec`: Defines how often the scheduler should run (in seconds) to trigger new tasks. The default value is 5 seconds.

* `max_dagruns_to_create_per_loop`: The maximum number of DAGs to create DAG runs for per scheduler loop. Decrease the value to free resources for scheduling tasks. The default value is 10.

* `max_tis_per_query`: Changes the batch size of queries to the metastore in the main scheduling loop. A higher value allows more `tis` to be processed per query, but your query may become too complex and cause performance issues. The default value is 16 queries. Note that the `scheduler.max_tis_per_query` value needs to be lower than the `core.parallelism` value.

### DAG-level Airflow settings

DAG-level settings apply only to specific DAGs and are defined in your DAG code. You should modify DAG-level settings if you want to performance tune a particular DAG, especially in cases where that DAG is hitting an external system such as an API or a database that might cause performance issues if hit too frequently. When a setting exists at both the DAG-level and environment-level, the DAG-level setting takes precedence.

There are three primary DAG-level Airflow settings that you can define in code:

* `max_active_runs`: The maximum number of active DAG runs allowed for the DAG. When this limit is exceeded, the scheduler won't create new active DAG runs. If this setting isn't defined, the value of the environment-level setting `max_active_runs_per_dag` is assumed.

  If you're utilizing `catchup` or `backfill` for your DAG, consider defining this parameter to ensure that you don't accidentally trigger a high number of DAG runs.
* `max_active_tasks`: The total number of tasks that can run at the same time across all DAG runs. It essentially controls the parallelism across all runs of a particular DAG. If this setting isn't defined, the value of the environment-level setting `max_active_tasks_per_dag` is assumed.
* `concurrency`: The maximum number of task instances allowed to run concurrently across all active DAG runs for a given DAG. This allows you to allow one DAG to run 32 tasks at once, and another DAG can be set to run 16 tasks at once. If this setting isn't defined, the value of the environment-level setting `max_active_tasks_per_dag` is assumed.

You can define any DAG-level settings within your DAG definition. For example:

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # Allow a maximum of concurrent 10 tasks across a max of 3 active DAG runs
  @dag("my_dag_id", concurrency=10, max_active_runs=3)
  def my_dag():
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # Allow a maximum of concurrent 10 tasks across a max of 3 active DAG runs
  with DAG("my_dag_id", concurrency=10, max_active_runs=3):
  ```
</details>

### Task-level Airflow settings

Task-level settings are defined by task operators that you can use to implement additional performance adjustments. Modify task-level settings when specific types of tasks are causing performance issues.

There are three primary task-level Airflow settings users can define in code:

* `max_active_tis_per_dag` (formerly `task_concurrency`): The maximum number of times that the same task can run concurrently across all DAG runs. For instance, if a task pulls from an external resource, such as a data table, that shouldn't be modified by multiple tasks at once, then you can set this value to 1.
* `max_active_tis_per_dagrun`: The maximum number of times that the same task can run concurrently within a single DAG run. This doesn't limit the concurrency of the task across simultaneous DAG runs, only within each DAG run. This setting won't affect single task runs (as there will always be one task instance per DAG run), but can be used to control concurrency of [Dynamically Mapped Tasks](/docs/learn/dynamic-tasks) within each DAG run.
* `pool`: Defines the amount of pools available for a task. Pools are a way to limit the number of concurrent instances of an arbitrary group of tasks. This setting is useful if you have a lot of workers or DAG runs in parallel, but you want to avoid an API rate limit or otherwise don't want to overwhelm a data source or destination. For more information, see the [Airflow Pools Guide](/docs/learn/airflow-pools).

The parameters above are inherited from the `BaseOperator`, so you can set them in any operator definition. For example:

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  @task(
      pool="my_custom_pool",
      max_active_tis_per_dag=14
  ) 
  def t1():
      pass
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  def t1_func():
      pass

  t1 = PythonOperator(
      task_id="t1",
      python_callable=t1_func,
      pool="my_custom_pool",
      max_active_tis_per_dag=14
  )
  ```
</details>

## Executors and scaling

Depending on which executor you choose for your Airflow environment, there are additional settings to keep in mind when scaling.

### Celery executor

The [Celery executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/celery.html) utilizes standing workers to run tasks. Scaling with the Celery executor involves choosing both the number and size of the workers available to Airflow. The more workers you have available in your environment, or the larger your workers are, the more capacity you have to run tasks concurrently.

You can also tune your `worker_concurrency` (environment variable: `AIRFLOW__CELERY__WORKER_CONCURRENCY`), which determines how many tasks each Celery worker can run at any given time. By default, the Celery executor runs a maximum of sixteen tasks concurrently. If you increase `worker_concurrency`, you might also need to provision additional CPU and/or memory for your workers.

### Kubernetes executor

The [Kubernetes executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html) launches a pod in a Kubernetes cluster for each task. Since each task runs in its own pod, resources can be specified on an individual task level.

When tuning performance with the Kubernetes executor, it is important to consider the supporting infrastructure of your Kubernetes cluster. Many users will enable auto-scaling on their cluster to ensure they get the benefit of Kubernetes' elasticity.

You can also tune your `worker_pods_creation_batch_size` (environment variable: `AIRFLOW__KUBERNETES__WORKER_PODS_CREATION_BATCH_SIZE`), which determines how many pods can be created per scheduler loop. The default is 1, but you'll want to increase this number for better performance, especially if you have concurrent tasks. The maximum value is determined by the tolerance of your Kubernetes cluster.

## Potential scaling issues

Scaling your Airflow environment is an art and not a science, and it's highly dependent on your supporting infrastructure and your DAGs. The following are some of the most common issues:

* Task scheduling latency is high.
  * The scheduler may not have enough resources to parse DAGs in order to then schedule tasks.
  * Change `worker_concurrency` (if using Celery), or `parallelism`.
* DAGs remain in queued state, but aren't running.
  * The number of tasks being scheduled may be beyond the capacity of your Airflow infrastructure.
  * If you're using the Kubernetes executor, check that there are available resources in the namespace and check if `worker_pods_creation_batch_size` can be increased. If using the Celery executor, check if `worker_concurrency` can be increased.
* An individual DAG is having trouble running tasks in parallel, while other DAGs are unaffected.
  * Possible DAG-level bottleneck.
  * Change `max_active_task_per_dag`, pools (if using them), or overall `parallelism`.

For help with scaling issues, consider joining the [Apache Airflow Slack](https://airflow.apache.org/community/) or contact [Astronomer support](https://www.astronomer.io/get-astronomer/).


# Use setup and teardown tasks in Airflow
Source: https://astronomer.io/docs/learn/airflow-setup-teardown

Learn how to use setup and teardown tasks to manage task resources in Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

In production Airflow environments, it's best practice to set up resources and configurations before certain tasks can run, then tear the resources down even if the tasks fail. This pattern can reduce resource utilization and save costs.

Starting in Airflow 2.7, you can use a special type of task to create and delete resources. In this guide, you will learn all about *setup* and *teardown tasks* in Airflow.

<Frame>
  <img alt="DAG with setup/ teardown - all successful" />
</Frame>

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Webinar: [What’s New in Airflow 2.7](https://www.astronomer.io/events/webinars/whats-new-in-airflow-2-7/).
  * Webinar: [Efficient data quality checks with Airflow 2.7](https://www.astronomer.io/events/webinars/efficient-data-quality-checks-with-airflow-2-7/).
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow decorators. See [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).
* Managing dependencies in Airflow. See [Manage task and task group dependencies in Airflow](/docs/learn/managing-dependencies).

## When to use setup/ teardown tasks

Setup/ teardown tasks ensure that the necessary resources to run an Airflow task are set up before a task is executed and that those resources are torn down after the task has completed, regardless of any task failures.

Any existing Airflow task can be designated as a setup or teardown task, with special behavior and added visibility of the setup/ teardown relationship in the Airflow UI.

There are many use cases for setup and teardown tasks. For example, you might want to:

* Manage a Spark cluster to run heavy workloads.
* Manage compute resources to train an ML model.
* Manage the resources to run [data quality](/docs/learn/data-quality) checks.
* Set up storage in your [custom XCom backend](/docs/learn/custom-xcom-backend-strategies) to hold data processed through Airflow tasks, then tear the extra storage down afterwards when the XCom data is no longer needed.

## Setup/ teardown concepts

Any task can be designated as a setup or a teardown task. A setup task, its teardown task, and the tasks in between constitute a *setup/ teardown workflow*.

Tasks that run after a setup task and before the associated teardown task are considered to be *in scope* of the setup/ teardown workflow. Usually these tasks will use the resources set up by the setup task and which the teardown task will dismantle.

Setup/ teardown tasks have different behavior from regular tasks:

* Clearing a task that is in scope of a setup/ teardown workflow will also clear and rerun the associated setup and teardown tasks, ensuring that all resources the task needs are created again for the task rerun and torn down after the task has completed.

* A teardown task will run as long as at least one of its associated setup tasks have completed successfully and all of its upstream tasks have completed, regardless of whether they were successful or not. If all associated setup tasks fail or are skipped, the teardown task will be failed or skipped respectively.

* A teardown task without any associated setup tasks will always run once all upstream worker tasks have completed running, independently of whether they were successful or not.

* When evaluating whether a DAG run was successful, Airflow will ignore teardown tasks by default. This means if a teardown task fails as the final task of a DAG, the DAG is still marked as having succeeded. In the example shown in the screenshot below, the DAG run state isn't impacted by the failure of `tear_down_cluster` and is marked as successful. You can change this behavior by setting `on_failure_fail_dagrun=True` in the [`.as_teardown()` method](#as_setup-and-as_teardown-methods) or [`@teardown` decorator](#@setup-and-@teardown-decorators).

  <Frame>
    <img alt="Successful DAG with failed teardown" />
  </Frame>

* When a teardown task is within a [task group](/docs/learn/task-groups) and a dependency is set on the task group, the teardown task will be ignored when evaluating if a dependency has been met. For example, `run_after_task_group`, which is dependent on the `work_in_the_cluster` task group, will run even if the teardown task has failed or is still running.

  <Frame>
    <img alt="Task group with teardown" />
  </Frame>

* You can have a setup task without an associated teardown task and vice versa. If you define a setup task without a teardown task, everything downstream of the setup task is considered in its scope and will cause the setup task to rerun when cleared.

### Before and after using setup and teardown tasks

Setup and teardown tasks can help you write more robust DAGs by making sure resources are set up at the right moment and torn down even when worker tasks fail.

The following DAG isn't using Airflow setup and teardown functionality. It sets up its resources using a standard task called `provision_cluster`, runs three worker tasks using those resources, and tears down the resources using the `tear_down_cluster` task.

<Frame>
  <img alt="DAG without Setup/ teardown - all successful" />
</Frame>

The way this DAG is set up, a failure in any of the worker tasks will lead to the `tear_down_cluster` task not running. This means that the resources won't be torn down and will continue to incur costs. Additionally, any downstream tasks depending on `tear_down_cluster` will also fail to run unless they have [trigger rules](/docs/learn/airflow-trigger-rules) to run independently of upstream failures.

<Frame>
  <img alt="DAG without setup/ teardown - upstream failure" />
</Frame>

In this example, you can turn the `provision_cluster` task into a setup task and the `tear_down_cluster` into a teardown task by using the code examples shown in [setup/ teardown implementation](#setup/-teardown-implementation).

After you convert the tasks, the **Grid** view shows your setup tasks with an upwards arrow and teardown tasks with a downwards arrow. After you configure the [setup/ teardown workflow](#create-setup/-teardown-workflows) between `provision_cluster` and `tear_down_cluster`, the tasks are connected by a dotted line. The tasks `worker_task_1`, `worker_task_2` and `worker_task_3` are in the scope of this setup/ teardown workflow.

<Frame>
  <img alt="DAG with setup/ teardown - all successful" />
</Frame>

Now, even if one of the worker tasks fails, like `worker_task_2` in the following screenshot, the `tear_down_cluster` task will still run, the resources will be torn down, and downstream tasks will run successfully.

<Frame>
  <img alt="DAG with setup/ teardown - upstream failure" />
</Frame>

Additionally, when you clear any of the worker tasks, both the setup and teardown tasks will also be cleared and rerun. This is useful when you are recovering from a pipeline issue and need to rerun one or more tasks that use a resource independent of the other tasks in the scope.

For example, in the previous DAG, consider if `worker_task_2` failed and `worker_task_3` was unable to run due to its upstream task having failed. If you cleared `worker_task_2` by clicking **Clear task**, both the setup task `provision_cluster` and the teardown task `tear_down_cluster` will be cleared and rerun in addition to `worker_task_2`, `worker_task_3` and `downstream_task`. This lets you completely recover without needing to rerun `worker_task_1` or manually rerun individual tasks.

<Frame>
  <img alt="DAG with setup/ teardown - recovery" />
</Frame>

## Setup/ teardown implementation

There are two ways to turn tasks into setup/ teardown tasks:

* Using the `.as_setup()` and `.as_teardown()` methods on TaskFlow API tasks or traditional operators.
* Using the `@setup` and `@teardown` decorators on a Python function.

Worker tasks can be added to the scope of a setup/ teardown workflow in two ways:

* By being between the setup and teardown tasks in the DAG dependency relationship.
* By using a context manager with the `.teardown()` method.

Which method you choose to add worker tasks to a setup/ teardown scope is a matter of personal preference.

You can define as many setup and teardown tasks in one DAG as you need. In order for Airflow to understand which setup and teardown tasks belong together, you need to [create setup/ teardown workflows](#create-setup/-teardown-workflows).

### `.as_setup()` and `.as_teardown()` methods

Any individual task can be turned into a setup or teardown task.

To turn a task into a setup task, call the `.as_setup()` method on the called task object.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  @task 
  def my_setup_task():
      return "Setting up resources!"

  my_setup_task_obj = my_setup_task()
  my_setup_task_obj.as_setup()

  # it is also possible to call `.as_setup()` directly on the function call
  # my_setup_task().as_setup()
  ```

  <Frame>
    <img alt="Setup task decorator" />
  </Frame>
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  def my_setup_task_func():
      return "Setting up resources!"

  my_setup_task_obj = PythonOperator(
      task_id="my_setup_task",
      python_callable=my_setup_task_func,
  )

  my_setup_task_obj.as_setup()
  ```

  <Frame>
    <img alt="Setup task traditional operator" />
  </Frame>
</details>

To turn a task into a teardown task, call the `.as_teardown()` method on the called task object. Note that you can't have a teardown task without at least one upstream worker task.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  @task
  def worker_task():
      return "Doing some work!"

  @task
  def my_teardown_task():
      return "Tearing down resources!"

  my_teardown_task_obj = my_teardown_task()
  worker_task() >> my_teardown_task_obj.as_teardown()

  # it is also possible to call `.as_teardown()` directly on the function call
  # worker_task() >> my_teardown_task().as_teardown()
  ```

  <Frame>
    <img alt="Teardown task decorator" />
  </Frame>
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  def worker_task_func():
      return "Doing some work!"

  worker_task_obj = PythonOperator(
      task_id="worker_task",
      python_callable=worker_task_func,
  )

  def my_teardown_task_func():
      return "Setting up resources!"

  my_teardown_task_obj = PythonOperator(
      task_id="my_teardown_task",
      python_callable=my_teardown_task_func,
  )

  worker_task_obj >> my_teardown_task_obj.as_teardown()
  ```

  <Frame>
    <img alt="Teardown task traditional" />
  </Frame>
</details>

After you have defined your setup and teardown tasks you need to [define their workflow](#create-setup/-teardown-workflows) in order for Airflow to know which setup and teardown tasks perform actions on the same resources.

### `@setup` and `@teardown` decorators

When working with the TaskFlow API you can also use the `@setup` and `@teardown` decorators to turn any Python function into a setup or teardown task.

```python wrap theme={null}
from airflow.decorators import setup

@setup 
def my_setup_task():
    return "Setting up resources!"

my_setup_task()
```

<Frame>
  <img alt="Setup task decorator" />
</Frame>

As with the `.as_teardown()` method you can't have a `@teardown` task without at least one upstream worker task. The worker task can use the `@task` decorator or be defined with a traditional operator.

```python wrap theme={null}
from airflow.decorators import task, teardown

@task 
def worker_task():
    return "Doing some work!"

@teardown
def my_teardown_task():
    return "Tearing down resources!"

worker_task() >> my_teardown_task()
```

<Frame>
  <img alt="Teardown task decorator" />
</Frame>

After you have defined your setup and teardown tasks you need to [create their workflows](#create-setup/-teardown-workflows) in order for Airflow to know which setup and teardown tasks perform actions on the same resources.

### Create setup/ teardown workflows

Airflow needs to know which setup and teardown tasks are related based on the resources they manage. Setup and teardown tasks can be defined in the same workflow by:

* Providing the setup task object to the `setups` argument in the `.as_teardown()` method of a teardown task object.
* Connecting a setup and a teardown task with a normal task dependency using the bit-shift operator (`>>`) or a dependency function like `chain()`.
* Providing the called object of a task created using the `@setup` decorator as an argument to a task created using the `@teardown` decorator.

Which method you use is a matter of personal preference. However, note that if you are using `@setup` and `@teardown` decorators, you can't use the `setups` argument.

You can have multiple sets of setup and teardown tasks in a DAG, both in [parallel](#parallel-setup/-teardown-workflows) and [nested](#nested-setup/-teardown-workflows) workflows.

There are no limits to how many setup and teardown tasks you can have, nor are there limits to how many worker tasks you can include in their scope.

For example, you could have one task that creates a cluster, a second task that modifies the environment within that cluster, and a third task that tears down the cluster. In this case you could define the first two tasks as setup tasks and the last one as a teardown task, all belonging to the same resource. In a second step, you could add 10 tasks performing actions on that cluster to the scope of the setup/ teardown workflow.

There are multiple methods for linking setup and teardown tasks.

<details>
  <summary>TaskFlow Setups</summary>

  Using the `@task` decorator, you can use the `.as_teardown()` method and the `setups` argument to define which setup tasks are in the same workflow as the teardown task. Note that it is also possible to use [`@setup` and `@teardown` decorators](#@setup-and-@teardown-decorators) instead and link them using direct dependencies.

  ```python wrap theme={null}
  @task
  def my_setup_task():
      return "Setting up resources!"

  @task
  def worker_task():
      return "Doing some work!"

  @task
  def my_teardown_task():
      return "Tearing down resources!"

  my_setup_task_obj = my_setup_task()

  (
      my_setup_task_obj#.as_setup() does not need to be called anymore
      >> worker_task()
      >> my_teardown_task().as_teardown(setups=my_setup_task_obj)
  )
  ```

  <Frame>
    <img alt="Setup/ teardown method decorator" />
  </Frame>
</details>

<details>
  <summary>Traditional Setups</summary>

  If you are using traditional Airflow operators, you can use the `.as_teardown()` method and the `setups` argument to define which setup tasks are in the same workflow as the teardown task.

  ```python wrap theme={null}
  def my_setup_task_func():
      return "Setting up resources!"

  def worker_task_func():
      return "Doing some work!"

  def my_teardown_task_func():
      return "Tearing down resources!"

  my_setup_task_obj = PythonOperator(
      task_id="my_setup_task",
      python_callable=my_setup_task_func,
  )

  worker_task_obj = PythonOperator(
      task_id="worker_task",
      python_callable=worker_task_func,
  )

  my_teardown_task_obj = PythonOperator(
      task_id="my_teardown_task",
      python_callable=my_teardown_task_func,
  )

  (
      my_setup_task_obj#.as_setup() does not need to be called anymore
      >> worker_task_obj
      >> my_teardown_task_obj.as_teardown(setups=my_setup_task_obj)
  )
  ```

  <Frame>
    <img alt="Setup/ teardown relationships traditional" />
  </Frame>
</details>

<details>
  <summary>TaskFlow Direct</summary>

  Instead of using the `setups` argument you can directly link the setup and teardown tasks with a traditional dependency. Whenever you define a direct dependency between a setup and a teardown task Airflow will interpret this as them being in the same workflow together, no matter what actions the tasks actually perform.

  ```python wrap theme={null}
  (
      my_setup_task_obj.as_setup()  # calling .as_setup() is necessary
      >> worker_task()
      >> my_teardown_task_obj.as_teardown()
  )

  my_setup_task_obj >> my_teardown_task_obj
  ```

  This code creates an identical DAG using the `setups` argument.

  ```python wrap theme={null}
  (
      my_setup_task_obj#.as_setup() is not necessary
      >> worker_task()
      >> my_teardown_task_obj.as_teardown(setups=my_setup_task_obj)
  )
  ```

  <Frame>
    <img alt="Setup/ teardown method decorator" />
  </Frame>
</details>

<details>
  <summary>Decorators</summary>

  With the`@setup` and `@teardown` decorators, you can define the setup/ teardown workflow between two tasks either by defining direct dependencies or by providing the object of the setup task as an argument to the teardown task.

  The latter pattern is often used to pass information like a resource id from the setup task to the teardown task.

  ```python wrap theme={null}
  from airflow.decorators import task, setup, teardown

  @setup
  def my_setup_task():
      print("Setting up resources!")
      my_cluster_id = "cluster-2319"
      return my_cluster_id

  @task
  def worker_task():
      return "Doing some work!"

  @teardown
  def my_teardown_task(my_cluster_id):
      return f"Tearing down {my_cluster_id}!"

  my_setup_task_obj = my_setup_task()
  my_setup_task_obj >> worker_task() >> my_teardown_task(my_setup_task_obj)
  ```

  <Frame>
    <img alt="Setup/ teardown method decorator" />
  </Frame>
</details>

<details>
  <summary>Context Manager</summary>

  You can also use a task that calls the `.as_teardown()` method to wrap a set of tasks that should be in scope of a setup/ teardown workflow. The code snippet below shows three tasks being in scope of the setup/ teardown workflow created by `my_cluster_setup_task` and `my_cluster_teardown_task`.

  ```python wrap theme={null}
  with my_cluster_teardown_task_obj.as_teardown(setups=my_cluster_setup_task_obj):
      worker_task_1() >> [worker_task_2(),  worker_task_3()]
  ```

  <Frame>
    <img alt="Setup/ teardown created using a context manager" />
  </Frame>

  Note that a task that was already instantiated outside of the context manager can still be added to the scope, but you have to do this explicitly using the `.add_task()` method on the context manager object.

  ```python wrap theme={null}
  # task instantiation outside of the context manager
  worker_task_1_obj = worker_task_1()

  with my_cluster_teardown_task_obj.as_teardown(
      setups=my_cluster_setup_task_obj
  ) as my_scope:
      # adding the task to the context manager
      my_scope.add_task(worker_task_1_obj)
  ```
</details>

#### Use multiple setup/ teardown tasks in one workflow

To define several setup tasks for one teardown task, you can pass a list of setup tasks to the `setups` argument. You don't need to call `.as_setup()` on any of the setup tasks.

```python wrap theme={null}
(
    [my_setup_task_obj_1, my_setup_task_obj_2, my_setup_task_obj_3]
    >> worker_task()
    >> my_teardown_task().as_teardown(
        setups=[my_setup_task_obj_1, my_setup_task_obj_2, my_setup_task_obj_3]
    )
)
```

<Frame>
  <img alt="Setup/ teardown relationships multiple setup" />
</Frame>

To define several teardown tasks for one setup task, you have to provide the setup task object to the `setups` argument of the `.as_teardown()` method of each teardown task.

```python wrap theme={null}
(
    my_setup_task_obj
    >> worker_task()
    >> [
        my_teardown_task_obj_1.as_teardown(setups=my_setup_task_obj),
        my_teardown_task_obj_2.as_teardown(setups=my_setup_task_obj),
        my_teardown_task_obj_3.as_teardown(setups=my_setup_task_obj),
    ]
)
```

<Frame>
  <img alt="Setup/ teardown relationships multiple setup" />
</Frame>

If your setup/ teardown workflow contains more than one setup and one teardown task, you need to define several dependencies, when not using the `setups` argument. Each setup task needs to be set as an upstream dependency to each teardown task. The example below shows a setup/ teardown workflow containing two setup tasks and two teardown tasks. To define the workflow, you need to set four dependencies.

```python wrap theme={null}
(
    [my_setup_task_obj_1.as_setup(), my_setup_task_obj_2.as_setup()]
    >> worker_task()
    >> [my_teardown_task_obj_1.as_teardown(), my_teardown_task_obj_2.as_teardown()]
)

# defining the dependency between each setup and each teardown task
my_setup_task_obj_1 >> my_teardown_task_obj_1
my_setup_task_obj_1 >> my_teardown_task_obj_2
my_setup_task_obj_2 >> my_teardown_task_obj_1
my_setup_task_obj_2 >> my_teardown_task_obj_2
```

This code creates an identical DAG using the `setups` argument.

```python wrap theme={null}
(
    [my_setup_task_obj_1, my_setup_task_obj_2]
    >> worker_task()
    >> [
        my_teardown_task_obj_1.as_teardown(
            setups=[my_setup_task_obj_1, my_setup_task_obj_2]
        ),
        my_teardown_task_obj_2.as_teardown(
            setups=[my_setup_task_obj_1, my_setup_task_obj_2]
        ),
    ]
)
```

<Frame>
  <img alt="Multiple setups/ teardowns" />
</Frame>

#### Parallel setup/ teardown workflows

You can have several independent sets of setup and teardown tasks in the same DAG. For example, you might have a workflow of tasks that sets up and tears down a cluster and another workflow that sets up and tears down a temporary database.

<details>
  <summary>Decorators</summary>

  ```python expandable wrap theme={null}
  from airflow.decorators import task, setup, teardown

  @setup
  def my_cluster_setup_task():
      print("Setting up resources!")
      my_cluster_id = "cluster-2319"
      return my_cluster_id

  @task
  def my_cluster_worker_task():
      return "Doing some work!"

  @teardown
  def my_cluster_teardown_task(my_cluster_id):
      return f"Tearing down {my_cluster_id}!"

  @setup
  def my_database_setup_task():
      print("Setting up my database!")
      my_database_name = "DWH"
      return my_database_name

  @task
  def my_database_worker_task():
      return "Doing some work!"

  @teardown
  def my_database_teardown_task(my_database_name):
      return f"Tearing down {my_database_name}!"

  my_setup_task_obj = my_cluster_setup_task()
  (
      my_setup_task_obj
      >> my_cluster_worker_task()
      >> my_cluster_teardown_task(my_setup_task_obj)
  )

  my_database_setup_obj = my_database_setup_task()
  (
      my_database_setup_obj
      >> my_database_worker_task()
      >> my_database_teardown_task(my_database_setup_obj)
  )
  ```
</details>

<details>
  <summary>Methods</summary>

  ```python expandable wrap theme={null}
  @task
  def my_cluster_setup_task():
      print("Setting up resources!")
      my_cluster_id = "cluster-2319"
      return my_cluster_id

  @task
  def my_cluster_worker_task():
      return "Doing some work!"

  @task
  def my_cluster_teardown_task(my_cluster_id):
      return f"Tearing down {my_cluster_id}!"

  @task
  def my_database_setup_task():
      print("Setting up my database!")
      my_database_name = "DWH"
      return my_database_name

  @task
  def my_database_worker_task():
      return "Doing some work!"

  @task
  def my_database_teardown_task(my_database_name):
      return f"Tearing down {my_database_name}!"

  my_setup_task_obj = my_cluster_setup_task()
  (
      my_setup_task_obj
      >> my_cluster_worker_task()
      >> my_cluster_teardown_task(my_setup_task_obj).as_teardown(
          setups=my_setup_task_obj
      )
  )

  my_database_setup_obj = my_database_setup_task()
  (
      my_database_setup_obj
      >> my_database_worker_task()
      >> my_database_teardown_task(my_database_setup_obj).as_teardown(
          setups=my_database_setup_obj
      )
  )
  ```
</details>

<Frame>
  <img alt="Parallel groups of setup/ teardown" />
</Frame>

#### Nested setup/ teardown workflows

You can nest setup and teardown tasks to have an outer and inner scope. This is useful if you have basic resources, such as a cluster that you want to set up once and then tear down after all the work is done, but you also have resources running on that cluster that you want to set up and tear down for individual groups of tasks.

The example below shows the dependency code for a simple structure with an outer and inner setup/ teardown workflow:

* `outer_setup` and `outer_teardown` are the outer setup and teardown tasks.
* `inner_setup` and `inner_teardown` are the inner setup and teardown tasks and both are in scope of the outer setup/ teardown workflow.
* `inner_worker_1` and `inner_worker_2` are worker tasks that are in scope of the inner setup/ teardown workflow. All tasks in scope of the inner setup/ teardown workflow will also be in scope of the outer setup/ teardown workflow.
* `outer_worker_1`, `outer_worker_2`, `outer_worker_3` are worker tasks that are in scope of the outer setup/ teardown workflow.

```python wrap theme={null}
outer_setup_obj = outer_setup()
inner_setup_obj = inner_setup()
outer_teardown_obj = outer_teardown()

(
    outer_setup_obj
    >> inner_setup_obj
    >> [inner_worker_1(), inner_worker_2()]
    >> inner_teardown().as_teardown(setups=inner_setup_obj)
    >> [outer_worker_1(), outer_worker_2()]
    >> outer_teardown_obj.as_teardown(setups=outer_setup_obj)
)

outer_setup_obj >> outer_worker_3() >> outer_teardown_obj
```

<Frame>
  <img alt="Setup/ teardown nesting" />
</Frame>

Clearing a task will clear all setups and teardowns the task is in scope of, in addition to all downstream tasks. For example:

* Clearing any of the outer worker tasks (`outer_worker_1`, `outer_worker_2`, `outer_worker_3`) will also clear `outer_setup`, `outer_teardown`.
* Clearing any of the inner worker tasks (`inner_worker_1`, `inner_worker_2`) will clear `inner_setup`, `inner_teardown`, `outer_setup`, and `outer_teardown`. Additionally `outer_worker_1` and `outer_worker_2` will be cleared because they are downstream of the inner worker tasks. `outer_worker_3` won't be cleared because it runs parallel to the inner worker tasks.

### Narrow the scope of a setup task

If you have a setup task with no associated downstream task, you can narrow the scope of the setup task by using an empty task as its teardown. For example, if `my_worker_task_3_obj` doesn't need the resources created by `my_setup_task` and shouldn't cause a rerun of the setup task when cleared, you can add an empty teardown task in the dependency chain:

```python wrap theme={null}
my_setup_task >> [my_worker_task_1_obj >> my_worker_task_2_obj] >> my_worker_task_3_obj

[my_worker_task_1_obj >> my_worker_task_2_obj] >> EmptyOperator(
    task_id="empty_task"
).as_teardown(setups=my_setup_task)
```

## Example DAG

The DAG shown in this example mimics a setup/ teardown pattern that you can run locally. The setup/ teardown workflow consists of the following tasks:

* The `create_csv` task is a setup task that creates a CSV file in a directory specified as a [DAG param](/docs/learn/airflow-params).
* The `write_to_csv` task is a setup task that writes data to the CSV file.
* The `fetch_data` task is a setup task that fetches data from a remote source and writes it to the CSV file.
* The `delete_csv` task is the associated teardown task and deletes the resource of the CSV file.
* The `get_average_age_obj` task is in scope of the setup/ teardown workflow. If this task fails, the DAG still needs to delete the "CSV file" afterwards (to make it more real, consider the CSV file to be an expensive cluster).

  To recover from a failure when rerunning the `get_average_age_obj` task, you always need the CSV file to be created again, as well as the data to be fetched again and written to the CSV file. Because the task is in scope of `create_csv`, `write_to_csv`, and `fetch_data`, these tasks will also rerun when you rerun `get_average_age_obj`.

The DAG contains 3 tasks which aren't in scope of the setup/ teardown workflow:

* The `start` task is an empty task at the start of the DAG.
* The `report_file_path` task is a task that prints the path of the CSV file to the logs.
* The `end` task is an empty task at the end of the DAG.

This DAG comes with a convenience parameter to test setup/ teardown functionality. Toggle `fetch_bad_data` in the **Trigger DAG** view to cause bad data to get into the pipeline and the `get_average_age_obj` to fail. You will see that `delete_csv` will still run and delete the CSV file. In a real-world scenario, after fixing the data issue you would clear the `get_average_age_obj` task and all tasks of the setup/ teardown workflow would rerun and complete successfully.

<details>
  <summary>Methods</summary>

  ```python expandable wrap theme={null}
  """
  ## Use `.as_teardown()` in a simple local example to enable setup/teardown functionality

  DAG that uses setup/teardown to prepare a CSV file to write to and then showcases the
  behavior in case faulty data is fetched.
  """

  from airflow.decorators import dag, task
  from airflow.models.baseoperator import chain
  from pendulum import datetime
  from airflow.models.param import Param
  from airflow.operators.empty import EmptyOperator
  import os
  import csv
  import time


  def get_params_helper(**context):
      folder = context["params"]["folder"]
      filename = context["params"]["filename"]
      cols = context["params"]["cols"]
      return folder, filename, cols


  @dag(
      start_date=datetime(2023, 7, 1),
      schedule=None,
      catchup=False,
      params={
          "folder": "include/my_data",
          "filename": "data.csv",
          "cols": ["id", "name", "age"],
          "fetch_bad_data": Param(False, type="boolean"),
      },
      tags=[".is_teardown()", "setup/teardown"],
  )
  def setup_teardown_csv_methods():
      start = EmptyOperator(task_id="start")
      end = EmptyOperator(task_id="end")

      @task
      def report_filepath(**context):
          folder, filename, cols = get_params_helper(**context)
          print(f"Filename: {folder}/{filename}")

      @task
      def create_csv(**context):
          folder, filename, cols = get_params_helper(**context)

          if not os.path.exists(folder):
              os.makedirs(folder)

          with open(f"{folder}/{filename}", "w", newline="") as f:
              writer = csv.writer(f)
              writer.writerows([cols])

      @task
      def fetch_data(**context):
          bad_data = context["params"]["fetch_bad_data"]

          if bad_data:
              return [
                  [1, "Joe", "Forty"],
                  [2, "Tom", 29],
                  [3, "Lea", 19],
              ]
          else:
              return [
                  [1, "Joe", 40],
                  [2, "Tom", 29],
                  [3, "Lea", 19],
              ]

      @task
      def write_to_csv(data, **context):
          folder, filename, cols = get_params_helper(**context)

          with open(f"{folder}/{filename}", "a", newline="") as f:
              writer = csv.writer(f)
              writer.writerows(data)

          time.sleep(10)

      @task
      def get_average_age(**context):
          folder, filename, cols = get_params_helper(**context)

          with open(f"{folder}/{filename}", "r", newline="") as f:
              reader = csv.reader(f)
              next(reader)
              ages = [int(row[2]) for row in reader]

          return sum(ages) / len(ages)

      @task
      def delete_csv(**context):
          folder, filename, cols = get_params_helper(**context)

          os.remove(f"{folder}/{filename}")

          if not os.listdir(f"{folder}"):
              os.rmdir(f"{folder}")

      start >> report_filepath() >> end

      create_csv_obj = create_csv()
      fetch_data_obj = fetch_data()
      write_to_csv_obj = write_to_csv(fetch_data_obj)
      get_average_age_obj = get_average_age()
      delete_csv_obj = delete_csv()

      chain(
          start,
          create_csv_obj,
          write_to_csv_obj,
          get_average_age_obj,
          delete_csv_obj.as_teardown(
              setups=[create_csv_obj, write_to_csv_obj, fetch_data_obj]
          ),
          end,
      )


  setup_teardown_csv_methods()
  ```
</details>

<details>
  <summary>Decorators</summary>

  ```python expandable wrap theme={null}
  """
  ## Use `@setup` and `@teardown` in a simple local example to enable setup/teardown functionality

  DAG that uses setup/teardown to prepare a CSV file to write to and then showcases the
  behavior in case faulty data is fetched.
  """

  from airflow.decorators import dag, task, setup, teardown
  from airflow.models.baseoperator import chain
  from pendulum import datetime
  from airflow.models.param import Param
  from airflow.operators.empty import EmptyOperator
  import os
  import csv
  import time


  def get_params_helper(**context):
      folder = context["params"]["folder"]
      filename = context["params"]["filename"]
      cols = context["params"]["cols"]
      return folder, filename, cols


  @dag(
      start_date=datetime(2023, 7, 1),
      schedule=None,
      catchup=False,
      params={
          "folder": "include/my_data",
          "filename": "data.csv",
          "cols": ["id", "name", "age"],
          "fetch_bad_data": Param(False, type="boolean"),
      },
      tags=["@setup", "@teardown", "setup/teardown"],
  )
  def setup_teardown_csv_decorators():
      start = EmptyOperator(task_id="start")
      end = EmptyOperator(task_id="end")

      @task
      def report_filepath(**context):
          folder, filename, cols = get_params_helper(**context)
          print(f"Filename: {folder}/{filename}")

      @setup
      def create_csv(**context):
          folder, filename, cols = get_params_helper(**context)

          if not os.path.exists(folder):
              os.makedirs(folder)

          with open(f"{folder}/{filename}", "w", newline="") as f:
              writer = csv.writer(f)
              writer.writerows([cols])

      @setup
      def fetch_data(**context):
          bad_data = context["params"]["fetch_bad_data"]

          if bad_data:
              return [
                  [1, "Joe", "Forty"],
                  [2, "Tom", 29],
                  [3, "Lea", 19],
              ]
          else:
              return [
                  [1, "Joe", 40],
                  [2, "Tom", 29],
                  [3, "Lea", 19],
              ]

      @setup
      def write_to_csv(data, **context):
          folder, filename, cols = get_params_helper(**context)

          with open(f"{folder}/{filename}", "a", newline="") as f:
              writer = csv.writer(f)
              writer.writerows(data)

          time.sleep(10)

      @task
      def get_average_age(**context):
          folder, filename, cols = get_params_helper(**context)

          with open(f"{folder}/{filename}", "r", newline="") as f:
              reader = csv.reader(f)
              next(reader)
              ages = [int(row[2]) for row in reader]

          return sum(ages) / len(ages)

      @teardown
      def delete_csv(**context):
          folder, filename, cols = get_params_helper(**context)

          os.remove(f"{folder}/{filename}")

          if not os.listdir(f"{folder}"):
              os.rmdir(f"{folder}")

      start >> report_filepath() >> end

      create_csv_obj = create_csv()
      fetch_data_obj = fetch_data()
      write_to_csv_obj = write_to_csv(fetch_data_obj)
      get_average_age_obj = get_average_age()
      delete_csv_obj = delete_csv()

      chain(
          start,
          create_csv_obj,
          write_to_csv_obj,
          get_average_age_obj,
          delete_csv_obj,
          end,
      )

      # when using @setup and @teardown the tasks can be linked using normal dependency syntax
      # or by leveraging task flow (see the complex example)
      create_csv_obj >> delete_csv_obj
      fetch_data_obj >> delete_csv_obj
      write_to_csv_obj >> delete_csv_obj


  setup_teardown_csv_decorators()
  ```
</details>

<Frame>
  <img alt="Setup/ teardown example DAG" />
</Frame>


# Using Airflow to Execute SQL
Source: https://astronomer.io/docs/learn/airflow-sql

Learn the best practices for executing SQL from your DAG. Get to know Airflow’s SQL-related operators and see how to use Airflow for common SQL use cases.

Executing SQL queries is one of the most common use cases for data pipelines. Whether you're extracting and loading data, calling a stored procedure, or executing a complex query for a report, Airflow can help you orchestrate the process.

In this guide you'll learn about the best practices for executing SQL from your DAG, review the most commonly used Airflow SQL-related operators, and then use sample code to implement a few common SQL use cases.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Snowflake basics. See [Introduction to Snowflake](https://docs.snowflake.com/en/user-guide-intro.html).

## Best practices for executing SQL from your DAG

No matter what database or SQL version you're using, there are many ways to execute your queries using Airflow. Once you determine how to execute your queries, the following tips will help you keep your DAGs clean, readable, and efficient for execution.

### Use hooks and operators

The [Common SQL provider](https://airflow.apache.org/registry/providers/common-sql/) is a great place to start when looking for SQL-related operators. It includes the [`SQLExecuteQueryOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLExecuteQueryOperator) operator, which is a generic operator that can be used with a variety of databases, including Snowflake and Postgres.

For some databases more specialized operators exist and are part of the service-specific provider package. For example the [`SnowflakeSqlApiOperator`](https://airflow.apache.org/registry/providers/snowflake#snowflake-snowflake-SnowflakeSqlApiOperator) supports submitting multiple SQL statements in a single request.

Hooks to use with a database in an `@task` decorated function or in a `PythonOperator` are typically contained in the provider package for that database. For example, the [`PostgresHook`](https://airflow.apache.org/registry/providers/postgres#postgres-postgres-PostgresHook) is part of the Postgres provider package.

<Tip>
  If you are looking to run data quality checks on your data, you can use the SQL check operators, see [Run data quality checks using SQL check operators](/docs/learn/airflow-sql-data-quality) for more information.
</Tip>

### Keep lengthy SQL code out of your DAG

Astronomer recommends avoiding long SQL statements in your DAG file. If you have a SQL query, you should keep it in its own .sql file.

If you use the Astro CLI, you can store supporting code like SQL scripts in the `include/` directory:

```bash wrap theme={null}
├─ dags/  
|    └─ example-dag.py  
├─ plugins/  
├─ include/  
|    ├─ query1.sql  
|    └─ query2.sql    
├─ Dockerfile  
├─ packages.txt  
└─ requirements.txt
```

An exception to this rule could be very short queries (such as `SELECT * FROM table`). Putting one-line queries like this directly in the DAG is fine if it makes your code more readable.

## Examples

This section contains a few examples of how to use Airflow to execute SQL queries. The examples are based on Snowflake, but the concepts apply to most relational databases.

### Example 1: Execute a query

In this first example, a DAG executes two simple interdependent queries using the [`SQLExecuteQueryOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLExecuteQueryOperator).

First, you need to make sure you have the Common SQL and the Snowflake provider installed. If you use the Astro CLI, you can add the following lines to your `requirements.txt` file. You need the Snowflake provider in order to connect to Snowflake, and the Common SQL provider to use the `SQLExecuteQueryOperator`:

```text wrap theme={null}
apache-airflow-providers-snowflake
apache-airflow-providers-common-sql
```

Next, you need to define your DAG:

```python wrap theme={null}
from airflow.sdk import chain, dag
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator


@dag(template_searchpath="/usr/local/airflow/include")  # Path to the SQL files
def execute_snowflake_queries():
    run_query1 = SQLExecuteQueryOperator(
        task_id="run_query1", conn_id="my_snowflake_conn", sql="query1.sql"
    )

    run_query2 = SQLExecuteQueryOperator(
        task_id="run_query2", conn_id="my_snowflake_conn", sql="query2.sql"
    )

    chain(run_query1, run_query2)


execute_snowflake_queries()
```

The `template_searchpath` argument in the DAG definition tells the DAG to look in the given folder for scripts, so you can now add two SQL scripts to your project. In this example, those scripts are `query1.sql` and `query2.sql`, which contain the following SQL code respectively:

```sql wrap theme={null}
CREATE OR REPLACE TABLE MY_DATABASE.MY_SCHEMA.MY_NEW_TABLE (
    id INT,
    grocery STRING
);
```

```sql wrap theme={null}
INSERT INTO MY_DATABASE.MY_SCHEMA.MY_NEW_TABLE (id, grocery)
VALUES 
    (1, 'Chocolate'),
    (2, 'Eggs'),
    (3, 'Cake');
```

Note that the SQL in these files could be any type of query you need to execute.

Finally, you need to set up a connection to your database service, in this case a connection to Snowflake with the connection ID `my_snowflake_conn`. There are a few ways to manage connections using Astronomer, see the [Connections guide](/docs/learn/connections) for more information on connections in general and the [Snowflake Connection](/docs/learn/connections/snowflake) guide for more information on how to set up a Snowflake connection.

### Example 2: Execute a query with parameters

Using Airflow, you can also parameterize your SQL queries to use information from the Airflow context. Consider when you have a query that selects data from a table for a date that you want to dynamically update. You can execute the query using the same setup as in Example 1, but with a few adjustments.

Your DAG will look like the following:

```python expandable wrap theme={null}
from airflow.decorators import dag
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
from pendulum import datetime, duration

default_args = {
    "owner": "airflow",
    "depends_on_past": False,
    "email_on_failure": False,
    "email_on_retry": False,
    "retries": 1,
    "retry_delay": duration(minutes=1),
}


@dag(
    start_date=datetime(2020, 6, 1),
    max_active_runs=3,
    schedule="@daily",
    default_args=default_args,
    template_searchpath="/usr/local/airflow/include",
    catchup=False,
)
def parameterized_query():
    opr_param_query = SQLExecuteQueryOperator(
        task_id="param_query", conn_id="snowflake", sql="param-query.sql"
    )

    opr_param_query


parameterized_query()
```

In this example, the query has been parameterized to dynamically select data for one day before the DAG's logical date.

Astronomer recommends using Airflow context information or macros whenever possible to increase flexibility and make your workflows [idempotent](https://en.wikipedia.org/wiki/Idempotence). The above example will work with any [Airflow context](/docs/learn/airflow-context) information. For example, you could access a [DAG-level param](/docs/learn/airflow-params) using the `params` dictionary:

```sql wrap theme={null}
SELECT *
FROM STATE_DATA
WHERE state = {{ params['my_state'] }}
```

You can also pass information to your SQL file using the `parameters` argument in the `SQLExecuteQueryOperator`. This is useful if you want to pass a value that is derived from another task in your DAG.

```python wrap theme={null}
from airflow.sdk import chain, dag, task
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator

@dag
def using_parameters_argument():

    @task
    def get_grocery():
        return "Chocolate"

    _get_grocery = get_grocery()

    opr_param_query = SQLExecuteQueryOperator(
        task_id="param_query",
        conn_id="my_snowflake_conn",
        sql="""
            SELECT *
            FROM DEMO_DB.DEMO_SCHEMA.MY_NEW_TABLE
            WHERE grocery = %(my_grocery)s;
        """,
        parameters={"my_grocery": _get_grocery},
    )

    chain(
        _get_grocery,
        opr_param_query,
    )

using_parameters_argument()

```


# Run data quality checks using SQL check operators
Source: https://astronomer.io/docs/learn/airflow-sql-data-quality

Learn how to use the SQLColumnCheckOperator, SQLTableCheckOperator and SQLCheckOperator.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

Data quality is key to the success of an organization's data systems. With in-DAG quality checks, you can halt pipelines and alert stakeholders before bad data makes its way to a production lake or warehouse.

The SQL check operators in the [Common SQL provider](https://airflow.apache.org/registry/providers/common-sql/) provide a simple and effective way to implement data quality checks in your Airflow DAGs. Using this set of operators, you can quickly develop a pipeline specifically for checking data quality, or you can add data quality checks to existing pipelines with just a few more lines of code.

This tutorial shows how to use three SQL check operators (`SQLColumnCheckOperator`, `SQLTableCheckOperator`, and `SQLCheckOperator`) to build a robust data quality suite for your DAGs.

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Webinar: [Implementing Data Quality Checks in Airflow](https://www.astronomer.io/events/webinars/implementing-data-quality-checks-in-airflow/).
  * Webinar: [Efficient data quality checks with Airflow 2.7](https://www.astronomer.io/events/webinars/efficient-data-quality-checks-with-airflow-2-7/).
  * Example repository: [data quality demo](https://github.com/astronomer/airflow-data-quality-demo/).
</Tip>

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, you should have an understanding of:

* How to design a data quality process. See [Data quality and Airflow](/docs/learn/data-quality).
* Running SQL from Airflow. See [Using Airflow to execute SQL](/docs/learn/airflow-sql).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* Access to a relational database. You can use an in-memory SQLite database for which you'll need to install the [SQLite provider](https://airflow.apache.org/registry/providers/sqlite/). Note that currently the operators can't support BigQuery `job_id`s.
* A love for birds.

## Step 1: Configure your Astro project

To use SQL check operators, install the [Common SQL provider](https://airflow.apache.org/registry/providers/common-sql/) in your Astro project.

1. Run the following commands to create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-sql-check-tutorial && cd astro-sql-check-tutorial
   $ astro dev init
   ```

2. Add the Common SQL provider and the SQLite provider to your Astro project `requirements.txt` file.

   ```text wrap theme={null}
   apache-airflow-providers-common-sql==1.5.2
   apache-airflow-providers-sqlite==3.4.2
   ```

## Step 2: Create a connection to SQLite

1. In the Airflow UI, go to **Admin** > **Connections** and click **+**.

2. Create a new connection named `sqlite_conn` and choose the `SQLite` connection type. Enter the following information:

   * **Connection Id**: `sqlite_conn`.
   * **Connection Type**: `SQLite`.
   * **Host**: `/tmp/sqlite.db`.

## Step 3: Add a SQL file with a custom check

1. In your `include` folder, create a file called `custom_check.sql`.

2. Copy and paste the following SQL statement into the file:

   ```sql wrap theme={null}
   WITH all_combinations_unique AS (
   SELECT DISTINCT bird_name, observation_year AS combos_unique
     FROM '{{ params.table_name }}'
     )
   SELECT CASE
     WHEN COUNT(*) = COUNT(combos_unique) THEN 1
     ELSE 0
   END AS is_unique
   FROM '{{ params.table_name }}' JOIN all_combinations_unique;
   ```

   This SQL statement returns 1 if all combinations of `bird_name` and `observation_year` in a templated table are unique, and 0 if not.

## Step 4: Create a DAG using SQL check operators

1. Start Airflow by running `astro dev start`.

2. Create a new file in your `dags` folder called `sql_data_quality.py`.

3. Copy and paste the following DAG code into the file:

   ```python expandable wrap theme={null}
   """
   ## Check data quality using SQL check operators

   This DAG creates a toy table about birds in SQLite to run data quality checks on using the
   SQLColumnCheckOperator, SQLTableCheckOperator, and SQLCheckOperator.
   """

   from airflow.decorators import dag
   from airflow.providers.common.sql.operators.sql import (
       SQLColumnCheckOperator,
       SQLTableCheckOperator,
       SQLCheckOperator,
   )
   from airflow.providers.sqlite.operators.sqlite import SqliteOperator
   from pendulum import datetime

   _CONN_ID = "sqlite_conn"
   _TABLE_NAME = "birds"


   @dag(
       start_date=datetime(2023, 7, 1),
       schedule=None,
       catchup=False,
       template_searchpath=["/usr/local/airflow/include/"],
   )
   def sql_data_quality():
       create_table = SqliteOperator(
           task_id="create_table",
           sqlite_conn_id=_CONN_ID,
           sql=f"""
           CREATE TABLE IF NOT EXISTS {_TABLE_NAME} (
               bird_name VARCHAR,
               observation_year INT,
               bird_happiness INT
           );
           """,
       )

       populate_data = SqliteOperator(
           task_id="populate_data",
           sqlite_conn_id=_CONN_ID,
           sql=f"""
               INSERT INTO {_TABLE_NAME} (bird_name, observation_year, bird_happiness) VALUES
                   ('King vulture (Sarcoramphus papa)', 2022, 9),
                   ('Victoria Crowned Pigeon (Goura victoria)', 2021, 10),
                   ('Orange-bellied parrot (Neophema chrysogaster)', 2021, 9),
                   ('Orange-bellied parrot (Neophema chrysogaster)', 2020, 8),
                   (NULL, 2019, 8),
                   ('Indochinese green magpie (Cissa hypoleuca)', 2018, 10);
           """,
       )

       column_checks = SQLColumnCheckOperator(
           task_id="column_checks",
           conn_id=_CONN_ID,
           table=_TABLE_NAME,
           partition_clause="bird_name IS NOT NULL",
           column_mapping={
               "bird_name": {
                   "null_check": {"equal_to": 0},
                   "distinct_check": {"geq_to": 2},
               },
               "observation_year": {"max": {"less_than": 2023}},
               "bird_happiness": {"min": {"greater_than": 0}, "max": {"leq_to": 10}},
           },
       )

       table_checks = SQLTableCheckOperator(
           task_id="table_checks",
           conn_id=_CONN_ID,
           table=_TABLE_NAME,
           checks={
               "row_count_check": {"check_statement": "COUNT(*) >= 3"},
               "average_happiness_check": {
                   "check_statement": "AVG(bird_happiness) >= 9",
                   "partition_clause": "observation_year >= 2021",
               },
           },
       )

       custom_check = SQLCheckOperator(
           task_id="custom_check",
           conn_id=_CONN_ID,
           sql="custom_check.sql",
           params={"table_name": _TABLE_NAME},
       )

       create_table >> populate_data >> [column_checks, table_checks, custom_check]


   sql_data_quality()
   ```

   This DAG creates and populates a small SQLite table `birds` with information about birds. Then, three tasks containing data quality checks are run on the table:

   * The `column_checks` task uses the `SQLColumnCheckOperator` to run the column-level checks provided to the `column_mapping` dictionary. The task also uses an operator-level [`partition_clause`](#partition_clause) to only run the checks on rows where the `bird_name` column isn't null.

   * The `table_checks` task uses the `SQLTableCheckOperator` to run two checks on the whole table:
     * `row_count_check`: This check makes sure the table has at least three rows.
     * `average_happiness_check`: This check makes sure the average happiness of the birds is at least 9. This check has a check-level `partition_clause` which means the check only runs on rows with observations from 2021 onward.

   * The `custom_check` task uses the `SQLCheckOperator`. This operator can run any SQL statement that returns a single row and will deem the data quality check failed if that row contains any value [Python bool casting](https://docs.python.org/3/library/stdtypes.html) evaluates as `False`, for example `0`. Otherwise, the data quality check and the task will be marked as successful. This task will run the SQL statement in the file `include/custom_check.sql` on the `table_name` passed as a parameter. Note that in order to run SQL stored in a file, the path to the SQL file has to be added to the `template_searchpath` parameter of the DAG.

4. Open Airflow at `http://localhost:8080/`. Run the DAG manually by clicking the play button, then click the DAG name to view the DAG in the **Grid** view. All checks are set up to pass.

   <Frame>
     <img alt="Data quality check DAG grid view" />
   </Frame>

5. View the logs of the SQL check operators to get detailed information about the checks that were run and their results:

   For example, the logs for the `column_checks` task show the five individual checks that were run on three columns:

   ```text wrap theme={null}
   [2023-07-04, 11:18:01 UTC] {sql.py:374} INFO - Running statement: SELECT col_name, check_type, check_result FROM (
         SELECT 'bird_name' AS col_name, 'null_check' AS check_type, bird_name_null_check AS check_result
         FROM (SELECT SUM(CASE WHEN bird_name IS NULL THEN 1 ELSE 0 END) AS bird_name_null_check FROM birds WHERE bird_name IS NOT NULL) AS sq
     UNION ALL
         SELECT 'bird_name' AS col_name, 'distinct_check' AS check_type, bird_name_distinct_check AS check_result
         FROM (SELECT COUNT(DISTINCT(bird_name)) AS bird_name_distinct_check FROM birds WHERE bird_name IS NOT NULL) AS sq
     UNION ALL
         SELECT 'observation_year' AS col_name, 'max' AS check_type, observation_year_max AS check_result
         FROM (SELECT MAX(observation_year) AS observation_year_max FROM birds WHERE bird_name IS NOT NULL) AS sq
     UNION ALL
         SELECT 'bird_happiness' AS col_name, 'min' AS check_type, bird_happiness_min AS check_result
         FROM (SELECT MIN(bird_happiness) AS bird_happiness_min FROM birds WHERE bird_name IS NOT NULL) AS sq
     UNION ALL
         SELECT 'bird_happiness' AS col_name, 'max' AS check_type, bird_happiness_max AS check_result
         FROM (SELECT MAX(bird_happiness) AS bird_happiness_max FROM birds WHERE bird_name IS NOT NULL) AS sq
     ) AS check_columns, parameters: None
   [2023-07-04, 11:18:01 UTC] {sql.py:397} INFO - Record: [('bird_name', 'null_check', 0), ('bird_name', 'distinct_check', 4), ('observation_year', 'max', 2022), ('bird_happiness', 'min', 8), ('bird_happiness', 'max', 10)]
   [2023-07-04, 11:18:01 UTC] {sql.py:420} INFO - All tests have passed
   ```

## How it works

The SQL check operators abstract SQL queries to streamline data quality checks. One difference between the SQL check operators and the standard [`BaseSQLOperator`](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/stable/_api/airflow/providers/common/sql/operators/sql/index.html#airflow.providers.common.sql.operators.sql.BaseSQLOperator) is that the SQL check operators respond with a boolean, meaning the task fails when any of the resulting queries fail. This is particularly helpful for stopping a data pipeline before bad data makes it to a given destination. The lines of code and values that fail the check are observable in the Airflow logs.

The following SQL check operators are recommended for implementing data quality checks:

* **[`SQLColumnCheckOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLColumnCheckOperator)**: Runs one or more predefined data quality checks on one or more columns within the same task.
* **[`SQLTableCheckOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLTableCheckOperator)**: Runs multiple user-defined checks which can involve one or more columns of a table.
* **[`SQLCheckOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLCheckOperator)**: Takes any SQL query and returns a single row that is evaluated to booleans. This operator is useful for more complicated checks that could span several tables of your database.
* **[`SQLIntervalCheckOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLIntervalCheckOperator)**: Checks current data against historical data.

Astronomer recommends using the `SQLColumnCheckOperator` and `SQLTableCheckOperator` over the older operators ([`SQLValueCheckOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLValueCheckOperator) and [`SQLThresholdCheckOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLThresholdCheckOperator)) whenever possible to improve code readability.

### `SQLColumnCheckOperator`

The `SQLColumnCheckOperator` has a `column_mapping` parameter which stores a dictionary of checks. Using this dictionary, it can run many checks within one task and still provide observability in the Airflow logs over which checks passed and which failed.

This operator is useful for:

* Ensuring all numeric values in a column are above a minimum, below a maximum or within a certain range (with or without a tolerance threshold).
* Null checks.
* Checking primary key columns for uniqueness.
* Checking the number of distinct values of a column.

The `SQLColumnCheckOperator` offers 5 options for column checks which are abstractions over SQL statements:

* `"min"`: `"MIN(column) AS column_min"`
* `"max"`: `"MAX(column) AS column_max"`
* `"unique_check"`: `"COUNT(column) - COUNT(DISTINCT(column)) AS column_unique_check"`
* `"distinct_check"`: `"COUNT(DISTINCT(column)) AS column_distinct_check"`
* `"null_check"`: `"SUM(CASE WHEN column IS NULL THEN 1 ELSE 0 END) AS column_null_check"`

The resulting values can be compared to an expected value using any of the following qualifiers:

* `greater_than`
* `geq_to` (greater or equal than)
* `equal_to`
* `leq_to` (lesser or equal than)
* `less_than`

Additionally, the `SQLColumnCheckOperator`:

* Allows you to specify a tolerance to the comparisons as a fraction (0.1 = 10% tolerance), see the [`partition_clause`](#partition_clause) section for an example.
* Converts a returned `result` of `None` to 0 by default and still runs the check. For example, if a column check for the `MY_COL` column is set to accept a minimum value of -10 or more but runs on an empty table, the check would still pass because the `None` result is treated as 0. You can toggle this behavior by setting `accept_none=False`, which will cause all checks returning `None` to fail.
* Accepts an operator-level `partition_clause` parameter that allows you to run checks on a subset of your table. See the [`partition_clause`](#partition_clause) section for more information.

### `SQLTableCheckOperator`

The `SQLTableCheckOperator` provides a way to check the validity of user defined SQL statements which can involve one or more columns of a table. There is no limit to the amount of columns these statements can involve or to their complexity. The statements are provided to the operator as a dictionary with the `checks` parameter.

The `SQLTableCheckOperator` is useful for:

* Checks that include aggregate values using the whole table (for example, comparing the average of one column to the average of another using the SQL `AVG()` function).
* Row count checks.
* Checking if a date is between certain bounds (for example, using `MY_DATE_COL BETWEEN '2019-01-01' AND '2019-12-31'` to make sure only dates in the year 2019 exist).
* Comparisons between multiple columns, both aggregated and not aggregated.

Similarly to the `SQLColumnCheckOperator`, you can pass a SQL `WHERE`-clause (without the `WHERE` keyword) to the operator-level [`partition_clause`](#partition_clause) parameter or as a check-level `partition_clause`.

### `SQLCheckOperator`

The `SQLCheckOperator` returns a single row from a provided SQL query and checks to see if any of the returned values in that row are a value that [Python bool casting](https://docs.python.org/3/library/stdtypes.html) evaluates as `False`, for example `0`. If any values are `False`, the task fails. This operator lets you check:

* A specific, single column value.
* Part of or an entire row compared to a known set of values.
* Options for categorical variables and data types.
* Comparisons between multiple tables.
* The results of any other function that can be written as a SQL query.

The target table(s) for the `SQLCheckOperator` has to be specified within the SQL statement. The `sql` parameter of this operator can be either a complete SQL query as a string or, as in this tutorial, a reference to a query stored in a local file.

### `partition_clause`

With the `SQLColumnCheckOperator` and `SQLTableCheckOperator`, you can run checks on a subset of your table using either a check-level or task-level `partition_clause` parameter. This parameter takes a SQL `WHERE`-clause (without the `WHERE` keyword) and uses it to filter your table before running a given check or group of checks in a task.

The code snippet below shows a `SQLColumnCheckOperator` defined with a `partition_clause` at the operator level, as well as a `partition_clause` in one of the two column checks defined in the `column_mapping`.

In the following example, the operator checks whether:

* `MY_NUM_COL_1` has a minimum value of 10 with a tolerance of 10%, meaning that the check will pass if the minimum value in this column is between 9 and 11.
* `MY_NUM_COL_2` has a maximum value less than 300. Only rows that fulfill the check-level `partition_clause` are checked (rows where `CUSTOMER_STATUS = 'active'`).

Both of the above checks only run on rows that fulfill the operator-level partition clause `CUSTOMER_NAME IS NOT NULL`. If both an operator-level `partition_clause` and a check-level `partition_clause` are defined for a check, the check will only run on rows fulfilling both clauses.

```python wrap theme={null}
column_checks = SQLColumnCheckOperator(
    task_id="column_checks",
    conn_id="MY_DB_CONNECTION",
    table="MY_TABLE",
    partition_clause="CUSTOMER_NAME IS NOT NULL",
    column_mapping={
        "MY_NUM_COL_1": {"min": {"equal_to": 10, "tolerance": 0.1}},
        "MY_NUM_COL_2": {
            "max": {"less_than": 300, "partition_clause": "CUSTOMER_STATUS = 'active'"}
        },
    },
)
```


# Synchronous Dag execution
Source: https://astronomer.io/docs/learn/airflow-synchronous-dag-execution

Learn about synchronous Dag execution in Airflow.

Synchronous Dag execution refers to the ability in Airflow 3.1+ to trigger a Dag run using an API call and wait for it to complete before returning [XCom](/docs/learn/airflow-passing-data-between-tasks) values pushed by one or more tasks in the Dag run. This is useful both for single DAG runs and for cases where the same DAG may be triggered multiple times in parallel.

<Note>
  Synchronous Dag execution was added as an [experimental feature](https://airflow.apache.org/docs/apache-airflow/stable/release-process.html#experimental-features) in Airflow 3.1.
</Note>

## Assumed knowledge

* Basic knowledge of Airflow. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Knowing how to use the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html).
* Basic understanding of XCom. See [Passing data between tasks](/docs/learn/airflow-passing-data-between-tasks).

## When to use synchronous Dag execution

Synchronous Dag execution is a way to use Airflow as the backend for services processing user requests coming from a frontend application like a website, mobile app, or slack bot. Common use cases include:

* Inference execution: A user provides input to a pipeline that interacts with one or more LLMs and/or AI agents to generate a response. The response is served back to the user as soon as the Dag has completed running.
* Ad-hoc requests: Non-technical stakeholders request data analyses that use a Dag to retrieve the desired result.
* Data submission: Non-technical users can submit their data to a Dag to be processed and get immediate feedback on the status of the request and the result.

## API endpoint

The endpoint to wait for a Dag run to complete is:

```text wrap theme={null}
GET api/v2/dags/{dag_id}/dagRuns/{dag_run_id}/wait
```

It includes the following path parameters:

* `dag_id`: (Mandatory) The id of the DAG to wait for.
* `dag_run_id`: (Mandatory) The id of the DAG run to wait for.

The query parameters are:

* `interval`: (Mandatory) Seconds to wait between Dag run state checks.
* `result`: (Optional) Array of strings or null. A list of task ids from which to pull the XCom value pushed under the `return_value` key. In Airflow 3.3+ you specify which task in a Dag returns the result in the Dag code, see [specify the Dag result](#specify-the-dag-result).

Calling this endpoint on any running Dag will start a waiting process until the Dag run completes. If any task is specified as returning a result or any XCom are requested in the `result` parameter, they are returned in the response upon Dag run completion.

## Example script

The following script creates a Dag run for the `my_dag` Dag and waits for it to complete. It includes XComs pushed under the `return_value` key of the `my_task` task in the response.

```python expandable wrap theme={null}
import requests
from datetime import datetime
import json

_USERNAME = "admin"
_PASSWORD = "admin"
_HOST = "http://localhost:8080"  # To learn how to send API requests to Airflow running on Astro see: https://www.astronomer.io/docs/astro/airflow-api/

_DAG_ID = "my_dag"
_TASK_ID = "my_task"


def _get_jwt_token():
    token_url = f"{_HOST}/auth/token"
    payload = {"username": _USERNAME, "password": _PASSWORD}
    headers = {"Content-Type": "application/json"}
    response = requests.post(token_url, json=payload, headers=headers)

    token = response.json().get("access_token")
    return token


def _trigger_dag_run(dag_id: str):
    url = f"{_HOST}/api/v2/dags/{dag_id}/dagRuns"
    headers = {
        "Authorization": f"Bearer {_get_jwt_token()}",
        "Content-Type": "application/json",
    }
    payload = {
        "logical_date": None,
    }
    response = requests.post(url, headers=headers, json=payload)
    return response.json()["dag_run_id"]


def _wait_for_dag_run_completion(dag_id: str, dag_run_id: str):
    url = f"{_HOST}/api/v2/dags/{dag_id}/dagRuns/{dag_run_id}/wait"
    headers = {
        "Authorization": f"Bearer {_get_jwt_token()}",
    }
    params = {
        "interval": 1,
        "result": [_TASK_ID],  # In Airflow 3.3+ the result task can also be defined in the Dag code
    }
    response = requests.get(url, headers=headers, params=params)
    print(f"Status Code: {response.status_code}")

    lines = response.text.strip().split("\n")
    json_objects = []

    for line in lines:
        if line.strip():
            json_obj = json.loads(line)
            json_objects.append(json_obj)
            print(f"Status: {json_obj.get('state', 'unknown')}")

    if json_objects:
        last_status_update = json_objects[-1]
        xcom_results = last_status_update.get("results", {})
        print("Last status update: ", last_status_update)
        print("XCom results: ", xcom_results)
        return xcom_results


if __name__ == "__main__":
    _dag_run_id = _trigger_dag_run(_DAG_ID)
    _wait_for_dag_run_completion(_DAG_ID, _dag_run_id)
```

Running the script above returns an output similar to the following:

```text wrap theme={null}
Status Code: 200
Status: queued
Status: running
Status: running
Status: success
Last status update:  {'state': 'success', 'results': {'my_task': 'Hello World!'}}
XCom results:  {'my_task': 'Hello World!'}
```

## Specify the Dag result

In Airflow 3.3+ you can define which task in a Dag returns the result the `wait` endpoint should return, using the `@result` decorator on top of a `@task` decorated function.

```python {3} wrap theme={null}
from airflow.sdk import task, result

@result
@task
def my_task():
    return "hello!"

my_task()
```

When using traditional operators, you can add a task's `.output` (the XCom pushed with the key `return_value`) to the Dag object.

```python {12} wrap theme={null}
from airflow.sdk import DAG, chain
from airflow.providers.standard.operators.python import PythonOperator


def _my_task_func():
    return "hello!"


with DAG("my_dag") as dag:
    _my_task = PythonOperator(task_id="my_task", python_callable=_my_task_func)

    dag.add_result(_my_task.output)
```


# Task state store in Apache Airflow®
Source: https://astronomer.io/docs/learn/airflow-task-state-store

Learn how to save information persisting between task retries in Airflow.

The task state store, added in [Apache Airflow®](https://airflow.apache.org/) 3.3, allows you to save information that persists between task retries. This makes the task state store suitable for storing external job IDs and intra-task checkpoints.

In this guide, you'll learn:

* When to use the task state store, and how it compares to XCom, deferrable operators, and the asset state store.
* How to set, get, delete, and clear values in the task state store.
* How to view and edit entries in the Airflow UI and through the Airflow REST API.
* How to create a custom operator that uses the task state store through the `ResumableJobMixin`.
* How to configure the task state store.

<Note>
  The task state store holds information for a specific task instance. Airflow 3.3 also introduced the asset state store, which holds information attached to a specific [Airflow asset](/docs/learn/airflow-datasets). See the [asset state store section](/docs/learn/airflow-advanced-asset-scheduling#asset-state-store) of the advanced asset-based scheduling guide for more information.
</Note>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Airflow operators. See [Airflow operators](/docs/learn/what-is-an-operator).
* Airflow decorators. See [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).

## When to use the task state store

Use the task state store to persist information about a task instance's own working state so that a later attempt of the same task instance, for example after a worker crash or retry, can access it. You can:

* Store the ID of a long-running external process, such as a Spark job or a model fine-tuning job, so a retry reconnects to the running job instead of submitting a duplicate.
* Store an intra-task checkpoint, such as the last page or offset processed, so a retry doesn't restart from the beginning.
* Store several pieces of state for one task instance, such as a status and a row count, and update them as the task runs.

When using the default task state store backend, Airflow saves entries to the metadata database, and stored values must be JSON serializable (except `None`). See [task state store configuration](#task-state-store-configuration) for information on implementing a custom task state store backend.

### Task state store vs. XCom

Both the task state store and [XCom](/docs/learn/airflow-passing-data-between-tasks) store small pieces of data attached to an individual task instance. The main difference is that Airflow clears XComs on retries, while task state entries persist.

|                          | Task state store                                                           | XCom                                                             |
| ------------------------ | -------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| Purpose                  | Persist a task instance's working state across retries                     | Pass data from a task to downstream tasks                        |
| Retry behavior           | Preserved across retries                                                   | Cleared on retries                                               |
| Read by                  | The same task instance, on a later attempt                                 | Downstream tasks in the same or other Dag runs                   |
| Access from within tasks | `context["task_state_store"].set`, `.get`, `.delete`, and `.clear` methods | `context["ti"].xcom_push` and `context["ti"].xcom_pull`          |
| Retention                | Time-based, `NEVER_EXPIRE`, or clear-on-success                            | Tied to the task instance, removed when the task instance reruns |

### Task state store vs. deferrable operators

The task state store, together with the `ResumableJobMixin`, and [deferrable operators](/docs/learn/deferrable-operators) are both used in the context of long-running external jobs. Which to choose is often a question of personal preference. Deferrable operators free the worker slot while waiting and require a triggerer. The `ResumableJobMixin` keeps the worker slot but makes a synchronous operator crash-safe/durable by reconnecting to a running job.

|                                     | Task state store with `ResumableJobMixin`                  | Deferrable operators                        |
| ----------------------------------- | ---------------------------------------------------------- | ------------------------------------------- |
| Best for                            | Retry safety for synchronous operators without a triggerer | Very long running tasks                     |
| Frees the worker slot while waiting | No                                                         | Yes                                         |
| Requires a triggerer                | No                                                         | Yes                                         |
| Programming model                   | Synchronous `execute` method                               | Async trigger and `execute_complete` method |

### Task state store vs. asset state store

The task state store and the asset state store have a similar key-value interface, but they are scoped differently.

|                  | Task state store                           | Asset state store                                                   |
| ---------------- | ------------------------------------------ | ------------------------------------------------------------------- |
| Attached to      | A single task instance                     | An [Airflow asset](/docs/learn/airflow-datasets)                         |
| Accessed through | `context["task_state_store"]`              | `context["asset_state_store"]`                                      |
| Read by          | The same task instance, on a later attempt | Any task with the asset in its `inlets` or `outlets` parameter      |
| Retention        | Configurable (default: 30 days)            | No retention limit, cleared explicitly or when you delete the asset |

For more information on the asset state store, see the [asset state store section](/docs/learn/airflow-advanced-asset-scheduling#asset-state-store) of the advanced asset-based scheduling guide.

## How to use the task state store

You have several options to interact with the task state store:

* [In your task code](#task-state-store-examples) using the [methods](#task-state-store-methods) available in the `task_state_store` key of the [Airflow context](/docs/learn/airflow-context) in:
  * A `@task`-decorated function using the TaskFlow API.
  * A `python_callable` passed to a `PythonOperator`.
  * The `execute` method of a custom operator, created by subclassing `BaseOperator` or any [existing traditional operator](https://airflow.apache.org/registry/).
* [In the Airflow UI](#task-state-store-in-the-airflow-ui)
* [With the Airflow REST API](#task-state-store-api-endpoints)

<Note>
  For custom operators built to resume a process after a retry a mixin is available, see [ResumableJobMixin](#resumablejobmixin).
</Note>

### Task state store methods

The task state store exposes four methods: `set`, `get`, `delete`, and `clear`.

`set(key, value, *, retention=None)` writes or overwrites the value for a key, in this case the PID of a long-running remote shell command. The keyword-only `retention` argument controls when the entry is automatically deleted:

* A `timedelta`, to delete the entry after the given duration from the time of the write. Airflow computes the expiry timestamp on the worker before sending the value to the API server.
* `NEVER_EXPIRE`, to exclude the entry from the automatic cleanup process and keep it until it is manually deleted.
* `None`, the default, to fall back to the environment's `AIRFLOW__STATE_STORE__DEFAULT_RETENTION_DAYS` configuration (default: 30 days).

```python wrap theme={null}
# task_state_store = context["task_state_store"]
task_state_store.set("remote_pid", 239, retention=timedelta(days=28))
```

`get(key, default=None)` returns the stored value, or `default` if the key doesn't exist. For example, if a previous try of a task instance already started a long running shell command and saved the PID in the task state store, a later retry can `get` the PID in order to check if the command is still running, and if that is the case, wait on the existing command to finish instead of rerunning it from the start.

```python wrap theme={null}
remote_pid = task_state_store.get("remote_pid", default=None)
```

`delete(key)` deletes a single key. It does nothing if the key doesn't exist.

```python wrap theme={null}
task_state_store.delete("remote_pid")
```

`clear()` deletes all task state store keys for the task instance.

```python wrap theme={null}
task_state_store.clear()
```

### Task state store examples

The following examples start a long-running shell command on a remote host and store its process ID (PID) in the task state store with the `set` method. If the first try fails and the task is retried, it uses the `get` method to reconnect to the still-running command instead of launching a new one.

<details>
  <summary>TaskFlow API</summary>

  ```python {17,21} expandable wrap theme={null}
  from pendulum import duration
  from airflow.sdk import task
  from airflow.providers.ssh.hooks.ssh import SSHHook


  def _ssh_exec(command):
      hook = SSHHook(ssh_conn_id="ssh_default")
      with hook.get_conn() as client:
          _, stdout, _ = client.exec_command(command)
          output = stdout.read().decode().strip()
          exit_status = stdout.channel.recv_exit_status()
      return exit_status, output


  @task(retries=10, retry_delay=duration(minutes=10))
  def run_ssh_command(launch_cmd, check_cmd, out_file, task_state_store=None):
      remote_pid = task_state_store.get("remote_pid")

      if remote_pid is None:
          _, remote_pid = _ssh_exec(launch_cmd.replace("{out_file}", out_file))
          task_state_store.set("remote_pid", remote_pid)

      print(f"Reconnecting to remote PID {remote_pid} from a previous attempt.")
      exit_status, output = _ssh_exec(
          check_cmd.replace("{pid}", remote_pid).replace("{out_file}", out_file)
      )
      print(f"Remote job output: {output}")
      if exit_status != 0:
          raise RuntimeError(f"Remote job {remote_pid} failed with exit status {exit_status}.")


  run_ssh_command(
      launch_cmd=(
          "nohup sh -c 'sleep 60; rc=$?; echo finished > {out_file}; "
          "echo $rc > {out_file}.rc' >/dev/null 2>&1 & echo $!"
      ),
      check_cmd=(
          "while kill -0 {pid} 2>/dev/null; do sleep 2; done; "
          "rc=$(cat {out_file}.rc 2>/dev/null); cat {out_file}; "
          '[ "$rc" = "0" ]'
      ),
      out_file="/tmp/log_{{ run_id }}.out",
  )
  ```
</details>

<details>
  <summary>`PythonOperator`</summary>

  ```python {17,21} expandable wrap theme={null}
  from pendulum import duration

  from airflow.providers.standard.operators.python import PythonOperator
  from airflow.providers.ssh.hooks.ssh import SSHHook


  def _ssh_exec(command):
      hook = SSHHook(ssh_conn_id="ssh_default")
      with hook.get_conn() as client:
          _, stdout, _ = client.exec_command(command)
          output = stdout.read().decode().strip()
          exit_status = stdout.channel.recv_exit_status()
      return exit_status, output


  def _run_ssh_command_func(launch_cmd, check_cmd, out_file, task_state_store=None):
      remote_pid = task_state_store.get("remote_pid")

      if remote_pid is None:
          _, remote_pid = _ssh_exec(launch_cmd.replace("{out_file}", out_file))
          task_state_store.set("remote_pid", remote_pid)

      print(f"Reconnecting to remote PID {remote_pid} from a previous attempt.")
      exit_status, output = _ssh_exec(
          check_cmd.replace("{pid}", remote_pid).replace("{out_file}", out_file)
      )
      print(f"Remote job output: {output}")
      if exit_status != 0:
          raise RuntimeError(f"Remote job {remote_pid} failed with exit status {exit_status}.")


  PythonOperator(
      task_id="run_ssh_command",
      python_callable=_run_ssh_command_func,
      op_kwargs={
          "launch_cmd": (
              "nohup sh -c 'sleep 60; rc=$?; echo finished > {out_file}; "
              "echo $rc > {out_file}.rc' >/dev/null 2>&1 & echo $!"
          ),
          "check_cmd": (
              "while kill -0 {pid} 2>/dev/null; do sleep 2; done; "
              "rc=$(cat {out_file}.rc 2>/dev/null); cat {out_file}; "
              '[ "$rc" = "0" ]'
          ),
          "out_file": "/tmp/log_{{ run_id }}.out",
      },
      retries=10,
      retry_delay=duration(minutes=10),
  )
  ```
</details>

<details>
  <summary>Custom operator</summary>

  ```python {27,28,32} expandable wrap theme={null}
  from pendulum import duration

  from airflow.sdk.bases.operator import BaseOperator
  from airflow.providers.ssh.hooks.ssh import SSHHook


  def _ssh_exec(command):
      hook = SSHHook(ssh_conn_id="ssh_default")
      with hook.get_conn() as client:
          _, stdout, _ = client.exec_command(command)
          output = stdout.read().decode().strip()
          exit_status = stdout.channel.recv_exit_status()
      return exit_status, output


  class MyCustomOperator(BaseOperator):

      template_fields = ("launch_cmd", "check_cmd", "out_file")

      def __init__(self, *, launch_cmd, check_cmd, out_file, **kwargs):
          super().__init__(**kwargs)
          self.launch_cmd = launch_cmd
          self.check_cmd = check_cmd
          self.out_file = out_file

      def execute(self, context):
          tss = context["task_state_store"]
          remote_pid = tss.get("remote_pid")

          if remote_pid is None:
              _, remote_pid = _ssh_exec(self.launch_cmd.replace("{out_file}", self.out_file))
              tss.set("remote_pid", remote_pid)

          self.log.info(f"Reconnecting to remote PID {remote_pid} from a previous attempt.")
          exit_status, output = _ssh_exec(
              self.check_cmd.replace("{pid}", remote_pid).replace("{out_file}", self.out_file)
          )
          self.log.info(f"Remote job output: {output}")
          if exit_status != 0:
              raise RuntimeError(f"Remote job {remote_pid} failed with exit status {exit_status}.")


  MyCustomOperator(
      task_id="run_ssh_command",
      launch_cmd=(
          "nohup sh -c 'sleep 60; rc=$?; echo finished > {out_file}; "
          "echo $rc > {out_file}.rc' >/dev/null 2>&1 & echo $!"
      ),
      check_cmd=(
          "while kill -0 {pid} 2>/dev/null; do sleep 2; done; "
          "rc=$(cat {out_file}.rc 2>/dev/null); cat {out_file}; "
          '[ "$rc" = "0" ]'
      ),
      out_file="/tmp/log_{{ run_id }}.out",
      retries=10,
      retry_delay=duration(minutes=10),
  )
  ```
</details>

<details>
  <summary>Extended traditional operator</summary>

  ```python {22,23,28} expandable wrap theme={null}
  """
  Custom extension of the SSHOperator that stores the remote PID
  in the task state store and reconnects to it on a retry.
  """

  from airflow.providers.ssh.operators.ssh import SSHOperator
  from pendulum import duration
  import base64


  class MyCustomSSHOperator(SSHOperator):

      template_fields = ("launch_cmd", "check_cmd", "out_file", *SSHOperator.template_fields)

      def __init__(self, *, launch_cmd, check_cmd, out_file, **kwargs):
          super().__init__(**kwargs)
          self.launch_cmd = launch_cmd
          self.check_cmd = check_cmd
          self.out_file = out_file

      def execute(self, context):
          tss = context["task_state_store"]
          remote_pid = tss.get("remote_pid")

          if remote_pid is None:
              self.command = self.launch_cmd.replace("{out_file}", self.out_file)
              remote_pid = base64.b64decode(super().execute(context)).decode().strip()
              tss.set("remote_pid", remote_pid)

          self.log.info(f"Reconnecting to remote PID {remote_pid} from a previous attempt.")
          self.command = self.check_cmd.replace("{pid}", remote_pid).replace("{out_file}", self.out_file)
          return super().execute(context)


  MyCustomSSHOperator(
      task_id="run_ssh_command",
      ssh_conn_id="ssh_default",
      launch_cmd=(
          "nohup sh -c 'sleep 60; rc=$?; echo finished > {out_file}; "
          "echo $rc > {out_file}.rc' >/dev/null 2>&1 & echo $!"
      ),
      check_cmd=(
          "while kill -0 {pid} 2>/dev/null; do sleep 2; done; "
          "rc=$(cat {out_file}.rc 2>/dev/null); cat {out_file}; "
          '[ "$rc" = "0" ]'
      ),
      out_file="/tmp/log_{{ run_id }}.out",
      cmd_timeout=120,
      retries=10,
      retry_delay=duration(minutes=10),
  )
  ```
</details>

### Task state store in the Airflow UI

You can view and edit task state store entries for any task instance in the Airflow UI.

<Frame>
  <img alt="Airflow UI task instance view with the Storage tab and Task State Store sub-tab selected, showing a my_num entry with its key, value, updated time, and expiry, and controls to add, clear, edit, and delete entries" />
</Frame>

To reach the task state store of a specific task instance, select the task instance square in the grid (1), open the **Storage** tab (2), then select the **Task State Store** tab. From here, you can:

* Add an entry with **Add Task State Store** (3).
* Remove all entries for the task instance with **Clear All Task State Store** (4).
* Edit an entry (5).
* Delete an entry (6).

When you select **Add Task State Store** (3), the **Add Task State Store** window appears. Enter a **Key** and **Value** for the entry, then choose when it expires:

* **Default (30 days)**: Use the default retention period for this Airflow environment, set by the `default_retention_days` configuration.
* **Never**: The entry never expires.
* **Custom**: The entry expires at the date and time you select.

<Frame>
  <img alt="Add Task State Store window in the Airflow UI with Key and Value fields and Expiration options for Default 30 days, Never, and Custom" />
</Frame>

Select **Save** to store the entry.

### Task state store API endpoints

The [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) exposes task state store operations under each task instance's `state-store` path:

```text wrap theme={null}
/dags/{dag_id}/dagRuns/{dag_run_id}/taskInstances/{task_id}/state-store
```

| Operation                             | Method and path                |
| ------------------------------------- | ------------------------------ |
| List all entries for a task instance  | `GET .../state-store`          |
| Get one entry                         | `GET .../state-store/{key}`    |
| Create or overwrite an entry          | `PUT .../state-store/{key}`    |
| Update an entry                       | `PATCH .../state-store/{key}`  |
| Delete one entry                      | `DELETE .../state-store/{key}` |
| Clear all entries for a task instance | `DELETE .../state-store`       |

The REST API limits each individual value to `max_value_storage_bytes` (64 KB by default) and rejects a larger value with a `422` error. For the request and response schemas of each operation, see the [Airflow REST API reference](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html).

### `ResumableJobMixin`

For custom operators that submit one long-running external job and poll for its completion, you can use the [`ResumableJobMixin`](https://airflow.apache.org/docs/task-sdk/stable/resumable-job-mixin.html). After submitting the job, the mixin persists the external job ID to the task state store before polling starts. On a retry, the mixin reads the stored ID and reconnects to the running job instead of submitting a duplicate. The `SparkSubmitOperator` in the Apache Spark provider (`6.2.0+`) uses this mixin to reconnect to a running Spark application after a retry.

To use the mixin, inherit from `ResumableJobMixin`, call `execute_resumable(context)` from your `execute` method, and write the following six methods to determine how to interact with your external system:

* `submit_job(context)`: Submit the job and return its external ID. The mixin stores this value in the task state store.
* `get_job_status(external_id, context)`: Query the external system and return its raw status string.
* `is_job_active(status)`: Return `True` if the job is still running and can be reconnected to.
* `is_job_succeeded(status)`: Return `True` if the job completed successfully.
* `poll_until_complete(external_id, context)`: Block until the job reaches a terminal state, and raise on failure.
* `get_job_result(external_id, context)`: Return the job result after completion, or `None` if not applicable.

```python wrap theme={null}
from typing import Any

from airflow.sdk import BaseOperator, ResumableJobMixin


class MyCustomResumableJobOperator(ResumableJobMixin, BaseOperator):
    external_id_key = "my_job_id"

    def execute(self, context):
        return self.execute_resumable(context)

    def submit_job(self, context) -> str:
        return self.hook.submit(...)  # Return the external job ID.

    def get_job_status(self, external_id, context) -> str:
        return self.hook.get_status(external_id)

    def is_job_active(self, status: str) -> bool:
        return status in ("RUNNING", "PENDING")

    def is_job_succeeded(self, status: str) -> bool:
        return status == "SUCCEEDED"

    def poll_until_complete(self, external_id, context) -> None:
        self.hook.wait(external_id)

    def get_job_result(self, external_id, context) -> Any:
        return None
```

On a retry, the mixin reads the stored ID and checks the current job status:

* If the job is still active, the mixin reconnects and continues polling.
* If the job already succeeded, the mixin returns the result without resubmitting.
* If the job is in a terminal failure state, the mixin submits a fresh job.

The `external_id_key` class attribute sets the key used to store the job ID. The default is `remote_job_id`.

<Note>
  There is a small window between `submit_job` returning and the mixin persisting the ID to the task state store. If the worker crashes in that window, the retry doesn't have the ID and submits a fresh job. For most workloads this window is negligible.
</Note>

## Task state store configuration

You can configure the task state store with the `[state_store]` configs.

| Environment variable                             | Default           | Description                                                                                                                                                                                                                                     |
| ------------------------------------------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AIRFLOW__STATE_STORE__BACKEND`                  | Metastore backend | Dotted path to the class that implements state storage.                                                                                                                                                                                         |
| `AIRFLOW__STATE_STORE__DEFAULT_RETENTION_DAYS`   | `30`              | Number of days after which entries written without an explicit retention expire. Set to `0` to disable time-based cleanup. Changing this value doesn't affect entries that already exist.                                                       |
| `AIRFLOW__STATE_STORE__CLEAR_ON_SUCCESS`         | `False`           | When `True`, delete all task state store entries for a task instance when it completes successfully.                                                                                                                                            |
| `AIRFLOW__STATE_STORE__STATE_CLEANUP_BATCH_SIZE` | `0`               | Number of rows deleted per batch during cleanup. `0` deletes all matching rows in a single statement.                                                                                                                                           |
| `AIRFLOW__STATE_STORE__MAX_VALUE_STORAGE_BYTES`  | `65535`           | Maximum size in bytes of a single value written through the public REST API. The API rejects a larger value with a `422` error. The execution API doesn't block worker writes; they log a warning and proceed. Set to `0` to disable the limit. |

<Note>
  The `clear_on_success` and `default_retention_days` options apply to the task state store only. They don't affect the [asset state store](/docs/learn/airflow-advanced-asset-scheduling#asset-state-store).
</Note>

By default, the task state store and asset state store persist entries in the Airflow metadata database. To store entries elsewhere, you can provide a custom backend. See the [Apache Airflow task and asset state store configuration documentation](http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/administration-and-deployment/task-and-asset-state-store.html) for more information.


# Apache Airflow® trigger rules
Source: https://astronomer.io/docs/learn/airflow-trigger-rules

Learn about available trigger rules and how to use them.

Trigger rules are used to determine when a task should run in relation to the previous task. By default, [Apache Airflow®](https://airflow.apache.org/) runs a task when all directly upstream tasks are successful. However, you can change this behavior using the `trigger_rule` parameter in the task definition.

<Info>
  Trigger rules define whether a task runs based on its direct upstream dependencies. To learn how to set task dependencies, see the [Manage task and task group dependencies in Airflow](/docs/learn/managing-dependencies) guide.
</Info>

## Define a trigger rule

You can override the default trigger rule by setting the `trigger_rule` parameter in the task definition.

```python {7} wrap theme={null}
# from airflow.sdk import chain, task

@task 
def upstream_task():
    return "Hello..."

@task(trigger_rule="all_success")
def downstream_task():
    return " World!"

chain(upstream_task(), downstream_task())
```

```python {7} wrap theme={null}
# from airflow.providers.standard.operators.empty import EmptyOperator
# from airflow.sdk import chain

upstream_task = EmptyOperator(task_id="upstream_task")
downstream_task = EmptyOperator(
    task_id="downstream_task",
    trigger_rule="all_success"
)
chain(upstream_task, downstream_task)
```

## Trigger rules in Airflow

The following trigger rules are available:

* `all_success`: (default) The task runs only when all upstream tasks have succeeded.
* `all_done`: The task runs once all upstream tasks are done with their execution.
* `all_done_min_one_success`: The task runs once all upstream tasks are done with their execution and at least one upstream task has succeeded. Note that `skipped` doesn't count as done for this rule and will cause the downstream task to be skipped as well.
* `all_failed`: The task runs only when all upstream tasks are in a `failed` or `upstream_failed` state.
* `all_skipped`: The task runs only when all upstream tasks have been skipped.
* `always`: The task runs as soon as the Dag run starts, independently of the status of any upstream tasks.
* `none_failed`: The task runs only when all upstream tasks have succeeded or been skipped.
* `none_failed_min_one_success`: The task runs only when all upstream tasks aren't in the state `failed` or `upstream_failed`, and at least one upstream task has succeeded.
* `none_skipped`: The task runs only when no upstream task is in a `skipped` state.
* `one_done`: The task runs as soon as at least one upstream task has either succeeded or failed.
* `one_failed`: The task runs as soon as at least one upstream task is in `failed` or `upstream_failed` state.
* `one_success`: The task runs as soon as at least one upstream task has succeeded.
* `all_done_setup_success`: Special trigger rule used in teardown tasks. The task runs when all upstream tasks have finished and at least one directly connected setup task has been successful. See the [Use setup and teardown tasks in Airflow](/docs/learn/airflow-setup-teardown) guide for more information.

<Info>
  You can define a Dag in which any task failure stops the Dag execution by setting the [Dag parameter](/docs/learn/airflow-dag-parameters) `fail_fast` to `True`. This will set all tasks that are still running to `failed` and mark any tasks that haven't run yet as `skipped`, as soon as any task in the Dag fails. Note that you can't have any trigger rule other than `all_success` (and `all_done_setup_success`) in a Dag with `fail_fast` set to `True` and that [teardown](/docs/learn/airflow-setup-teardown) tasks are exempt and will still run.
</Info>

## Branching and trigger rules

One common scenario where you might need to implement trigger rules is if your Dag contains conditional logic such as [branching](/docs/learn/airflow-branch-operator). In these cases, for the task directly after the branches, `none_failed_min_one_success` or `none_failed` are likely more helpful than `all_success`, because unless all branches are run, at least one upstream task will always be in a `skipped` state.

In the following example Dag there is a simple branch with a downstream task that needs to run if either of the branches are followed. With the `all_success` rule, the `end` task never runs because all but one of the `branch` tasks is always ignored and therefore doesn't have a success state. If you change the trigger rule to `none_failed_min_one_success`, then the `end` task can run so long as one at least one of the branches has succeeded and none of the branches have failed.

```python {10} wrap theme={null}
import random
from airflow.sdk import dag, task
from airflow.providers.standard.operators.empty import EmptyOperator


@dag
def branching_dag():
    # EmptyOperators to start and end the DAG
    start = EmptyOperator(task_id="start")
    end = EmptyOperator(task_id="end", trigger_rule="none_failed_min_one_success")

    # Branching task
    @task.branch
    def branching(**kwargs):
        branches = ["branch_0", "branch_1", "branch_2"]
        return random.choice(branches)

    branching_task = branching()

    start >> branching_task

    # set dependencies
    for i in range(0, 3):
        d = EmptyOperator(task_id="branch_{0}".format(i))
        branching_task >> d >> end


branching_dag()
```

This image shows the resulting Dag:

<Frame>
  <img alt="Branch Dependencies" />
</Frame>

<details>
  <summary>Traditional Syntax</summary>

  ```python {15} wrap theme={null}
  import random
  from airflow.sdk import DAG
  from airflow.providers.standard.operators.empty import EmptyOperator
  from airflow.providers.standard.operators.python import BranchPythonOperator


  def return_branch(**kwargs):
      branches = ["branch_0", "branch_1", "branch_2"]
      return random.choice(branches)


  with DAG(dag_id="branching_dag"):
      # EmptyOperators to start and end the DAG
      start = EmptyOperator(task_id="start")
      end = EmptyOperator(task_id="end", trigger_rule="none_failed_min_one_success")

      # Branching task
      branching = BranchPythonOperator(task_id="branching", python_callable=return_branch)

      start >> branching

      # set dependencies
      for i in range(0, 3):
          d = EmptyOperator(task_id="branch_{0}".format(i))
          branching >> d >> end
  ```
</details>

## Airflow trigger rules in detail

### `all_success`

This is the default trigger rule. A task with the trigger rule `all_success` only runs when all upstream tasks have succeeded.

<Frame>
  <img alt="Graph view of a Dag where all four upstream tasks succeeded and the downstream authorize_flight task ran with the all_success trigger rule" />
</Frame>

As soon as any upstream tasks are in the state of `failed`, `upstream_failed`, the downstream task is set to the state `upstream_failed` and doesn't run.

<Frame>
  <img alt="Graph view of a Dag where the clear_paris task failed, setting the downstream authorize_flight task to upstream_failed under the all_success trigger rule" />
</Frame>

Similarly, as soon as any upstream task is in the state `skipped`, the downstream task is set to the state `skipped` and doesn't run.

<Frame>
  <img alt="Graph view of a Dag where the clear_solo task was skipped, setting the downstream authorize_flight task to skipped under the all_success trigger rule" />
</Frame>

If a task with the trigger rule `all_success` has one upstream task that is `skipped` and one that is `failed` / `upstream_failed`, whether the downstream task is set to `skipped` or `upstream_failed` depends on which of the upstream tasks finishes first. If the first upstream task that isn't successful ends with the state `skipped` the downstream task is skipped, if it ends in `failed` or `upstream_failed` the downstream task is set to `upstream_failed`. If both upstream tasks, `skipped` and `failed`/`upstream_failed` finish in the same scheduler evaluation period, the downstream task will be set to `upstream_failed`.

### `all_done`

The `all_done` trigger rule will make a task wait until all upstream tasks are done with their execution.

<Frame>
  <img alt="Graph view of a Dag where the trajectory_mercury task is still running, so the downstream log_trajectory task has no status yet and waits under the all_done trigger rule" />
</Frame>

As soon as all tasks finish, no matter what their state is, the downstream task will run.

<Frame>
  <img alt="Graph view of a Dag where the upstream tasks finished in failed, upstream_failed, and skipped states and the downstream log_trajectory task still ran under the all_done trigger rule" />
</Frame>

### `all_failed`

The `all_failed` trigger rule will make a task wait until all upstream tasks are in a `failed` or `upstream_failed` state.

<Frame>
  <img alt="Graph view of a Dag where the check_vesta_orbit task is still running, so the downstream end_mission task has no status yet and waits under the all_failed trigger rule" />
</Frame>

<Frame>
  <img alt="Graph view of a Dag where all upstream tasks are in failed or upstream_failed states and the downstream end_mission task ran under the all_failed trigger rule" />
</Frame>

As soon as any upstream task is in the state `success` or `skipped`, the downstream task is set to the state `skipped` and doesn't run.

<Frame>
  <img alt="Graph view of a Dag where the check_pallas_orbit task succeeded, setting the downstream end_mission task to skipped under the all_failed trigger rule" />
</Frame>

<Frame>
  <img alt="Graph view of a Dag where the check_pallas_orbit task was skipped, setting the downstream end_mission task to skipped under the all_failed trigger rule" />
</Frame>

### `all_skipped`

A task with the trigger rule `all_skipped` waits for all its upstream tasks to be `skipped`.

<Frame>
  <img alt="Graph view of a Dag where three survey tasks are skipped and survey_ganymede is still running, so the downstream return_to_orbit task has no status yet and waits under the all_skipped trigger rule" />
</Frame>

<Frame>
  <img alt="Graph view of a Dag where all four survey tasks are skipped and the downstream return_to_orbit task ran under the all_skipped trigger rule" />
</Frame>

As soon as any upstream task is in the state `success`, `failed`, or `upstream_failed`, the downstream task with the trigger rule `all_skipped` is set to the state `skipped` and doesn't run.

<Frame>
  <img alt="Graph view of a Dag where the survey_io task succeeded, setting the downstream return_to_orbit task to skipped under the all_skipped trigger rule" />
</Frame>

### `all_done_min_one_success`

Tasks using the `all_done_min_one_success` trigger rule run only when three conditions are met:

1. All upstream tasks are in either `success`, `failed` or `upstream_failed` state.
2. At least one upstream task is in the success state.
3. No upstream task is in the `skipped` state.

<Frame>
  <img alt="Graph view of a Dag where ping_delta_relay and ping_ds9 failed, ping_voyager is upstream_failed, ping_enterprise succeeded, and ping_farragut is still running, so the downstream generate_report task has no status yet and waits under the all_done_min_one_success trigger rule" />
</Frame>

<Frame>
  <img alt="Graph view of a Dag where all upstream tasks finished with ping_enterprise succeeded and the others failed or upstream_failed, so the downstream generate_report task ran under the all_done_min_one_success trigger rule" />
</Frame>

If all upstream tasks finish in either `failed` or `upstream_failed` state, the task using the `all_done_min_one_success` trigger rule is set to `upstream_failed`.

<Frame>
  <img alt="Graph view of a Dag where all upstream tasks finished in failed or upstream_failed states, setting the downstream generate_report task to upstream_failed under the all_done_min_one_success trigger rule" />
</Frame>

As soon as any upstream task is in a `skipped` state the task using the `all_done_min_one_success` trigger rule is `skipped` as well.

<Frame>
  <img alt="Graph view of a Dag where the ping_enterprise task was skipped, setting the downstream generate_report task to skipped under the all_done_min_one_success trigger rule" />
</Frame>

### always

A task with the trigger rule `always` runs as soon as the Dag run is started, regardless of the state of its upstream tasks.

<Frame>
  <img alt="Graph view of a Dag where the downstream send_telemetry task already succeeded while all its upstream tasks are still running or have no status yet under the always trigger rule" />
</Frame>

### `none_failed`

The `none_failed` trigger rule makes a task run only when all upstream tasks have either succeeded or been skipped.

<Frame>
  <img alt="Graph view of a Dag where calibrate_phobos and calibrate_io are skipped and calibrate_deimos and calibrate_luna succeeded, so the downstream initialize_navigation task ran under the none_failed trigger rule" />
</Frame>

As soon as any upstream task is in the state `failed` or `upstream_failed`, the downstream task is set to the state `upstream_failed` and doesn't run.

<Frame>
  <img alt="Graph view of a Dag where the calibrate_luna task failed, setting the downstream initialize_navigation task to upstream_failed under the none_failed trigger rule" />
</Frame>

### `none_failed_min_one_success`

Tasks using the `none_failed_min_one_success` trigger rule run only when three conditions are met:

1. All upstream tasks are finished.
2. No upstream tasks are in the `failed` or `upstream_failed` state.
3. At least one upstream task is in the `success` state.

<Frame>
  <img alt="Graph view of a Dag where invite_stevens is still running, so the downstream launch_mission task has no status yet and waits under the none_failed_min_one_success trigger rule" />
</Frame>

<Frame>
  <img alt="Graph view of a Dag where ping and invite_baldwin succeeded and the remaining invite tasks are skipped, so the downstream launch_mission task ran under the none_failed_min_one_success trigger rule" />
</Frame>

If any upstream task is in the `failed` or `upstream_failed` state, the downstream task is set to the state `upstream_failed` and doesn't run.

<Frame>
  <img alt="Graph view of a Dag where the ping task failed, setting invite_baldwin and the downstream launch_mission task to upstream_failed under the none_failed_min_one_success trigger rule" />
</Frame>

If all upstream tasks are in the `skipped` state, the downstream task is set to the state `skipped` and doesn't run.

<Frame>
  <img alt="Graph view of a Dag where all upstream tasks are skipped, setting the downstream launch_mission task to skipped under the none_failed_min_one_success trigger rule" />
</Frame>

### `none_skipped`

Tasks using the `none_skipped` trigger rule run only when no upstream task is in the `skipped` state. Upstream tasks can be in any other state: `success`, `failed`, or `upstream_failed`.

<Frame>
  <img alt="Graph view of a Dag where verify_hull is still running, so the downstream authorize_launch task has no status yet and waits under the none_skipped trigger rule" />
</Frame>

<Frame>
  <img alt="Graph view of a Dag where the verify tasks finished in success and failed states with none skipped, so the downstream authorize_launch task ran under the none_skipped trigger rule" />
</Frame>

If any upstream task is in the `skipped` state, the downstream task is set to the state `skipped` and doesn't run.

<Frame>
  <img alt="Graph view of a Dag where the verify_life_support task was skipped, setting the downstream authorize_launch task to skipped under the none_skipped trigger rule" />
</Frame>

### `one_done`

The `one_done` trigger rule makes a task run as soon as at least one of its upstream tasks is in either the `success` or `failed` state. Upstream tasks with `skipped` or `upstream_failed` states aren't considered "done."

<Frame>
  <img alt="Graph view of a Dag where contact_gamma_q and contact_alpha_q are running, contact_beta_q is skipped, and contact_delta_q is upstream_failed, so no upstream task is success or failed yet and the downstream record_uplink task waits under the one_done trigger rule" />
</Frame>

Once one upstream task finishes (either in the `success` or `failed` state), the downstream task runs.

<Frame>
  <img alt="Graph view of a Dag where contact_gamma_q succeeded while the other upstream tasks are skipped or upstream_failed, so the downstream record_uplink task ran under the one_done trigger rule" />
</Frame>

<Frame>
  <img alt="Graph view of a Dag where contact_gamma_q failed while the other upstream tasks are skipped or upstream_failed, so the downstream record_uplink task ran under the one_done trigger rule" />
</Frame>

If all upstream tasks are either in `skipped` or `upstream_failed` states, the downstream task with the `one_done` trigger rule is set to the state `skipped`.

<Frame>
  <img alt="Graph view of a Dag where all upstream tasks are skipped, setting the downstream record_uplink task to skipped under the one_done trigger rule" />
</Frame>

### `one_failed`

The `one_failed` trigger rule will make a task run as soon as at least one of its upstream tasks is in either the `failed` or `upstream_failed` state.

<Frame>
  <img alt="Graph view of a Dag where diagnose_razorback failed while the other upstream tasks are still running, so the downstream alert_engineering task ran under the one_failed trigger rule" />
</Frame>

<Frame>
  <img alt="Graph view of a Dag where ping failed, setting diagnose_donnager to upstream_failed, so the downstream alert_engineering task ran under the one_failed trigger rule" />
</Frame>

If all upstream tasks have completed and none of them are in the `failed` or `upstream_failed` state, the downstream task will be set to the state `skipped`.

<Frame>
  <img alt="Graph view of a Dag where all upstream tasks succeeded with none failed or upstream_failed, setting the downstream alert_engineering task to skipped under the one_failed trigger rule" />
</Frame>

### `one_success`

The `one_success` trigger rule will make a task run as soon as at least one of its upstream tasks is in the `success` state.

<Frame>
  <img alt="Graph view of a Dag where ping_sojourner succeeded while the other upstream tasks are still running, so the downstream launch_supplies task ran under the one_success trigger rule" />
</Frame>

If all upstream tasks have been `skipped`, the downstream task with the `one_success` trigger rule is set to the state `skipped` as well.

<Frame>
  <img alt="Graph view of a Dag where all upstream tasks are skipped, setting the downstream launch_supplies task to skipped under the one_success trigger rule" />
</Frame>

If all upstream tasks have completed and at least one of them is in the `failed` or `upstream_failed` state, the downstream task will be set to the state `upstream_failed`.

<Frame>
  <img alt="Graph view of a Dag where the upstream tasks finished with failures and a skip but no success, setting the downstream launch_supplies task to upstream_failed under the one_success trigger rule" />
</Frame>


# An introduction to the Airflow UI
Source: https://astronomer.io/docs/learn/airflow-ui

Explore the Airflow UI, which helps you monitor and troubleshoot your data pipelines. Learn about some of its key features and visualizations.

The Apache Airflow [user interface (UI)](https://airflow.apache.org/docs/apache-airflow/stable/ui.html) is the web-based hub for monitoring, managing, and troubleshooting your data pipelines served by the [API server](/docs/learn/airflow-components). With a significant redesign in Airflow 3 focused on improving the developer experience, the UI is now React-based and comes with a more intuitive [plugin](/docs/learn/using-airflow-plugins) interface to add custom functionality.

The UI not only provides deep insights into your Dags and Dag runs but also allows you to manage core Airflow elements like [variables](/docs/learn/airflow-variables), [connections](/docs/learn/connections), and [pools](/docs/learn/airflow-pools). You can interact directly with your pipelines, for example, to run your Dags, [backfill](/docs/learn/rerunning-dags#backfill) them, clear task instances, or generate [asset events](/docs/learn/airflow-datasets). The UI is also where you can view previous [versions](/docs/learn/airflow-dag-versioning) of your Dags.

This guide provides an overview of the most useful features and visualizations in the Airflow UI. To follow along with the examples, you can get a local Airflow environment running in minutes using the [Astro CLI](/docs/cli/v1.43/get-started-cli).

<Info>This guide is based on the Airflow 3.3 UI. If you are on a different version, some elements may appear different. Astronomer recommends upgrading your Airflow environment frequently to take advantage of the latest features and improvements.</Info>

<Tip>
  You can customize and extend the Airflow UI by using [Airflow plugins](/docs/learn/using-airflow-plugins). In Airflow 3.2+ you can change the colors and CSS styling of the UI with the [AIRFLOW\_\_API\_\_THEME](http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/configurations-ref.html#theme) configuration.
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Airflow Dags. See [Introduction to Airflow Dags](/docs/learn/dags).

## Importance

While you can write, schedule, and run Dags without ever touching the Airflow UI, it is an indispensable tool for Dag development, monitoring, and operations. The UI serves as the primary window into your Airflow environment, providing the visibility and control necessary to manage complex data workflows effectively.

<Frame>
  <img alt="Screenshot of the Airflow UI Home view in light and dark mode." />
</Frame>

<Tip>Switch between light and dark mode under **User** → **Appearance**.</Tip>

The Airflow UI is the core feature for the following scenarios:

<CardGroup>
  <Card title="Observability" icon="chart-simple">
    Get a real-time status overview of your Dags and tasks. Use the grid, graph, and Gantt views to find performance bottlenecks and access task logs to troubleshoot failures. When using [assets](/docs/learn/airflow-datasets) to schedule your Dags, the Airflow UI also visualizes data-driven dependencies between assets and tasks.
  </Card>

  <Card title="Operational Management" icon="gear">
    Perform daily operations like pausing Dags, clearing tasks, [backfilling](/docs/learn/rerunning-dags#backfill), or managing Airflow connections. The UI makes pipeline management accessible to less technical data practitioners.
  </Card>

  <Card title="Debugging" icon="circle-play">
    Visually confirm your Dag's structure and dependencies as you build. Inspect rendered templates and XComs to debug issues, speeding up the development process.
  </Card>
</CardGroup>

<Note>
  For a video-based introduction to the Airflow UI, see the [Airflow: UI](https://academy.astronomer.io/path/airflow-101/airflow-ui) module in the Astronomer Academy.
</Note>

## Views

The Airflow UI is organized into several main views, accessible from the navigation bar. These views provide dedicated spaces for monitoring Dags and Dag runs, observing assets, browsing Airflow objects, and managing your environment.

While the UI is designed to be intuitive, many views contain powerful features that you may not know about. The following sections provide a walkthrough of each primary view and highlight specific features and advanced concepts to help you get the most out of the Airflow UI.

The most important views are:

* **[Home/Dashboard](#home/dashboard)**: A page to give you an overview of your Airflow instance and quick links to different views.
* **[Dags](#dags)**: A list of all Dags in your instance. From here you can access individual Dags and explore their tasks and runs.
* **[Assets](#assets)**: A list of all [assets](/docs/learn/airflow-datasets) in your instance. From here you can access asset graphs that show data-driven dependencies between assets and Dags.
* **[Browse](#browse)**: Access to lists of audit logs, XComs, deadlines, and required actions for human-in-the-loop operators.
* **[Admin](#admin)**: Management access to elements such as Airflow connections, variables, or pools.

## Home/Dashboard

The **Home** view is your landing page in Airflow and serves as a high-level dashboard for the entire environment. It provides a great overview of your pipelines' status and system health, allowing you to assess the state of your Airflow instance at a glance.

<Frame>
  <img alt="Screenshot of the Airflow UI Home view showing the dashboard." />
</Frame>

The Home view is composed of several key widgets:

* **Stats**: A quick summary of your environment's workload, with counts of active Dags and running or failed task instances. Each statistic is a clickable link, taking you to the pre-filtered Dags view.
* **Favorites**: A list of your favorite Dags. You can add a Dag to your favorites by clicking the star icon next to its name in the Dags view or within the individual Dag view.
* **Health**: See the overall health of your [Airflow components](/docs/learn/airflow-components).
* **Pool Slots**: Shows the accumulated number of slots of all configured [pools](/docs/learn/airflow-pools) and their utilization.
* **History**: See summary statistics of recent Dag runs and task instances at a glance.
* **Asset Events**: Shows a list of newest or oldest asset events.

## Dags

The **Dags** view is your central control panel for all data pipelines in your Airflow environment. It allows you to quickly assess the status of your Dags, monitor their recent performance, and perform key operational actions.

<Frame>
  <img alt="Screenshot of the Airflow UI Dags view showing several Dags with their run history and current run status." />
</Frame>

This view provides a searchable and filterable list of all your Dags. It shows you an overview of key metadata, including:

* **Schedule**: Whether a Dag runs on a cron expression, a timetable, or if it's triggered by updates to specific [assets](/docs/learn/airflow-datasets).
* **Next Run**: When the Dag is scheduled to run next.
* **Latest Run**: The status and date of the most recent Dag run.
* **Tags**: Any tags applied to the Dag, which can be used for filtering and organization.
* **Actions**: Quick access to pause or unpause a Dag, trigger a Dag run, star a Dag as a favorite to display it in the Home view, or delete it.

The **card view** also includes a visual history of recent Dag runs as a series of vertical bars. The height of each bar corresponds to the run's duration, while its color indicates the Dag's state, allowing you to spot anomalies quickly.

For a more condensed list of Dags, you can switch to the **list view**, which presents the information in a compact table ideal for environments with many Dags. You can toggle between the views directly below the total number of Dags.

<Tip>
  Use the keyboard shortcut <kbd>⌘</kbd>+<kbd>K</kbd> (or <kbd>Ctrl</kbd>+<kbd>K</kbd> on Windows/Linux) to jump to the advanced search bar. This allows you to filter and navigate through hundreds of Dags instantly without waiting for page reloads.
</Tip>

### Individual Dag

The individual Dag view gives you detailed insights into a specific Dag, including its Dag runs, task instances, and required actions (*when using the human-in-the-loop feature*). You can also trigger a backfill, or reparse the Dag.

<Frame>
  <img alt="Screenshot of the individual Dag view." />
</Frame>

### Actions

In the top-right corner of the individual Dag view, a set of action buttons provides direct control over the Dag's lifecycle. In addition to responding to open required actions, favoriting, reparsing, or deleting the Dag, the **Trigger** button is your primary tool for initiating runs outside the regular schedule.

<Frame>
  <img alt="Screenshot of the individual Dag actions." />
</Frame>

Clicking **Trigger** opens a dialog where you can start a single Dag run or a [backfill](/docs/learn/rerunning-dags#backfill) operation. For both types of runs, you can supply custom configuration in the **Advanced options** area. For a [backfill](/docs/learn/rerunning-dags#backfill), you define a date range and select one of three reprocessing behaviors.

<Frame>
  <img alt="Screenshot of the backfill dialog." />
</Frame>

You can also control whether the backfill operation runs backwards and how many active runs to apply. While a backfill is in progress, a status bar appears at the top of the Dag view to visualize its progress. Individual runs created by a backfill are also marked with a dedicated icon in the grid, distinguishing them from regularly scheduled or manually triggered runs.

<Frame>
  <img alt="Screenshot of a running backfill operation." />
</Frame>

#### Required Actions

The **Required Actions** button is central to managing interactive workflows that use Airflow's human-in-the-loop capabilities. This button is only visible for Dags with pending required actions.

<Frame>
  <img alt="The individual Dag run view with an arrow highlighting the Required Actions button." />
</Frame>

It displays a list of all task instances that are currently awaiting an action or have required one in the past. Select any pending required action to view its details and respond to it.

<Frame>
  <img alt="Screenshot of the Required Actions list." />
</Frame>

### Dag views

The two main visualizations of a Dag, displayed on the left side of the individual Dag view, are the grid view and the graph view.

In the screenshot below, on the left you can see a grid representation of the Dag's previous runs, including their duration and the outcome of all individual task instances. Each column represents a Dag run, and each square represents a task instance in that Dag run. Task instances are color-coded according to their status. A small play icon on a Dag run indicates that a run was triggered manually, a backwards turning arrow indicates a [backfilled](/docs/learn/rerunning-dags#backfill) Dag run, and a small asset icon shows that a run was triggered by an [asset update](https://astronomer.io/guides/airflow-datasets). If no icon is shown, the Dag ran according to its schedule.

<Frame>
  <img alt="Screenshot of the individual Dag grid view." />
</Frame>

On the right side of the grid/graph view, you can see the different tabs you can select to access more information about the Dag, Dag run, task, or task instance you selected in the grid or graph view.

After clicking the upper bar of an individual run in the grid representation, you have the option to enable the Gantt chart by clicking on the **Show Gantt** button.

<Frame>
  <img alt="The individual Dag view with the Gantt chart enabled, showing task durations across a timeline for a selected Dag run." />
</Frame>

<Tip>
  The **Gantt chart** provides a timeline view of a single Dag run, showing when each task started, how long it ran, and where tasks ran in parallel. This visualization is excellent for identifying bottlenecks in your pipeline. Look for long bars, as these represent the longest-running tasks. Optimizing these tasks can significantly reduce the overall duration of your Dag run.
</Tip>

Clicking a task instance box in the grid takes you directly to the task instance's logs for quick debugging.

<Frame>
  <img alt="Screenshot of the individual Dag view with a task instance selected and the logs tab open." />
</Frame>

In the top-left corner, you can switch between the grid and graph views. The graph view shows your Dag, its tasks, and their dependencies. You can open the graph view from the individual Dag page to see the graph independent of Dag runs. Alternatively, open it for a specific Dag run to also visualize the task states.

<Frame>
  <img alt="Screenshot highlighting the toggle between graph and grid views." />
</Frame>

<Tip>
  Press <kbd>g</kbd> to quickly toggle between the graph and grid views.
</Tip>

Use the gear icon in the graph view to select a specific Dag version. This lets you review your Dag's history and see that version's tasks and dependencies.

<Frame>
  <img alt="Screenshot highlighting the Dag version switch in the graph view." />
</Frame>

### Tabs

There are several tabs available within the individual **Dag** view:

<CardGroup>
  <Card title="Overview" icon="house-chimney" href="#overview">
    Get a summary of the most recent Dag runs and any related asset events.
  </Card>

  <Card title="Runs" icon="person-running-fast" href="#runs">
    View a detailed list of all historical and active Dag runs with their status and duration.
  </Card>

  <Card title="Tasks" icon="list-check" href="#tasks">
    Inspect detailed information about each task within the Dag and its dependencies.
  </Card>

  <Card title="Calendar" icon="calendar-days" href="#calendar">
    See past run history and future scheduled runs in a monthly calendar grid.
  </Card>

  <Card title="Required Actions" icon="hand-pointer" href="#required-actions">
    View and respond to any tasks that are currently paused and waiting for human input.
  </Card>

  <Card title="Backfills" icon="clock-rotate-left" href="#backfills">
    Review detailed information and logs for any backfill jobs that have been executed.
  </Card>

  <Card title="Audit Log" icon="file-waveform" href="#audit-log">
    See a complete log of all events related to the Dag, including run triggers and task clears.
  </Card>

  <Card title="Code" icon="code" href="#code">
    View the Python source code that defines the current version of your Dag.
  </Card>

  <Card title="Details" icon="circle-info" href="#details">
    See all metadata about the Dag, including its owner, tags, and parameters.
  </Card>
</CardGroup>

### Overview

The **Overview** tab shows a basic summary of the Dag, including recent runs and related asset events. You can filter this view by different time ranges.

<Frame>
  <img alt="Screenshot of the Overview tab." />
</Frame>

### Runs

The **Runs** tab shows detailed information about Dag runs, including the run after date, state, run type, triggering user, start and end dates, duration, Dag version, and additional configuration values. You can open filters for the list by clicking on **+ Filter**.

Click a run to open the individual [**Dag run**](#dag-run) view.

<Frame>
  <img alt="Screenshot of the Runs tab." />
</Frame>

### Tasks

The **Tasks** tab details information on all of the tasks in the Dag, including the task names, the operator type, the trigger rule, number of retries, and whether a task is [dynamically mapped](/docs/learn/dynamic-tasks) or not. You can click the individual task names to get back to the task overview page.

<Frame>
  <img alt="Screenshot of the Tasks tab." />
</Frame>

### Calendar

The **Calendar** tab provides a long-term perspective on your Dag's execution history. It presents a grid that visualizes run activity, which can be toggled between an hourly view for a given month and a daily view for an entire year.

Each colored square on the grid represents a time slot where one or more Dag runs occurred. The intensity of the color corresponds to the number of runs within that hour or day, making it easy to spot periods of high activity.

<Frame>
  <img alt="Screenshot of the Calendar tab." />
</Frame>

### Backfills

Under the **Backfills** tab, you can see detailed information about the backfills that were run on the Dag. This includes information on when it was run from/to, the reprocessing behavior, when it was created and completed, the duration, and the max active runs.

<Frame>
  <img alt="Screenshot of the Backfills tab." />
</Frame>

### Audit Log

The **Audit Log** tab shows a list of events that have occurred in your Airflow environment related to the Dag, Dag run, or task instance you have selected.

<Frame>
  <img alt="Screenshot of the Audit Log tab." />
</Frame>

### Code

The **Code** tab allows you to inspect the code for the Dag itself. It also shows the date and time of the last parse. You can copy the code, select which [Dag version](/docs/learn/airflow-dag-versioning) you want to see, and even compare the **Diff** between different Dag versions.

<Frame>
  <img alt="Screenshot of the Code tab." />
</Frame>

### Details

The **Details** tab displays detailed information about the Dag. This includes information such as the Dag ID, description, timezone, file location, last parsed time, latest version information, and start time. You can also see Dag configuration information such as the concurrency number, max active runs/tasks, if catchup is enabled, and any params.

<Frame>
  <img alt="Screenshot of the Details tab." />
</Frame>

### Dag run

The **Dag run** view represents a single Dag run. It offers similar tabs to those in the Dag view:

* **Task Instances**: Shows metadata for every task instance within this specific Dag run. Clicking on a task ID brings you to the details of the instance, including the logs and XComs.
* **Asset Events**: Displays the source [asset events](/docs/learn/airflow-datasets) with their details.
* **Audit Log**: Provides a detailed audit trail related specifically to this Dag run.
* **Code**: Shows the exact version of the Dag code that was executed for this run, which is critical for debugging historical runs.
* **Details**: Displays metadata specific to this run.

The Dag run view also offers an important set of functions for operational management, including responding to any pending required actions, adding a note to the Dag run, clearing the Dag run, marking it as successful (**Checkmark**) or failed (**X**), or deleting the Dag run entirely.

<Frame>
  <img alt="Screenshot of actions for a single Dag run." />
</Frame>

### Task Instance

Drilling down one level further, selecting an entry from the **Task Instances** tab opens the individual **Task Instance** view. This is the most granular interface for debugging in Airflow, focusing on a single execution of a single task.

The centerpiece of this view is the **Logs** tab, which provides direct access to the log output for that specific task run. For easier debugging, the logs are syntax-highlighted, you can filter them by log level to quickly isolate important messages, and use the **Search** field to run a freetext search. In addition to logs, other tabs offer further information about your task instance, for example, the data it pushed to XCom.

<Frame>
  <img alt="Screenshot of the Logs tab for an individual task instance." />
</Frame>

## Assets

The **Assets** tab allows you to see a list of the [assets](/docs/learn/airflow-datasets) associated with your Airflow instance.

By default, assets are shown in a list that displays key information for each one:

* **Last Asset Event**: The date of the most recent asset event, indicating data freshness.
* **Group**: The asset group it belongs to.
* **Scheduled Dags**: Which Dags have schedules that include the asset.
* **Producing Tasks**: The specific tasks that updated the asset.

From this view, you can also use the play button to manually generate an asset event for testing or to kick off a data-aware workflow including one or more Dags.

<Frame>
  <img alt="Screenshot of the assets list view." />
</Frame>

There are two ways to create an asset event:

* **Materialize**: Trigger the underlying producing task of this asset.
* **Manual**: Directly create the asset event, without running the producing task. Allows you to optionally attach extra information and/or a [partition key](/docs/learn/airflow-partitioned-runs) to the asset event.

<Frame>
  <img alt="Screenshot of the create asset event dialog." />
</Frame>

By clicking an individual asset, you can switch to the **asset graph**, which visualizes the dependencies between assets and Dags, providing an overview of how your Dags and your data depend on each other. On the right side of an asset graph, you can see more details about the related asset events, as well as the [**Asset State Store**](/docs/learn/airflow-advanced-asset-scheduling#asset-state-store).

You can also generate an asset event from this view by using the play button in the top right corner.

<Frame>
  <img alt="Screenshot of the assets graph view." />
</Frame>

## Browse

The **Browse** tab provides detailed information from your Audit Log, Deadlines, Airflow Jobs, required human-in-the-loop actions, and your XComs.

<Frame>
  <img alt="Screenshot of the Browse tab." />
</Frame>

The Audit Log lists all your logged events. This includes information such as when an event occurred, which user is associated with the event, and extra information about the associated Dag and task.

The **Deadlines** view contains a list of all [deadline alerts](https://airflow.apache.org/docs/apache-airflow/stable/howto/deadline-alerts.html) alongside their Dag run, deadline time and status.

The **Jobs** view shows a list of all Airflow Jobs currently running in your environment. An Airflow Job is a Python process for an internal Airflow component, such as a `SchedulerJob` or a `TriggererJob`. These jobs record heartbeats and track the lifecycle of those components.

The **XComs** view provides a centralized location to inspect all XComs pushed by tasks across your Airflow environment. It is an essential tool for debugging data-passing issues between tasks. The view displays a filterable list where each entry shows the XCom's key, its value, and the exact task instance that pushed it, identified by its Dag ID, run ID, task ID, and map index.

Under **Required Actions**, you can find the global list for managing all human-in-the-loop tasks across your entire Airflow instance, including all task instances that are either pending user input or have been previously reviewed. You can filter the list to focus on a specific required action state.

To take action on a pending task, simply click the **Open review drawer** icon next to the action's state and respond in the drawer to the right.

<Frame>
  <img alt="The Browse Required Actions list with the review drawer open, with arrows highlighting the Open review drawer icon and the response buttons in the drawer." />
</Frame>

## Admin

The **Admin** tab provides you with tools for operational management not specific to any particular Dag. You can use them to view and modify your Airflow environment.

* **Variables**: View and manage [Airflow variables](/docs/learn/airflow-variables).
* **Pools**: View and manage [Airflow pools](/docs/learn/airflow-pools).
* **Providers**: View all installed [Provider packages](https://airflow.apache.org/registry/) in your environment.
* **Plugins**: View any [Airflow plugins](https://airflow.apache.org/docs/apache-airflow/stable/plugins.html) defined in your environment.
* **Connections**: View and manage [Airflow connections](/docs/learn/connections).
* **Config**: View the contents of your `airflow.cfg` file.

<Info>
  Note that the **Config** view is often disabled for security reasons. You can control this behavior by setting the [api.`expose_config`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#expose-config) configuration.
</Info>

## Docs

The **Docs** tab provides links to external Airflow resources, such as the official [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/index.html), the [GitHub repository](https://github.com/apache/airflow), and the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) reference.

## Timezone

The **Timezone** tab lets you change the time zone used for time displays in the Airflow UI. Note that this setting doesn't influence your Dag schedules.

## User

The bottom-most icon on the navigation is the **User** tab. This allows you to:

* Select your preferred language
* Toggle Light/Dark Mode
* Default to Graph View
* Generate an API or CLI JWT token.
* Log out

<Note>
  On Astro you don't need to generate a JWT token to programmatically interact with your Airflow environment. For CLI access use the [Astro CLI](https://www.astronomer.io/docs/astro/cli/v-1-43/overview) and authenticate with `astro login`. See [Using the Airflow REST API with Astro](/docs/astro/airflow-api) for information on how to get API access to Airflow environments running on Astro.
</Note>

## Conclusion

This guide provided a basic overview of some of the most commonly used features of the Airflow UI. The UI has become much easier and more intuitive from 2.x to 3.x.

<Tip>
  [Airflow plugins](https://airflow.apache.org/docs/apache-airflow/stable/plugins.html) are external features that can be added to customize your Airflow installation. In [this guide](/docs/learn/using-airflow-plugins), you learn how to extend the functionality of the Airflow UI using plugins.
</Tip>

The Airflow UI is a dynamic and evolving part of Airflow, with the open-source community continuously working to improve the user experience and add new functionality. To take full advantage of these enhancements, make sure to upgrade your Airflow environment frequently.

If you have ideas for improving the UI or want to help build its next generation of features, the Apache Airflow community welcomes your [contributions](https://github.com/apache/airflow?tab=readme-ov-file#contributing).


# Upgrade from Apache Airflow® 2 to 3
Source: https://astronomer.io/docs/learn/airflow-upgrade-2-3

Learn how upgrade your Airflow 2 environment to Airflow 3.

Airflow 3 is a major release of [Apache Airflow®](https://airflow.apache.org/) that includes a completely new UI and significant architectural changes, improving Airflow's security posture and enabling new features. While Airflow developers took great care to keep as much backward compatibility as possible, making the upgrade process as efficient and smooth as it
can be, there are some breaking changes that you need to be aware of. Additionally, the Airflow project has tools to help you upgrade your DAG code and Airflow configuration to be compatible with Airflow 3.

This guide provides a checklist for upgrading from Airflow 2 to Airflow 3, including:

* How to check your DAG code for compatibility with Airflow 3
* How to check your Airflow config for compatibility with Airflow 3
* A list of important breaking changes between Airflow 2 and Airflow 3

<Tip>
  This guide covers important breaking changes between Airflow 2 and Airflow 3, as well as upgrading instructions for open-source Airflow. Astronomer customers should also refer to the Astro documentation for specific upgrade instructions. For more in depth instructions on how to upgrade from Airflow 2 to Airflow 3, see the free eBook [Practical guide: Upgrade from Apache Airflow 2 to Airflow 3](https://www.astronomer.io/ebooks/upgrade-from-airflow-2-to-3?utm_source=website\&utm_medium=learn-guides\&utm_campaign=airflow-upgrade-2-3).
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Basic knowledge of Airflow components. See [Airflow Components](/docs/learn/airflow-components).

## Upgrade checklist

The following checklist provides a high-level overview of the steps you need to take to upgrade your Airflow 2 environment to Airflow 3. The steps are described in more detail in the sections below.

1. Make sure you are at least on Astro Runtime 13.7.0+ (Airflow 2.11) as this is the minimum version required to update Deployments on Astro.
2. Use the [Airflow ruff rules](#check-your-dag-code-with-ruff) to check your Airflow DAG code, and make all necessary changes. If you are using the Astro CLI, the [`astro dev upgrade-test`](/docs/cli/v1.43/astro-dev-upgrade-test) command contains ruff check options.
3. Use the [Airflow config linter](#check-your-airflow-config) to check your Airflow config, and make all
   necessary changes.
4. Assess whether you need to make additional changes to the DAGs based on the list of [breaking changes](#breaking-changes).
5. Upgrade your local development Airflow environment to Airflow 3. If you are using the Astro CLI:
   * Check your Astro CLI version with `astro version`. You need to be at least on version 1.34.0 to run Airflow 3. You can upgrade the Astro CLI with `brew upgrade astro`.
   * Change the Astro Runtime version in your project's Dockerfile to `FROM astrocrpublic.azurecr.io/runtime:<astro-runtime-version>` (see the [Astro Runtime release notes](/docs/runtime/runtime-release-notes) for the latest version available).
6. Run your updated DAGs with Airflow 3 locally to test them.
7. Upgrade your production environment to Airflow 3. Astronomer customers should refer to the Astro documentation for upgrade instructions.
8. Deploy your updated DAGs to the cloud environment.

<Info>
  If you are still using Airflow 1, we highly recommend upgrading to Airflow 2 as soon as possible. Support for Airflow 1 ended on June 17, 2021, so no further updates are being made, and potential security issues in Airflow 1 aren't being addressed. After upgrading to Airflow 2, upgrade to Airflow 2.6.3+; then upgrade to Airflow 3 as explained in this chapter. For information on upgrading from Airflow 1 to Airflow 2, see the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/howto/upgrading-from-1-10/index.html).
</Info>

## Check your DAG code with ruff

In Airflow 3 several deprecated parameters and import paths have been removed. This means that if you have been using deprecated parameters or import paths in your DAG code, you will need to update them to be compatible with Airflow 3. The [ruff](https://ruff.rs/) linter is a Python linter and code transformation tool that can be used to check your Airflow DAG code for compatibility with Airflow 3. There are two sets of ruff rules available for upgrading from Airflow 2 to Airflow 3:

* AIR30: These rules check your Airflow code for removed parameters and imports that are no longer available in Airflow 3. Making these changes is mandatory for your DAGs to work in Airflow 3.
* AIR31: These rules check your Airflow code for deprecated parameters and imports that are still available in Airflow 3, but will be removed in future versions. Making these changes is recommended to ensure that your DAGs will continue to work in future versions of Airflow.

To use the ruff linter, you need to install the latest version of ruff. You can do this with pip:

```bash wrap theme={null}
pip install --upgrade ruff
```

Then, you can run the ruff linter on your Airflow DAG code with the following command:

```bash wrap theme={null}
ruff check --preview --select AIR30 <path_to_your_dag_code>
```

You can add `--fix` to the command to automatically fix issues that ruff finds, note that not all issues can be fixed automatically.

After running this command you will see a list of issues that ruff found in your code in your terminal, with a suggestion for how to fix them. For example, if you have a DAG that uses the `fail_stop` [DAG parameter](/docs/learn/airflow-dag-parameters), which was renamed to `fail_fast`, you will see an error message like this:

```bash wrap theme={null}
dags/my_dag.py:19:5: AIR301 [*] `fail_stop` is removed in Airflow 3.0
   |
17 |     start_date=datetime(2025, 1, 1),
18 |     schedule="@daily",
19 |     fail_stop=True
   |     ^^^^^^^^^ AIR301
20 | ):
   |
   = help: Use `fail_fast` instead

Found 1 error.
[*] 1 fixable with the `--fix` option.
```

<Tip>
  If you are using the Astro CLI, the ruff check is included in the `astro dev upgrade-test` command. See [the Astro CLI documentation](/docs/cli/v1.43/astro-dev-upgrade-test) for more information.
</Tip>

<Info>
  The ruff linter is a great tool to help you update your DAG code for Airflow 3. However, it can't detect all potential compatibility issues. After running the ruff linter, you should still read through the [breaking changes](#breaking-changes) section of this guide and the [Airflow release notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html) to ensure that your code is compatible with Airflow 3.
</Info>

## Check your Airflow config

In Airflow 3, some changes have been made to configuration options. For upgrading purposes, four categories of changes are relevant:

* **Default changes**: Some defaults have changed, for example the default for `[scheduler].catchup_by_default` has changed from True to False.
* **Renamed options**: Some options have been renamed and/or moved to another section, such as `[webserver].web_server_host` which has been renamed and moved to `[api].host`.
* **Removed options**: Some previously deprecated options have been removed, such as `[webserver].error_logfile`.
* **Previously valid options are now invalid**: Some options that were previously valid are now invalid. For example, `0` used to be a valid input to `[core].parallelism`, but now a positive integer is required.

You can learn more about all valid configuration options in the [Airflow configuration reference](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html).

The `airflow config lint` command of the [Airflow CLI](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html) that can be used to check your Airflow configuration for compatibility with Airflow 3 and `airflow config update` will make the necessary changes to your configuration file for it to be compatible with Airflow 3.

Astro CLI users first need to export their `airflow.cfg` file and potentially make a change for it to parse through the linter correctly:

1. Run `astro dev start` to start your Astro CLI project.
2. Run `astro dev run config list | awk '/^\[core\]/ {found=1} found' > airflow.cfg` to export your current configuration file. This command removes any additional lines at the beginning of the file that might cause a parsing error in the linter.
3. Copy the `airflow.cfg` file into your scheduler container with `docker cp airflow.cfg  <scheduler container>:/usr/local/airflow`.
4. Enter the scheduler container with `astro dev bash`.
5. Run `airflow config lint` inside the container to check your configuration file for compatibility with Airflow 3.
6. Make the necessary changes to your environment variables related to configuration options. See [Environment variables](/docs/astro/environment-variables) in the Astro documentation for more information on how to set environment variables in your Astro project.

After running the `airflow config lint` command, you will see a list of issues that were found in your configuration file like the example below:

```bash wrap theme={null}
Found issues in your airflow.cfg:
  - `base_url` configuration parameter moved from `webserver` section to `api` section as `base_url`.
  - Removed `error_logfile` configuration parameter from `webserver` section.

Please update your configuration file accordingly.
```

## Breaking changes

Being a new major version, Airflow 3 comes with a number of breaking changes that can affect some of your DAGs, depending on which features you are using. This section lists the most important breaking changes that you need to be aware of when upgrading from Airflow 2 to Airflow 3.

<Tip>
  The list of breaking changes in this guide focuses on the most relevant ones but isn't exhaustive. For a full list of changes between Airflow 2 and Airflow 3, see the [Airflow release notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html).
</Tip>

### Removed direct metadata database access

In Airflow 2, all tasks had direct access to the Airflow metadata database. This access was removed in Airflow 3, greatly improving Airflow’s security posture. If you are accessing the Airflow metadata database directly in any of your task or trigger code, such as by using the SQLAlchemy connection environment variable, that process will error in Airflow 3. This includes custom operators making such a connection.

**Recommendation**: Directly accessing the Airflow metadata database from within tasks is an antipattern because it could lead to accidental modifying or dropping of information that is vital to Airflow’s functioning, up to and including corruption of your entire Airflow instance. To interact with and retrieve information about your Airflow instance, use the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) instead.

### Changes related to scheduling

In Airflow 3, the following changes were made to scheduling parameters and utilities:

* `schedule_interval` and `timetable` were deprecated in favor of `schedule`. The default schedule is now `None`.
* `catchup` was set to `False` by default at the configuration level. This means that if you don't set a value for `catchup`, Airflow won't try to catch up on missed runs. You can enable the Airflow 2 behavior by setting the `[scheduler].catchup_by_default` configuration option to `True`.
* The `days_ago` function was removed in favor of `pendulum.today('UTC').add(days=-N, ...)`.

If you pass raw cron strings to your DAG's `schedule`, for example `0 0 * * *`, by default it used to be interpreted with the `CronDataIntervalTimetable` timetable under the hood. In Airflow 3, this behavior was changed to use the `CronTriggerTimetable` timetable instead. You can change this behavior back to the Airflow 2 behavior by setting the `[scheduler].create_cron_data_intervals` configuration option to `True`. For more information on the differences between the two timetables, see the [Timetables comparisons in the Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timetable.html#differences-between-the-two-cron-timetables).

The `logical_date` attribute of the DAG run was changed from being equivalent to the `data_interval_start` in Airflow 2 to being equivalent to the `run_after` date in Airflow 3. This means that the `logical_date` is now equivalent to the `run_after` date and the `run_id` takes its timestamp from the moment in time when the DAG run is queued. It is also now possible to pass `None` as the `logical_date`. The deprecated `execution_date` attribute was removed. This change is mostly relevant for users that utilize a time-dependent context element in the logic of their DAGs, for example to partition their data in a SQL query. See [Schedule DAGs in Apache Airflow®](/docs/learn/scheduling-in-airflow) for more information.

### Other changes

Airflow 3 introduces other improvements and changes that may affect you if you use any of the related features. The following list summarizes the most important ones:

* Airflow 3 uses a new `v2` version of the Airflow REST API. If you are using the REST API to interact with Airflow you'll likely need to update your scripts. See the [Airflow REST API documentation](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) for more information.
* There were changes to the Airflow context that included the removal of deprecated keys such as `execution_date`. See the [Airflow context guide](/docs/learn/airflow-context) for more information.
* The long deprecated subdags feature was removed. If you are using subdags in your DAGs, you will need to refactor them to use [task groups](/docs/learn/task-groups) instead.
* The SLA feature of Airflow 2 was removed. Astro customers should use [Astro Alerts](/docs/astro/alerts) for simple SLAs and [Astro Observe ](/docs/astro/create-data-products) for advanced SLAs instead. For open source Airflow users, a [new deadline alerts](https://airflow.apache.org/docs/apache-airflow/stable/howto/deadline-alerts.html) feature is available in Airflow 3.2+.
* Given the [new React-based U](/docs/learn/airflow-ui), Flask-AppBuilder (FAB) was removed in Airflow 3. The default auth manager was changed to `SimpleAuthManager`. If you need FAB integration, install the [FAB provider](https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/index.html). For more information on auth managers, see the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/auth-manager.html). Support for FAB-based plugins is limited in Airflow 3.0 but will be available in a future release.
* The deprecated Astro Python SDK package isn't compatible with Airflow 3.
* [XCom](/docs/learn/airflow-passing-data-between-tasks) pickling is no longer allowed when using the default XCom backend in Airflow 3 for [security reasons](https://docs.python.org/3/library/pickle.html). Existing pickled XComs are moved to an archive table. Use a [custom XCom backend](/docs/learn/custom-xcom-backend-strategies) with custom serialization to pass data between tasks that Airflow can't serialize by default.
* The email/SMTP integration in Airflow core was deprecated and will be removed in Airflow 4. For more information on how to set up email notifications see the [Manage Apache Airflow® DAG notifications guide](/docs/learn/error-notifications-in-airflow).
* Python 3.8 is no longer supported in Airflow 3. You need to be on Python 3.9 or higher to run Airflow 3.
* PostgreSQL 12 is no longer supported in Airflow 3. You need to be on PostgreSQL 13 or higher to run Airflow 3.

Depending on how you run Airflow, you may find that some Airflow providers (such as FTP, HTTP, and IMAP) that used to be preinstalled in your image/package for Airflow 2 aren't preinstalled in Airflow 3. Pip-install the needed providers in your Airflow environment. See the Airflow documentation for a [list of officially supported providers](https://airflow.apache.org/docs).


# Use Airflow variables
Source: https://astronomer.io/docs/learn/airflow-variables

Create and use Airflow variables.

An Airflow variable is a key-value pair that can be used to store information in your Airflow environment. They are commonly used to store instance level information that rarely changes, including secrets like an API key or the path to a configuration file.

There are two distinct types of Airflow variables: regular values and JSON serialized values.

<Frame>
  <img alt="Variables in the Airflow UI" />
</Frame>

This guide covers how to create Airflow variables and access them programmatically.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow DAGs. See [Introduction to Airflow DAGs](/docs/learn/dags).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).

## Best practices for storing information in Airflow

[Airflow variables](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/variables.html#variables) store key-value pairs or short JSON objects that need to be accessible in your whole Airflow instance. They are Airflow’s runtime configuration concept and defined using the [`airflow.sdk.definitions.variable.Variable`](https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/definitions/variable.py) object.

There are some best practices to keep in mind when using Airflow variables:

* Airflow variables should be used for information that is runtime dependent but doesn't change too frequently.
* You should avoid using Airflow variables outside of tasks in top-level DAG code, as they will create a connection to the Airflow metastore every time the DAG is parsed, which can lead to performance issues. See [DAG writing best practices in Apache Airflow](/docs/learn/dag-best-practices#avoid-top-level-code-in-your-dag-file).
* If you do use Airflow variables in top-level DAG code, use the [Jinja template](/docs/learn/templating) syntax so that your Airflow variables are only rendered when a task executes.
* Airflow variables are encrypted with [Fernet](https://github.com/fernet/spec/) when they are written to the Airflow metastore. To mask Airflow variables in the UI and logs, include a substring indicating a sensitive value in your Airflow variable name. See [Hiding sensitive information](#hide-sensitive-information-in-airflow-variables).

See the Airflow documentation for examples of code showing [good and bad practices for accessing Airflow variables in a DAG](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#airflow-variables).

Aside from Airflow variables, there are other ways of storing information in Airflow. The ideal option depends on what type of information you are storing and where and how you want to access it:

* Environment variables store small pieces of information that are available to the whole Airflow environment. There is no direct way to see environment variables in the Airflow UI but they can be accessed using `os.getenv("MY_ENV_VAR")` inside of Airflow DAGs and tasks. Environment variables are very versatile, as they can be used to both store arbitrary information and [configure Airflow](https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html). One advantage of using environment variables is that you can include their creation in your CI/CD process. They are also often used to store credentials for local development.
* [Params](/docs/learn/airflow-params) can be used to store information specific to a DAG or DAG run. You can define defaults for params at the DAG or task level and override them at runtime. Params aren't encrypted and shouldn't be used to store secrets.
* [XComs](/docs/learn/airflow-passing-data-between-tasks) can be used to pass small pieces of information between Airflow tasks. Use XComs when the information is likely to change with each DAG run and mostly needs to be accessed by individual tasks in or outside of the DAG from within which the XCom is created. Default XComs aren't encrypted and shouldn't be used to store secrets.

## Create an Airflow variable

There are several ways to create Airflow variables:

* Using the Airflow UI
* Using the Airflow CLI.
* Using an environment variable.
* Programmatically from within an Airflow task.

### Use the Airflow UI

To create an Airflow variable in the UI, click the **Admin** tab and select **Variables**. Then click the **+** button and enter a key, value and an optional description for your Airflow variable. You also have the option to **Import Variables** from a file.

<Frame>
  <img alt="UI" />
</Frame>

### Use the Airflow CLI

The Airflow CLI contains options to set, get and delete [Airflow variables](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#variables). To create an Airflow variable through the CLI, use the following command:

<details>
  <summary>Astro</summary>

  ```sh wrap theme={null}
  astro dev run variables set my_var my_value
  astro dev run variables set -j my_json_var '{"key": "value"}'
  ```

  Note that [`astro dev run`](/docs/cli/v1.43/astro-dev-run) executes Airflow commands only in your local Airflow environment and can't be used on Astro Deployments. To set Airflow variables for an Astro Deployment, use the [Astro Environment Manager](/docs/astro/manage-connections-variables) or [environment variables](#using-environment-variables).
</details>

<details>
  <summary>Airflow</summary>

  ```sh wrap theme={null}
  airflow variables set my_var my_value
  airflow variables set -j my_json_var '{"key": "value"}'
  ```
</details>

### Using environment variables

To set Airflow variables using an environment variable, create an environment variable with the prefix `AIRFLOW_VAR_` and the name of the Airflow variable you want to set. For example:

```text wrap theme={null}
AIRFLOW_VAR_MYREGULARVAR='my_value'
AIRFLOW_VAR_MYJSONVAR='{"hello":"world"}'
```

To fetch the Airflow variable in the DAG, you can then use the following methods:

* `Variable.get('<VAR_NAME>', '<default-value>')`: This method is recommended as it is the most secure way to fetch secret values. However, if used in top-level DAG code or as an argument in the operator, this method can affect the performance because it makes a request to the Airflow metadata database every time your DAGs are parsed, which can occur every 30 seconds. An alternative approach is to use the [Jinja template](/docs/learn/templating) `{{ var.value.get(<var_name>, '<default-value>') }}`, which is evaluated only at runtime. See [DAG writing best practices](/docs/learn/dag-best-practices#avoid-top-level-code-in-your-dag-file) for more information about avoiding repeated requests in top level code.

* `os.getenv('AIRFLOW_VAR_<VAR_NAME>','<default-value>')`: This method is faster because it reduces the number of Airflow metadata database requests. However, it's less secure. Astronomer doesn't recommend using `os.getenv` with secret values because retrieving environment variables with this method can print them to your logs.

If Airflow can't find the environment variable, replace `<default_value>` with a default value.

To learn more about how to set environment variables on Astro, see [Environment Variables](/docs/astro/manage-env-vars).

### Programmatically from a DAG or task

Lastly, you can programmatically set Airflow variables within your Airflow tasks using the [`Variable` model](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/variable/index.html#module-airflow.models.variable). If you want to serialize a JSON value, make sure to set `serialize_json=True`.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # from airflow.sdk import task

  @task
  def set_var():
      from airflow.sdk import Variable
      Variable.set(key="my_regular_var", value="Hello!")
      Variable.set(key="my_json_var", value={"num1": 23, "num2": 42}, serialize_json=True)
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # from airflow.providers.standard.operators.python import PythonOperator

  def set_var_func():
      from airflow.sdk import Variable
      Variable.set(key="my_regular_var", value="Hello!")
      Variable.set(key="my_json_var", value={"num1": 23, "num2": 42}, serialize_json=True)

  PythonOperator(
      task_id="set_var",
      python_callable=set_var_func,
  )
  ```
</details>

Updating an Airflow variable works the same way by using the `.update()` method.

## Retrieve an Airflow variable

To programmatically retrieve an Airflow variable, you can either use the `.get()` method of the Airflow variable model or you can pull the Airflow variable value directly from the [Airflow context](/docs/learn/airflow-context).

When retrieving a JSON serialized Airflow variable, make sure to set `deserialize_json=True` in the `.get()` method or access the `json` key from the `var` dictionary in the Airflow context.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # from airflow.sdk import task

  @task
  def get_var_regular():
      from airflow.sdk import Variable
      my_regular_var = Variable.get("my_regular_var", default=None)
      my_json_var = Variable.get(
          "my_json_var", deserialize_json=True, default=None
      )["num1"]
      print(my_regular_var)
      print(my_json_var)

  @task
  def get_var_from_context(**context):
      my_regular_var = context["var"]["value"].get("my_regular_var")
      my_json_var = context["var"]["json"].get("my_json_var")["num2"]
      print(my_regular_var)
      print(my_json_var)
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # from airflow.providers.standard.operators.python import PythonOperator

  def get_var_regular_func():
      from airflow.sdk import Variable
      my_regular_var = Variable.get("my_regular_var", default=None)
      my_json_var = Variable.get(
          "my_json_var", deserialize_json=True, default=None
      )["num1"]
      print(my_regular_var)
      print(my_json_var)

  def get_var_from_context_func(**context):
      my_regular_var = context["var"]["value"].get("my_regular_var")
      my_json_var = context["var"]["json"].get("my_json_var")["num2"]
      print(my_regular_var)
      print(my_json_var)

  PythonOperator(
      task_id="get_var_regular",
      python_callable=get_var_regular_func,
  )

  PythonOperator(
      task_id="get_var_from_context",
      python_callable=get_var_from_context_func,
  )
  ```
</details>

When using traditional Airflow operators, it's often easier to use a [Jinja template](/docs/learn/templating) to retrieve Airflow variables. See [Airflow variables in Templates](https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#airflow-variables-in-templates).

```python wrap theme={null}
# from airflow.providers.standard.operators.bash import BashOperator

get_var_jinja = BashOperator(
    task_id="get_var_jinja",
    bash_command='echo "{{ var.value.my_regular_var }} {{ var.json.my_json_var.num2 }}"',
)
```

You can also retrieve an Airflow variable using the Airflow CLI's [`get`](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#get_repeat3) and [`list`](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#list_repeat8) commands.

## Hide sensitive information in Airflow variables

Airflow variables are [Fernet](https://github.com/fernet/spec/) encrypted in the Airflow metastore.

As seen in the screenshot at the beginning of this guide, some Airflow variables are additionally masked in the Airflow UI and logs. By default, the [`hide_sensitive_var_conn_fields`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#hide-sensitive-var-conn-fields) configuration is set to `True`, which automatically masks all Airflow variables that contain the following strings:

* `access_token`
* `api_key`
* `apikey`
* `authorization`
* `passphrase`
* `passwd`
* `password`
* `private_key`
* `secret`
* `token`

This list can be extended by adding comma separated strings to the [`sensitive_var_conn_names`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#sensitive-var-conn-names) configuration. See [Masking sensitive data](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/secrets/mask-sensitive-values.html).

On Astro you can also manually mark Airflow variables as secrets when creating them as an environment variable. See [Set environment variables on Astro](/docs/astro/environment-variables).

<Info>
  If you need to access the same sensitive information in several Airflow instances, consider using a [Secrets Backend](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html).
</Info>


# Set up Remote Execution Agents on Astro
Source: https://astronomer.io/docs/learn/astro-remote-execution

Learn how to set up Remote Execution on Astro.

Remote Execution is a feature in Airflow 3 that allows you to run your Airflow tasks on any machine, in the cloud or on-premises. When using Remote Execution, only the information that’s essential for running the task, such as scheduling details and heartbeat pings, is available to Airflow system components. Everything else stays within the remote environment, making this a key feature in highly regulated industries.

This tutorial covers when to use Remote Execution and how to set it up on Astro with Remote Execution Agents running on AWS EKS or on-premises infrastructure. While this guide focuses on these specific environments, the concepts and steps can be adapted for other Kubernetes clusters, for example running on GCP or Azure.

<Note>
  Remote execution on Astro is only available for Airflow 3.x Deployments on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).
</Note>

## When to use remote execution

You might want to use Remote Execution in the following situations:

* Running tasks that need to access and/or use sensitive data that can't leave a particular environment, such as an on-premises server. This requirement is common in highly regulated industries like financial services and health care.
* Running tasks that require specialized compute, such as a GPU or TPU machine to train neural networks.

You can accomplish Remote Execution in two ways:

* When running Airflow on Astro, you can use [Remote Execution Agents](/docs/astro/remote-execution-configure-agents) with the [AstroExecutor](/docs/astro/astro-executor).
* When running open source Airflow, you can use the EdgeExecutor, which is part of the [edge3 provider package](https://airflow.apache.org/docs/apache-airflow-providers-edge3/stable/index.html).

This tutorial covers the steps for setting up Remote Execution Agents on Astro to run on AWS EKS and on-premises.

## Time to complete

This tutorial takes approximately one hour to complete.

## Assumed knowledge

To get the most out of this tutorial, you should have an understanding of:

* The [Airflow components](/docs/learn/airflow-components) and how they work together.

## Prerequisites

* An Astronomer account on the [Enterprise tier](https://www.astronomer.io/pricing/).
* Access to [AWS EKS](https://docs.aws.amazon.com/eks/) or an on-premises Kubernetes environment.
* [kubectl](https://kubernetes.io/docs/tasks/tools/) installed.
* An [S3](https://docs.aws.amazon.com/s3/) bucket to store [XComs](/docs/learn/airflow-passing-data-between-tasks).
* If you are using AWS EKS, you need to have [eksctl](https://github.com/eksctl-io/eksctl) and the [AWS CLI](https://docs.aws.amazon.com/cli/latest/) installed.

## Step 1: Create a Remote Execution Deployment

To start registering Remote Execution Agents, you first need to create a dedicated Remote Execution Deployment on Astro.

1. Make sure you have a **dedicated cluster** in your Astro Workspace. If you don't, you can [create a new dedicated cluster](/docs/astro/create-dedicated-cluster). When creating a new cluster, you can leave the VPC Subnet range at its default setting (`172.20.0.0/19`) or customize it for your needs. Note that it can take up to an hour for a new cluster to be provisioned. If you later want to use customer managed workload identity to read logs from Remote Execution Agents running on AWS EKS, you need to create your dedicated cluster on AWS.
2. [Create a Remote Execution Deployment](/docs/astro/create-deployment) in your Astro Workspace.

   * Select **Remote Execution** as the execution mode.
   * Select your dedicated cluster.

   <Frame>
     <img alt="Create a Remote Execution Deployment" />
   </Frame>

## Step 2: Create an agent token

Your Remote Execution Agents will need to authenticate themselves to your Astro Deployment. To do this, you need to create an Agent Token.

1. In the Astro UI, select the Remote Execution Deployment you created in the previous step and click the **Remote Agents** tab.

2. Select **Tokens**.

3. Click **+Agent Token** and create a new Agent Token.

   <Frame>
     <img alt="Create an Agent Token" />
   </Frame>

4. Make sure to save the Agent Token in a secure location as you will need it later.

## Step 3: Create a Deployment API Token

Your Remote Execution Agents will also need to fetch the right images from your Astro Deployment. To do this, you need to create a Deployment API Token.

1. In the Astro UI, select the Remote Execution Deployment you created in [Step 1](#step-1-create-a-remote-execution-deployment) and click the **Access** tab.

2. Select **API Tokens**.

3. Click **+ API Token**.

4. Select **Add Deployment API Token** and create a new Deployment API Token with Admin permissions.

   <Frame>
     <img alt="Create a Deployment API Token" />
   </Frame>

5. Make sure to save the Deployment API Token in a secure location as you will need it later.

## Step 4: Retrieve your `values.yaml` file

1. In the Astro UI, select the Remote Execution Deployment you created in [Step 1](#step-1-create-a-remote-execution-deployment) and click the **Remote Agents** tab.

2. Click **Register a Remote Agent**.

   <Frame>
     <img alt="Register a Remote Agent" />
   </Frame>

3. Download the `values.yaml` file you are given.

Note that no Remote Execution Agents show up in the list yet, they will only appear in the **Remote Agents** tab when they start heartbeating!

## Step 5A: Set up your Kubernetes cluster on EKS

This step covers the setup for deploying the Remote Execution Agent on AWS EKS. For a simple on-premises setup see [Step 5B](#step-5b-set-up-your-local-kubernetes-cluster).

1. Authenticate your machine to your AWS account. If your organization uses SSO, use `aws configure sso` and sign in through the browser. Make sure to set the `AWS_PROFILE` environment variable to the profile (`CLI profile name`) you used to sign in with `export AWS_PROFILE=<your-profile-name>`. You can verify your profile by running `aws sts get-caller-identity`.

2. To create a new EKS cluster, you need to define its parameters in a `my-cluster.yaml` file. Make sure the `workers` node group is large enough to support your intended workload and the Agent specifications in your `values.yaml` file for all 3 Agents. You can use the below example as a starting point, make sure to update `<your-cluster-name>` and `<your-region>` with your own values.

   ```yaml wrap theme={null}
   apiVersion: eksctl.io/v1alpha5
   kind: ClusterConfig

   metadata:
     name: <your-cluster-name>
     region: <your-region>          # it is recommended to use the same region as your Astro Cluster
     version: "1.33"               

   cloudWatch:
     clusterLogging:
       enableTypes: ["api", "audit", "authenticator", "controllerManager", "scheduler"] 

   iam:
     withOIDC: true                 # This setting is important for the IRSA role that will interact with S3 to save logs/xcom

   nodeGroups:
     - name: workers
       instanceType: m5.xlarge      # 4 vCPUs, 16 GiB RAM - minimum for 3x1CPU + k8s overhead
       desiredCapacity: 2           # Number of nodes to start with
       minSize: 0                   # Minimum number of nodes
       maxSize: 4                   # Maximum number of nodes
       volumeSize: 50               # EBS volume size in GB
       amiFamily: AmazonLinux2023      
       labels: { role: worker }
       tags:
         k8s.io/cluster-autoscaler/enabled: "true"
         k8s.io/cluster-autoscaler/remote-execution-airflow-cluster: "owned"
   ```

3. Create the EKS cluster by running the following command. Note the cluster creation can take up to 15-25 minutes.

   ```bash wrap theme={null}
   eksctl create cluster -f my-cluster.yaml
   ```

4. Configure `kubectl` to use your new EKS cluster by running the following command. Replace `<your-cluster-name>` with the name of your cluster.

   ```bash wrap theme={null}
   aws eks update-kubeconfig --name <your-cluster-name>
   ```

5. Verify that `kubectl` is aimed at the right cluster by running:

   ```bash wrap theme={null}
   kubectl get nodes
   ```

   The output should look similar to this:

   ```bash wrap theme={null}
   NAME                           STATUS   ROLES    AGE   VERSION
   ip-123-45-67-89.ec2.internal   Ready    <none>   16m   v1.33.4-eks-99d6cc0
   ip-123-45-67-90.ec2.internal   Ready    <none>   16m   v1.33.4-eks-99d6cc0
   ```

## Step 5B: Set up your local Kubernetes cluster

Alternatively, you can deploy the Remote Execution Agent on your on-premises cluster. If you want to test Remote Execution locally, a good option is to use the `Kubernetes` feature of [Orbstack](https://www.orbstack.dev/) or [Docker Desktop](https://docs.docker.com/desktop/kubernetes/). In this step we'll use Orbstack as an example.

1. Enable the `Kubernetes` feature in Orbstack.

   <Frame>
     <img alt="Enable Kubernetes in Orbstack" />
   </Frame>

2. Switch to the `orbstack` context:

   ```bash wrap theme={null}
   kubectl config use-context orbstack
   ```

## Step 6: Deploy the Remote Execution Agent

1. Create a new namespace for the Remote Execution Agent by running:

   ```bash wrap theme={null}
   kubectl create namespace <your-namespace>
   ```

2. Create a secret containing the Agent Token named `my-agent-token` by running the following command. Replace `<your-agent-token>` with the Agent Token you created in [Step 2](#step-2-create-an-agent-token). Replace `<your-namespace>` with the namespace you created.

   ```bash wrap theme={null}
   kubectl create secret generic my-agent-token \
   --from-literal=token=<your-agent-token> \
   --namespace <your-namespace>
   ```

3. Create a secret containing the Deployment API Token named `my-astro-registry-secret` by running the following command. Replace `<your-deployment-api-token>` with the Deployment API Token you created in [Step 3](#step-3-create-a-deployment-api-token) and replace `<your-namespace>` with your namespace.

   ```bash wrap theme={null}
   kubectl create secret docker-registry my-astro-registry-secret \
   --namespace <your-namespace> \
   --docker-server=images.astronomer.cloud \
   --docker-username=cli \
   --docker-password=<your-deployment-api-token>
   ```

4. Modify your `values.yaml` file to add `<your-namespace>`, as well as the names for your agent token (`agentTokenSecretName`) and deployment API token (`imagePullSecretName`).

   ```yaml wrap theme={null}
   resourceNamePrefix: "astro-agent"  # you can choose any prefix you want
   namespace: <your-namespace>
   imagePullSecretName: my-astro-registry-secret
   agentTokenSecretName: my-agent-token
   ```

5. Modify your `values.yaml` file to add your Dag bundle configuration to the `dagBundleConfigList` section.

   ```yaml wrap theme={null}
   dagBundleConfigList: <your-dag-bundle-config>
   ```

   Note that you need to store your Dags in a Dag bundle accessible to your Remote Execution Agents. Below is an example of a `GitDagBundle` configuration working with a Git connection named `git_default` (set in the `commonEnv` section later in this tutorial).

   ```yaml wrap theme={null}
   dagBundleConfigList: '[{"name": "gitbundle-1", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"git_conn_id": "git_default", "subdir": "dags", "tracking_ref": "main", "refresh_interval": 10}}]'
   ```

6. Modify your `values.yaml` file to add your XCom backend configuration to the `xcomBackend` section. For this tutorial we'll use the [Object Storage XCom Backend](/docs/learn/custom-xcom-backends-tutorial). The credentials are set in the `commonEnv` section later in this tutorial.

   ```yaml wrap theme={null}
   xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"
   ```

7. Modify your `values.yaml` file to set a secrets backend, we'll use the [Local Filesystem Secrets Backend](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/local-filesystem-secrets-backend.html) as a placeholder. Note that if you want to install an external secrets backend, you need to provide the relevant provider packages to the worker containers and credentials in `commonEnv`. For more information on how to interact with secrets backends, see [Configure a secrets backend](/docs/astro/secrets-backend).

   ```yaml wrap theme={null}
   secretBackend: "airflow.secrets.local_filesystem.LocalFilesystemBackend"
   ```

8. Modify your `values.yaml` file to add necessary environment variables to the `commonEnv` section. Make sure to replace all placeholders with your own values.

   ```yaml wrap theme={null}
   commonEnv:
     - name: ASTRONOMER_ENVIRONMENT
       value: "cloud"

     # This is the connection used in the GitDagBundle. If you want to access a private repo you need an access token with read and write permissions.
     - name: AIRFLOW_CONN_GIT_DEFAULT
       value: '{"conn_type": "git", "login": "<your GH login>", "password": "<access_token>", "host": "https://github.com/<account>/<repo>"}'

     # Update with your credentials that have access to your XCom S3 bucket!
     - name: AIRFLOW_CONN_AWS_DEFAULT  
       value: '{"conn_type": "aws", "login": "<your-access-key>", "password": "<your-secret-key>", "extra": {"region_name": "<your-region>"}}'

     # These two environment variables are needed for the custom XCom backend
     - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
       value: "s3://aws_default@<your-bucket>/xcom"  # replace the bucket with your XCom bucket. Uses the aws_default connection
     - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
       value: "0"  # all XCom will be stored in Object storage

     # Add any necessary environment variables for your secrets backend
   ```

9. Install the Helm chart by running the following command. Replace `<your-namespace>` with your namespace.

   ```bash wrap theme={null}
   helm repo add astronomer https://helm.astronomer.io/ 
   helm repo update
   helm install astro-agent astronomer/astro-remote-execution-agent --namespace <your-namespace> --values values.yaml
   ```

10. Verify that the 3 Remote Execution Agent pods are running by running the following command. Replace `<your-namespace>` with your namespace.

    ```bash wrap theme={null}
    kubectl get pods -n <your-namespace>
    ```

    The output should look similar to this:

    ```bash wrap theme={null}
    NAME                                                  READY   STATUS    RESTARTS   AGE
    astro-agent--dag-processor-7b46c75566-dsdlq           1/1     Running   0          87s
    astro-agent--triggerer-6cb88c8db7-kx9d2               1/1     Running   0          87s
    astro-agent--worker-default-worker-779c98cfb5-7chg2   1/1     Running   0          86s
    ```

On Astro you can see the 3 Remote Execution Agent pods happily heartbeating to your Astro Deployment. When you open the Airflow UI on this Astro Deployment, you'll be able to see and interact with all Dags contained in the configured Dag bundles.

<Frame>
  <img alt="Remote Execution Agent pods heartbeating" />
</Frame>

You can now run tasks on the remote EKS cluster! In order to be able to use XCom, see [Step 7](#step-7-configure-xcom) for more information.

<Note>
  If you ever need to update the Helm chart you can use the following command. Replace `<your-namespace>` with your namespace.

  ```bash wrap theme={null}
  helm upgrade astro-agent astronomer/astro-remote-execution-agent --namespace <your-namespace> --values values.yaml
  ```
</Note>

## Step 7: Configure XCom

If you want to use XCom to pass information between tasks running using Remote Execution, you need to configure a custom XCom backend. You already laid the foundation for this in [Step 6](#step-6-deploy-the-remote-execution-agent) when setting the following:

```yaml wrap theme={null}
xcomBackend: "airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"
commonEnv:
  # ...
  - name: AIRFLOW_CONN_AWS_DEFAULT  
    value: '{"conn_type": "aws", "extra": {"region_name": "us-east-1"}}'

  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH
    value: "s3://aws_default@<your bucket>/xcom"
  - name: AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD
    value: "0"
```

But in order for the worker pod to be able to use the XCom backend, you need to install the necessary Airflow provider packages on it. To make installation faster we recommend using a constraints file.

1. Create your `constraints.txt` file (see [GitHub](https://github.com/astronomer/webinar-demos/blob/apache-airflow-3-stronger-security-and-remote-execution/remote_execution_eks_templates/constraints.txt) for an example). Make sure that it includes the [Airflow Common IO provider](https://airflow.apache.org/registry/providers/common-io/) and the [Amazon provider](https://airflow.apache.org/registry/providers/amazon/) with the `s3fs` extra.

2. Add the constraints file as a configmap to the k8s cluster. Replace `<your-namespace>` with the namespace you created in [Step 6](#step-6-deploy-the-remote-execution-agent).

   ```bash wrap theme={null}
   kubectl create configmap constraints-configmap --from-file=constraints.txt -n <your-namespace>
   ```

3. Update your `values.yaml` file to install the necessary provider packages in the `workers` section. Update the versions as needed. Note that you also need to update the `PYTHONPATH` environment variable to include the shared packages. Note that your `image` version likely differs from the one in the example below.

   ```yaml wrap theme={null}
   initContainers:
     - name: install-amazon-provider-s3fs
       image: images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.12-astro-agent-1.0.2
       command: 
         - "pip"
         - "install" 
         - "--target"
         - "/shared/packages"
         - "--prefer-binary"
         - "--constraint"
         - "/constraints/constraints.txt"
         - "apache-airflow-providers-amazon[s3fs]==9.9.0"
         - "apache-airflow-providers-common-io==1.6.1"
       volumeMounts:
         - name: shared-packages
           mountPath: /shared/packages
         - name: constraints
           mountPath: /constraints

   env:
     - name: PYTHONPATH
       value: "/shared/packages:$PYTHONPATH"
   ```

4. Update your `values.yaml` file to mount the constraints file.

   ```yaml wrap theme={null}
       volumes:
       - name: shared-packages
         emptyDir: {}
       - name: constraints
         configMap:
           name: constraints-configmap

       volumeMounts:
       - name: shared-packages
         mountPath: /shared/packages
         readOnly: true
   ```

5. Update the Helm chart by running the following command. Replace `<your-namespace>` with your namespace.

   ```bash wrap theme={null}
   helm upgrade astro-agent astronomer/astro-remote-execution-agent --namespace <your-namespace> --values values.yaml
   ```

6. Run a Dag that uses XCom to verify the setup. Remember that you need to push the Dag to your Dag bundle location for it to be accessible to the Remote Execution Agent.

<Note>
  If you'd like to see your task logs displayed in the Airflow UI, see our docs on [Task logging for remote Deployments](/docs/astro/remote-execution-logging-overview).
</Note>

## Step 8: (optional, AWS only) use a secrets backend

If you want to use a secrets backend to store your connections and variables, you need to configure the Remote Execution Agent to use it.

1. First, you need an IAM role to attach this policy to. The IAM role's trust policy needs to include the EKS OIDC ID, so you need to fetch that first. Replace `<YOUR_EKS_CLUSTER_NAME>` with the name of your EKS cluster and `<YOUR_AWS_REGION>` with the region of your EKS cluster.

   ```bash wrap theme={null}
   OIDC_ISSUER_URL=$(aws eks describe-cluster --name <YOUR_EKS_CLUSTER_NAME> --query "cluster.identity.oidc.issuer" --output text)
   EKS_OIDC_ID=$(echo "$OIDC_ISSUER_URL" | sed -e 's|https://oidc.eks.<YOUR_AWS_REGION>.amazonaws.com/id/||')
   echo $EKS_OIDC_ID
   ```

2. Create a new file called `my-airflow-trust-policy.json` and add the following trust policy. Replace `<your-account-id>` with your AWS account ID, `<your-region>` with the region of your EKS cluster, `<your-namespace>` with the namespace you created in [Step 6](#step-6-deploy-the-remote-execution-agent), and `<your-cluster-oidc-id>` with the EKS OIDC ID you fetched in the previous substep.

   ```json wrap theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Federated": "arn:aws:iam::<your-account-id>:oidc-provider/oidc.eks.<your-region>.amazonaws.com/id/<your-cluster-oidc-id>"
               },
               "Action": "sts:AssumeRoleWithWebIdentity",
               "Condition": {
                   "StringEquals": {
                       "oidc.eks.<your-region>.amazonaws.com/id/<your-cluster-oidc-id>:sub": "system:serviceaccount:<your-namespace>:*",
                       "oidc.eks.<your-region>.amazonaws.com/id/<your-cluster-oidc-id>:aud": "sts.amazonaws.com"
                   }
               }
           }
       ]
   }
   ```

3. Create a new IAM role called `RemoteAgentsRole` with the trust policy you created in the previous step.

   ```bash wrap theme={null}
   aws iam create-role \
   --role-name RemoteAgentsRole \
   --assume-role-policy-document file://my-airflow-trust-policy.json
   ```

4. Create a new file called `my-airflow-secrets-policy.json` and add the following policy. Replace `<your-region>` with the region of your EKS cluster and `<your-account-id>` with your AWS account ID.

   ```json wrap theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "secretsmanager:GetSecretValue",
                   "secretsmanager:DescribeSecret",
                   "secretsmanager:ListSecrets"
               ],
               "Resource": "arn:aws:secretsmanager:<your-region>:<your-account-id>:secret:airflow/*"
           }
       ]
   }
   ```

   Create the policy using the following command.

   ```bash wrap theme={null}
   aws iam create-policy \
   --policy-name AirflowSecretsManagerAccess \
   --policy-document file://my-airflow-secrets-policy.json
   ```

5. Attach the `AirflowSecretsManagerAccess` policy to the `RemoteAgentsRole` role.

   ```bash wrap theme={null}
   aws iam attach-role-policy \
   --role-name RemoteAgentsRole \
   --policy-arn arn:aws:iam::<your-account-id>:policy/AirflowSecretsManagerAccess
   ```

6. Update the `serviceAccount` section in your `values.yaml` file to annotate the role to your service accounts. Replace `<your-account-id>` with your AWS account ID.

   ```yaml wrap theme={null}
   serviceAccount:
     workers:
       annotations:
         eks.amazonaws.com/role-arn: arn:aws:iam::<your-account-id>:role/RemoteAgentsRole

     dagProcessor:
       annotations:
         eks.amazonaws.com/role-arn: arn:aws:iam::<your-account-id>:role/RemoteAgentsRole

     triggerer:
       annotations:
         eks.amazonaws.com/role-arn: arn:aws:iam::<your-account-id>:role/RemoteAgentsRole 
   ```

7. Update the `commonEnv` section in your `values.yaml` file to configure the secrets backend. Replace `<your-role-arn>` with the ARN of the IAM role you created in the previous step.

   ```yaml wrap theme={null}
       secretBackend: "airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend"

       commonEnv:
       - name: AIRFLOW__SECRETS__BACKEND_KWARGS
         value: '{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}'
       - name: AWS_DEFAULT_REGION
         value: '<your-region>'
   ```

8. Since the secrets backend is also used in the Dag processor and Triggerer components and part of the [Airflow Amazon provider](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/secrets-backends/aws-secrets-manager.html), you need to install the necessary provider packages on these components as well, like you did for the worker pods when configuring the XCom backend in [Step 7](#step-7-configure-xcom). Note that your `image` version likely differs from the one in the example below.

   ```yaml expandable wrap theme={null}
   dagProcessor:
     # ... other dagProcessor config ...

     # Add PYTHONPATH to dagProcessor env
     env:
       - name: PYTHONPATH
         value: "/shared/packages:$PYTHONPATH"

     # Add initContainers (replace initContainers: [])
     initContainers:
       - name: install-amazon-provider-s3fs
         image: images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.12-astro-agent-1.0.2
         command: 
           - "pip"
           - "install" 
           - "--target"
           - "/shared/packages"
           - "--prefer-binary"
           - "--constraint"
           - "/constraints/constraints.txt"
           - "apache-airflow-providers-amazon[s3fs]==9.9.0"
           - "apache-airflow-providers-common-io==1.6.1"
         volumeMounts:
           - name: shared-packages
             mountPath: "/shared/packages"
           - name: constraints
             mountPath: "/constraints"

     # Add volumes (replace volumes: [])
     volumes:
       - name: shared-packages
         emptyDir: {}
       - name: constraints
         configMap:
           name: constraints-configmap

     # Add volumeMounts (replace volumeMounts: [])
     volumeMounts:
       - name: shared-packages
         mountPath: /shared/packages

   # In values.yaml, under triggerer section
   triggerer:
     # ... other triggerer config ...

     # Add PYTHONPATH to triggerer env 
     env:
       - name: PYTHONPATH
         value: "/shared/packages:$PYTHONPATH"

     # Add initContainers (replace initContainers: [])
     initContainers:
       - name: install-amazon-provider-s3fs
         image: images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.0-4-python-3.12-astro-agent-1.0.2
         command: 
           - "pip"
           - "install" 
           - "--target"
           - "/shared/packages"
           - "--prefer-binary"
           - "--constraint"
           - "/constraints/constraints.txt"
           - "apache-airflow-providers-amazon[s3fs]==9.9.0"
           - "apache-airflow-providers-common-io==1.6.1"
         volumeMounts:
           - name: shared-packages
             mountPath: "/shared/packages"
           - name: constraints
             mountPath: "/constraints"

     # Add volumes (replace volumes: [])
     volumes:
       - name: shared-packages
         emptyDir: {}
       - name: constraints
         configMap:
           name: constraints-configmap

     # Add volumeMounts (replace volumeMounts: [])
     volumeMounts:
       - name: shared-packages
         mountPath: /shared/packages
   ```

9. Update the Helm chart with the new `values.yaml` file.

   ```bash wrap theme={null}
   helm upgrade astro-agent astronomer/astro-remote-execution-agent --namespace <your-namespace> --values values.yaml
   ```

10. Now your tasks have access to the secrets backend! You can store [connections](/docs/learn/connections) under `airflow/connections` and [variables](/docs/learn/airflow-variables) under `airflow/variables`.

## Step 9: (optional, AWS only) Configure logs in the Airflow UI

When using Remote Execution with a Deployment running on AWS and the Remote Execution Agent running on AWS, you can configure your task logs to be read from an S3 bucket using a [customer workload identity](/docs/astro/authorize-deployments-to-your-cloud#step-1-authorize-the-deployment-to-your-iam-role-aws).

1. Create a new IAM policy called `AirflowS3Access` and attach the following policy. Replace `<your-logging-bucket>` with the name of your logging bucket. Make sure to record the policy ARN `arn:aws:iam::<your-acccoun-id>:policy/AirflowS3Access` from the output of the command.

   ```bash wrap theme={null}
   aws iam create-policy \
   --policy-name AirflowS3Access \
   --policy-document file://my-airflow-s3-policy.json 
   ```

   This is the policy you need to create in the `my-airflow-s3-policy.json` file.

   ```json expandable wrap theme={null}
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Action": [
                   "s3:ListBucket"
               ],
               "Resource": [
                   "arn:aws:s3:::<your-logging-bucket>"
               ],
               "Effect": "Allow",
               "Sid": "ListObjectsInBucket"
           },
           {
               "Action": [
                   "s3:GetObject",
                   "s3:PutObject",
                   "s3:DeleteObject"
               ],
               "Resource": [
                   "arn:aws:s3:::<your-logging-bucket>/*"
               ],
               "Effect": "Allow",
               "Sid": "AllObjectActions"
           },
           {
               "Sid": "AssumeRole",
               "Effect": "Allow",
               "Action": "sts:AssumeRole",
               "Resource": "*"
           }
       ]
   }
   ```

2. Attach the `AirflowS3Access` policy to the `RemoteAgentsRole` role you created and add to the service account annotations in [Step 8](#step-8-optional-aws-only-use-a-secrets-backend). Replace `<your-account-id>` with your AWS account ID.

   ```bash wrap theme={null}
   aws iam attach-role-policy \
   --role-name RemoteAgentsRole \
   --policy-arn arn:aws:iam::<your-account-id>:policy/AirflowS3Access
   ```

3. Update the `commonEnv` section in your `values.yaml` file to configure the logs to be written to S3. Replace `<your-logging-bucket>` with the name of your logging bucket and `<your-deployment-id>` with the ID of your deployment.

   ```yaml wrap theme={null}
   commonEnv:
       # ...
     - name: AIRFLOW__LOGGING__REMOTE_LOGGING
       value: "True"
     - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
       value: "astro_aws_logging"
     - name: AIRFLOW_CONN_ASTRO_AWS_LOGGING
       value: "s3://" # means the credentials are fetched from IRSA
     - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
       value: "s3://<your-logging-bucket>/<your-deployment-id>"
     - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
       value: "astronomer.runtime.logging.logging_config"
     - name: ASTRONOMER_ENVIRONMENT
       value: "cloud"
   ```

4. Update the Helm chart with the new `values.yaml` file. Upon the next Dag run you should be able to see the logs in your S3 bucket.

5. To see the logs in the Airflow UI, you need to configure the Astro Deployment to use the S3 bucket for task logs. In the Astro UI, navigate to your Deployment and click the **Details** tab. Click **Edit** in the **Advanced** section.

   <Frame>
     <img alt="Astro UI showing where to configure the task logs." />
   </Frame>

   Select **Bucket Storage** in the **Task Logs** field and add the **Bucket URL** as `s3://<your-logging-bucket>/<your-deployment-id>`. Select **Customer Managed Identity** in the **Workload Identity for Bucket Storage** field and use your `RemoteAgentsRole` IAM role ARN for the **Workload Identity ARN** before running the provided bash script.

   <Frame>
     <img alt="Astro UI showing the task logs configuration." />
   </Frame>

6. Now you should be able to see the task logs in the Airflow UI.


# Using the BashOperator
Source: https://astronomer.io/docs/learn/bashoperator

Learn how to use the BashOperator to run bash commands and bash scripts. Review examples of how to run scripts in languages other than Python.

The [`BashOperator`](https://airflow.apache.org/registry/providers/standard#standard-bash-BashOperator) is one of the most commonly used operators in Airflow. It executes bash commands or a bash script from within your Airflow DAG.

In this guide you'll learn:

* When to use the `BashOperator`.
* How to use the `BashOperator` and `@task.bash` decorator.
* How to use the `BashOperator` including executing bash commands and bash scripts.
* How to run scripts in non-Python programming languages using the `BashOperator`.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow decorators. See [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).
* Basic bash commands. See the [Bash Reference Manual](https://www.gnu.org/software/bash/manual/bash.html).

## How to use the `BashOperator` and `@task.bash` decorator

The `BashOperator` is part of core Airflow and can be used to execute a single bash command, a set of bash commands, or a bash script ending in `.sh`. The `@task.bash` decorator can be used to create bash statements using Python functions and is available as of Airflow 2.9.

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.providers.standard.operators.bash import BashOperator 

  bash_task = BashOperator(
      task_id="bash_task",
      bash_command="echo $MY_VAR",
      env={"MY_VAR": "Hello World"}
  )
  ```
</details>

<details>
  <summary>Taskflow</summary>

  ```python wrap theme={null}
  from airflow.sdk import task

  @task.bash(env={"MY_VAR": "Hello World"})
  def bash_task():
      return "echo $MY_VAR"  # the returned string is executed as a bash command

  bash_task()
  ```
</details>

The following parameters can be provided to the operator and decorator:

* `bash_command`: Defines a single bash command, a set of commands, or a bash script to execute. This parameter is required.
* `env`: Defines environment variables in a dictionary for the bash process. By default, the defined dictionary overwrites all existing environment variables in your Airflow environment, including those not defined in the provided dictionary. To change this behavior, you can set the `append_env` parameter. If you leave this parameter blank, the `BashOperator` inherits the environment variables from your Airflow environment.
* `append_env`: Changes the behavior of the `env` parameter. If you set this to `True`, the environment variables you define in `env` are appended to existing environment variables instead of overwriting them. The default is `False`.
* `output_encoding`: Defines the output encoding of the bash command. The default is `utf-8`.
* `skip_on_exit_code`: Defines which bash exit code should cause the `BashOperator` to enter a `skipped` state. The default is `99`.
* `cwd`: Changes the working directory where the bash command is run. The default is `None` and the bash command runs in a temporary directory.

The behavior of a `BashOperator` task is based on the status of the bash shell:

* Tasks succeed if the whole shell exits with an exit code of 0.
* Tasks are skipped if the exit code is 99 (unless otherwise specified in `skip_exit_code`).
* Tasks fail in case of all other exit codes.

<Tip>
  If you expect a non-zero exit from a sub-command you can add the prefix `set -e;` to your bash command to make sure that the exit is captured as a task failure.
</Tip>

Both the `bash_command` and the `env` parameter can accept [Jinja templates](/docs/learn/templating). However, the input given through Jinja templates to `bash_command` isn't escaped or sanitized. If you are concerned about potentially harmful user input you can use the setup shown in the [`BashOperator` documentation](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/bash.html).

## When to use the `BashOperator`

The following are common use cases for the `BashOperator` and `@task.bash` decorator in Airflow DAGs:

* Creating and running bash commands based on complex Python logic.
* Running a single or multiple bash commands in your Airflow environment.
* Running a previously prepared bash script.
* Running scripts in a programming language other than Python.
* Running commands to initialize tools that lack specific operator support. For example [Soda Core](/docs/learn/soda-data-quality).

## Example: Using Python to create bash commands

You can use `@task.bash` to create bash statements using Python functions. This decorator is especially useful when you want to run bash commands based on complex Python logic, including inputs from upstream tasks. The following example demonstrates how to use the `@task.bash` decorator to conditionally run different bash commands based on the output of an upstream task.

```python expandable wrap theme={null}
from airflow.sdk import task

@task
def upstream_task():
    dog_owner_data = {
        "names": ["Trevor", "Grant", "Marcy", "Carly", "Philip"],
        "dogs": [1, 2, 2, 0, 4],
    }

    return dog_owner_data

@task.bash
def bash_task(dog_owner_data):
    names_of_dogless_people = []
    for name, dog in zip(dog_owner_data["names"], dog_owner_data["dogs"]):
        if dog < 1:
            names_of_dogless_people.append(name)

    if names_of_dogless_people:
        if len(names_of_dogless_people) == 1:
            # this bash command is executed if only one person has no dog
            return f'echo "{names_of_dogless_people[0]} urgently needs a dog!"'
        else:
            names_of_dogless_people_str = " and ".join(names_of_dogless_people)
            # this bash command is executed if more than one person has no dog
            return f'echo "{names_of_dogless_people_str} urgently need a dog!"'
    else:
        # this bash command is executed if everyone has at least one dog
        return f'echo "All good, everyone has at least one dog!"'

bash_task(dog_owner_data=upstream_task())
```

## Example: Execute two bash commands using one `BashOperator`

The `BashOperator` can execute any number of bash commands separated by `&&`.

In this example, you run two bash commands in a single task:

* `echo Hello $MY_NAME!` prints the environment variable `MY_NAME` to the console.
* `echo $A_LARGE_NUMBER | rev  2>&1 | tee $AIRFLOW_HOME/include/my_secret_number.txt` takes the environment variable `A_LARGE_NUMBER`, pipes it to the `rev` command which reverses any input, and saves the result in a file called `my_secret_number.txt` located in the `/include` directory. The reversed number will also be printed to the console.

The second command uses an environment variable from the Airflow environment, `AIRFLOW_HOME`. This is only possible because `append_env` is set to `True`.

```python wrap theme={null}
from airflow.decorators import dag
from airflow.operators.bash import BashOperator
from pendulum import datetime


@dag(start_date=datetime(2022, 8, 1), schedule=None, catchup=False)
def bash_two_commands_example_dag():
    say_hello_and_create_a_secret_number = BashOperator(
        task_id="say_hello_and_create_a_secret_number",
        bash_command="echo Hello $MY_NAME! && echo $A_LARGE_NUMBER | rev  2>&1\
                     | tee $AIRFLOW_HOME/include/my_secret_number.txt",
        env={"MY_NAME": "<my name>", "A_LARGE_NUMBER": "231942"},
        append_env=True,
    )

    say_hello_and_create_a_secret_number


bash_two_commands_example_dag()
```

It is also possible to use two separate BashOperators to run the two commands, which can be useful if you want to assign different dependencies to the tasks.

## Example: Execute a bash script

The `BashOperator` can also be provided with a bash script (ending in `.sh`) to be executed.

For this example, you run a bash script which iterates over all files in the `/include` folder and prints their names to the console.

```bash wrap theme={null}
#!/bin/bash

echo "The script is starting!"
echo "The current user is $(whoami)"
files = $AIRFLOW_HOME/include/*

for file in $files
do
    echo "The include folder contains $(basename $file)"
done

echo "The script has run. Have an amazing day!"
```

Make sure that your bash script (`my_bash_script.sh` in this example) is available to your Airflow environment. If you use the Astro CLI, you can make this file accessible to Airflow by placing it in the `/include` directory of your Astro project.

It is important to make the bash script executable by running the following command before making the script available to your Airflow environment:

```bash wrap theme={null}
chmod +x my_bash_script.sh
```

If you use the Astro CLI, you can run this command before running `astro dev start`, or you can add the command to your project's Dockerfile with the following `RUN` command:

```docker wrap theme={null}
RUN chmod +x /usr/local/airflow/include/my_bash_script.sh
```

Astronomer recommends running this command in your Dockerfile for production builds such as Astro Deployments or in production CI/CD pipelines.

After making the script available to Airflow, you only have to provide the path to the script in the `bash_command` parameter. Be sure to add a space character at the end of the filepath, or else the task will fail with a Jinja exception!

```python wrap theme={null}
from airflow.decorators import dag
from airflow.operators.bash import BashOperator
from pendulum import datetime


@dag(start_date=datetime(2022, 8, 1), schedule=None, catchup=False)
def bash_script_example_dag():
    execute_my_script = BashOperator(
        task_id="execute_my_script",
        # Note the space at the end of the command!
        bash_command="$AIRFLOW_HOME/include/my_bash_script.sh ",
        # since the env argument is not specified, this instance of the
        # BashOperator has access to the environment variables of the Airflow
        # instance like AIRFLOW_HOME
    )

    execute_my_script


bash_script_example_dag()
```

## Example: Run a script in another programming language

Using the `BashOperator` is a straightforward way to run a script in a non-Python programming language in Airflow. You can run a script in any language that can be run with a bash command.

In this example, you run some JavaScript to query a public API providing the [current location of the international Space Station](http://open-notify.org/Open-Notify-API/ISS-Location-Now/). The query result is pushed to XCom so that a second task can extract the latitude and longitude information in a script written in R and print the data to the console.

The following setup is required:

* Install the JavaScript and R language packages at the OS level.
* Write a JavaScript file.
* Write a R script file.
* Make the scripts available to the Airflow environment.
* Execute the files from within a DAG using the `BashOperator`.

If you use the Astro CLI, the programming language packages can be installed at the OS level by adding them to the `packages.txt` file of your Astro project.

```text wrap theme={null}
r-base
nodejs
```

The following JavaScript file contains code for sending a GET request to the `/iss-now` path at `api.open-notify.org` and returning the results to `stdout`, which will both be printed to the console and pushed to XCom by the `BashOperator`.

```javascript wrap theme={null}
// specify that a http API is queried
const http = require('http');

// define the API to query
const options = {
  hostname: 'api.open-notify.org',
  port: 80,
  path: '/iss-now',
  method: 'GET',
};

const req = http.request(options, res => {
  // log the status code of the API response
  console.log(`statusCode: ${res.statusCode}`);

  // write the result of the GET request to stdout
  res.on('data', d => {
    process.stdout.write(d);
  });
});

// in case of an error print the error statement to the console
req.on('error', error => {
  console.error(error);
});

req.end();
```

The second task runs a script written in R that uses a regex to filter and print the longitude and latitude information from the API response.

```r wrap theme={null}
# Print outputs to the console
options(echo = TRUE)

# Read all trailing command line args 
myargs <- commandArgs(trailingOnly = TRUE)

# Reassemble them into a single JSON string
json_string <- paste(myargs, collapse = " ")

# Use regex to extract latitude & longitude 
latitude_str <- sub('.*"latitude"\\s*:\\s*"([^"]+)".*', '\\1', json_string)
longitude_str <- sub('.*"longitude"\\s*:\\s*"([^"]+)".*', '\\1', json_string)

latitude <- as.numeric(latitude_str)
longitude <- as.numeric(longitude_str)

sprintf("The current ISS location: lat: %s / long: %s.", latitude, longitude)
```

To run these scripts using the `BashOperator`, ensure that they are accessible to your Airflow environment. If you use the Astro CLI, you can place these files in the `/include` directory of your Astro project.

The DAG uses the `BashOperator` to execute both files defined above sequentially.

```python expandable wrap theme={null}
from airflow.decorators import dag
from airflow.operators.bash import BashOperator
from pendulum import datetime


@dag(
    dag_id="print_ISS_info_dag",
    start_date=datetime(2022, 8, 1),
    schedule=None,
    catchup=False,
)
def print_ISS_info_dag():
    # Use the node command to execute the JavaScript file from the command line
    get_ISS_coordinates = BashOperator(
        task_id="get_ISS_coordinates",
        bash_command="node $AIRFLOW_HOME/include/my_java_script.js",
    )

    # Use the Rscript command to execute the R file which is being provided
    # with the result from task one via an environment variable via XComs
    print_ISS_coordinates = BashOperator(
        task_id="print_ISS_coordinates",
        bash_command="Rscript $AIRFLOW_HOME/include/my_R_script.R $ISS_COORDINATES",
        env={
            "ISS_COORDINATES": "{{ task_instance.xcom_pull(\
                               task_ids='get_ISS_coordinates', \
                               key='return_value') }}"
        },
        # set append_env to True to be able to use env variables
        # like AIRFLOW_HOME from the Airflow environment
        append_env=True,
    )

    get_ISS_coordinates >> print_ISS_coordinates


print_ISS_info_dag()
```


# Blueprint: A Dag writing abstraction with a no-code interface in the Astro IDE
Source: https://astronomer.io/docs/learn/blueprint-overview

Learn about blueprint, a template-based system for composing Apache Airflow Dags from reusable Python templates and YAML configuration.

<Info>
  **Preview**

  The no-code UI for blueprint within the Astro IDE is in [Preview](/docs/astro/feature-previews).
</Info>

Blueprint is a template-based Dag authoring system built on the open-source [airflow-blueprint](https://github.com/astronomer/blueprint) package, which is compatible with any Airflow 3 environment. When using this package, data engineers define reusable *blueprints* in Python, and other team members can compose those blueprints into Dags, either through YAML configuration or in a no-code drag-and-drop interface in the [Astro IDE](/docs/astro/ide-overview).

<Frame>
  <img alt="Astro IDE Blueprint tab with the visual workflow graph, library panel, and node configuration." />
</Frame>

<CardGroup>
  <Card title="Write Blueprint templates tutorial" icon="code" href="/learn/blueprint-writer-tutorial">
    For data engineers and platform teams. Learn how to define reusable blueprints in Python and make them available to your team members in the [Astro IDE](/docs/astro/ide-overview).
  </Card>

  <Card title="Use Blueprint on Astro tutorial" icon="browser" href="/learn/blueprint-user-tutorial">
    For analysts, data scientists, and other team members, who prefer a no-code interface for defining workflows. Learn how to build pipelines from templates using the [Astro IDE](/docs/astro/ide-overview).
  </Card>
</CardGroup>

## What can a blueprint do?

A blueprint can contain any logic you can write in Python. If you are familiar with Airflow, you can think of a blueprint as a self-contained [task group](/docs/learn/task-groups) containing one or more tasks defined with [operators](/docs/learn/what-is-an-operator) or [decorators](/docs/learn/airflow-decorators). The blueprint author writes the logic in Python and end users assemble one or more blueprints into an Airflow Dag either using YAML or the drag-and-drop no-code interface in the [Astro IDE](/docs/astro/ide-overview).

Here are some examples of what you can do with a blueprint:

* Execute SQL queries against a data warehouse
* Run a [dbt](/docs/learn/airflow-dbt) project using [Cosmos](https://github.com/astronomer/astronomer-cosmos)
* Orchestrate an AI agent or large language model (LLM) call
* Add [human-in-the-loop](/docs/learn/airflow-human-in-the-loop) approval steps
* Run [data quality](/docs/learn/data-quality) checks on a table
* Execute a complex multi-step workflow while only exposing a few key configuration parameters

The blueprint author decides which configuration options to expose, for example, to hide Airflow internals like retries, or entire (sets of) operators, for example, performing clean up tasks after an ETL pipeline. End users see only the fields relevant to their use case, for example, the SQL query they'd like to execute or the prompt they want to give to an AI agent.

## When to use Blueprint

Blueprint is a good fit when:

* You want to encode workflow patterns in Python so analysts, data scientists, and other team members can compose and change Dags through YAML or the no-code UI without writing Python code.
* You have recurring pipeline patterns (extract-transform-load, model training, report generation) that differ only in configuration and benefit from standardization.


# How to use blueprint on Astro to write Apache Airflow® Dags in a no-code interface
Source: https://astronomer.io/docs/learn/blueprint-user-tutorial

Learn how to build data pipelines from pre-built templates using the Blueprint visual builder in the Astro IDE.

<Info>
  **Preview**

  The no-code UI for Blueprint on Astro is in [Preview](/docs/astro/feature-previews).
</Info>

Blueprint on Astro lets you build data pipelines from templates provided by your data engineering team. You browse the template library, configure each template using form fields, and connect them into a pipeline through a drag-and-drop interface in the [Astro IDE](/docs/astro/ide-overview).

<Frame>
  <img alt="Astro IDE Blueprint tab with the visual workflow graph, library panel, and node configuration." />
</Frame>

This tutorial walks you through exploring the [blueprint onboarding project](https://github.com/astronomer/templates) and adding a new ETL pipeline consisting of three templates in order to aggregate moon-merch sales revenue. No knowledge of Airflow or Python is required.

## Step 1a: Sign up for a free trial of Astro

If you don't have an Astro account yet, sign up for a [free trial of Astro](https://www.astronomer.io/lp/signup/), which gives access to the Astro IDE, including the blueprint interface. If you already have an Astro account, skip to [Step 1b](#step-1b-add-the-tutorial-project-to-the-astro-ide).

1. In the onboarding flow, after giving your Organization and Workspace names, select **Start with a template**.

<Frame>
  <img alt="Astro onboarding flow with the &#x22;Start with a template&#x22; option selected." />
</Frame>

2. Choose the **Blueprint** template and click **Continue**.

<Frame>
  <img alt="Astro onboarding flow with the &#x22;Blueprint&#x22; template selected." />
</Frame>

You enter the Astro IDE where the **Blueprint** tab opens the no-code interface to define your pipeline. Continue with [Step 2](#step-2-explore-an-existing-blueprint-pipeline).

<Frame>
  <img alt="Astro IDE with the Blueprint tab selected, the Dag list, and New DAG." />
</Frame>

## Step 1b: Add the tutorial project to the Astro IDE

If you already have an Astro account, you can add the blueprint tutorial project by navigating to the Astro IDE (1) and then clicking **Build DAGs visually with Blueprint templates**.

<Frame>
  <img alt="Astro workspace with Astro IDE in the sidebar and the Build DAGs visually with Blueprint templates card." />
</Frame>

## Step 2: Explore an existing blueprint pipeline

There are eight pre-existing pipelines in the onboarding project, each using one or more of 11 blueprint templates to accomplish different data engineering tasks.

1. Click one of the existing pipelines, for example `moon_missions_country_stats` to open the drag-and-drop view, which shows you the library of blueprint templates (1), the workflow graph (2), and the **DAG Properties** button (3) to modify the schedule (4) on which the pipeline should run.

<Frame>
  <img alt="Astro IDE Blueprint editor with the template library, workflow canvas, DAG Properties, and schedule field." />
</Frame>

2. If you click any nodes in the workflow graph, a panel opens on the right in which you can make changes to the configuration of the template. For example, you can change the `quality checks` task, which uses the `sql data quality check` blueprint to make sure the source data has at least 10 rows.

<Frame>
  <img alt="Astro IDE Blueprint canvas with the quality checks node selected and DAG Properties for min rows and column checks." />
</Frame>

## Step 3: Run the pipeline

To be able to run any pipeline in the Astro IDE, you need to start a test Deployment.

1. Click **Start Test Deployment** in the top right corner of the IDE. The test Deployment might take a few minutes to start.

<Frame>
  <img alt="Astro IDE with the Start Test Deployment button highlighted." />
</Frame>

2. Once the test Deployment has spun up, you can run the Dag. Click the **Test** tab (1), select the Dag from the dropdown menu (2), and click **Run DAG** (3) to start a run.

<Frame>
  <img alt="Astro IDE with the Test tab selected, the Dag dropdown menu, and the Run DAG button." />
</Frame>

3. Click **+ X TASKS** on any blueprint node to see the individual Airflow tasks contained in a blueprint as they complete. You can see the output of the task by clicking on it and then on **Task Logs** on the bar at the bottom of the Astro IDE.

<Frame>
  <img alt="Astro IDE Blueprint view with expanded tasks, a selected task, and Task Logs with tabular output." />
</Frame>

<Note>
  If you make any changes to a blueprint pipeline, you need to click **Sync to Test** to deploy your changes to the test Deployment before running the changed pipeline.
</Note>

## Step 4: Create a new pipeline

Now it is time to use Blueprint to build your own pipeline.

1. Back on the **Blueprint** tab, click **Blueprint** in the breadcrumb to return to the overview of all Blueprint Dags in this project.

<Frame>
  <img alt="Astro IDE Blueprint tab with the Blueprint breadcrumb, workflow canvas, library, and Task Logs panel." />
</Frame>

2. Click **+ New DAG**, give your Dag a name (an ID unique within the project, for example `my_first_blueprint_dag`) and click **Generate DAG**.

<Frame>
  <img alt="Astro IDE Blueprint Dag list with New DAG and the onboarding project Dags." />
</Frame>

3. Now you have an empty canvas. Add the first blueprint by dragging the **extract and aggregate** blueprint to the canvas and configure it in the form on the right. For example, you can change the `GRAIN` to `quarter` instead of `month`, which changes how the blueprint aggregates moon merch sales.

<Frame>
  <img alt="Astro IDE Blueprint canvas with extract and aggregate on the canvas, library, and configuration form with source CSV and grain." />
</Frame>

<Tip>
  If you want to change blueprints, either to change the form options, or get more blueprints entirely, see the [write blueprint templates](/docs/learn/blueprint-writer-tutorial) tutorial. Any action that can be defined in Python code can be part of a blueprint.
</Tip>

4. Next, add a second blueprint to the pipeline to perform a data quality check. Drag the **row count check** blueprint from the library into the canvas and enter the minimum number of rows you are expecting. At a quarterly grain for one year of data, that would be 4.

<Frame>
  <img alt="Astro IDE Blueprint canvas with row count check selected, library, and min rows in the configuration form." />
</Frame>

5. You can set the dependency between the blueprints by hovering on the bottom edge of the **extract and aggregate** node, clicking and then dragging your cursor to the bottom edge of the **row count check** node. A blue line appears to indicate that the aggregation blueprint needs to run before the quality check.

<Frame>
  <img alt="Screen recording of drawing a dependency edge from extract and aggregate to row count check on the Blueprint canvas." />
</Frame>

6. Lastly, add a third blueprint that prints the results after the data quality check passes. Drag the **print results** blueprint from the library (1) into the canvas, add a dependency (2) and fill in the source variable field using `merch_revenue_by_period`, the target variable of the first blueprint.

<Frame>
  <img alt="Astro IDE Blueprint canvas with print results, three connected nodes, library, and source variable in the configuration form." />
</Frame>

7. Click **Sync to Test** to deploy your changes to the test Deployment. After the sync process has finished you can run your pipeline!

<Tip>
  The test Deployment is a fully functional Airflow environment. You can access the regular [Airflow UI](/docs/learn/airflow-ui) of your Deployment by clicking on the dropdown arrow next to the **Sync to Test** button and selecting **Open Airflow**.

  <Frame>
    <img alt="Astro IDE with the Sync to Test menu open showing Test Deployment Details, Open Airflow, and Stop Test Deployment." />
  </Frame>
</Tip>

## Conclusion

Congratulations! You created an Airflow Dag processing data, performing a data quality check, and printing the results, without writing any Python code! A good next step is to send the [How to write blueprint templates tutorial](/docs/learn/blueprint-writer-tutorial) to your data engineering team to write more blueprints for you to use.


# How to write blueprint templates
Source: https://astronomer.io/docs/learn/blueprint-writer-tutorial

Learn how to define reusable blueprint templates in Python and compose them into Apache Airflow Dags using YAML.

The open-source [Blueprint](https://github.com/astronomer/blueprint) package lets data engineers define reusable Dag building blocks called *blueprints* in Python. Each blueprint wraps an [Airflow task group](/docs/learn/task-groups) containing one or more Airflow operators, decorators, or nested task groups into a configurable template that other team members can use without needing to write Airflow code.

Team members who don't know Airflow can create [Dags](/docs/learn/dags) by chaining blueprints together either [using YAML](#step-5-write-a-dag-using-the-blueprint-with-yaml) or the no-code interface in the [Astro IDE](/docs/learn/blueprint-user-tutorial).

In this tutorial, you'll learn how to create new blueprints for your team from scratch.

## Assumed knowledge

To get the most out of this tutorial, you should have an understanding of:

* Basic knowledge of [Python](https://docs.python.org/3/tutorial/index.html).
* How to write [Airflow Dags](/docs/learn/dags) in Python.
* [Airflow task groups](/docs/learn/task-groups) and [Airflow operators](/docs/learn/what-is-an-operator).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli) using at least version 1.40.

## Step 1: Set up the project

1. Create a new Astro project. Delete the `dags/example_astronauts.py` file.

   ```bash wrap theme={null}
   mkdir blueprint-tutorial && cd blueprint-tutorial
   astro dev init
   ```

2. Add the [blueprint package](https://github.com/astronomer/blueprint) to your `requirements.txt` file. Make sure to pin the latest version.

   ```text wrap theme={null}
   airflow-blueprint==<version>
   ```

## Step 2: Write a template class

A blueprint template is a Python class that inherits from the `Blueprint` class and defines a `render()` method. The `render()` method returns an Airflow `TaskGroup` or a single operator.

1. In your Dags folder, create a subdirectory called `templates` with one file `math_etl.py` and add the following scaffolding code.

   ```python title="dags/templates/math_etl.py" wrap theme={null}
   from airflow.sdk import TaskGroup
   from blueprint import BaseModel, Blueprint, Field

   class MyMathETLConfig(BaseModel):
       my_string_config: str = Field(
           default="",
           description="",
       )


   class MyMathETLBlueprint(Blueprint[MyMathETLConfig]):

       def render(self, config: MyMathETLConfig) -> TaskGroup:
           pass
   ```

   The `MyMathETLConfig` class contains the definition of each configuration `Field` that is available to the end user using the template in a Dag.

   The template class `MyMathETLBlueprint` inherits from `Blueprint[MyMathETLConfig]`, which ties the blueprint to that configuration model. The class's `render()` method returns a `TaskGroup` that contains the tasks to be executed when the blueprint is used in a Dag.

2. Fill the `MyMathETLConfig` class with two fields: `my_number` and `my_name`.

   ```python title="dags/templates/math_etl.py" wrap theme={null}
   from blueprint import BaseModel, Field

   class MyMathETLConfig(BaseModel):
       my_number: int = Field(
           default=2,
           description="Number to multiply the source number by",
       )

       my_name: str = Field(
           default="Rémy",
           description="Name to print",
       )
   ```

3. Add a `TaskGroup` to the `render()` method that contains three tasks: `extract`, `multiply` and `print`. Make sure the `render()` method returns the task group object. Note how you can access the configs provided by the end user inside the blueprint template by using `config.my_number` and `config.my_name`.

   ```python title="dags/templates/math_etl.py" expandable wrap theme={null}
   from airflow.sdk import TaskGroup, chain

   from blueprint import BaseModel, Blueprint, Field
   from airflow.providers.standard.operators.bash import BashOperator
   from airflow.providers.standard.operators.python import PythonOperator


   def extract_data_function():
       import random

       return {"my_source_number": random.randint(1, 100)}


   def multiply_by_x_function(x: int, input_data: dict) -> dict:
       result = input_data["my_source_number"] * x
       return {"my_result": result}


   class MyMathETLConfig(BaseModel):
       my_number: int = Field(
           default=2,
           description="Number to multiply the source number by",
       )

       my_name: str = Field(
           default="Rémy",
           description="Name to print",
       )


   class MyMathETLBlueprint(Blueprint[MyMathETLConfig]):

       def render(self, config: MyMathETLConfig) -> TaskGroup:
           with TaskGroup(group_id=self.step_id) as group:
               _extract = PythonOperator(
                   task_id="extract",
                   python_callable=extract_data_function,
               )
               _multiply = PythonOperator(
                   task_id="multiply",
                   python_callable=multiply_by_x_function,
                   op_kwargs={"x": config.my_number, "input_data": _extract.output},
               )
               _print_result = BashOperator(
                   task_id="print_result",
                   bash_command=(
                       f"echo 'Hello {config.my_name}! The result is "
                       "{{ task_instance.xcom_pull(task_ids='my_math_etl.multiply') }}'"
                   ),
               )

               chain(_extract, _multiply, _print_result)
           return group

   ```

## Step 3: Generate the blueprint schema JSON

If your end users are using the [Astro IDE](/docs/learn/blueprint-user-tutorial) to create blueprint Dags, you need to generate a JSON schema file that describes the blueprint configuration model. This file is used by the Astro IDE to validate the configuration fields and provide a visual interface for the end user to configure the blueprint.

1. Create a new folder at the root of your project called `blueprint` and create a subfolder called `generated-schemas`.

   ```bash wrap theme={null}
   mkdir -p blueprint/generated-schemas
   ```

2. Run the following command to generate the blueprint schema JSON file for your template.

   ```bash wrap theme={null}
   uvx --from airflow-blueprint blueprint schema my_math_etl_blueprint -o blueprint/generated-schemas/my_math_etl_blueprint.schema.json
   ```

<Accordion title="Generated schema file">
  ```json title="blueprint/generated-schemas/my_math_etl_blueprint.schema.json" expandable wrap theme={null}
  {                                                                                        
    "properties": {                                                                        
      "my_number": {                                                                       
        "default": 2,                                                                      
        "description": "Number to multiply the source number by",                          
        "title": "My Number",                                                              
        "type": "integer"                                                                  
      },                                                                                   
      "my_name": {                                                                         
        "default": "R\u00e9my",                                                            
        "description": "Name to print",                                                    
        "title": "My Name",                                                                
        "type": "string"                                                                   
      },                                                                                   
      "blueprint": {                                                                       
        "type": "string",                                                                  
        "const": "my_math_etl_blueprint",                                                  
        "description": "The blueprint template to use"                                     
      },                                                                                   
      "version": {                                                                         
        "type": "integer",                                                                 
        "const": 1,                                                                        
        "description": "The blueprint version"                                             
      }                                                                                    
    },                                                                                     
    "title": "MyMathETLBlueprint",                                                         
    "type": "object",                                                                      
    "required": [                                                                          
      "blueprint",                                                                         
      "version"                                                                            
    ],                                                                                     
    "$schema": "http://json-schema.org/draft-07/schema#"                                   
  }                                                                                        
  ```
</Accordion>

Once the schema file is present in the `blueprint/generated-schemas` directory, importing this Astro project into the [Astro IDE](/docs/astro/ide-overview) will automatically generate an entry in the `Library` of the blueprint interface, for users to build Dags using [drag-and-drop](/docs/learn/blueprint-user-tutorial). Users can drag the blueprint node (1) to the canvas and configure all input fields in the form to the right (2).

<Frame>
  <img alt="Astro IDE Blueprint with MyMathETLBlueprint in the library and My Number and My Name in the configuration form." />
</Frame>

## Step 4: Add a Dag loader file

When you [create a Dag using blueprint in the Astro IDE](/docs/learn/blueprint-user-tutorial), the Astro IDE automatically creates a YAML file for the Dag. This YAML file references the blueprint using the `blueprint` key. To make Airflow aware of this Dag, you need to add the Dag loader file.

1. Create a new file in the `dags` folder called `loader.py` and add the following code. Note that for Airflow to parse the file, it needs to include either the string `airflow` or `dag` (case-insensitive). You can toggle this behavior by setting the [`[core].dag_discovery_safe_mode`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#core-dag-discovery-safe-mode) configuration to `False`.

   ```python title="dags/loader.py" wrap theme={null}
   """Register YAML-defined Dags with Airflow (see *.dag.yaml next to this file)."""

   from blueprint import build_all

   build_all()
   ```

   This function call discovers all `*.dag.yaml` files in the `dags` folder and resolves the referenced blueprints, validates configurations, and creates Dag objects that can be picked up by Airflow.

## Step 5: Write a Dag using the blueprint with YAML

Of course, you can also directly use blueprints in YAML without using the Astro IDE.

1. Create a new YAML file in the `dags` folder called `my_math_etl.dag.yaml` and add the following code. Note that the filename needs to end with `.dag.yaml` for the blueprint loader to pick it up by default.

   ```yaml title="dags/my_math_etl.dag.yaml" wrap theme={null}
   dag_id: my_math_etl
   schedule: "@daily"

   steps:
     my_math_etl:
       blueprint: my_math_etl_blueprint
       my_number: 23
       my_name: "Kathryn"
   ```

2. You can add as many blueprints within the `steps` key as you want. Dependencies are set using the `depends_on` key.

   ```yaml title="dags/my_math_etl.dag.yaml" wrap theme={null}
   dag_id: my_math_etl
   schedule: "@daily"

   steps:
     my_math_etl:
       blueprint: my_math_etl_blueprint
       my_number: 23
       my_name: "Kathryn"

     my_second_math_etl:
       blueprint: my_math_etl_blueprint
       my_number: 19
       my_name: "Dominik"
       depends_on:
         - my_math_etl
   ```

3. (Optional) You can test your blueprint Dag like any other Dag in a local Airflow environment. Start Airflow using `astro dev start` and run your Dag in the [Airflow UI](/docs/learn/airflow-ui).

<Tip>
  Every task generated by Blueprint includes two extra fields visible in the **Rendered Template** tab in the Airflow UI: `blueprint_step_config` (the resolved YAML configuration) and `blueprint_step_code` (the Python source of the blueprint class). You can use these fields to trace any task back to its configuration.
</Tip>

## (Optional) Step 6: Version a blueprint

As your blueprints evolve, you might need to introduce breaking changes to a configuration schema. Blueprint supports versioning so existing Dag YAML files continue to work while new ones can use the updated schema. You apply the same pattern to `MyMathETLBlueprint` when you publish a `MyMathETLBlueprintV2` (or later) class.

Each version is a separate Python class. The initial version uses a clean class name (implicitly version 1). Later versions add a `V{N}` suffix:

1. To add a second version of your blueprint, create a new class called `MyMathETLBlueprintV2` and make any changes to the contents that you want.

```python title="dags/templates/math_etl.py" wrap theme={null}
class MyMathETLBlueprint(Blueprint[MyMathETLConfig]):
    # ...

class MyMathETLBlueprintV2(Blueprint[MyMathETLConfig]):
    # ... 
```

2. To use the new version in your YAML, add the `version` key to the blueprint step.

```yaml title="dags/my_math_etl.dag.yaml" wrap theme={null}
  my_second_math_etl:
    blueprint: my_math_etl_blueprint
    my_number: 19
    my_name: "Dominik"
    version: 2
    depends_on: [my_math_etl]
```

## Conclusion

Congratulations! You created a blueprint template and used it to create a Dag using YAML. You can now create blueprints for common data engineering patterns and provide them in an Astro project for your team members to build Dags without writing Python code.


# Clean up the Airflow metadata database using Dags
Source: https://astronomer.io/docs/learn/cleanup-dag-tutorial

Learn how to remove unnecessary data from the Airflow metadata database by calling an Airflow Plugin from a Dag.

In addition to storing configurations about your Airflow environment, the Airflow [metadata database](https://docs.astronomer.io/learn/airflow-database) stores data about past and present task runs. Airflow never automatically removes metadata, so the longer you use it, the more task run data is stored in your metadata DB. Over a long enough time, this can result in a bloated metadata DB, which can affect performance across your Airflow environment.

When a table in the metadata DB is larger than 50GiB, you might start to experience degraded scheduler performance. This can result in:

* Slow task scheduling
* Slow dag parsing
* Gunicorn timing out when using the Celery executor
* Slower Airflow UI load times

The following tables in the database are at risk of becoming too large over time:

* `dag_run`
* `job`
* `log`
* `rendered_task_instance_fields`
* `task_instance`
* `task_state_store`
* `xcom`

To keep your Airflow environment running at optimal performance, you can clean the metadata DB using the Airflow CLI `airflow db clean` command. This command was created as a way to safely clean up your metadata DB without querying it directly.

In Airflow 3, this command can't be called from a Dag because tasks can no longer directly access the metadata DB. Instead you can expose Airflow's utility function used by the command through an HTTP API using an Airflow Plugin.
This tutorial describes how to implement the cleanup Dag and corresponding plugin in Airflow so that you can clean your database using the command directly from the Airflow UI.

<Danger>
  Even when using Airflow's DB clean utilities, deleting data from the metadata database can destroy important data. Read the [Warnings](#warnings) section carefully before implementing this tutorial Dag in any production Airflow environment.
</Danger>

## Warnings

Deleting data from the metadata database can be an extremely destructive action. If you delete data that future task runs depend on, it's difficult to restore the database to its previous state without interrupting your data pipelines. Before implementing the Dag in this tutorial, consider the following:

* ⚠️ When specifying the `clean_before_timestamp` value, use as old a date as possible. The older the deleted data, the less likely it is to affect your currently running Dags.
* ⚠️ For `task_state_store`, the cutoff applies to the `expires_at` column set at write time (controlled by `[state_store] default_retention_days`), not the row creation date. Rows written with `NEVER_EXPIRE` (`expires_at=NULL`) are never deleted regardless of the cutoff.
* ⚠️ The Dag in this tutorial isn't designed to keep archived tables. It drops the archived tables it created in the cleanup process by default using the `skip_archive=True` argument, and doesn't maintain any history. If the task fails (for example if it runs for longer than five minutes), the archive tables aren't cleared. By calling `drop_archived_tables` in the second task of the Dag, we ensure all archive tables are dropped even in the event of the first task failing.

## Prerequisites

* An Airflow project

  <Warning>
    This Dag has been designed and optimized for Airflow environments running on Astro. Consider adjusting the parameters and code if you're running the Dag in any other type of Airflow environment.
  </Warning>
* The [HTTP Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-http/stable/index.html) installed

## Step 1: Create your Dag and plugin

1. In your dags folder, create a file called `db_cleanup.py`.

2. Copy the following code into the Dag file.

   ```python expandable wrap theme={null}
   """A DB cleanup dag maintained by Astronomer."""

   from datetime import UTC, datetime, timedelta

   from airflow.cli.commands.db_command import all_tables
   from airflow.providers.http.hooks.http import HttpHook
   from airflow.sdk import Param, dag, task


   def get_tables() -> list[str]:
       tables = []

       for table in all_tables:
           # can't delete dag versions which may be older than corresponding task instances
           # in order to keep dag_version untouched we also need to ignore the dag table
           # https://github.com/apache/airflow/issues/56192
           if table in {
               "dag_version",
               "dag",
           }:
               continue
           tables.append(table)

       return tables


   @task
   def get_chunked_timestamps(**context) -> list[datetime]:
       from plugins.db_cleanup import OldestTimestampResponse

       http_conn_id = context["params"]["http_conn_id"]
       tables = context["params"]["tables"]
       batches = []

       response = HttpHook("GET", http_conn_id).run(
           "/db_cleanup/api/oldest_timestamp",
           data={"table_names": tables},
       )
       start_chunk_time = OldestTimestampResponse.model_validate_json(response.content).oldest_timestamp

       if start_chunk_time is not None:
           start_ts = start_chunk_time
           end_ts = datetime.fromisoformat(context["params"]["clean_before_timestamp"])
           batch_size_days = context["params"]["batch_size_days"]

           while start_ts < end_ts:
               batch_end = min(start_ts + timedelta(days=batch_size_days), end_ts)
               batches.append(batch_end)
               start_ts += timedelta(days=batch_size_days)
       return batches


   @task(map_index_template="ts {{ clean_before_timestamp }}")
   def db_cleanup(clean_before_timestamp: datetime, **context) -> None:
       context["clean_before_timestamp"] = clean_before_timestamp.isoformat()
       tables = context["params"]["tables"]
       http_conn_id = context["params"]["http_conn_id"]
       HttpHook("DELETE", http_conn_id).run(
           "/db_cleanup/api/records",
           params={
               "clean_before_timestamp": clean_before_timestamp.isoformat(),
               "dry_run": context["params"]["dry_run"],
               "skip_archive": True,
               "table_names": tables,
           },
       )


   @task(trigger_rule="all_done")
   def clean_archive_tables(**context) -> None:
       tables = context["params"]["tables"]
       http_conn_id = context["params"]["http_conn_id"]
       HttpHook("DELETE", http_conn_id).run(
           "/db_cleanup/api/archived",
           params={"table_names": tables},
       )


   @dag(
       schedule=None,
       catchup=False,
       description=__doc__,
       doc_md=__doc__,
       render_template_as_native_obj=True,
       max_active_tasks=1,
       max_active_runs=1,
       tags=["astronomer", "cleanup"],
       params={
           "clean_before_timestamp": Param(
               default=(datetime.now(tz=UTC) - timedelta(days=90)).isoformat(),
               type="string",
               format="date-time",
               description="Delete records older than this timestamp. Default is 90 days ago.",
           ),
           "tables": Param(
               default=get_tables(),
               type=["null", "array"],
               examples=get_tables(),
               description="List of tables to clean. Default is all tables.",
           ),
           "dry_run": Param(
               default=False,
               type="boolean",
               description="Show a summary of which tables would be deleted in the api-server logs without actually deleting the records. Default is False.",
           ),
           "batch_size_days": Param(
               default=7,
               type="integer",
               description="Number of days in each batch for the cleanup. Default is 7 days.",
           ),
           "http_conn_id": Param(
               default="http_default",
               type="string",
               description="The HTTP connection ID for calling the cleanup API. Default is 'http_default'.",
           ),
       },
   )
   def astronomer_db_cleanup():

       db_cleanup.expand(clean_before_timestamp=get_chunked_timestamps()) >> clean_archive_tables()


   astronomer_db_cleanup()
   ```

   Rather than running on a schedule, this Dag is triggered manually by default and includes params so that you're in full control over how you clean the metadata DB.

   It includes three tasks:

   * `get_chunked_timestamps`: creates a list of timestamps to process in batches.
   * `db_cleanup`: calls the `run_cleanup` utility.
   * `clean_archive_tables`: calls the `drop_archived_tables` utility.

   These three tasks run with params you specify at runtime. The params let you specify:

   * `clean_before_timestamp`: What age of data to delete. Any data that was created before the specified time will be deleted. The default is to delete all data older than 90 days.
   * `tables`: Which tables to delete data from. By default all tables supported by the DB cleanup utilities are included except for the `dag` and `dag_version` table.
   * `dry_run`: Whether to run the cleanup as a dry run, meaning that no data is deleted. The dag will instead return the SQL that would be executed based on other parameters you have specified. The default is to run the deletion without a dry run.
   * `batch_size_days`: What batch size to use in order to cleanup data in batches.
   * `http_conn_id`: Which HTTP connection to use for calling the API exposing the DB cleanup utilities.

3. In your plugins folder, create a file called `db_cleanup.py`.

4. Copy the following code into the plugin file.

   ```python expandable wrap theme={null}
   """A DB cleanup plugin maintained by Astronomer."""

   import logging
   import os
   from collections.abc import Generator
   from datetime import datetime
   from typing import Annotated

   import pendulum
   from airflow.api_fastapi.common.router import AirflowRouter
   from airflow.api_fastapi.core_api.security import requires_access_configuration
   from airflow.plugins_manager import AirflowPlugin
   from airflow.utils.db import reflect_tables
   from airflow.utils.db_cleanup import _effective_table_names, drop_archived_tables, run_cleanup
   from airflow.utils.session import create_session
   from fastapi import Depends, FastAPI, Query
   from pydantic import BaseModel
   from sqlalchemy import func
   from sqlalchemy.orm.session import Session


   def _get_session() -> Generator[Session, None]:
       with create_session() as session:
           yield session


   logger = logging.getLogger(__name__)


   class TableInfo(BaseModel):
       table_name: str
       row_estimate: int = 0
       table_bytes: int = 0
       index_bytes: int = 0
       toast_bytes: int = 0
       total_bytes: int = 0


   class InfoResponse(BaseModel):
       tables: list[TableInfo] = []


   class OldestTimestampResponse(BaseModel):
       oldest_timestamp: datetime | None = None


   api = AirflowRouter(
       tags=["DB API"],
       dependencies=[Depends(requires_access_configuration("GET"))],
   )


   @api.get("/info")
   def info(
       *,
       order_by: str = "total_bytes",
       order_desc: bool = True,
       session: Annotated[Session, Depends(_get_session)],
   ) -> InfoResponse:
       """
       Provides information about the size of tables in the metadata database.
       """
       if order_by not in {
           "table_name",
           "row_estimate",
           "table_bytes",
           "index_bytes",
           "toast_bytes",
           "total_bytes",
       }:
           raise ValueError(f"Invalid order_by value: {order_by}")
       query = f"""
       SELECT
           table_name,
           row_estimate,
           total_bytes - index_bytes - COALESCE(toast_bytes, 0) AS table_bytes,
           index_bytes,
           toast_bytes,
           total_bytes
       FROM (
           SELECT
               relname AS table_name,
               c.reltuples::int AS row_estimate,
               pg_indexes_size(c.oid) AS index_bytes,
               pg_total_relation_size(reltoastrelid) AS toast_bytes,
               pg_total_relation_size(c.oid) AS total_bytes
           FROM pg_class c
           LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
           WHERE relkind = 'r'
           AND nspname = :table_schema
       ) a
       ORDER BY {order_by} {"DESC" if order_desc else "ASC"};
       """
       table_schema = "public" if os.getenv("ASTRONOMER_ENVIRONMENT") == "local" else "airflow"

       result = session.execute(query, {"table_schema": table_schema})
       response = InfoResponse()

       for row in result:
           response.tables.append(TableInfo(**{k: v for k, v in row._mapping.items() if v is not None}))

       return response


   @api.get("/oldest_timestamp")
   def get_oldest_timestamp(
       *,
       table_names: Annotated[list[str] | None, Query()] = None,
       session: Annotated[Session, Depends(_get_session)],
   ) -> OldestTimestampResponse:
       oldest_timestamp_list = []
       existing_tables = reflect_tables(tables=None, session=session).tables
       _, effective_config_dict = _effective_table_names(table_names=table_names)
       for table_name, table_config in effective_config_dict.items():
           if table_name in existing_tables:
               orm_model = table_config.orm_model
               recency_column = table_config.recency_column
               oldest_execution_date = session.query(func.min(recency_column)).select_from(orm_model).scalar()
               if oldest_execution_date:
                   oldest_timestamp_list.append(oldest_execution_date)
               else:
                   logging.info("No data found for %s, skipping...", table_name)
           else:
               logging.warning("Table %s not found. Skipping.", table_name)

       response = OldestTimestampResponse()
       if oldest_timestamp_list:
           response.oldest_timestamp = min(oldest_timestamp_list)
       return response


   @api.delete("/records")
   def delete_records(
       *,
       clean_before_timestamp: datetime,
       table_names: Annotated[list[str] | None, Query()] = None,
       dry_run: bool = False,
       verbose: bool = False,
       skip_archive: bool = False,
       batch_size: int | None = None,
       session: Annotated[Session, Depends(_get_session)],
   ):
       # The `batch_size` argument to `run_cleanup` is only supported in Airflow 3.1
       # and later. Pass it through only when set so the plugin remains compatible
       # with Airflow 3.0.
       extra_kwargs = {"batch_size": batch_size} if batch_size is not None else {}
       run_cleanup(
           clean_before_timestamp=pendulum.instance(clean_before_timestamp),
           table_names=table_names,
           dry_run=dry_run,
           verbose=verbose,
           confirm=False,
           skip_archive=skip_archive,
           session=session,
           **extra_kwargs,
       )


   @api.delete("/archived")
   def delete_archived(
       *,
       table_names: Annotated[list[str] | None, Query()] = None,
       session: Annotated[Session, Depends(_get_session)],
   ):
       drop_archived_tables(
           table_names=table_names,
           needs_confirm=False,
           session=session,
       )


   app = FastAPI()
   app.include_router(api, prefix="/api")


   class AstronomerDBCleanupPlugin(AirflowPlugin):
       name = "AstronomerDBCleanupPlugin"
       fastapi_apps = [
           {
               "app": app,
               "url_prefix": "/db_cleanup",
               "name": "Astronomer DB Cleanup Plugin",
           }
       ]
   ```

   The plugin uses `requires_access_configuration("GET")` from Airflow's core API security module to restrict access to users with Airflow configuration access, which is equivalent to admin access. It exposes the following API endpoints:

   * `GET /db_cleanup/api/info`: Provide a list of tables with their corresponding sizes and row count estimates. This endpoint isn't used by the Dag, but can be useful to get insights into table sizes.
   * `GET /db_cleanup/api/oldest_timestamp`: Return the oldest timestamp for the tables to cleanup used for calculating batches.
   * `DELETE /db_cleanup/api/records`: Call the `run_cleanup` utility.
   * `DELETE /db_cleanup/api/archived`: Call the `drop_archived_tables` utility.

<Warning>
  Because the DB cleanup utilities are running on the api-server, the corresponding logs will show up in the api-server logs.
</Warning>

<Note>
  The `batch_size` query parameter on the `DELETE /db_cleanup/api/records` endpoint is forwarded to Airflow's `run_cleanup` utility, which only accepts `batch_size` in Airflow 3.1 and later. When using the API endpoint directly with Airflow 3.0, omit the parameter, otherwise the request fails with `TypeError: run_cleanup() got an unexpected keyword argument 'batch_size'`. The `db_cleanup` Dag doesn't pass `batch_size` and is compatible with Airflow 3.0 and 3.1+.
</Note>

## Step 2: Configure an HTTP connection

Add an HTTP connection used for calling the API endpoints.

* `host`: Set this to the deployment's URL. For example on Astro this would look like something like `https://cmls9yey09fpw01ncvse41m4n.4n.astronomer.run/dse41m4n`. When running locally in `astro dev` this should be set to `http://api-server:8080`.
* `extra`: If needed, set the authorization header. On Astro with an [API token](/docs/astro/api/v-1/overview#authentication) this would look something like `{"Authorization": "Bearer mytoken1234...abc1234"}`.

## Step 3: Practice running the Dag

In this step, run the Dag in a local Airflow environment to practice the workflow for cleaning metadata DB records. If you completed Step 1 in your production environment, you will need to repeat it here before starting your local Airflow project. Typically in a fresh local Airflow environment there isn't much to clean up. When completing this process in a production environment which has been running for a while, there are more historic records to cleanup.

1. Run `astro dev start` in your Astro project to start Airflow, then open the Airflow UI at `localhost:8080`.

2. Ensure the Airflow connection `http_default` with host `http://api-server:8080` is set.

   <Tip>
     Instead of creating an Airflow connection, you can also define it as an environment variable `AIRFLOW_CONN_HTTP_DEFAULT=http://api-server:8080` in your local `.env` file.
   </Tip>

3. In the Airflow UI, run the `astronomer_db_cleanup` Dag by clicking the play button and configure the following params:

   * `dry_run` is enabled
   * Choose an appropriate cutoff date for `clean_before_timestamp`

<Note>
  In some older 3.0 versions a bug causes malformed timestamps when using `date-time` param fields in the Airflow UI. If you are on Airflow 3.0 and encounter an error like `ValueError: Invalid isoformat string: '2026-03-06T13:19:00.000Z:00+00:00'` set your `clean_before_timestamp` directly in the Configuration JSON under **Advanced Options** instead of using the Run Parameter.
</Note>

4. Click **Trigger**.
5. In a local terminal run `astro dev logs --api-server -f` to show the api-server logs.
6. Check that the `run_cleanup` utility completed successfully. Note that if you created a new Astro project for this tutorial, the run won't show much data to be deleted.

You can now use this Dag to periodically clean data from the Airflow metadata DB as needed.


# Manage connections in Apache Airflow
Source: https://astronomer.io/docs/learn/connections

Learn how to set up, manage, and maintain different types of connections in Apache Airflow. Use example connection configurations as the basis for your own connections.

Connections in Airflow are sets of configurations used to connect with other tools in the data ecosystem. Because most hooks and operators rely on connections to send and retrieve data from external systems, understanding how to create and configure them is essential for running Airflow in a production environment.

In this guide you'll:

* Learn about Airflow connections.
* Learn how to define connections using the Airflow UI.
* Learn how to define connections using environment variables.
* Add sample Snowflake and Slack Webhook connections to a DAG.

<Info>
  For Astro customers, Astronomer recommends using the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager) to store connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).
</Info>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow hooks. See [Hooks 101](/docs/learn/what-is-a-hook).

## Airflow connection basics

An Airflow connection is a set of configurations that send requests to the API of an external tool. In most cases, a connection requires sign-in credentials or a private key to authenticate Airflow to the external tool.

Airflow connections can be created by using one of the following methods:

* The [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager), which is the recommended way for Astro customers to manage connections.
* The [Airflow UI](/docs/learn/airflow-ui).
* [Environment variables](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#environment-variables).
* The [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Connection).
* A [secrets backend](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html) (a system for managing secrets external to Airflow).
* The [Airflow CLI](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#connection-cli).
* The [`airflow_settings.yaml` file](/docs/cli/v1.43/develop-project#configure-airflow_settings-yaml-local-development-only) for Astro CLI users.

This guide focuses on adding connections using the Airflow UI and environment variables. For more in-depth information on configuring connections using other methods, see the [REST API reference](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Connection), [Managing Connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html) and [Secrets Backend](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html).

Each connection has a unique `conn_id` which can be provided to operators and [hooks](/docs/learn/what-is-a-hook) that require a connection.

To standardize connections, Airflow includes many different connection types. There are general connection types for connecting to large clouds, such as `aws_default` and `gcp_default`, as well as connection types for specific services like `azure_service_bus_default`.

Each connection type requires different configurations and values based on the service it's connecting to. There are a couple of ways to find the information you need to provide for a particular connection type:

* Open the relevant provider page in the [Airflow Registry](https://airflow.apache.org/registry/providers/) and go to the first link under **Helpful Links** to access the Apache Airflow documentation for the provider. Most commonly used providers will have documentation on each of their associated connection types. For example, you can find information on how to set up different connections to Azure in the [Azure provider docs](https://airflow.apache.org/registry/providers/microsoft-azure).
* Check the documentation of the external tool you are connecting to and see if it offers guidance on how to authenticate.
* Refer to the source code of the hook that is being used by your operator.

<Tip>
  If you use a mix of strategies for managing connections, it's important to understand that if the same connection is defined in multiple ways, Airflow uses the following order of precedence:

  1. Secrets Backend
  2. Astro Environment Manager
  3. Environment Variables
  4. Airflow's metadata database (Airflow UI)

  See [How Airflow finds connections](/docs/astro/manage-connections-variables#how-airflow-finds-connections) for more information.
</Tip>

## Defining connections in the Airflow UI

The most common way of defining a connection is using the Airflow UI. Go to **Admin** > **Connections** and then click **+ Add Connection**.

<Frame>
  <img alt="Create a new connection by clicking on Admin, Connections and Add connection." />
</Frame>

In the connection form, select a **Connection Type** from the dropdown list. The connection type determines which fields are available in the form, each connection type requires different kinds of information. More connection types appear in the dropdown list as you install more providers. For example, if you install the [Snowflake provider](https://airflow.apache.org/registry/providers/snowflake), a new connection type called `Snowflake` appears in the dropdown list. If no connection type is available for your connection you can always use the `generic` connection type.

<Frame>
  <img alt="Empty Connection form in the Airflow UI" />
</Frame>

You don't have to specify every field for most connections. However, the values marked as required in the Airflow UI can be misleading. For example, to set up a connection to a PostgreSQL database, you need to reference the [PostgreSQL provider documentation](https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/connections/postgres.html) to learn that the connection requires a `Host`, a user name as `login`, and a password in the `password` field. Most of the time you'll also need to provide your `Port`.

<Frame>
  <img alt="Example PostgreSQL connection" />
</Frame>

Any parameters that don't have specific fields in the connection form can be defined in the **Extra** field as a JSON dictionary. For example, you can add the `sslmode` or a client `sslkey` in the **Extra** field of your PostgreSQL connection.

## Define connections with environment variables

Connections can also be defined using environment variables. If you use the Astro CLI, you can use the `.env` file for local development or specify environment variables in your project's Dockerfile.

<Note>
  If you are synchronizing your project to a remote repository, don't save sensitive information in your Dockerfile. In this case, using either a secrets backend, Airflow connections defined in the UI, or `.env` locally are preferred to avoid exposing secrets in plain text.
</Note>

The environment variable used for the connection must be formatted as `AIRFLOW_CONN_YOURCONNID` and can be provided as a Uniform Resource Identifier (URI) or in JSON.

[URI](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier) is a format designed to contain all necessary connection information in one string, starting with the connection type, followed by `login`, `password`, and `host`. In many cases a specific port, schema, and additional parameters must be added.

```docker wrap theme={null}
# the general format of a URI connection that is defined in your Dockerfile
ENV AIRFLOW_CONN_MYCONNID='my-conn-type://login:password@host:port/schema?param1=val1&param2=val2'

# an example of a connection to snowflake defined as a URI
ENV AIRFLOW_CONN_SNOWFLAKE_CONN='snowflake://LOGIN:PASSWORD@/?account=xy12345&region=eu-central-1'
```

Connections can also be provided to an environment variable as a JSON dictionary:

```json wrap theme={null}
# example of a connection defined as a JSON file in your `.env` file
AIRFLOW_CONN_MYCONNID='{
    "conn_type": "my-conn-type",
    "login": "my-login",
    "password": "my-password",
    "host": "my-host",
    "port": 1234,
    "schema": "my-schema",
    "extra": {
        "param1": "val1",
        "param2": "val2"
    }
}'
```

Connections that are defined using environment variables don't appear in the list of available connections in the Airflow UI.

<Info>
  To store a connection in JSON as an Astro environment variable, remove all line breaks in your JSON object so that the value is a single, unbroken line. See [Add Airflow connections and variables using environment variables](/docs/astro/environment-variables)
</Info>

## Mask sensitive information

Connections often contain sensitive credentials. By default, Airflow hides the `password` field in the UI and in the Airflow logs. If `AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS` is set to `True`, values from the connection's `Extra` field are also hidden if their keys contain any of the words listed in `AIRFLOW__CORE__SENSITIVE_VAR_CONN_NAMES`. You can find more information on masking, including a list of the default values in this environment variable, in the Airflow documentation on [Masking sensitive data](https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/mask-sensitive-values.html).

## Test connections in the Airflow UI

You can enable testing of Airflow connections in the Airflow UI by setting the `AIRFLOW__CORE__TEST_CONNECTION` environment variable to `Enabled`.

<Frame>
  <img alt="Enable testing of Airflow connections in the Airflow UI" />
</Frame>

See [Testing connections](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#testing-connections) in the Airflow documentation for more information.

## Example: Configure the `SqlToSlackWebhookOperator`

In this example, you'll configure the `SqlToSlackWebhookOperator`, which requires connections to a SQL database (we will use Snowflake) and Slack. You'll define the connections using the Airflow UI.

Before starting Airflow, you need to install the [Snowflake](https://airflow.apache.org/registry/providers/snowflake) and the [Slack](https://airflow.apache.org/registry/providers/slack) providers. If you use the Astro CLI, you can install the packages by adding the following lines to your Astro project's `requirements.txt` file:

```text wrap theme={null}
apache-airflow-providers-snowflake>=6.2.0
apache-airflow-providers-slack>=9.0.3
```

Open the Airflow UI and create a new connection. Set the **Connection Type** to `Snowflake`. There are different ways to define a Snowflake connection, see [Create a Snowflake Connection in Airflow](/docs/learn/connections/snowflake) for more information.

Next you'll set up a connection to Slack. To post a message to a Slack channel, you need to create a Slack app for your server and configure incoming webhooks. See the [Slack Documentation](https://api.slack.com/messaging/webhooks) for setup steps.

To connect to Slack from Airflow, you need to provide the following parameters:

* Connection Id: `slack_conn` (or another string that hasn't been used for a different connection already)
* Connection Type: `Slack Webhook`
* Host: `https://hooks.slack.com.services`, which is the first part of your Webhook URL
* Password: The second part of your Webhook URL in the format `T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX`

The last step is writing the DAG using the `SqlToSlackWebhookOperator` to run a SQL query on a Snowflake table and post the result as a message to a Slack channel. The `SqlToSlackWebhookOperator` requires both the connection id for the Snowflake connection (`sql_conn_id`) and the connection id for the Slack connection (`slack_webhook_conn_id`).

```python wrap theme={null}
from airflow.decorators import dag
from pendulum import datetime
from airflow.providers.snowflake.transfers.snowflake_to_slack import (
    SnowflakeToSlackOperator,
)


@dag(start_date=datetime(2022, 7, 1), schedule=None, catchup=False)
def snowflake_to_slack_dag():
    transfer_task = SnowflakeToSlackOperator(
        task_id="transfer_task",
        # the two connections are passed to the operator here:
        snowflake_conn_id="snowflake_conn",
        slack_conn_id="slack_conn",
        params={"table_name": "ORDERS", "col_to_sum": "O_TOTALPRICE"},
        sql="""
            SELECT
              COUNT(*) AS row_count,
              SUM({{ params.col_to_sum }}) AS sum_price
            FROM {{ params.table_name }}
        """,
        slack_message="""The table {{ params.table_name }} has
            => {{ results_df.ROW_COUNT[0] }} entries
            => with a total price of {{results_df.SUM_PRICE[0]}}""",
    )

    transfer_task


snowflake_to_slack_dag()
```


# Create an Azure Blob Storage connection in Airflow
Source: https://astronomer.io/docs/learn/connections/azure-blob-storage

Learn how to create an Azure Blob Storage connection in Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview) provides the storage for all of your Azure Storage data objects, including blobs, files, queues, and tables. Integrating your Azure storage account with Airflow lets you perform different kind of operations on blob objects stored in the cloud. For example, you can create or delete a container, upload or read a blob, or download blobs using Airflow.

This guide explains how to set up an Azure Blob Storage connection using the **Azure Blob Storage** connection type. Astronomer recommends using this connection type because it utilizes the `wasb` protocol, which means you can connect with any Azure Storage account including Azure Data Lake Gen 1 and Azure Data Lake Gen 2.

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* A locally running [Astro project](/docs/cli/v1.43/get-started-cli).
* An [Azure storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal).
* [Permissions to access](https://learn.microsoft.com/en-us/azure/storage/blobs/assign-azure-role-data-access?tabs=portal) blob data from your local Airflow environment.

## Get connection details

To create an Azure Blob Storage connection in Airflow, you can use any of the following methods:

<details>
  <summary>Shared Access Key</summary>

  Microsoft generates two [shared access keys](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal) by default for every storage account. You can use them to give Airflow access to the data in your storage account.

  An Azure Blob Storage connection using a shared access key requires the following information:

  * Name of the storage account
  * Shared access key

  Complete the following steps to retrieve these values:

  1. In your Azure portal, open your storage account.
  2. Copy the name of your storage account.
  3. Follow [Microsoft documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal#view-account-access-keys) to copy the storage account **Key**.
</details>

<details>
  <summary>Connection String</summary>

  A [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string) for a storage account includes the authorization information required to access data in your storage account.

  An Azure blob storage connection using connection string requires the following information:

  * Storage account name
  * Storage account connection string

  Complete the following steps to retrieve these values:

  1. In your Azure portal, open your storage account.
  2. Copy the name of your storage account.
  3. Follow [Microsoft documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-get-info?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json\&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json\&tabs=portal#get-a-connection-string-for-the-storage-account) to copy the **Connection string**.
</details>

<details>
  <summary>SAS Token</summary>

  A [shared access signature (SAS) token](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview) provides granular access for a storage account.

  An Azure blob storage connection using SAS token requires the following information:

  * Storage account name
  * SAS token

  Complete the following steps to retrieve these values:

  1. In your Azure portal, navigate to your Storage account view and select your subscription.
  2. Copy the name of your storage account.
  3. Follow the [Microsoft documentation](https://learn.microsoft.com/en-us/azure/cognitive-services/translator/document-translation/how-to-guides/create-sas-tokens?tabs=Containers#create-sas-tokens-in-the-azure-portal) to generate your SAS token. Copy the SAS token.
</details>

<details>
  <summary>Azure App Service Principal</summary>

  A [service principal for an Azure app](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview) provides granular access for a storage account.

  An Azure Blob Storage connection using a service principal requires the following information:

  * Storage account URL
  * Application Client ID
  * Tenant ID
  * Client secret

  Complete the following steps to retrieve these values:

  1. In your Azure portal, open your storage account.
  2. Follow [Azure documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-get-info?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json\&bc=%2Fazure%2Fstorage%2Fblobs%2Fbreadcrumb%2Ftoc.json\&tabs=portal#get-service-endpoints-for-the-storage-account) to copy your **Blob Service URL**. It should be in the format `https://mystorageaccount.blob.core.windows.net/`.
  3. Open your Microsoft Entra ID application. Then, from the **Overview** tab, copy the **Application (client) ID** and **Directory (tenant) ID**.
  4. [Create a new client secret](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#option-3-create-a-new-application-secret) for your application to be used in the Airflow connection. Copy the **VALUE** of the client secret that appears.
  5. [Assign](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal#assign-a-role-to-the-application) the [Storage Blob Data Contributor](https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) role to your app so that Airflow can access blob objects in your storage account.
</details>

## Create your connection

<Info>
  Astro users can also create connections using the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager), which stores connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).
</Info>

<details>
  <summary>Shared Access Key</summary>

  1. Open your Astro project and add the following line to your `requirements.txt` file:

     ```text wrap theme={null}
     apache-airflow-providers-microsoft-azure
     ```

     This installs the Microsoft Azure provider package, which makes the Azure Blob Storage connection type available in Airflow.

  2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

  3. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then choose the **Azure Blob Storage** connection type.

  4. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

     * **Connection Id**: Enter a name for the connection.
     * **Blob Storage Login**: Enter your storage account name.
     * **Blob Storage Key**: Enter your storage account **Key**.

  5. Click **Test**. After the connection test succeeds, click **Save**.

     <Frame>
       <img alt="azure-connection-storage-access-key" />
     </Frame>
</details>

<details>
  <summary>Connection String</summary>

  1. Open your Astro project and add the following line to your `requirements.txt` file:

     ```text wrap theme={null}
     apache-airflow-providers-microsoft-azure
     ```

     This installs the Microsoft Azure provider package, which makes the Azure Blob Storage connection type available in Airflow.

  2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

  3. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then choose the **Azure Blob Storage** connection type.

  4. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

     * **Connection Id**: Enter a name for the connection.
     * **Blob Storage Connection String**: Enter your storage account connection string.

  5. Click **Test**. After the connection test succeeds, click **Save**.

     <Frame>
       <img alt="azure-connection-storage-conn-string" />
     </Frame>

  <Tip>
    If you want, you can replace the value in **Blob Storage Connection String** with the connection string for an SAS token.
  </Tip>
</details>

<details>
  <summary>SAS Token</summary>

  1. Open your Astro project and add the following line to your `requirements.txt` file:

     ```text wrap theme={null}
     apache-airflow-providers-microsoft-azure
     ```

     This installs the Microsoft Azure provider package, which makes the Azure Blob Storage connection type available in Airflow.

  2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

  3. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then choose the **Azure Blob Storage** connection type.

  4. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

     * **Connection Id**: Enter a name for the connection.
     * **Blob Storage Login**: Enter the name of your storage account.
     * **SAS Token**: Enter your SAS token.

  5. Click **Test**. After the connection test succeeds, click **Save**.

     <Frame>
       <img alt="azure-connection-storage-sas-token" />
     </Frame>
</details>

<details>
  <summary>Azure App Service Principal</summary>

  1. Open your Astro project and add the following line to your `requirements.txt` file:

     ```text wrap theme={null}
     apache-airflow-providers-microsoft-azure
     ```

     This installs the Microsoft Azure provider package, which makes the Azure Blob Storage connection type available in Airflow.

  2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

  3. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then choose the **Azure Blob Storage** connection type.

  4. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

     * **Connection Id**: Enter a name for the connection.
     * **Account Name**: Enter **Blob Service URL** for your storage account.
     * **Blob Storage Login**: Enter your **Application (client) ID**.
     * **Blob Storage Key**: Enter your client secret **Value**.
     * **Tenant Id**: Enter your **Directory (tenant) ID**.

  5. Click **Test**. After the connection test succeeds, click **Save**.

     <Frame>
       <img alt="azure-blob-storage-app-secret" />
     </Frame>
</details>

## How it works

Airflow uses the [Azure SDK for Python](https://github.com/Azure/azure-sdk-for-python) to connect to Azure services through the [`WasbHook`](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/_api/airflow/providers/microsoft/azure/hooks/wasb/index.html).

## See also

* [Apache Airflow Microsoft Azure provider package documentation](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/connections/wasb.html).
* Azure blob storage modules in the [Airflow Registry](https://airflow.apache.org/registry/).
* [Import and export Airflow connections using Astro CLI](/docs/astro/import-export-connections-variables#from-environment-variables).


# Create a BigQuery connection in Airflow
Source: https://astronomer.io/docs/learn/connections/bigquery

Learn how to create a BigQuery connection in Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[BigQuery](https://cloud.google.com/bigquery) is Google's fully managed and serverless data warehouse. Integrating BigQuery with Airflow lets you execute BigQuery jobs from a DAG.

There are multiple ways to connect Airflow and BigQuery, all of which require a [GCP Service Account](https://cloud.google.com/docs/authentication#service-accounts):

* Use the contents of a service account key file directly in an Airflow connection.
* Copy the service account key file to your Airflow project.
* Store the contents of a service account key file in a secrets backend.
* Use a Kubernetes service account to integrate Airflow and BigQuery. This is possible only if you run Airflow on Astro or Google Kubernetes Engine (GKE).

Using a Kubernetes service account is the most secure method because it doesn't require storing a secret in Airflow's metadata database, on disk, or in a secrets backend. The next most secure connection method is to store the contents of your service account key file in a secrets backend.

<Tip>
  If you're an Astro user, Astronomer recommends using workload identity to authorize your Deployments to BigQuery. This eliminates the need to specify secrets in your Airflow connections or copying credentials file to your Astro project. See [Authorize Deployments to your cloud](/docs/astro/authorize-deployments-to-your-cloud).
</Tip>

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* A locally running [Astro project](/docs/cli/v1.43/get-started-cli).
* A Google Cloud project with [BigQuery API](https://cloud.google.com/bigquery/docs/enable-transfer-service#enable-api) enabled.
* Permissions to create an IAM service account or use an existing one. See [Google documentation](https://cloud.google.com/iam/docs/manage-access-service-accounts).

## Get connection details

A connection from Airflow to Google BigQuery requires the following information:

* Service account name
* Service account key file
* Google Cloud Project ID

Complete one of the following sets of steps to retrieve these values:

<details>
  <summary>Key File Value</summary>

  This method requires you to save the contents of your service account key file in your Airflow connection.

  1. In your Google Cloud console, select your Google Cloud project and copy its **ID**.
  2. [Create a new service account](https://cloud.google.com/iam/docs/service-accounts-create).
  3. [Grant roles](https://cloud.google.com/iam/docs/grant-role-console) to your service account so that it can access BigQuery. See [BigQuery roles](https://cloud.google.com/bigquery/docs/access-control#bigquery) for a list of available roles and the permissions.
  4. [Add a new JSON key file](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) to the service account.
  5. Copy the contents of the key file.
</details>

<details>
  <summary>Key File In Container</summary>

  This method requires you to mount your service account key file to your Airflow containers.

  1. In your Google Cloud console, select your Google Cloud project and copy its **ID**.
  2. [Create a new service account](https://cloud.google.com/iam/docs/service-accounts-create).
  3. [Grant roles](https://cloud.google.com/iam/docs/grant-role-console) to your service account so that it can access BigQuery. See [BigQuery roles](https://cloud.google.com/bigquery/docs/access-control#bigquery) for a list of available roles and the permissions.
  4. [Add a new JSON key file](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) to the service account.
  5. Download the key file.
</details>

<details>
  <summary>Key File In Secrets Backend</summary>

  You can save your service account key file to any secrets backend. See [Configure a secrets backend](/docs/astro/secrets-backend) for steps on how to configure several popular secrets backend services to use with Airflow on Astro. For example, if you use Google Secret Manager as a secrets backend:

  1. In the Google Cloud console, select your Google Cloud project and copy its **ID**.
  2. [Create a new service account](https://cloud.google.com/iam/docs/service-accounts-create).
  3. [Grant roles](https://cloud.google.com/iam/docs/grant-role-console) to your service account so that it can access BigQuery. See [BigQuery roles](https://cloud.google.com/bigquery/docs/access-control#bigquery) for a list of available roles and the permissions.
  4. [Add a new JSON key file](https://cloud.google.com/iam/docs/keys-create-delete#iam-service-account-keys-create-console) to the service account.
  5. Download the key file.
  6. [Create a secret](https://cloud.google.com/secret-manager/docs/create-secret-quickstart) in Google Secret Manager and upload the key file from Step 5 as a secret value. Then, copy the ID of your secret name.
  7. Follow Astronomer's documentation to [configure secrets backend](/docs/astro/secrets-backend) for your Astro project.

  You can now use this secret in your Airflow connections.
</details>

<details>
  <summary>Kubernetes Service Account</summary>

  A [Kubernetes service account](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/) provides an identity to the processes running in a Pod. The process running inside a Pod can use this identity of its associated service account to authenticate to the cluster's API server. This is also referred to as Workload Identity in [GCP](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity) and [Azure](https://learn.microsoft.com/en-us/azure/aks/learn/tutorial-kubernetes-workload-identity).

  This method can't be used in a local Airflow environment. It is available to use with Airflow on Astro or OSS Airflow running on Kubernetes clusters.

  If you're running Airflow in a GKE cluster, complete the following steps:

  1. In your Google Cloud console, open the Google Cloud project where you're running BigQuery and copy its **ID**.
  2. [Enable Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) and [configure Airflow to use workload identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to). Copy the name for the Kubernetes service account that you create.
  3. Go to **IAM**, then click **Service Accounts** and search for your Kubernetes service account. If you don't see your service account, click **+ ADD** to add your service account to your Google Cloud project.
  4. [Grant roles](https://cloud.google.com/iam/docs/grant-role-console) to your service account to access BigQuery. See [BigQuery roles](https://cloud.google.com/bigquery/docs/access-control#bigquery) for a list of available roles and the permissions.

  After you complete these steps, any Google Cloud connection you create in the Deployment will use your workload identity by default to access BigQuery.
</details>

## Create your connection

<Info>
  Astro users can also create connections using the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager), which stores connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).
</Info>

<details>
  <summary>Key File Value</summary>

  1. Open your Astro project and add the following line to your `requirements.txt` file:

     ```text wrap theme={null}
     apache-airflow-providers-google
     ```

     This installs the Google Cloud provider package, which makes the Google Cloud connection type available in Airflow.

  2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

  3. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then choose the **Google Cloud** connection type.

  4. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

     * **Connection Id**: Enter a name for the connection.
     * **Keyfile JSON**: Enter the contents of the key file.

  5. Click **Test**. After the connection test succeeds, click **Save**.

     <Frame>
       <img alt="GCP-connection-key-in-ui" />
     </Frame>
</details>

<details>
  <summary>Key File In Container</summary>

  1. Open your Astro project and add the following line to your `requirements.txt` file:

     ```text wrap theme={null}
     apache-airflow-providers-google
     ```

     This installs the Google Cloud provider package, which makes the Google Cloud connection type available in Airflow.

  2. Add the key file to your `include` folder. This will make it available to Airflow at `/usr/local/airflow/include/<your-key-file>.json`.

  3. Restart or start your local Airflow using `astro dev restart` to apply your changes in `requirements.txt`.

  4. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then choose the **Google Cloud** connection type.

  5. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

     * **Connection Id**: Enter a name for the connection.
     * **Keyfile Path**: Enter the path of your key file.

  6. Click **Test connection**. After the connection test succeeds, click **Save**.

     <Frame>
       <img alt="GCP-connection-key-in-airflow-container" />
     </Frame>
</details>

<details>
  <summary>Key File In Secrets Backend</summary>

  1. Open your Astro project and add the following line to your `requirements.txt` file:

     ```text wrap theme={null}
     apache-airflow-providers-google
     ```

     This will install the Google Cloud provider package, which makes the Google Cloud connection type available in Airflow.

  2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

  3. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then choose the **Google Cloud** connection type.

  4. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

     * **Connection Id**: Enter a name for the connection.
     * **Keyfile Secret Project Id**: Enter the **ID** of the Google Cloud project.
     * **Keyfile Secret Name**: Enter the ID of your secret name.

  5. Click **Test connection**. After the connection test succeeds, click **Save**.

     <Frame>
       <img alt="GCP-connection-key-in-secret-manager" />
     </Frame>
</details>

<details>
  <summary>Kubernetes Service Account</summary>

  1. Open your Airflow project and add the following line to your `requirements.txt` file:

     ```text wrap theme={null}
     apache-airflow-providers-google
     ```

     This will install the Google Cloud provider package, which makes the Google Cloud connection type available in Airflow.

  2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

  3. In your Airflow UI, go to **Admin** > **Connections**. Click the **+** sign to add a new connection, select the connection type as **Google Cloud**.

  4. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

     * **Connection Id**: Enter a name for the connection.
     * **Project Id**: Enter the **ID** of the Google Cloud project.

  5. Click **Test connection**. After the connection test succeeds, click **Save**.

     <Frame>
       <img alt="GCP-connection-using-workload-identity" />
     </Frame>
</details>

## How it works

Airflow uses the [`python-bigquery`](https://github.com/googleapis/python-bigquery) library to connect to GCP BigQuery through the [`BigQueryHook`](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/cloud/hooks/bigquery/index.html). If you don't define specific key credentials in the connection, Google defaults to using [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials). This means when you use Workload Identity to connect to BigQuery, Airflow relies on ADC to authenticate.

## See also

* [Apache Airflow Google provider package documentation](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html)
* BigQuery Modules in the [Airflow Registry](https://airflow.apache.org/registry/)
* [Import and export Airflow connections using the Astro CLI](/docs/astro/import-export-connections-variables#using-the-astro-cli-local-environments-only)


# Create a Databricks connection in Airflow
Source: https://astronomer.io/docs/learn/connections/databricks

Learn how to create a Databricks connection in Airflow.

[Databricks](https://databricks.com/) is a popular unified data and analytics platform built around [Apache Spark](https://spark.apache.org/) that provides users with fully managed Apache Spark clusters and interactive workspaces.

This guide provides the basic setup for creating a Databricks connection. For a complete integration tutorial, see [Orchestrate Databricks jobs with Airflow](/docs/learn/airflow-databricks).

## Prerequisites

* An Airflow environment with the [Airflow Databricks provider](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/index.html) (`apache-airflow-providers-databricks`) installed.
* A [Databricks account](https://www.databricks.com/try-databricks?itm_data=NavBar-TryDatabricks-Trial#account).

<Note>
  Astro users can also create connections using the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager), which stores connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).
</Note>

## Connect with an OAuth connection

An OAuth connection from Airflow to Databricks requires the following information:

* **Host**: Databricks URL
* **Service Principal Client ID** / **Login**: Service Principal Client ID
* **Service Principal Client Secret** / **Password**: Service Principal Client Secret

Complete the following steps to retrieve these values:

1. In the Databricks Cloud UI, copy the URL of your Databricks workspace. It should be formatted as either `https://dbc-75fc7ab7-96a6.cloud.databricks.com/` or `https://your-org.cloud.databricks.com/`.
2. Create a service principal in Databricks and copy the Client ID and Client Secret, see [Authorize service principal access to Databricks with OAuth](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m).

## Connect with a personal access token

A Personal Access Token (PAT) connection from Airflow to Databricks requires the following information:

* **Host**: Databricks URL
* **Personal Access Token** / **Password**: Personal access token

Complete the following steps to retrieve these values:

1. In the Databricks Cloud UI, copy the URL of your Databricks workspace. It should be formatted as either `https://dbc-75fc7ab7-96a6.cloud.databricks.com/` or `https://your-org.cloud.databricks.com/`.
2. To use a personal access token for a user, follow the [Databricks documentation](https://docs.databricks.com/dev-tools/auth.html#databricks-personal-access-tokens-for-users) to generate a new token. To generate a personal access token for a service principal, see [Manage personal access tokens for a service principal](https://docs.databricks.com/administration-guide/users-groups/service-principals.html#manage-personal-access-tokens-for-a-service-principal). Copy the personal access token.

## See also

* [Apache Airflow Databricks provider package documentation](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/index.html)
* [Databricks modules](https://airflow.apache.org/registry/providers/databricks/) in the Airflow Registry


# Create a dbt Cloud connection in Airflow
Source: https://astronomer.io/docs/learn/connections/dbt-cloud

Learn how to create a dbt Cloud connection in Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[dbt Cloud](https://www.getdbt.com/product/what-is-dbt/) is a SaaS product that runs SQL-first transformation workflows. Integrating dbt Cloud with Airflow allows you to trigger dbt cloud jobs and check their status from an Airflow DAG.

This guide provides the basic setup for creating a dbt Cloud Airflow connection. For a complete integration tutorial, see [Orchestrate dbt Cloud jobs with Airflow](/docs/learn/2.x/airflow-dbt-cloud). To run your dbt core jobs using Airflow, see [Orchestrate dbt-core Jobs with Airflow](/docs/learn/2.x/airflow-dbt).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* A locally running [Astro project](/docs/cli/v1.43/get-started-cli).
* A [dbt Cloud account](https://cloud.getdbt.com/)

## Get connection details

A connection from Airflow to dbt Cloud requires the following information:

* dbt Cloud URL
* API token
* Account ID

Complete the following steps to retrieve these values:

1. In the dbt Cloud UI, copy the URL of your dbt Cloud account. It should be formatted as `https://cloud.getdbt.com`. Your URL might be different based on the hosted region of your dbt Cloud. See [dbt Cloud URIs](https://docs.getdbt.com/docs/cloud/manage-access/sso-overview#auth0-multi-tenant-uris) for more details.

2. If you're using a dbt Developer account, follow the [dbt documentation](https://docs.getdbt.com/docs/dbt-cloud-apis/user-tokens#user-api-tokens) to copy the API token for your user account. To generate a token for a service account, see [Generating service account tokens](https://docs.getdbt.com/docs/dbt-cloud-apis/service-tokens#generating-service-account-tokens).

3. To retrieve the Account ID of your dbt account, go to the settings page in dbt Cloud UI. Then, from the URL of the settings page, copy the account ID, which is an integer appearing after `accounts`. For example, in the URL `https://cloud.getdbt.com/settings/accounts/88348`, the account ID is `88348`.

## Create your connection

<Info>
  Astro users can also create connections using the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager), which stores connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).
</Info>

1. Open your Astro project and add the following line to your `requirements.txt` file:

   ```text wrap theme={null}
   apache-airflow-providers-dbt-cloud
   ```

   This will install the dbt Cloud provider package, which makes the dbt Cloud connection type available in Airflow.

2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

3. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then select the connection type as **dbt Cloud**.

4. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

   * **Connection Id**: Enter a name for the connection.
   * **Tenant**: Enter the dbt Cloud URL.
   * **API Token**: Enter your user token or service access token.
   * **Account ID**: Enter your dbt cloud account ID. This field is optional. If you skip this, you must pass the account ID to the dbt cloud operator or hook.

5. Click **Test**. After the connection test succeeds, click **Save**.

   <Frame>
     <img alt="dbt Cloud connection test" />
   </Frame>

<Info>
  Note that the connection test only tests the **Tenant** and **API Token** fields. It won't validate your **Account ID**.
</Info>

## How it works

Airflow uses Python's `requests` library to connect to dbt Cloud through the [`DbtCloudHook`](https://airflow.apache.org/docs/apache-airflow-providers-dbt-cloud/stable/_api/airflow/providers/dbt/cloud/hooks/dbt/index.html).

## See also

* [Apache Airflow dbt cloud provider package documentation](https://airflow.apache.org/docs/apache-airflow-providers-dbt-cloud/stable/connections.html)
* dbt Cloud modules in the [Airflow Registry](https://airflow.apache.org/registry/)
* [Import and export Airflow connections using Astro CLI](/docs/astro/import-export-connections-variables#using-the-astro-cli-local-environments-only)


# Create a Microsoft Entra Workload ID connection in Airflow
Source: https://astronomer.io/docs/learn/connections/entra-workload-identity

Learn how to create an Azure Workload Identity connection in Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

A [workload identity](https://learn.microsoft.com/en-us/entra/workload-id/workload-identities-overview) is an identity you can assign to your Airflow environment which is authorized to access external services and resources. On Azure, a single workload identity can be authorized to multiple Azure resources through Azure resource groups.

The new generic **Azure** connection type lets you assign a workload identity to your Airflow environment so that Airflow can access multiple Azure resources using a single Airflow connection. This configuration greatly simplifies the number of credentials and connections you need to manage for Azure workflows.

This guide explains how to set up an Azure Workload Identity connection using the **Azure** connection type on Astro. Astronomer recommends using this connection type for most Azure workflows.

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* The [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/) or access to the Azure web portal.
* An [Astro project](/docs/cli/v1.43/get-started-cli).
* (Optional) An [Astro Deployment](/docs/astro/create-deployment).
* A Microsoft Entra [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp).

<Info>
  If you want to use Microsoft Entra Workload ID with a generic Apache Airflow project, your setup steps might vary. See [Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/connections/azure.html).
</Info>

## Get connection details

To create a workload identity for your Airflow environment, you first need to link your Airflow environment to your Entra ID managed identity.

If you're using Astro, follow the steps in [Authorize Deployments to Cloud resources](/docs/astro/authorize-deployments-to-your-cloud?tab=azure#setup) to create a workload identity for your Deployment.

If you're using Apache Airflow outside of Astro, your setup will vary based on your cloud and the environment you're running Airflow in. Generally speaking, the setup will be similar to the following:

1. In your Azure portal, open the **Managed Identities** menu.
2. Search for your managed identity, click **Properties**, then copy its **Name**, **Client ID**, **Tenant ID**, and **Resource group name**.
3. Run the following command to create a workload identity for your Airflow environment, replacing the `<managed-identity>` and `<resource-group>` values with your managed identity **Name** and **Resource group name** respectively.

   ```bash wrap theme={null}
   workloads=( scheduler triggerer worker )
   for workload in "${workloads[@]}"; do
       az identity federated-credential create --name <credential-name>-$workload --identity-name <managed-identity> --resource-group <resource-group> --issuer <your-issuer> --subject <your-service-account>
   done
   az identity federated-credential create --name <credential-name> --identity-name <managed-identity> --resource-group <resource-group> --issuer <your-issuer> --subject <your-service-account>
   ```

## Create your connection

To create your connection in Astro, follow the steps to [create a new connection in the Astro Environment Manager](/docs/astro/create-and-link-connections). Select the **Azure workload identity** connection type and enter your **Client ID** and **Tenant ID**. If you need to specify a **Subscription ID** for a specific service, you can open the **More options** dropdown menu and add it there.

<Frame>
  <img alt="Example Azure workload identity connection" />
</Frame>

Alternatively, to create your connection in the Airflow UI:

1. In the Airflow UI, go to **Admin** > **Connections**.
2. Click **+** to add a new connection, then select **Azure** as the connection type.
3. Enter the `clientId` and `tenantId` fields you retrieved from [Get connection details](#get-connection-details) and enter them into the **Managed Identity Client ID** and **Workload Identity Tenant ID** fields respectively. You can also specify a `subscriptionId` for a specific service if required.
4. Click **Save**.

After you create your connection, any DAGs using the connection will have the same permissions and access you defined in your managed identity.


# Create a Microsoft SQL Server connection in Airflow
Source: https://astronomer.io/docs/learn/connections/ms-sqlserver

Learn how to create a Microsoft SQL Server connection in Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Microsoft SQL Server](https://www.microsoft.com/en-in/sql-server/sql-server-downloads) is a proprietary relational database management system developed by Microsoft. Integrating SQL Server with Airflow allows you to interact with the database or export the data from a SQL server to an external system using an Airflow DAG.

This guide provides the basic setup for creating a Microsoft SQL Server connection.

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* A locally running [Astro project](/docs/cli/v1.43/get-started-cli).
* A Microsoft SQL Server database hosted in cloud or on-premises.
* [Permissions](https://www.w3computing.com/sqlserver2012/managing-permissions-using-management-studio/) to access Microsoft SQL Server from your local Airflow environment.

## Get connection details

A connection from Airflow to Microsoft SQL Server requires the following information:

* Host (also known as the endpoint URL, server name, or instance ID depending on your cloud provider)
* Port (default is 1433)
* Username
* Password
* Schema (default is `dbo`)

The method to retrieve these values will vary based on which cloud provider you use to host Microsoft SQL Server. Refer to the following documents for more information about retrieving these values.

* AWS: Connect to Microsoft SQL Server running [on RDS](https://aws.amazon.com/getting-started/hands-on/create-microsoft-sql-db/)
* GCP: Connect to Microsoft SQL Server running [on Cloud SQL](https://cloud.google.com/sql/docs/sqlserver/quickstarts)
* Azure: Connect to Microsoft SQL Server running on an [Azure SQL database](https://learn.microsoft.com/en-us/azure/azure-sql/database/connect-query-ssms?view=azuresql-mi) or [on a VM](https://learn.microsoft.com/en-us/azure/azure-sql/virtual-machines/windows/ways-to-connect-to-sql?view=azuresql-vm)

For example, if you are running Microsoft SQL Server in a Relational Data Store (RDS) in AWS, complete the following steps to retrieve these values:

1. In your AWS console, select your region, then go to the RDS service and select your SQL Server database.
2. Open the **Connectivity & security** tab and copy the **Endpoint** and **Port**.
3. Follow the documentation for Microsoft SQL Server to [create a new database user](https://learn.microsoft.com/en-us/sql/relational-databases/security/authentication-access/create-a-database-user?view=sql-server-ver16). Copy the username and password.
4. (Optional) To use a specific schema, copy the name of the schema. If you skip this, Airflow uses the default schema `dbo`.

## Create your connection

<Info>
  Astro users can also create connections using the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager), which stores connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).
</Info>

1. Open your Astro project and add the following line to your `requirements.txt` file:

   ```text wrap theme={null}
   apache-airflow-providers-microsoft-mssql
   ```

   This will install the Microsoft provider package, which makes the MS SQL Server connection type available in Airflow.

   <Info>
     To install `apache-airflow-providers-microsoft-mssql` to Airflow 2.6+, you must also add the following lines to `packages.txt` and restart your Astro project.

     ```text wrap theme={null}
     build-essential
     freetds-dev
     libkrb5-dev
     default-libmysqlclient-dev
     ```
   </Info>

2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

3. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then choose **Microsoft SQL Server** as the connection type.

4. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

   * **Connection Id**: Enter a name for the connection.
   * **Host**: Enter your host, endpoint URL, server name, or instance ID.
   * **Schema**: Enter your schema name.
   * **Login**: Enter your username.
   * **Password**: Enter your password.
   * **Port**: Enter your **Port**.

5. Click **Test**. After the connection test succeeds, click **Save**.

   <Frame>
     <img alt="connection-mssqlserver" />
   </Frame>

## How it works

Airflow uses [PyMSSQL](https://pypi.org/project/pymssql/) to connect to Microsoft SQL Server through the [MsSqlhook](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-mssql/1.0.0/_api/airflow/providers/microsoft/mssql/hooks/mssql/index.html). You can also directly use the MsSqlhook to create your own custom operators.

## See also

* [Apache Airflow Microsoft provider package documentation](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-mssql/stable/index.html)
* MS SQL Server Modules in the [Airflow Registry](https://airflow.apache.org/registry/)
* [Import and export Airflow connections using Astro CLI](/docs/astro/import-export-connections-variables#using-the-astro-cli-local-environments-only)


# Cross-DAG dependencies
Source: https://astronomer.io/docs/learn/cross-dag-dependencies

How to implement dependencies between your Airflow DAGs.

When designing Airflow DAGs, it is often best practice to put all related tasks in the same DAG. However, it's sometimes necessary to create dependencies between your DAGs. In this scenario, one node of a DAG is its own complete DAG, rather than just a single task. Throughout this guide, the following terms are used to describe DAG dependencies:

* **Upstream DAG**: A DAG that must reach a specified state before a downstream DAG can run
* **Downstream DAG**: A DAG that can't run until an upstream DAG reaches a specified state

The Airflow topic [Cross-DAG Dependencies](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/external_task_sensor.html#cross-dag-dependencies), indicates cross-DAG dependencies can be helpful in the following situations:

* A DAG should only run after one or more assets have been updated by tasks in other DAGs.
* Two DAGs are dependent, but they have different schedules.
* Two DAGs are dependent, but they are owned by different teams.
* A task depends on another task but for a different execution date.

In this guide, you'll review the methods for implementing cross-DAG dependencies, including how to implement dependencies if your dependent DAGs are located in different Airflow deployments.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Dependencies in Airflow. See [Managing Dependencies in Apache Airflow](/docs/learn/managing-dependencies).
* Airflow DAGs. See [Introduction to Airflow DAGs](/docs/learn/dags).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow sensors. See [Sensors 101](/docs/learn/what-is-a-sensor).

## Implement cross-DAG dependencies

There are multiple ways to implement cross-DAG dependencies in Airflow, including:

* [Data-aware scheduling using assets](/docs/learn/airflow-datasets).
* The [`TriggerDagRunOperator`](https://airflow.apache.org/registry/providers/standard#standard-trigger_dagrun-TriggerDagRunOperator).
* The [`ExternalTaskSensor`](https://airflow.apache.org/registry/providers/standard#standard-external_task-ExternalTaskSensor).
* The [Airflow API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html).

In this section, you'll learn how and when you should use each method and how to view dependencies in the Airflow UI.

### Asset dependencies

The most common way to define cross-DAG dependencies is by using [assets](/docs/learn/airflow-datasets). DAGs that access the same data can have explicit, visible relationships, and DAGs can be scheduled based on updates to this data.

You should use this method if you have a downstream DAG that should only run after an asset has been updated by an upstream DAG, especially if those updates are irregular. This type of dependency also provides you with increased observability into the dependencies between your DAGs and assets in the Airflow UI.

Using assets requires knowledge of the following scheduling concepts:

* **Producing task**: A task that updates a specific asset, defined by its `outlets` parameter.
* **Consuming DAG**: A DAG that runs as soon as a specific asset is updated.

Any task can be made into a producing task by providing one or more assets to the `outlets` parameter. For example:

```python wrap theme={null}
from airflow.sdk import Asset 
from airflow.providers.standard.operators.empty import EmptyOperator 

asset1 = Asset('asset1')

# producing task in the upstream DAG
EmptyOperator(
    task_id="producing_task",
    outlets=[asset1]  # flagging to Airflow that asset1 was updated
)
```

The following downstream DAG is scheduled to run after `asset1` has been updated by providing it to the `schedule` parameter.

```python wrap theme={null}
from airflow.sdk import Asset, dag

asset1 = Asset('asset1')

# consuming DAG
@dag(schedule=[asset1])
```

See [Assets and data-aware scheduling in Airflow](/docs/learn/airflow-datasets) to learn more.

### `TriggerDagRunOperator`

The `TriggerDagRunOperator` is a straightforward method of implementing cross-DAG dependencies from an upstream DAG. This operator allows you to have a task in one DAG that triggers another DAG in the same Airflow environment. For more information about this operator, see [`TriggerDagRunOperator`](https://airflow.apache.org/registry/providers/standard#standard-trigger_dagrun-TriggerDagRunOperator).

You can trigger a downstream DAG with the `TriggerDagRunOperator` from any point in the upstream DAG. If you set the operator's `wait_for_completion` parameter to `True`, the upstream DAG pauses and then resumes only after the downstream DAG has finished running. This waiting process can be deferred to the triggerer by setting the parameter, `deferrable`, to True. This setting turns the operator into a [deferrable operator](/docs/learn/deferrable-operators), which increases Airflow's scalability and can reduce cost.

A common use case for this implementation is when an upstream DAG fetches new testing data for a machine learning pipeline, runs and tests a model, and publishes the model's prediction. In case of the model underperforming, the `TriggerDagRunOperator` is used to start a separate DAG that retrains the model while the upstream DAG waits. Once the model is retrained and tested by the downstream DAG, the upstream DAG resumes and publishes the new model's results.

The [schedule](/docs/learn/scheduling-in-airflow) of the downstream DAG is independent of the runs triggered by the `TriggerDagRunOperator`. To run a DAG solely with the `TriggerDagRunOperator`, set the DAG's `schedule` parameter to `None`. Note that the dependent DAG must be unpaused to get triggered.

The following example DAG implements the `TriggerDagRunOperator` to trigger a DAG with the `dag_id` `dependent_dag` between two other tasks. Since both the `wait_for_completion` and the `deferrable` parameters of the `trigger_dependent_dag` task in the `trigger_dagrun_dag` are set to `True`, the task is deferred until the `dependent_dag` has finished its run. Once the `trigger_dagrun_dag` task completes, the `end_task` will run.

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  from airflow.decorators import dag, task
  from airflow.operators.trigger_dagrun import TriggerDagRunOperator
  from pendulum import datetime, duration


  @task
  def start_task(task_type):
      return f"The {task_type} task has completed."


  @task
  def end_task(task_type):
      return f"The {task_type} task has completed."


  # Default settings applied to all tasks
  default_args = {
      "owner": "airflow",
      "depends_on_past": False,
      "email_on_failure": False,
      "email_on_retry": False,
      "retries": 1,
      "retry_delay": duration(minutes=5),
  }


  @dag(
      start_date=datetime(2023, 1, 1),
      max_active_runs=1,
      schedule="@daily",
      default_args=default_args,
      catchup=False,
  )
  def trigger_dagrun_dag():
      trigger_dependent_dag = TriggerDagRunOperator(
          task_id="trigger_dependent_dag",
          trigger_dag_id="dependent_dag",
          wait_for_completion=True,
          deferrable=True,  # Note that this parameter only exists in Airflow 2.6+
      )

      start_task("starting") >> trigger_dependent_dag >> end_task("ending")


  trigger_dagrun_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  from airflow import DAG
  from airflow.operators.python import PythonOperator
  from airflow.operators.trigger_dagrun import TriggerDagRunOperator
  from pendulum import datetime, duration


  def print_task_type(**kwargs):
      print(f"The {kwargs['task_type']} task has completed.")


  # Default settings applied to all tasks
  default_args = {
      "owner": "airflow",
      "depends_on_past": False,
      "email_on_failure": False,
      "email_on_retry": False,
      "retries": 1,
      "retry_delay": duration(minutes=5),
  }

  with DAG(
      dag_id="trigger_dagrun_dag",
      start_date=datetime(2023, 1, 1),
      max_active_runs=1,
      schedule="@daily",
      default_args=default_args,
      catchup=False,
  ) as dag:
      start_task = PythonOperator(
          task_id="start_task",
          python_callable=print_task_type,
          op_kwargs={"task_type": "starting"},
      )

      trigger_dependent_dag = TriggerDagRunOperator(
          task_id="trigger_dependent_dag",
          trigger_dag_id="dependent_dag",
          wait_for_completion=True,
          deferrable=True,  # Note that this parameter only exists in Airflow 2.6+
      )

      end_task = PythonOperator(
          task_id="end_task",
          python_callable=print_task_type,
          op_kwargs={"task_type": "ending"},
      )

      start_task >> trigger_dependent_dag >> end_task
  ```
</details>

If your dependent DAG requires a config input or a specific logical date, you can specify them in the operator using the `conf` and `logical_date` params respectively.

<Tip>
  You can set `skip_when_already_exists` to `True` to keep the operator from attempting to trigger runs that have already occurred, and failing as a result. This can happen when trying to rerun DAGs and tasks. Additionally, you can set `fail_when_dag_is_paused` to fail the task instantiated with the `TriggerDagRunOperator` if the dependent DAG is paused.
</Tip>

### `ExternalTaskSensor`

To create cross-DAG dependencies from a downstream DAG, consider using one or more [ExternalTaskSensors](https://airflow.apache.org/registry/providers/standard#standard-external_task-ExternalTaskSensor). The downstream DAG will pause until a task is completed in the upstream DAG before resuming.

This method of creating cross-DAG dependencies is especially useful when you have a downstream DAG with different branches that depend on different tasks in one or more upstream DAGs. Instead of defining an entire DAG as being downstream of another DAG as you do with assets, you can set a specific task in a downstream DAG to wait for a task to finish in an upstream DAG.

For example, you could have upstream tasks modifying different tables in a data warehouse and one downstream DAG running one branch of data quality checks for each of those tables. You can use one `ExternalTaskSensor` at the start of each branch to make sure that the checks running on each table only start after the update to the specific table is finished.

You use the `ExternalTaskSensor` in deferrable mode using `deferrable=True`. For more info on deferrable operators and their benefits, see [Deferrable Operators](/docs/learn/deferrable-operators)

The following example DAG uses three ExternalTaskSensors at the start of three parallel branches in the same DAG.

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  from airflow.decorators import dag, task
  from airflow.sensors.external_task import ExternalTaskSensor
  from airflow.operators.empty import EmptyOperator
  from pendulum import datetime, duration


  @task
  def downstream_function_branch_1():
      print("Upstream DAG 1 has completed. Starting tasks of branch 1.")


  @task
  def downstream_function_branch_2():
      print("Upstream DAG 2 has completed. Starting tasks of branch 2.")


  @task
  def downstream_function_branch_3():
      print("Upstream DAG 3 has completed. Starting tasks of branch 3.")


  default_args = {
      "owner": "airflow",
      "depends_on_past": False,
      "email_on_failure": False,
      "email_on_retry": False,
      "retries": 1,
      "retry_delay": duration(minutes=5),
  }


  @dag(
      start_date=datetime(2022, 8, 1),
      max_active_runs=3,
      schedule="*/1 * * * *",
      catchup=False,
  )
  def external_task_sensor_taskflow_dag():
      start = EmptyOperator(task_id="start")
      end = EmptyOperator(task_id="end")

      ets_branch_1 = ExternalTaskSensor(
          task_id="ets_branch_1",
          external_dag_id="upstream_dag_1",
          external_task_id="my_task",
          allowed_states=["success"],
          failed_states=["failed", "skipped"],
      )

      task_branch_1 = downstream_function_branch_1()

      ets_branch_2 = ExternalTaskSensor(
          task_id="ets_branch_2",
          external_dag_id="upstream_dag_2",
          external_task_id="my_task",
          allowed_states=["success"],
          failed_states=["failed", "skipped"],
      )

      task_branch_2 = downstream_function_branch_2()

      ets_branch_3 = ExternalTaskSensor(
          task_id="ets_branch_3",
          external_dag_id="upstream_dag_3",
          external_task_id="my_task",
          allowed_states=["success"],
          failed_states=["failed", "skipped"],
      )

      task_branch_3 = downstream_function_branch_3()

      start >> [ets_branch_1, ets_branch_2, ets_branch_3]

      ets_branch_1 >> task_branch_1
      ets_branch_2 >> task_branch_2
      ets_branch_3 >> task_branch_3

      [task_branch_1, task_branch_2, task_branch_3] >> end


  external_task_sensor_taskflow_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  from airflow import DAG
  from airflow.operators.python import PythonOperator
  from airflow.sensors.external_task import ExternalTaskSensor
  from airflow.operators.empty import EmptyOperator
  from pendulum import datetime, duration


  def downstream_function_branch_1():
      print("Upstream DAG 1 has completed. Starting tasks of branch 1.")


  def downstream_function_branch_2():
      print("Upstream DAG 2 has completed. Starting tasks of branch 2.")


  def downstream_function_branch_3():
      print("Upstream DAG 3 has completed. Starting tasks of branch 3.")


  default_args = {
      "owner": "airflow",
      "depends_on_past": False,
      "email_on_failure": False,
      "email_on_retry": False,
      "retries": 1,
      "retry_delay": duration(minutes=5),
  }

  with DAG(
      "external-task-sensor-dag",
      start_date=datetime(2022, 8, 1),
      max_active_runs=3,
      schedule="*/1 * * * *",
      catchup=False,
  ) as dag:
      start = EmptyOperator(task_id="start")
      end = EmptyOperator(task_id="end")

      ets_branch_1 = ExternalTaskSensor(
          task_id="ets_branch_1",
          external_dag_id="upstream_dag_1",
          external_task_id="my_task",
          allowed_states=["success"],
          failed_states=["failed", "skipped"],
      )

      task_branch_1 = PythonOperator(
          task_id="task_branch_1",
          python_callable=downstream_function_branch_1,
      )

      ets_branch_2 = ExternalTaskSensor(
          task_id="ets_branch_2",
          external_dag_id="upstream_dag_2",
          external_task_id="my_task",
          allowed_states=["success"],
          failed_states=["failed", "skipped"],
      )

      task_branch_2 = PythonOperator(
          task_id="task_branch_2",
          python_callable=downstream_function_branch_2,
      )

      ets_branch_3 = ExternalTaskSensor(
          task_id="ets_branch_3",
          external_dag_id="upstream_dag_3",
          external_task_id="my_task",
          allowed_states=["success"],
          failed_states=["failed", "skipped"],
      )

      task_branch_3 = PythonOperator(
          task_id="task_branch_3",
          python_callable=downstream_function_branch_3,
      )

      start >> [ets_branch_1, ets_branch_2, ets_branch_3]

      ets_branch_1 >> task_branch_1
      ets_branch_2 >> task_branch_2
      ets_branch_3 >> task_branch_3

      [task_branch_1, task_branch_2, task_branch_3] >> end
  ```
</details>

In this DAG:

* `ets_branch_1` waits for the `my_task` task of `upstream_dag_1` to complete before moving on to execute `task_branch_1`.
* `ets_branch_2` waits for the `my_task` task of `upstream_dag_2` to complete before moving on to execute `task_branch_2`.
* `ets_branch_3` waits for the `my_task` task of `upstream_dag_3` to complete before moving on to execute `task_branch_3`.

These processes happen in parallel and are independent of each other. The graph view shows the state of the DAG after `my_task` in `upstream_dag_1` has finished which caused `ets_branch_1` and `task_branch_1` to run. `ets_branch_2` and `ets_branch_3` are still waiting for their upstream tasks to finish.

<Frame>
  <img alt="ExternalTaskSensor 3 Branches" />
</Frame>

If you want the downstream DAG to wait for the entire upstream DAG to finish instead of a specific task, you can set the `external_task_id` to `None`. In the example above, you specified that the external task must have a state of `success` for the downstream task to succeed, as defined by the `allowed_states` and `failed_states`.

In the previous example, the upstream DAG (`example_dag`) and downstream DAG (`external-task-sensor-dag`) must have the same start date and schedule interval. This is because the `ExternalTaskSensor` will look for completion of the specified task or DAG at the same `logical_date`. To look for completion of the external task at a different date, you can make use of either of the `execution_delta` or `execution_date_fn` parameters (these are described in more detail in the documentation linked above).

### Airflow API

The [Airflow API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) is another way of creating cross-DAG dependencies. To use the API to trigger a DAG run, you can make a POST request to the `DAGRuns` endpoint.

The following script shows how to trigger a DAG run using the Airflow API, for cross-dag dependencies, you would run this code inside an `@task` decorated function in your upstream DAG.

```python expandable wrap theme={null}
import requests

# Replace with your Airflow instance details
USERNAME = "admin"
PASSWORD = "admin"
HOST = "http://localhost:8080/"

MY_DAG = "example_dag" # The id of the DAG you want to trigger

def get_jwt_token():
    token_url = f"{HOST}/auth/token"
    payload = {"username": USERNAME, "password": PASSWORD}
    headers = {"Content-Type": "application/json"}
    response = requests.post(token_url, json=payload, headers=headers)

    token = response.json().get("access_token")
    return token

def run_dag(dag_id, logical_date=None):
    event_payload = {"conf": {"param1": "Hello World"}, "logical_date": logical_date}
    token = get_jwt_token()

    if token:
        url = f"{HOST}/api/v2/dags/{dag_id}/dagRuns"
        headers = {"Authorization": f"Bearer {token}"}
        response = requests.post(url, json=event_payload, headers=headers)

        print(response.status_code)
        print(response.json())
    else:
        raise Exception("Failed to get JWT token")

if __name__ == "__main__":
    run_dag(dag_id=MY_DAG, logical_date=None)
```

You can also update an asset using the API by making a POST request to the `Assets` endpoint.

## Cross-deployment dependencies

To implement cross-DAG dependencies on two different Airflow environments on Astro, follow the guidance in [Cross-deployment dependencies](/docs/astro/best-practices/cross-deployment-dependencies).


# Create DAG documentation in Apache Airflow
Source: https://astronomer.io/docs/learn/custom-airflow-ui-docs-tutorial

Use Apache Airflow's built-in documentation features to generate documentation for your DAGs in the Airflow UI.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

One of the more powerful and lesser-known features of Airflow is that you can create Markdown-based DAG documentation that appears in the Airflow UI

<Frame>
  <img alt="DAG Docs Intro Example" />
</Frame>

After you complete this tutorial, you'll be able to:

* Add custom doc strings to an Airflow DAG.
* Add custom doc strings to an Airflow task.

## Time to complete

This tutorial takes approximately 15 minutes to complete.

## Assumed knowledge

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Basic Python. See the [Python Documentation](https://docs.python.org/3/tutorial/index.html).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/install-cli).

## Step 1: Create an Astro project

To run Airflow locally, you first need to create an Astro project.

1. Create a new directory for your Astro project:

   ```sh wrap theme={null}
   mkdir <your-astro-project-name> && cd <your-astro-project-name>
   ```

2. Run the following Astro CLI command to initialize an Astro project in the directory:

   ```sh wrap theme={null}
   astro dev init
   ```

3. To enable raw HTML in your Markdown DAG descriptions, add the following Airflow config environment variable to your `.env` file. This will allow you to use HTML in your DAG descriptions. If you don't want to enable this setting due to security concerns you will still be able to use Markdown in your DAG descriptions and the HTML shown in this tutorial will be displayed as raw content.

   ```text wrap theme={null}
   AIRFLOW__WEBSERVER__ALLOW_RAW_HTML_DESCRIPTIONS=True
   ```

4. Start your Airflow instance by running:

   ```sh wrap theme={null}
   astro dev start
   ```

## Step 2: Create a new DAG

1. In your `dags` folder, create a file named `docs_example_dag.py`.

2. Copy and paste one of the following DAGs based on which coding style you're most comfortable with.

<details>
  <summary>TaskFlow API</summary>

  ```python wrap theme={null}
  from airflow.decorators import task, dag
  from pendulum import datetime
  import requests

  @dag(
      start_date=datetime(2022,11,1),
      schedule="@daily",
      catchup=False
  )
  def docs_example_dag():

      @task
      def tell_me_what_to_do():
          response = requests.get("https://bored-api.appbrewery.com/random")
          return response.json()["activity"]

      tell_me_what_to_do()

  docs_example_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.models.dag import DAG
  from airflow.operators.python import PythonOperator
  from pendulum import datetime
  import requests

  def query_api():
      response = requests.get("https://bored-api.appbrewery.com/random")
      return response.json()["activity"]

  with DAG(
      dag_id="docs_example_dag",
      start_date=datetime(2022,11,1),
      schedule=None,
      catchup=False,
  ):

      tell_me_what_to_do = PythonOperator(
          task_id="tell_me_what_to_do",
          python_callable=query_api,
      )
  ```
</details>

This DAG has one task called `tell_me_what_to_do`, which queries an [API](https://bored-api.appbrewery.com/random) that provides a random activity for the day and prints it to the logs.

## Step 3: Add docs to your DAG

You can add Markdown-based documentation to your DAGs that will render in the **Grid**, **Graph** and **Calendar** pages of the Airflow UI.

1. In your `docs_example_dag.py` file, add the following doc string above the definition of your DAG:

   ```python wrap theme={null}
   doc_md_DAG = """
   ### The Activity DAG

   This DAG will help me decide what to do today. It uses the [BoredAPI](https://bored-api.appbrewery.com/random) to do so.

   Before I get to do the activity I will have to:

   - Clean up the kitchen.
   - Check on my pipelines.
   - Water the plants.

   Here are some happy plants:

   <img src="https://www.publicdomainpictures.net/pictures/80000/velka/succulent-roses-echeveria.jpg" alt="plants" width="300"/>
   """
   ```

   This doc string is written in Markdown. It includes a title, a link to an external website, a bulleted list, as well as an image which has been formatted using HTML. To learn more about Markdown, see [The Markdown Guide](https://www.markdownguide.org/).

2. Add the documentation to your DAG by passing `doc_md_DAG` to the `doc_md` parameter of your DAG class as shown in the code snippet below:

<details>
  <summary>TaskFlow API</summary>

  ```python wrap theme={null}
  @dag(
      start_date=datetime(2022,11,1),
      schedule="@daily",
      catchup=False,
      doc_md=doc_md_DAG
  )
  def docs_example_dag():
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  with DAG(
      dag_id="docs_example_dag",
      start_date=datetime(2022,11,1),
      schedule="@daily",
      catchup=False,
      doc_md=doc_md_DAG
  ):
  ```
</details>

3. Go to the **Grid** view and click the **DAG Docs** banner to view the rendered documentation.

   <Frame>
     <img alt="DAG Docs" />
   </Frame>

<Tip>
  Airflow will automatically pick up a doc string written directly beneath the definition of the DAG context and add it as **DAG Docs**.
  Additionally, using `with DAG():` lets you pass the filepath of a Markdown file to the `doc_md` parameter. This can be useful if you want to add the same documentation to several of your DAGs.
</Tip>

## Step 4: Add docs to a task

You can also add docs to specific Airflow tasks using Markdown, Monospace, JSON, YAML or reStructuredText. Note that only Markdown will be rendered and other formats will be displayed as rich content.

To add documentation to your task, follow these steps:

1. Add the following code with a string in Markdown format:

   ```python wrap theme={null}
   doc_md_task = """

   ### Purpose of this task

   This task **boldly** suggests a daily activity.
   """
   ```

2. Add the following code with a string written in monospace format:

   ```python wrap theme={null}
   doc_monospace_task = """
   If you don't like the suggested activity you can always just go to the park instead.
   """
   ```

3. Add the following code with a string in JSON format:

   ```python wrap theme={null}
   doc_json_task = """
   {
       "previous_suggestions": {
           "go to the gym": ["frequency": 2, "rating": 8],
           "mow your lawn": ["frequency": 1, "rating": 2],
           "read a book": ["frequency": 3, "rating": 10],
       }
   }
   """
   ```

4. Add the following code with a string written in YAML format:

   ```python wrap theme={null}
   doc_yaml_task = """
   clothes_to_wear: sports
   gear: |
       - climbing: true
       - swimming: false
   """
   ```

5. Add the following code containing reStructuredText:

   ```python wrap theme={null}
   doc_rst_task = """
   ===========
   This feature is pretty neat
   ===========

   * there are many ways to add docs
   * luckily Airflow supports a lot of them

   .. note:: `Learn more about rst here! <https://gdal.org/contributing/rst_style.html#>`__
   """
   ```

6. Create a task definition as shown in the following snippet. The task definition includes parameters for specifying each of the documentation strings you created. Pick the coding style you're most comfortable with.

<details>
  <summary>TaskFlow API</summary>

  ```python wrap theme={null}
  @task(
      doc_md=doc_md_task,
      doc=doc_monospace_task,
      doc_json=doc_json_task,
      doc_yaml=doc_yaml_task,
      doc_rst=doc_rst_task
  )
  def tell_me_what_to_do():
      response = requests.get("https://bored-api.appbrewery.com/random")
      return response.json()["activity"]

  tell_me_what_to_do()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  tell_me_what_to_do = PythonOperator(
      task_id="tell_me_what_to_do",
      python_callable=query_api,
      doc_md=doc_md_task,
      doc=doc_monospace_task,
      doc_json=doc_json_task,
      doc_yaml=doc_yaml_task,
      doc_rst=doc_rst_task
  )
  ```
</details>

3. Go to the Airflow UI and run your DAG.

4. In the **Grid** view, click the green square for your task instance.

5. Click **Task Instance Details**.

   <Frame>
     <img alt="Task Instance Details" />
   </Frame>

6. See the docs under their respective attribute:

   <Frame>
     <img alt="All Task Docs" />
   </Frame>

In Airflow 2.10+, task docs provided to `doc_md` or as a doc string in a `@task` decorated task are rendered in the task details in the Airflow UI.

<Frame>
  <img alt="Task docs rendered in the Airflow 2.10 UI" />
</Frame>

## Step 5: Add notes to a task instance and DAG run

You can add notes to task instances and DAG runs from the **Grid** view in the Airflow UI. This feature is useful if you need to share contextual information about a DAG or task run with your team, such as why a specific run failed.

1. Go to the **Grid View** of the `docs_example_dag` DAG you created in [Step 2](#step-2-create-a-new-dag).

2. Select a task instance or DAG run.

3. Click **Details** > **Task Instance Notes** or **DAG Run Notes** > **Add Note**.

4. Write a note and click **Save Note**.

   <Frame>
     <img alt="Add task note" />
   </Frame>

## Conclusion

Congratulations! You now know how to add fancy documentation to both your DAGs and your Airflow tasks.


# Strategies for custom XCom backends in Airflow
Source: https://astronomer.io/docs/learn/custom-xcom-backend-strategies

Use this guide to learn about different ways you can set up custom XCom backends.

Airflow [XComs](/docs/learn/airflow-passing-data-between-tasks) allow you to pass data between tasks. By default, Airflow uses the [metadata database](/docs/learn/airflow-database) to store XComs, which works well for local development but has limited performance. If you configure a custom XCom backend, you can define where and how Airflow stores XComs, as well as customize serialization and deserialization methods.

In this guide you'll learn:

* When you should use a custom XCom backend.
* How to set up a custom XCom backend using the Object Storage XCom Backend.
* How to use a custom XCom backend class with custom serialization and deserialization methods.

<Warning>
  While a custom XCom backend allows you to store virtually unlimited amounts of data as XComs, you will need to scale other Airflow components to pass large amounts of data between tasks. For help running Airflow at scale, [contact Astronomer](https://www.astronomer.io/lp/signup/).
</Warning>

## Assumed knowledge

To get the most benefits from this guide, you need an understanding of:

* XCom basics. See [Pass data between tasks](/docs/learn/airflow-passing-data-between-tasks).
* Basic knowledge of a cloud-based object storage service like [AWS S3](https://aws.amazon.com/s3/), [GCP Cloud Storage](https://cloud.google.com/storage) or [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/).

## Why use a custom XCom backend?

Common reasons to use a custom XCom backend include:

* You need more storage space for XComs than the Airflow metadata database can offer.
* You're running a production environment where you require custom retention, deletion, and backup policies for XComs.
* You want to access XComs without accessing the metadata database.
* You want to restrict types of allowed XCom values.
* You want to save XComs in multiple locations simultaneously.

You can also use custom XCom backends to define custom serialization and deserialization methods for XComs if you need to add a serialization method to a class, or if registering a custom serializer isn't feasible. See [Custom serialization and deserialization](#custom-serialization-and-deserialization) for more information.

## How to set up a custom XCom backend

There are two main ways to set up a custom XCom backend:

* **Object Storage XCom Backend**: Use this method to create a custom XCom backend when you want to store XComs in a cloud-based object storage service like AWS S3, GCP Cloud Storage, or Azure Blob Storage. This option is recommended if you need to store XComs in a single remote location, and the Object Storage XCom Backend threshold and compression options meet your requirements.
* **Custom XCom backend class**: Use this method when you want to further customize how XComs are stored, for example to simultaneously store XComs in two different locations.

Additionally, some provider packages offer custom XCom backends that you can use out of the box. For example, the [Snowpark provider](/docs/learn/airflow-snowpark) contains a custom XCom backend for Snowflake.

### Use the Object Storage XCom Backend

You can create a custom XCom backend using object storage. The Object Storage XCom Backend is part of the [Common IO](https://airflow.apache.org/registry/providers/common-io/) provider and can be defined using the following environment variables:

* `AIRFLOW__CORE__XCOM_BACKEND`: The XCom backend to use. Set this to `airflow.providers.common.io.xcom.backend.XComObjectStorageBackend` to use the Object Storage XCom Backend.
* `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH`: The path to the object storage where XComs are stored. The path should be in the format `<your-scheme>://<your-connection-id@<your-bucket>/xcom`. For example, `s3://my-s3-connection@my-bucket/xcom`. The most common schemes are `s3`, `gs`, and `abfs` for Amazon S3, Google Cloud Storage, and Azure Blob Storage, respectively.
* `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD`: The threshold in bytes for XComs to be stored in the object storage. All objects smaller or equal to this threshold are stored in the metadata database. All objects larger than this threshold are stored in the object storage. The default value is `-1`, meaning all XComs are stored in the metadata database.
* `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION`: Optional. The compression algorithm to use when storing XComs in the object storage, for example `zip`. The default value is `None`.

For a step-by-step tutorial on how to set up a custom XCom backend using the Object Storage XCom Backend for Amazon S3, Google Cloud Storage and Azure Blob Storage, see the [Set up a custom XCom backend using object storage](/docs/learn/custom-xcom-backends-tutorial).

### Use a custom XCom backend class

To create a custom XCom backend, you need to define an XCom backend class which inherits from the `BaseXCom` class.

The code below shows an example `MyCustomXComBackend` class that only allows JSON-serializeable XComs and stores them in both, Amazon S3 and Google Cloud Storage using a custom `serialize_value()` method. The `deserialize_value()` method retrieves the XComs from the Amazon S3 bucket and returns the value.

The Airflow metadata database stores a reference string to the XCom, which is displayed in the XCom tab of the Airflow UI. The reference string is prefixed with `s3_and_gs://` to indicate that the XCom is stored in both Amazon S3 and Google Cloud Storage. You can add any serialization and deserialization logic to the `serialize_value()` and `deserialize_value()` methods that you need, see [Custom serialization and deserialization](#custom-serialization-and-deserialization) for more information.

<details>
  <summary>Click to view the full custom XCom backend class example code</summary>

  ```python expandable wrap theme={null}
  from airflow.sdk.bases.xcom import BaseXCom
  from airflow.providers.amazon.aws.hooks.s3 import S3Hook
  from airflow.providers.google.cloud.hooks.gcs import GCSHook
  import json
  import uuid
  import os

  class MyCustomXComBackend(BaseXCom):
      # the prefix is optional and used to make it easier to recognize
      # which reference strings in the Airflow metadata database
      # refer to an XCom that has been stored in remote storage
      PREFIX = "s3_and_gs://"
      S3_BUCKET_NAME = "s3-xcom-backend-example"
      GS_BUCKET_NAME = "gcs-xcom-backend-example"

      @staticmethod
      def serialize_value(
          value,
          key=None,
          task_id=None,
          dag_id=None,
          run_id=None,
          map_index=None,
          **kwargs,
      ):

          # make sure the value is JSON-serializable
          try:
              serialized_value = json.dumps(value)
          except TypeError as e:
              raise ValueError(f"XCom value is not JSON-serializable!: {e}")

          # instantiate a context with the value as a temporary JSON file
          with tempfile.NamedTemporaryFile(mode="w+", delete=False) as tmp_file:
              tmp_file.write(serialized_value)
              tmp_file.flush()
              tmp_file_name = tmp_file.name

              # the connection to AWS is created by using the S3 hook
              hook = S3Hook(aws_conn_id="my_aws_conn_id")
              # make sure the file_id is unique, either by using combinations of
              # the task_id, run_id and map_index parameters or by using a uuid
              filename = "data_" + str(uuid.uuid4()) + ".json"
              # define the full S3 key where the file should be stored
              key = f"{dag_id}/{run_id}/{task_id}/{map_index}/{key}_{filename}"

              # load the local JSON file into the S3 bucket
              hook.load_file(
                  filename=tmp_file_name,
                  key=key,
                  bucket_name=MyCustomXComBackend.S3_BUCKET_NAME,
                  replace=True,
              )

              # the connection to GCS is created by using the GCS hook
              hook = GCSHook(gcp_conn_id="my_gcs_conn_id")

              if hook.exists(MyCustomXComBackend.GS_BUCKET_NAME, key):
                  print(
                      f"File {key} already exists in the bucket {MyCustomXComBackend.GS_BUCKET_NAME}."
                  )
              else:
                  # load the local JSON file into the GCS bucket
                  hook.upload(
                      filename=tmp_file_name,
                      object_name=key,
                      bucket_name=MyCustomXComBackend.GS_BUCKET_NAME,
                  )

          # define the string that will be saved to the Airflow metadata
          # database to refer to this XCom
          reference_string = MyCustomXComBackend.PREFIX + key

          # use JSON serialization to write the reference string to the
          # Airflow metadata database (like a regular XCom)
          return BaseXCom.serialize_value(value=reference_string)

      @staticmethod
      def deserialize_value(result):
          import logging

          reference_string = BaseXCom.deserialize_value(result=result)
          hook = S3Hook(aws_conn_id="my_aws_conn")
          key = reference_string.replace(MyCustomXComBackend.PREFIX, "")

          # Use a temporary directory to download the file
          with tempfile.TemporaryDirectory() as tmp_dir:
              local_file_path = hook.download_file(
                  key=key,
                  bucket_name=MyCustomXComBackend.S3_BUCKET_NAME,
                  local_path=tmp_dir,
              )

              # ensure the file is not empty and log its size
              file_size = os.path.getsize(local_file_path)
              logging.info(f"Downloaded file size: {file_size} bytes.")
              if file_size == 0:
                  raise ValueError(
                      f"The downloaded file is empty. Check the content of the S3 object at {key}."
                  )

              with open(local_file_path, "r") as file:
                  try:
                      output = json.load(file)
                  except json.JSONDecodeError as e:
                      logging.error(f"Error decoding JSON from the file: {e}")
                      raise

          return output
  ```
</details>

To use a custom XCom backend class, you need to save it in a Python file in the `include` directory of your Airflow project. Then, set the `AIRFLOW__CORE__XCOM_BACKEND` environment variable in your Airflow instance to the path of the custom XCom backend class. If you run Airflow locally with the Astro CLI, you can set the environment variable in the `.env` file of your Astro project. On Astro, you can [set the environment variable in the Astro UI](https://docs.astronomer.io/astro/environment-variables).

```text wrap theme={null}
AIRFLOW__CORE__XCOM_BACKEND=include.<your-file-name>.MyCustomXComBackend
```

If you want to further customize the functionality for your custom XCom backend, you can override additional methods of the [XCom module](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html) ([source code](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/models/xcom.py)).

## Custom serialization and deserialization

By default, Airflow includes [serialization](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/serializers.html) methods for common object types like [JSON](https://www.json.org/json-en.html), [pandas DataFrames](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) and [NumPy](https://numpy.org/).

If you need to pass data objects through XCom that aren't supported, you have several options:

* Register a custom serializer, see [Serialization](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/serializers.html).
* Add a `serialize()` and `deserialize()` method to the class of the object you want to pass through XCom, see [Serialization](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/serializers.html).
* Use a custom XCom backend to define custom serialization and deserialization methods, see [Use a custom XCom backend class](#use-a-custom-xcom-backend-class).


# Set up a custom XCom backend using object storage
Source: https://astronomer.io/docs/learn/custom-xcom-backends-tutorial

Use this tutorial to learn how to set up a custom XCom backend with object storage.

By default, Airflow uses the [metadata database](/docs/learn/airflow-database) to store XComs, which works well for local development but has limited performance. For production environments that use XCom to pass data between tasks, Astronomer recommends using a custom XCom backend. [Custom XCom backends](/docs/learn/custom-xcom-backend-strategies) allow you to configure where Airflow stores information that is passed between tasks using [XComs](/docs/learn/airflow-passing-data-between-tasks#xcom).

The Object Storage XCom Backend available in the [Common IO provider](https://airflow.apache.org/docs/apache-airflow-providers-common-io/stable/index.html) is the easiest way to store XComs in a remote object storage solution.

This tutorial will show you how to set up a custom XCom backend using object storage for [AWS S3](https://aws.amazon.com/s3/), [GCP Cloud Storage](https://cloud.google.com/storage) or [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs/).

To learn more about other options for setting custom XCom backends, see [Strategies for custom XCom backends in Airflow](/docs/learn/custom-xcom-backend-strategies).

<Warning>
  While a custom XCom backend allows you to store virtually unlimited amounts of data as XComs, you will also need to scale other Airflow components to pass large amounts of data between tasks. For help running Airflow at scale, [contact Astronomer](https://www.astronomer.io/lp/signup/).
</Warning>

## Time to complete

This tutorial takes approximately 45 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* XCom basics. See [Passing data between Airflow tasks](/docs/learn/airflow-passing-data-between-tasks).
* Airflow connections. See [Manage connections in Apache Airflow](/docs/learn/connections).

## Prerequisites

* The [Astro CLI](https://docs.astronomer.io/astro/cli/install-cli) with an Astro project running Astro Runtime 11.5.0 or higher (Airflow 2.9.2 or higher). To set up a custom XCom backend with older versions of Airflow, see [Custom XCom backends](/docs/learn/custom-xcom-backend-strategies).
* An account in either [AWS](https://aws.amazon.com/), [GCP](https://cloud.google.com/), or [Azure](https://azure.microsoft.com/) with permissions to create and configure an object storage container.

## Step 1: Set up your object storage container

First, you need to set up the object storage container in your cloud provider where Airflow will store the XComs.

<details>
  <summary>AWS</summary>

  1. Log into your AWS account and [create a new S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). Ensure that public access to the bucket is blocked. You don't need to enable bucket versioning.

  2. [Create a new IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html) for Airflow to access your bucket. You can use the JSON configuration below or use the AWS GUI to replicate what you see in the screenshot. Replace `<your-bucket-name>` with the name of your S3 bucket.

     ```json wrap theme={null}
     {
         "Version": "2012-10-17",
         "Statement": [
             {
                 "Sid": "VisualEditor0",
                 "Effect": "Allow",
                 "Action": [
                     "s3:ReplicateObject",
                     "s3:PutObject",
                     "s3:GetObject",
                     "s3:RestoreObject",
                     "s3:ListBucket",
                     "s3:DeleteObject"
                 ],
                 "Resource": [
                     "arn:aws:s3:::<your-bucket-name>/*",
                     "arn:aws:s3:::<your-bucket-name>"
                 ]
             }
         ]
     }
     ```

  3. Save your policy under the name `AirflowXComBackendAWSS3`.

  4. [Create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) called `airflow-xcom` with the AWS credential type `Access key - Programmatic access` and [attach](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html) the `AirflowXComBackendAWSS3` policy to this user.

  5. [Create an access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) of the type `Third-party-service` for your `airflow-xcom` user. Make sure to save the Access Key ID and the Secret Access Key in a secure location to use in [Step 3](#step-3-set-up-your-airflow-connection).

  <Info>
    For other ways to set up a connection between Airflow and AWS, see the [Amazon provider](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html) documentation.
  </Info>
</details>

<details>
  <summary>GCP</summary>

  1. Log into your Google Cloud account and [create a new project](https://cloud.google.com/resource-manager/docs/creating-managing-projects).

  2. [Create a new bucket](https://cloud.google.com/storage/docs/creating-buckets) in your project with Uniform Access Control. Enforce public access prevention.

  3. [Create a custom IAM role](https://cloud.google.com/iam/docs/creating-custom-roles) called `AirflowXComBackendGCS` for Airflow to access your bucket. Assign 6 permissions:

     * `storage.buckets.list`
     * `storage.objects.create`
     * `storage.objects.delete`
     * `storage.objects.get`
     * `storage.objects.list`
     * `storage.objects.update`

  4. [Create a new service account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) called `airflow-xcom` and grant it access to your project by granting it the `AirflowXComBackendGCS` role.

  5. [Create a new key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) for your `airflow-xcom` service account and make sure to download the credentials in JSON format.

  <Info>
    For other ways to set up a connection between Airflow and Google Cloud, see the [Google provider](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html) documentation.
  </Info>
</details>

<details>
  <summary>Azure</summary>

  1. Log into your Azure account and [create a storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create). Ensure that public access to the bucket is blocked.

  2. In the storage account, [create a new container](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal).

  3. [Create a shared access token](https://learn.microsoft.com/en-us/azure/cognitive-services/Translator/document-translation/how-to-guides/create-sas-tokens?tabs=Containers) for your container.

     In the **Permissions** dropdown menu, enable the following permissions:

     * Read
     * Add
     * Create
     * Write
     * Delete
     * List

     Set the duration the token will be valid and set **Allowed Protocols** to `HTTPS only`. Provide the IP address of your Airflow instance. If you are running Airflow locally with the Astro CLI, use the IP address of your computer.

  4. Go to your Storage account and navigate to [**Access keys**](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage). Copy the **Key** and **Connection string** values and save them in a secure location to use in [step 3](#step-3-set-up-your-airflow-connection).

  <Info>
    For other ways to set up a connection between Airflow and Azure Blob Storage, see the [Microsoft Azure provider](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/connections/wasb.html) documentation.
  </Info>
</details>

## Step 2: Install the required provider packages

To use the Object Storage XCom Backend, you need to install the Common IO provider package and the provider package for your object storage container provider.

<details>
  <summary>AWS</summary>

  Add the [Common IO](https://airflow.apache.org/registry/providers/common-io/) and [Amazon](https://airflow.apache.org/registry/providers/amazon/) provider packages to your `requirements.txt` file. Note that you need to install the `s3fs` extra to use the Amazon provider package with the [object storage](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html) feature.

  ```text wrap theme={null}
  apache-airflow-providers-common-io
  apache-airflow-providers-amazon[s3fs]
  ```
</details>

<details>
  <summary>GCP</summary>

  Add the [Common IO](https://airflow.apache.org/registry/providers/common-io/) and [Google](https://airflow.apache.org/registry/providers/google/) provider packages to your `requirements.txt` file.

  ```text wrap theme={null}
  apache-airflow-providers-common-io
  apache-airflow-providers-google
  ```
</details>

<details>
  <summary>Azure</summary>

  Add the [Common IO](https://airflow.apache.org/registry/providers/common-io/) and [Microsoft Azure](https://airflow.apache.org/registry/providers/microsoft-azure/) provider packages to your `requirements.txt` file.

  ```text wrap theme={null}
  apache-airflow-providers-common-io
  apache-airflow-providers-microsoft-azure
  ```
</details>

## Step 3: Set up your Airflow connection

An Airflow connection is necessary to connect Airflow with your object storage container provider. In this tutorial, you'll use the Airflow UI to configure your connection.

1. Start your Astro project by running:

   ```bash wrap theme={null}
   astro dev start
   ```

<details>
  <summary>AWS</summary>

  2. In the Airflow UI, go to **Admin** > **Connections** and click **Create**. Fill in the following fields:

     * **Conn Id**: `my_aws_conn`
     * **Conn Type**: `Amazon Web Services`
     * **AWS Access Key ID**: `<your access key>`
     * **AWS Secret Access Key**: `<your secret key>`

     To learn more about configuration options for the AWS connection, see the [Amazon provider](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html) documentation.
</details>

<details>
  <summary>GCP</summary>

  2. In the Airflow UI, go to **Admin** > **Connections** and click **Create**. Fill in the following fields:

     * **Conn Id**: `my_gcp_conn`
     * **Conn Type**: `Google Cloud`
     * **Project Id**: `<your project id>`
     * **Keyfile JSON**: `<the contents from your keyfile JSON that you downloaded in step 1>`

     To learn more about configuration options for the Google connection, see the [Google provider](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html) documentation.
</details>

<details>
  <summary>Azure</summary>

  2. In the Airflow UI, go to **Admin** > **Connections** and click **Create**. Fill in the following fields:

     * **Conn Id**: `my_azure_conn`
     * **Conn Type**: `azure_container_volume`
     * **Subscription ID**: `<your Azure subscription ID>`
     * **Blob Storage Connection String**: `<connection string to your Azure Storage account>`

     There are many more options for connecting to Azure. To learn more, see the [Microsoft Azure provider](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/connections/wasb.html) documentation.
</details>

## Step 4: Configure your custom XCom backend

Configuring a custom XCom backend with object storage can be done by setting environment variables in your Astro project.

<Info>
  If you are setting up a custom XCom backend for an Astro deployment, you have to set the following environment variables for your deployment. See [Environment variables](https://docs.astronomer.io/astro/environment-variables) for instructions.
</Info>

1. Add the `AIRFLOW__CORE__XCOM_BACKEND` environment variable to your `.env` file. It defines the class to use for the custom XCom backend implementation.

   ```text wrap theme={null}
   AIRFLOW__CORE__XCOM_BACKEND="airflow.providers.common.io.xcom.backend.XComObjectStorageBackend"
   ```

<details>
  <summary>AWS</summary>

  2. Add the `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH` environment variable to your `.env` file to define the path in your S3 bucket where the XComs will be stored in the form of `<connection id>@<bucket name>/<path>`. Use the connection id of the Airflow connection you defined in [step 3](#step-3-set-up-your-airflow-connection) and replace `<my-bucket>` with your S3 bucket name.

     ```text wrap theme={null}
     AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH="s3://my_aws_conn@<my-bucket>/xcom"
     ```
</details>

<details>
  <summary>GCP</summary>

  2. Add the `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH` environment variable to your `.env` file to define the path in your GCS bucket where the XComs will be stored in the form of `<connection id>@<bucket name>/<path>`. Use the connection id of the Airflow connection you defined in [step 3](#step-3-set-up-your-airflow-connection) and replace `<my-bucket>` with your GCS bucket name.

     ```text wrap theme={null}
     AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH="gs://my_gcp_conn@<my-bucket>/xcom"
     ```
</details>

<details>
  <summary>Azure</summary>

  2. Add the `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH` environment variable to your `.env` file to define the path in your Azure blob container where the XComs will be stored in the form of `<connection id>@<blob name>/<path>`. Use the connection id of the Airflow connection you defined in [step 3](#step-3-set-up-your-airflow-connection) and replace `<my-blob>` with your Azure blob container name.

     ```text wrap theme={null}
     AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH="abfs://my_azure_conn@<my-blob>/xcom"
     ```
</details>

3. Add the `AIRFLOW__COMMON.IO__XCOM_OBJECTSTORAGE_THRESHOLD` environment variable to your `.env` file to determine when Airflow will store XComs in the object storage vs the metadata database. The default value is `-1` which will store all XComs in the metadata database. Set the value to `0` to store all XComs in the object storage. Any positive value means any XCom with a byte size greater than the threshold will be stored in the object storage and any XCom with a size equal to or less than the threshold will be stored in the metadata database.
   For this tutorial we will set the threshold to `1000` bytes, which means any XCom larger than 1KB will be stored in the object storage.

   ```text wrap theme={null}
   AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD="1000"
   ```

4. Optional. Define the `AIRFLOW__COMMON_IO__XCOM_OBJECTSTORE_COMPRESSION` environment variable to compress the XComs stored in the object storage with [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) supported compression algorithms like `zip`. The default value is `None`.

   ```text wrap theme={null}
   AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_COMPRESSION="zip"
   ```

5. Restart your Airflow project by running:

   ```bash wrap theme={null}
   astro dev restart
   ```

## Step 5: Test your custom XCom backend

We will use a simple DAG to test your custom XCom backend.

1. Create a new file in the `dags` directory of your Astro project called `custom_xcom_backend_test.py` and add the following code:

   ```python expandable wrap theme={null}
   """
   ## Toy DAG to show size dependant custom XCom serialization

   This DAG pushes two dicts to XCom, one below, one above 1000 bytes.
   It then pulls them and prints their sizes.
   """

   from airflow.decorators import dag, task
   from airflow.models.baseoperator import chain


   @dag(
       start_date=None,
       schedule=None,
       catchup=False,
       doc_md=__doc__,
       tags=["xcom", "2-9", "toy"],
   )
   def custom_xcom_backend_test():
       @task
       def push_objects(**context) -> None:
           """Create a small and a big dictionary, print their sizes and push them to XCom."""

           small_obj = {"a": 23}
           big_obj = {f"key{i}": "x" * 100 for i in range(100)}
           print(f"Size of small object: {small_obj.__sizeof__()}")
           print(f"Size of big object: {big_obj.__sizeof__()}")

           context["ti"].xcom_push(key="small_obj", value=small_obj)
           context["ti"].xcom_push(key="big_obj", value=big_obj)

       @task
       def pull_objects(**context) -> None:
           """Pull the small and big dictionaries from XCom and print their sizes."""

           small_obj = context["ti"].xcom_pull(task_ids="push_objects", key="small_obj")
           big_obj = context["ti"].xcom_pull(task_ids="push_objects", key="big_obj")

           print(f"Size of small object: {small_obj.__sizeof__()}")
           print(f"Size of big object: {big_obj.__sizeof__()}")

       chain(push_objects(), pull_objects())


   custom_xcom_backend_test()
   ```

2. Manually trigger the `custom_xcom_backend_test` DAG in the Airflow UI and navigate to the XCom tab of the `push_objects` task. You should see that the `small_obj` XCom shows its value, meaning it was stored in the metadata database, since it is smaller than 1KB. The `big_obj` XCom shows the path to the object in the object storage containing the serialized value of the XCom.

   <Frame>
     <img alt="XCom tab of the push_objects task showing two key-value pairs showing the &#x22;big_obj&#x22; being serialized to the custom XCom backend and the &#x22;small_obj&#x22;: a dictionary containing 'a': 23, which was stored in the metadata database." />
   </Frame>

## Conclusion

Congratulations, you learned how to set up a custom XCom backend using object storage! Learn more about other options to set up custom XCom backends in the [Strategies for custom XCom backends in Airflow](/docs/learn/custom-xcom-backend-strategies) guide.


# DAG writing best practices in Apache Airflow
Source: https://astronomer.io/docs/learn/dag-best-practices

Keep up to date with the best practices for developing efficient, secure, and scalable DAGs using Airflow. Learn about DAG design and data orchestration.

Because Airflow is 100% code, knowing the basics of Python is all it takes to get started writing DAGs. However, writing DAGs that are efficient, secure, and scalable requires some Airflow-specific finesse. In this guide, you'll learn how you can develop DAGs that make the most of what Airflow has to offer.

In general, best practices fall into one of two categories:

* DAG design
* Using Airflow as an orchestrator

For an in-depth walk through and examples of some of the concepts covered in this guide, it's recommended that you review the [DAG Writing Best Practices in Apache Airflow](https://www.astronomer.io/blog/dag-writing-best-practices-in-apache-airflow) webinar and the [GitHub repository](https://github.com/astronomer/webinar-dag-writing-best-practices) for DAG examples.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).

## Review idempotency

[Idempotency](https://en.wikipedia.org/wiki/Idempotence) is the foundation for many computing practices, including the Airflow best practices in this guide. A program is considered idempotent if, for a set input, running the program once has the same effect as running the program multiple times.

In the context of Airflow, a DAG is considered idempotent if rerunning the same DAG Run with the same inputs multiple times has the same effect as running it only once. This can be achieved by designing each individual task in your DAG to be idempotent. Designing idempotent DAGs and tasks decreases recovery time from failures and prevents data loss.

Idempotency paves the way for one of Airflow's most useful features: Retries.

## Set retries

In a distributed environment where task containers are executed on shared hosts, it's possible for tasks to be killed off unexpectedly. When this happens, you might see  a [zombie process](https://en.wikipedia.org/wiki/Zombie_process) in the Airflow logs.

You can resolve issues like zombies by using task retries. Retries can be set at different levels with the following precedence:

1. **Tasks:**  Pass the `retries` parameter to the task's Operator.
2. **DAGs:** Include `retries` in a DAG's `default_args` object.
3. **Deployments:** Set the environment variable `AIRFLOW__CORE__DEFAULT_TASK_RETRIES`.

Setting retries to `2` will protect a task from most problems common to distributed environments. For more on using retries, see [Rerun DAGs and Tasks](/docs/learn/rerunning-dags).

## DAG design

The following DAG design principles will help to make your DAGs idempotent, efficient, and readable.

### Keep tasks atomic

When organizing your pipeline into individual tasks, each task should be responsible for one operation that can be re-run independently of the others. In an atomized task, a success in part of the task means a success of the entire task.

For example, in an ETL pipeline you would ideally want your Extract, Transform, and Load operations covered by three separate tasks. Atomizing these tasks allows you to rerun each operation in the pipeline independently, which supports idempotence.

### Use template fields, variables, and macros

By using templated fields in Airflow, you can pull values into DAGs using environment variables and Jinja templating. Compared to using Python functions, using templated fields helps keep your DAGs idempotent and ensures you aren't executing functions on every Scheduler heartbeat. See [Avoid top level code in your DAG file](#avoid-top-level-code-in-your-dag-file).

Contrary to our best practices, the following example defines variables based on `datetime` Python functions:

```python wrap theme={null}
# Variables used by tasks
# Bad example - Define today's and yesterday's date using datetime module
today = datetime.today()
yesterday = datetime.today() - timedelta(1)
```

If this code is in a DAG file, these functions are executed on every Scheduler heartbeat, which may not be performant. Even more importantly, this doesn't produce an idempotent DAG. You can't rerun a previously failed DAG run for a past date because `datetime.today()` is relative to the current date, not the DAG execution date.

A better way of implementing this is by using an Airflow variable:

```python wrap theme={null}
# Variables used by tasks
# Good example - Fetch the start timestamp of the previous successful DAG run
yesterday = {{ prev_start_date_success }}
```

You can use one of the Airflow built-in [variables and macros](https://airflow.apache.org/docs/apache-airflow/stable/macros-ref.html), or you can create your own templated field to pass information at runtime. For more information on this topic, see [templating and macros in Airflow](/docs/learn/templating).

### Incremental record filtering

You should break out your pipelines into incremental extracts and loads wherever possible. For example, if you have a DAG that runs hourly, each DAG run should process only records from that hour, rather than the whole dataset. When the results in each DAG run represent only a small subset of your total dataset, a failure in one subset of the data won't prevent the rest of your DAG Runs from completing successfully. If your DAGs are idempotent, you can rerun a DAG for only the data that failed rather than reprocessing the entire dataset.

There are multiple ways you can achieve incremental pipelines.

#### Last modified date

Using a last modified date is recommended for incremental loads. Ideally, each record in your source system has a column containing the last time the record was modified. With this design, a DAG run looks for records that were updated within specific dates from this column.

For example, with a DAG that runs hourly, each DAG run is responsible for loading any records that fall between the start and end of its hour. If any of those runs fail, it doesn't affect other Runs.

#### Sequence IDs

When a last modified date is unavailable, a sequence or incrementing ID can be used for incremental loads. This logic works best when the source records are only being appended to and not updated. Although implementing a last modified date system in your records is considered best practice, basing your incremental logic off of a sequence ID can be a sound way to filter pipeline records without a last modified date.

### Avoid top-level code in your DAG file

In the context of Airflow, top-level code refers to any code that is run at the time the DAG is parsed, as opposed to the time the task is run.

Code that is part of an operator or a decorated task is run by Airflow only when the task runs, not when the DAG is parsed. For example, in the following code, `call_external_systems()` is considered top-level code because it runs when the DAG is parsed. `x + y` isn't top-level code, because it is part of the task definition and only runs when the task runs.

```python wrap theme={null}
@dag(...)
def the_dag():

    @task
    def do_thing():
        x + y

    num_of_things = call_external_system()    # this is "top level code"

    chain(do_thing() for _ in range(num_of_things))

the_dag()

```

Generally, any code that isn't part of your DAG or operator instantiations and that makes requests to external systems is of concern. Airflow executes all code in the `dags_folder` on every [`min_file_process_interval`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#min-file-process-interval), which defaults to 30 seconds. Therefore, any code that is run when the DAG is parsed and makes requests to external systems, like an API or a database, or makes function calls outside of your tasks can cause performance issues since these requests and connections are being made every 30 seconds rather than only when the DAG is scheduled to run.

To see another example, the following DAG example dynamically generates tasks using the `PostgresOperator` based on records pulled from a different database.

In the **Bad practice** example the connection to the other database is made outside of an operator instantiation as top-level code. When the scheduler parses this DAG, it will use the `hook` and `result` variables to query the `grocery_list` table. This query is run every time the DAG is parsed, which can cause performance issues.

The version shown under the **Good practice** DAG wraps the connection to the database into its own task, the `get_list_of_results` task. Now the connection is only made at when the DAG actually runs, preventing performance issues.

<details>
  <summary>Bad Practice</summary>

  ```python expandable wrap theme={null}
  """WARNING: This DAG is used as an example for _bad_ Airflow practices. Do not
  use this DAG."""

  from airflow.decorators import dag
  from airflow.providers.postgres.operators.postgres import PostgresOperator
  from airflow.providers.postgres.hooks.postgres import PostgresHook
  from pendulum import datetime

  # Bad practice: top-level code in a DAG file
  hook = PostgresHook("database_conn")
  results = hook.get_records("SELECT * FROM grocery_list;")

  sql_queries = []

  for result in results:
      grocery = result[0]
      amount = result[1]
      sql_query = f"INSERT INTO purchase_order VALUES ('{grocery}', {amount});"

      sql_queries.append(sql_query)


  @dag(
      start_date=datetime(2023, 1, 1), max_active_runs=3, schedule="@daily", catchup=False
  )
  def bad_practices_dag_1():
      insert_into_purchase_order_postgres = PostgresOperator.partial(
          task_id="insert_into_purchase_order_postgres",
          postgres_conn_id="postgres_default",
      ).expand(sql=sql_queries)


  bad_practices_dag_1()
  ```
</details>

<details>
  <summary>Good Practice</summary>

  ```python expandable wrap theme={null}
  from airflow.decorators import dag, task
  from airflow.providers.postgres.operators.postgres import PostgresOperator
  from airflow.providers.postgres.hooks.postgres import PostgresHook
  from pendulum import datetime


  @dag(
      start_date=datetime(2023, 1, 1), max_active_runs=3, schedule="@daily", catchup=False
  )
  def good_practices_dag_1():
      @task
      def get_list_of_results():
          # good practice: wrap database connections into a task
          hook = PostgresHook("database_conn")
          results = hook.get_records("SELECT * FROM grocery_list;")
          return results

      @task
      def create_sql_query(result):
          grocery = result[0]
          amount = result[1]
          sql = f"INSERT INTO purchase_order VALUES ('{grocery}', {amount});"
          return sql

      sql_queries = create_sql_query.expand(result=get_list_of_results())

      insert_into_purchase_order_postgres = PostgresOperator.partial(
          task_id="insert_into_purchase_order_postgres",
          postgres_conn_id="postgres_default",
      ).expand(sql=sql_queries)


  good_practices_dag_1()
  ```
</details>

### Treat your DAG file like a config file

Including code that isn't part of your DAG or operator instantiations in your DAG file makes the DAG harder to read, maintain, and update. When possible, leave all of the heavy lifting to the hooks and operators that you instantiate within the file. If your DAGs need to access additional code such as a SQL script or a Python function, consider keeping that code in a separate file that can be read into a DAG run.

### Use a consistent method for task dependencies

In Airflow, task dependencies can be set multiple ways. You can use `set_upstream()` and `set_downstream()` functions, or you can use `<<` and `>>` operators. Which method you use is a matter of personal preference, but for readability it's best practice to choose one method and use it consistently.

For example, instead of mixing methods like this:

```python wrap theme={null}
task_1.set_downstream(task_2)
task_3.set_upstream(task_2)
task_3 >> task_4
```

Try to be consistent with something like this:

```python wrap theme={null}
task_1 >> task_2 >> [task_3, task_4]
```

## Use Airflow features

To get the most out of Airflow, use built-in features and the broader Airflow ecosystem, namely provider packages for third-party integrations, to fulfill specific use cases. Using Airflow in this way makes it easier to scale and pull in the right tools based on your needs.

### Make use of provider packages

One of the best aspects of Airflow is its robust and active community, which has resulted in integrations between Airflow and other tools known as [provider packages](https://airflow.apache.org/docs/apache-airflow-providers/).

Provider packages let you orchestrate third party data processing jobs directly from Airflow. Wherever possible, it's recommended that you make use of these integrations rather than writing Python functions yourself. This makes it easier for organizations using existing tools to adopt Airflow, and you don't have to write new code.

For more information about the available provider packages, see the [Airflow Registry](https://airflow.apache.org/registry/).

### Decide where to run data processing jobs

There are many options available for implementing data processing. For small to medium scale workloads, it is typically safe to do your data processing within Airflow as long as you allocate enough resources to your Airflow infrastructure. Large data processing jobs are typically best offloaded to a framework specifically optimized for those use cases, such as [Apache Spark](https://spark.apache.org/). You can then use Airflow to orchestrate those jobs.

Astronomer recommends that you consider the size of your data now and in the future when deciding whether to process data within Airflow or offload to an external tool. Follow these recommendations if your use case is well suited to processing data within Airflow:

* Ensure your Airflow infrastructure has the necessary resources.
* Use the Kubernetes Executor to isolate task processing and have more control over resources at the task level.
* Use a [custom XCom backend](/docs/learn/custom-xcom-backend-strategies) or intermediary data storage if you need to pass any data between the tasks so you don't overload your metadata database.

## Other best practices

Here are a few other noteworthy best practices that you should follow.

### Use a consistent file structure

Having a consistent file structure for Airflow projects keeps things organized and easy to adopt. This is the structure that Astronomer uses:

```bash wrap theme={null}
├── dags/ # Where your DAGs go
│   └── example-dag.py # An example dag that comes with the initialized project
├── Dockerfile # For Astronomer's Docker image and runtime overrides
├── include/ # For any other files you'd like to include
├── plugins/ # For any custom or community Airflow plugins
├── packages.txt # For OS-level packages
└── requirements.txt # For any Python packages

```


# Use DAG Factory to create dags
Source: https://astronomer.io/docs/learn/dag-factory

Learn how to dynamically convert YAML files into Apache Airflow® dags with DAG Factory, an open source project that makes creating dags easy.

[DAG Factory](https://astronomer.github.io/dag-factory/latest/) is an open source tool managed by Astronomer that allows you to [dynamically generate](/docs/learn/dynamically-generating-dags) [Apache Airflow®](https://airflow.apache.org/) dags from [YAML](https://yaml.org/). While Airflow dags are traditionally written exclusively in Python, DAG Factory makes it easy for people who don't know Python to use Airflow.

This guide provides a complete walkthrough of using DAG Factory package to build production-ready pipelines in a modern Airflow project. You will learn to install the library, structure your project according to best practices, and define a multi-task pipeline entirely in YAML. The example demonstrates powerful features like using the TaskFlow API, organizing tasks with task groups, and passing data between tasks, all from your configuration file. By the end, you'll be ready to apply these patterns to your own dynamic dags.

DAG Factory can be used with all Astronomer products and any Apache Airflow installation. To view the source code of the project, have a look at the [dag-factory](https://github.com/astronomer/dag-factory) GitHub repository.

## When to use DAG Factory

While writing dags directly in Python is powerful and flexible, it's not always the most efficient approach for every use case. DAG Factory offers a configuration-driven alternative where you define the structure of your pipelines in YAML. This is particularly useful in several key scenarios:

<CardGroup>
  <Card title="Empowering Teams" icon="users">
    YAML is often more approachable than Python. DAG Factory allows team members like analysts or junior engineers, who may not be Airflow experts, to create and manage their own dags with a simplified, declarative syntax.
  </Card>

  <Card title="Standardizing Repetitive Dags" icon="clone">
    If you have dozens of dags that follow the same pattern (like a standard extract-and-load job), DAG Factory is ideal. You can create a standard template and then generate numerous dags just by changing the parameters in a YAML file, which reduces code duplication and simplifies maintenance.
  </Card>

  <Card title="Separating Logic from Structure" icon="layer-group">
    DAG Factory helps you separate the *what* from the *how*. The YAML clearly defines the dag's structure and dependencies, while the underlying Python functions handle the actual business logic. This makes your dags easier to read at a glance and your Python code more modular and testable.
  </Card>
</CardGroup>

While DAG Factory offers significant advantages for some use cases, **there are scenarios when using other ways of dag authoring like writing dags directly in Python are more appropriate**.

When your data pipelines require complex conditional logic, branching, or sophisticated error handling that goes beyond what YAML can express cleanly, native Python is generally the better approach. Additionally, YAML-based dags can be more challenging to debug compared to native Python code, as you are missing extensive logging or step-through debugging capabilities. Finally, your way of orchestrating workflows should match your team environment, so consider existing expertise.

<Note>
  While DAG Factory is a flexible product that supports all the main concepts of Airflow, newer features like [asset-aware scheduling](/docs/learn/airflow-datasets) may work but aren't as user-friendly or as well integrated as others.
</Note>

## Assumed knowledge

To get the most out of this tutorial, you should have an understanding of:

* The [Airflow components](/docs/learn/airflow-components) and how they work together.
* [Airflow fundamentals](/docs/learn/get-started-with-airflow), such as writing dags and defining tasks.
* Basic understanding of [Airflow operators](/docs/learn/what-is-an-operator).

## Prerequisites

* Python 3.9.0+
* The [Astro CLI](/docs/cli/v1.43/install-cli)

## Step 1: Initialize your Airflow project with the Astro CLI

First, create a new project directory and initialize an Astro project using the [Astro CLI](/docs/cli/v1.43/install-cli).

```bash wrap theme={null}
mkdir my-dag-factory-project && cd my-dag-factory-project
astro dev init
```

The `init` command creates a standard Airflow project structure. Since this tutorial focuses on DAG Factory, let's remove the example dag that's included by default.

```bash wrap theme={null}
rm dags/exampledag.py
```

Next, add the `dag-factory` library as a project dependency. Open `requirements.txt` and add the following line:

```text wrap theme={null}
dag-factory==1.0.1
```

Now, start your local Airflow environment. The Astro CLI will build your project, installing `dag-factory` in the process.

```bash wrap theme={null}
astro dev start
```

Once the project is running, the Airflow UI should open automatically at `http://localhost:8080` and you will be presented with an empty dags list.

## Step 2: Organize the project

A key to building a maintainable and performant Airflow project is proper organization. While you could put all your YAML configs, Python scripts, and SQL files into the `dags/` folder, this can quickly become messy and put unnecessary strain on the dag processor.

<Tip>
  Astronomer recommends placing Python, SQL, and other scripts that aren't dag definitions in the `include/` folder. Files in this folder are available to your dags but aren't parsed by the Airflow dag processor, which reduces overhead and improves performance.
</Tip>

For this tutorial, we'll use a structure that is also a great starting point for real-world projects:

* `dags/`: This folder will contain only the YAML configuration files and the Python script that generates the dags from them. This keeps all dag definitions encapsulated.
* `include/`: We will create a `tasks` subfolder here to hold the Python functions that our operators will call. Any other supporting scripts (for example, SQL queries) would also live in sub-folders within `include/`.

We will apply this principle in the next steps.

<Tip>
  For larger projects with a mix of dynamically generated and standard Python dags, consider organizing further. For example, you could create a `dags/configs` subfolder to hold all your DAG Factory YAML files, keeping them separate from your other `.py` dag files.
</Tip>

To separate our business logic from the rest of our orchestration logic, create a new folder named `tasks` inside `include`. There, we'll add Python scripts that define the functions our YAML-based pipelines will call in the next steps.

```bash wrap theme={null}
mkdir -p include/tasks
```

## Step 3: Prepare functions

Our example dag will orchestrate a simple pipeline using both a `PythonOperator` using the [TaskFlow API](/docs/learn/airflow-decorators) and a `BashOperator` using the [traditional operator](/docs/learn/what-is-an-operator).

<Note>
  DAG Factory supports both traditional operators and the modern TaskFlow API. This tutorial targets Airflow 3.x and will use the TaskFlow API decorator syntax whenever possible.
</Note>

Before defining the dag in YAML, let's write the Python functions that our tasks will execute. Following our plan from Step 2, we'll place these functions in the `include/tasks/` folder.

Create a file named `include/tasks/basic_example_tasks.py` with the following content:

```python wrap theme={null}
def _extract_data() -> list[int]:
    return [1, 2, 3, 4]

def _store_data(processed_at: str, data_a: list[int], data_b: list[int]) -> None:
    print(f"Storing {len(data_a + data_b)} records at {processed_at}")
```

<Tip>
  Design your Python functions to be small, self-contained, and independently testable, which aligns with best practices for both DAG Factory and general Airflow development.
</Tip>

## Step 4: Define a basic dag in YAML

Now we can create the YAML definition for our dag. Create a new YAML file in the `dags` folder named `basic_example.yml` and add the following content:

```yaml expandable wrap theme={null}
basic_example_dag:
  default_args:
    owner: "astronomer"
    start_date: 2025-09-01
  description: "Basic example DAG"
  tags: ["demo", "etl"]
  schedule: "@hourly"

  task_groups:
    extract:
      tooltip: "data extraction"

  tasks:
    extract_data_from_a:
      decorator: airflow.sdk.task
      python_callable: include.tasks.basic_example_tasks._extract_data
      task_group_name: extract

    extract_data_from_b:
      decorator: airflow.sdk.task
      python_callable: include.tasks.basic_example_tasks._extract_data
      task_group_name: extract

    store_data:
      decorator: airflow.sdk.task
      python_callable: include.tasks.basic_example_tasks._store_data
      processed_at: "{{ logical_date }}"
      data_a: +extract_data_from_a
      data_b: +extract_data_from_b
      dependencies: [extract]

    validate_data:
      operator: airflow.providers.standard.operators.bash.BashOperator
      bash_command: "echo data is valid"
      dependencies: [store_data]
```

This YAML file defines the dag's structure and its tasks. Note how `+extract_data_from_a` and `+extract_data_from_b` are used to pass the return value of the `extract` tasks to the `store_data` task, and how Jinja templating (`{{ logical_date }}`) is used to pass the logical date.

## Step 5: Implement the generator script

The final step to make our dag appear, is to create the Python script that Airflow will parse. This script uses the DAG Factory library to find our YAML file and generate the actual Airflow dag object from it. This approach gives you full control over the generation process and allows for extensive customization in advanced use cases.

Create a Python file named `dags/basic_example_dag_generation.py` with the following content:

```python wrap theme={null}
import os
from pathlib import Path

from dagfactory import load_yaml_dags

DEFAULT_CONFIG_ROOT_DIR = "/usr/local/airflow/dags/"
CONFIG_ROOT_DIR = Path(os.getenv("CONFIG_ROOT_DIR", DEFAULT_CONFIG_ROOT_DIR))

config_file = str(CONFIG_ROOT_DIR / "basic_example.yml")

load_yaml_dags(
    globals_dict=globals(),
    config_filepath=config_file,
)
```

Once the dag processor parses this file, your dag with the ID `basic_example_dag` will appear in the UI. It has 4 tasks in its pipeline, 2 of them within a task group:

* `extract_data`: Uses the TaskFlow API to call the `_extract_data` function from our `include/tasks/basic_example_tasks.py` script. We create 2 different tasks in this scenario. Both return a list of numbers.
* `store_data`: Uses the TaskFlow API to call the `_store_data` function from our `include/tasks/basic_example_tasks.py` script. To pass parameters with this approach, just set them with the appropriate name directly in the YAML configuration. With `+extract_data` we tell DAG Factory to reference the return value of the `extract_data` task. Also as shown in the example, you can use Jinja templating including [variables, macros and filters](https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html).
* `validate_data`: Here, we use the classic approach to use the `BashOperator`, just printing the sentence *data is valid*.

<Note>
  The `load_yaml_dags` function is responsible for generating the dags. You can point it to a specific file, or to a folder that it will scan recursively for `.yml` or `.yaml` files. It uses the provided `globals_dict` to add the generated dags to the Airflow context. For more options, see the [official documentation](https://astronomer.github.io/dag-factory/latest/configuration/load_yaml_dags/).
</Note>

<Frame>
  <img alt="Basic generated dag example" />
</Frame>

With this, you already know the basics of how to orchestrate a dag with YAML, including using task groups, passing data between tasks, using the TaskFlow API and classic operators, and setting basic dag attributes like the schedule or tags.

<Tip>
  Dags defined with DAG Factory automatically receive the `dagfactory` tag. Also, if you select **Dag Docs** from the individual dag view, it will by default show the YAML file that created the dag, which is very useful for debugging.
</Tip>

<Frame>
  <img alt="Dag docs showing YAML definition" />
</Frame>

This is a great starting point, and the following steps will cover more advanced features to prepare your DAG Factory knowledge for real-world use cases.

## (Optional) Step 6: Asset-aware scheduling with YAML

Now, let's explore one of Airflow's most powerful features, [asset-aware scheduling](/docs/learn/airflow-datasets), and how to implement it using DAG Factory. We will create two dags: a producer that updates an asset, and a consumer that runs whenever that asset is updated.

First, let's create the Python functions that our tasks will execute. These functions will fetch data from an API, save it to a file, and then read it back.

Create a new file named `include/tasks/asset_example_tasks.py` with the following content:

<Accordion title="`include/tasks/asset_example_tasks.py`">
  ```python wrap theme={null}
  import json
  import tempfile
  import requests

  def _get_iss_coordinates_file_path() -> str:
      return tempfile.gettempdir() + "/iss_coordinates.txt"

  def _update_iss_coordinates() -> None:
      placeholder = {"latitude": "0.0", "longitude": "0.0"}

      try:
          response = requests.get("http://api.open-notify.org/iss-now.json", timeout=5)
          response.raise_for_status()
          data = response.json()
          coordinates = data.get("iss_position", placeholder)
      except Exception:
          coordinates = placeholder

      with open(_get_iss_coordinates_file_path(), "w") as f:
          f.write(json.dumps(coordinates))

  def _read_iss_coordinates() -> None:
      path = _get_iss_coordinates_file_path()
      with open(path, "r") as f:
          print("::group::ISS Coordinates")
          print(f.read())
          print("::endgroup::")
  ```
</Accordion>

The `_update_iss_coordinates` function retrieves data from an API and writes it to a file, while `_read_iss_coordinates` reads this file and prints the content to a dedicated log group.

Now that we have our Python logic, we can define the two dags that will orchestrate it. Create a new YAML file at `dags/asset_example.yml`:

```yaml wrap theme={null}
default:
  start_date: 2025-09-01

update_iss_coordinates:
  schedule: "@daily"
  tasks:
    update_coordinates:
      decorator: airflow.sdk.task
      python_callable: include.tasks.asset_example_tasks._update_iss_coordinates
      outlets:
        - __type__: airflow.sdk.Asset
          name: "iss_coordinates"

process_iss_coordinates:
  schedule:
    - __type__: airflow.sdk.Asset
      name: "iss_coordinates"
  tasks:
    read_coordinates:
      decorator: airflow.sdk.task
      python_callable: include.tasks.asset_example_tasks._read_iss_coordinates
```

This single YAML file defines both the `update_iss_coordinates` (*producer*) and `process_iss_coordinates` (*consumer*) dags. For the producing dag we define an outlet of type `airflow.sdk.Asset` and name it `iss_coordinates`. The consuming dag then uses this same asset identifier for its `schedule` attribute, which creates the dependency.

Also, take note of the YAML top-level `default` block. **This configuration affects all the dags defined in the YAML file**, allowing you to share standard settings and configurations, for improved consistency, maintainability and simplicity.

Finally, to generate these dags in Airflow, we need to create a corresponding generator script. Create a new file named `dags/asset_example_dag_generation.py` with the following content:

<Accordion title="`dags/asset_example_dag_generation.py`">
  ```python wrap theme={null}
  import os
  from pathlib import Path

  from dagfactory import load_yaml_dags

  DEFAULT_CONFIG_ROOT_DIR = "/usr/local/airflow/dags/"
  CONFIG_ROOT_DIR = Path(os.getenv("CONFIG_ROOT_DIR", DEFAULT_CONFIG_ROOT_DIR))

  config_file = str(CONFIG_ROOT_DIR / "asset_example.yml")

  load_yaml_dags(
      globals_dict=globals(),
      config_filepath=config_file,
  )
  ```
</Accordion>

And that's it! Once the scheduler parses this file, you will see two new dags in the Airflow UI, connected by the `iss_coordinates` asset. When you run the `update_iss_coordinates` dag, the `process_iss_coordinates` dag will be triggered automatically upon its completion.

<Frame>
  <img alt="Asset consumer task logs" />
</Frame>

<Tip>
  For a simpler approach to creating one dag with one task updating an asset, you could use [@asset syntax](/docs/learn/airflow-datasets#asset-definition), adding `@asset(schedule="@daily")` directly to the `_update_iss_coordinates` function in your Python file. This would allow you to remove the `update_iss_coordinates` dag definition from your YAML entirely. This tutorial defines both in YAML to fully demonstrate how DAG Factory handles asset producers and consumers.
</Tip>

## (Optional) Step 7: Alternative YAML loading

In the previous steps, we used a dedicated Python dag generation script for each dag to parse the YAML with DAG Factory. This is a useful approach for maximum control over the generation process, and to avoid any unexpected workload when teams work with many YAML files.

However, it also adds complexity. The `load_yaml_dags` function therefore also supports a more pragmatic way, to parse all YAML files in your dags folder recursively. To illustrate this, delete the two generator scripts `dags/basic_example_dag_generation.py` and `dags/asset_example_dag_generation.py`. Then create a new file `dags/dag_generation.py`:

```python wrap theme={null}
# keep import to ensure the dag processor parses the file
from airflow.sdk import dag
from dagfactory import load_yaml_dags
load_yaml_dags(globals_dict=globals())
```

In this particular case, we need to add the dag import as an indicator for Airflow, to not skip this file during parsing. You will notice, the result will be the same as before, and all additional YAML files added will now be automatically processed.

<Tip>
  When searching for dags inside the dag bundle, Airflow only considers Python files that contain the strings `airflow` and `dag` (case-insensitively) as an optimization. Because of these optimizations, you might need to add the `dag` import to ensure your file is parsed. To consider all Python files instead, disable the `DAG_DISCOVERY_SAFE_MODE` configuration flag.
</Tip>

<Tip>
  In case you want to outsource your YAML definitions, you can overwrite the `dags_folder` argument when calling the `load_yaml_dags` function to set a custom folder to process recursively.
</Tip>

## (Optional) Step 8: Configuration and inheritance

As you create more dags, you'll want to avoid repeating the same configuration. DAG Factory includes powerful features for centralized configuration and inheritance to help you keep your dag definitions clean, consistent, and easy to maintain across your project.

This feature allows you to set default values for both dag-level arguments (like `schedule`) and task-level arguments (like `retries` using `default_args`).

In our `dags/asset_example.yml` file, you already discovered one way to configure dags in a centralized way within the YAML definition:

```yaml wrap theme={null}
default:
  start_date: 2025-09-01

update_iss_coordinates:
  # ...

process_iss_coordinates:
  # ...
```

With this approach both dags, `update_iss_coordinates` and `process_iss_coordinates`, will use the `start_date` from the `default` block. This feature becomes even more powerful, when using global defaults in combination with inheritance.

To illustrate this, let's imagine a real-world scenario with a set of company-wide data pipeline standards:

* All dags should have a default `start_date` of `2025-09-01`.
* All dags should be owned by `astronomer`, unless they belong to a specific department.
* All tasks should have 2 retries by default.
* The default schedule for all dags should be daily at midnight (`@daily`), unless specified otherwise.

DAG Factory automatically looks for a file named `defaults.yml` in your dags folder and applies its configuration to all dags within that folder and its subfolders. This creates a single source of truth for your global defaults.

<Note>
  `load_yaml_dags` uses the same default path for both the configurations and the YAML files: the path set as `dags_folder`. You can override only the path where DAG Factory looks for configurations, by setting the `defaults_config_path` parameter.
</Note>

To implement our company standards, create a new file at `dags/defaults.yml` with the following content:

```yml wrap theme={null}
schedule: "@daily"  # dag-specific arguments at root level

default_args:
  owner: "astronomer"
  retries: 2
```

The real power of this feature comes from inheritance. DAG Factory applies `defaults.yml` files hierarchically. A `defaults.yml` in a subfolder will inherit from its parent and can override any of the parent's settings.

Let's apply this to our scenario. We want to override the default `owner` for our Marketing and Finance departments, and also change the default `schedule` just for the Marketing department, to run dags at 1 AM rather than midnight for this department.

First, let's create the folder structure:

```text wrap theme={null}
airflow
└── dags
    ├── defaults.yml
    ├── marketing
    │   ├── defaults.yml
    │   └── marketing_dag.yml
    └── finance
        ├── defaults.yml
        └── finance_dag.yml
```

Now, create `dags/marketing/defaults.yml` to set a new `schedule` and `owner`:

```yml wrap theme={null}
schedule: "0 1 * * *"

default_args:
  owner: "astronomer-marketing"
```

And for the Finance department, create `dags/finance/defaults.yml` to override only the `owner`:

```yml wrap theme={null}
default_args:
  owner: "astronomer-finance"
```

Now that our defaults are in place, creating the actual dags is incredibly simple and clean.

Create `dags/marketing/marketing_dag.yml`:

```yml wrap theme={null}
marketing_dag:
  tasks:
    some_process:
      operator: airflow.providers.standard.operators.bash.BashOperator
      bash_command: "echo processing data"
```

And similarly, create `dags/finance/finance_dag.yml`:

```yml wrap theme={null}
finance_dag:
  tasks:
    some_process:
      operator: airflow.providers.standard.operators.bash.BashOperator
      bash_command: "echo processing data"
```

Notice how concise these definitions are. We don't need to specify `start_date`, `owner`, or `retries` because they are all handled by our layered `defaults.yml` files. This allows you to write minimal dag configurations while maintaining centralized control over your project's standards.

In the Airflow UI, you will see two new dags, each with a different set of inherited properties:

* `marketing_dag`: Inherits the `schedule` (`0 1 * * *`) and `owner` (`astronomer-marketing`) from its local `defaults.yml`, and `retries` from the global `defaults.yml`.
* `finance_dag`: Inherits the `owner` (`astronomer-finance`) from its local `defaults.yml`, and both the `schedule` (`@daily`) and `retries` from the global `defaults.yml`.

<Frame>
  <img alt="Dags with inherited properties" />
</Frame>

<Note>
  If any `defaults.yml` files are inside your `dag_folder`, DAG Factory might try to parse them as dags, which can cause errors in your task logs. To prevent this, keep `dags_folder` and `defaults_config_path` separate. Configuration inheritance still works as expected, and these errors are non-critical.
</Note>

## Advanced usage: Dynamic task mapping

DAG Factory also supports [dynamic task mapping](/docs/learn/dynamic-tasks), to dynamically generate parallel tasks at runtime. The following example shows how to apply this principle using the TaskFlow API. Let's assume we have the following Python functions defined in `include/tasks/dtm_tasks.py`:

```python wrap theme={null}
def _generate_data():
    return [1, 2, 3, 4, 5]

def _process_data(processing_date, value):
    print(f"Processing {value} at {processing_date}")
```

We can now simply reference arguments under `partial` and `expand` in our YAML, to let DAG Factory apply dynamic task mapping:

```yaml wrap theme={null}
dtm_example:
  default_args:
    owner: "astronomer"
    start_date: 2025-09-01
  schedule: "@hourly"

  tasks:
    generate_data:
      decorator: airflow.sdk.task
      python_callable: include.tasks.dtm_tasks._generate_data

    process_data:
      decorator: airflow.sdk.task
      python_callable: include.tasks.dtm_tasks._process_data
      partial:
        processing_date: "{{ logical_date }}"
      expand:
        value: +generate_data
      dependencies: [generate_data]
```

With this, we will use the output of `generate_data` to generate parallel task instances.

<Frame>
  <img alt="YAML-generated dynamic tasks." />
</Frame>

## Advanced usage: Dynamic YAML generation

The examples above show how to use DAG Factory to create dags based on static YAML files. For use cases where you'd like to create several dags with a similar structure it is possible to create them [dynamically](/docs/learn/dynamically-generating-dags) based on a template YAML file to avoid code duplication.

Creating a dag dynamically with DAG Factory simply means that you use Python code to create the YAML configurations instead of writing them manually.

There are two files that you need:

* A **template YAML file** that contains the structure of the dags you want to create with placeholders for the values that will change.
* A **Python script** that creates DAG Factory YAML file by replacing the placeholders in the template YAML file with the actual values.

Since Airflow uses Jinja2 internally already, we can use this library for a more robust generation process.

The template YAML file provides the structure for all the dags you will generate dynamically with placeholders for values that vary in between the dags. Create a file called `include/template.yml`:

<Accordion title="`include/template.yml`">
  ```text wrap theme={null}
  {{ dag_id }}:
    schedule: "{{ schedule }}"
    tasks:
      task_1:
        operator: airflow.providers.standard.operators.bash.BashOperator
        bash_command: "{{ bash_command_task_1 }}"
      task_2:
        operator: airflow.providers.standard.operators.bash.BashOperator
        bash_command: "{{ bash_command_task_2 }}"
        dependencies: [task_1]
  ```
</Accordion>

The Python script reads the template YAML file, replaces the placeholders with the actual values, and writes the resulting YAML files to the `dags` directory. Place this script in the top-level of your project for now. You can run this script manually to generate your dags for local development or automatically as part of your CI/CD pipeline.

<Accordion title="`generate_yaml.py`">
  ```python expandable wrap theme={null}
  from pathlib import Path
  import yaml
  from jinja2 import Environment, FileSystemLoader

  TEMPLATE_DIR = "include"
  TEMPLATE_NAME = "template.yml"
  OUTPUT_FILE = "dags/dynamic_dags.yml"
  TEMPLATE_VARIABLES = [{
      "dag_id": "example_1",
      "schedule": "@daily",
      "bash_command_task_1": "echo task 1 from example 1",
      "bash_command_task_2": "echo task 2 from example 1",
  }, {
      "dag_id": "example_2",
      "schedule": "@weekly",
      "bash_command_task_1": "echo task 1 from example 2",
      "bash_command_task_2": "echo task 2 from example 2",
  }]

  def generate_dags_from_template():
      # setup Jinja2
      env = Environment(loader=FileSystemLoader(TEMPLATE_DIR), autoescape=True)
      template = env.get_template(TEMPLATE_NAME)

      # render dags from template
      all_dags = {}
      for variables in TEMPLATE_VARIABLES:
          rendered_yaml_str = template.render(variables)
          dag_config = yaml.safe_load(rendered_yaml_str)
          all_dags.update(dag_config)

      # write to file
      output_path = Path(OUTPUT_FILE)
      with open(output_path, "w") as f:
          yaml.dump(all_dags, f, sort_keys=False)

      print(f"Successfully generated {len(TEMPLATE_VARIABLES)} dags into {OUTPUT_FILE}")

  if __name__ == "__main__":
      generate_dags_from_template()
  ```
</Accordion>

As a result, you will see the dynamically generated `dags/dynamic_dags.yml` file:

<Accordion title="`dags/dynamic_dags.yml`">
  ```yaml wrap theme={null}
  example_1:
    schedule: '@daily'
    tasks:
      task_1:
        operator: airflow.providers.standard.operators.bash.BashOperator
        bash_command: echo task 1 from example 1
      task_2:
        operator: airflow.providers.standard.operators.bash.BashOperator
        bash_command: echo task 2 from example 1
        dependencies:
        - task_1
  example_2:
    schedule: '@weekly'
    tasks:
      task_1:
        operator: airflow.providers.standard.operators.bash.BashOperator
        bash_command: echo task 1 from example 2
      task_2:
        operator: airflow.providers.standard.operators.bash.BashOperator
        bash_command: echo task 2 from example 2
        dependencies:
        - task_1
  ```
</Accordion>

## Conclusion

In this tutorial, you've journeyed from defining a single dag in a YAML file to building a complete framework for dynamically generating your pipelines.

You've learned how to:

* Define dags, tasks, and task groups using a simple, declarative syntax.
* Pass data between tasks and use the TaskFlow API.
* Implement Airflow features like asset-aware scheduling and dynamic task mapping.
* Manage configuration at scale using hierarchical `defaults.yml` files for inheritance.
* Dynamically generate your YAML configurations using a templating engine.

Whether your goal is to empower analysts, standardize repetitive ETL jobs, or simply separate your pipeline's structure from its logic, DAG Factory provides a robust, configuration-driven approach to Airflow development.

To continue your journey, explore the official [DAG Factory repository](https://github.com/astronomer/dag-factory/tree/main/dev/dags), which contains many more examples and advanced use cases. You now have all the tools to start building your own dynamic dags.


# Introduction to Apache Airflow® Dags
Source: https://astronomer.io/docs/learn/dags

Learn how to write Dags and get tips on how to define an Apache Airflow® Dag in Python. Learn all about Dag parameters and their settings.

In [Apache Airflow®](https://airflow.apache.org/), a **Dag** is a data pipeline or workflow. Dags are the main organizational unit in Airflow; they contain a collection of tasks and dependencies that you want to execute on a schedule.

Without a Dag, pipeline steps run independently with no awareness of each other. If an extraction step fails, downstream transformations might still run on stale or missing data. Dags solve this by defining explicit dependencies between tasks, so failures halt dependent steps and alert you to the problem.

A Dag is defined in Python code and visualized in the Airflow UI. Dags can be as simple as a single task or as complex as hundreds or thousands of tasks with complicated dependencies.

The following screenshot shows a [complex Dag run graph](#complex-dag-runs) in the Airflow UI. After reading this guide, you'll be able to understand the elements in this graph, as well as know how to define Dags and use Dag parameters.

<Frame>
  <img alt="Screenshot of a complex DAG run graph with dynamically mapped tasks, task groups and setup/teardown tasks." />
</Frame>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* What Airflow is. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).

## What is a Dag?

A *Dag* (directed acyclic graph) is a mathematical structure consisting of nodes and edges. In Airflow, a Dag represents a data pipeline or workflow with a start and an end.

<Note>
  The term "Dag" was historically written as the acronym "DAG". The Airflow project now uses "Dag" as a standalone word.
</Note>

The mathematical properties of Dags make them useful for building data pipelines:

* **Directed**: There is a clear direction of flow between tasks. A task can be either upstream, downstream, or parallel to another task.

  <Frame>
    <img alt="Visualization of two graphs with 3 nodes each. The first graph is directed, the arrow in between the nodes always points into one direction. The second graph isn't directed, the arrow between the second and third node points in both directions. Only the first graph would be possible to define in Airflow." />
  </Frame>

* **Acyclic**: There are no circular dependencies in a Dag. This means that a task can't depend on itself, nor can it depend on a task that ultimately depends on it.

  <Frame>
    <img alt="Visualization of two graphs with 4 nodes each. The first graph is acyclic, there are no circles defined between the nodes. In the second graph a dependency is added between task 4 and task 1, meaning task 1 depends on task 4. This creates a circle because task 4 is downstream of task 1. Only the first graph would be possible to define in Airflow." />
  </Frame>

* **Graph**: A Dag is a graph, which is a structure consisting of nodes and edges. In Airflow, nodes are tasks and edges are dependencies between tasks. Defining workflows as graphs helps you visualize the entire workflow in a way that's easy to navigate and conceptualize.

Beyond these requirements, a Dag can be as simple or as complicated as you need. You can define tasks that run in parallel or sequentially, implement conditional branches, and visually group tasks together in [task groups](/docs/learn/task-groups).

For example, a common Dag might extract data from an API, anonymize sensitive fields, check for duplicate records, insert cleaned data into a database, and run a SQL query to update a dashboard. Each of these steps is a task, and the Dag ensures they run in the correct order.

Each task in a Dag should perform one unit of work. Tasks can be anything from a simple Python function to a complex data transformation or a call to an external service. They are defined using [Airflow operators](/docs/learn/what-is-an-operator) or [Airflow decorators](/docs/learn/airflow-decorators). The dependencies between tasks can be set in different ways (see [Managing Dependencies](/docs/learn/managing-dependencies)).

The following screenshot shows a simple Dag graph with 3 sequential tasks.

<Frame>
  <img alt="A simple Dag graph is shown with 3 sequential tasks, get_astronauts, print_astronaut_craft (which is a dynamically mapped task) and print_reaction." />
</Frame>

<details>
  <summary>Click to view the full Dag code used to create the Dag in the screenshot</summary>

  ```python expandable wrap theme={null}
  """
  ## Astronaut ETL example DAG

  This DAG queries the list of astronauts currently in space from the
  Open Notify API and prints each astronaut's name and flying craft.

  There are three tasks, one to get the data from the API and save the results,
  one to print the results and a final task to react. The first two tasks are
  written in Python using Airflow's TaskFlow API, which allows you to easily turn
  Python functions into Airflow tasks, and automatically infer dependencies and pass data.

  The second task uses dynamic task mapping to create a copy of the task for
  each Astronaut in the list retrieved from the API. This list will change
  depending on how many Astronauts are in space, and the DAG will adjust
  accordingly each time it runs.

  The third task is defined using the BashOperator, which is a traditional
  operator, allowing you to run a bash command.
  """

  from airflow.decorators import dag, task
  from airflow.operators.bash import BashOperator
  from airflow.models.baseoperator import chain
  from pendulum import datetime
  import requests


  # Define the basic parameters of the DAG, like schedule and start_date
  @dag(
      start_date=datetime(2024, 1, 1),
      schedule="@daily",
      catchup=False,
      doc_md=__doc__,
      default_args={"owner": "Astro", "retries": 3},
      tags=["example"],
  )
  def example_astronauts_three_tasks():
      # Define tasks
      @task
      def get_astronauts(**context) -> list[dict]:
          """
          This task uses the requests library to retrieve a list of Astronauts
          currently in space. The results are pushed to XCom with a specific key
          so they can be used in a downstream pipeline. The task returns a list
          of Astronauts to be used in the next task.
          """
          r = requests.get("http://api.open-notify.org/astros.json")
          number_of_people_in_space = r.json()["number"]
          list_of_people_in_space = r.json()["people"]

          context["ti"].xcom_push(
              key="number_of_people_in_space", value=number_of_people_in_space
          )
          return list_of_people_in_space

      @task
      def print_astronaut_craft(greeting: str, person_in_space: dict) -> None:
          """
          This task creates a print statement with the name of an
          Astronaut in space and the craft they are flying on from
          the API request results of the previous task, along with a
          greeting which is hard-coded in this example.
          """
          craft = person_in_space["craft"]
          name = person_in_space["name"]

          print(f"{name} is currently in space flying on the {craft}! {greeting}")

      # define a task using a traditional operator
      # this task will run a bash command
      print_reaction = BashOperator(
          task_id="print_reaction",
          bash_command="echo This is awesome!",
      )

      # Use dynamic task mapping to run the print_astronaut_craft task for each
      # Astronaut in space
      print_astronaut_craft_obj = print_astronaut_craft.partial(
          greeting="Hello! :)"
      ).expand(
          person_in_space=get_astronauts()  # Define dependencies using TaskFlow API syntax
      )

      # set the dependency between the second and third task explicitly
      chain(print_astronaut_craft_obj, print_reaction)


  # Instantiate the DAG
  example_astronauts_three_tasks()
  ```
</details>

## Why use Dags for data pipelines?

Structuring your data pipelines as Dags provides several advantages over running disconnected scripts or using separate interface-driven tools:

* **Reliability**: Tasks execute in a guaranteed order every run. If a task fails, dependent downstream tasks don't execute, preventing data corruption from stale or incomplete data.
* **Visibility**: The Dag structure gives you a visual map of your entire pipeline in the Airflow UI, making it easier to debug failures and understand data flow at a glance.
* **Testability**: Because Dags define deterministic execution paths, you can test individual tasks in isolation and validate expected outcomes against known inputs.

## What is a Dag run?

A *Dag run* is an instance of a Dag running at a specific point in time. A *task instance* is an instance of a task running at a specific point in time. Each Dag run has a unique `run_id` and contains one or more task instances. The history of previous Dag runs is stored in the [Airflow metadata database](/docs/learn/airflow-database).

In the Airflow UI, you can view previous runs of a Dag in the **Grid** view and select individual Dag runs by clicking on their respective duration bar.

<Frame>
  <img alt="Gridview 3 task Dag." />
</Frame>

A Dag run graph looks similar to the Dag graph, but includes additional information about the status of each task instance in the Dag run.

<Frame>
  <img alt="Screenshot of a Dag run graph with 3 tasks, get_astronauts, print_astronaut_craft (which is a dynamically mapped task with 12 mapped task instances) and print_reaction." />
</Frame>

Learn more about how to navigate the Airflow UI in the [An introduction to the Airflow UI](/docs/learn/airflow-ui) guide.

Each Dag run is associated with a Dag version. Each time you make structural changes to a Dag and create a new Dag run, the Dag version is incremented. This allows you to track changes to the Dag over time and better understand previous Dag runs. For more information on Dag versions, see [Dag Versioning and Dag Bundles](/docs/learn/airflow-dag-versioning).

### Dag run properties

A Dag run graph in the Airflow UI contains information about the Dag run, as well as the status of each task instance in the Dag run. The following screenshot shows the same Dag as in the previous section, but with annotations explaining the different elements of the graph.

<Frame>
  <img alt="Screenshot of the Airflow UI. A Dag run with 3 tasks is shown. The annotations show the location of the dag_id and logical date (top of the screenshot), the task_id, task state and operator/decorator used in the nodes of the graph, as well as the number of dynamically mapped task instances in [] behind the task id and the Dag dependency layout to the right of the screen." />
</Frame>

* `dag_id`: The unique identifier of the Dag.
* `logical date`: The point in time after which this particular Dag run can run. This date and time isn't necessarily the same as the actual moment the Dag run is executed. See [Scheduling](/docs/learn/scheduling-in-airflow) for more information.
* `task_id`: The unique identifier of the task.
* `task state`: The status of the task instance in the Dag run. Possible states are `running`, `success`, `failed`, `skipped`, `restarting`, `up_for_retry`, `upstream_failed`, `queued`, `scheduled`, `none`, `removed`, `deferred`, and `up_for_reschedule`, they each cause the border of the node to be colored differently. See the OSS documentation on [task instances](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#task-instances) for an explanation of each state.

There are four ways you can trigger a Dag run:

* **Backfill**: [Backfilling](/docs/learn/rerunning-dags#backfill) is a mechanism by which you can create several Dag runs for dates in the past using the Airflow UI, API or CLI. Backfilled Dag runs include a curved arrow on their Dag run duration bar.
* **Scheduled**: Dag runs created based on a Dag's schedule (for example `@daily`, `@hourly`) are created by the Airflow scheduler. The Dag run duration bar doesn't have an additional icon.
* **Manual**: You can trigger manual runs of a Dag in the Airflow UI, or by using the Airflow CLI or API. Manually triggered Dag runs include a play icon on the Dag run duration bar.
* **Asset triggered**: Dags can be scheduled using data-aware scheduling. This means a Dag runs as soon as one or more [Airflow assets](/docs/learn/airflow-datasets) are updated. These updates can come from tasks inside of the same Airflow instance, a call to the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html), be made manually using the Airflow UI, or triggered based on [messages in a message queue](/docs/learn/airflow-event-driven-scheduling). The Dag run duration bar includes an asset icon.

A Dag run can have the following statuses:

* **Queued**: The time after which the Dag run can be created has passed but the scheduler hasn't created task instances for it yet.
* **Running**: The Dag run is eligible to have task instances scheduled.
* **Success**: All task instances are in a terminal state (`success`, `skipped`, `failed` or `upstream_failed`) and all leaf tasks (tasks with no downstream tasks) are either in the state `success` or `skipped`. The duration bar of a successful Dag run is green.
* **Failed**: All task instances are in a terminal state and at least one leaf task is in the state `failed` or `upstream_failed`. The duration bar of a failed Dag run is red.

### Complex Dag runs

When you start writing more complex Dags, you will see additional Airflow features that are visualized in the Dag run graph. The following screenshot shows the same complex Dag as in the overview but with annotations explaining the different elements of the graph. Don't worry if you don't know about all these features yet. You will learn about them as you become more familiar with Airflow.

<Frame>
  <img alt="Screenshot of a complex Dag run graph with annotations showing a dynamically mapped task, a branching task, an edge label, a dynamically mapped task group, regular task groups, setup/ teardown tasks as well as an asset." />
</Frame>

<details>
  <summary>Click to view the full Dag code used for the screenshot</summary>

  The following code creates the same DAG structure as shown in the previous screenshot.

  ```python expandable wrap theme={null}
  """
  ## Example of a complex DAG structure

  This DAG demonstrates how to set up a complex structures including:
  - Branches with Labels
  - Task Groups
  - Dynamically mapped tasks
  - Dynamically mapped task groups

  The tasks themselves are empty or simple bash statements.
  """

  from airflow.providers.standard.operators.bash import BashOperator
  from airflow.providers.standard.operators.empty import EmptyOperator
  from airflow.sdk import Asset, Label, chain, chain_linear, dag, task, task_group


  # Define the DAG
  @dag
  def complex_dag_structure():

      start = EmptyOperator(task_id="start")

      # Dynamically mapped tasks: .partial() contains all parameters that stay the same
      # between mapped task instances. .expand() contains the parameters that change
      # one mapped task instance will be created for each value in the list provided to `bash_command`
      sales_data_extract = BashOperator.partial(task_id="sales_data_extract").expand(
          bash_command=["echo 1", "echo 2", "echo 3", "echo 4"]
      )
      internal_api_extract = BashOperator.partial(task_id="internal_api_extract").expand(
          bash_command=["echo 1", "echo 2", "echo 3", "echo 4"]
      )

      # Branch task that picks which branch to follow
      @task.branch
      def determine_load_type() -> str:
          """Randomly choose a branch. The return value is the task_id of the branch to follow."""
          import random

          if random.choice([True, False]):
              return "internal_api_load_full"
          return "internal_api_load_incremental"

      sales_data_transform = EmptyOperator(task_id="sales_data_transform")

      # When using the TaskFlow API it is common to assign the called task to an object
      # to use in several dependency definitions without creating several instances of the same task
      determine_load_type_obj = determine_load_type()

      sales_data_load = EmptyOperator(task_id="sales_data_load")
      internal_api_load_full = EmptyOperator(task_id="internal_api_load_full")
      internal_api_load_incremental = EmptyOperator(
          task_id="internal_api_load_incremental"
      )

      # defining a task group that task a parameter (a) to allow for dynamic task group mapping
      @task_group
      def sales_data_reporting(a):
          # the trigger_rule of the first task in the task group can be set to "all_done"
          # to ensure that it runs even if one of the upstream tasks (`internal_api_load_full`)
          # is skipped due to the branch task
          prepare_report = EmptyOperator(
              task_id="prepare_report", trigger_rule="all_done"
          )
          publish_report = EmptyOperator(task_id="publish_report")

          # setting dependencies within the task group
          chain(prepare_report, publish_report)

      # dynamically mapping the task group with `a` being the mapped parameter
      # one task group instance will be created for each value in the list provided to `a`
      sales_data_reporting_obj = sales_data_reporting.expand(a=[1, 2, 3, 4, 5, 6])

      # defining a task group that does not use any additional parameters
      @task_group
      def cre_integration():
          # the trigger_rule of the first task in the task group can be set to "all_done"
          # to ensure that it runs even if one of the upstream tasks (`internal_api_load_full`)
          # is skipped due to the branch task
          cre_extract = EmptyOperator(task_id="cre_extract", trigger_rule="all_done")
          cre_transform = EmptyOperator(task_id="cre_transform")
          cre_load = EmptyOperator(task_id="cre_load")

          # setting dependencies within the task group
          chain(cre_extract, cre_transform, cre_load)

      # calling the task group to instantiate it and assigning it to an object
      # to use in dependency definitions
      cre_integration_obj = cre_integration()

      @task_group
      def mlops():
          # the trigger_rule of the first task in the task group can be set to "all_done"
          # to ensure that it runs even if one of the upstream tasks (`internal_api_load_incremental`)
          # is skipped due to the branch task
          set_up_cluster = EmptyOperator(
              task_id="set_up_cluster", trigger_rule="all_done"
          )
          # the outlets parameter is used to define which datasets are updated by this task
          train_model = EmptyOperator(task_id="train_model")
          tear_down_cluster = EmptyOperator(task_id="tear_down_cluster")

          # setting dependencies within the task group
          chain(set_up_cluster, train_model, tear_down_cluster)

          # turning the `tear_down_cluster`` task into a teardown task and set
          # the `set_up_cluster` task as the associated setup task
          tear_down_cluster.as_teardown(setups=set_up_cluster)

      mlops_obj = mlops()

      end = EmptyOperator(task_id="end", outlets=[Asset("dag_completed")])

      # --------------------- #
      # Defining dependencies #
      # --------------------- #

      chain(
          start,
          sales_data_extract,
          sales_data_transform,
          sales_data_load,
          [sales_data_reporting_obj, cre_integration_obj],
          end,
      )
      chain(
          start,
          internal_api_extract,
          determine_load_type_obj,
          [internal_api_load_full, internal_api_load_incremental],
          mlops_obj,
          end,
      )

      chain_linear(
          [sales_data_load, internal_api_load_full],
          [sales_data_reporting_obj, cre_integration_obj],
      )

      # Adding labels to two edges
      chain(
          determine_load_type_obj, Label("additional data"), internal_api_load_incremental
      )
      chain(
          determine_load_type_obj, Label("changed existing data"), internal_api_load_full
      )


  # Calling the DAG function will instantiate the DAG
  complex_dag_structure()
  ```
</details>

Some more complex features visible in this Dag graph are:

* **Dynamically mapped tasks**: A dynamically mapped task is [created dynamically](/docs/learn/dynamic-tasks) at runtime based on user-defined input. The number of dynamically mapped task instances is shown in brackets (`[]`) behind the task ID.
* **Branching tasks**: A branching task creates a conditional branch in the Dag. See [Branching in Airflow](/docs/learn/airflow-branch-operator) for more information.
* **Edge labels**: Edge labels appear on the edge between two tasks. These labels are often helpful to annotate branch decisions in a Dag graph.
* **Task groups**: A task group is a tool to logically and visually group tasks in an Airflow Dag. See [Airflow task groups](/docs/learn/task-groups) for more information.
* **Setup/teardown tasks**: When using Airflow to manage infrastructure, it can be helpful to define tasks as setup and teardown tasks to take advantage of additional intelligent dependency behavior. Setup and teardown tasks appear with diagonal arrows next to their task IDs and are connected with a dotted line. See [Use setup and teardown tasks in Airflow](/docs/learn/airflow-setup-teardown) for more information.
* **Assets**: Assets are shown in the Dag graph. If a Dag is scheduled on an asset, it is shown upstream of the first task of the Dag. If a task in the Dag updates an asset, it is shown after the respective task as in the previous screenshot. See [Airflow assets](/docs/learn/airflow-datasets) for more information.

You can learn more about how to set complex dependencies between tasks and task groups in the [Managing Dependencies](/docs/learn/managing-dependencies) guide.

## Write a Dag

A Dag can be defined with a Python file placed in an Airflow project's [Dag bundle](/docs/learn/airflow-dag-versioning). When using the [Astro CLI](/docs/cli/v1.43/get-started-cli) with default settings this is your `dags` folder. Airflow automatically parses all files in this folder every 5 minutes to check for new Dags, and it parses existing Dags for code changes every 30 seconds. You can force a new Dag parse using [`airflow dags reserialize`](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#reserialize), or `astro dev run dags reserialize` using the Astro CLI.

There are two types of syntax you can use to structure your Dag:

* **TaskFlow API**: The TaskFlow API contains the `@dag` decorator. A function decorated with `@dag` defines a Dag. Note that you need to call the function at the end of the script for Airflow to register the Dag. All tasks are defined within the context of the Dag function.
* **Traditional syntax**: You can create a Dag by instantiating a Dag context using the `DAG` class and defining tasks within that context.

TaskFlow API and traditional syntax can be freely mixed. See [Mixing TaskFlow decorators with traditional operators](/docs/learn/airflow-decorators#mixing-taskflow-decorators-with-traditional-operators) for more information. Additionally, it is also possible to create one-task Dags with the `@asset` decorator, for more information see [Airflow assets](/docs/learn/airflow-datasets#basic-asset-concepts).

The following is an example of the same Dag written using each type of syntax.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # Import all packages needed at the top level of the DAG
  from airflow.sdk import dag, task, chain
  from pendulum import datetime


  # Define the DAG function a set of parameters
  @dag(
      start_date=datetime(2025, 4, 1),
      schedule="@daily",
  )
  def taskflow_dag():
      # Define tasks within the DAG context
      @task
      def my_task_1():
          import time  # import packages only needed in the task function

          time.sleep(5)
          print(1)

      @task
      def my_task_2():
          print(2)

      # Define dependencies and call task functions
      chain(my_task_1(), my_task_2())


  # Call the DAG function
  taskflow_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  # Import all packages needed at the top level of the DAG
  from airflow.sdk import DAG
  from airflow.providers.standard.operators.python import PythonOperator
  from pendulum import datetime


  def my_task_1_func():
      import time  # import packages only needed in the task function

      time.sleep(5)
      print(1)


  # Instantiate the DAG
  with DAG(
      dag_id="traditional_syntax_dag",
      start_date=datetime(2025, 4, 1),
      schedule="@daily",
  ):
      # Instantiate tasks within the DAG context
      my_task_1 = PythonOperator(
          task_id="my_task_1",
          python_callable=my_task_1_func,
      )

      my_task_2 = PythonOperator(
          task_id="my_task_2",
          python_callable=lambda: print(2),
      )

      # Define dependencies
      my_task_1 >> my_task_2
  ```
</details>

<Tip>
  Astronomer recommends creating one Python file for each Dag and naming it after the `dag_id` as a best practice for organizing your Airflow project. For certain advanced use cases it may be appropriate to dynamically generate Dags using Python code, see [Dynamically generate Dags in Airflow](/docs/learn/dynamically-generating-dags) for more information.
</Tip>

### Dag-level parameters

In Airflow, you can configure when and how your Dag runs by setting parameters in the Dag object. Dag-level parameters affect how the entire Dag behaves, as opposed to task-level parameters which only affect a single task.

The Dags in the previous section have the following basic parameters defined:

* `dag_id`: The name of the Dag. This must be unique for each Dag in the Airflow environment. When using the `@dag` decorator and not providing the `dag_id` parameter name, the function name is used as the `dag_id`.
* `start_date`: The date and time after which the Dag starts being scheduled. Defaults to `None`.
* `schedule`: The schedule for the Dag. There are many different ways to define a schedule, see [Scheduling in Airflow](/docs/learn/scheduling-in-airflow) for more information. Defaults to `None`.

There are many more Dag-level parameters that let you configure anything from resource usage to the Dag's appearance in the Airflow UI. See [Dag-level parameters](/docs/learn/airflow-dag-parameters) for a complete list.

## FAQ

<AccordionGroup>
  <Accordion title="What does Dag stand for?">
    Dag stands for "directed acyclic graph," a mathematical term for a graph with directed edges and no cycles. The Airflow project historically wrote this as the acronym "DAG" but now treats "Dag" as a standalone term.
  </Accordion>

  <Accordion title="What is a Dag used for?">
    In Airflow, a Dag defines a data workflow as a series of tasks with explicit dependencies. This gives you control over task ordering, visibility into pipeline execution through the Airflow UI, and automatic handling of failures across dependent tasks.
  </Accordion>

  <Accordion title="Can a Dag have parallel tasks?">
    Yes. Tasks without dependencies between them run in parallel, assuming your Airflow instance is set up to support parallel tasks. Only tasks with explicit upstream or downstream relationships run sequentially. You control parallelism by how you define task dependencies.
  </Accordion>
</AccordionGroup>

## See also

* [Get started with Apache Airflow](/docs/learn/get-started-with-airflow) tutorial for a hands-on introduction to writing your first simple Dag.
* [Airflow operators](/docs/learn/what-is-an-operator) and [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators) for more information on how to define tasks in a Dag.


# Debug Apache Airflow® Dags with AI
Source: https://astronomer.io/docs/learn/debug-dags-with-ai

Debug Apache Airflow® Dags locally with the help of AI agents.

Almost every data engineer uses AI to debug Airflow failures, from Dag import errors to failed tasks to Airflow infrastructure issues. Pasting a stack trace to an agent with "fix this" works some of the time, but there are better, systematic ways to let AI find the root cause of issues in your Airflow pipelines. This guide explains how to use AI agents to debug systematically in loops, or automatically as soon as a Dag running on Astro fails.

## Assumed knowledge

To get the most out of this guide, you should have:

* A local Airflow environment. See [Run Airflow locally](/docs/learn/run-airflow-locally).
* Basic familiarity with an AI coding agent harness, such as Claude Code or Cursor.

## Otto

<Info>
  **Labs**

  Otto is in [Labs](/docs/astro/feature-previews).
</Info>

Debugging is one of the tasks where Otto has the biggest advantage over a general-purpose agent. Otto investigates with access to the operational history in Astro: component logs, recent deploys, connections, lineage, and a compatibility knowledge base drawn from Astronomer's experience running Airflow at scale. A general-purpose harness only sees what's in your local repository and whatever you added to the context manually.

You can ask Otto to investigate a failure interactively from the Astro CLI, trigger an investigation from the Astro UI, or automatically using an Astro alert and Otto's API endpoint. See [Investigate with Otto](/docs/astro/otto-investigate) and [How to use Otto to automatically investigate Dag failures and PR a fix](/docs/learn/airflow-otto-rca-auto-fix) for more information.

An investigation reports a root cause and a suggested fix, among additional information such as estimated severity, a confidence score, and the evidence behind the diagnosis. The following example shows the same failure investigated from different locations: a `copy_to_staging` task that fails because Snowflake can't cast the percentage string `"88.1%"` into a numeric column.

<Tabs>
  <Tab title="Astro UI">
    <Frame>
      <img alt="An Otto investigation in the Astro UI for a failed Dag run, labeled HIGH, P2, and PERMANENT. Otto reports 95 percent confidence that the task fails because Snowflake can't cast the percentage string 88.1% into a numeric column, and suggests emitting a numeric value instead. Dag-level checks rule out a Dag run timeout and a recent deploy as causes." />
    </Frame>
  </Tab>

  <Tab title="Astro CLI">
    <Frame>
      <img alt="Otto investigating the same failure in a terminal. In response to the question &#x22;Why did fetch_cargo_manifests fail in the prod deployment?&#x22;, Otto searches the copy_to_staging task logs for errors, reads the Dag source, and reports the root cause: cargo_fill_pct is built as a percent-formatted string, but the Snowflake column it lands in is DECIMAL(5,2). Otto names the file and line to change and offers to apply the fix and re-trigger the run." />
    </Frame>

    The `af` command Otto uses comes from the [`astro-airflow-mcp` package](https://github.com/astronomer/agents/blob/main/astro-airflow-mcp/README.md#airflow-cli-tool), which includes both an MCP server and `af`, a CLI for interacting with Airflow instances from your terminal.
  </Tab>
</Tabs>

## Debug systematically

Using AI doesn't change what good debugging looks like. When you use an agent to debug a Dag, use the same systematic approach you would use when debugging manually. Astronomer recommends the following debugging ladder:

1. **Infrastructure**: Is the scheduler running, is the database reachable, and are the worker queues sized adequately?
2. **Dag parsing**: Does the Dag file import without errors? Check with `astro dev parse`.
3. **Dag and task run**: Is the Dag schedule correct, and are the tasks running with the correct dependencies?
4. **Logic**: Is the task code doing what it should?
5. **Prevention**: After you fix the bug, what test or check would have caught it earlier?

<Info>
  To see an example of the debugging ladder in practice, watch the recording of the [Best practices for debugging your Airflow Dags](https://www.astronomer.io/events/webinars/best-practices-for-debugging-your-airflow-dags-video/) webinar.
</Info>

Encode this ladder as a skill or rule file for your harness, so every debugging session follows the same steps without you repeating them in each prompt.

## Give your agent context about the failure

For debugging, your agent needs access to the context surrounding a failure:

* **Task logs**: The full output of the failed task run, if applicable, including the exception, stack trace, and surrounding log lines.
* **Run history and task state**: Whether this is the first failure, whether upstream tasks actually succeeded, and what schedule and trigger rule the task uses.
* **Source history**: What changed in the Dag file recently, for example using `git log` and `git blame`.
* **Lineage**: What data sources upstream of the Dag might influence its functioning, and which downstream assets can be affected. When running Airflow on Astro, Otto can access information gathered as part of [Astro Observe](/docs/astro/astro-observe).
* **Additional variables**: Sometimes Dags depend on values that aren't defined in code. This could be environment variables, Airflow variables, Airflow connections, or configuration information fetched at runtime from a third-party service. If possible gather, the values for the failed Dag run and make them available to your agent.

### Gather task logs

For a local project in container mode, Airflow writes task logs to `$AIRFLOW_HOME/logs/` inside the scheduler container, so your agent needs a shell in the container to read them. You can use `astro dev bash` to open one. Use `astro dev logs` for component logs from the scheduler, triggerer, API server, and Dag processor.

For a Dag that ran on an Astro Deployment, Astronomer recommends using the `astro-airflow-mcp` as a wrapper around the Airflow REST API. See [Airflow MCP Plugin](/docs/astro/astro-mcp-server#airflow-mcp-plugin) for more information.

<Warning>
  Commands such as `astro dev run tasks test <dag-id> <task-id>`, and triggering a Dag run, execute real task code against whatever connections the task uses. Before you let an agent run tasks while debugging, make sure running tasks is safe, for example by pointing the project at a test environment.
</Warning>

For stepping through task logic line by line, see [Debug with `dag.test()`](/docs/learn/set-up-your-ide-for-data-engineering#debug-with-dag-test).

## Debug in a loop

Debugging in a loop works the same way as [writing Dags in a loop](/docs/learn/develop-dags-with-ai#let-agents-write-in-a-loop). Turn the bug into a regression test first: a test that reproduces the failure and fails until the fix is correct. Then let the agent iterate against that test the same way it would against any other [success criteria](/docs/learn/develop-dags-with-ai#define-success-criteria).

The same [safety guardrails](/docs/learn/develop-dags-with-ai#safety) apply here: scope tool permissions, require approval for anything you can't easily undo, and commit frequently so errors can be reverted.


# Debug DAGs
Source: https://astronomer.io/docs/learn/debugging-dags

Troubleshoot Airflow DAGs

This guide explains how to identify and resolve common Airflow DAG issues. It also includes resources to try out if you can't find a solution to an Airflow issue.
While the focus of the troubleshooting steps provided lies on local development, much of the information is also relevant for running Airflow in a production context.

<Tip>
  Consider implementing systematic testing of your DAGs to prevent common issues. See the [Test Airflow DAGs](/docs/learn/testing-airflow) guide.
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Get started with Airflow tutorial](/docs/learn/get-started-with-airflow).
* Basic knowledge of Airflow DAGs. See [Introduction to Airflow DAGs](/docs/learn/dags).

## General Airflow debugging approach

To give yourself the best possible chance of fixing a bug in Airflow, contextualize the issue by asking yourself the following questions:

* Is the problem with Airflow, or is it with an external system connected to Airflow? Test if the action can be completed in the external system without using Airflow.
* What is the state of your [Airflow components](/docs/learn/airflow-components)? Inspect the logs of each component and restart your Airflow environment if necessary.
* Does Airflow have access to all relevant files? This is especially relevant when running Airflow in a containerized service like the [Astro CLI](/docs/cli/v1.43/overview).
* Are your [Airflow connections](/docs/learn/connections) set up correctly with correct credentials? See [Troubleshooting connections](#troubleshooting-connections).
* Is the issue with all DAGs, or is it isolated to one DAG?
* Can you collect the relevant logs? For more information on log location and configuration, see the [Airflow logging](/docs/learn/logging) guide.
* Which versions of Airflow and Airflow providers are you using? Make sure that you're using the correct version of the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/index.html).
* Can you reproduce the problem in a new local Airflow instance using the [Astro CLI](/docs/cli/v1.43/overview)?

Answering these questions will help you narrow down what kind of issue you're dealing with and inform your next steps.

## Airflow isn't starting on the Astro CLI

The 3 most common ways to run Airflow locally are using the [Astro CLI](/docs/cli/v1.43/install-cli), running a [standalone instance](https://airflow.apache.org/docs/apache-airflow/stable/start.html), or running [Airflow in Docker](https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html). This guide focuses on troubleshooting the Astro CLI, which is an open source tool for quickly running Airflow on a local machine.

The most common issues related to the Astro CLI are:

* The Astro CLI wasn't correctly installed. Run `astro version` to confirm that you can successfully run Astro CLI commands. If a newer version is available, consider upgrading.
* There are errors caused by custom commands in the Dockerfile, or dependency conflicts with the packages in `packages.txt` and `requirements.txt`.
* Airflow components are in a crash-loop because of errors in custom plugins or XCom backends. View scheduler logs using `astro dev logs -s` to troubleshoot.

To troubleshoot infrastructure issues when running Airflow on other platforms, for example in Docker, on Kubernetes using the [Helm Chart](https://airflow.apache.org/docs/helm-chart/stable/index.html) or on managed services, refer to the relevant documentation and customer support.

You can learn more about [testing and troubleshooting locally](/docs/cli/v1.43/test-your-astro-project-locally) with the Astro CLI in the Astro documentation.

## Common DAG issues

This section covers common issues related to DAG code that you might encounter when developing.

### DAGs don't appear in the Airflow UI

If a DAG isn't appearing in the Airflow UI, it's typically because Airflow is unable to parse the DAG. If this is the case, you'll see an `Import Error` in the Airflow UI.

<Frame>
  <img alt="Import Error" />
</Frame>

The message in the import error can help you troubleshoot and resolve the issue.

To view import errors in your terminal, run `astro dev run dags list-import-errors` with the Astro CLI, or run `airflow dags list-import-errors` with the Airflow CLI.

If you don't see an import error message but your DAGs still don't appear in the UI, try these debugging steps:

* Make sure all of your DAG files are located in the `dags` folder.
* Airflow scans the `dags` folder for new DAGs every [`dag_dir_list_interval`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-dir-list-interval), which defaults to 5 minutes but can be modified. You can force reparsing of all files with `astro dev run dags reserialize` / `airflow dags reserialize`.
* Ensure that you have permission to see the DAGs, and that the permissions on the DAG file are correct.
* Run `astro dev run dags list` with the Astro CLI or `airflow dags list` with the Airflow CLI to make sure that Airflow has registered the DAG in the metadata database. If the DAG appears in the list but not in the UI, try restarting Airflow with `astro dev restart`.
* Try restarting the Airflow scheduler with `astro dev restart`.
* If you see an error in the Airflow UI indicating that the scheduler isn't running, check the scheduler logs to see if an error in a DAG file is causing the scheduler to crash. If you are using the Astro CLI, run `astro dev logs -s` and then try restarting.

  <Frame>
    <img alt="No Scheduler" />
  </Frame>

At the code level, ensure that each DAG:

* Has a unique `dag_id`.
* Is called when defined with the `@dag` decorator. See also [Introduction to Airflow decorators](/docs/learn/airflow-decorators).

You can configure an Airflow listener as a plugin to run any Python code, either when a new import error appears (`on_new_dag_import_error`) or when the dag processor finds a known import error (`on_existing_dag_import_error`). See [Airflow listeners](/docs/learn/airflow-listeners) for more information.

### Import errors due to dependency conflicts

A frequent cause of DAG import errors isn't having the necessary packages installed in your Airflow environment. You might be missing [provider packages](https://airflow.apache.org/registry/providers/) that are required for using specific operators or hooks, or you might be missing Python packages used in Airflow tasks.

In an Astro project, you can install OS-level packages by adding them to your `packages.txt` file. You can install Python-level packages, such as provider packages, by adding them to your `requirements.txt` file. If you need to install packages using a specific package manager, consider doing so by adding a bash command to your Dockerfile.

To prevent compatibility issues when new packages are released, Astronomer recommends pinning a package version to your project. For example, adding `apache-airflow-providers-amazon==9.6.0` to your `requirements.txt` file ensures that no future releases of `apache-airflow-providers-amazon` causes compatibility issues. If no version is pinned, Airflow will always use the latest available version.

If you are using the Astro CLI, packages are installed in the scheduler container. You can confirm that a package is installed correctly by running:

```sh wrap theme={null}
astro dev bash --scheduler "pip freeze | grep <package-name>"
```

If you have conflicting package versions or need to run multiple Python versions, you can run tasks in different environments using a few different operators:

* [`KubernetesPodOperator`](/docs/learn/kubepod-operator): Runs a task in a separate Kubernetes Pod.
* [`ExternalPythonOperator`](/docs/learn/airflow-isolated-environments): Runs a task in a predefined virtual environment.
* [`PythonVirtualEnvOperator`](https://airflow.apache.org/registry/providers/standard#standard-python-PythonVirtualenvOperator): Runs a task in a temporary virtual environment.

If many Airflow tasks share a set of alternate package and version requirements a common pattern is to run them in two or more separate Airflow deployments.

### DAGs aren't running correctly

If your DAGs are either not running or running differently than you intended, consider checking the following common causes:

* DAGs need to be unpaused in order to run on their schedule. You can unpause a DAG by clicking the toggle on the left side of the Airflow UI or by using the [Airflow CLI](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#unpause).

  <Frame>
    <img alt="Location of unpause toggle in the Airflow UI" />
  </Frame>

  If you want all DAGs unpaused by default, you can set [`dags_are_paused_at_creation=False`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dag-dir-list-interval) in your Airflow config.

* Double check that each DAG has a unique `dag_id`. If two DAGs with the same id are present in one Airflow instance the scheduler will pick one at random every 30 seconds to display.

* Make sure your DAG has a `start_date` in the past. A DAG with a `start_date` in the future will result in a successful DAG run with no task runs. Don't use `datetime.now()` as a `start_date`.

* Test the DAG using `astro dev dags test <dag_id>`. With the Airflow CLI, run `airflow dags test <dag_id>`.

* If no DAGs are running, check the state of your scheduler
  using `astro dev logs -s`.

If your DAG is running, but not on the schedule you expected, review the [DAG Schedule DAGs in Airflow](/docs/learn/scheduling-in-airflow) guide. If you are using a custom timetable, ensure that the data interval for your DAG run doesn't precede the DAG start date.

## Common task issues

This section covers common issues related to individual tasks you might encounter. If your entire DAG isn't working, see the [DAGs aren't running correctly](#dags-aren’t-running-correctly) section above.

<Tip>
  There were significant changes to the Airflow architecture between Airflow 2 and Airflow 3, greatly improving Airflow's security posture and enabling new features such as remote execution. The most important impact of those changes for DAG authors is that directly accessing the metadata database from within Airflow tasks isn't possible anymore. See the [Upgrade from Apache Airflow® 2 to 3 guide](/docs/learn/airflow-upgrade-2-3) and the [Airflow release notes](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html) for more information.
</Tip>

### Tasks aren't running correctly

It is possible for a DAG to start but its tasks to be stuck in various states or to not run in the desired order. If your tasks aren't running as intended, try the following debugging methods:

* Double check that your DAG's `start_date` is in the past. A future `start_date` will result in a successful DAG run even though no tasks ran.
* If your tasks stay in a `scheduled` or `queued` state, ensure your scheduler is running properly. If needed, restart the scheduler or increase scheduler resources in your Airflow infrastructure.
* If your tasks have the `depends_on_past` parameter set to `True`, those newly added tasks won't run until you set the state of prior task runs.
* When running many instances of a task or DAG, be mindful of scaling parameters and configurations. Airflow has default settings that limit the amount of concurrently running DAGs and tasks. See [Scaling Airflow to optimize performance](/docs/learn/airflow-scaling-workers) to learn more.
* If you are using task decorators and your tasks aren't showing up in the **Graph** and **Grid**, make sure you are calling your tasks. See also [Introduction to Airflow decorators](/docs/learn/airflow-decorators).
* Check your task dependencies and trigger rules. See [Manage DAG and task dependencies in Airflow](/docs/learn/managing-dependencies) and [Airflow trigger rules](/docs/learn/airflow-trigger-rules). Consider recreating your DAG structure with EmptyOperators to ensure that your dependencies are structured as expected.
* The [`task_queued_timeout`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#task-queued-timeout) parameter controls how long tasks can be in queued state before they are either retried or marked as failed. The default is 600 seconds.
* If you are using the CeleryExecutor in an Airflow version earlier than 2.6 and tasks get stuck in the `queued` state, consider turning on [`stalled_task_timeout`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#stalled-task-timeout).

### Tasks are failing

Most task failure issues fall into one of 3 categories:

* Issues with operator parameter inputs.
* Issues within the operator.
* Issues in an external system.

Failed tasks appear as red squares in the **Grid** view, where you can also directly access task logs.

<Frame>
  <img alt="Grid View Task failure" />
</Frame>

The task logs provide information about the error that caused the failure.

To help identify and resolve task failures, you can set up error notifications. See [Error Notifications in Airflow](/docs/learn/error-notifications-in-airflow).

Task failures in newly developed DAGs with error messages such as `Task exited with return code Negsignal.SIGKILL` or containing a `-9` error code are often caused by a lack of memory. Increase the resources for your scheduler, webserver, or pod, depending on whether you're running the Local, Celery, or Kubernetes executors respectively.

<Info>
  After resolving your issue you may want to rerun your DAGs or tasks, see [Rerunning DAGs](/docs/learn/rerunning-dags).
</Info>

### Issues with dynamically mapped tasks

[Dynamic task mapping](/docs/learn/dynamic-tasks) is a powerful feature that allows you to dynamically adjust the number of tasks at runtime based on changing input parameters. It is also possible to [dynamically map over task groups](/docs/learn/task-groups#generate-task-groups-dynamically-at-runtime).

Possible causes of issues when working with dynamically mapped tasks include:

* You didn't provide a keyword argument to the `.expand()` function.
* When using `.expand_kwargs()`, you didn't provide mapped parameters in the form of a `List(Dict)`.
* You tried to map over an empty list, which causes the mapped task to be skipped.
* You exceeded the limit for how many mapped task instances you can create. This limit depends on the Airflow core config [`max_map_length`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#max-map-length) and is 1024 by default.
* The number of mapped task instances of a specific task that can run in parallel across all runs of a given DAG depend on the task level parameter `max_active_tis_per_dag`.
* Not all parameters are mappable. If a parameter doesn't support mapping you will see an import error.

When creating complex patterns with dynamically mapped tasks, we recommend first creating your DAG structure using EmptyOperators or decorated Python operators. Once the structure works as intended, you can start adding your tasks. Refer to the [Create dynamic Airflow tasks](/docs/learn/dynamic-tasks) guide for code examples.

<Tip>
  It is very common that the output of an upstream operator is in a slightly different format than what you need to map over. Use [`.map()`](/docs/learn/dynamic-tasks#transform-outputs-with-map) to transform elements in a list using a Python function.
</Tip>

## Missing logs

When you check your task logs to debug a failure, you may not see any logs. On the log page in the Airflow UI, you may see a spinning wheel, or you may just see a blank file.

Generally, logs fail to appear when a process dies in your scheduler or worker and communication is lost. The following are some debugging steps you can try:

* Try rerunning the task by [clearing the task instance](/docs/learn/rerunning-dags#manually-rerun-tasks-or-dags) to see if the logs appear during the rerun.
* Increase your `log_fetch_timeout_sec` parameter to greater than the 5 second default. This parameter controls how long the webserver waits for the initial handshake when fetching logs from the worker machines, and having extra time here can sometimes resolve issues.
* Increase the resources available to your workers (if using the Celery executor) or scheduler (if using the local executor).
* If you're using the Kubernetes executor and a task fails very quickly (in less than 15 seconds), the pod running the task spins down before the webserver has a chance to collect the logs from the pod. If possible, try building in some wait time to your task depending on which operator you're using. If that isn't possible, try to diagnose what could be causing a near-immediate failure in your task. This is often related to either lack of resources or an error in the task configuration.
* Increase the CPU or memory for the task.
* Ensure that your logs are retained until you need to access them. If you are an Astronomer customer see our documentation on how to [View logs](/docs/astro/view-logs).
* Check your scheduler and webserver logs for any errors that might indicate why your task logs aren't appearing.

## Troubleshooting connections

Typically, Airflow connections are needed to allow Airflow to communicate with external systems. Most hooks and operators expect a defined connection parameter. Because of this, improperly defined connections are one of the most common issues Airflow users have to debug when first working with their DAGs.

While the specific error associated with a poorly defined connection can vary widely, you will typically see a message with "connection" in your task logs. If you haven't defined a connection, you'll see a message such as `'connection_abc' is not defined`.

The following are some debugging steps you can try:

* Review [Manage connections in Apache Airflow](/docs/learn/connections) to learn how connections work.
* Make sure you have the necessary provider packages installed to be able to use a specific connection type.
* Change the `<external tool>_default` connection to use your connection details or define a new connection with a different name and pass the new name to the hook or operator.
* Define connections using Airflow environment variables instead of adding them in the Airflow UI. Make sure you're not defining the same connection in multiple places. If you do, the environment variable takes precedence.
* Test if your credentials work when used in a direct API call to the external tool.

To find information about what parameters are required for a specific connection:

* Read the provider documentation in the [Airflow Registry](https://airflow.apache.org/registry/providers/?page=1) to access the Apache Airflow documentation for the provider. Most commonly used providers will have documentation on each of their associated connection types. For example, you can find information on how to set up different connections to Azure in the Azure provider docs.
* Check the documentation of the external tool you are connecting to and see if it offers guidance on how to authenticate.
* View the source code of the hook that is being used by your operator.

## I need more help

The information provided here should help you resolve the most common issues. If your issue wasn't covered in this guide, try the following resources:

* If you are an Astronomer customer contact our [customer support](https://support.astronomer.io/).
* Post your question to [Stack Overflow](https://stackoverflow.com/), tagged with `airflow` and other relevant tools you are using. Using Stack Overflow is ideal when you are unsure which tool is causing the error, since experts for different tools will be able to see your question.
* Join the [Apache Airflow Slack](https://apache-airflow-slack.herokuapp.com/) and open a thread in `#newbie-questions` or `#troubleshooting`. The Airflow slack is the best place to get answers to more complex Airflow specific questions.
* If you found a bug in Airflow or one of its core providers, open an issue in the [Airflow GitHub repository](https://github.com/apache/airflow/issues). For bugs in Astronomer open source tools open an issue in the relevant [Astronomer repository](https://github.com/astronomer).

To get more specific answers to your question, include the following information in your question or issue:

* Your method for running Airflow (Astro CLI, standalone, Docker, managed services).
* Your Airflow version and the version of relevant providers.
* The full error with the error trace if applicable.
* The full code of the DAG causing the error if applicable.
* What you are trying to accomplish in as much detail as possible.
* What you changed in your environment when the problem started.


# Running asynchronous processes in Apache Airflow®
Source: https://astronomer.io/docs/learn/deferrable-operators

Run asynchronous Python in Airflow tasks and use deferrable operators to free up worker resources.

Apache Airflow supports two approaches for running asynchronous Python code: async tasks, which refers to running async functions provided to the `@task` decorator or the `PythonOperator`, and deferrable operators. Async tasks run concurrent async code directly on workers, while deferrable operators offload long-running polling to the [triggerer component](/docs/learn/airflow-components), releasing the worker slot. Both approaches use Python's [asyncio](https://docs.python.org/3/library/asyncio.html) library.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow sensors. See [Sensors 101](/docs/learn/what-is-a-sensor).
* Python's [asyncio](https://docs.python.org/3/library/asyncio.html) library.

## When to use async tasks vs deferrable operators

Airflow provides two distinct mechanisms for asynchronous execution. The right choice depends on what your task does while it waits.

|                  | Async tasks                                                                                                                                                                                | Deferrable operators                                                                                                                                          |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Use when**     | You want to run concurrent operations within a single task, for example making multiple API calls in parallel and/or fetching larger amounts of data asynchronously from multiple sources. | You want a task to wait for an external condition such as a file landing or a job completing and you expect long wait times.                                  |
| **How it works** | The worker runs your `async def` function directly, including `await` and `asyncio.gather` calls.                                                                                          | The task pauses, releases the worker slot, and submits a trigger to the triggerer process.                                                                    |
| **Runs on**      | Worker                                                                                                                                                                                     | Triggerer                                                                                                                                                     |
| **Worker slot**  | Occupied for the duration of the task                                                                                                                                                      | Released while the task is deferred                                                                                                                           |
| **Requires**     | Airflow 3.2+                                                                                                                                                                               | A running triggerer process and a deferrable operator for the use case. You can create your own [custom deferrable operators](#create-a-deferrable-operator). |

## Async tasks

In Airflow 3.2+, workers can run `async def` functions natively. You can define an async function as a task using the `@task` decorator or the `PythonOperator`. The worker executes the function in an asyncio event loop, allowing you to use `await`, `asyncio.gather`, and other asyncio patterns directly.

### Async @task decorator

The following example Dag defines an async task that fetches data from two endpoints concurrently using `asyncio.gather`. Both requests run in parallel, completing in roughly 10 seconds instead of the 15 seconds it would take to fetch them sequentially.

```python wrap theme={null}
from airflow.sdk import dag, task

async def fetch_one():
    import httpx
    async with httpx.AsyncClient(timeout=30) as client:
        response = await client.get("https://httpbin.org/delay/5")
        return response.json()

async def fetch_two():
    import httpx
    async with httpx.AsyncClient(timeout=30) as client:
        response = await client.get("https://httpbin.org/delay/10")
        return response.json()

@dag
def async_example():

    @task
    async def fetch_concurrently():
        import asyncio
        import time
        start = time.monotonic()
        slow, fast = await asyncio.gather(fetch_one(), fetch_two())
        elapsed = time.monotonic() - start
        print(f"Both done in {elapsed:.1f}s")

    fetch_concurrently()

async_example()
```

### Async `PythonOperator`

You can also pass an async callable to the `PythonOperator`:

```python wrap theme={null}
from airflow.sdk import dag
from airflow.providers.standard.operators.python import PythonOperator

async def my_async_function():
    import asyncio
    await asyncio.sleep(1)
    return "done"

@dag
def async_python_operator_example():

    PythonOperator(
        task_id="async_task",
        python_callable=my_async_function,
    )

async_python_operator_example()
```

## Deferrable operators

Deferrable operators use the Python [asyncio](https://docs.python.org/3/library/asyncio.html) library to efficiently run tasks waiting for an external resource to finish. When a task is deferred, it releases its worker slot and submits a trigger to the triggerer process. This frees up your workers and allows you to use resources more effectively.

### Terms and concepts

Review the following terms and concepts to gain a better understanding of deferrable operator functionality:

* [asyncio](https://docs.python.org/3/library/asyncio.html): A Python library used as the foundation for multiple asynchronous frameworks. This library is core to deferrable operator functionality, and is used when writing triggers.
* Triggers: Small, asynchronous sections of Python code. Due to their asynchronous nature, they coexist efficiently in a single process known as the triggerer.
* Triggerer: An Airflow service similar to a scheduler or a worker that runs an [asyncio event loop](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio-event-loop) in your Airflow environment. Running a triggerer is essential for using deferrable operators.
* Deferred: An Airflow task state indicating that a task has paused its execution, released the worker slot, and submitted a trigger to be picked up by the triggerer process.

### How deferrable operators work

With traditional operators, a task submits a job to an external system such as a Spark cluster and then polls the job status until it is completed. Although the task isn't doing significant work, it still occupies a worker slot during the polling process. As worker slots are occupied, tasks are queued and start times are delayed. The following image illustrates this process:

<Frame>
  <img alt="Classic Worker" />
</Frame>

With deferrable operators, worker slots are released when a task is polling for the job status. When the task is deferred, the polling process is offloaded as a trigger to the triggerer, and the worker slot becomes available. The triggerer can run many asynchronous polling tasks concurrently, and this prevents polling tasks from occupying your worker resources. When the terminal status for the job is received, the operator resumes the task, taking up a worker slot while it finishes. The following image illustrates the process:

<Frame>
  <img alt="Deferrable Worker" />
</Frame>

<Info>
  Some deferrable operators directly enter a deferred state without going to a worker first, see [Triggering Deferral from Start](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html#triggering-deferral-from-task-start).
</Info>

### Benefits

There are numerous benefits to using deferrable operators:

* Reduced resource consumption: Depending on the available resources and the workload of your triggers, you can run hundreds to thousands of deferred tasks in a single triggerer process. This can lead to a reduction in the number of workers needed to run tasks during periods of high concurrency. With fewer workers needed, you can scale down the underlying infrastructure of your Airflow environment.
* Resiliency against restarts: Triggers are stateless by design. This means your deferred tasks aren't set to a failure state if a triggerer needs to be restarted due to a deployment or infrastructure issue. When a triggerer is back up and running in your environment, your deferred tasks resume.

<Tip>
  When you can't use a deferrable operator for a longer running sensor task, such as when you can't run a triggerer, Astronomer recommends using a sensor in [`reschedule` mode](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/sensors.html) to reduce unnecessary resource overhead. See the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html#difference-between-mode-reschedule-and-deferrable-true-in-sensors) for details about the differences between deferrable operators and sensors in `reschedule` mode.
</Tip>

### Use deferrable operators

Deferrable operators should be used whenever you have tasks that occupy a worker slot while polling for a condition in an external system. For example, using deferrable operators for sensor tasks can provide efficiency gains and reduce operational costs.

#### Start a triggerer

To use deferrable operators, you must have a triggerer running in your Airflow environment. On Astro the triggerer is automatically configured for every Deployment. If you are using Astro Private Cloud, see [Configure a Deployment on Astro Private Cloud - Triggerer](/docs/astro-private-cloud/v-2-x/configure-deployment#triggerer).

If you aren't using Astro, run `airflow triggerer` to start a triggerer process in your Airflow environment. Your output should look similar to the following image:

<Frame>
  <img alt="Triggerer Logs" />
</Frame>

As tasks are raised into a deferred state, triggers are registered in the triggerer. You can set the number of concurrent triggers that can run in a single triggerer process with the [`default_capacity`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#triggerer) configuration setting in Airflow. This config can also be set with the `AIRFLOW__TRIGGERER__DEFAULT_CAPACITY` environment variable. The default value is `1000`.

#### Use deferrable versions of operators

Many Airflow operators, such as the [`TriggerDagRunOperator`](https://airflow.apache.org/registry/providers/standard#standard-trigger_dagrun-TriggerDagRunOperator) and the [`WasbBlobSensor`](https://airflow.apache.org/registry/providers/microsoft-azure#microsoft-azure-wasb-WasbBlobSensor), can be set to run in deferrable mode using the `deferrable` parameter. You can check if the operator you want to use has a `deferrable` parameter in the [Airflow Registry](https://airflow.apache.org/registry/).

To always use the deferrable version of an operator if it's available, set the Airflow config `operators.default_deferrable` to `True`. You can do so by defining the following environment variable in your Airflow environment:

```text wrap theme={null}
AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE=True
```

After you set the variable, all operators with a `deferrable` parameter run as their deferrable version by default. You can override the config setting at the operator level using the `deferrable` parameter directly:

```python wrap theme={null}
trigger_dag_run = TriggerDagRunOperator(
   task_id="task_in_downstream_dag",
   trigger_dag_id="downstream_dag",
   wait_for_completion=True,
   poke_interval=20,
   deferrable=False,  # turns off deferrable mode just for this operator instance
)
```

You can find a list of operators that support deferrable mode in the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/deferrable-operator-ref.html).

Previously, before the `deferrable` parameter was available in regular operators, deferrable operators were implemented as standalone operators, usually with an `-Async` suffix. Some of these operators are still available. For example, the `DateTimeSensor` doesn't have a `deferrable` parameter, but has a deferrable version called `DateTimeSensorAsync`.

<Info>
  The [Astronomer providers](https://github.com/astronomer/astronomer-providers) package, which contained many `-Async` operators, is deprecated. The functionality from most of these operators is integrated into their original operator version in the relevant Airflow provider package.
</Info>

### Example: Deferrable sensor

The following example Dag is scheduled to run every minute between its `start_date` and its `end_date`. Every Dag run contains one sensor task that will potentially take up to 20 minutes to complete.

```python wrap theme={null}
from airflow.decorators import dag
from airflow.sensors.date_time import DateTimeSensor
from pendulum import datetime


@dag(
    start_date=datetime(2024, 5, 23, 20, 0),
    end_date=datetime(2024, 5, 23, 20, 19),
    schedule="* * * * *",
    catchup=True,
)
def sync_dag_2():
    DateTimeSensor(
        task_id="sync_task",
        target_time="""{{ macros.datetime.utcnow() + macros.timedelta(minutes=20) }}""",
    )


sync_dag_2()
```

Using `DateTimeSensor`, one worker slot is taken up by every sensor that runs. By using the deferrable version of this sensor, `DateTimeSensorAsync`, you can achieve full concurrency while freeing up your workers to complete additional tasks across your Airflow environment.

In the following image, running the Dag produces 16 running task instances, each containing one active `DateTimeSensor` taking up one worker slot.

<Frame>
  <img alt="Standard sensor Grid View" />
</Frame>

Because Airflow imposes default limits on the number of active runs of the same Dag or number of active tasks in a Dag across all runs, you'll have to scale up Airflow to concurrently run any other Dags and tasks as described in the [Scaling Airflow to optimize performance](/docs/learn/airflow-scaling-workers) guide.

Switching out the `DateTimeSensor` for `DateTimeSensorAsync` creates 16 running Dag instances, but the tasks for these Dags are in a deferred state which doesn't take up a worker slot. The only difference in the Dag code is using the deferrable operator `DateTimeSensorAsync` over `DateTimeSensor`:

```python wrap theme={null}
from airflow.decorators import dag
from pendulum import datetime
from airflow.sensors.date_time import DateTimeSensorAsync


@dag(
    start_date=datetime(2024, 5, 23, 20, 0),
    end_date=datetime(2024, 5, 23, 20, 19),
    schedule="* * * * *",
    catchup=True,
)
def async_dag_2():
    DateTimeSensorAsync(
        task_id="async_task",
        target_time="""{{ macros.datetime.utcnow() + macros.timedelta(minutes=20) }}""",
    )


async_dag_2()
```

In the following image, all tasks are shown in a deferred (violet) state. Tasks in other Dags can use the available worker slots, making the deferrable operator more cost and time-efficient.

<Frame>
  <img alt="Deferrable sensor Grid View" />
</Frame>

### High availability

Triggers are designed to be highly available. You can implement this by starting multiple triggerer processes. Similar to the [HA scheduler](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html#running-more-than-one-scheduler), Airflow ensures that they co-exist with correct locking and high availability. See [High Availability](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html#high-availability) for more information on this topic.

### Create a deferrable operator

If you have an operator that would benefit from being asynchronous but doesn't yet exist in OSS Airflow, you can create your own by writing a deferrable operator and trigger class. You can also defer a task several times if needed.

Get a template for a custom deferrable operator and custom trigger class by clicking the dropdown. Make sure to adjust the classpath for your trigger's `.serialize` method (currently `include.deferrable_operator_template.MyTrigger`) to match your file structure.

<details>
  <summary>Click to view the template code</summary>

  ```python expandable wrap theme={null}
  from __future__ import annotations
  import asyncio
  import time
  from asgiref.sync import sync_to_async
  from typing import Any, Sequence, AsyncIterator
  from airflow.configuration import conf
  from airflow.models.baseoperator import BaseOperator
  from airflow.triggers.base import BaseTrigger, TriggerEvent
  from airflow.utils.context import Context


  class MyTrigger(BaseTrigger):
      """
      This is an example of a custom trigger that waits for a binary random choice
      between 0 and 1 to be 1.
      Args:
          poll_interval (int): How many seconds to wait between async polls.
          my_kwarg_passed_into_the_trigger (str): A kwarg that is passed into the trigger.
      Returns:
          my_kwarg_passed_out_of_the_trigger (str): A kwarg that is passed out of the trigger.
      """

      def __init__(
          self,
          poll_interval: int = 60,
          my_kwarg_passed_into_the_trigger: str = "notset",
          my_kwarg_passed_out_of_the_trigger: str = "notset",
          # you can add more arguments here
      ):
          super().__init__()
          self.poll_interval = poll_interval
          self.my_kwarg_passed_into_the_trigger = my_kwarg_passed_into_the_trigger
          self.my_kwarg_passed_out_of_the_trigger = my_kwarg_passed_out_of_the_trigger

      def serialize(self) -> tuple[str, dict[str, Any]]:
          """
          Serialize MyTrigger arguments and classpath.
          All arguments must be JSON serializable.
          This will be returned by the trigger when it is complete and passed as `event` to the
          `execute_complete` method of the deferrable operator.
          """

          return (
              "include.deferrable_operator_template.MyTrigger",  # this is the classpath for the Trigger
              {
                  "poll_interval": self.poll_interval,
                  "my_kwarg_passed_into_the_trigger": self.my_kwarg_passed_into_the_trigger,
                  "my_kwarg_passed_out_of_the_trigger": self.my_kwarg_passed_out_of_the_trigger,
                  # you can add more kwargs here
              },
          )

      # The run method is an async generator that yields TriggerEvents when the desired condition is met
      async def run(self) -> AsyncIterator[TriggerEvent]:
          while True:
              result = (
                  await self.my_trigger_function()
              )  # The my_trigger_function is awaited and where the condition is checked
              if result == 1:
                  self.log.info(f"Result was 1, thats the number! Triggering event.")

                  self.log.info(
                      f"Kwarg passed in was: {self.my_kwarg_passed_into_the_trigger}"
                  )
                  # This is how you pass data out of the trigger, by setting attributes that get serialized
                  self.my_kwarg_passed_out_of_the_trigger = "apple"
                  self.log.info(
                      f"Kwarg to be passed out is: {self.my_kwarg_passed_out_of_the_trigger}"
                  )
                  # Fire the trigger event! This gets a worker to execute the operator's `execute_complete` method
                  yield TriggerEvent(self.serialize())
                  return  # The return statement prevents the trigger from running again
              else:
                  self.log.info(
                      f"Result was not the one we are waiting for. Sleeping for {self.poll_interval} seconds."
                  )
                  # If the condition is not met, the trigger sleeps for the poll_interval
                  # this code can run multiple times until the condition is met
                  await asyncio.sleep(self.poll_interval)

      # This is the function that is awaited in the run method
      @sync_to_async
      def my_trigger_function(self) -> str:
          """
          This is where what you are waiting for goes For example a call to an
          API to check for the state of a cloud resource.
          This code can run multiple times until the condition is met.
          """

          import random

          randint = random.choice([0, 1])
          self.log.info(f"Random number: {randint}")

          return randint


  class MyOperator(BaseOperator):
      """
      Deferrable operator that waits for a binary random choice between 0 and 1 to be 1.
      Args:
          wait_for_completion (bool): Whether to wait for the trigger to complete.
          poke_interval (int): How many seconds to wait between polls,
              both in deferrable or sensor mode.
          deferrable (bool): Whether to defer the operator. If set to False,
              the operator will act as a sensor.
      Returns:
          str: A kwarg that is passed through the trigger and returned by the operator.
      """

      template_fields: Sequence[str] = (
          "wait_for_completion",
          "poke_interval",
      )
      ui_color = "#73deff"

      def __init__(
          self,
          *,
          # you can add more arguments here
          wait_for_completion: bool = False,
          poke_interval: int = 60,
          deferrable: bool = conf.getboolean(
              "operators", "default_deferrable", fallback=False
          ),  # this default is a convention to be able to set the operator to deferrable in the config
          # using AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE=True
          **kwargs,
      ) -> None:
          super().__init__(**kwargs)

          self.wait_for_completion = wait_for_completion
          self.poke_interval = poke_interval
          self._defer = deferrable

      def execute(self, context: Context):

          # Add code you want to be executed before the deferred part here (this code only runs once)

          # turns operator into sensor/deferred operator
          if self.wait_for_completion:
              # Starting the deferral process
              if self._defer:
                  self.log.info(
                      "Operator in deferrable mode. Starting the deferral process."
                  )
                  self.defer(
                      trigger=MyTrigger(
                          poll_interval=self.poke_interval,
                          my_kwarg_passed_into_the_trigger="lemon",
                          # you can pass information into the trigger here
                      ),
                      method_name="execute_complete",
                      kwargs={"kwarg_passed_to_execute_complete": "tomato"},
                      # kwargs get passed through to the execute_complete method
                  )
              else:  # regular sensor part
                  while True:
                      self.log.info("Operator in sensor mode. Polling.")
                      time.sleep(self.poke_interval)
                      import random

                      # This is where you would check for the condition you are waiting for
                      # when using the operator as a regular sensor
                      # This code can run multiple times until the condition is met

                      randint = random.choice([0, 1])

                      self.log.info(f"Random number: {randint}")
                      if randint == 1:
                          self.log.info("Result was 1, thats the number! Continuing.")
                          return randint
                      self.log.info(
                          "Result was not the one we are waiting for. Sleeping."
                      )
          else:
              self.log.info("Not waiting for completion.")

          # Add code you want to be executed after the deferred part here (this code only runs once)
          # you can have as many deferred parts as you want in an operator

      def execute_complete(
          self,
          context: Context,
          event: tuple[str, dict[str, Any]],
          kwarg_passed_to_execute_complete: str,  # make sure to add the kwargs you want to pass through
      ):
          """Execute when the trigger is complete. This code only runs once."""

          self.log.info("Trigger is complete.")
          self.log.info(f"Event: {event}")  # printing the serialized event

          # you can push additional data to XCom here
          context["ti"].xcom_push(
              "message_from_the_trigger", event[1]["my_kwarg_passed_out_of_the_trigger"]
          )

          return kwarg_passed_to_execute_complete  # the returned value gets pushed to XCom as `return_value`
  ```
</details>

Note that when developing a custom trigger, you need to restart your triggerer to pick up any changes you make, since the triggerer caches the trigger classes. Additionally, all information you pass between the triggerer and the worker must be JSON serializable.

See [Writing Deferrable Operators](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html#writing-deferrable-operators) for more information.

You can implement direct deferral without the task ever being picked back up by a worker. The following code shows a deferrable operator circumventing the `.execute()` method. When using this template, make sure to adjust the classpath for your trigger (currently `include.deferrable_operator_template.MyTrigger`) in both the `.serialize` method and the `StartTriggerArgs` to match your file structure.

<details>
  <summary>Click to view the template code</summary>

  ```python expandable wrap theme={null}
  from __future__ import annotations
  import asyncio
  import time
  from asgiref.sync import sync_to_async
  from typing import Any, Sequence, AsyncIterator
  from airflow.configuration import conf
  from airflow.models.baseoperator import BaseOperator
  from airflow.triggers.base import BaseTrigger, TriggerEvent
  from airflow.utils.context import Context
  from airflow.triggers.base import StartTriggerArgs


  class MyTrigger(BaseTrigger):
      """
      This is an example of a custom trigger that waits for a binary random choice
      between 0 and 1 to be 1.
      Args:
          poll_interval (int): How many seconds to wait between async polls.
          my_kwarg_passed_into_the_trigger (str): A kwarg that is passed into the trigger.
      Returns:
          my_kwarg_passed_out_of_the_trigger (str): A kwarg that is passed out of the trigger.
      """

      def __init__(
          self,
          poll_interval: int = 60,
          my_kwarg_passed_into_the_trigger: str = "notset",
          my_kwarg_passed_out_of_the_trigger: str = "notset",
          # you can add more arguments here
      ):
          super().__init__()
          self.poll_interval = poll_interval
          self.my_kwarg_passed_into_the_trigger = my_kwarg_passed_into_the_trigger
          self.my_kwarg_passed_out_of_the_trigger = my_kwarg_passed_out_of_the_trigger

      def serialize(self) -> tuple[str, dict[str, Any]]:
          """
          Serialize MyTrigger arguments and classpath.
          All arguments must be JSON serializable.
          This will be returned by the trigger when it is complete and passed as `event` to the
          `execute_complete` method of the deferrable operator.
          """

          return (
              "include.custom_deferrable_operator.MyTrigger",  # this is the classpath for the Trigger
              {
                  "poll_interval": self.poll_interval,
                  "my_kwarg_passed_into_the_trigger": self.my_kwarg_passed_into_the_trigger,
                  "my_kwarg_passed_out_of_the_trigger": self.my_kwarg_passed_out_of_the_trigger,
                  # you can add more kwargs here
              },
          )

      # The run method is an async generator that yields TriggerEvents when the desired condition is met
      async def run(self) -> AsyncIterator[TriggerEvent]:
          while True:
              result = (
                  await self.my_trigger_function()
              )  # The my_trigger_function is awaited and where the condition is checked
              if result == 1:
                  self.log.info(f"Result was 1, thats the number! Triggering event.")

                  self.log.info(
                      f"Kwarg passed in was: {self.my_kwarg_passed_into_the_trigger}"
                  )
                  # This is how you pass data out of the trigger, by setting attributes that get serialized
                  self.my_kwarg_passed_out_of_the_trigger = "apple"
                  self.log.info(
                      f"Kwarg to be passed out is: {self.my_kwarg_passed_out_of_the_trigger}"
                  )
                  # Fire the trigger event! This gets a worker to execute the operator's `execute_complete` method
                  yield TriggerEvent(self.serialize())
                  return  # The return statement prevents the trigger from running again
              else:
                  self.log.info(
                      f"Result was not the one we are waiting for. Sleeping for {self.poll_interval} seconds."
                  )
                  # If the condition is not met, the trigger sleeps for the poll_interval
                  # this code can run multiple times until the condition is met
                  await asyncio.sleep(self.poll_interval)

      # This is the function that is awaited in the run method
      @sync_to_async
      def my_trigger_function(self) -> str:
          """
          This is where what you are waiting for goes For example a call to an
          API to check for the state of a cloud resource.
          This code can run multiple times until the condition is met.
          """

          import random

          randint = random.choice([0, 1])
          self.log.info(f"Random number: {randint}")

          return randint


  class MyDeferrableOperator(BaseOperator):
      """
      Deferrable operator that waits for a binary random choice between 0 and 1 to be 1.
      Args:
          wait_for_completion (bool): Whether to wait for the trigger to complete.
          poke_interval (int): How many seconds to wait between polls,
              both in deferrable or sensor mode.
          deferrable (bool): Whether to defer the operator. If set to False,
              the operator will act as a sensor.
      Returns:
          str: A kwarg that is passed through the trigger and returned by the operator.
      """

      template_fields: Sequence[str] = (
          "wait_for_completion",
          "poke_interval",
      )
      ui_color = "#73deff"

      # --------------------------------------------------------- #
      # New implementation directly starting the trigger - Part 1 #
      # --------------------------------------------------------- #

      start_trigger_args = StartTriggerArgs(
          trigger_cls="include.custom_deferrable_operator.MyTrigger",
          trigger_kwargs={
              "poll_interval": 60,
              "my_kwarg_passed_into_the_trigger": "lemon",
          },
          next_method="execute_complete",
          next_kwargs={"kwarg_passed_to_execute_complete": "tomato"},
          timeout=None,
      )
      start_from_trigger = True

      def __init__(
          self,
          *,
          # you can add more arguments here
          wait_for_completion: bool = False,
          poke_interval: int = 60,
          deferrable: bool = conf.getboolean(
              "operators", "default_deferrable", fallback=False
          ),  # this default is a convention to be able to set the operator to deferrable in the config
          # using AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE=True
          **kwargs,
      ) -> None:
          super().__init__(**kwargs)

          self.wait_for_completion = wait_for_completion
          self.poke_interval = poke_interval
          self._defer = deferrable

          # --------------------------------------------------------- #
          # New implementation directly starting the trigger - Part 2 #
          # --------------------------------------------------------- #

          self.start_trigger_args.trigger_kwargs = dict(
              poll_interval=self.poke_interval,
              my_kwarg_passed_into_the_trigger="lemon",
          )

      def execute_complete(
          self,
          context: Context,
          event: tuple[str, dict[str, Any]],
          kwarg_passed_to_execute_complete: str,  # make sure to add the kwargs you want to pass through
      ):
          """Execute when the trigger is complete. This code only runs once."""

          self.log.info("Trigger is complete.")
          self.log.info(f"Event: {event}")  # printing the serialized event

          # you can push additional data to XCom here
          context["ti"].xcom_push(
              "message_from_the_trigger", event[1]["my_kwarg_passed_out_of_the_trigger"]
          )

          return kwarg_passed_to_execute_complete  # the returned value gets pushed to XCom as `return_value`
  ```
</details>

See [Triggering Deferral from Start](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html#triggering-deferral-from-task-start) for more details and code examples.


# Develop Apache Airflow® Dags with AI
Source: https://astronomer.io/docs/learn/develop-dags-with-ai

Develop Apache Airflow® Dags locally with the help of AI agents.

AI coding agents can draft an Apache Airflow® Dag for you. Whether that draft is useful depends on the agent's context, the quality of your spec, and whether your agent was able to iteratively improve its work by testing against given success criteria.

Astronomer's data engineering agent, [Otto](/docs/astro/otto-overview), has the best practices described in this guide built in, plus additional advanced capabilities for local data engineering. See [Working with Otto](#working-with-otto).

This guide covers:

* How to use Otto to write best practice Dags
* Writing a spec with defined success criteria
* Letting an agent run in a loop to iteratively write Dags

<Info>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Blog post: [Best practices for writing Airflow Dags with AI](https://www.astronomer.io/blog/best-practices-for-writing-airflow-dags-with-ai/).
</Info>

## Assumed knowledge

To get the most out of this guide, you should have:

* A local Airflow environment. See [Run Airflow locally](/docs/learn/run-airflow-locally).
* An agent that knows about your Airflow environment. See [AI context for data engineering](/docs/learn/ai-context-for-data-engineering).
* Basic familiarity with an AI coding agent harness, such as Claude Code or Cursor.

## Working with Otto

<Info>
  **Labs**

  This feature is in [Labs](/docs/astro/feature-previews).
</Info>

The easiest way to develop best practice Dags with AI is by using Otto, Astronomer's data engineering agent. You can interact with Otto directly or delegate tasks to Otto from another coding agent like Claude Code, Codex, or Gemini CLI.

### Work with Otto directly

You can work with Otto in the Astro UI, in the Astro IDE, through an API endpoint, or using the Astro CLI. This guide focuses on how to use the Astro CLI to interact with Otto. For other interfaces, see the [Otto documentation](/docs/astro/otto-overview).

To interact with Otto from your terminal:

1. Install the [Astro CLI](/docs/cli/v1.45/overview). Make sure you are on the latest version.

2. Sign in to your Astro account with `astro login` (a [free trial](https://www.astronomer.io/lp/signup/) is available). This is necessary to work with Otto because Otto uses an LLM gateway hosted by Astronomer.

3. Make sure you are in your Astro project directory, or create a new one by running `astro dev init`.

4. Start your Airflow project by running `astro dev start`.

<Note>
  If your Dag connects to other systems, you'll need to provide the credentials in Airflow connections. See [Manage connections in Apache Airflow](/docs/learn/connections) for general information on Airflow connections, and [Manage Airflow connections, variables, and environment variables](/docs/astro/manage-connections-variables) for how to add Airflow connections to Astro Deployments.
</Note>

5. Run `astro otto` to start your interactive Otto session in the terminal. Type `/model` to choose your preferred [model](/docs/astro/otto-models-and-regions).

   <Frame>
     <img alt="Terminal session showing the astro otto command starting an interactive Otto session, with the project, Airflow version, Runtime version, and model listed." />
   </Frame>

6. Ask Otto to write you a spec and describe the pipeline you'd like to create in natural language, for example, "Write me a spec to have an AI agent answer support tickets".

7. Answer any clarifying questions Otto asks.

   <Frame>
     <img alt="Otto asking a clarifying question about how much autonomy the agent should have, with options ranging from draft-only to fully autonomous." />
   </Frame>

8. Read the spec Otto writes for the planned Dag and suggest any necessary changes. The following is a section of a detailed Otto-written spec for an AI orchestration pipeline.

   <Frame>
     <img alt="Section of an Otto-written spec for a support-ticket AI agent, showing an autonomy model table and an architecture diagram from the Zendesk webhook to the ticket agent and logging store." />
   </Frame>

9. Once you are happy with the spec, tell Otto to implement it. Otto automatically develops in an agentic loop, testing its own changes with `astro dev parse`, while being aware of your current Airflow environment, following the conventions in your project, and having access to all relevant Airflow-related skills.

   <Frame>
     <img alt="Otto implementing a Dag in an agentic loop, inspecting installed providers with the Astro CLI, reading the current Airflow instance, and editing requirements.txt to add required provider packages." />
   </Frame>

10. Go to `localhost:8080` to inspect and run your Dag. Ask Otto to make any necessary changes.

    <Frame>
      <img alt="Graph view of the Otto-generated support_ticket_agent Dag in the Airflow UI, showing a successful run through fetch_new_tickets, build_context, generate_reply, assess_guardrails, route, human_review, auto_send, reject_ticket, and send_approved_reply." />
    </Frame>

### Delegate to Otto

If you prefer to work from within your existing AI agent setup, such as Claude Code, you can still use Otto to develop your Dags by delegating Dag-writing related tasks to Otto.

1. Add the [Delegating to Otto](https://github.com/astronomer/agents/blob/main/skills/delegating-to-otto/SKILL.md) skill to your AI agent.

2. Now, whenever you mention "use Otto" or a similar phrase, your AI agent delegates the work to Otto, running as a sub-agent.

   <Frame>
     <img alt="Claude Code loading the delegating-to-otto skill after being asked to use Otto to write a Dag that extracts structured data from unstructured invoices and customer feedback, checking that local Airflow is running, then delegating the task to astro otto." />
   </Frame>

## Using other coding agents

If you cannot use Otto, you can manually follow this section's steps to improve the Dag-writing capabilities of your generic AI agent.

### Give your agent context

Since generic AI agents don't have Otto's skills assessing your Airflow environment, you need to give them this context explicitly. Relevant information includes your Airflow version, your provider versions, and the conventions your team has already settled on. See [AI context for data engineering](/docs/learn/ai-context-for-data-engineering).

With the context in place, use your harness's *planning mode*, where the agent writes a plan without permission to edit any files. After you review and approve that plan, have it build a spec as described in the following section.

### Write the spec

Once you are happy with your agent's plan, tell it to turn it into a structured spec. You can think of a spec like a skill or workflow to follow for a specific goal, in this case to write an Airflow pipeline.

<Tip>
  There are several tools that help your agent with writing specs. A commonly used one is [`spec-kit`](https://github.com/github/spec-kit).
</Tip>

Some important considerations when writing a spec (or reviewing an AI-written spec) are:

* Structure the spec with Markdown headings so the model works through requirements in a fixed order: goal, context, data contract (between tasks), and constraints.
* Instruct the agent to write a skeleton Dag first using `EmptyOperator` placeholders for the task graph, then optionally stop to ask you to review the Dag structure before filling in the task logic.
* Specify the format of the input and the output as well as the action you want performed. For example: input is a DataFrame with columns `x` and `y`, output is a dictionary with fields `a` and `b`.
* State [idempotency](/docs/learn/dag-best-practices#review-idempotency) requirements explicitly.
* Keep raw data separate from instructions. If the prompt includes an example payload, a schema, or sample output, wrap it in a fenced code block or an XML-style tag so the model doesn't confuse the data with the instructions around it. Explicitly state that the example is a sample of the general structure.

The [example at the end of this guide](#example-write-an-ai-orchestration-dag) shows a full spec written this way: a goal, the environment and its constraints, the data contracts for what goes in and out of each task, the dependencies between tasks, and the acceptance criteria the finished Dag has to fulfill.

### Define success criteria

A spec only helps if you can check whether the agent fulfilled it. Before you ask an agent to write a Dag, decide what "done" means, ideally in the form of programmatic tests. Tests give you and the agent something concrete to check against, and they let an agent iterate independently.

<Tip>
  To learn more about different options for testing Airflow Dags, see the [Best practices for testing Apache Airflow® Dags](https://www.astronomer.io/ebooks/best-practices-for-testing-apache-airflow-dags) eBook.
</Tip>

When using the Astro CLI, you can define tests in the `tests` directory and run them with the `astro dev pytest` command (independently of whether they use the `pytest` package or another testing framework such as `unittest`).

There are five main types of tests in Airflow:

* **Parsing tests** check whether the Dag parses correctly or results in an import error. Your agent can run one with `astro dev parse`.
* **Unit tests** check the logic in the Python functions an agent writes, such as a custom hook, a custom operator, or functions used in an `@task` or `PythonOperator` task. Write these the same way you would for any Python code. See [Unit testing](/docs/learn/testing-airflow#unit-testing) for examples.
* **Dag validation tests** define rules for your Dags, for example requiring `tags` to be defined, or only allowing specific operators or Dag schedules. See [Write Dag validation tests](/docs/learn/testing-airflow#write-dag-validation-tests).
* **Integration tests** check that individual tasks work against the systems they talk to, rather than against mocks. Often the easiest way to run one is with a helper Dag that interacts with an external system, for example querying a few rows from a specific table in a database and checking the schema against what your Airflow tasks expect.
* **End-to-end (E2E) tests** run a whole Dag or a set of connected Dags. If your agent has access to the Astro CLI, it can trigger these itself to check its work. These Dag runs change data and incur cost in external systems; therefore, Astronomer recommends pointing your development environment at a development replica rather than at production.

<Note>
  Testing Airflow Dag code is a separate concept from testing the quality of the data orchestrated with Airflow Dags. For the latter, see [Data quality and Airflow](/docs/learn/data-quality).
</Note>

### Safety

Before you let an agent iterate on its own, decide what it can do without you.

* Decide which tools and commands your agent can run without asking. Allowlist safe commands such as parsing Dags or running tests.
* Decide which actions need your approval every time. Anything you can't easily undo belongs here, along with anything that writes outside your project folder.

Use version control and commit often when working with AI agents. Review diffs closely: agents sometimes refactor code you didn't ask them to change.

For more information, see [Safety](/docs/learn/local-data-engineering-with-ai-overview#safety).

### Let agents write in a loop

Once your spec, success criteria, and guardrails are in place, it is time to let the agent write your Dags while you grab a nice cup of tea.

With tests defined, an agent can check its own output and only hand you a draft after its code passes the success criteria. The loop is the same one as in regular test-driven development: the agent writes code, runs a check, reads the result, and tries again until the check passes or until it hits a pre-defined timeout in terms of time or tokens spent.

#### Agentic hooks

For Dags, the fastest check in that loop is `astro dev parse`, which catches import errors. Beyond the command permissions covered in [Restrict agent commands](/docs/learn/run-airflow-locally#restrict-agent-commands), Claude Code and similar harnesses let you enforce this check automatically with a hook that runs after every file edit. A `PostToolUse` hook, for example, can run `astro dev parse` after every `Edit` or `Write` call and feed the result back to the agent for the next iteration.

<AccordionGroup>
  <Accordion title="Claude Code settings.json example">
    The following `settings.json` example shows this in practice for Claude Code: an `allow` list for a small set of low-risk commands, a `PreToolUse` hook that blocks destructive commands like `rm -rf` or `DROP TABLE` outright, and a `PostToolUse` hook that runs `astro dev parse` after every `Edit` or `Write` call and logs the result, feeding a pass or fail message straight back to the agent.

    The `env` and `enabledPlugins` entries enable a Python language server. This lets the agent see unresolved imports and type errors from the provider versions installed in your project.

    ```json theme={null}
    {
      "permissions": {
        "allow": [
          "Bash(astro dev parse)",
          "Bash(astro dev pytest *)",
          "Bash(astro dev logs *)",
          "Bash(astro dev run dags list)",
          "Bash(git diff *)",
          "Bash(git log *)"
        ]
      },
      "hooks": {
        "PostToolUse": [
          {
            "matcher": "Edit|Write",
            "hooks": [
              {
                "type": "command",
                "command": "ts=$(date '+%Y-%m-%d %H:%M:%S'); file=$(jq -r '.tool_input.file_path'); echo \"[$ts] PostToolUse Edit|Write on $file\" >> .claude/hook-activity.log"
              },
              {
                "type": "command",
                "command": "ts=$(date '+%Y-%m-%d %H:%M:%S'); file=$(jq -r '.tool_input.file_path'); case \"$file\" in *dags/*.py) output=$(astro dev parse 2>&1); if [ $? -eq 0 ]; then echo \"[$ts] PASS parse $file\" >> .claude/hook-activity.log; jq -n --arg msg \"Dag parse passed: $file\" '{decision: \"block\", reason: $msg}'; else echo \"[$ts] FAIL parse $file\" >> .claude/hook-activity.log; echo \"$output\" >> .claude/hook-activity.log; jq -n --arg msg \"Dag parse failed: $file\\n\\n$output\" '{decision: \"block\", reason: $msg}'; fi ;; esac"
              },
              {
                "type": "command",
                "command": "file=$(jq -r '.tool_input.file_path'); case \"$file\" in *dags/*.py) echo \"Auto-tested Dag: $file\" >> .claude/dag-changelog.md ;; esac"
              }
            ]
          }
        ],
        "PreToolUse": [
          {
            "matcher": "Bash",
            "hooks": [
              {
                "type": "command",
                "command": "cmd=$(jq -r '.tool_input.command'); if echo \"$cmd\" | grep -qE '(rm -rf|DROP TABLE|DROP DATABASE)'; then echo 'BLOCKED: Dangerous command detected' >&2; exit 2; fi"
              }
            ]
          }
        ]
      },
      "env": {
        "ENABLE_LSP_TOOL": "1"
      },
      "enabledPlugins": {
        "pyright-lsp@claude-plugins-official": true
      }
    }
    ```

    <Note>
      This example is for Claude Code. Other harnesses such as [OpenAI Codex](https://openai.com/codex/) or [Google Gemini CLI](https://geminicli.com/) support similar automation, but with different configuration formats. See the relevant harness documentation for more information.
    </Note>
  </Accordion>
</AccordionGroup>

## Example: Write an AI orchestration Dag

The following example shows a spec written with the structure from [Write the spec](#write-the-spec), and the Dag an agent (Claude Code using Sonnet 4.6 with the `astronomer/agent` skills available) produced from that spec in a single pass.

<AccordionGroup>
  <Accordion title="Spec">
    ```markdown theme={null}
    # Spec: AI-Drafted Support Ticket Responder

    ## Goal
    An Airflow Dag that drafts an LLM reply to a support ticket, routes it through a human reviewer, and sends the approved/edited reply. Teaching example for **agentic LLM tasks + human-in-the-loop (HITL) gates**.

    ## Stack / constraints
    - Airflow 3.2, TaskFlow SDK (`@dag`, `@task`, `chain`, `Param`)
    - LLM via `@task.agent` (`llm_conn_id="pydanticai_default"`)
    - HITL via `HITLOperator` (standard provider)
    - Structured LLM output validated by Pydantic
    - Input: one ticket JSON file on disk

    ## Data contracts
    - **Ticket (in):** `{ ticket_id, customer, customer_id, subject, body }`
    - **DraftAnswer (LLM out):** `{ answer: str, urgency: "P0".."P4" }`

    ## Flow
    1. **fetch_support_ticket**: read ticket JSON → dict. `TICKET_PATH = Path(__file__).resolve().parents[1] / "include" / "support_ticket.json"`
    2. **draft_answer** (`@task.agent`): ticket → `DraftAnswer`. Persona: friendly Starfleet support, short replies for busy captains.
    3. **human_in_the_loop** (`HITLOperator`): render ticket + draft. Options: `Approve AI answer` (default), `Request Changes`, `Manual Response`. Free-text param `override_llm_response` carries edit instructions or the manual reply.
    4. **route** (`@task.branch`): reviewer choice → dispatch: approve→send; changes→rewrite; manual→process_manual_response.
    5. **rewrite** (`@task.agent`): apply ONLY requested changes; preserve the rest verbatim.
    6. **process_manual_response**: wrap reviewer's text as the reply (no LLM).
    7. **send_email** (×3, one per branch): emit the final reply.

    ## Requirements
    - Exactly one send task fires per run (mutually exclusive branch).
    - LLM output MUST conform to `DraftAnswer`; retry on violation.
    - Rewrite preserves unrequested content unchanged.
    - HITL defaults to approve so an unattended run still completes.

    ## Non-goals
    - No real email send (print/log is fine).
    - No live ticket queue or batching (single static file).
    - No reply persistence.

    ## Acceptance criteria
    - Parses and runs end-to-end on Airflow 3.x.
    - Each reviewer choice reaches its correct send task; the other two skip.
    - Draft and rewrite both produce schema-valid output.
    ```
  </Accordion>

  <Accordion title="Result">
    ```python theme={null}
    import json
    from pathlib import Path
    from typing import Literal

    from airflow.providers.standard.operators.hitl import HITLOperator
    from airflow.sdk import Param, chain, dag, task
    from pydantic import BaseModel, Field

    TICKET_PATH = Path(__file__).resolve().parents[1] / "include" / "support_ticket.json"


    class DraftAnswer(BaseModel):
        answer: str = Field(description="The reply to send to the customer")
        urgency: Literal["P0", "P1", "P2", "P3", "P4"] = Field(
            description="Ticket priority: P0 is critical, P4 is low"
        )


    @dag
    def answer_support_tickets_sonnet_4_6_with_skills():
        @task
        def fetch_support_ticket() -> dict:
            return json.loads(TICKET_PATH.read_text())

        @task.agent(
            llm_conn_id="pydanticai_default",
            system_prompt=(
                "You are a friendly Starfleet support officer. "
                "Keep replies short and practical — the captain is busy."
            ),
            output_type=DraftAnswer,
        )
        def draft_answer(ticket: dict) -> str:
            return (
                f"Draft a support reply for ticket #{ticket['ticket_id']} "
                f"from {ticket['customer']}.\n\n"
                f"Subject: {ticket['subject']}\n\n"
                f"{ticket['body']}"
            )

        review = HITLOperator(
            task_id="human_in_the_loop",
            subject="Review AI-drafted support reply",
            body=(
                "**Ticket**\n\n"
                "{{ ti.xcom_pull(task_ids='fetch_support_ticket') }}\n\n"
                "**AI Draft**\n\n"
                "{{ ti.xcom_pull(task_ids='draft_answer') }}"
            ),
            options=["Approve AI answer", "Request Changes", "Manual Response"],
            defaults="Approve AI answer",
            params={
                "override_llm_response": Param(
                    "",
                    type=["string", "null"],
                    title="Edit instructions (for changes) or full reply (for manual)",
                )
            },
        )

        @task.branch
        def route(decision: dict) -> str:
            choice = decision["chosen_options"][0]
            if choice == "Request Changes":
                return "rewrite"
            if choice == "Manual Response":
                return "process_manual_response"
            return "send_email_approved"

        @task.agent(
            llm_conn_id="pydanticai_default",
            system_prompt=(
                "You are a friendly Starfleet support officer. "
                "Apply ONLY the requested changes to the draft. "
                "Preserve all other content verbatim."
            ),
            output_type=DraftAnswer,
        )
        def rewrite(draft: dict, decision: dict) -> str:
            instructions = decision["params_input"].get("override_llm_response", "")
            return (
                f"Rewrite the following support reply applying ONLY these changes: {instructions}\n\n"
                f"Original answer:\n{draft['answer']}\n\n"
                f"Original urgency: {draft['urgency']}"
            )

        @task
        def process_manual_response(decision: dict) -> dict:
            reply = decision["params_input"].get("override_llm_response", "")
            return {"answer": reply, "urgency": "P2"}

        @task
        def send_email(reply: dict):
            print(f"[SEND] urgency={reply['urgency']}\n\n{reply['answer']}")

        ticket = fetch_support_ticket()
        draft = draft_answer(ticket)

        chain(draft, review)

        router = route(review.output)

        rewritten = rewrite(draft=draft, decision=review.output)
        manual_reply = process_manual_response(decision=review.output)

        send_approved = send_email.override(task_id="send_email_approved")(reply=draft)
        send_rewritten = send_email.override(task_id="send_email_rewritten")(reply=rewritten)
        send_manual = send_email.override(task_id="send_email_manual")(reply=manual_reply)

        chain(router, [send_approved, rewritten, manual_reply])


    answer_support_tickets_sonnet_4_6_with_skills()
    ```

    <Frame>
      <img alt="Graph view of the Dag showing the task dependencies." />
    </Frame>

    This is exactly what the spec produced, with no edits. Two things a human reviewer would likely change: the human-in-the-loop step renders the raw JSON payload instead of formatted Markdown, and `process_manual_response` hardcodes the urgency to `P2` for every manually written reply, regardless of what the reply says.
  </Accordion>
</AccordionGroup>


# Create dynamic Airflow tasks
Source: https://astronomer.io/docs/learn/dynamic-tasks

How to dynamically create tasks at runtime in your Airflow DAGs.

With **dynamic task mapping**, you can write DAGs that dynamically generate parallel tasks at runtime. This feature is a paradigm shift for DAG design in Airflow, since it allows you to create tasks based on the current runtime environment without having to change your DAG code.

In this guide, you'll learn about dynamic task mapping and complete an example implementation for a common use case.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow Operators. See [Operators 101](/docs/learn/what-is-an-operator).
* How to use Airflow decorators to define tasks. See [Introduction to Airflow Decorators](/docs/learn/airflow-decorators).
* XComs in Airflow. See [Passing Data Between Airflow Tasks](/docs/learn/airflow-passing-data-between-tasks).

## Dynamic task concepts

The Airflow dynamic task mapping feature is based on the [MapReduce](https://en.wikipedia.org/wiki/MapReduce) programming model. Dynamic task mapping creates a single task for each input. The reduce procedure, which is optional, allows a task to operate on the collected output of a mapped task. In practice, this means that your DAG can create an arbitrary number of parallel tasks at runtime based on some input parameter (the map), and then if needed, have a single task downstream of your parallel mapped tasks that depends on their output (the reduce).

Airflow tasks have two functions available to implement the map portion of dynamic task mapping. For the task you want to map, you must pass all operator parameters through one of the following functions.

* `expand()`: This function passes the parameters that you want to map. A separate parallel task is created for each input. For some instances of [mapping over multiple parameters](#mapping-over-multiple-parameters), `.expand_kwargs()` is used instead.
* `partial()`: This function passes any parameters that remain constant across all mapped tasks which are generated by `expand()`.

In the following example, the task uses both, `.partial()` and `.expand()`, to dynamically generate three task runs.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import task

  @task
  def add(x: int, y: int):
      return x + y

  added_values = add.partial(y=10).expand(x=[1, 2, 3])
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.providers.standard.operators.python import PythonOperator

  def add_function(x: int, y: int):
      return x + y

  added_values = PythonOperator.partial(
      task_id="add",
      python_callable=add_function,
      op_kwargs={"y": 10},
  ).expand(op_args=[[1], [2], [3]])
  ```
</details>

<Frame>
  <img alt="Screenshot of the Airflow UI showing three dynamically mapped task instances created with the code snippets above." />
</Frame>

This `expand` function creates three mapped `add` tasks, one for each entry in the `x` input list. The `partial` function specifies a value for `y` that remains constant in each task.

When you work with mapped tasks, keep the following in mind:

* You can use the results of an upstream task as the input to a mapped task. The upstream task must return a value in a `dict` or `list` form. If you're using traditional operators and not [decorated tasks](/docs/learn/airflow-decorators), the mapping values must be stored in XComs.
* You can map over multiple parameters.
* You can use the results of a mapped task as input to a downstream mapped task.
* You can have a mapped task that results in no task instances. For example, when your upstream task that generates the mapping values returns an empty list. In this case, the mapped task is marked skipped, and downstream tasks are run according to the trigger rules you set. By default, downstream tasks are also skipped.
* Some parameters can't be mapped. For example, `task_id`, `pool`, and many `BaseOperator` arguments.
* `expand()` only accepts keyword arguments.
* The maximum amount of mapped task instances is determined by the `max_map_length` parameter in the [Airflow configuration](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#max-map-length). By default it is set to 1024.
* You can limit the number of mapped task instances for a particular task that run in parallel by setting the following parameters in your dynamically mapped task:
  * Set a limit across all DAG runs with the `max_active_tis_per_dag` parameter.
  * Set a limit for parallel runs within a single DAG with the `max_active_tis_per_dagrun` parameter.
* XComs created by mapped task instances are stored in a list and can be accessed by using the map index of a specific mapped task instance. For example, to access the XComs created by the third mapped task instance (map index of 2) of `my_mapped_task`, use `ti.xcom_pull(task_ids=['my_mapped_task'])[2]`. The `map_indexes` parameter in the `.xcom_pull()` method allows you to specify a list of map indexes of interest (`ti.xcom_pull(task_ids=['my_mapped_task'], map_indexes=[2])`).

For additional examples of how to apply dynamic task mapping functions, see [Dynamic Task Mapping](https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html) in the official Airflow documentation.

The Airflow UI provides observability for mapped tasks in the **Grid View**.

Mapped tasks are identified with a set of brackets `[ ]` followed by the task ID. All mapped task instances are combined into one row on the grid.

The number in the brackets show in the DAG run graph, is updated for each DAG run to reflect how many mapped instances were created. The following screenshot shows a DAG run graph with two tasks, the latter having 49 dynamically mapped task instances.

<Frame>
  <img alt="Screenshot of the Graph tab in the Airflow UI Grid view showing a DAG run graph with two tasks, the latter having been mapped 49 times." />
</Frame>

To see the logs for, and XCom pushed by each dynamically mapped task instance, click the grid square of a dynamically mapped task and then an individual task instance.

## Map over the result of another operator

You can use the output of an upstream operator as the input data for a dynamically mapped downstream task.

In this section you'll learn how to pass mapping information to a downstream task for each of the following scenarios:

* **TaskFlow over TaskFlow**: Both tasks are defined using the TaskFlow API.
* **TaskFlow over traditional operator**: The upstream task is defined using a traditional operator and the downstream task is defined using the TaskFlow API.
* **Traditional operator over TaskFlow**: The upstream task is defined using the TaskFlow API and the downstream task is defined using a traditional operator.
* **Traditional operator over traditional operator**: Both tasks are defined using traditional operators.

<details>
  <summary>Two Flow</summary>

  If both tasks are defined using the TaskFlow API, you can provide a function call to the upstream task as the argument for the `expand()` function.

  ```python wrap theme={null}
  from airflow.sdk import task

  @task
  def one_two_three_TF():
      return [1, 2, 3]

  @task
  def plus_10_TF(x):
      return x + 10

  plus_10_TF.partial().expand(x=one_two_three_TF())
  ```
</details>

<details>
  <summary>Flow Traditional</summary>

  If you are mapping over the results of a traditional operator, you need to provide the argument for `expand()` using the `.output` attribute of the task object.

  ```python wrap theme={null}
  from airflow.sdk import task
  from airflow.providers.standard.operators.python import PythonOperator

  def one_two_three_traditional():
      return [1, 2, 3]

  @task
  def plus_10_TF(x):
      return x + 10

  one_two_three_task = PythonOperator(
      task_id="one_two_three_task", python_callable=one_two_three_traditional
  )

  plus_10_TF.partial().expand(x=one_two_three_task.output)
  ```
</details>

<details>
  <summary>Traditional Flow</summary>

  When mapping a traditional `PythonOperator` over results from an upstream TaskFlow task you need to modify the format of the output to be accepted by the `op_args` argument of the traditional `PythonOperator`.

  ```python wrap theme={null}
  from airflow.sdk import task
  from airflow.providers.standard.operators.python import PythonOperator

  @task
  def one_two_three_TF():
      # this adjustment is due to op_args expecting each argument as a list
      return [[1], [2], [3]]

  def plus_10_traditional(x):
      return x + 10

  plus_10_task = PythonOperator.partial(
      task_id="plus_10_task", python_callable=plus_10_traditional
  ).expand(op_args=one_two_three_TF())
  ```
</details>

<details>
  <summary>Two Traditional</summary>

  When mapping a traditional `PythonOperator` over the result of another `PythonOperator` use the `.output` attribute on the task object and make sure the format returned by the upstream task matches the format expected by the `op_args` parameter.

  ```python wrap theme={null}
  from airflow.providers.standard.operators.python import PythonOperator

  def one_two_three_traditional():
      # this adjustment is due to op_args expecting each argument as a list
      return [[1], [2], [3]]

  def plus_10_traditional(x):
      return x + 10

  one_two_three_task = PythonOperator(
      task_id="one_two_three_task", python_callable=one_two_three_traditional
  )

  plus_10_task = PythonOperator.partial(
      task_id="plus_10_task", python_callable=plus_10_traditional
  ).expand(op_args=one_two_three_task.output)

  # when only using traditional operators, define dependencies explicitly
  one_two_three_task >> plus_10_task
  ```
</details>

### Map over combined results of upstream tasks

You can combine the output lists of upstream tasks using the `.concat()` method. In previous Airflow versions, you needed an intermediate task combining the lists to achieve this.

The following code snippet creates 7 dynamically mapped task instances for the `map_me` task.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import task
  import time

  @task
  def t1():
      return [1, 2, 3]

  t1_obj = t1()

  @task
  def t2():
      return [4, 5, 6, 7]

  t2_obj = t2()

  @task
  def map_me(input):
      print(f"Sleeping for {input} seconds!")
      time.sleep(input)
      print("Waking up!")

  map_me.expand(input=t1_obj.concat(t2_obj))
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.providers.standard.operators.python import PythonOperator
  import time

  def t1_func():
      return [[1], [2], [3]]

  def t2_func():
      return [[4], [5], [6], [7]]

  def map_me_func(input):
      print(f"Sleeping for {input} seconds!")
      time.sleep(input)
      print("Waking up!")

  t1 = PythonOperator(task_id="t1", python_callable=t1_func)

  t2 = PythonOperator(task_id="t2", python_callable=t2_func)

  map_me = PythonOperator.partial(
      task_id="map_me", python_callable=map_me_func
  ).expand(op_args=t1.output.concat(t2.output))
  ```
</details>

## Mapping over multiple parameters

You can use one of the following methods to map over multiple parameters:

* **Cross-product**: Mapping over two or more keyword arguments results in a mapped task instance for each possible combination of inputs. This type of mapping uses the `expand()` function.
* **Sets of keyword arguments**: Mapping over two or more sets of one or more keyword arguments results in a mapped task instance for every defined set, rather than every combination of individual inputs. This type of mapping uses the `expand_kwargs()` function.
* **Zip**: Mapping over a set of positional arguments created with Python's built-in `zip()` function or with the `.zip()` method of an XComArg results in one mapped task for every set of positional arguments. Each set of positional arguments is passed to the same keyword argument of the operator. This type of mapping uses the `expand()` function.

### Cross-product

The default behavior of the `expand()` function is to create a mapped task instance for every possible combination of all provided inputs. For example, if you map over three keyword arguments and provide two options to the first, four options to the second, and five options to the third, you would create 2x4x5=40 mapped task instances. One common use case for this method is tuning model hyperparameters.

The following task definition maps over three options for  the `bash_command` parameter and three options for the `env` parameter. This will result in 3x3=9 mapped task instances. Each bash command runs with each definition for the environment variable `WORD`.

```python wrap theme={null}
from airflow.providers.standard.operators.bash import BashOperator

cross_product_example = BashOperator.partial(
    task_id="cross_product_example"
).expand(
    bash_command=[
        "echo $WORD", # prints the env variable WORD
        "echo `expr length $WORD`", # prints the number of letters in WORD
        "echo \\${WORD//e/X}" # replaces each "e" in WORD with "X"
    ],
    env=[
        {"WORD": "hello"},
        {"WORD": "tea"},
        {"WORD": "goodbye"}
    ]
)
```

The nine mapped task instances of the task `cross_product_example` run all possible combinations of the bash command with the `env` variable:

* Map index 0: `hello`
* Map index 1: `tea`
* Map index 2: `goodbye`
* Map index 3: `5`
* Map index 4: `3`
* Map index 5: `7`
* Map index 6: `hXllo`
* Map index 7: `tXa`
* Map index 8: `goodbyX`

### Sets of keyword arguments

To map over sets of inputs to two or more keyword arguments (kwargs), you can use the `expand_kwargs()` function. You can provide sets of parameters as a list containing a dictionary or as an `XComArg`. The operator gets 3 sets of commands, resulting in 3 mapped task instances.

```python wrap theme={null}
from airflow.providers.standard.operators.bash import BashOperator

# input sets of kwargs provided directly as a list[dict]
t1 = BashOperator.partial(task_id="t1").expand_kwargs(
    [
        {"bash_command": "echo $WORD", "env" : {"WORD": "hello"}},
        {"bash_command": "echo `expr length $WORD`", "env" : {"WORD": "tea"}},
        {"bash_command": "echo \\${WORD//e/X}", "env" : {"WORD": "goodbye"}}
    ]
)
```

The task `t1` will have three mapped task instances printing their results into the logs:

* Map index 0: `hello`
* Map index 1: `3`
* Map index 2: `goodbyX`

### Zip

In dynamic task mapping, you can provide sets of positional arguments to the same keyword argument. For example, the `op_args` argument of the `PythonOperator`. You can use the built-in [`zip()`](https://docs.python.org/3/library/functions.html#zip) Python function if your inputs are in the form of iterables such as tuples, dictionaries, or lists. If your inputs come from XCom objects, you can use the `.zip()` method of the `XComArg` object.

#### Provide positional arguments with the built-in Python `zip()`

The `zip()` function takes in an arbitrary number of iterables and uses their elements to create a zip-object containing tuples. There will be as many tuples as there are elements in the shortest iterable. Each tuple contains one element from every iterable provided. For example:

* `zip(["a", "b", "c"], [1, 2, 3], ["hi", "bye", "tea"])` results in a zip object containing: `("a", 1, "hi"), ("b", 2, "bye"), ("c", 3, "tea")`.
* `zip(["a", "b"], [1], ["hi", "bye"], [19, 23], ["x", "y", "z"])` results in a zip object containing only one tuple: `("a", 1, "hi", 19, "x")`. This is because the shortest list provided only contains one element.
* It is also possible to zip together different types of iterables. `zip(["a", "b"], {"hi", "bye"}, (19, 23))` results in a zip object containing: `('a', 'hi', 19), ('b', 'bye', 23)`.

The following code snippet shows how a list of zipped arguments can be provided to the `expand()` function in order to create mapped tasks over sets of positional arguments. In the TaskFlow API version of the DAG, each set of positional arguments is passed to the argument `zipped_x_y_z`. In the DAG using a traditional `PythonOperator` each set of positional arguments is unpacked due to `op_args` expecting an iterable and passed to the arguments `x`, `y` and `z`.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import task

  # use the zip function to create three-tuples out of three lists
  zipped_arguments = list(zip([1, 2, 3], [10, 20, 30], [100, 200, 300]))
  # zipped_arguments contains: [(1,10,100), (2,20,200), (3,30,300)]

  # creating the mapped task instances using the TaskFlow API
  @task
  def add_numbers(zipped_x_y_z):
      return zipped_x_y_z[0] + zipped_x_y_z[1] + zipped_x_y_z[2]

  add_numbers.expand(zipped_x_y_z=zipped_arguments)
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.providers.standard.operators.python import PythonOperator

  # use the zip function to create three-tuples out of three lists
  zipped_arguments = list(zip([1, 2, 3], [10, 20, 30], [100, 200, 300]))
  # zipped_arguments contains: [(1,10,100), (2,20,200), (3,30,300)]

  # function for the PythonOperator
  def add_numbers_function(x, y, z):
      return x + y + z

  # dynamically mapped PythonOperator
  add_numbers = PythonOperator.partial(
      task_id="add_numbers",
      python_callable=add_numbers_function,
  ).expand(op_args=zipped_arguments)
  ```
</details>

The task `add_numbers` will have three mapped task instances one for each tuple of positional arguments:

* Map index 0: `111`
* Map index 1: `222`
* Map index 2: `333`

#### Provide positional arguments with XComArg.zip()

It is also possible to zip `XComArg` objects. If the upstream task has been defined using the TaskFlow API, provide the function call. If the upstream task uses a traditional operator, provide `task_object.output` or `XcomArg(task_object)`. In the following example, you can see the results of three tasks being zipped together to form the `zipped_arguments` (`[(1, 10, 100), (2, 1000, 200), (1000, 1000, 300)]`).

To mimic the behavior of the [`zip_longest()`](https://docs.python.org/3/library/itertools.html#itertools.zip_longest) function, you can add the optional `fillvalue` keyword argument to the `.zip()` method. If you specify a default value with `fillvalue`, the method produces as many tuples as the longest input has elements and fills in missing elements with the default value. If `fillvalue` wasn't specified in the example below, `zipped_arguments` would only contain one tuple `[(1, 10, 100)]` since the shortest list provided to the `.zip()` method is only one element long.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import task

  @task
  def one_two_three():
      return [1, 2]

  @task
  def ten_twenty_thirty():
      return [10]

  @task
  def one_two_three_hundred():
      return [100, 200, 300]

  zipped_arguments = one_two_three().zip(
      ten_twenty_thirty(), one_two_three_hundred(), fillvalue=1000
  )
  # zipped_arguments contains [(1, 10, 100), (2, 1000, 200), (1000, 1000, 300)]

  # creating the mapped task instances using the TaskFlow API
  @task
  def add_nums(zipped_x_y_z):
      return zipped_x_y_z[0] + zipped_x_y_z[1] + zipped_x_y_z[2]

  add_nums.expand(zipped_x_y_z=zipped_arguments)
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  from airflow.providers.standard.operators.python import PythonOperator

  def one_two_three_function():
      return [1, 2]

  def ten_twenty_thirty_function():
      return [10]

  def one_two_three_hundred_function():
      return [100, 200, 300]

  one_two_three = PythonOperator(
      task_id="one_two_three", python_callable=one_two_three_function
  )

  ten_twenty_thirty = PythonOperator(
      task_id="ten_twenty_thirty", python_callable=ten_twenty_thirty_function
  )

  one_two_three_hundred = PythonOperator(
      task_id="one_two_three_hundred", python_callable=one_two_three_hundred_function
  )

  zipped_arguments = one_two_three.output.zip(
      ten_twenty_thirty.output, one_two_three_hundred.output, fillvalue=1000
  )
  # zipped_arguments contains [(1, 10, 100), (2, 1000, 200), (1000, 1000, 300)]

  # function that will be used in the dynamically mapped PythonOperator
  def add_nums_function(x, y, z):
      return x + y + z

  add_nums = PythonOperator.partial(
      task_id="add_nums", python_callable=add_nums_function
  ).expand(op_args=zipped_arguments)
  ```
</details>

The `add_nums` task will have three mapped instances with the following results:

* Map index 0: `111` (1+10+100)
* Map index 1: `1202` (2+1000+200)
* Map index 2: `2300` (1000+1000+300)

## Repeated mapping

You can dynamically map an Airflow task over the output of another dynamically mapped task. This results in one mapped task instance for every mapped task instance of the upstream task.

The following example shows three dynamically mapped tasks.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import task

  @task
  def multiply_by_2(num):
      return num * 2

  @task
  def add_10(num):
      return num + 10

  @task
  def multiply_by_100(num):
      return num * 100

  multiplied_value_1 = multiply_by_2.expand(num=[1, 2, 3])
  summed_value = add_10.expand(num=multiplied_value_1)
  multiply_by_100.expand(num=summed_value)
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.providers.standard.operators.python import PythonOperator

  def multiply_by_2_func(num):
      return [num * 2]

  def add_10_func(num):
      return [num + 10]

  def multiply_by_100_func(num):
      return num * 100

  multiply_by_2 = PythonOperator.partial(
      task_id="multiply_by_2",
      python_callable=multiply_by_2_func
  ).expand(op_args=[[1], [2], [3]])

  add_10 = PythonOperator.partial(
      task_id="add_10",
      python_callable=add_10_func
  ).expand(op_args=multiply_by_2.output)

  multiply_by_100 = PythonOperator.partial(
      task_id="multiply_by_100",
      python_callable=multiply_by_100_func
  ).expand(op_args=add_10.output)

  multiply_by_2 >> add_10 >> multiply_by_100
  ```
</details>

In the example above, the `multiply_by_2` task is dynamically mapped over a list of three elements (`[1, 2, 3]`). The task has three mapped task instances containing the following values:

* Map index 0: `2` (1\*2)
* Map index 1: `4` (2\*2)
* Map index 2: `6` (3\*2)

The `add_10` task is dynamically mapped over the output of the `multiply_by_2` task. It has 3 mapped task instances (one for each mapped instance of the previous task) which contain the following values:

* Map index 0: `12` (2+10)
* Map index 1: `14` (4+10)
* Map index 2: `16` (6+10)

The `multiply_by_100` task is dynamically mapped over the output of the `add_10` task, which results in three mapped task instances with the following outputs:

* Map index 0: `1200` (12\*100)
* Map index 1: `1400` (14\*100)
* Map index 2: `1600` (16\*100)

You can chain an arbitrary number of dynamically mapped tasks in this manner. It is currently not possible to exponentially increase the number of mapped task instances.

## Map over task groups

[Task groups](/docs/learn/task-groups) defined with the `@task_group` decorator can be dynamically mapped as well. The syntax for dynamically mapping over a task group is the same as dynamically mapping over a single task.

```python wrap theme={null}
from airflow.sdk import task, task_group

# creating a task group using the decorator with the dynamic input my_num
@task_group(group_id="group1")
def tg1(my_num):
    @task
    def print_num(num):
        return num

    @task
    def add_42(num):
        return num + 42

    print_num(my_num) >> add_42(my_num)

# creating 6 mapped task group instances of the task group group1
tg1_object = tg1.expand(my_num=[19, 23, 42, 8, 7, 108])
```

You can also dynamically map over multiple task group input parameters as you would for regular tasks using a cross-product, `zip` function, or sets of keyword arguments. For more on this, see [Mapping over multiple parameters](#mapping-over-multiple-parameters).

## Transform outputs with .map

There are use cases where you want to transform the output of an upstream task before another task dynamically maps over it. For example, if the upstream traditional operator returns its output in a fixed format or if you want to skip certain mapped task instances based on a logical condition.

The `.map()` method accepts a Python function and uses it to transform an iterable input before a task dynamically maps over it.

You can call `.map()` directly on a task using the TaskFlow API (`my_upstream_task_flow_task().map(mapping_function)`) or on the output object of a traditional operator (`my_upstream_traditional_operator.output.map(mapping_function)`).

The downstream task is dynamically mapped over the object created by the `.map()` method using either `.expand()` for a single keyword argument or `.expand_kwargs()` for list of dictionaries containing sets of keyword arguments.

The code snippet below shows how to use `.map()` to skip specific mapped tasks based on a logical condition.

* `list_strings` is the upstream task returning a list of strings.
  -`skip_strings_starting_with_skip` transforms a list of strings into a list of modified strings and `AirflowSkipExceptions`. In this DAG, the function transforms `list_strings` into a new list called `transformed_list`. This function won't appear as an Airflow task.
* `mapped_printing_task` dynamically maps over the `transformed_list` object.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import task
  from airflow.sdk.exceptions import AirflowSkipException

  # an upstream task returns a list of outputs in a fixed format
  @task
  def list_strings():
      return ["skip_hello", "hi", "skip_hallo", "hola", "hey"]

  # the function used to transform the upstream output before
  # a downstream task is dynamically mapped over it
  def skip_strings_starting_with_skip(string):
      if len(string) < 4:
          return string + "!"
      elif string[:4] == "skip":
          raise AirflowSkipException(f"Skipping {string}; as I was told!")
      else:
          return string + "!"

  # transforming the output of the first task with the map function.
  transformed_list = list_strings().map(skip_strings_starting_with_skip)

  # the task using dynamic task mapping on the transformed list of strings
  @task
  def mapped_printing_task(string):
      return "Say " + string

  mapped_printing_task.partial().expand(string=transformed_list)
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  from airflow.providers.standard.operators.python import PythonOperator
  from airflow.sdk.exceptions import AirflowSkipException

  # an upstream task returns a list of outputs in a fixed format
  def list_strings():
      return ["skip_hello", "hi", "skip_hallo", "hola", "hey"]

  listed_strings = PythonOperator(
      task_id="list_strings",
      python_callable=list_strings,
  )

  # the function used to transform the upstream output before
  # a downstream task is dynamically mapped over it
  def skip_strings_starting_with_skip(string):
      if len(string) < 4:
          return [string + "!"]
      elif string[:4] == "skip":
          raise AirflowSkipException(f"Skipping {string}; as I was told!")
      else:
          return [string + "!"]

  # transforming the output of the first task with the map function.
  # since `op_args` expects a list of lists it is important
  # each element of the list is wrapped in a list in the map function.
  transformed_list = listed_strings.output.map(skip_strings_starting_with_skip)

  # function to use in the dynamically mapped PythonOperator
  def mapped_printing_function(string):
      return "Say " + string

  mapped_printing = PythonOperator.partial(
      task_id="mapped_printing",
      python_callable=mapped_printing_function,
  ).expand(op_args=transformed_list)
  ```
</details>

In the **\[] Mapped Tasks** tab, you can see how the mapped task instances 0 and 2 have been skipped.

<Frame>
  <img alt="Skipped Mapped Tasks" />
</Frame>

## Example implementation

For this example, you'll implement one of the most common use cases for dynamic tasks: processing files in Amazon S3. In this scenario, you'll use an ELT framework to extract data from files in Amazon S3, load the data into Snowflake, and transform the data using Snowflake's built-in compute. It's assumed that the files will be dropped daily, but it's unknown how many will arrive each day. You'll use dynamic task mapping to create a unique task for each file at runtime. This gives you the benefit of atomicity, better observability, and easier recovery from failures.

All code used in this example is located in the [dynamic-task-mapping-tutorial repository](https://github.com/astronomer/dynamic-task-mapping-tutorial).

The example DAG completes the following steps:

* Use a decorated Python operator to get the current list of files from Amazon S3. The Amazon S3 prefix passed to this function is parameterized with `ds_nodash` so it pulls files only for the execution date of the DAG run. For example, for a DAG run on April 12th 2025, you assume the files landed in a folder named `20250412/`.
* Use the results of the first task, map an `S3ToSnowflakeOperator` for each file.
* Move the daily folder of processed files into a `processed/` folder while,
* Simultaneously runs a Snowflake query that transforms the data. The query is located in a separate SQL file in our `include/` directory.
* Deletes the folder of daily files now that it has been moved to `processed/` for record keeping.

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  from airflow.decorators import dag, task
  from airflow.providers.snowflake.transfers.copy_into_snowflake import (
      CopyFromExternalStageToSnowflakeOperator,
  )
  from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
  from airflow.providers.amazon.aws.hooks.s3 import S3Hook
  from airflow.providers.amazon.aws.operators.s3 import S3CopyObjectOperator
  from airflow.providers.amazon.aws.operators.s3 import S3DeleteObjectsOperator

  from pendulum import datetime


  @dag(
      start_date=datetime(2024, 4, 2),
      catchup=False,
      template_searchpath="/usr/local/airflow/include",
      schedule="@daily",
  )
  def mapping_elt():

      @task
      def get_s3_files(current_prefix):
          s3_hook = S3Hook(aws_conn_id="s3")

          current_files = s3_hook.list_keys(
              bucket_name="my-bucket",
              prefix=current_prefix + "/",
              start_after_key=current_prefix + "/",
          )

          return [[file] for file in current_files]

      copy_to_snowflake = CopyFromExternalStageToSnowflakeOperator.partial(
          task_id="load_files_to_snowflake",
          stage="MY_STAGE",
          table="COMBINED_HOMES",
          schema="MYSCHEMA",
          file_format="(type = 'CSV',field_delimiter = ',', skip_header=1)",
          snowflake_conn_id="snowflake",
      ).expand(files=get_s3_files(current_prefix="{{ ds_nodash }}"))

      move_s3 = S3CopyObjectOperator(
          task_id="move_files_to_processed",
          aws_conn_id="s3",
          source_bucket_name="my-bucket",
          source_bucket_key="{{ ds_nodash }}" + "/",
          dest_bucket_name="my-bucket",
          dest_bucket_key="processed/" + "{{ ds_nodash }}" + "/",
      )

      delete_landing_files = S3DeleteObjectsOperator(
          task_id="delete_landing_files",
          aws_conn_id="s3",
          bucket="my-bucket",
          prefix="{{ ds_nodash }}" + "/",
      )

      transform_in_snowflake = SnowflakeOperator(
          task_id="run_transformation_query",
          sql="/transformation_query.sql",
          snowflake_conn_id="snowflake",
      )

      copy_to_snowflake >> [move_s3, transform_in_snowflake]
      move_s3 >> delete_landing_files


  mapping_elt()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  from airflow import DAG
  from airflow.providers.snowflake.transfers.copy_into_snowflake import (
      CopyFromExternalStageToSnowflakeOperator,
  )
  from airflow.operators.python import PythonOperator
  from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
  from airflow.providers.amazon.aws.hooks.s3 import S3Hook
  from airflow.providers.amazon.aws.operators.s3 import S3CopyObjectOperator
  from airflow.providers.amazon.aws.operators.s3 import S3DeleteObjectsOperator

  from pendulum import datetime


  def get_s3_files(current_prefix):
      s3_hook = S3Hook(aws_conn_id="s3")

      current_files = s3_hook.list_keys(
          bucket_name="my-bucket",
          prefix=current_prefix + "/",
          start_after_key=current_prefix + "/",
      )

      return [[file] for file in current_files]


  with DAG(
      "mapping_elt_traditional",
      start_date=datetime(2024, 4, 2),
      catchup=False,
      template_searchpath="/usr/local/airflow/include",
      schedule="@daily",
  ):
      get_s3_files_task = PythonOperator(
          task_id="get_s3_files",
          python_callable=get_s3_files,
          op_kwargs={"current_prefix": "{{ ds_nodash }}"},
      )

      copy_to_snowflake = CopyFromExternalStageToSnowflakeOperator.partial(
          task_id="load_files_to_snowflake",
          stage="MY_STAGE",
          table="COMBINED_HOMES",
          schema="MYSCHEMA",
          file_format="(type = 'CSV',field_delimiter = ',', skip_header=1)",
          snowflake_conn_id="snowflake",
      ).expand(files=get_s3_files_task.output)

      move_s3 = S3CopyObjectOperator(
          task_id="move_files_to_processed",
          aws_conn_id="s3",
          source_bucket_name="my-bucket",
          source_bucket_key="{{ ds_nodash }}" + "/",
          dest_bucket_name="my-bucket",
          dest_bucket_key="processed/" + "{{ ds_nodash }}" + "/",
      )

      delete_landing_files = S3DeleteObjectsOperator(
          task_id="delete_landing_files",
          aws_conn_id="s3",
          bucket="my-bucket",
          prefix="{{ ds_nodash }}" + "/",
      )

      transform_in_snowflake = SnowflakeOperator(
          task_id="run_transformation_query",
          sql="/transformation_query.sql",
          snowflake_conn_id="snowflake",
      )

      copy_to_snowflake >> [move_s3, transform_in_snowflake]
      move_s3 >> delete_landing_files
  ```
</details>

The graph of this DAG looks similar to this image:

<Frame>
  <img alt="ELT Graph" />
</Frame>

When dynamically mapping tasks, make note of the format needed for the parameter you are mapping. In the previous example, you wrote your own Python function to get the Amazon S3 keys because the `S3toSnowflakeOperator` requires each `s3_key` parameter to be in a list format, and the `s3_hook.list_keys` function returns a single list with all keys. By writing your own simple function, you can turn the hook results into a list of lists that can be used by the downstream operator.


# Dynamically generate DAGs in Airflow
Source: https://astronomer.io/docs/learn/dynamically-generating-dags

Get to know the best ways to dynamically generate DAGs in Apache Airflow. Use examples to generate DAGs using single- and multiple-file methods.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

In Airflow, [DAGs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html) are defined as Python code. Airflow executes all Python code in the `dags_folder` and loads any `DAG` objects that appear in `globals()`. The simplest way to create a DAG is to write it as a static Python file.

Sometimes, manually writing DAGs isn't practical. Maybe you have hundreds or thousands of DAGs that do similar things with just a parameter changing between them. Or maybe you need a set of DAGs to load tables, but don't want to manually update DAGs every time the tables change. In these cases, and others, it makes more sense to dynamically generate DAGs.

Because everything in Airflow is code, you can dynamically generate DAGs using Python alone. As long as a `DAG` object in `globals()` is created by Python code that is stored in the `dags_folder`, Airflow will load it. In this guide, you'll learn how to dynamically generate DAGs. You'll learn when DAG generation is the preferred option and what pitfalls to avoid.

All code used in this guide is located in the [Astronomer Registry](https://github.com/astronomer/dynamic-dags-tutorial).

<Tip>
  You can use [dynamic task mapping](/docs/learn/dynamic-tasks) to write DAGs that dynamically generate parallel tasks at runtime. Dynamic task mapping is a first-class Airflow feature, and suitable for many dynamic use cases. Due to its higher degree of support and stability, Astronomer recommends exploring dynamic task mapping for your use case before implementing the dynamic DAG generation methods described in this guide.
</Tip>

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Astronomer Academy: [Airflow: Dynamic DAGs](https://academy.astronomer.io/astro-runtime-dynamic-dags).
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow DAGs. See [Introduction to Airflow DAGs](/docs/learn/dags).

## Single-file methods

One method for dynamically generating DAGs is to have a single Python file which generates DAGs based on some input parameter(s). For example, a list of APIs or tables. A common use case for this is an ETL or ELT-type pipeline where there are many data sources or destinations. This requires creating many DAGs that all follow a similar pattern.

Some benefits of the single-file method:

* It's straightforward to implement.
* It can accommodate input parameters from many different sources.
* Adding DAGs is nearly instantaneous since it requires only changing the input parameters.

The single-file method has the following disadvantages:

* Your visibility into the code behind any specific DAG is limited because a DAG file isn't created.
* Generation code is executed every time the DAG is parsed because this method requires a Python file in the `dags_folder`. How frequently this occurs is controlled by the [`min_file_process_interval`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#min-file-process-interval) parameter. This can cause performance issues if the total number of DAGs is large, or if the code is connecting to an external system such as a database.

In the following examples, the single-file method is implemented differently based on which input parameters are used for generating DAGs.

### Example: Use a `create_dag` function

To dynamically create DAGs from a file, you need to define a Python function that will generate the DAGs based on an input parameter. In this case, you're going to define a DAG template within a `create_dag` function. The code here is very similar to what you would use when creating a single DAG, but it is wrapped in a function that allows for custom parameters to be passed in.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.decorators import dag, task

  def create_dag(dag_id, schedule, dag_number, default_args):
      @dag(dag_id=dag_id, schedule=schedule, default_args=default_args, catchup=False)
      def hello_world_dag():
          @task()
          def hello_world():
              print("Hello World")
              print("This is DAG: {}".format(str(dag_number)))

          hello_world()

      generated_dag = hello_world_dag()

      return generated_dag
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.models.dag import DAG
  from airflow.operators.python import PythonOperator

  def create_dag(dag_id, schedule, dag_number, default_args):
      def hello_world_py(*args):
          print("Hello World")
          print("This is DAG: {}".format(str(dag_number)))

      generated_dag = DAG(dag_id, schedule=schedule, default_args=default_args, catchup=False)

      with generated_dag:
          t1 = PythonOperator(
              task_id="hello_world", python_callable=hello_world_py
          )

      return generated_dag
  ```
</details>

In this example, the input parameters can come from any source that the Python script can access. You can then set a simple loop (`range(1, 4)`) to generate these unique parameters and pass them to the global scope, thereby registering them as valid DAGs with the Airflow scheduler:

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.decorators import dag, task
  from pendulum import datetime


  def create_dag(dag_id, schedule, dag_number, default_args):
      @dag(dag_id=dag_id, schedule=schedule, default_args=default_args, catchup=False)
      def hello_world_dag():
          @task()
          def hello_world(*args):
              print("Hello World")
              print("This is DAG: {}".format(str(dag_number)))

          hello_world()

      generated_dag = hello_world_dag()

      return generated_dag


  # build a dag for each number in range(1, 4)
  for n in range(1, 4):
      dag_id = "loop_hello_world_{}".format(str(n))

      default_args = {"owner": "airflow", "start_date": datetime(2023, 7, 1)}

      schedule = "@daily"

      dag_number = n

      globals()[dag_id] = create_dag(dag_id, schedule, dag_number, default_args)
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from pendulum import datetime
  from airflow import DAG
  from airflow.operators.python import PythonOperator


  def create_dag(dag_id, schedule, dag_number, default_args):
      def hello_world_py():
          print("Hello World")
          print("This is DAG: {}".format(str(dag_number)))

      generated_dag = DAG(dag_id, schedule=schedule, default_args=default_args)

      with generated_dag:
          PythonOperator(task_id="hello_world", python_callable=hello_world_py)

      return generated_dag


  # build a dag for each number in range(1, 4)
  for n in range(1, 4):
      dag_id = "loop_hello_world_{}".format(str(n))

      default_args = {"owner": "airflow", "start_date": datetime(2023, 7, 1)}

      schedule = "@daily"
      dag_number = n

      globals()[dag_id] = create_dag(dag_id, schedule, dag_number, default_args)
  ```
</details>

The DAGs appear in the Airflow UI:

<Frame>
  <img alt="DAGs from Loop" />
</Frame>

### Example: Generate DAGs from environment variables

As mentioned previously, the input parameters don't have to exist in the DAG file. Another common form of generating DAGs is by setting values using environment variables.

You can define environment variables locally in the `.env` file of your Astro project or in the [Astro UI](/docs/astro/manage-env-vars) for your Astro deployments.

```text wrap theme={null}
DYNAMIC_DAG_NUMBER=10
```

You can retrieve this value by fetching the environment variable and passing it into your `range`. The `default` is set to 3 because you want the interpreter to register this file as valid regardless of whether the variable exists.

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  from airflow.decorators import dag, task
  from pendulum import datetime
  import os


  def create_dag(dag_id, schedule, dag_number, default_args):
      @dag(dag_id=dag_id, schedule=schedule, default_args=default_args, catchup=False)
      def hello_world_dag():
          @task()
          def hello_world(*args):
              print("Hello World")
              print("This is DAG: {}".format(str(dag_number)))

          hello_world()

      generated_dag = hello_world_dag()

      return generated_dag


  number_of_dags = os.getenv("DYNAMIC_DAG_NUMBER", default=3)
  number_of_dags = int(number_of_dags)

  for n in range(1, number_of_dags):
      dag_id = "variable_hello_world_{}".format(str(n))

      default_args = {"owner": "airflow", "start_date": datetime(2023, 7, 1)}

      schedule = "@daily"
      dag_number = n

      globals()[dag_id] = create_dag(dag_id, schedule, dag_number, default_args)
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow import DAG
  from airflow.operators.python import PythonOperator
  from pendulum import datetime
  import os


  def create_dag(dag_id, schedule, dag_number, default_args):
      def hello_world_py():
          print("Hello World")
          print("This is DAG: {}".format(str(dag_number)))

      generated_dag = DAG(dag_id, schedule=schedule, default_args=default_args)

      with generated_dag:
          PythonOperator(task_id="hello_world", python_callable=hello_world_py)

      return generated_dag


  number_of_dags = os.getenv("DYNAMIC_DAG_NUMBER", default=3)
  number_of_dags = int(number_of_dags)

  for n in range(1, number_of_dags):
      dag_id = "hello_world_{}".format(str(n))

      default_args = {"owner": "airflow", "start_date": datetime(2023, 7, 1)}

      schedule = "@daily"
      dag_number = n
      globals()[dag_id] = create_dag(dag_id, schedule, dag_number, default_args)
  ```
</details>

The DAGs appear in the Airflow UI:

<Frame>
  <img alt="DAGs from Variables in the Airflow UI" />
</Frame>

## Multiple-file methods

Another method for dynamically generating DAGs is to use code to generate full Python files for each DAG. The end result of this method is having one Python file per generated DAG in your `dags_folder`.

One way of implementing this method in production is to have a Python script that generates DAG files when executed as part of a CI/CD workflow. The DAGs are generated during the CI/CD build and then deployed to Airflow. You could also have another DAG that runs the generation script periodically.

Some benefits of this method:

* It's more scalable than single-file methods. Because the DAG files aren't being generated by parsing code in the `dags_folder`, the DAG generation code isn't executed on every scheduler heartbeat.
* Since DAG files are being explicitly created before deploying to Airflow, you have full visibility into the DAG code, including from the **Code** button in the Airflow UI.

Some disadvantages of this method:

* It can be complex to set up.
* Changes to DAGs or additional DAGs won't be generated until the script is run, which in some cases requires a deployment.

### Example: Generate DAGs from JSON config files

One way of implementing the multiple-file method is by using a Python script to generate DAG files based on a set of JSON configuration files. For this example, you'll assume that all DAGs have a single task that uses the [`BashOperator`](https://airflow.apache.org/registry/providers/standard#standard-bash-BashOperator) to run a Bash command. This use case might be relevant for a team of analysts who need to schedule Bash commands, where the DAG is largely the same, but the command, an environment variable and the schedule change.

To start, you'll create a DAG 'template' file that defines the DAG's structure. This looks just like a regular DAG file, but specific variables have been added where information is going to be dynamically generated, namely `dag_id_to_replace`, `schedule_to_replace`, `bash_command_to_replace` and `env_var_to_replace`.

```python wrap theme={null}
from airflow.decorators import dag
from airflow.operators.bash import BashOperator
from pendulum import datetime

@dag(
    dag_id=dag_id_to_replace,
    start_date=datetime(2023, 7, 1),
    schedule=schedule_to_replace,
    catchup=False,
)
def dag_from_config():
    BashOperator(
        task_id="say_hello",
        bash_command=bash_command_to_replace,
        env={"ENVVAR": env_var_to_replace},
    )

dag_from_config()

```

Next, you create a `dag-config` folder that will contain a JSON config file for each DAG. The config file should define the parameters discussed previously, the DAG ID, schedule, Bash command and environment variable to be used.

```json wrap theme={null}
{
    "dag_id": "dag_file_1",
    "schedule": "'@daily'",
    "bash_command": "'echo $ENVVAR'",
    "env_var": "'Hello! :)'"
}
```

Finally, you create a Python script that will create the DAG files based on the template and the config files. The script loops through every config file in the `dag-config` folder, makes a copy of the template in the `include` folder, and then overwrites the parameters in that file with the ones from the config file.

```python wrap theme={null}
import json
import os
import shutil
import fileinput

config_filepath = "include/dag-config/"
dag_template_filename = "include/dag-template.py"

for filename in os.listdir(config_filepath):
    f = open(config_filepath + filename)
    config = json.load(f)

    new_filename = "dags/" + config["dag_id"] + ".py"
    shutil.copyfile(dag_template_filename, new_filename)

    for line in fileinput.input(new_filename, inplace=True):
        line = line.replace("dag_id_to_replace", "'" + config["dag_id"] + "'")
        line = line.replace("schedule_to_replace", config["schedule"])
        line = line.replace("bash_command_to_replace", config["bash_command"])
        line = line.replace("env_var_to_replace", config["env_var"])
        print(line, end="")
```

To generate your DAG files, you can either run this script on demand or as part of your CI/CD workflow. After running the script, your project structure will be similar to the example below, where the `include` directory contains the files from which the DAGs are generated, and the `dags` directory contains the two dynamically generated DAGs:

```text wrap theme={null}
.
├── dags
│   ├── dag_file_1.py
│   └── dag_file_2.py
└── include
    ├── dag-config
    │   ├── dag1-config.json
    │   └── dag2-config.json
    ├── dag-template.py
    └── generate-dag-files.py
```

This is a straightforward example that works only if all of the DAGs follow the same pattern. However, it could be expanded to have dynamic inputs for tasks, dependencies, different operators, and so on.

## Tools for dynamically creating DAGs

### gusty

A popular tool for dynamically creating DAGs is [gusty](https://github.com/chriscardillo/gusty). gusty is an open source Python library for dynamically generating Airflow DAGs. Tasks can be created from YAML, Python, SQL, R Markdown, and Jupyter Notebooks.

You can install gusty in your Airflow environment by running `pip install gusty` from your command line. If you use the Astro CLI, you can alternatively add `gusty` to your Astro project `requirements.txt` file.

To use gusty, create a new directory in your `dags` folder that will contain all gusty DAGs. Subdirectories of this folder will define DAGs, while nested subdirectories will define task groups within their respective DAGs.

The following file structure will lead to the creation of 2 DAGs from the contents of the `my_gusty_dags` directory. `my_dag_1` contains two tasks each defined in their own YAML file. `my_dag_2` contains one task, `task_0`, defined from a YAML file, as well as the two task groups `my_task_group_1` and `my_task_group_2`, containing two tasks each. The latter task group contains two tasks defined from SQL files.

```text wrap theme={null}
.
└── dags
    ├── my_gusty_dags
    │   ├── my_dag_1
    │   │   ├── METADATA.yml
    │   │   ├── task_1.yaml
    │   │   └── task_2.yaml
    │   └── my_dag_2
    │       ├── METADATA.yml
    │       ├── task_0.yaml
    │       ├── my_taskgroup_1
    │       │   ├── task_1.yaml
    │       │   └── task_2.yaml
    │       └── my_taskgroup_2
    │           ├── task_3.sql
    │           └── task_4.sql
    ├── creating_gusty_dags.py
    └── my_regular_dag.py
```

To create DAGs from the `my_gusty_dags` directory, you need a Python script that calls gusty's `create_dags` function. In this example, a script called `creating_gusty_dags.py` in the project's `dags` directory contains the following code.

```python wrap theme={null}
from gusty import create_dags

dag = create_dags(
    # provide the path to your gusty DAGs directory
    '/usr/local/airflow/dags/my_gusty_dags',
    # provide the namespace for gusty to use
    globals(),
    # By default, gusty places a LatestOnlyOperator at the root of the DAG.
    # We can disable this behavior by setting latest_only=False
    latest_only=False
)
```

DAG-level parameters can be defined in the `METADATA.yml` file:

```yaml wrap theme={null}
description: "An example of a DAG created using gusty!"
schedule_interval: "1 0 * * *"
default_args:
    owner: airflow
    depends_on_past: False
    start_date: !days_ago 1
    email: airflow@example.com
    email_on_failure: False
    email_on_retry: False
    retries: 1
    retry_delay: !timedelta 'minutes: 5'
```

Tasks can be defined in YAML for any standard and custom Airflow operator. The example below shows how to use gusty to define a `BashOperator` task in YAML. The `dependencies` parameter was set to make this task dependent on `task_1` having completed successfully.

```yaml wrap theme={null}
operator: airflow.operators.bash.BashOperator
bash_command: echo $MY_ENV_VAR
dependencies:
  - task_1
env:
    MY_ENV_VAR: "Hello!"
```

Note that to use gusty-generated DAGs and standard DAGs in the same Airflow environment, ensure that your standard DAGs are in your `dags` directory outside of the `my_gusty_dags` folder.

Learn more about gusty features in the [repository README](https://github.com/chriscardillo/gusty/blob/main/README). Additionally, you can explore two fully functional gusty environments: The [gusty-demo](https://github.com/chriscardillo/gusty-demo) and the [gusty-demo-lite](https://github.com/chriscardillo/gusty-demo-lite).

### DAG Factory

Another open source tool for dynamic DAG generation is [DAG Factory](/docs/learn/dag-factory). The [dag-factory package](https://github.com/astronomer/dag-factory) allows users to create DAGs from YAML files which contain both DAG and task-level parameters, removing the necessity to know about Airflow specific syntax.

To learn more about DAG Factory, including a full walkthrough of features like assets and configuration inheritance, and how to generate YAML files based on a template, see the [DAG Factory tutorial](/docs/learn/dag-factory).

## Scalability

Dynamically generating DAGs can cause performance issues when used at scale. Whether or not any particular method will cause problems is dependent on your total number of DAGs, your Airflow configuration, and your infrastructure. Keep the following considerations in mind when considering dynamically generating DAGs:

* Any code in the `dags_folder` is executed either every `min_file_processing_interval` or as fast as the DAG file processor can, whichever is less frequent. Methods where the code is dynamically generating DAGs, such as the single-file method, are more likely to cause performance issues at scale.
* If you are reaching out to a database to create your DAGs, you will be querying frequently. Be conscious of your database's ability to handle such frequent connections and any costs you may incur for each request from your data provider.
* To help with potential performance issues, you can increase the `min_file_processing_interval` to a higher value. Consider this option if you know that your DAGs aren't changing frequently and if you can tolerate some delay in the dynamic DAGs changing in response to the external source that generates them.

[Fine-tuning your scheduler](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html#fine-tuning-your-scheduler-performance) helps resolve potential performance issues. There is no single right way to implement or scale dynamically generated DAGs, but the flexibility of Airflow means there are many ways to arrive at a solution that works for your organization.


# Manage Apache Airflow® Dag notifications
Source: https://astronomer.io/docs/learn/error-notifications-in-airflow

Master the basics of Apache Airflow® notifications. Learn how to set up automatic email and Slack notifications to be alerted of events in your Dags.

When you're using a data orchestration tool, how do you know when something has gone wrong? [Apache Airflow®](https://airflow.apache.org/) users can check the Airflow UI to determine the status of their Dags, but this is an inefficient way of managing errors systematically, especially if certain failures need to be addressed promptly or by multiple team members. Fortunately, Airflow has several notification mechanisms that can be used to configure error notifications in a way that works for your organization.

In this guide, you'll learn how to set up common Airflow notification mechanisms including [email (SMTP) notifications](#email-smtp-notifications), [Airflow callbacks](#airflow-callbacks) and [notifiers](#pre-built-notifiers).

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow Dags. See [Introduction to Airflow Dags](/docs/learn/dags).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow decorators. See [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).
* Airflow connections. See [Manage connections in Apache Airflow](/docs/learn/connections).

## Notification types

When setting up Airflow notifications, you must first decide between using Airflow's built-in notification system, an external monitoring service, or a combination of both. The three types of notifications available when running Airflow on Astro are:

<CardGroup>
  <Card title="Airflow notifications" icon="bell" href="#airflow-notification-concepts">
    Airflow notifications are available in open-source Airflow itself and defined using callback parameters and/or configuration variables relating to email and SMTP.
  </Card>

  <Card title="Astro alerts" icon="alarm-clock" href="/astro/alerts">
    Astro alerts are a feature of Astro that allows you to configure alerts for many Dags and Deployments at once.
  </Card>

  <Card title="Astro Observe" icon="radar" href="/astro/astro-observe">
    Astro Observe is a product provided by Astronomer that includes the ability to define data products spanning multiple Dags and Deployments, and define Service Level Agreements (SLAs) on them.
  </Card>
</CardGroup>

The advantage of Airflow notifications is that you can define them directly in your Dag code. The downside is that you need Airflow to be running to send notifications, which means you might run into silent failures if there is an issue with your Airflow infrastructure. Airflow notifications also have some limitations, for example relating to defining SLAs and timeouts.

For the cases where Airflow notifications aren't sufficient, [Astro alerts](/docs/astro/alerts) and [Astro Observe](/docs/astro/astro-observe) provide an additional level of observability. For guidance on when to choose Airflow notifications or Astro alerts, see [When to use Airflow or Astro alerts for your pipelines on Astro](/docs/astro/best-practices/airflow-vs-astro-alerts).

## Airflow notification concepts

When defining notifications in Airflow you should understand the following concepts:

<CardGroup>
  <Card title="Email (SMTP)" icon="envelope" href="#email-smtp-notifications">
    Airflow allows you to send email alerts using an external SMTP server. There are different ways to configure email notifications.
  </Card>

  <Card title="Airflow Callbacks" icon="phone-flip" href="#airflow-callbacks">
    Dag- and task-level parameters that allow you to define code that should be executed when a Dag or task reaches a specific state. Can use plain Python functions or notifiers.
  </Card>

  <Card title="Airflow Notifiers" icon="bell" href="#pre-built-notifiers">
    A type of Airflow class like operators or hooks, that can be used to standardize your code. Each notifier has a `.notify()` method for sending notifications.
  </Card>

  <Card title="Airflow Listeners" icon="ear-listen" href="https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/listeners.html">
    An advanced Airflow feature that runs in the background and executes code when certain events occur anywhere in your Airflow environment.
  </Card>
</CardGroup>

### Choose your Airflow notification method

It's best practice to use pre-built solutions whenever possible. This approach makes your Dags more robust by reducing custom code and standardizing notifications across different Airflow environments.

If you want to deliver notifications to email, use the [SmtpNotifier](#use-the-smtpnotifier) or [`EmailOperator`](#use-the-emailoperator). If you want to use another email service like SendGrid or Amazon SES, see the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/howto/email-config.html) for more information.

If you want to be notified using another system, check if a [notifier class](#pre-built-notifiers) exists for your use case. See the Airflow documentation for [an up-to-date list of available Notifiers](https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html) and the [Apprise wiki](https://github.com/caronc/apprise/wiki) for a list of services the [AppriseNotifier](https://airflow.apache.org/docs/apache-airflow-providers-apprise/stable/_api/airflow/providers/apprise/notifications/apprise/index.html) can connect to.

Only use custom [callback functions](#airflow-callbacks) when no notifier is available for your use case. Consider writing a [custom notifier](#pre-built-notifiers) to standardize the code you use to send notifications.

If you want to execute code based on events happening anywhere in your Airflow environment, for example whenever any asset is updated, a Dag run fails, or a new import error is detected, you can use [Airflow listeners](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/listeners.html#listeners).

## Email (SMTP) notifications

Airflow email notifications can be set up in three different ways:

* (Recommended) You can provide the [`SmtpNotifier`](#use-the-smtpnotifier) with any [callback parameter](#airflow-callbacks) to send emails when a Dag or task reaches a specific state.
* (Recommended) You can use the [`EmailOperator`](#use-the-emailoperator) to create dedicated tasks in your Dags to send emails.
* (Legacy) You can configure email notifications using the `email` task parameter in combination with Airflow configuration variables in the `SMTP` section. This approach has limitations and will be removed in a future version. See [(Legacy) Email notifications using configuration variables](#legacy-email-notifications-using-configuration-variables).

All email notifications require you to install the [SMTP provider](https://airflow.apache.org/registry/providers/smtp/) by adding it to your `requirements.txt` file.

```text wrap theme={null}
apache-airflow-providers-smtp
```

### Use the SmtpNotifier

The `SmtpNotifier` is a pre-built notifier that can be provided to any [callback parameter](#airflow-callbacks) to send emails when a Dag or task reaches a specific state.

To connect the notifier to your SMTP server, you need to create an [Airflow connection](/docs/learn/connections), for example by setting the following environment variable to create the `smtp_default` connection:

```text wrap theme={null}
AIRFLOW_CONN_SMTP_DEFAULT='{
   "conn_type":"smtp",
   "host":"smtp.yourdomain.com",
   "port":<your-port>,
   "login":"<your-username>",
   "password":"<your-password>",
   "extra":{
      "disable_ssl":<your-setting>,
      "disable_tls":<your-setting>
   }
}'
```

The main parameters to configure for the [SmtpNotifier](https://airflow.apache.org/registry/providers/smtp#smtp-smtp-SmtpNotifier) are:

* `smtp_conn_id`: The ID of the Airflow connection to your SMTP server. Default: `smtp_default`.
* `to`: The email address to send the email to. You can provide a single email address as a string or multiple in a list. Default: `None`. This parameter is required.
* `cc`: The email address to send the email to as a carbon copy. You can provide a single email address as a string or multiple in a list. Default: `None`.
* `bcc`: The email address to send the email to as a blind carbon copy. You can provide a single email address as a string or multiple in a list. Default: `None`.
* `from_email`: The email address to send the email from. Default: `None`.
* `subject`: The subject of the email. Default: `None`.
* `html_content`: The HTML content of the email. Default: `None`.
* `files`: The files to attach to the email as a list of file paths. Default: `None`.
* `custom_headers`: A dictionary of custom headers to add to the email. Default: `None`.

You provide the instantiated notifier class directly to any [callback parameter](#airflow-callbacks) to send emails when that callback is triggered. To add information about the Dag run to the email, use [Jinja templating](/docs/learn/templating). All parameters listed above other than `smtp_conn_id` are templatable.

For example, to send an email notification when a task fails that includes information about the task, as well as the error message (`{{ exception }}`) and a link to the task's log (`{{ ti.log_url }}`), you can use the SmtpNotifier as shown in the code example below.

```python expandable wrap theme={null}
@task(
    on_failure_callback=SmtpNotifier(
        from_email="testnotifier@test.com",
        to=["primary@test.com"],
        cc=["manager@test.com", "team-lead@test.com"],
        bcc=["audit@test.com", "monitoring@test.com"],
        subject="{{ ti.task_id }} failed in {{ dag.dag_id }}",
        html_content="""
            <html>
                <body>
                    <h2 style="color: red;">Task Failure Alert</h2>
                    <p><strong>Task:</strong> {{ ti.task_id }}</p>
                    <p><strong>DAG:</strong> {{ dag.dag_id }}</p>
                    <p><strong>Execution Date:</strong> {{ ts }}</p>
                    <p><strong>Log URL:</strong> {{ ti.log_url }}</p>
                    <hr>
                    <h3>Error Details:</h3>
                    <pre>{{ exception }}</pre>
                </body>
            </html>
        """,
        files=["include/debug_info.json"],
        custom_headers={
            "X-Priority": "1",
            "X-Airflow-DAG": "{{ dag.dag_id }}",
            "X-Airflow-Task": "{{ ti.task_id }}",
            "Reply-To": "airflow-support@test.com"
        }
    )
)
def test_notifier_advanced():
    raise Exception("Oops, too much vibe coding!")
```

The resulting email looks like this:

<Frame>
  <img alt="Example email notification using the SmtpNotifier" />
</Frame>

<Note>
  If you'd like to test email formatting locally without connecting to a real SMTP server, you can use [MailHog](https://github.com/mailhog/MailHog) in a local Docker container to catch emails and view them in a web interface. Start the MailHog server using `docker run -d -p 1025:1025 -p 8025:8025 mailhog/mailhog` and use the following connection string:

  ```text wrap theme={null}
  AIRFLOW_CONN_SMTP_DEFAULT='{
     "conn_type":"smtp",
     "host":"localhost",
     "port":1025,
     "login":"",
     "password":"",
     "extra":{
        "disable_ssl":true,
        "disable_tls":true
     }
  }'
  ```
</Note>

### Use the `EmailOperator`

You can use the `EmailOperator` to create dedicated tasks in your Dags to send emails. As with the `SmtpNotifier`, you need to have the [SMTP provider](https://airflow.apache.org/registry/providers/smtp/) installed and create an [Airflow connection](/docs/learn/connections) to your SMTP server. The main parameters are analogous to the `SmtpNotifier`.

```python wrap theme={null}
from airflow.providers.smtp.operators.smtp import EmailOperator

EmailOperator(
    task_id="send_email",
    conn_id="smtp_default",
    from_email="caller@mydomain.io",
    to="receiver@mydomain.io",     
    subject="Test Email",
    html_content="This is a test email"
)
```

### (Legacy) Email notifications using configuration variables

In older Airflow versions it was common to configure email notifications using a mix of configuration variables and task parameters. This approach is being deprecated in Airflow 3.0 and will be removed in a future version.

To configure email notifications using configuration variables, both the [SMTP](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#smtp) configuration variables and the `email` task parameter are needed. Note that you can't use an `AIRFLOW_CONN_` connection with the email configuration parameters in Airflow 3.

The SMTP configuration variables define the connection to your SMTP server.

```text wrap theme={null}
AIRFLOW__SMTP__SMTP_HOST=<your-smtp-host>
AIRFLOW__SMTP__SMTP_PORT=<your-port>
AIRFLOW__SMTP__SMTP_USER=<your-username>
AIRFLOW__SMTP__SMTP_PASSWORD=<your-password>
AIRFLOW__SMTP__SMTP_SSL=<your-setting>
AIRFLOW__SMTP__SMTP_TLS=<your-setting>
AIRFLOW__SMTP__SMTP_MAIL_FROM=<your-from-email>
```

In order for a task to be able to send an email if it fails or retries, you need to provide the `email` task parameter to the task to specify who to send the email to.

It is common to provide this in the `default_args` parameter of a Dag to apply it to all tasks in the Dag.

```python wrap theme={null}
from airflow.sdk import dag

@dag(
    default_args={
        "email": ["myname@mydomain.com"],
    }
)
def my_dag():
```

But you can also provide it at the task level to override the default.

```python wrap theme={null}
@task(email=["myfriend@mydomain.com"])
def test():
    print("Test")
    raise Exception("Test Exception")
```

If you want a task to only send emails when it fails, set the `email_on_retry` parameter to `False`, if you want it to only send emails when it retries, set the `email_on_failure` parameter to `False`.

<Note>
  Most of the `AIRFLOW__EMAIL__` configuration variables are no longer supported in Airflow 3.0 for SMTP-based email notifications. Some of those parameters are still used when utilizing other email notification methods such as SendGrid or Amazon SES, see [Email Configuration](https://airflow.apache.org/docs/apache-airflow/stable/howto/email-config.html) in the Airflow documentation for more information.
</Note>

## Airflow callbacks

In Airflow you can define actions to be taken based on different Dag or task states using `*_callback` parameters:

* `on_success_callback`: Invoked when a task or Dag succeeds.
* `on_failure_callback`: Invoked when a task or Dag fails.
* `on_skipped_callback` : Invoked when a task is skipped. This callback only exists at the task level, and is only invoked when an `AirflowSkipException` is raised, not when a task is skipped due to other reasons, like a trigger rule.
* `on_execute_callback`: Invoked right before a task begins executing. This callback only exists at the task level.
* `on_retry_callback`: Invoked when a task is retried. This callback only exists at the task level.

You can provide any Python callable or [Airflow notifiers](#pre-built-notifiers) to the `*_callback` parameters. To execute multiple functions, you can provide several callback items to the same callback parameter in a list.

### Set Dag-level callbacks

To define a notification at the Dag level, you can set the `*_callback` parameter in your Dag instantiation. Dag-level notifications will trigger callback functions based on the terminal state of the entire Dag run. The example below shows one function being executed when the Dag succeeds and two functions being executed when the Dag fails (one custom function and one `SlackNotifier`).

```python wrap theme={null}
from airflow.sdk import dag
from airflow.providers.slack.notifications.slack_notifier import SlackNotifier

def my_success_callback_function(context):
    pass

def my_failure_callback_function(context):
    pass

@dag(
    on_success_callback=my_success_callback_function,
    on_failure_callback=[
        my_failure_callback_function,
        SlackNotifier(
            slack_conn_id="slack_conn",
            text="Dag failed",
            channel="alerts"
        )
    ],
)
```

<Note>
  Deadline alerts, which are executed when a Dag run exceeds a user-defined time threshold replace the removed SLA feature used with the `sla` and `sla_miss_callback` parameters. See [Deadline alerts](https://airflow.apache.org/docs/apache-airflow/stable/howto/deadline-alerts.html) in the Airflow documentation for more information.

  Astronomer customers should use [Astro alerts](/docs/astro/alerts) and [Astro Observe](/docs/astro/astro-observe) to define timeliness and freshness SLAs.
</Note>

### Set task-level callbacks

To apply a task-level callback to each task in your Dag, you can pass the callback function to the `default_args` parameter. Items listed in the dictionary provided to the `default_args` parameter will be set for each task in the Dag. While the example shows one callback function being assigned to each callback parameter, you can provide multiple callback functions and/or notifiers to the same callback parameter in a list as well.

```python wrap theme={null}
from airflow.sdk import dag

def my_execute_callback_function(context):
    pass

def my_retry_callback_function(context):
    pass

def my_success_callback_function(context):
    pass

def my_failure_callback_function(context):
    pass

def my_skipped_callback_function(context):
    pass

@dag(
    default_args={
        "on_execute_callback": my_execute_callback_function,
        "on_retry_callback": my_retry_callback_function,
        "on_success_callback": my_success_callback_function,
        "on_failure_callback": my_failure_callback_function,
        "on_skipped_callback": my_skipped_callback_function,
    }
)
```

For use cases where an individual task should use a specific callback, the task-level callback parameters can be defined in the task instantiation. Callbacks defined at the individual task level will override callbacks passed in using `default_args`.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import task

  def my_execute_callback_function(context):
      pass

  def my_retry_callback_function(context):
      pass

  def my_success_callback_function(context):
      pass

  def my_failure_callback_function(context):
      pass

  def my_skipped_callback_function(context):
      pass

  @task(
      on_execute_callback=my_execute_callback_function,
      on_retry_callback=my_retry_callback_function,
      on_success_callback=my_success_callback_function,
      on_failure_callback=my_failure_callback_function,
      on_skipped_callback=my_skipped_callback_function,
  )
  def t1():
      return "hello"
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.providers.standard.operators.python import PythonOperator

  def my_execute_callback_function(context):
      pass

  def my_retry_callback_function(context):
      pass

  def my_success_callback_function(context):
      pass

  def my_failure_callback_function(context):
      pass

  def my_skipped_callback_function(context):
      pass

  def say_hello():
      return "hello"

  t1 = PythonOperator(
      task_id="t1",
      python_callable=say_hello,
      on_execute_callback=my_execute_callback_function,
      on_retry_callback=my_retry_callback_function,
      on_success_callback=my_success_callback_function,
      on_failure_callback=my_failure_callback_function,
      on_skipped_callback=my_skipped_callback_function,
  )
  ```
</details>

### Pre-built notifiers

[Airflow notifiers](https://airflow.apache.org/docs/apache-airflow/stable/howto/notifications.html) are pre-built or custom classes and can be used to standardize and modularize the functions you use to send notifications. Notifiers can be passed to the relevant `*_callback` parameter of your Dag depending on what event you want to trigger the notification.

<Info>
  You can find a full list of all pre-built notifiers created for Airflow providers [here](https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/notifications.html) and connect to [many more services](https://github.com/caronc/apprise/wiki) through the [AppriseNotifier](https://airflow.apache.org/docs/apache-airflow-providers-apprise/stable/_api/airflow/providers/apprise/notifications/apprise/index.html).
</Info>

Notifiers are defined in provider packages or imported from the `include` folder and can be used across any of your Dags. This feature has the advantage that community members can define and share functionality previously used in callback functions as Airflow modules, creating pre-built callbacks to send notifications to other data tools.

#### Example pre-built notifier: Slack

An example of a community provided pre-built notifier is the [SlackNotifier](https://airflow.apache.org/docs/apache-airflow-providers-slack/stable/_api/airflow/providers/slack/notifications/slack/index.html#module-airflow.providers.slack.notifications.slack).

It can be imported from the Slack provider package and used with any `*_callback` function:

```python expandable wrap theme={null}
"""
Example showing how to use the SlackNotifier. Needs a Slack connection set
up with Slack API Token for a Slack bot (starts with 'xoxb-...')
"""

from airflow.sdk import dag, task
from pendulum import datetime
from airflow.providers.slack.notifications.slack_notifier import SlackNotifier

SLACK_CONNECTION_ID = "slack_conn"
SLACK_CHANNEL = "alerts"
SLACK_MESSAGE = """
Hello! The {{ ti.task_id }} task is saying hi :wave: 
Today is the {{ ds }} and this task finished with the state: {{ ti.state }} :tada:.
"""


@dag
def slack_notifier_example_dag():
    @task(
        on_success_callback=SlackNotifier(
            slack_conn_id=SLACK_CONNECTION_ID,
            text=SLACK_MESSAGE,
            channel=SLACK_CHANNEL,
        ),
    )
    def post_to_slack():
        return 10

    post_to_slack()


slack_notifier_example_dag()
```

The Dag above has one task sending a notification to Slack. It uses a Slack [Airflow connection](/docs/learn/connections) with the connection ID `slack_conn`.

<Frame>
  <img alt="Slack notification" />
</Frame>

### Custom notifiers

If no notifier exists for your use case you can write your own! An Airflow notifier can be created by inheriting from the `BaseNotifier` class and defining the action which should be taken in case the notifier is used in the `.notify()` method.

```python wrap theme={null}
from airflow.sdk import BaseNotifier

class MyNotifier(BaseNotifier):
    """
    Basic notifier, prints the task_id, state and a message.
    """

    template_fields = ("message",)

    def __init__(self, message):
        self.message = message

    def notify(self, context):
        t_id = context["ti"].task_id
        t_state = context["ti"].state
        print(
            f"Hi from MyNotifier! {t_id} finished as: {t_state} and says {self.message}"
        )
```

To use the custom notifier in a Dag, provide its instantiation to any callback parameter. For example:

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import task

  def say_hello():
      return "hello"

  @task(
      on_failure_callback=MyNotifier(message="Hello failed!"),
  )
  def t1():
      return "hello"
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.providers.standard.operators.python import PythonOperator

  def say_hello():
      return "hello"

  t1 = PythonOperator(
      task_id="t1",
      python_callable=say_hello,
      on_failure_callback=MyNotifier(message="Hello failed!"),
  )
  ```
</details>


# Microsoft Teams notifications
Source: https://astronomer.io/docs/learn/example-ms-teams-callback

Configure notifications in Microsoft teams for DAG runs and tasks using Airflow callbacks.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

This example shows how to set up Airflow notifications in a [Microsoft Teams](https://www.microsoft.com/en-us/microsoft-teams/group-chat-software) channel by using [Airflow callbacks](/docs/learn/error-notifications-in-airflow#airflow-callbacks). Teams notifications about DAG runs and tasks let you quickly inform many team members about the status of your data pipelines.

## Before you start

Before trying this example, make sure you have:

* [Teams](https://www.microsoft.com/en-us/microsoft-teams/log-in) with a Business account supporting team channels.
* The [Astro CLI](/docs/cli/v1.43/install-cli).
* An Astro project running locally on your computer. See [Getting started with the Astro CLI](/docs/cli/v1.43/get-started-cli).

## Send task failure notifications to MS Teams

Follow these steps to receive notifications in MS Teams for failed tasks in an example DAG. Refer to the [Airflow callbacks section](/docs/learn/error-notifications-in-airflow#airflow-callbacks) of our notifications guide to learn how to set up notifications for other types of events.

1. Open the folder containing your Astro Project. Copy the contents of the `include` folder in the [project GitHub repository](https://github.com/astronomer/cs-tutorial-msteams-callbacks/tree/main/include) to your Astro project `include` folder.

   ```text wrap theme={null}
   ├── .astro
   ├── dags
   └── include
       ├── hooks
       │   └── ms_teams_webhook_hook.py
       ├── operators
       │   └── ms_teams_webhook_operator.py
       ├── ms_teams_callback_functions.py
       └── ms_teams_callback_functions_with_partial.py
   ```

2. Create a [Microsoft Teams Incoming Webhook](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook?tabs=dotnet#create-incoming-webhooks-1) for the channel where you want to receive notifications. Copy and save the webhook URL.

3. In the Airflow UI, go to **Admin** > **Connections** and create a new connection with the following parameters. Note that you won't be able to test this connection from the Airflow UI.

   * **Connection Id**: `ms_teams_callbacks`
   * **Connection Type**: `HTTP`
   * **Host**: `<your-organization>.office.com/webhook/<your-webhook-id>`
   * **Schema**: `https`

   <Frame>
     <img alt="Connection" />
   </Frame>

   <Info>
     Some corporate environments use outbound proxies. If you're behind an outbound proxy for internet access, put the proxy details in the **Extra** field when creating the HTTP connection in the Airflow UI (for example, `{"proxy":"http://my-proxy:3128"}`).
   </Info>

   <Info>
     If the `HTTP` connection type isn't available, double check that the [HTTP provider](https://airflow.apache.org/registry/providers/http/) is installed in your Airflow environment.
   </Info>

4. Import the failure callback function at the top of your DAG file.

   ```python wrap theme={null}
   from include.ms_teams_callback_functions import failure_callback
   ```

5. Set the `on_failure_callback` keyword of your DAG's `default_args` parameter to the imported `failure_callback` function.

   ```python wrap theme={null}
   @dag(
       start_date=datetime(2023, 7, 1),
       schedule="@daily",
       default_args={
           "on_failure_callback": failure_callback,
       }
   )
   ```

6. Run your DAG. Any failed task will trigger the `failure_callback` function which sends a notification message to your Teams channel.

The `include` folder of the [project repository](https://github.com/astronomer/cs-tutorial-msteams-callbacks) also contains callback functions for other triggers in addition to the failure callback shown here. You can modify any of the functions to customize the notification message. To learn more about all available callback parameters, see [Airflow callbacks](/docs/learn/error-notifications-in-airflow#airflow-callbacks).

<Frame>
  <img alt="Notification" />
</Frame>

## See also

* [MS Teams developer documentation](https://learn.microsoft.com/en-us/microsoftteams/platform/mstdd-landing)
* [Manage Airflow DAG notifications](/docs/learn/error-notifications-in-airflow)


# Execute a Jupyter notebook with Airflow
Source: https://astronomer.io/docs/learn/execute-notebooks

Run a parameterized Jupyter notebook using Airflow and the Astro CLI.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

Jupyter notebooks are a popular open source notebook tool for quickly developing code and presenting data visualizations. They are frequently used in exploratory data analysis, data science, analytics, and reporting. This tutorial demonstrates how to run a Jupyter notebook from Airflow using the [Papermill provider package](https://airflow.apache.org/registry/providers/papermill). The `PapermillOperator` contained in this package executes a notebook as an Airflow task.

After you complete this tutorial, you'll be able to:

* Add a Jupyter notebook to your Astro CLI project.
* Run your Jupyter notebook from an Airflow DAG.
* Pass parameters to your Jupyter notebook from Airflow.
* Understand what use cases are ideal for orchestrating Jupyter notebooks with Airflow.

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Creating Jupyter notebooks. See [Try Jupyter](https://docs.jupyter.org/en/latest/start/index.html).

## Prerequisites

To complete this tutorial, you need:

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).
* The [Jupyter Notebook](https://jupyter.org/install) package.

## Step 1: Create an Astro project and a Jupyter notebook

To run a DAG that executes a Jupyter notebook, you first need to create an Astro project, which contains the set of files necessary to run Airflow locally.

1. Create a new directory for your Astro project:

   ```sh wrap theme={null}
   mkdir <your-astro-project-name>
   ```

2. Open the directory:

   ```sh wrap theme={null}
   cd <your-astro-project-name>
   ```

3. Run the following Astro CLI command to initialize an Astro project in the directory:

   ```sh wrap theme={null}
   astro dev init
   ```

Next, create a Jupyter notebook called `example_notebook.ipynb` and save it to the `include/` directory of the Astro project you created.

## Step 2: Optionally parameterize your Jupyter notebook

Parameterize any cells in your notebook as needed. If you need to pass any information to your notebook at run time, tag the cell in your notebook as described in the [Papermill usage documentation](https://papermill.readthedocs.io/en/latest/usage-parameterize.html).

The following notebook prints a simple statement with the current date. The second cell is parameterized so that the `execution_date` is dynamic.

<Frame>
  <img alt="Notebook param" />
</Frame>

## Step 3: Install supporting packages

Install the Papermill provider and supporting packages required to run the notebook kernel.

Add the following to the `requirements.txt` file of your Astro project:

```text wrap theme={null}
apache-airflow-providers-papermill
ipykernel
```

The `PapermillOperator` is designed to run a notebook locally, so you need to supply a kernel engine for your Airflow environment to execute the notebook code. This tutorial uses the `ipykernel` package to run the kernel, but there are other options available such as the `jupyter` package.

## Step 4: Create your DAG

Create your DAG with the `PapermillOperator` to execute your notebook. Use your favorite code editor or text editor to copy-paste the following code into a `.py` file in your project's `dags/` directory:

```python wrap theme={null}
from datetime import datetime, timedelta

from airflow.models.dag import DAG
from airflow.providers.papermill.operators.papermill import PapermillOperator

with DAG(
    dag_id='example_papermill_operator',
    default_args={
        'retries': 0
    },
    schedule='0 0 * * *',
    start_date=datetime(2022, 10, 1),
    template_searchpath='/usr/local/airflow/include',
    catchup=False
) as dag_1:

    notebook_task = PapermillOperator(
        task_id="run_example_notebook",
        input_nb="include/example_notebook.ipynb",
        output_nb="include/out-{{ execution_date }}.ipynb",
        parameters={"execution_date": "{{ execution_date }}"},
    )
```

The `PapermillOperator` requires the following arguments:

* `input_nb`: The notebook you want to run.
* `output_nb`: The path to your output notebook (that is, the notebook which shows the results of the notebook execution).
* `parameters`: A JSON dictionary of any parameters you are passing to your notebook.

Note that the DAG uses the built-in `execution_date` Airflow variable so that it's idempotent. Parameters for your notebook can come from anywhere, but Astronomer recommends using Airflow macros and environment variables to avoid hard-coding values in your DAG file.

## Step 5: Run your DAG to execute your notebook

Trigger your DAG to execute the `example_notebook.ipynb` and generate an output notebook with a name that includes the execution date. Open the output notebook in your `include/` directory to see the results of the run:

<Frame>
  <img alt="Output notebook" />
</Frame>

<Info>
  With some versions of `papermill` you might encounter a bug when writing grammar tables as described in [this GitHub issue](https://github.com/psf/black/issues/1143). The error would say something like `Writing failed: [Errno 2] No such file or directory: '/home/astro/.cache/black/21.7b0/tmpzpsclowd'`. If this occurs, a workaround is to manually add that directory to your Airflow environment. If using an Astro project, you can add `RUN mkdir -p /home/astro/.cache/black/21.7b0/` to your project's `Dockerfile`.
</Info>

## Additional considerations

Running Jupyter notebooks from Airflow is a great way to accomplish many common data science and data analytics use cases like generating data visualizations, performing exploratory data analysis, and training small machine learning models.

However, there are several cases where this might not be the best approach:

* Because the Jupyter notebook runs within your Airflow environment, this method isn't recommended for notebooks that process large data sets. For notebooks that are computationally intensive, [Databricks](/docs/learn/airflow-databricks) or notebook instances from cloud providers like AWS or GCP may be more appropriate.
* Notebooks are run in their entirety during each DAG run and don't maintain state between runs. This means you run every cell in your notebook on every DAG run. For this reason, if you have code that takes a long time to run (such as a large ML model), a better approach may be to break up the code into distinct Airflow tasks using other tools.


# Intro to Airflow tutorial: Get started and run your first pipeline
Source: https://astronomer.io/docs/learn/get-started-with-airflow

A hands-on Airflow tutorial to help you run your first data pipeline. Learn core concepts and build ETL workflows with Apache Airflow in minutes.

This tutorial will get you started as quickly as possible while explaining the core concepts of Apache Airflow. You will explore galaxies 🌌 while extending an existing workflow with modern Airflow features, setting you up for diving into the world of data orchestration with Apache Airflow.

<Tip>
  No matter if you are an absolute Airflow beginner or already know about certain concepts, in 5 minutes from now, you will have your first data pipeline (a Dag) running in a fully functional Airflow environment.
</Tip>

<CardGroup>
  <Card title="Set up in minutes" icon="rocket">
    Get a fully functional Airflow environment running in your browser with zero local setup using Astro IDE.
  </Card>

  <Card title="Build your first pipeline" icon="diagram-project">
    Create and run an ETL pipeline that processes galaxy data with extraction, transformation, and loading steps.
  </Card>

  <Card title="Master core concepts" icon="graduation-cap">
    Learn Dags, tasks, operators, dependencies, and asset-aware scheduling through hands-on practice.
  </Card>
</CardGroup>

## Step 1: Set up your Astro trial and Astro IDE

1. The first step is to [start a free Astro trial](https://www.astronomer.io/lp/signup/?utm_source=website\&utm_medium=learn-guides\&utm_campaign=learn-intro-tutorial-11-25).

   All Astro accounts have access to the Astro IDE, which is the easiest way to develop Airflow Dags right in your browser. You can directly deploy your Dags from the Astro IDE to an Astro Deployment, an Airflow environment running in the cloud. After entering your email address, starting the trial includes 4 steps:

   1. Choose between professional and personal. The choice has no impact on this tutorial.
   2. Enter an organization and workspace name. Each customer has a dedicated organization on Astro. Each team or project has a workspace, which is a collection of deployments. A deployment is an Airflow environment hosted on Astro. For this tutorial, you can use any names.
   3. You can choose to upload Dags, use a template, or start with an empty workspace. For this tutorial, choose **Start with a template**.
   4. Choose the **ETL** template.

   <Frame>
     <img alt="Astro trial flow" />
   </Frame>

   <Info>
     **Astro Concepts**

     * **Astro**: Fully-managed platform that helps teams write and run data pipelines with Airflow at any scale.
     * **Astro IDE**: In-browser IDE with context-aware AI and zero local setup.
     * **Organization**: Each customer has a dedicated org on Astro.
     * **Workspace**: Each team or project has a dedicated workspace, containing a collection of deployments.
     * **Deployment**: Airflow environment hosted on Astro.
     * **Summary**: 1 **Organization** → n **Workspace** → n **Deployment** → 1 **Airflow instance**.
   </Info>

   After your environment is created, you'll find yourself in the Astro IDE with your very first ETL Dag, ready to be deployed. The Python code is a programmatic representation of your workflow. Clicking **Start Test Deployment** in the top right starts a fully functional Airflow environment and deploys your code.

2. Click **Start Test Deployment** and wait for the deployment to finish.

   <Frame>
     <img alt="Astro trial flow" />
   </Frame>

3. Your first Airflow Dag is deployed and ready to be executed. Click the dropdown menu next to **Sync to Test** and select **Open Airflow**.

   <Frame>
     <img alt="Open Airflow from Astro IDE" />
   </Frame>

   The [Airflow UI](/docs/learn/airflow-ui) home dashboard of your Airflow instance will open in a new browser tab.

   <Frame>
     <img alt="Airflow home dashboard" />
   </Frame>

## Step 2: Run your first Dag

1. Within the navbar on the left, click **Dags**.

   This view shows all your Dags defined in your Python code. The ETL template comes with one Dag named `example_etl_galaxies`.

   <Frame>
     <img alt="Dags view" />
   </Frame>

   This ETL (Extract, Transform, Load) pipeline retrieves data about galaxies, filters them based on their distance from the Milky Way, and stores the results in a [DuckDB](https://duckdb.org/) database.

   <Frame>
     <img alt="Graph representation of the Dag" />
   </Frame>

   The Dag has these tasks:

   * **`create_galaxy_table_in_duckdb`**: Creates a table in DuckDB with columns for galaxy name, distances, type, and characteristics.
   * **`extract_galaxy_data`**: Retrieves raw data about 20 galaxies and returns it as a pandas DataFrame.
   * **`transform_galaxy_data`**: Filters the galaxy data to keep only galaxies within a specified distance from the Milky Way (default: 500,000 light years).
   * **`load_galaxy_data`**: Inserts the filtered galaxy data into the DuckDB table and produces an Airflow Asset update.
   * **`print_loaded_galaxies`**: Queries and prints all stored galaxies from DuckDB, sorted by distance from the Milky Way.

   The tasks have these dependencies:

   * `create_galaxy_table_in_duckdb` → `load_galaxy_data` (table must exist before loading)
   * `extract_galaxy_data` → `transform_galaxy_data` (raw data is needed for filtering)
   * `transform_galaxy_data` → `load_galaxy_data` (filtered data is needed for loading)
   * `load_galaxy_data` → `print_loaded_galaxies` (data must be loaded before printing)

2. Run the pipeline by clicking **Play** next to the Dag.

   <Frame>
     <img alt="Trigger Dag run via the Dags view" />
   </Frame>

   This opens a trigger dialog, allowing you to trigger a single run or a [backfill](/docs/learn/rerunning-dags#backfill) to process a range of dates right from the UI. Dags can also have parameters that can be used within the implementation to keep certain parts of your pipeline configurable.

3. Select **Single Run**, keep the parameters at their defaults, and click **Trigger**.

   <Frame>
     <img alt="Trigger Dag run dialog" />
   </Frame>

   Your Dag starts, and under **Latest Run** in the **Dags** view you'll see the current running instance of it.

4. Click that run date to go to the individual Dag run view.

   <Frame>
     <img alt="Latest run in the Dags view" />
   </Frame>

   Watch how the Dag run finishes and explore the grid and graph views (buttons on the top left), two different representations of your pipeline.

5. After all tasks have finished successfully, open the grid view and click the `print_loaded_galaxies` task, the last step in your pipeline graph.

   <Frame>
     <img alt="Task selection of a Dag run in the grid view" />
   </Frame>

   This opens the logs of this task instance, showing a table of galaxies with their distance from the Milky Way and from the solar system, as well as the type of galaxy.

   <Frame>
     <img alt="Task logs in the Airflow UI" />
   </Frame>

<Tip>
  You just set up your Airflow development environment, started your first Airflow environment, and deployed and ran your first Dag. Take a moment to check the time and see how quickly you got there.
</Tip>

Take your time to explore the UI, trigger more runs, check the logs of other tasks, and make yourself familiar with the interface. Feel free to read the [Airflow UI guide](/docs/learn/airflow-ui) for a deep dive into its different views and functionality.

## Step 3: Understand the basic concepts

After you've finished your exploration, switch back to the Astro IDE and have a look at the Python code inside `example_etl_galaxies.py`. The code contains a lot of comments explaining each step in detail. Here's an overview before you dive into details.

The Python file contains the following key elements:

* **Imports**: All modules, classes, and functions needed for your implementation. Always use the Airflow Task SDK by importing from `airflow.sdk`, as this is the user-facing SDK.
* **Constants**: Any constants, like the connection string for the DuckDB instance.
* **Dag definition**: The data pipeline together with settings like its `schedule`.
* **Tasks**: The units of work. Tasks should be atomic and idempotent (producing the same result when run multiple times with the same inputs).
* **Dependencies**: How the tasks are connected, so Airflow knows how to construct the graph.

```python expandable wrap theme={null}
# imports
from airflow.sdk import Asset, chain, Param, dag, task
# ...

# constants
_DUCKDB_INSTANCE_NAME = os.getenv("DUCKDB_INSTANCE_NAME", "include/astronomy.db")
_DUCKDB_TABLE_NAME = os.getenv("DUCKDB_TABLE_NAME", "galaxy_data")
_DUCKDB_TABLE_URI = f"duckdb://{_DUCKDB_INSTANCE_NAME}/{_DUCKDB_TABLE_NAME}"
# ...

# Dag definition
@dag(...)
def example_etl_galaxies():

    # tasks
    @task(retries=2)
    def create_galaxy_table_in_duckdb(...):
        # ...

    @task
    def extract_galaxy_data(...):
        # ...

    @task
    def transform_galaxy_data(...):
        # ..

    @task
    def load_galaxy_data(...):
        # ...

    @task
    def print_loaded_galaxies(...):
        # ...

    # create task instances and define implicit dependencies
    create_galaxy_table_in_duckdb_obj = create_galaxy_table_in_duckdb()
    extract_galaxy_data_obj = extract_galaxy_data()
    transform_galaxy_data_obj = transform_galaxy_data(extract_galaxy_data_obj)
    load_galaxy_data_obj = load_galaxy_data(transform_galaxy_data_obj)

    # define explicit dependencies
    chain(
        create_galaxy_table_in_duckdb_obj, load_galaxy_data_obj, print_loaded_galaxies()
    )

# Instantiate the Dag
example_etl_galaxies()
```

<Tip icon="circle-info">
  **Airflow Concepts**

  * **[Dag](/docs/learn/dags)**: Your entire pipeline from start to finish, consisting of one or more tasks.
  * **[Task](/docs/learn/intro-to-airflow#airflow-concepts)**: A unit of work within your pipeline.
  * **[Operator/Decorator](/docs/learn/what-is-an-operator)**: The template/class that defines what work a task does, serving as the building blocks of pipelines.
    * Traditional: `task = PythonOperator(...)` → returns operator directly.
    * [TaskFlow API](/docs/learn/airflow-decorators): `@task def my_task(): ...` → creates operator, wrapped in `XComArg`.
    * Many decorators available: `@task`, `@task.bash`, `@task.docker`, `@task.kubernetes`, etc.
    * `XComArg`: Wrapper enabling automatic data passing and dependency inference.
</Tip>

## Step 4: Extend the demo project

Now that you've run your first Dag, extend the project by adding a second Dag that builds on top of the first one.

You'll create a `galaxy_maintenance` Dag that allows you to manually enter new galaxy data through an interactive form. The data is automatically added to the database and validated with automated quality checks.

What you'll learn:

<CardGroup>
  <Card title="Extend functionality" icon="puzzle-piece" href="https://airflow.apache.org/docs/apache-airflow-providers/">
    Add provider packages to extend Airflow with new operators and integrations for databases and external systems.
  </Card>

  <Card title="Connect to databases" icon="database" href="/learn/connections">
    Set up proper Airflow connections to manage credentials and configurations for external tools.
  </Card>

  <Card title="Human-in-the-loop (HITL)" icon="user-check" href="/learn/airflow-human-in-the-loop">
    Implement human-in-the-loop workflows that pause for manual data entry and human decision-making.
  </Card>

  <Card title="SQL operations" icon="code" href="/learn/airflow-sql">
    Use common SQL operators to run parameterized queries across different database systems.
  </Card>

  <Card title="Data quality" icon="shield-check" href="/learn/airflow-sql-data-quality">
    Add automated data quality checks to ensure data integrity throughout your pipelines.
  </Card>

  <Card title="Asset-aware scheduling" icon="calendar-clock" href="/learn/airflow-datasets">
    Trigger Dags based on asset-aware scheduling rather than time schedules for data-driven workflows.
  </Card>
</CardGroup>

By the end of this section, you'll have a powerful toolbox of concepts to explore Airflow further and confidently jump into your first real-world ETL/ELT project!

### Step 4.1: Add provider packages

The `example_etl_galaxies` Dag currently connects directly to the DuckDB database using:

```python wrap theme={null}
cursor = duckdb.connect(duckdb_instance_name)
```

While this works, Airflow offers a better approach: common SQL operators that execute queries using [Airflow connections](/docs/learn/connections). This unifies and simplifies SQL workloads across your pipelines. The following steps set this up.

Airflow's core functionality can be extended with [provider packages](https://airflow.apache.org/docs/apache-airflow-providers/) for specific use cases. This tutorial uses two providers for the DuckDB connection.

1. Open the `requirements.txt` file in the Astro IDE.

2. Add the following lines at the bottom:

   ```text wrap theme={null}
   apache-airflow-providers-common-sql==1.28.2
   airflow-provider-duckdb==0.2.0
   ```

3. Since you added new dependencies, sync the changes. Click **Sync to Test** and wait for the changes to be deployed.

### Step 4.2: Set up a connection

An [**Airflow connection**](/docs/learn/connections) stores configuration details for connecting to external tools in your data ecosystem. Most hooks ([what is a hook?](/docs/learn/what-is-a-hook)) and operators that interact with external systems require a connection.

To create the connection:

1. Open Airflow and click **Admin** in the left navbar

2. Select **Connections**

3. Click **Add Connection** (top right)

4. Enter the following details:

   * **Connection ID**: `duckdb_astronomy`
   * **Connection Type**: DuckDB
   * **Host**: `include/astronomy.db`
   * Leave the remaining fields empty.

   <Frame>
     <img alt="Add connection to DuckDB database" />
   </Frame>

5. Save the connection and you're now ready to connect! You can find the Airflow task that uses this connection in the example code in [Step 4.4](#step-4-4-implement-dag-with-human-in-the-loop).

<Info>
  **Astro Concepts**

  You just added a connection to this deployment (a single Airflow instance). If you deployed your Dags to another environment or recreated the test deployment, you'd need to add the connection again. Astro offers a helpful solution: under **Environment** → **Connections** in the Astro platform, you can set up workspace-wide connections that are available across all your Airflow instances. See [Manage Airflow connections and variables](/docs/astro/manage-connections-variables) in the Astro documentation.
</Info>

<Tip icon="circle-info">
  **Airflow Concepts**

  * **Provider package**: Provider packages are installable modules that contain pre-built decorators, operators, hooks, and sensors for integrating with external services and extending Airflow functionality.
  * **Connection**: Connections in Airflow are sets of configurations used to connect with other tools in the data ecosystem.
</Tip>

### Step 4.3: Prepare test deployment for advanced usage

The test deployment is a fully functional but minimal Airflow setup. To enable advanced features like asset-aware scheduling (explained [later](#step-4-4-implement-dag-with-human-in-the-loop)), you need to apply a quick configuration change.

1. In the Astro IDE, click the dropdown menu next to **Sync to Test** (top right).
2. Select **Test Deployment Details**.
3. Navigate to the **Environment** tab, click **Edit Deployment Variables**, and remove `AIRFLOW__SCHEDULER__USE_JOB_SCHEDULE` by clicking the trash bin icon next to it.
4. Click **Update Environment Variables** (bottom right) and you're ready to go! Head back to the Astro IDE.

<Frame>
  <img alt="Change test deployment environment" />
</Frame>

### Step 4.4: Implement Dag with human-in-the-loop

1. Within the Astro IDE, create a new file by right-clicking on the `dags` folder → **New File...** and name it `galaxy_maintenance.py`.

2. Paste the following content:

   ```python expandable wrap theme={null}
   from airflow.sdk import chain, dag, Asset, Param
   from airflow.providers.standard.operators.hitl import HITLEntryOperator
   from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator, SQLColumnCheckOperator

   _DUCKDB_TABLE_URI = "duckdb://include/astronomy.db/galaxy_data"
   _DUCKDB_CONN_ID = "duckdb_astronomy"

   galaxy_table_asset = Asset(_DUCKDB_TABLE_URI)

   @dag(schedule=galaxy_table_asset)
   def galaxy_maintenance():

       _enter_galaxy_details = HITLEntryOperator(
           task_id="enter_galaxy_details",
           subject="Please provide required information: ",
           params={
               "name": Param("", type="string"),
               "distance_from_milkyway": Param(10000, type="number"),
               "distance_from_solarsystem": Param(10000, type="number"),
               "type_of_galaxy": Param("Dwarf", type="string", enum=[
                   "Dwarf Spheroidal",
                   "Dwarf",
                   "Irregular",
                   "Spiral"
               ]),
               "characteristics": Param("", type="string")
           }
       )

       _insert_galaxy_details = SQLExecuteQueryOperator(
           task_id="insert_galaxy_details",
           conn_id=_DUCKDB_CONN_ID,
           show_return_value_in_logs=True,
           sql="""
               -- in case db was removed due to sync
               CREATE TABLE IF NOT EXISTS galaxy_data (
                   name STRING PRIMARY KEY,
                   distance_from_milkyway INT,
                   distance_from_solarsystem INT,
                   type_of_galaxy STRING,
                   characteristics STRING
               );
               INSERT OR IGNORE INTO galaxy_data BY NAME
               SELECT
                   $name AS name,
                   $distance_from_milkyway AS distance_from_milkyway,
                   $distance_from_solarsystem AS distance_from_solarsystem,
                   $type_of_galaxy AS type_of_galaxy,
                   $characteristics AS characteristics
           """,
           parameters={
               "name": "{{ task_instance.xcom_pull('enter_galaxy_details')['params_input']['name'] }}",
               "distance_from_milkyway": "{{ task_instance.xcom_pull('enter_galaxy_details')['params_input']['distance_from_milkyway'] }}",
               "distance_from_solarsystem": "{{ task_instance.xcom_pull('enter_galaxy_details')['params_input']['distance_from_solarsystem'] }}",
               "type_of_galaxy": "{{ task_instance.xcom_pull('enter_galaxy_details')['params_input']['type_of_galaxy'] }}",
               "characteristics": "{{ task_instance.xcom_pull('enter_galaxy_details')['params_input']['characteristics'] }}"
           }
       )

       _galaxy_dq_checks = SQLColumnCheckOperator(
           task_id="dq_checks",
           conn_id=_DUCKDB_CONN_ID,
           table="galaxy_data",
           column_mapping={
               "distance_from_milkyway": {
                   "min": {"geq_to": 10000},
                   "max": {"leq_to": 900000},
               },
               "distance_from_solarsystem": {
                   "min": {"geq_to": 10000},
                   "max": {"leq_to": 900000},
               },
           },
       )

       chain(_enter_galaxy_details, _insert_galaxy_details, _galaxy_dq_checks)

   galaxy_maintenance()
   ```

   This maintenance pipeline is triggered automatically whenever the galaxy data table is updated. It allows manual entry of new galaxy data through a human-in-the-loop interface, inserts the data into DuckDB, and runs data quality checks to ensure the values are within acceptable ranges.

   <Frame>
     <img alt="Graph representation of the maintenance Dag" />
   </Frame>

   The Dag has these tasks:

   * **`enter_galaxy_details`**: Pauses the pipeline and prompts a user to manually enter galaxy information (name, distances, type, and characteristics) through a form interface.
   * **`insert_galaxy_details`**: Inserts the user-provided galaxy data into the DuckDB table using the values collected from the previous task.
   * **`dq_checks`**: Validates the data quality by checking that distance values are within acceptable ranges (between 10,000 and 900,000 light years).

   The tasks have these dependencies:

   * `enter_galaxy_details` → `insert_galaxy_details` (user input needed before insertion)
   * `insert_galaxy_details` → `dq_checks` (data must be inserted before validation)

   <Tip icon="circle-info">
     **Airflow Concepts**

     * **Human-in-the-loop**: [Human-in-the-loop](/docs/learn/airflow-human-in-the-loop) workflows are processes that require human intervention, for example, to approve or reject an AI generated output, or choose a [branch](/docs/learn/airflow-branch-operator) in a Dag depending on the result of an upstream task.
     * **SQL operators**: The [common SQL provider](https://airflow.apache.org/registry/providers/common-sql/) is a great place to start when looking for [SQL-related operators](/docs/learn/airflow-sql). It includes the [`SQLExecuteQueryOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLExecuteQueryOperator) operator, which is a generic operator that can be used with a variety of databases, including Snowflake and Postgres. It also comes with data quality related operators, like the [`SQLColumnCheckOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLColumnCheckOperator).
     * **Parameters**: You can use `parameters` to have dynamic queries with placeholders. These will be handled on database-driver level.
   </Tip>

3. Click **Sync to Test** (top right) to sync your changes to the test deployment.

4. After the sync process finishes, head back to the Airflow UI.

5. Open the **Dags** view, and a new Dag should appear in the list.

Notice how the schedule is set to be triggered whenever the asset named `duckdb://include/astronomy.db/galaxy_data` is updated.

<Frame>
  <img alt="Second Dag in the Dags view" />
</Frame>

The first Dag updates this asset when data is loaded to DuckDB by using the `outlets` parameter:

```python wrap theme={null}
    @task(outlets=[Asset(_DUCKDB_TABLE_URI)])
    def load_galaxy_data(
        filtered_galaxy_df: pd.DataFrame,
        duckdb_instance_name: str = _DUCKDB_INSTANCE_NAME,
        table_name: str = _DUCKDB_TABLE_NAME,
    ):
    # ...
```

<Tip icon="circle-info">
  **Airflow Concepts**

  * **Asset** (object): Logical representation of data (table, model, file) used to establish dependencies. Can be used imperatively (code-based) or declaratively (implicit definition via the [`@asset`](/docs/learn/airflow-datasets#asset-definition) decorator). It is an abstract representation of data.
  * **Asset event**: Each time an asset is updated, the system creates an asset event object. This object includes the ID of the Dag that produced the update, the update timestamp, and optional custom information.
  * **Asset-aware scheduling**: Set the `schedule` of a Dag to one or more assets, optionally with a logical expression using AND (`&`) and OR (`|`) operators, so that the Dag is triggered when these assets receive asset update events.
  * **Producer task**: a task that produces updates to one or more assets provided to its `outlets` parameter, creating asset events when it completes successfully.
  * **Materialize**: Running a producer task, which updates an asset.
  * **@asset**: Declarative shortcut (Dag + task + asset(s) in one).
</Tip>

### Step 4.5: Try your advanced Dag

Time to see asset-aware scheduling and your new Dag in action.

<Warning>
  This tutorial stores the DuckDB database in a project file (`include/astronomy.db`). The `example_etl_galaxies` Dag creates a table in this database, but the file isn't included in the auto-generated project repository. As a result, each time you sync changes to the deployment, the database file disappears. To handle this, the `insert_galaxy_details` task in the second Dag uses `CREATE TABLE IF NOT EXISTS` in case the database file was removed between runs. To improve this, you could use a persistent database service, for example Snowflake or BigQuery.
</Warning>

1. Trigger `example_etl_galaxies` and observe what happens.

   You'll notice that `galaxy_maintenance` starts when `example_etl_galaxies` finishes. More precisely, when it updates the asset that triggers the other Dag.

2. While `galaxy_maintenance` is running, open the latest run and you'll notice there's a required action. This is part of the human-in-the-loop feature: your task is waiting for user input.

   <Frame>
     <img alt="Required actions for a Dag run" />
   </Frame>

3. Take time to explore the Airflow UI and see where these required actions are visible!

4. Open the required action to see the form defined in the code, and enter the following details:

   * **name**: Astro
   * **`distance_from_milkyway`**: 10000
   * **`distance_from_solarsystem`**: 10000
   * **`type_of_galaxy`**: Dwarf
   * **characteristics**: Looks amazing

   <Frame>
     <img alt="Human-in-the-loop form" />
   </Frame>

5. Click **OK** and observe how the pipeline proceeds. Pay close attention to the `dq_checks` task, which successfully validates the data.

6. Try it again by running `galaxy_maintenance` once more. This time, enter **42** as the distance and observe how the `dq_checks` task fails because the data quality check detected an issue with your galaxy data.

## Conclusion and next steps

**Congratulations 🎉!** You've just built two interconnected data pipelines using Apache Airflow, and along the way you've learned the fundamental concepts that power modern data orchestration.

In this tutorial, you:

* Set up a complete Airflow development environment in minutes using Astro IDE.
* Built and ran your first ETL pipeline with extraction, transformation, and loading steps.
* Mastered core Airflow concepts: Dags, tasks, operators, and dependencies.
* Extended your project with provider packages and database connections.
* Implemented human-in-the-loop workflows for manual data entry.
* Added automated data quality checks to ensure data integrity.
* Used asset-aware scheduling to create a dependency between two Dags.

Ready to dive deeper?

**Explore more guides:**

* [An introduction to Apache Airflow®](/docs/learn/intro-to-airflow)
* [Introduction to Dags](/docs/learn/dags)
* [An introduction to the Airflow UI](/docs/learn/airflow-ui)
* [Using Airflow to Execute SQL](/docs/learn/airflow-sql)
* [Assets and data-aware scheduling in Airflow](/docs/learn/airflow-datasets)
* [Get started with Airflow using the Astro CLI](/docs/cli/v1.43/get-started-cli)
* [Apache Airflow® GenAI Quickstart](/docs/learn/airflow-quickstart-genai)

**Join the academy and get certified:**

* [Airflow 101 Learning Path](https://academy.astronomer.io/path/airflow-101)
* [Airflow Dag Authoring Learning Path](https://academy.astronomer.io/path/airflow-dag-authoring)
* [Astro Onboarding Learning Path](https://academy.astronomer.io/path/astro-onboarding)


# Use the KubernetesPodOperator
Source: https://astronomer.io/docs/learn/kubepod-operator

Use the KubernetesPodOperator in Airflow to run tasks in Kubernetes Pods

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

The `KubernetesPodOperator` (KPO) runs a Docker image in a dedicated Kubernetes Pod. By abstracting calls to the Kubernetes API, the `KubernetesPodOperator` lets you start and run Pods from Airflow using DAG code.

In this guide, you'll learn:

* The requirements for running the `KubernetesPodOperator`.
* When to use the `KubernetesPodOperator`.
* How to configure the `KubernetesPodOperator`.
* The differences between the `KubernetesPodOperator` and the Kubernetes executor.

You'll also learn how to use the `KubernetesPodOperator` to run a task in a language other than Python, how to use the `KubernetesPodOperator` with XComs, and how to launch a Pod in a remote AWS EKS Cluster.

<Tip>
  On Astro, all of the infrastructure required to run the `KubernetesPodOperator` is hosted by Astronomer and managed automatically. Therefore, some of the use cases on this page might be simplified if you're running the `KubernetesPodOperator` on Astro. See [Run the `KubernetesPodOperator` on Astro](/docs/astro/kubernetespodoperator) to learn more.
</Tip>

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Astronomer Academy: [Airflow: The `KubernetesPodOperator`](https://academy.astronomer.io/astro-runtime-the-kubernetespodoperator-1) module.
  * Webinar: [Running Airflow Tasks in Isolated Environments](https://www.astronomer.io/events/webinars/running-airflow-tasks-in-isolated-environments/).
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Kubernetes basics. See the [Kubernetes Documentation](https://kubernetes.io/docs/home/).

## Prerequisites

To use the `KubernetesPodOperator` you need to install the Kubernetes provider package. To install it with pip, run:

```bash wrap theme={null}
pip install apache-airflow-providers-cncf-kubernetes==<version>
```

If you use the [Astro CLI](/docs/cli/v1.43/overview), you can alternatively install the package by adding the following line to your Astro project:

```text wrap theme={null}
apache-airflow-providers-cncf-kubernetes==<version>
```

Review the [Airflow Kubernetes provider Documentation](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/index.html#requirements) to make sure you install the correct version of the provider package for your version of Airflow.

You also need an existing Kubernetes cluster to connect to. This is commonly the same cluster that Airflow is running on, but it doesn't have to be.

You don't need to use the Kubernetes executor to use the `KubernetesPodOperator`. You can choose one of the following executors:

* Local executor
* LocalKubernetes executor
* Celery executor
* Kubernetes executor
* CeleryKubernetes executor

On Astro, the infrastructure needed to run the `KubernetesPodOperator` with the Celery executor is included with all clusters by default. For more information, see [Run the `KubernetesPodOperator` on Astro](/docs/astro/kubernetespodoperator).

### Run the `KubernetesPodOperator` locally

Setting up your local environment to use the `KubernetesPodOperator` can help you avoid time consuming deployments to remote environments.

Use the steps below to quickly set up a local environment for the `KubernetesPodOperator` using the [Astro CLI](/docs/cli/v1.43/overview). Alternatively, you can use the [Helm Chart for Apache Airflow](https://airflow.apache.org/docs/helm-chart/stable/index.html) to run open source Airflow within a local Kubernetes cluster. See [Getting Started With the Official Airflow Helm Chart](https://www.youtube.com/watch?v=39k2Sz9jZ2c\&ab_channel=Astronomer).

#### Step 1: Set up Kubernetes

<details>
  <summary>Windows And Mac</summary>

  The latest versions of Docker for Windows and Mac let you run a single node Kubernetes cluster locally. If you are using Windows, see [Setting Up Docker for Windows and WSL to Work Flawlessly](https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly). If you are using Mac, see [Docker Desktop for Mac user manual](https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly). It isn't necessary to install Docker Compose.

  1. Open Docker and go to **Settings** > **Kubernetes**.

  2. Select the `Enable Kubernetes` checkbox.

  3. Click **Apply and Restart**.

  4. Click **Install** in the **Kubernetes Cluster Installation** dialog.

     Docker restarts and the status indicator changes to green to indicate Kubernetes is running.
</details>

<details>
  <summary>Linux</summary>

  1. Install Microk8s. See [Microk8s](https://microk8s.io/).

  2. Run `microk8s.start` to start Kubernetes.
</details>

#### Step 2: Update the kubeconfig file

<details>
  <summary>Windows And Mac</summary>

  1. Use the following commands to copy the `docker-desktop` context from the Kubernetes configuration file and save it as a separate file in the `/include/.kube/` folder in your Astro project. The `config` file contains all the information the `KubernetesPodOperator` uses to connect to your cluster.

     ```bash wrap theme={null}
     kubectl config use-context docker-desktop
     kubectl config view --minify --raw > <Astro project directory>/include/.kube
     ```

     After running these commands, you will find a `config` file in the `/include/.kube/` folder of your Astro project which resembles this example:

     ```yaml wrap theme={null}
     apiVersion: v1
     clusters:
     - cluster:
         certificate-authority-data: <certificate-authority-data>
         server: https://kubernetes.docker.internal:6443/
       name: docker-desktop
     contexts:
     - context:
         cluster: docker-desktop
         user: docker-desktop
       name: docker-desktop
     current-context: docker-desktop
     kind: Config
     preferences: {}
     users:
     - name: docker-desktop
       user:
         client-certificate-data: <client-certificate-data>
         client-key-data: <client-key-data>
     ```

  2. If you have issues connecting, check the server configuration in the `kubeconfig` file. If `server: https://localhost:6445` is present, change to `server: https://kubernetes.docker.internal:6443` to identify the localhost running Kubernetes Pods. If this doesn't work, try `server: https://host.docker.internal:6445`.

  3. (Optional) Add the `.kube` folder to `.gitignore` if your Astro project is hosted in a GitHub repository and you want to prevent the file from being tracked by your version control tool.

  4. (Optional) Add the `.kube` folder to `.dockerignore` to exclude it from the Docker image.
</details>

<details>
  <summary>Linux</summary>

  In a `.kube` folder in your Astro project, create a config file with:

  ```bash wrap theme={null}
  microk8s.config > /include/.kube/config
  ```
</details>

#### Step 3: Create Kubernetes Connection in the Airflow UI

To run a Kubernetes pod locally, you can use the following .json template to create a .json connection string that you can then use to create a Kubernetes connection using the local Airflow UI. First, edit the template with the values you gathered in the previous step:

```json expandable wrap theme={null}
{
    "apiVersion": "v1",
    "clusters": [
        {
            "cluster": {
                "certificate-authority-data": "<certificate-authority-data>",
                "server": "https://kubernetes.docker.internal:6443"
            },
            "name": "docker-desktop"
        }
    ],
    "contexts": [
        {
            "context": {
                "cluster": "docker-desktop",
                "user": "docker-desktop"
            },
            "name": "docker-desktop"
        }
    ],
    "current-context": "docker-desktop",
    "kind": "Config",
    "preferences": {},
    "users": [
        {
            "name": "docker-desktop",
            "user": {
                "client-certificate-data": "<client-certificate-data>",
                "client-key-data": "<client-key-data>"
            }
        }
    ]
}
```

Then, run `astro dev start` with the Astro CLI to spin up a local Airflow environment. Once your environment has been created, open up the connection management UI, and create a new connection of the `Kubernetes Cluster Connection` type. Within the connection creation menu, copy the .json file you created using the above template into the `Kube config (JSON format)` field, and save the connection with the connection id `k8s_conn`. If you'd like to use another connection id, make sure to alter the following example DAG code.

#### Step 4: Run your container

To use the `KubernetesPodOperator`, you must define the configuration of each task and the Kubernetes Pod in which it runs, including its namespace and Docker image.

This example DAG runs a `hello-world` Docker image using the `k8s_conn` connection you defined in the previous step to run it on your local Kubernetes cluster.

```python wrap theme={null}
from pendulum import datetime
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
    KubernetesPodOperator,
)

with DAG(
    dag_id="example_kubernetes_pod",
    schedule="@once",
    start_date=datetime(2023, 3, 30),
) as dag:
    example_kpo = KubernetesPodOperator(
        kubernetes_conn_id="k8s_conn",
        image="hello-world",
        name="airflow-test-pod",
        task_id="task-one",
        is_delete_operator_pod=True,
        get_logs=True,
    )

    example_kpo
```

#### Step 4: View Kubernetes logs

(Optional) Use the `kubectl` command line tool to review the logs for any Pods that were created by the operator for issues and help with troubleshooting. If you haven't installed the `kubectl` command line tool, see [Install Tools](https://kubernetes.io/docs/tasks/tools/#kubectl).

<details>
  <summary>Windows And Mac</summary>

  Run `kubectl get pods -n $namespace` or `kubectl logs {pod_name} -n $namespace` to examine the logs for the Pod that just ran. By default, `docker-for-desktop` runs Pods in the `default` namespace.
</details>

<details>
  <summary>Linux</summary>

  Run `microk8s.kubectl get pods -n $namespace` or `microk8s.kubectl logs {pod_name} -n $namespace` to examine the logs for the pod that just ran. By default, `microk8s` runs pods in the `default` namespace.
</details>

## When to use the `KubernetesPodOperator`

The `KubernetesPodOperator` runs any Docker image provided to it. Frequent use cases are:

* Running a task in a language other than Python. This guide includes an example of how to run a Haskell script with the `KubernetesPodOperator`.
* Having full control over how much compute resources and memory a single task can use.
* Executing tasks in a separate environment with individual packages and dependencies.
* Running tasks that use a version of Python not supported by your Airflow environment.
* Running tasks with specific Node (a virtual or physical machine in Kubernetes) constraints, such as only running on Nodes located in the European Union.

### A comparison of the `KubernetesPodOperator` and the Kubernetes executor

[Executors](/docs/learn/airflow-executors-explained) determine how your Airflow tasks are executed. The Kubernetes executor and the `KubernetesPodOperator` both dynamically launch and terminate Pods to run Airflow tasks. As the name suggests, the Kubernetes executor affects how all tasks in an Airflow instance are executed. The `KubernetesPodOperator` launches only its own task in a Kubernetes Pod with its own configuration. It doesn't affect any other tasks in the Airflow instance. To configure the Kubernetes executor, see [Kubernetes Executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html).

The following are the primary differences between the `KubernetesPodOperator` and the Kubernetes executor:

* The `KubernetesPodOperator` requires a Docker image to be specified, while the Kubernetes executor doesn't.
* The `KubernetesPodOperator` defines one isolated Airflow task. In contrast, the Kubernetes executor is implemented at the configuration level of the Airflow instance, which means all tasks run in their own Kubernetes Pod. This might be desired in some use cases that require auto-scaling, but it's not ideal for environments with a high volume of shorter running tasks.
* In comparison to the `KubernetesPodOperator`, the Kubernetes executor has less abstraction over Pod configuration. All task-level configurations have to be passed to the executor as a dictionary using the `BaseOperator's` `executor_config` argument, which is available to all operators.
* If a custom Docker image is passed to the Kubernetes executor's `base` container by providing it to either the `pod_template_file` or the `pod_override` key in the dictionary for the `executor_config` argument, Airflow must be installed or the task won't run. A possible reason for customizing this Docker image would be to run a task in an environment with different versions of packages than other tasks running in your Airflow instance. This isn't the case with the `KubernetesPodOperator`, which can run any valid Docker image.

Both the `KubernetesPodOperator` and the Kubernetes executor can use the Kubernetes API to create Pods for running tasks. Typically, the `KubernetesPodOperator` is ideal for controlling the environment in which the task runs, while the Kubernetes executor is ideal for controlling resource optimization. It's common to use both the Kubernetes executor and the `KubernetesPodOperator` in the same Airflow environment, where all tasks need to run on Kubernetes but only some tasks require additional environment configurations.

## How to configure the `KubernetesPodOperator`

The `KubernetesPodOperator` launches any valid Docker image provided to it in a dedicated Kubernetes Pod on a Kubernetes cluster. The `KubernetesPodOperator` supports arguments for some of the most common Pod settings. For advanced use cases, you can specify a [Pod template file](https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates) that supports all possible Pod settings.

The `KubernetesPodOperator` can be instantiated like any other operator within the context of a DAG.

### Required arguments

* `task_id`: A unique string identifying the task within Airflow.
* `namespace`: The namespace within your Kubernetes cluster to which the new Pod is assigned.
* `name`: The name of the Pod being created. This name must be unique for each Pod within a namespace.
* `image`: The Docker image to launch. Images from [hub.docker.com](https://hub.docker.com/) can be passed with just the image name, but you must provide the full URL for custom repositories.

### Optional arguments

* `random_name_suffix`: Generates a random suffix for the Pod name if set to `True`. Avoids naming conflicts when running a large number of Pods.

* `labels`: A list of key and value pairs which can be used to logically group decoupled objects together.

* `ports`: Ports for the Pod.

* `reattach_on_restart`: Defines how to handle losing the worker while the Pod is running. When set to `True`, the existing Pod reattaches to the worker on the next try. When set to `False`, a new Pod will be created for each try. The default is `True`.

* `is_delete_operator_pod`: Determines whether to delete the Pod when it reaches its final state or when the execution is interrupted. The default is `True`.

* `get_logs`: Determines whether to use the `stdout` of the container as task-logs to the Airflow logging system.

* `log_events_on_failure`: Determines whether events are logged in case the Pod fails. The default is `False`.

* `env_vars`: A dictionary of environment variables for the Pod.

* `container_resources`: A [`k8s.V1ResourceRequirements`](https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1ResourceRequirements) object containing the resource requests and/or limits for the Pod.

  ```python wrap theme={null}
  # from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
  #     KubernetesPodOperator,
  # )
  # from kubernetes.client import CoreV1Api, V1Pod, models as k8s

  KubernetesPodOperator(
      # other arguments
      container_resources=k8s.V1ResourceRequirements(
          requests={"cpu": "100m", "memory": "64Mi", "ephemeral-storage": "1Gi"},
          limits={"cpu": "200m", "memory": "420Mi", "ephemeral-storage": "2Gi"},
      )
  )
  ```

  See the [Kubernetes Documentation on Resource Management for Pods and Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for more information.

  <Info>
    Astronomer customers can set default resource requests and limits for all KPO tasks in their deployment settings, see [Configure Kubernetes Pod resources](/docs/astro/deployment-resources#configure-kubernetes-pod-resources). Setting the `container_resources` argument in the KPO task will override the default settings. Note that using `ephemeral-storage` for Astro Hosted is currently in [Preview](/docs/astro/feature-previews).
  </Info>

* `volumes`: A list of `k8s.V1Volumes`, see also this [Kubernetes example DAG](https://github.com/apache/airflow/blob/providers-cncf-kubernetes/10.0.0/providers/tests/system/cncf/kubernetes/example_kubernetes.py).

* `affinity` and `tolerations`: Dictionaries of rules for [Pod to Node assignments](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/). Like the `volumes` parameter, these also require a `k8s` object.

* `pod_template_file`: The path to a Pod template file.

* `full_pod_spec`: A complete Pod configuration formatted as a Python `k8s` object.

You can also use many other arguments to configure the Pod and pass information to the Docker image. For a list of the available `KubernetesPodOperator` arguments, see the [`KubernetesPodOperator` source code](https://github.com/apache/airflow/blob/main/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py).

The following `KubernetesPodOperator` arguments can be used with Jinja templates: `image`, `cmds`, `arguments`, `env_vars`, `labels`, `config_file`, `pod_template_file`, and `namespace`.

### Configure a Kubernetes connection

If you leave `in_cluster=True`, you only need to specify the `KubernetesPodOperator`'s `namespace` argument to establish a connection with your Kubernetes cluster. The Pod specified by the `KubernetesPodOperator` runs on the same Kubernetes cluster as your Airflow instance.

If you aren't running Airflow on Kubernetes, or want to send the Pod to a different cluster than the one currently hosting your Airflow instance, you can create a Kubernetes Cluster [connection](/docs/learn/connections) which uses the [Kubernetes hook](https://airflow.apache.org/registry/providers/cncf-kubernetes#cncf-kubernetes-kubernetes-KubernetesHook) to connect to the [Kubernetes API](https://kubernetes.io/docs/reference/kubernetes-api/) of a different Kubernetes cluster. This connection can be passed to the `KubernetesPodOperator` using the `kubernetes_conn_id` argument and requires the following components to work:

* A `KubeConfig` file, provided as either a path to the file or in JSON format.
* The cluster context from the provided `KubeConfig` file.

The following image shows how to set up a Kubernetes cluster connection in the Airflow UI.

<Frame>
  <img alt="Kubernetes Cluster Connection" />
</Frame>

The components of the connection can also be set or overwritten at the task level by using the arguments `config_file` (to specify the path to the `KubeConfig` file) and `cluster_context`. Setting these parameters in `airflow.cfg` is deprecated.

<Info>
  **Launching Pods in external clusters**

  If some of your tasks require specific resources such as a GPU, you might want to run them in a different cluster than your Airflow instance.

  The way that you connect to an external cluster will vary based on where your cluster is hosted and where your Airflow environment is hosted, but generally the following conditions must be met to launch a Pod in an external cluster:

  * Your Airflow environment must have a network connection to the external cluster.
  * Your Airflow environment must have permissions to spin up Pods in the external cluster.
  * Your cluster configuration must be passed to your `KubernetesPodOperator` tasks either through a task-level configuration or a Kubernetes connection.

  See the [Astro documentation](/docs/astro/kubernetespodoperator) for a more detailed example of how to configure a `KubernetesPodOperator` task to launch a Pod in an external EKS cluster.
</Info>

## Use the `@task.kubernetes` decorator

The `@task.kubernetes` decorator provides an alternative to the traditional `KubernetesPodOperator` when you run Python scripts in a separate Kubernetes Pod. The Docker image provided to the `@task.kubernetes` decorator must support executing Python scripts.

Like regular `@task` decorated functions, XComs can be passed to the Python script running in the dedicated Kubernetes pod. If `do_xcom_push` is set to `True` in the decorator parameters, the value returned by the decorated function is pushed to XCom. You can learn more about decorators in the [Introduction to Airflow decorators](/docs/learn/airflow-decorators) guide.

Astronomer recommends using the `@task.kubernetes` decorator instead of the `KubernetesPodOperator` when using XCom with Python scripts in a dedicated Kubernetes pod.

```python expandable wrap theme={null}
from pendulum import datetime
from airflow.configuration import conf
from airflow.decorators import dag, task
import random

# get the current Kubernetes namespace Airflow is running in
namespace = conf.get("kubernetes", "NAMESPACE")


@dag(
    start_date=datetime(2023, 1, 1),
    catchup=False,
    schedule="@daily",
)
def kubernetes_decorator_example_dag():
    @task
    def extract_data():
        # simulating querying from a database
        data_point = random.randint(0, 100)
        return data_point

    @task.kubernetes(
        # specify the Docker image to launch, it needs to be able to run a Python script
        image="python",
        # launch the Pod on the same cluster as Airflow is running on
        in_cluster=True,
        # launch the Pod in the same namespace as Airflow is running in
        namespace=namespace,
        # Pod configuration
        # naming the Pod
        name="my_pod",
        # log stdout of the container as task logs
        get_logs=True,
        # log events in case of Pod failure
        log_events_on_failure=True,
        # enable pushing to XCom
        do_xcom_push=True,
    )
    def transform(data_point):
        multiplied_data_point = 23 * int(data_point)
        return multiplied_data_point

    @task
    def load_data(**context):
        # pull the XCom value that has been pushed by the KubernetesPodOperator
        transformed_data_point = context["ti"].xcom_pull(
            task_ids="transform", key="return_value"
        )
        print(transformed_data_point)

    load_data(transform(extract_data()))


kubernetes_decorator_example_dag()
```

## Example: Use the `KubernetesPodOperator` to run a script in another language

A frequent use case for the `KubernetesPodOperator` is running a task in a language other than Python. To do this, you build a custom Docker image containing the script.

In the following example, the Haskell script runs and the value `NAME_TO_GREET` is printed on the console:

```haskell wrap theme={null}
import System.Environment

main = do
        name <- getEnv "NAME_TO_GREET"
        putStrLn ("Hello, " ++ name)
```

The Dockerfile creates the necessary environment to run the script and then executes it with a `CMD` command:

```docker wrap theme={null}
FROM haskell
WORKDIR /opt/hello_name
RUN cabal update
COPY ./haskell_example.cabal /opt/hello_name/haskell_example.cabal
RUN cabal build --only-dependencies -j4
COPY . /opt/hello_name
RUN cabal install
CMD ["haskell_example"]
```

After making the Docker image available, it can be run from the `KubernetesPodOperator` with the `image` argument. The following example DAG showcases a variety of arguments of the `KubernetesPodOperator`, including how to pass `NAME_TO_GREET` to the Haskell code.

```python expandable wrap theme={null}
from airflow import DAG
from pendulum import datetime
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
    KubernetesPodOperator,
)
from airflow.configuration import conf

# get the current Kubernetes namespace Airflow is running in
namespace = conf.get("kubernetes", "NAMESPACE")

# set the name that will be printed
name = "your_name"

# instantiate the DAG
with DAG(
    start_date=datetime(2022, 6, 1),
    catchup=False,
    schedule="@daily",
    dag_id="KPO_different_language_example_dag",
) as dag:
    say_hello_name_in_haskell = KubernetesPodOperator(
        # unique id of the task within the DAG
        task_id="say_hello_name_in_haskell",
        # the Docker image to launch
        image="<image location>",
        # launch the Pod on the same cluster as Airflow is running on
        in_cluster=True,
        # launch the Pod in the same namespace as Airflow is running in
        namespace=namespace,
        # Pod configuration
        # name the Pod
        name="my_pod",
        # give the Pod name a random suffix, ensure uniqueness in the namespace
        random_name_suffix=True,
        # attach labels to the Pod, can be used for grouping
        labels={"app": "backend", "env": "dev"},
        # reattach to worker instead of creating a new Pod on worker failure
        reattach_on_restart=True,
        # delete Pod after the task is finished
        is_delete_operator_pod=True,
        # get log stdout of the container as task logs
        get_logs=True,
        # log events in case of Pod failure
        log_events_on_failure=True,
        # pass your name as an environment var
        env_vars={"NAME_TO_GREET": f"{name}"},
    )
```

## Example: Use the `KubernetesPodOperator` with XComs

[XCom](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html) is a commonly used Airflow feature for passing small amounts of data between tasks. You can use the `KubernetesPodOperator` to both receive values stored in XCom and push values to XCom.

The following example DAG shows an ETL pipeline with an `extract_data` task that runs a query on a database and returns a value. The [TaskFlow API](https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html#tutorial-on-the-taskflow-api) automatically pushes the return value to XComs.

The `transform` task is a `KubernetesPodOperator` which requires that the XCom data is pushed from the upstream task before it, and then launches an image created with the following Dockerfile:

```docker wrap theme={null}
FROM python

WORKDIR /

# creating the file to write XComs to
RUN mkdir -p airflow/xcom
RUN echo "" > airflow/xcom/return.json

COPY multiply_by_23.py ./

CMD ["python", "./multiply_by_23.py"]
```

When using XComs with the `KubernetesPodOperator`, you must create the file `airflow/xcom/return.json` in your Docker container (ideally from within your Dockerfile), because Airflow can only look for XComs to pull at that specific location. In the following example, the Docker image contains a simple Python script to multiply an environment variable by 23, package the result into JSON, and then write that JSON to the correct file to be retrieved as an XCom. The XComs from the `KubernetesPodOperator` are pushed only if the task is marked successful.

```python wrap theme={null}
import os

# import the result of the previous task as an environment variable
data_point = os.environ["DATA_POINT"]

# multiply the data point by 23 and package the result into a json
multiplied_data_point = str(23 * int(data_point))
return_json = {"return_value": f"{multiplied_data_point}"}

# write to the file checked by Airflow for XComs
f = open("./airflow/xcom/return.json", "w")
f.write(f"{return_json}")
f.close()
```

The `load_data` task pulls the XCom returned from the `transform` task and prints it to the console.

The full DAG code is provided in the following example. To avoid task failure, turn on `do_xcom_push` after you create the `airflow/xcom/return.json` within the Docker container run by the `KubernetesPodOperator`.

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  from pendulum import datetime
  from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
      KubernetesPodOperator,
  )
  from airflow.configuration import conf
  from airflow.decorators import dag, task

  import random

  # get the current Kubernetes namespace Airflow is running in
  namespace = conf.get("kubernetes", "NAMESPACE")


  # instantiate the DAG
  @dag(
      start_date=datetime(2022, 6, 1),
      catchup=False,
      schedule="@daily",
  )
  def KPO_XComs_example_dag():
      @task
      def extract_data():
          # simulating querying from a database
          data_point = random.randint(0, 100)
          return data_point

      transform = KubernetesPodOperator(
          # set task id
          task_id="transform",
          # specify the Docker image to launch
          image="<image location>",
          # launch the Pod on the same cluster as Airflow is running on
          in_cluster=True,
          # launch the Pod in the same namespace as Airflow is running in
          namespace=namespace,
          # Pod configuration
          # naming the Pod
          name="my_pod",
          # log stdout of the container as task logs
          get_logs=True,
          # log events in case of Pod failure
          log_events_on_failure=True,
          # pull a variable from XComs using Jinja templating and provide it
          # to the Pod as an environment variable
          env_vars={
              "DATA_POINT": """{{ ti.xcom_pull(task_ids='extract_data',
                                                   key='return_value') }}"""
          },
          # push the contents from xcom.json to Xcoms. Remember to only set this
          # argument to True if you have created the `airflow/xcom/return.json`
          # file within the Docker container run by the KubernetesPodOperator.
          do_xcom_push=True,
      )

      @task
      def load_data(**context):
          # pull the XCom value that has been pushed by the KubernetesPodOperator
          transformed_data_point = context["ti"].xcom_pull(
              task_ids="transform", key="return_value"
          )
          print(transformed_data_point)

      # set dependencies (tasks defined using Decorators need to be called)
      extract_data() >> transform >> load_data()


  KPO_XComs_example_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  from airflow import DAG
  from pendulum import datetime
  from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
      KubernetesPodOperator,
  )
  from airflow.configuration import conf
  from airflow.operators.python import PythonOperator

  import random

  # get the current Kubernetes namespace Airflow is running in
  namespace = conf.get("kubernetes", "NAMESPACE")


  def extract_data_function():
      # simulating querying from a database
      data_point = random.randint(0, 100)
      return data_point


  def load_data_function(**context):
      # pull the XCom value that has been pushed by the KubernetesPodOperator
      transformed_data_point = context["ti"].xcom_pull(
          task_ids="transform", key="return_value"
      )
      print(transformed_data_point)


  # instantiate the DAG
  with DAG(
      dag_id="KPO_XComs_example_dag",
      start_date=datetime(2022, 6, 1),
      catchup=False,
      schedule="@daily",
  ):
      extract_data = PythonOperator(
          task_id="extract_data", python_callable=extract_data_function
      )

      transform = KubernetesPodOperator(
          # set task id
          task_id="transform",
          # specify the Docker image to launch
          image="<image location>",
          # launch the Pod on the same cluster as Airflow is running on
          in_cluster=True,
          # launch the Pod in the same namespace as Airflow is running in
          namespace=namespace,
          # Pod configuration
          # naming the Pod
          name="my_pod",
          # log stdout of the container as task logs
          get_logs=True,
          # log events in case of Pod failure
          log_events_on_failure=True,
          # pull a variable from XComs using Jinja templating and provide it
          # to the Pod as an environment variable
          env_vars={
              "DATA_POINT": """{{ ti.xcom_pull(task_ids='extract_data',
                                                   key='return_value') }}"""
          },
          # push the contents from xcom.json to Xcoms. Remember to only set this
          # argument to True if you have created the `airflow/xcom/return.json`
          # file within the Docker container run by the KubernetesPodOperator.
          do_xcom_push=True,
      )

      load_data = PythonOperator(
          task_id="load_data",
          python_callable=load_data_function,
      )

      # set dependencies (tasks defined using Decorators need to be called)
      extract_data >> transform >> load_data
  ```
</details>


# Data engineering with AI
Source: https://astronomer.io/docs/learn/local-data-engineering-with-ai-overview

Learn how to do local Airflow data engineering with AI tools.

Running AI agents for local engineering work is, in essence, a context engineering problem. The prompt, your local files, installed skills, available tools, and the output of every command the agent runs all become context, and the code the agent writes is only as good as the information you give it access to.

This Learn section covers the basics of using AI to write [Apache Airflow®](https://airflow.apache.org/) pipelines:

* **[Foundational concepts and terms](#basic-concepts-in-agentic-engineering)**: The core terms in agentic data engineering, plus what to know about [safety](#safety) and [cost](#cost) of using AI agents.
* **[Run Airflow locally](/docs/learn/run-airflow-locally)**: How to use the Astro CLI to spin up a local Airflow project so your agent can run Dags to test them in a realistic environment.
* **[IDE setup for data engineering](/docs/learn/set-up-your-ide-for-data-engineering)**: How to attach your code editor to the same container your Dags run in, so autocomplete, type checking, and debugging match the Airflow and provider versions in your Airflow project.
* **[AI context for data engineering](/docs/learn/ai-context-for-data-engineering)**: How to give your AI agents access to advanced knowledge about Airflow to write better Dags.
* **[Develop Dags with AI](/docs/learn/develop-dags-with-ai)**: How to write instructions for AI agents that allow them to iteratively develop Airflow pipelines.
* **[Debug Dags with AI](/docs/learn/debug-dags-with-ai)**: How to systematically debug failed tasks, Dags, and other Airflow issues with AI.

<Tip>
  [Otto](/docs/astro/otto-overview), Astronomer's data engineering agent, has advanced capabilities for local data engineering with a focus on writing Dags, debugging, and upgrading Airflow.

  To work with Otto locally, install the [Astro CLI](/docs/cli/v1.45/overview), sign in to your Astro account with `astro login` (a [free trial](https://www.astronomer.io/lp/signup/) is available), and then run `astro otto`.
</Tip>

<Info>
  For a hands-on demo of using AI agents in local data engineering, watch the recording of the [Local data engineering in the agentic era](https://www.astronomer.io/events/webinars/local-data-engineering-in-the-agentic-era-video/) webinar.
</Info>

## The mental shift from typing code to orchestrating agents

The way people get machines to do what they want has moved through increasingly abstract layers: machine code, assembly, high-level languages, and now natural language prompts. The move from typing code to directing an AI is a bigger jump than the ones before it. Every earlier layer still had you writing deterministic instructions. An AI model generates those deterministic instructions through a non-deterministic process.

Developers adopting AI for code often follow a similar progression: first copying snippets out of a chat based AI application, then working with an assistant inline while coding, then directing a single agent, then coordinating several at once.

Each step moves the work away from writing code by hand and toward two other jobs: [engineering the agent's context](/docs/learn/ai-context-for-data-engineering) and [writing a spec for the outcome you want](/docs/learn/develop-dags-with-ai). Deciding what to build in the first place, and whether it's worth building at all, is still your job.

## Basic concepts in agentic engineering

There are several core concepts you should understand when using AI in your local engineering processes:

* **Model**: The LLM (large language model) itself. Your model runs either in the cloud (for example, Claude Opus 4.8) or on your computer (for example, a Llama variant). The model takes in input (context) and produces output.
* **Harness**: The code around the model that decides what to send to it as input and how to handle different outputs. The harness determines which files are read automatically, which tools an AI has access to, and which AI outputs require human approval. The model running in the harness, plus all available tools, is your AI agent.
* **Interface**: How you interact with your AI agent. Most developers use a terminal interface, an integrated development environment (IDE), or a combination of both.
* **Context window**: Everything the model receives as input for a single call, including tool descriptions, conversation history, and any files or command output the agent pulled in. It has a fixed maximum size.

Information (context) given to your AI agent comes in several forms:

* **Auto-loaded context**: Most agent harnesses automatically load some files into the context window, for example [`AGENTS.md`](https://agents.md/) files.
* **Prompt**: What you directly type and paste into your AI agent interface, plus what the AI harness decides to send with your input, for example the conversation history.
* **Skills**: Step-by-step instructions or workflows the agent can load on explicit demand or automatically based on keywords in the prompt. Skills are Markdown files with structured frontmatter. Astronomer maintains the [`astronomer/agents` skills](https://github.com/astronomer/agents), a set of skills covering Airflow and data engineering best practices.
* **Available MCP servers**: Structured interfaces to other systems. An MCP server exposes a set of tools the agent can call, for example your data warehouse's MCP. The [`astro-airflow-mcp` server](/docs/astro/astro-mcp-server#airflow-mcp-plugin) exists for Airflow Deployments running on Astro. Tool descriptions from MCP servers are part of context the moment the server connects, so connected MCP servers cost tokens even when you don't use their tools.
* **Available tools**: Any script can be a tool. Both the tool description and the tool output can be part of AI context. Tools are what let an agent act on your computer instead of only producing text, which also makes them where most of the risk sits. See [Safety](#safety).

## Safety

A local agent running on your computer can be a safety issue if left running without supervision.

* **Tools can be destructive.** As soon as an agent has access to a tool and no approval step, it can use that tool. An agent with unrestricted Bash access can wipe your entire hard drive while it thinks it's deleting files inside a container. Restrict which commands your agent can run without asking. See [Restrict agent commands](/docs/learn/run-airflow-locally#restrict-agent-commands) for how to set those restrictions.
* **Incoming context can be wrong, or even deliberately malicious.** Anything your agent reads can act as an instruction rather than as data. A web page can tell your agent to ignore what you asked and hand over your credentials, and the agent has no reliable way to separate that from a legitimate request. Treat anything retrieved from the web or another external source as untrusted.
* **Anything in the context window can leave it.** Credentials, connection strings, and query results all become part of context the moment the agent reads them, and an agent that can reach the internet can send them elsewhere, be that by accident or because of a malicious prompt injection on a website.
* **Your Agent might spill your secrets.**. For that reason you shouldn't give your agents plain credentials and rather sign into to a CLI yourself and let the agent run commands through the session you authenticated. Where the system supports it, give your agent its own account with restricted permissions. Otto works this way: you run `astro login` yourself, and Otto uses that authenticated session to interact with Astro Deployments.

To guard against these issues you have three options:

* **Restrict permissions**: Allow only specific CLI sub-commands, and deny unrestricted web access entirely.
* **Require human review** for potentially destructive commands and actions.
* **Run in a sandboxed environment**: Running agents in an environment separate from your local machine allows you to be more permissive, for example allowing file deletion, as long as deletion is only possible within the sandbox.

What you can't do is enforce a rule by writing it into a prompt. Models are non-deterministic and might not follow your instructions, and compaction can drop the instruction from context partway through a long session.

<Info>
  For more on what can go wrong with AI agents, see the ContextOops chapter in the [AI Context Engineering with Apache Airflow®](https://www.astronomer.io/ebooks/ai-context-engineering-with-apache-airflow/) eBook.
</Info>

## Cost

Agentic loops can read and write a lot of tokens, which can get expensive over time. A few strategies to lower cost are:

* **Choose the right model for the task.** Use a more capable model for planning and for problems that need reasoning across a whole project. Smaller, cheaper models handle well-scoped individual tasks.
* **You pay for what you send, as well as for what the model writes.** Skills, rules files, MCP tool descriptions, and pasted documentation all form part of the input on every turn. Keep your skills and instructions short and specific.
* **Avoid paying to generate the same code twice.** If your agent keeps writing the same script, move that script into your project so the agent can call it instead of regenerating it each time.
* **Use a Dag-as-a-tool.** Many skills are step-based workflows with some configuration. If you often need the same workflow, move it into an Airflow Dag and have your agent trigger that Dag and use its output. Running a *Dag-as-a-tool* is more reproducible, more robust, and cheaper, because the pipeline keeps the context needed for each step small, and deterministic steps can be accomplished using deterministic code. See [Skills vs. Pipeline: Two Ways to Build the Same AI Workflow](https://medium.com/apache-airflow/skills-vs-pipeline-two-ways-to-build-the-same-ai-workflow-b5dd7088e122) by Vikram Koka for more information on skills as Airflow Dags.


# Airflow logging
Source: https://astronomer.io/docs/learn/logging

An introduction to Airflow logging.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

Airflow provides an extensive logging system for monitoring and debugging your data pipelines. Your webserver, scheduler, metadata database, and individual tasks all generate logs. You can export these logs to a local file, your console, or to a specific remote storage solution.

In this guide, you'll learn the basics of Airflow logging, including:

* Where to find logs for different Airflow components.
* How to add custom task logs from within a DAG.
* When and how to configure logging settings.
* How to set up remote logging in OSS Airflow.

In addition to standard logging, Airflow provides observability features that you can use to collect metrics, trigger callback functions with task events, monitor Airflow health status, and track errors and user activity. For more information, see:

* [Deployment metrics](/docs/astro/deployment-metrics) for Astro customers to use built-in features providing you with detailed metrics about how your tasks run and use resources in your cloud.
* [Metrics Configuration](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html) for monitoring options for self-hosted Airflow including StatsD and OpenTelemetry.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Airflow core components. See [Airflow's components](/docs/learn/airflow-components).

## Airflow logging

Logging in Airflow leverages the [Python stdlib `logging` module](https://docs.python.org/3/library/logging.html). The `logging` module includes the following classes:

* Loggers (`logging.Logger`): The interface that the application code directly interacts with. Airflow defines 4 loggers by default: `root`, `flask_appbuilder`, `airflow.processor` and `airflow.task`.
* Handlers (`logging.Handler`): Send log records to their destination. By default, Airflow uses `RedirectStdHandler`, `FileProcessorHandler` and `FileTaskHandler`.
* Filters (`logging.Filter`): Determine which log records are emitted. Airflow uses `SecretsMasker` as a filter to prevent sensitive information from being printed into logs.
* Formatters (`logging.Formatter`): Determine the layout of log records. Two formatters are predefined in Airflow:
  * `airflow_colored`: `"[%(blue)s%(asctime)s%(reset)s] {%(blue)s%(filename)s:%(reset)s%(lineno)d} %(log_color)s%(levelname)s%(reset)s - %(log_color)s%(message)s%(reset)s"` (this formatting relates to [colored logs in a TTY terminal](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#colored-log-format))
  * `airflow`: `"[%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s"` (this format is used to display logs in the Airflow UI)

See [Logging facility for Python](https://docs.python.org/3/library/logging.html) for more information on the methods available for these classes, including the attributes of a LogRecord object and the 6 available levels of logging severity (`CRITICAL`, `ERROR`, `WARNING`, `INFO`, `DEBUG`, and `NOTSET`).

The four default loggers in Airflow each have a handler with a predefined log destination and formatter:

* `root` (level: `INFO`): Uses `RedirectStdHandler` and `airflow_colored`. It outputs to `sys.stderr/stout` and acts as a catch-all for processes that have no specific logger defined.
* `flask_appbuilder` (level: `WARNING`): Uses `RedirectStdHandler` and `airflow_colored`. It outputs to `sys.stderr/stout`. It handles logs from the webserver.
* `airflow.processor` (level: `INFO`): Uses `FileProcessorHandler` and `airflow`. It writes logs from the scheduler to the local file system.
* `airflow.task` (level: `INFO`): Uses `FileTaskHandlers` and `airflow`. It writes task logs to the local file system.

By default, log file names have the following format:

* For standard tasks: `dag_id={dag_id}/run_id={run_id}/task_id={task_id}/attempt={try_number}.log`
* For [dynamically mapped tasks](/docs/learn/dynamic-tasks): `dag_id={dag_id}/run_id={run_id}/task_id={task_id}/map_index={map_index}/attempt={try_number}.log`

These filename formats can be reconfigured using `log_filename_template` in `airflow.cfg`.

You can view the full default logging configuration under `DEFAULT_LOGGING_CONFIG` in the [Airflow source code](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/config_templates/airflow_local_settings.py).

The Airflow UI shows logs using a `read()` method on task handlers that isn't part of stdlib. `read()` checks for available logs and displays them in a predefined order:

* Remote logs (if remote logging is enabled)
* Logs on the local filesystem
* Logs from [worker specific webserver subprocesses](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#worker-log-server-port)

When using the Kubernetes Executor and a worker pod still exists, `read()` shows the first 100 lines from the Kubernetes pod logs. If a worker pod spins down, the logs are no longer available. For more information, see [Logging Architecture](https://kubernetes.io/docs/concepts/cluster-administration/logging/).

## Log locations

By default, Airflow outputs logs to the `base_log_folder` configured in `airflow.cfg`, which is located in your `$AIRFLOW_HOME` directory.

Airflow also makes event logs/audit logs available. For example, listing which user triggered a DAG run.

You can view the full cluster Audit log containing all audit logs for your Airflow instance under the "Browse" tab.

<Frame>
  <img alt="Audit Logs List View in the Airflow UI." />
</Frame>

Audit logs for individual DAGs, DAG runs and tasks can be found in the `Event Log` tab after selecting the DAG, DAG run or task. This view offers advanced filtering options such as filtering for time periods or including/excluding types of events.

<Frame>
  <img alt="Event Logs for a DAG in the Airflow UI." />
</Frame>

For details about the log levels and events tracked, see [Audit logs in Airflow](https://airflow.apache.org/docs/apache-airflow/stable/security/audit_logs.html).

### Local Airflow environment

If you run Airflow locally, logging information is accessible in the following locations:

* Scheduler: Logs are printed to the console and accessible in `$AIRFLOW_HOME/logs/scheduler`.
* Webserver and Triggerer: Logs are printed to the console. Individual triggers' log messages can be found in the logs of tasks that use deferrable operators.
* Task: Logs can be viewed in the Airflow UI or at `$AIRFLOW_HOME/logs/`. To view task logs directly in your terminal, run `astro dev run tasks test <dag_id> <task_id>` with the [Astro CLI](/docs/cli/v1.43/overview) or `airflow tasks test <dag_id> <task_id>` if you are running Airflow with other tools. Note that as of Airflow 2.9, `Pre task execution logs` and `Post task execution logs` are grouped in the Airflow UI and need to be expanded to view the full logs, see [Airflow task log customization](#airflow-task-log-customization).
* Metadata database: Logs are handled differently depending on which database you use.

### Containerized Airflow environment

If you run Airflow in containers using the [Astro CLI](/docs/cli/v1.43/install-cli), you can find the logs for each Airflow component in the following locations:

* Scheduler: Logs are in `/usr/local/airflow/logs/scheduler` within the scheduler container by default. To enter a container in a bash session, run `podman exec -it <container_id> /bin/bash` (or `docker exec -it <container_id> /bin/bash` if you are using Docker).
* Webserver: Logs appear in the console by default. You can access the logs by running `podman logs <webserver_container_id>`.
* Metadata database: Logs appear in the console by default. You can access the logs by running `podman logs <postgres_container_id>`.
* Triggerer: Logs appear in the console by default. You can access the logs by running `podman logs <triggerer_container_id>`. Individual triggers' log messages can be found in the logs of tasks that use deferrable operators.
* Task: Logs appear in `/usr/local/airflow/logs/` within the scheduler container. To access task logs in the Airflow UI, click the square of a task instance in the **Grid** view and then select the **Logs** tab.

<Frame>
  <img alt="Logs in Grid View" />
</Frame>

The Astro CLI includes a command to show webserver, scheduler, triggerer and Celery worker logs from the local Airflow environment. For more information, see [astro dev logs](/docs/cli/v1.43/astro-dev-logs).

<Info>
  In newer Airflow versions, logs from other Airflow components, such as the scheduler (2.8+) or executor (2.10+), will be forwarded to the task logs if an error in the component causes the task to fail. For example, if a task runs out of memory, causing a [zombie process](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#zombie-undead-tasks), information about the zombie is printed to the task logs. You can disable this behavior by setting [`AIRFLOW__LOGGING__ENABLE_TASK_CONTEXT_LOGGER=False`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#enable-task-context-logger).
</Info>

## Add custom task logs from a DAG

All hooks and operators in Airflow generate logs when a task is run. You can't modify logs from within other operators or in the top-level code, but you can add custom logging statements from within your Python functions by accessing the `airflow.task` logger.

The advantage of using a logger over print statements is that you can log at different levels and control which logs are emitted to a specific location. For example, by default the `airflow.task` logger is set at the level of `INFO`, which means that logs at the level of `DEBUG` aren't logged. To see `DEBUG` logs when debugging your Python tasks, you need to set `AIRFLOW__LOGGING__LOGGING_LEVEL=DEBUG` or change the value of `logging_level` in `airflow.cfg`. After debugging, you can change the `logging_level` back to `INFO` without modifying your DAG code.

The following example DAG shows how to instantiate an object using the existing `airflow.task` logger, how to add logging statements of different severity levels from within a Python function, and what the log output would be with default Airflow logging settings. There are many use cases for adding additional logging statements from within DAGs, ranging from logging warnings when a specific set of conditions appear over additional debugging messages to catching exceptions but still keeping a record of their having occurred.

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  from pendulum import datetime, duration
  from airflow.decorators import dag, task
  from airflow.operators.bash import BashOperator

  # import the logging module
  import logging

  # get the airflow.task logger
  task_logger = logging.getLogger("airflow.task")


  @task
  def extract():
      # with default airflow logging settings, DEBUG logs are ignored
      task_logger.debug("This log is at the level of DEBUG")

      # each of these lines produces a log statement
      print("This log is created with a print statement")
      task_logger.info("This log is informational")
      task_logger.warning("This log is a warning")
      task_logger.error("This log shows an error!")
      task_logger.critical("This log shows a critical error!")

      data = {"a": 19, "b": 23, "c": 42}

      # Using the Task flow API to push to XCom by returning a value
      return data


  # logs outside of tasks will not be processed
  task_logger.warning("This log will not show up!")

  # command to create a file and write the data from the extract task into it
  # these commands use Jinja templating within {{}}
  commands = """
      touch /usr/local/airflow/{{ds}}.txt
      echo {{ti.xcom_pull(task_ids='extract')}} > /usr/local/airflow/{{ds}}.txt
      """


  @dag(
      start_date=datetime(2022, 6, 5),
      schedule="@daily",
      dagrun_timeout=duration(minutes=10),
      catchup=False,
  )
  def more_logs_dag():
      write_to_file = BashOperator(task_id="write_to_file", bash_command=commands)

      # logs outside of tasks will not be processed
      task_logger.warning("This log will not show up!")

      extract() >> write_to_file


  more_logs_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  from pendulum import datetime, duration
  from airflow import DAG
  from airflow.operators.python import PythonOperator
  from airflow.operators.bash import BashOperator

  # import the logging module
  import logging

  # get the airflow.task logger
  task_logger = logging.getLogger("airflow.task")


  def extract_function():
      # with default airflow logging settings, DEBUG logs are ignored
      task_logger.debug("This log is at the level of DEBUG")

      # each of these lines produces a log statement
      print("This log is created with a print statement")
      task_logger.info("This log is informational")
      task_logger.warning("This log is a warning")
      task_logger.error("This log shows an error!")
      task_logger.critical("This log shows a critical error!")

      data = {"a": 19, "b": 23, "c": 42}

      # Using the Task flow API to push to XCom by returning a value
      return data


  # logs outside of tasks will not be processed
  task_logger.warning("This log will not show up!")

  # command to create a file and write the data from the extract task into it
  # these commands use Jinja templating within {{}}
  commands = """
      touch /usr/local/airflow/{{ds}}.txt
      echo {{ti.xcom_pull(task_ids='extract')}} > /usr/local/airflow/{{ds}}.txt
      """

  with DAG(
      dag_id="more_logs_dag",
      start_date=datetime(2022, 6, 5),
      schedule="@daily",
      dagrun_timeout=duration(minutes=10),
      catchup=False,
  ) as dag:
      extract = PythonOperator(task_id="extract", python_callable=extract_function)

      write_to_file = BashOperator(task_id="write_to_file", bash_command=commands)

      # logs outside of tasks will not be processed
      task_logger.warning("This log will not show up!")

      extract >> write_to_file
  ```
</details>

For the previous DAG, the logs for the `extract` task show the following lines under the default Airflow logging configuration (set at the level of `INFO`):

```bash wrap theme={null}
[2022-06-06, 07:25:09 UTC] {logging_mixin.py:115} INFO - This log is created with a print statement
[2022-06-06, 07:25:09 UTC] {more_logs_dag.py:15} INFO - This log is informational
[2022-06-06, 07:25:09 UTC] {more_logs_dag.py:16} WARNING - This log is a warning
[2022-06-06, 07:25:09 UTC] {more_logs_dag.py:17} ERROR - This log shows an error!
[2022-06-06, 07:25:09 UTC] {more_logs_dag.py:18} CRITICAL - This log shows a critical error!
```

### Airflow task log customization

Recent Airflow versions added options to customize task logs displayed in the UI by allowing log groups (Airflow 2.9) and to use keywords to turn lines red or yellow (Airflow 2.10).

By default, Airflow groups all `Pre task execution logs` as well as all `Post task execution logs` in the Airflow UI. To see the full logs, click the triangle to expand the log groups.
To add your own log groups, use the `::group::` syntax:

```text wrap theme={null}
t_log.info("::group::<log group name>")
t_log.info("<log in log group>")
t_log.info("::endgroup::")
```

The task below creates one new log group containing one hidden log line:

```python wrap theme={null}
# from airflow.decorators import dag, task
# import logging

t_log = logging.getLogger("airflow.task")

@task
def log_groups():
    t_log.info("I'm a log that is always shown.")
    t_log.info("::group::My log group!")
    t_log.info("hi! I'm a hidden log! :)")
    t_log.info("::endgroup::")
    t_log.info("I'm not hidden either.")

log_groups()

```

In the Airflow UI you can collapse and expand this log group:

<Frame>
  <img alt="Gif showing collapsing and expanding of the 2 default task log groups and the custom My log group! group in the Airflow UI" />
</Frame>

Airflow 2.10 added log line highlighting based on keywords in the Airflow UI.

By default, any lines that are logged with the level `error` or contain the words `error` or `exception` are logged in red. Any lines emitted with the level `warn` or containing the word `warn` are yellow. You can change this behavior by changing the [`AIRFLOW__LOGGING__COLOR_LOG_ERROR_KEYWORDS`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#color-log-error-keywords) and [`AIRFLOW__LOGGING__COLOR_LOG_WARNING_KEYWORDS`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#color-log-warning-keywords) Airflow configurations.

For example, after setting `AIRFLOW__LOGGING__COLOR_LOG_ERROR_KEYWORDS=error,exception,important` in an Airflow environment the task below logs 4 lines in red and 2 lines in yellow:

```python wrap theme={null}
@task
def log_coloring():
    t_log.info("If I say 'error' the log line turns red!")
    t_log.info("I warn you, this line turns yellow!")
    t_log.info("By changing the keywords in the config I can make this important line turn red as well!")
    print("Also works for regular but important print statements.")
    print("This is a normal print statement.")

    t_log.error("The log-level also affects the output, in this case turning it red.")
    t_log.warn("This log-level turns the line yellow.")
    t_log.critical("This line is not red. But can be made red by adding 'critical' to the keyword list.")
```

<Frame>
  <img alt="Screenshot of the task logs in the Airflow UI showing the above lines in red and yellow" />
</Frame>

For color customization of logs sent to a TTY terminal, see the [Airflow configuration reference](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#colored-log-format).

## When to configure logging

Logging in Airflow is ready to use without any additional configuration. However, there are many use cases where customization of logging is beneficial. For example:

* Changing the format of existing logs to contain additional information: for example, the full pathname of the source file from which the logging call was made.
* Adding additional handlers: for example, to log all critical errors in a separate file.
* Storing logs remotely.
* Adding your own custom handlers: for example, to log remotely to a destination not yet supported by existing providers.

## How to configure logging

<Info>
  Options for configuring logging on Astro differ from those described in this section. For guidance specific to Astro, see: [View Airflow component and task logs for a Deployment](/docs/astro/view-logs).
</Info>

Logging in Airflow can be configured in `airflow.cfg` or by providing a custom `log_config.py` file. It is best practice not to declare configs or variables within the `.py` handler files except for testing or debugging purposes.

In the Airflow CLI, run the following commands to return the current task handler and logging configuration. If you're running Airflow in containers, make sure to enter your scheduler container before running the commands:

```console wrap theme={null}
airflow info          # shows the current handler
airflow config list   # shows current parameters under [logging]
```

A full list of parameters relating to logging that can be configured in `airflow.cfg` can be found in the [`base_log_folder`](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#logging) reference. They can also be configured by setting their corresponding environment variables.

For example, to change your logging level from the default `INFO` to `LogRecords` with a level of `ERROR` or above, you set `logging_level = ERROR` in `airflow.cfg` or define an environment variable `AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR`.

[Advanced configuration](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html#custom-logger-for-operators-hooks-and-tasks) might necessitate the logging config class to be overwritten. To enable custom logging, you need to create the configuration file `~/airflow/config/log_config.py` and specify your modifications to `DEFAULT_LOGGING_CONFIG`. You might need to do this to add a custom handler.

## Remote logging

<Info>
  Options for configuring remote logging on Astro differ from those described in this section. For guidance specific to Astro, see:

  * [Export metrics and logs to Datadog](/docs/astro/export-datadog).
  * [Export logs to AWS CloudWatch](/docs/astro/export-cloudwatch).
</Info>

When scaling your Airflow environment, you might produce more logs than your Airflow environment can store. In this case, you need reliable, resilient, and auto-scaling storage. The easiest solution is to use remote logging to a remote service which is already supported by the following community-managed providers:

* Alibaba: `OSSTaskHandler` (`oss://`)
* Amazon: `S3TaskHandler` (`s3://`), `CloudwatchTaskHandler` (`cloudwatch://`)
* [Elasticsearch](https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/stable/logging/index.html): `ElasticsearchTaskHandler` (further configured with `elasticsearch` in `airflow.cfg`)
* [Google](https://airflow.apache.org/docs/apache-airflow-providers-google/stable/logging/index.html): `GCSTaskHandler` (`gs://`), `StackdriverTaskHandler` (`stackdriver://`)
* [Microsoft Azure](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/logging/index.html): `WasbTaskHandler` (`wasb`)

By configuring `REMOTE_BASE_LOG_FOLDER` with the prefix of a supported provider, you can override the default task handler (`FileTaskHandler`) to send logs to a remote destination task handler. For example, `GCSTaskHandler`.

If you want different behavior or to add several handlers to one logger, you need to make changes to `DEFAULT_LOGGING_CONFIG`.

Logs are sent to remote storage only once a task has been completed or failed. This means that logs of currently running tasks are accessible only from your local Airflow environment.

## Remote logging example: Send task logs to Amazon S3

<Info>
  Options for configuring remote logging on Astro differ from those described in this section. For guidance specific to Astro, see:

  * [Export metrics and logs to Datadog](/docs/astro/export-datadog)
  * [Export logs to AWS CloudWatch](/docs/astro/export-cloudwatch)
  * [Export logs to a Secondary S3 Bucket](/docs/astro/export-secondary-s3-bucket)
  * [Export logs to a Secondary WASB Container](/docs/astro/export-secondary-wasb)
</Info>

1. Add the `apache-airflow-providers-amazon` provider package to `requirements.txt`.

2. Start your Airflow environment and go to **Admin** > **Connections** in the Airflow UI.

3. Create a connection of type **Amazon S3** and set `login` to your AWS access key ID and `password` to your AWS secret access key. See [AWS Account and Access Keys](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) for information about retrieving your AWS access key ID and AWS secret access key.

4. Add the following commands to the Dockerfile. Include the double underscores around `LOGGING`:

   ```docker wrap theme={null}
   # allow remote logging and provide a connection ID (see step 2)
   ENV AIRFLOW__LOGGING__REMOTE_LOGGING=True
   ENV AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID=${AMAZONS3_CON_ID}

   # specify the location of your remote logs using your bucket name
   ENV AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=s3://${S3BUCKET_NAME}/logs

   # optional: serverside encryption for S3 logs
   ENV AIRFLOW__LOGGING__ENCRYPT_S3_LOGS=True
   ```

   These environment variables configure remote logging to one S3 bucket (`S3BUCKET_NAME`). Behind the scenes, Airflow uses these configurations to create an `S3TaskHandler` that overrides the default `FileTaskHandler`.

5. Restart your Airflow environment and run any task to verify that the task logs are copied to your S3 bucket.

   <Frame>
     <img alt="Logs in S3 bucket" />
   </Frame>


# Manage Airflow code
Source: https://astronomer.io/docs/learn/managing-airflow-code

Learn best practices for Airflow project organization, such as when to separate out DAGs into multiple projects and how to manage code used across different projects.

One of the tenets of Apache Airflow is that pipelines are defined as code. This allows you to treat your pipelines as you would any other piece of software and use best practices such as version control and CI/CD. As you scale the use of Airflow within your organization, it becomes important to manage your Airflow code in a way that is organized and sustainable.

In this guide, you'll learn how to organize your Airflow projects, when to separate your DAGs into multiple projects, how to manage code that is used in different projects, and review a typical development flow.

Throughout this guide, the term project is used to denote any set of DAGs and supporting files that are deployed to a single Airflow Deployment. For example, your organization might have a finance team and a data science team each with their own separate Airflow deployment, and they each have a separate Airflow project that contains all of their code.

<Tip>
  Depending on how you run Airflow you can configure multiple DAG bundles to pull DAGs from multiple sources into one Airflow environment. See [DAG versioning and DAG bundles](/docs/learn/airflow-dag-versioning) for more information.
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).

## Project structure

When working with Airflow, a consistent project structure helps keep all DAGs and supporting code organized and easy to understand, and it makes it easier to scale Airflow horizontally within your organization.

The ideal setup is to keep one directory and repository for each project. This means that you can use a version control tool such as GitHub or Bitbucket to package everything together.

Astronomer uses the following project structure:

```bash wrap theme={null}
.
├── dags                        # Folder where all your DAGs go
│   ├── example-dag.py
│   └── redshift_transforms.py
├── Dockerfile                  # For Astronomer's Docker image and runtime overrides
├── include                     # For any scripts that your DAGs might need to access
│   └── sql
│       └── transforms.sql
├── packages.txt                # For OS-level packages
├── plugins                     # For any custom or community Airflow plugins
│   └── example-plugin.py
└── requirements.txt            # For any Python packages
```

To create a project with this structure automatically, install the [Astro CLI](/docs/cli/v1.43/install-cli) and initialize a project with `astro dev init`.

If you aren't running Airflow with Docker or have different requirements for your organization, your project structure might look different. Choose a structure that works for your organization and keep it consistent so that anyone working with Airflow can easily transition between projects without having to re-learn a new structure.

## When to separate projects

The most common setup for Airflow projects is to keep all code for a given deployment in the same repository. However, there are some circumstances where it makes sense to separate DAGs into multiple projects. In these scenarios, it's best practice to have a separate Airflow deployment for each project. You might implement this project structure for the following reasons:

* Access control: Role-based access control (RBAC) should be managed at the Airflow Deployment level and not at the DAG level. If Team A and Team B should only have access to their own team's DAGs, it would make sense to split them up into two projects and deployments.
* Infrastructure considerations: If you have a set of DAGs that are well suited to the Kubernetes executor and another set that are well suited to the Celery executor, you may want to separate them into two different projects that feed Airflow deployments with different infrastructure. See [Configure Deployment resources](/docs/astro/deployment-settings).
* Dependency management: If DAGs have conflicting Python or OS dependencies, one way of managing this can be separating them into separate projects so they are isolated from one another.

Occasionally, some use cases require DAGs from multiple projects to be deployed to the same Airflow deployment. This is a less common pattern and isn't recommended for project organization unless it is specifically required. In this case, deploying the files from different repositories together into one Airflow deployment should be managed by your CI/CD tool. If you are implementing this use case with Astro Private Cloud, you need to use the [NFS](https://www.astronomer.io/docs/software/deploy-nfs) or [Git-Sync](https://www.astronomer.io/docs/software/deploy-git-sync) deployment methods.

## Reuse code

Code, such as custom hooks, operators, or DAG templates that are reused between projects, should be stored in a repository that's separate from your individual project repositories. This ensures that any changes to the re-used code only need to be made once and are applied across all projects where that code is used. See [Sharing code across projects](/docs/learn/sharing-code-multiple-projects) for more on how to structure and set up a project for shared code.

You can pull code from separate repositories into your Airflow projects. If you are working with Astronomer, see [Install Python packages from private sources](/docs/cli/v1.43/private-python-packages). Depending on your repository setup, it may also be possible to manage this with your CI/CD tool.

## Development flow

You can extend the project structure to work for developing DAGs and promoting code through `dev`, `QA`, and `production` environments. The most common method for managing code promotion with this project structure is to use branching. You still maintain one project and repository, and create `dev` and `qa` branches for any code in development or testing. Your `main` branch should correspond to code that is deployed to production. You can then use your CI/CD tool to manage promotion between these three branches.

There are many ways of implementing a development flow for your Airflow code. For example, you might work with feature branches instead of a specific `dev` branch. How you choose to implement this will depend on the needs of your organization, but the method described here is a good place to start if you aren't currently using CI/CD with Airflow.


# Manage task and task group dependencies in Airflow
Source: https://astronomer.io/docs/learn/managing-dependencies

Learn how to manage dependencies between tasks and TaskGroups in Apache Airflow, including how to set dynamic dependencies.

[Dependencies](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#relationships) are a powerful and popular Airflow feature. In Airflow, your pipelines are defined as Directed Acyclic Graphs (DAGs). Each task is a node in the graph and dependencies are the directed edges that determine how to move through the graph. Because of this, dependencies are key to following data engineering best practices because they help you define flexible pipelines with atomic tasks.

Throughout this guide, the following terms are used to describe task dependencies:

* **Upstream task**: A task that must reach a specified state before a dependent task can run.
* **Downstream task**: A dependent task that can't run until an upstream task reaches a specified state.

In this guide you'll learn about the many ways you can implement dependencies in Airflow, including:

* Basic task dependencies.
* Dependency functions.
* Dynamic dependencies.
* Dependencies with task groups.
* Dependencies with the TaskFlow API.
* Trigger rules.

To view a video presentation of these concepts, see [Manage Dependencies Between Airflow Deployments, DAGs, and Tasks](https://www.astronomer.io/events/webinars/manage-dependencies-between-airflow-deployments-dags-tasks/).

The focus of this guide is dependencies between tasks in the same DAG. If you need to implement dependencies between DAGs, see [Cross-DAG dependencies](/docs/learn/cross-dag-dependencies).

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).

## Basic dependencies

Basic dependencies between Airflow tasks can be set in the following ways:

* Using bit-shift operators (`<<` and `>>`)
* Using the `set_upstream` and `set_downstream` methods

For example, if you have a DAG with four sequential tasks, the dependencies can be set in four ways:

* Using `set_downstream()`:

  ```python wrap theme={null}
  t0.set_downstream(t1)
  t1.set_downstream(t2)
  t2.set_downstream(t3)
  ```

* Using `set_upstream()`:

  ```python wrap theme={null}
  t3.set_upstream(t2)
  t2.set_upstream(t1)
  t1.set_upstream(t0)
  ```

* Using `>>`:

  ```python wrap theme={null}
  t0 >> t1 >> t2 >> t3
  ```

* Using `<<`:

  ```python wrap theme={null}
  t3 << t2 << t1 << t0
  ```

All of these methods are equivalent and result in the DAG shown in the following image:

<Frame>
  <img alt="Basic Dependencies" />
</Frame>

Astronomer recommends using a single method consistently. Using both bit-shift operators and `set_upstream`/`set_downstream` in your DAGs can overly complicate your code.

To set a dependency where two downstream tasks are dependent on the same upstream task, use lists or tuples. For example:

```python wrap theme={null}
# Dependencies with lists
t0 >> t1 >> [t2, t3]

# Dependencies with tuples
t0 >> t1 >> (t2, t3)
```

These statements are equivalent and result in the DAG shown in the following image:

<Frame>
  <img alt="List Dependencies" />
</Frame>

When you use bit-shift operators and the `.set_upstream` and `.set_downstream` method, you can't set dependencies between two lists. For example, `[t0, t1] >> [t2, t3]` returns an error. To set dependencies between lists, use the dependency functions described in the following section.

## Dependency functions

Dependency functions are utilities that let you set dependencies between several tasks or lists of tasks. A common reason to use dependency functions over bit-shift operators is to create dependencies for tasks that were created in a loop and are stored in a list.

```python wrap theme={null}
from airflow.sdk import chain

list_of_tasks = []
for i in range(5):
    if i % 3 == 0:
        ta = EmptyOperator(task_id=f"ta_{i}")
        list_of_tasks.append(ta)
    else:
        ta = EmptyOperator(task_id=f"ta_{i}")
        tb = EmptyOperator(task_id=f"tb_{i}")
        tc = EmptyOperator(task_id=f"tc_{i}")
        list_of_tasks.append([ta, tb, tc])

chain(*list_of_tasks)
```

This code creates the following DAG structure:

<Frame>
  <img alt="List Dependencies" />
</Frame>

### Use `chain()`

To set parallel dependencies between tasks and lists of tasks of the same length, use the `chain()` function. For example:

```python wrap theme={null}
# from airflow.sdk import chain
chain(t0, t1, [t2, t3, t4], [t5, t6, t7], t8)
```

This code creates the following DAG structure:

<Frame>
  <img alt="Chain Dependencies" />
</Frame>

When you use the `chain` function, any lists or tuples that are set to depend directly on each other need to be the same length.

```python wrap theme={null}
chain([t0, t1], [t2, t3])  # this code will work
chain([t0, t1], [t2, t3, t4])  # this code will cause an error
chain([t0, t1], t2, [t3, t4, t5])  # this code will work
```

### Use `chain_linear()`

To set interconnected dependencies between tasks and lists of tasks, use the `chain_linear()` function.

Replacing `chain` in the previous example with `chain_linear` creates dependencies where each element in the downstream list will depend on each element in the upstream list.

```python wrap theme={null}
# from airflow.sdk import chain_linear
chain_linear(t0, t1, [t2, t3, t4], [t5, t6, t7], t8)
```

<Frame>
  <img alt="Chain Linear Dependencies 2" />
</Frame>

The `chain_linear()` function can accept lists of any length in any order. For example, the following arguments are valid:

```python wrap theme={null}
chain_linear([t0, t1], [t2, t3, t4])
```

<Frame>
  <img alt="Chain Linear Dependencies 1" />
</Frame>

## Dependencies in dynamic task mapping

Dependencies for [dynamically mapped tasks](/docs/learn/dynamic-tasks) can be set in the same way as regular tasks. Note that when using the default [trigger rule](#trigger-rules) `all_success`, all mapped task instances need to be successful for the downstream task to run. For the purpose of trigger rules, mapped task instances behave like a set of parallel upstream tasks.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # from airflow.providers.standard.operators.empty import EmptyOperator
  # from airflow.sdk import task 

  start=EmptyOperator(task_id="start")

  @task 
  def multiply(x,y):
      return x*y

  multiply_obj = multiply.partial(x=2).expand(y=[1,2,3])

  # end will only run if all mapped task instances of the multiply task are successful
  end = EmptyOperator(task_id="end")

  start >> multiply_obj >> end

  # all of the following ways of setting dependencies are valid
  # multiply_obj.set_downstream(end)
  # end.set_upstream(multiply_obj)
  # chain(start, multiply_obj, end)

  ```

  <Frame>
    <img alt="Dependencies dynamic tasks" />
  </Frame>
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # from airflow.providers.standard.operators.empty import EmptyOperator
  # from airflow.providers.standard.operators.python import PythonOperator

  start=EmptyOperator(task_id="start")

  def multiply_func(x,y):
      return x*y

  multiply_obj = PythonOperator.partial(
      task_id="multiply",
      python_callable=multiply_func,
      op_args=[2]
  ).expand(op_kwargs=[{"y": 1}, {"y": 2}, {"y": 3}])

  # end will only run if all mapped task instances of the multiply task are successful
  end = EmptyOperator(task_id="end")

  start >> multiply_obj >> end

  # all of the following ways of setting dependencies are valid
  # multiply_obj.set_downstream(end)
  # end.set_upstream(multiply_obj)
  # chain(start, multiply_obj, end)

  ```

  <Frame>
    <img alt="Dependencies dynamic tasks" />
  </Frame>
</details>

## Task group dependencies

[Task groups](/docs/learn/task-groups) logically group tasks in the Airflow UI and can be [mapped dynamically](/docs/learn/task-groups#generate-task-groups-dynamically-at-runtime). This section will explain how to set dependencies between task groups.

Dependencies can be set both inside and outside of a task group. For example, in the following DAG code there is a start task, a task group with two dependent tasks, and an end task. All of these tasks need to happen sequentially. The dependencies between the two tasks in the task group are set within the task group's context (`t1 >> t2`). The dependencies between the task group and the start and end tasks are set within the DAG's context (`t0 >> tg1() >> t3`).

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # from airflow.providers.standard.operators.empty import EmptyOperator
  # from airflow.sdk import task_group

  t0 = EmptyOperator(task_id="start")

  # Start task group definition
  @task_group(
      group_id="group1"
  )
  def tg1():
      t1 = EmptyOperator(task_id="task1")
      t2 = EmptyOperator(task_id="task2")

      t1 >> t2
  # End task group definition

  t3 = EmptyOperator(task_id="end")

  # Set task group's (tg1) dependencies
  t0 >> tg1() >> t3
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # from airflow.providers.standard.operators.empty import EmptyOperator
  # from airflow.sdk import TaskGroup

  t0 = EmptyOperator(task_id="start")

  # Start task group definition
  with TaskGroup(group_id="group1") as tg1:
      t1 = EmptyOperator(task_id="task1")
      t2 = EmptyOperator(task_id="task2")

      t1 >> t2
  # End task group definition

  t3 = EmptyOperator(task_id="end")

  # Set task group's (tg1) dependencies
  t0 >> tg1 >> t3
  ```
</details>

This image shows the resulting DAG:

<Frame>
  <img alt="Task Group Dependencies" />
</Frame>

You can also set dependencies between task groups, between tasks inside and out of task groups, and even between tasks in different (nested) task groups.

The image below shows types of dependencies that can be set between tasks and task groups. You can find the code that created this DAG in a GitHub repository both for the [TaskFlow API](https://github.com/astronomer/webinar-task-groups/blob/main/dags/example_complex_dependencies_1.py) and [traditional version](https://github.com/astronomer/webinar-task-groups/blob/main/dags/example_complex_dependencies_2.py).

<Frame>
  <img alt="Task Group Dependencies" />
</Frame>

## TaskFlow API dependencies

The [TaskFlow API](/docs/learn/airflow-decorators) `@task` decorator allows you to easily turn Python functions into Airflow tasks.

If your DAG has several tasks that are defined with the `@task` decorator and use each other's output, you can use inferred dependencies through the TaskFlow API. For example, in the following DAG there are two dependent tasks, `get_a_cat_fact` and `print_the_cat_fact`. To set the dependencies, you pass the called function of the upstream task as a positional argument to the downstream task (`print_the_cat_fact(get_a_cat_fact())`):

```python expandable wrap theme={null}
from airflow.decorators import dag, task
from datetime import datetime

import requests
import json

url = "http://catfact.ninja/fact"

default_args = {"start_date": datetime(2021, 1, 1)}


@dag(schedule="@daily", default_args=default_args, catchup=False)
def xcom_taskflow_dag():
    @task
    def get_a_cat_fact():
        """
        Gets a cat fact from the CatFacts API
        """
        res = requests.get(url)
        return {"cat_fact": json.loads(res.text)["fact"]}

    @task
    def print_the_cat_fact(cat_fact: str):
        """
        Prints the cat fact
        """
        print("Cat fact for today:", cat_fact)
        # run some further cat analysis here

    # Invoke functions to create tasks and define dependencies
    print_the_cat_fact(get_a_cat_fact())


xcom_taskflow_dag()
```

This image shows the resulting DAG:

<Frame>
  <img alt="TaskFlow Dependencies" />
</Frame>

Note that you can also assign the called function to an object and then pass that object to the downstream task. This way of defining dependencies is often easier to read and allows you to set the same task as an upstream dependency to multiple other tasks.

```python wrap theme={null}
# from airflow.sdk import task

@task 
def get_num():
    return 42

@task 
def add_one(num):
    return num + 1

@task
def add_two(num):
    return num + 2

num = get_num()
add_one(num)
add_two(num)
```

<Frame>
  <img alt="TaskFlow Dependencies" />
</Frame>

If your DAG contains a mix of Python function tasks defined with decorators and tasks defined with traditional operators, you can set the dependencies by assigning the decorated task invocation to a variable and then defining the dependencies normally. For example, in the DAG below the `my_task_1` task is defined by the `@task` decorator and invoked with `_my_task_1 = my_task_1()`. The `my_task_1` variable is used in the last line to define dependencies.

```python wrap theme={null}
# from airflow.sdk import dag, task 
# from airflow.providers.standard.operators.empty import EmptyOperator

@dag
def my_dag():

    @task
    def my_task_1():
        pass

    _my_task_1 = my_task_1()

    my_task_2 = EmptyOperator(task_id="my_task_2")

    chain(
        _my_task_1,
        my_task_2,
    )

my_dag()

```

To learn how to pass information between TaskFlow decorators and traditional tasks, see [Mixing TaskFlow decorators with traditional operators](/docs/learn/airflow-decorators#mixing-taskflow-decorators-with-traditional-operators).

## Trigger rules

When you set dependencies between tasks, the default Airflow behavior is to run a task only when all upstream tasks have succeeded. You can use [trigger rules](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#trigger-rules) to change this default behavior.

The following options are available:

* `all_success`: (default) The task runs only when all upstream tasks have succeeded.
* `all_failed`: The task runs only when all upstream tasks are in a failed or upstream\_failed state.
* `all_done`: The task runs once all upstream tasks are done with their execution.
* `all_skipped`: The task runs only when all upstream tasks have been skipped.
* `one_failed`: The task runs when at least one upstream task has failed.
* `one_success`: The task runs when at least one upstream task has succeeded.
* `one_done`: The task runs when at least one upstream task has either succeeded or failed.
* `none_failed`: The task runs only when all upstream tasks have succeeded or been skipped.
* `none_failed_min_one_success`: The task runs only when all upstream tasks haven't failed or `upstream_failed`, and at least one upstream task has succeeded.
* `none_skipped`: The task runs only when no upstream task is in a skipped state.
* `always`: The task runs at any time.

Learn more about trigger rules in the [Airflow trigger rules](/docs/learn/airflow-trigger-rules) guide.


# Integrate OpenLineage and Airflow with Marquez
Source: https://astronomer.io/docs/learn/marquez

Use OpenLineage and Marquez to get lineage metadata locally from your Airflow DAGs.

[OpenLineage](https://openlineage.io/) is the open source industry standard framework for data lineage. Integrating OpenLineage with Airflow gives you greater observability over your data pipelines and helps with everything from data governance to tracking the affected area of a task failure across DAGs to managing PII.

Viewing and interacting with lineage metadata requires running a lineage front end. [Marquez](https://github.com/MarquezProject/marquez) is the most common open source choice for this purpose, and integrates easily with Airflow.

In this tutorial, you'll run OpenLineage with Airflow locally using Marquez as a lineage front end. You'll then generate and interpret lineage metadata using two DAGs that process data in Postgres.

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of OpenLineage. See [Integrate OpenLineage and Airflow](/docs/learn/airflow-openlineage).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* [PostgreSQL](https://www.postgresql.org/download/).

## Step 1: Run Marquez locally

1. Clone the Marquez repository:

   ```sh wrap theme={null}
   git clone https://github.com/MarquezProject/marquez && cd marquez
   ```

2. Run the following command in the `marquez` directory to start Marquez:

   ```sh wrap theme={null}
   ./docker/up.sh
   ```

For more details, see the quickstart in the [Marquez README](https://github.com/MarquezProject/marquez#quickstart).

## Step 2: Configure your Astro project

Use the Astro CLI to create and run an Airflow project locally that will integrate with Marquez.

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-marquez-tutorial && cd astro-marquez-tutorial
   $ astro dev init
   ```

2. Add the Airflow OpenLineage provider to your `requirements.txt` file. Note that the minimum Apache Airflow version required is 2.7.0 (Astro Runtime 9.0.0+).

   ```text wrap theme={null}
   apache-airflow-providers-openlineage==1.10.0
   ```

3. Add the following environment variables below to your Astro project `.env` file:

   ```bash wrap theme={null}
   OPENLINEAGE_URL=http://host.docker.internal:5000
   OPENLINEAGE_NAMESPACE=example
   ```

   These variables allow Airflow to connect with the OpenLineage API and send your lineage metadata to Marquez.

   By default, Marquez uses port 5000 when you run it using Docker. If you are using a different OpenLineage front end instead of Marquez, or you are running Marquez remotely, you can modify the `OPENLINEAGE_URL` as needed.

4. Marquez also uses Postgres, so Airflow needs to use a different port than the default 5432, which is already allocated to Airflow. Run the following command to use a port 5435 for Postgres:

   ```sh wrap theme={null}
   astro config set postgres.port 5435
   ```

5. Run the following command to start your local project:

   ```sh wrap theme={null}
   astro dev start
   ```

6. Confirm Airflow is running by going to `http://localhost:8080`, and Marquez is running by going to `http://localhost:3000`.

## Step 3: Configure your database

To show the lineage metadata that can result from Airflow DAG runs, you'll use two sample DAGs that process data in Postgres. To run this example in your local environment, complete the following steps:

1. Using `psql`, create a local Postgres database in the same container as the Airflow metastore:

   ```bash wrap theme={null}
   psql -h localhost -p 5435 -U postgres
   # enter password `postgres` when prompted
   create database lineagetutorial;
   \c lineagetutorial;
   ```

   If you already have a Postgres database or are using a different type of database you can skip this step. Note that this database should be separate from the Airflow and Marquez metastores.

2. Run the following SQL statements in your new database to create and populate two source tables:

   ```sql wrap theme={null}
   CREATE TABLE IF NOT EXISTS adoption_center_1
   (date DATE, type VARCHAR, name VARCHAR, age INTEGER);

   CREATE TABLE IF NOT EXISTS adoption_center_2
   (date DATE, type VARCHAR, name VARCHAR, age INTEGER);

   INSERT INTO
       adoption_center_1 (date, type, name, age)
   VALUES
       ('2022-01-01', 'Dog', 'Bingo', 4),
       ('2022-02-02', 'Cat', 'Bob', 7),
       ('2022-03-04', 'Fish', 'Bubbles', 2);

   INSERT INTO
       adoption_center_2 (date, type, name, age)
   VALUES
       ('2022-06-10', 'Horse', 'Seabiscuit', 4),
       ('2022-07-15', 'Snake', 'Stripes', 8),
       ('2022-08-07', 'Rabbit', 'Hops', 3);
   ```

## Step 4: Configure your Airflow connection

The connection you configure will connect to the Postgres database you created in [Step 3](#step-3-configure-your-database).

1. In the Airflow UI, go to **Admin** > **Connections**.

2. Create a new connection named `postgres_default` and choose the `postgres` connection type. Enter the following information:

   * **Host:** `host.docker.internal`
   * **Login:** `postgres`
   * **Password:** `postgres`
   * **Port:** `5435`

   If you are working with a database other than local Postgres, you may need to provide different information to the connection.

## Step 5: Create your DAGs

For this tutorial, you create two DAGs to generate and interpret lineage metadata.

1. In your Astro project `dags` folder, create a new file called `lineage-combine.py`. Paste the following code into the file:

   ```python expandable wrap theme={null}
   from datetime import datetime, timedelta

   from airflow.models.dag import DAG
   from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator

   create_table_query= '''
       CREATE TABLE IF NOT EXISTS animal_adoptions_combined (
           date DATE,
           type VARCHAR,
           name VARCHAR,
           age INTEGER
           );
   '''

   combine_data_query= '''
       INSERT INTO animal_adoptions_combined (date, type, name, age) 
           SELECT * 
           FROM adoption_center_1
           UNION 
           SELECT *
           FROM adoption_center_2;
   '''

   with DAG(
       'lineage-combine-postgres',
       start_date=datetime(2022, 12, 1),
       max_active_runs=1,
       schedule='@daily',
       default_args = {
           'retries': 1,
           'retry_delay': timedelta(minutes=1)
       },
       catchup=False
   ):

       create_table = SQLExecuteQueryOperator(
           task_id='create_table',
           postgres_conn_id='postgres_default',
           sql=create_table_query
       ) 

       insert_data = SQLExecuteQueryOperator(
           task_id='combine',
           postgres_conn_id='postgres_default',
           sql=combine_data_query
       ) 

       create_table >> insert_data
   ```

2. Create another file in your `dags` folder and call it `lineage-reporting.py`. Paste the following code into the file:

   ```python expandable wrap theme={null}
   from datetime import datetime, timedelta

   from airflow.models.dag import DAG
   from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator

   aggregate_reporting_query = '''
       INSERT INTO adoption_reporting_long (date, type, number)
       SELECT c.date, c.type, COUNT(c.type)
       FROM animal_adoptions_combined c
       GROUP BY date, type;
   '''

   with DAG(
       'lineage-reporting-postgres',
       start_date=datetime(2020, 6, 1),
       max_active_runs=1,
       schedule='@daily',
       default_args={
           'retries': 1,
           'retry_delay': timedelta(minutes=1)
       },
       catchup=False
   ):

       create_table = SQLExecuteQueryOperator(
           task_id='create_reporting_table',
           postgres_conn_id='postgres_default',
           sql='''
               CREATE TABLE IF NOT EXISTS adoption_reporting_long (
                   date DATE,
                   type VARCHAR,
                   number INTEGER
                   );
           ''',
       ) 

       insert_data = SQLExecuteQueryOperator(
           task_id='reporting',
           postgres_conn_id='postgres_default',
           sql=aggregate_reporting_query
       ) 

       create_table >> insert_data
   ```

The first DAG creates and populates a table (`animal_adoptions_combined`) with data aggregated from the two source tables (`adoption_center_1` and `adoption_center_2`) you created in [Step 3](#step-3-configure-your-database). The second DAG creates and populates a reporting table (`adoption_reporting_long`) using data from the aggregated table (`animal_adoptions_combined`) created in your first DAG. Both of these DAGs use the `SQLExecuteQueryOperator` - supported by OpenLineage, so lineage is generated automatically.

You might want to make adjustments to these DAGs if you are working with different source tables, or if your Postgres connection id isn't `postgres_default`.

## Step 6: Run your DAGs and view lineage metadata

You can trace the data through the DAGs you created in Step 5 by viewing their lineage metadata in Marquez.

1. Run the `lineage-combine-postgres` DAG.

2. Run the `lineage-reporting-postgres` DAG.

3. Go to the Marquez UI at `localhost:3000` and view the jobs created by each task instance. You should see something like this:

   <Frame>
     <img alt="Marquez Jobs" />
   </Frame>

4. Click one of the jobs from your DAGs to see the full lineage graph.

   <Frame>
     <img alt="Marquez Graph" />
   </Frame>

   The lineage graph shows:

   * Two origin datasets that are used to populate the combined data table.
   * The four jobs (tasks) from your DAGs that create new tables and result in new combined datasets: `combine` and `reporting`.
   * Two new datasets that are created by those jobs.

The lineage graph shows you how these two DAGs are connected and how data flows through the entire pipeline, giving you insight you wouldn't have if you were to view these DAGs in the Airflow UI alone.

## Conclusion

Congratulations! You can now run Marquez and Airflow locally and trace data through your DAGs by viewing their lineage. As a great next step, try other Airflow operators that generate lineage metadata. Or, if you are an Astronomer customer, check out [lineage in Astro](/docs/astro/create-data-products).


# Rerun Airflow Dags and tasks
Source: https://astronomer.io/docs/learn/rerunning-dags

How to configure retries, catchup, backfill, and clear task instances in Airflow.

You can set when to run Airflow Dags using a wide variety of [scheduling](/docs/learn/scheduling-in-airflow) options. Some use cases where you might want tasks or Dags to run outside of their regular schedule include:

* You want one or more tasks to automatically run again if they fail.
* You need to manually rerun a failed task for one or multiple Dag runs.
* You want to deploy a Dag with a start date of one year ago and trigger all Dag runs that would have been scheduled in the past year.
* You have a running Dag and realize you need it to process data for two months prior to the Dag's start date.

In this guide, you'll learn how to configure automatic retries, rerun tasks or Dags, trigger historical Dag runs, and review the Airflow concepts of catchup and backfill.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Dag scheduling. See [Schedule Dags in Airflow](/docs/learn/scheduling-in-airflow)

## Automatically retry tasks

In Airflow, you can configure individual tasks to retry automatically in case of a failure. The default number of times a task will retry before failing permanently can be defined at the Airflow configuration level using the core config `default_task_retries`. You can set this configuration either in `airflow.cfg` or with the environment variable `AIRFLOW__CORE__DEFAULT_TASK_RETRIES`.
You can overwrite the `default_task_retries` of an Airflow environment at the task level by using the `retries` parameter.

The `retry_delay` parameter (default: `timedelta(seconds=300)`) defines the time spent between retries. You can set a maximum value for the retry delay in the core Airflow config, `max_task_retry_delay` (`AIRFLOW__CORE__MAX_TASK_RETRY_DELAY`), which, by default, is set at 24 hours. Or, for individual tasks, you can set the maximum retry delay with the parameter, `max_retry_delay`.

To progressively increase the wait time between retries until `max_retry_delay` is reached, set `retry_exponential_backoff` to `True` for the delay to double with each retry. In Airflow 3.2+ you can set `retry_exponential_backoff` to a float to directly specify the factor by which the retry delay should be multiplied between retries. For example, to multiply the retry delay by 3 between retries, set `retry_exponential_backoff` to `3.0`.

It is common practice to set the number of retries for all tasks in a Dag by using `default_args` and override it for specific tasks as needed. To override specific tasks, provide a different value to the task level `retries` parameter.

The Dag below contains 4 tasks that will always fail. Each of the tasks uses a different retry parameter configuration.

```python expandable wrap theme={null}
from airflow.decorators import dag
from airflow.operators.bash import BashOperator
from pendulum import datetime, duration


@dag(
    start_date=datetime(2023, 4, 1),
    schedule="@daily",
    catchup=False,
    default_args={
        "retries": 3,
        "retry_delay": duration(seconds=2),
        "retry_exponential_backoff": True,
        "max_retry_delay": duration(hours=2),
    },
)
def retry_example():
    t1 = BashOperator(task_id="t1", bash_command="echo I get 3 retries! && False")

    t2 = BashOperator(
        task_id="t2",
        bash_command="echo I get 6 retries and never wait long! && False",
        retries=6,
        max_retry_delay=duration(seconds=10),
    )

    t3 = BashOperator(
        task_id="t3",
        bash_command="echo I wait exactly 20 seconds between each of my 4 retries! && False",
        retries=4,
        retry_delay=duration(seconds=20),
        retry_exponential_backoff=False,
    )

    t4 = BashOperator(
        task_id="t4",
        bash_command="echo I have to get it right the first time! && False",
        retries=0,
    )


retry_example()
```

### Retry policies

In Airflow 3.3+, rather than applying a fixed retry count as described previously, you can attach a retry policy that determines whether, when, and how a task is retried based on the type of failure that occurs.

For example, for exceptions that you know may be transient (like a 503 error), you can retry the task multiple times with a backoff. For exceptions that require manual intervention (like an authentication issue), you can fail the task immediately.

You can define your retry logic using `ExceptionRetryPolicy`. Each rule maps an exception type to an action, with an optional custom retry delay and a reason that gets logged. You can define the policy in a Dag file directly, or add it to your include/ folder and import it into multiple Dags. Any task errors that aren't covered by your policy will use the default retry behavior that you've set for your task, Dag, or Airflow environment.

```python expandable wrap theme={null}
from datetime import timedelta

from airflow.sdk import (
    ExceptionRetryPolicy,
    RetryAction,
    RetryRule,
    task,
)

SNOWFLAKE_RETRY_POLICY = ExceptionRetryPolicy(
    rules=[
        RetryRule(
            exception="snowflake.connector.errors.OperationalError",
            action=RetryAction.RETRY,
            retry_delay=timedelta(minutes=2),
            reason="Transient connection or warehouse issue",
        ),
        RetryRule(
            exception="snowflake.connector.errors.ProgrammingError",
            action=RetryAction.FAIL,
            reason="SQL or compilation error, query edit needed",
        ),
    ],
)

@task(
    retries=4,
    retry_delay=timedelta(seconds=30),
    retry_policy=SNOWFLAKE_RETRY_POLICY,
)
def daily_aggregation():
    ...
```

In this example, the Dag runs queries against Snowflake. If an operational error is raised, the task retries after a two-minute delay. Since this kind of error is usually caused by something like a network blip or an account-level concurrency limit, waiting and trying again makes sense. But if a programming error is raised, the task fails immediately. Programming errors are usually a problem with the SQL itself, so retrying won't produce a different result.

For more on this feature, including exception matching and composition with existing retry parameters, see the [Airflow docs](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#retry-policies).

### Custom retry policies

You can also implement a custom retry policy by subclassing `RetryPolicy` and implementing its `evaluate()` method. This lets you inspect exception attributes or use context that `ExceptionRetryPolicy` doesn't have access to.

Building on the example in the previous section, some database errors can only be distinguished by an attribute on the exception rather than by the exception class itself. A custom policy can implement retry logic on those attributes:

```python wrap theme={null}
from datetime import timedelta

from airflow.sdk import RetryDecision, RetryPolicy


class SnowflakeErrorCodeRetryPolicy(RetryPolicy):
    """Route Snowflake errors by error code rather than exception class."""

    RESOURCE_PRESSURE_CODES = {605, 606}     # statement canceled, resource shortage
    SCHEMA_ERROR_CODES = {2003, 2043, 3001}  # missing object, cannot drop, invalid identifier

    def evaluate(self, exception, try_number, max_tries, context=None):
        errno = getattr(exception, "errno", None)
        if errno in self.RESOURCE_PRESSURE_CODES:
            return RetryDecision.retry(retry_delay=timedelta(minutes=5))
        if errno in self.SCHEMA_ERROR_CODES:
            return RetryDecision.fail(reason=f"Snowflake error {errno}, retries won't fix it")
        return RetryDecision.default()
```

Another common example would be customizing retry behavior based on how much time has passed since the Dag started running. If your Dag produces time-sensitive data products, you may want the task to fail sooner so the team is notified, rather than waiting on multiple retries.

```python wrap theme={null}
from datetime import timedelta

from airflow.sdk import RetryDecision, RetryPolicy
from airflow.utils import timezone


class StaleRunRetryPolicy(RetryPolicy):
    """Stop retrying once the run is too stale to be useful."""

    STALENESS_THRESHOLD = timedelta(hours=6)

    def evaluate(self, exception, try_number, max_tries, context=None):
        if context is None:
            return RetryDecision.default()
        logical_date = context["dag_run"].logical_date
        if timezone.utcnow() - logical_date > self.STALENESS_THRESHOLD:
            return RetryDecision.fail(
                reason="Run is past the staleness threshold, escalating"
            )
        return RetryDecision.default()
```

## Automatically pause a failing Dag

You can configure Airflow to automatically pause a Dag after a certain number of failed Dag runs, preventing a failing Dag from continuing to run and potentially causing more issues.

To set the maximum number of consecutive failed Dag runs for all your Dags in your Airflow environment, set the `core.max_consecutive_failed_dag_runs_per_dag` config. For example, to automatically pause all your Dags after they had 5 failed Dag runs in a row, set:

```text wrap theme={null}
AIRFLOW__CORE__MAX_CONSECUTIVE_FAILED_DAG_RUNS_PER_DAG=5
```

You can override this setting for a specific Dag by setting the `max_consecutive_failed_dag_runs` parameter in the Dag instantiation. For example, to pause a specific Dag after 3 failed Dag runs in a row, set:

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  # from airflow.sdk import dag
  # from pendulum import datetime

  @dag(
      start_date=datetime(2024, 4, 1),
      schedule="@daily",
      max_consecutive_failed_dag_runs=3,
      catchup=False,
  )
  def my_dag():
      # Define your tasks here

  my_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  # from airflow.sdk import DAG
  # from pendulum import datetime

  with DAG(
      dag_id="my_dag",
      start_date=datetime(2024, 4, 1),
      schedule="@daily",
      max_consecutive_failed_dag_runs=3,
      catchup=False,
  ):
      # Define your tasks here
  ```
</details>

<Warning>
  The `max_consecutive_failed_dag_runs` config and Dag-level parameter is currently experimental and might be subject to breaking changes in future releases.
</Warning>

## Manually rerun tasks or Dags

[Rerunning tasks](https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html#re-run-tasks) or full Dags in Airflow is a common workflow.

To rerun a task in Airflow you clear the task status to update the `max_tries` and current task instance state values in the metastore. After the task reruns, the `max_tries` value updates to `0`, and the current task instance state updates to `None`.

To clear the task status, go to your Dag in the Airflow UI, select the task instance you want to rerun and click the **Clear Task Instance** button.

<Frame>
  <img alt="Clear Task Status" />
</Frame>

A popup window appears, giving you the following options to clear and rerun additional task instances related to the selected task:

* Past: Clears any instances of the task in Dag runs with a logical date before the selected task instance.
* Future: Clears any instances of the task in Dag runs with a logical date after the selected task instance.
* Upstream: Clears any tasks in the current Dag run which are upstream from the selected task instance.
* Downstream: Clears any tasks in the current Dag run which are downstream from the selected task instance.
* Only Failed: Clears only failed instances of any task instances selected based on the above options.

The window shows which task instances will be cleared with the current settings. Click **Confirm**  and the task(s) will be cleared and rescheduled for another run.

<Frame>
  <img alt="Task Instance Summary" />
</Frame>

You can also use the [Airflow CLI](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#clear) or [API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/patch_task_instance) to programmatically clear task instances.

To clear a full Dag run, click the Dag run, and then click **Clear Run** as shown in the following image.

<Frame>
  <img alt="Clear DAG Status" />
</Frame>

<Warning>
  Don't clear or change task statuses directly in the Airflow metastore. This can cause unexpected behavior in Airflow.
</Warning>

### Add notes to cleared tasks and Dags

You can add notes to task instances and Dag runs in the Airflow UI. This feature is useful for tracking manual changes to task instances, such as reruns or task status changes. Astronomer recommends leaving a note on a task or Dag whenever you manually update a task instance through the Airflow UI.

To add a note to a task instance or Dag run:

1. Go to your Dag in the Airflow UI.
2. Select a task instance or Dag run.
3. Click **Add a note**.
4. Write a note and click **Confirm**.

<Frame>
  <img alt="Add task note" />
</Frame>

We recommend using this feature for tracking and maintaining visibility of manual changes made to task instances such as rerunning or changing the task status.

## Catchup

You can use the built-in [catchup](https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html#catchup) Dag argument to process data for logical dates between the set `start_date` of a Dag and the current date.

When the catchup parameter for a Dag is set to `True`, at the time the Dag is turned on in Airflow, the scheduler starts a Dag run for every data interval that hasn't been run between the Dag's `start_date` and the current data interval. For example, if your Dag is scheduled to run daily and has a `start_date` of 1/1/2025, and you deploy that Dag and turn it on 2/1/2025, Airflow will schedule and start all of the daily Dag runs for January. Catchup is also triggered when you turn a Dag off for a period and then turn it on again.

Catchup can be controlled by setting the parameter in your Dag's arguments. By default, catchup is set to `False`. This example Dag has catchup turned on:

```python wrap theme={null}
@dag(
    dag_id="example_dag",
    start_date=datetime(2025, 4, 23),
    max_active_runs=1,
    schedule="@daily",
    default_args={
        "retries": 1,
        "retry_delay": timedelta(minutes=3),
    },
    catchup=True
)
```

Catchup is a powerful feature, but it should be used with caution. For example, if you deploy a Dag that runs every 5 minutes with a start date of 1 year ago and set catchup to `True`, Airflow will schedule numerous Dag runs all at once. When using catchup, keep in mind what resources Airflow has available and how many Dag runs you can support at one time. To avoid overloading your scheduler or external systems, you can use the following parameters in conjunction with catchup:

* `max_active_runs`: Set at the Dag level and limits the number of Dag runs that Airflow will execute for that particular Dag at any given time. For example, if you set this value to 3 and the Dag had 15 catchup runs to complete, they would be executed in 5 chunks of 3 runs.
* `depends_on_past`: Set at the task level or as a `default_arg` for all tasks at the Dag level. When set to `True`, the task instance must wait for the same task in the most recent Dag run to be successful. This ensures sequential data loads and allows only one Dag run to be executed at a time in most cases.
* `wait_for_downstream`: Set at the Dag level and similar to a Dag-level implementation of `depends_on_past`. The entire Dag needs to run successfully for the next Dag run to start.

If you want to set catchup to True by default for all Dags in your Airflow environment, for example for migration purposes, you can set the Airflow config `AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT` to `True`.

If you want to deploy your Dag with catchup enabled but there are some tasks you don't want to run during the catchup, you can use the [`LatestOnlyOperator`](https://airflow.apache.org/registry/providers/standard#standard-latest_only-LatestOnlyOperator) in your Dag. This operator only runs during the Dag's most recent scheduled interval. In every other Dag run it is ignored, along with any tasks downstream of it.

## Backfill

[Backfilling](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dag-run.html#backfill) is the concept of running a Dag for a specified period in the past to re-process historical or missed data. Unlike catchup, which triggers missed Dag runs from the Dag's `start_date` through the current data interval, backfill periods can be specified explicitly and can include periods prior to the Dag's `start_date`.

In Airflow 3, backfills are managed by the scheduler and can be triggered through the UI, API, or CLI. To run a backfill from the UI, click the blue **Trigger** button and select **Backfill**. Choose the date range you want to backfill for, and which runs you want to reprocess. You also have the option to select the number of max active runs for the backfill, whether you want to run backwards or forwards, specify run parameters and select one of three reprocessing behaviors:

* **Missing Runs**: Creates and runs only the Dag runs that don't already exist for the selected period.
* **Missing and Errored Runs**: Creates any missing runs and also re-runs any existing Dag runs that previously failed.
* **All Runs**: Clears and re-runs all existing Dag runs within the date range, in addition to creating any that are missing.

Execute the backfill by clicking **Run Backfill**. The UI tells you how many runs will be triggered and the backfill uses the latest [DAG version](/docs/learn/airflow-dag-versioning) that is available for your Dag.

<Frame>
  <img alt="Trigger backfill" />
</Frame>

Once the backfill has started, you can pause or cancel it at any time in the UI. Backfilled DAG runs are indicated in your grid by the u-turn arrow.

To see an example of backfilling using the CLI, see the [Airflow docs](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#backfill). For information on how to backfill using the Airflow REST API see the [Airflow REST API docs](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/Backfill).

When using backfill, make sure to consider your available resources. If your backfill will trigger many Dag runs, and/or you have many other Dags running at the same time, you should set a max active runs that won't overload your scheduler.


# Run Apache Airflow® locally
Source: https://astronomer.io/docs/learn/run-airflow-locally

Run Apache Airflow® on your local computer with the Astro CLI.

When developing [Apache Airflow®](https://airflow.apache.org/) Dags locally using AI agents, you need to give your agents a way to test their Dag changes in an Airflow environment. The easiest way for humans and AI agents alike to run Airflow on your computer is by using the [Astro CLI](/docs/cli/v1.45/overview), which gives you a fully functional Airflow environment, as well as options to test your Dag code. While the Astro CLI is freely available and you don't need to be an Astronomer customer to install and use it, you can use additional functionality if you have an Astronomer account.

<Tip>
  The Astro CLI contains the `astro otto` command, which lets you interact with Otto, Astronomer's data engineering agent. You can run Otto with models from Anthropic, OpenAI, and Google, and it's available as part of the [free Astro trial](https://www.astronomer.io/lp/signup/). See [Otto overview](/docs/astro/otto-overview) for more information.
</Tip>

<Info>
  For a hands-on demo of using AI agents in local data engineering, watch the recording of the [Local data engineering in the agentic era](https://www.astronomer.io/events/webinars/local-data-engineering-in-the-agentic-era-video/) webinar.
</Info>

## Assumed knowledge

To get the most out of this guide, you should have existing knowledge of:

* Airflow basics. See [Introduction to Apache Airflow®: A Technical Overview for Beginners](/docs/learn/intro-to-airflow).
* Basic use of a terminal. See [GNU Bash manual](https://www.gnu.org/software/bash/manual/).

## Install the Astro CLI

You can install the Astro CLI using [Homebrew](https://formulae.brew.sh/formula/astro) or directly with `curl`.

```bash theme={null}
curl -sSL install.astronomer.io | sudo bash -s
```

If you installed the Astro CLI previously, rerun the install command to upgrade to the latest version and access all commands mentioned in this document.

## Create and start an Airflow project

Use the `astro dev init` command to create a full Airflow project, including an example Dag, in any empty folder on your computer. Start the project with `astro dev start` in one of two modes.

```bash theme={null}
astro dev init                # create a new Airflow project in the current folder
astro dev start               # start Airflow (container mode by default, --standalone for standalone mode)
```

Container mode runs Airflow in five containers, one for each core component (Docker or Podman required):

* **Scheduler**: Monitors Dags and task instances, and schedules a task to run as soon as its dependencies are fulfilled, using the `LocalExecutor`, which runs tasks inside the scheduler process itself. This is the container you want to use as a dev container when developing Airflow Dags. See [Set up your IDE for data engineering](/docs/learn/set-up-your-ide-for-data-engineering).
* **API server**: A FastAPI server that serves the Airflow UI and the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html). It also allows workers to interact with the metadata database through an internal API.
* **Dag processor**: Parses the Dag files in your project and stores a serialized version of each Dag in the metadata database.
* **Metadata database**: Stores operational information for Airflow, such as serialized Dags and the history of Dag runs and task instances. The Astro CLI in container mode uses Postgres for this database.
* **Triggerer**: Runs asynchronous triggers for [deferrable operators](/docs/learn/deferrable-operators) and [event-driven scheduling](/docs/learn/airflow-event-driven-scheduling).

See [Apache Airflow components](/docs/learn/airflow-components) for how these components interact with each other.

<Info>
  You can still write Airflow Dags even if you can't install the Astro CLI on your computer. Astro includes the [Astro IDE](/docs/astro/ide-overview), an in-browser Airflow development environment, where you can write Dags (with optional assistance from Otto), push them to a test Deployment, and commit them to a Git repository. The Astro IDE is included in every [free Astro trial](https://www.astronomer.io/lp/signup/).
</Info>

### Container mode vs standalone mode

|                   | Container mode (default)                | Standalone mode (experimental)            |
| ----------------- | --------------------------------------- | ----------------------------------------- |
| Requires          | Docker or Podman                        | -                                         |
| Runs Airflow as   | Five containers, one per core component | Processes in a Python virtual environment |
| Start project     | `astro dev start`                       | `astro dev start --standalone`            |
| Executor          | `LocalExecutor`                         | `LocalExecutor`                           |
| Metadata database | Postgres                                | SQLite                                    |

The Airflow UI is where you view Dag runs, trigger Dags manually, and inspect task logs. By default, `astro dev start` runs a built-in reverse proxy that serves the Airflow UI at `<project>.localhost:6563`, based on your project folder name. This applies in both container mode and standalone mode, and lets you run multiple Airflow projects at the same time without manually managing ports. Use `astro dev start --no-proxy` to serve the Airflow UI directly at `localhost:8080` instead. See [`astro dev proxy`](/docs/cli/v1.44/astro-dev-proxy) for more information.

The first Airflow project you start is still available at `localhost:8080` even with the proxy enabled. If you start an additional project while the first one is still running, that project gets a random port instead and is only reachable through its own `<project>.localhost:6563` address.

See [Standalone mode](/docs/cli/v1.44/astro-dev-start#standalone-mode) in the `astro dev start` reference for more information.

Astronomer recommends container mode unless you can't run Docker or Podman.

## Astro CLI commands for local development

The Astro CLI has many commands for your AI agent (and you) to interact with your Airflow environments that run locally and on Astro. The following commands are especially useful when working locally:

```bash theme={null}
astro completion <shell>         # tab completions for Astro CLI commands
astro api airflow                # query the Airflow Registry API
astro dev bash                   # exec into the scheduler container
astro dev kill                   # remove all containers, metadata database, connections
astro dev logs                   # component logs: -s scheduler, -t triggerer,
                                 # --api-server, --dag-processor, -f to follow
astro dev parse                  # check for Dag import errors
astro dev pytest                 # run tests in /tests inside a container
astro dev run                    # run any Airflow CLI command
astro dev run dags reserialize   # force re-parse of all Dags, including new ones
```

The `astro dev run` command can run any [Airflow CLI](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html) command. Replace `airflow` with `astro dev run`.

If you have an Astro account, you have access to many more commands that interact with Deployments on Astro. Otto, Astronomer's data engineering AI agent, uses a large language model (LLM) gateway running on Astro, which is why you need to sign in to work with Otto.

```bash theme={null}
astro login   # sign in to your Astro account
astro otto    # launch Otto, Astronomer's data engineering agent
astro deploy  # push your local project to an Astro Deployment

# start locally with the same connections you configured in the 
# Astro Environment Manager for a Deployment
astro dev start --deployment-id <deployment-id>
```

## Restrict agent commands

Most AI coding agents can call the command-line tools already installed on your computer. If you allow your harness to run Bash commands without human approval, for example in Claude Code's auto mode, where a classifier by Anthropic decides whether a command appears to be safe, your agent can run CLI commands that have unintended consequences.

Agent harnesses typically offer you a way to give more fine-grained permissions for specific CLI commands. [Claude Code](https://code.claude.com/docs/en/permissions), for example, defines command permissions in `settings.json` under a `permissions` object with three lists:

* `allow`: Commands Claude Code runs without asking
* `deny`: Commands Claude Code never runs, even if a broader `allow` rule matches
* `ask`: Commands Claude Code confirms with you before running

A `deny` rule always overrides a matching `allow` rule.

For example, this configuration lets Claude Code run any `astro dev` sub-command without asking, but requires confirmation before it runs `astro deploy`:

```json theme={null}
{
  "permissions": {
    "allow": [
      "Bash(astro dev *)"
    ],
    "ask": [
      "Bash(astro deploy *)"
    ]
  }
}
```

Any commands that don't match a rule use the current general setting, for example asking for permission if the harness is in manual mode and potentially running without asking for approval if the harness is in auto mode.

<Note>
  This example is for Claude Code. If you're using another harness such as [OpenAI Codex](https://openai.com/codex/) or [Google Gemini CLI](https://geminicli.com/), you have different options to restrict CLI command usage. See the relevant harness documentation for more information.
</Note>


# Schedule DAGs in Apache Airflow®
Source: https://astronomer.io/docs/learn/scheduling-in-airflow

Get to know Airflow scheduling concepts and different ways to schedule a DAG.

One of the fundamental features of [Apache Airflow®](https://airflow.apache.org/) is the ability to schedule DAGs. Airflow offers many different options for scheduling, from simple cron-based schedules, over [data-aware scheduling](#data-aware-scheduling) with assets to event-driven scheduling based on messages in a queue.

In this guide, you'll learn:

* How to interpret the timestamps associated with a DAG run.
* How to set DAG parameters that control scheduling.
* The options available for scheduling DAGs.

<Info>
  This guide gives an overview of scheduling options. There are a number of related guides that cover specific types of scheduling in more detail:

  * [Assets and data-aware scheduling](/docs/learn/airflow-datasets)
  * [Event-driven scheduling](/docs/learn/airflow-event-driven-scheduling)
  * [Rerun Airflow DAGs and tasks](/docs/learn/rerunning-dags) (including backfilling)
</Info>

## Assumed knowledge

To get the most out of this guide, you should have an existing knowledge of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Airflow DAGs. See [Introduction to Airflow DAGs](/docs/learn/dags).
* Date and time modules in Python3. See the [Python documentation on the `datetime` package](https://docs.python.org/3/library/datetime.html) and the [`pendulum` documentation](https://pendulum.eustace.io/docs/).

### DAG run timestamps

A DAG run is a single execution of a DAG attached to a point in time. On the DAG run details page you can see different timestamps associated with your DAG run.

<Frame>
  <img alt="Screenshot DAG run details page" />
</Frame>

* **Logical Date**: The point in time after which a specific DAG run can start. This timestamp is displayed prominently in the Airflow UI as the main date the DAG run is associated with. The logical date can be set to `None` by the user for DAGs that are triggered using the Airflow REST API or Airflow UI.
* **Run after**: The point in time after which a specific DAG run can start. If a logical date is supplied, the run after date is set to the logical date. If the logical date is `None` the run after date is set to the current time as soon as the DAG run is triggered.
* **Start** and **Start Date**: The time the DAG run actually started. This timestamp is unrelated to the [DAG parameter](#schedule-dag-parameters) `start_date`.
* **End** and **End Date**: The time the DAG run finished. This timestamp is unrelated to the [DAG parameter](#schedule-dag-parameters) `end_date`.
* **Duration** and **Run Duration**: The time it took for the DAG run to complete, difference between the start and end date timestamps.
* **Run ID**: The unique identifier for the DAG run. The run ID is a combination of the type of DAG run, for example `scheduled` and the logical date. If the logical date is `None`, the run after date is used with an added random string suffix to ensure uniqueness. The run ID is used to identify the DAG run in the Airflow metadata database.
* **Last Scheduling Decision**: The last time a scheduler attempted to schedule task instances for this DAG run.
* **Queued at**: The time the first task instance was queued for this DAG run.

There are two additional timestamps that are only meaningful when using the `CronDataIntervalTimetable`.

* **Data Interval Start**: When the `CronDataIntervalTimetable` is used, the data interval start timestamp of a DAG run is equivalent to the run after date of the previous scheduled DAG run of the same DAG. When using other schedules, the data interval start timestamp is equivalent to the run after date of the current DAG run. If the DAG run is triggered with the logical date set to `None`, the data interval start timestamp is also `None`.
* **Data Interval End**: When the `CronDataIntervalTimetable` is used, the data interval start timestamp of a DAG run is equivalent to the run after date of the current scheduled DAG run. If the logical date is set to `None`, the data interval end timestamp is also `None`.

For manual Dag runs via the Airflow UI you have the option to choose **Specify Manually** to provide a custom **Data Interval**.

<Frame>
  <img alt="Image of the Trigger Dag form with a manually specified data interval." />
</Frame>

This will create one Dag run where the context variables `data_interval_start` and `data_interval_end` are set to the custom values you provided. This is different from creating a [backfill](/docs/learn/rerunning-dags#backfill) where one Dag run per schedule interval is created for the time period you specified.

For more information on the data intervals and the differences between the `CronDataIntervalTimetable` and the `CronTriggerTimetable`, see the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timetable.html#timetables-comparisons).

### Schedule DAG parameters

The following parameters ensure your DAGs run at the correct time:

* **`start_date`**: The timestamp after which this DAG can start running. When using the `CronDataIntervalTimetable`, the `start_date` is the moment in time after which the first data interval can start. Default: `None`.
* **`schedule`**: Defines the rules according to which DAG runs are scheduled. This parameter accepts cron expressions, timedelta objects, timetables, and lists of assets. Default: `None`.
* **`end_date`**: The date beyond which the DAG won't run. Default: `None`.
* **`catchup`**: A boolean that determines whether the DAG should automatically fill all runs between its `start_date` and the current date. Default: `False`. Aside from this automatic behavior, you can also manually trigger DAG runs for any date in the past. See [Backfills](/docs/learn/rerunning-dags#backfill) for more information.

The following code snippet defines a DAG with a `start_date` of April 1, 2025, a `schedule` of `@daily`, and an `end_date` of April 1, 2026. The DAG will run every day at midnight UTC, starting on April 1, 2025, and ending on March 31, 2026. It doesn't catch up on any missed runs automatically.

```python wrap theme={null}
from pendulum import datetime
from airflow.sdk import dag 

@dag( 
    start_date=datetime(2025, 4, 1),
    schedule="@daily", 
    end_date=datetime(2026, 4, 1),
)
```

<Warning>
  Don't make your DAG's schedule dynamic (for example, `datetime.now()`)! This will cause an error in the Scheduler.
</Warning>

## Time-based schedules

For pipelines with straightforward scheduling needs, you can define a `schedule` in your DAG using:

* A cron expression.
* A cron preset.
* A timedelta object.

Cron expressions are passed to a timetable under the hood. The default timetable used is the CronTriggerTimetable. You can use the [`[scheduler].create_cron_data_intervals` configuration](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#create-cron-data-intervals) option to switch to using the `CronDataIntervalTimetable` instead, which was the behavior in previous Airflow versions. See [Timetable comparisons](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timetable.html#timetables-comparisons) for more information.

### Cron expressions

You can pass any cron expression as a string to the `schedule` parameter in your DAG. For example, if you want to schedule your DAG at 4:05 AM every day, you would use `schedule='5 4 * * *'`.

If you need help creating the correct cron expression, see [crontab guru](https://crontab.guru/).

### Cron presets

Airflow can utilize cron presets for common, basic schedules. For example, `schedule='@hourly'` will schedule the DAG to run at the beginning of every hour. For the full list of presets, see [Cron Presets](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/cron.html#cron-presets).

### Timedelta objects

If you want to schedule your DAG on a particular cadence (hourly, every 5 minutes, etc.) rather than at a specific time, you can pass a `timedelta` object imported from the [`datetime` package](https://docs.python.org/3/library/datetime.html) or a `duration` object from the [`pendulum` package](https://pendulum.eustace.io/docs/) to the `schedule` parameter. For example, `schedule=timedelta(minutes=30)` will run the DAG every thirty minutes, and `schedule=timedelta(days=1)` will run the DAG every day.

### Limitations of cron-based schedules

Cron-based schedules run into limitations when dealing with irregular time-based schedules. For example when:

* Scheduling a DAG at different times on different days. For example, 2:00 PM on Thursdays and 4:00 PM on Saturdays.
* Scheduling a DAG daily except for holidays.
* Scheduling a DAG at multiple times daily with uneven intervals. For example, 1:00 PM and 4:30 PM.

Such schedules can be created using [timetables](#timetables).

## Data-aware scheduling

With **Assets**, you can make Airflow aware of updates to data objects. Using that awareness, Airflow can schedule other DAGs when there are updates to these assets. To create an asset-based schedule, pass the names of the asset(s) to the `schedule` parameter. You can create schedules based on conditional logic involving multiple assets, or even combine asset-based schedules with time-based schedules.

<details>
  <summary>Simple</summary>

  ```python wrap theme={null}
  # from airflow.sdk import Asset

  my_asset_1 = Asset("my_asset_1")
  my_asset_2 = Asset("my_asset_2")

  @dag(
      schedule=[my_asset_1, my_asset_2],  
      # Passing a list of datasets will create an AND condition
  )
  ```

  This DAG runs when both `my_asset_1` and `my_asset_2` are updated at least once.
</details>

<details>
  <summary>Conditional</summary>

  ```python wrap theme={null}
  # from airflow.sdk import Asset

  my_asset_1 = Asset("my_asset_1")
  my_asset_2 = Asset("my_asset_2")

  @dag(
      schedule=(my_asset_1 | my_asset_2),  
      # Use () instead of [] to be able to use conditional dataset scheduling!
  )
  ```

  This DAG runs when either `my_asset_1` or `my_asset_2` is updated.
</details>

<details>
  <summary>Time</summary>

  ```python wrap theme={null}
  # from airflow.sdk import Asset
  # from airflow.timetables.assets import AssetOrTimeSchedule
  # from airflow.timetables.trigger import CronTriggerTimetable

  my_asset_1 = Asset("my_asset_1")
  my_asset_2 = Asset("my_asset_2")

  @dag(
      schedule=AssetOrTimeSchedule(
          timetable=CronTriggerTimetable("0 0 * * *", timezone="UTC"),
          assets=(my_asset_1 | my_asset_2),
      ), 
      # Use () instead of [] to be able to use conditional dataset scheduling!
  )
  ```

  This DAG runs every day at midnight UTC and, additionally, whenever either `my_asset_1` or `my_asset_2` is updated.
</details>

Assets can be updated by any tasks in any DAG of the same Airflow environment, by calls to the [asset endpoint of the Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html), or manually in the Airflow UI.

To learn more about assets and data driven scheduling, see [Assets and Data-Aware Scheduling in Airflow](/docs/learn/airflow-datasets) guide.
Assets can be combined with AssetWatchers to create event-driven schedules. For more information, see [Event-Driven Scheduling](/docs/learn/airflow-event-driven-scheduling).

## Timetables

Each time-based schedule in Airflow is implemented using a timetable under the hood. There are a couple of built-in timetables, including the `CronTriggerTimetable` and `CronDataIntervalTimetable`. If no timetable exists for your use case, you have the option to [create your own custom](https://airflow.apache.org/docs/apache-airflow/stable/howto/timetable.html).

### Continuous timetable

You can run a DAG continuously with a pre-defined timetable. To use the [ContinuousTimetable](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/timetables/simple/index.html#module-airflow.timetables.simple.ContinuousTimetable), set the schedule of your DAG to `"@continuous"` and set `max_active_runs` to 1.

```python wrap theme={null}
@dag(
    start_date=datetime(2025, 4, 1),
    schedule="@continuous",
    max_active_runs=1,  
)
```

This schedule will create one continuous DAG run, with a new run starting as soon as the previous run has completed, regardless of whether the previous run succeeded or failed. Using a ContinuousTimetable is especially useful when [sensors](/docs/learn/what-is-a-sensor) or [deferrable operators](/docs/learn/deferrable-operators) are used to wait for highly irregular events in external data tools.

<Warning>
  Airflow is designed to handle orchestration of data pipelines in batches, and this feature isn't intended for streaming or low-latency processes. If you need to run pipelines more frequently than every minute, consider using Airflow in combination with tools designed specifically for that purpose like [Apache Kafka](/docs/learn/airflow-kafka).
</Warning>


# Set up your IDE for data engineering
Source: https://astronomer.io/docs/learn/set-up-your-ide-for-data-engineering

Set up your IDE for local Airflow data engineering with AI tools.

After you use the Astro CLI to spin up an [Airflow project locally](/docs/learn/run-airflow-locally), you can develop your Dags by editing the files in the project folder in any way you like.

Some developers prefer to use an integrated development environment (IDE), such as [VS Code](https://code.visualstudio.com/docs) or [PyCharm](https://www.jetbrains.com/help/pycharm/getting-started.html), to edit their files. These code editors come with a lot of additional functionality and extensions. A common way to improve debugging for both you and your AI agent is to run your editor's tooling inside a container built from your Airflow project, so it resolves code against the same Airflow version and Airflow provider versions your Dags run on. This gives you access to IDE features such as type checking, autocomplete, and debugging tools.

<Info>
  For a hands-on demo of using AI agents in local data engineering, watch the recording of the [Local data engineering in the agentic era](https://www.astronomer.io/events/webinars/local-data-engineering-in-the-agentic-era-video/) webinar.
</Info>

## Assumed knowledge

To get the most out of this guide, you should have:

* An Astro project on your computer. See [Run Airflow locally](/docs/learn/run-airflow-locally).

## Dev containers

A dev container runs your editor's tooling (language servers, linters, debuggers) inside the same container as your code. There are two ways to get one: attach your editor to the scheduler container that `astro dev start` is already running, or define the container declaratively in a `devcontainer.json` file that builds from your project's `Dockerfile`.

A dev container gives your editor:

* Autocomplete and type checking against the exact classes and provider packages installed in your Airflow project.
* Warnings for deprecated or unused imports before you run the Dag.
* Breakpoints and step-through debugging.

An AI agent running in that container has access to the same information, which lets it check code against the right Airflow and provider versions. See [Agentic hooks](/docs/learn/develop-dags-with-ai#agentic-hooks) for feeding those errors back to the agent automatically.

<Tip>
  If you only need a shell inside the container, `astro dev bash` is faster than attaching an IDE.
</Tip>

### VS Code

The [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) can attach to a container that's already running.

1. Start Airflow with `astro dev start`.
2. Open the command palette (Shift+Cmd+P on macOS, Shift+Ctrl+P on Windows/Linux) and run **Dev Containers: Attach to Running Container**. Select `<project>-scheduler-1`.
3. In the new window, open `/usr/local/airflow`.
4. Install the [Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python) inside the container. To skip this step on future attaches, set `dev.containers.defaultExtensions` in your VS Code settings.

You can also define your container in a `devcontainer.json` file. This is useful for sharing one VS Code setup across a team and builds a single container from your project's `Dockerfile` that you and your AI agent can use for editing, type checking, and `dag.test()`. Because the dev container doesn't start all five containers from `astro dev start`, the following example uses a dedicated, disposable SQLite metadata database for any functionality that requires database interaction.

1. In your Astro project, create a `.devcontainer` folder with a `devcontainer.json` file:

   ```json theme={null}
   {
     "name": "Astro Runtime Devcontainer",
     "build": {
       "context": "..",
       "dockerfile": "../Dockerfile"
     },
     "remoteEnv": {
       "AIRFLOW_HOME": "${containerWorkspaceFolder}",
       "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN": "sqlite:////tmp/airflow.db"
     },
     "postCreateCommand": "rm -rf /tmp/airflow.db && airflow db migrate",
     "customizations": {
       "vscode": {
         "settings": {
           "python.defaultInterpreterPath": "/usr/local/bin/python"
         },
         "extensions": [
           "ms-python.python",
           "ms-python.vscode-pylance",
           "ms-vscode.live-server"
         ]
       }
     }
   }
   ```

2. Open the command palette and run **Dev Containers: Reopen in Container**.

The `extensions` list decides which tooling your agent can see. For example, `ms-python.vscode-pylance` adds type checking and import resolution. See the [VS Code Extension Marketplace](https://marketplace.visualstudio.com/vscode) for more information.

### PyCharm

The PyCharm Dev Containers feature can use the same `.devcontainer/devcontainer.json` file from the preceding VS Code section.

1. Connect PyCharm to Docker: open **Settings**, go to **Build, Execution, Deployment** > **Docker**, click `+`, and [connect to your Docker daemon](https://www.jetbrains.com/help/pycharm/docker.html#connect_to_docker).

2. Add a new folder called exactly `.devcontainer` to your Airflow project's root and create a `devcontainer.json` file in it, with the same contents as in the preceding VS Code section. Open the file.

3. Click the Dev Container icon in the editor's left gutter, next to the file's first line, and select **Create Dev Container and Mount Sources…**, then choose your backend IDE.

   <Frame>
     <img alt="The Dev Container gutter icon appears next to the first line of an open devcontainer.json file" />
   </Frame>

   <Note>
     If the icon doesn't appear, restart PyCharm. It might not detect a `devcontainer.json` file created while the project was already open.
   </Note>

4. Watch the build progress in the **Services** tool window (**View** > **Tool Windows** > **Services**), then click **Open Project** after it finishes.

## Debug with `dag.test()`

`dag.test()` runs every task in a Dag inside a single Python process, without requiring a running Airflow environment. Because it runs as regular Python code, you can set breakpoints and step through task logic with your IDE's debugger.

<details>
  <summary>Decorator</summary>

  ```python {11-12} theme={null}
  from airflow.sdk import dag
  from airflow.providers.standard.operators.empty import EmptyOperator

  @dag
  def my_dag():

      t1 = EmptyOperator(task_id="t1")

  dag_object = my_dag()

  if __name__ == "__main__":
      dag_object.test()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python {10-11} theme={null}
  from airflow.sdk import DAG
  from airflow.providers.standard.operators.empty import EmptyOperator

  with DAG(
      dag_id="my_dag",
  ) as dag:  # assigning the context to an object is mandatory for using dag.test()

      t1 = EmptyOperator(task_id="t1")

  if __name__ == "__main__":
      dag.test()
  ```
</details>

For more information, see [Debug interactively with `dag.test()`](/docs/learn/testing-airflow#debug-interactively-with-dag-test).


# How to share code between multiple Airflow projects
Source: https://astronomer.io/docs/learn/sharing-code-multiple-projects

A description of the various ways to reuse and share code between multiple projects, with pros and cons of each solution.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

After you [set up an Astro project](/docs/cli/v1.43/develop-project), and you start implementing some pipelines, extracting several Python functions for reusability, and scaling other operations, you've now got multiple Airflow deployments. How do you reuse code between your projects? In this guide, you'll learn about various options for reusing code and their pros and cons.

Specifically, this guide demonstrates three options, ordered from simple implementation, but poor reusability, to comprehensive implementation, but excellent reusability:

| Solution                                                 | When to use                                                               |
| -------------------------------------------------------- | ------------------------------------------------------------------------- |
| Shared Python code in same file                          | When reusing code only within a single script                             |
| Shared Python code in `/include` folder                  | When reusing code in multiple scripts, but within the same Git repository |
| Shared Python code in Python package in separate project | When reusing code in multiple Git projects                                |

The following DAG, which queries a database and returns results, is used in each section below to highlight the different solutions. The tasks in this DAG perform the same business logic twice. Both functions (`_get_locations()` and `_get_purchases()`) instantiate a database client, execute a query, and return the result. The only difference is the query being executed. Any change to the database connection logic now requires two changes. If you continue copy-pasting these functions for running more queries, you have multiple copies of the same business logic, which also requires multiple code changes when you want to modify the database connection logic.

```python {7-9,11-12,14-16,18-19} wrap theme={null}
import datetime

from airflow import DAG
from airflow.operators.python import PythonOperator

with DAG(dag_id="example", schedule=None, start_date=datetime.datetime(2023, 1, 1)):
    def _get_locations():
        # Pseudocode:
        client = db.client()
        query = "SELECT store_id, city FROM stores"
        result = client.execute(query)
        return result

    def _get_purchases():
        # Pseudocode:
        client = db.client()
        query = "SELECT customer_id, string_agg(store_id, ',') FROM customers GROUP BY customer_id"
        result = client.execute(query)
        return result

    PythonOperator(task_id="get_locations", python_callable=_get_locations)
    PythonOperator(task_id="get_purchases", python_callable=_get_purchases)
```

## Shared Python code in same file

To reduce the burden of maintaining the same business logic multiple times, and have only one way of querying the database, you can extract the code into a single function within the same DAG file:

```python wrap theme={null}
def query_db(query):
    # Pseudocode:
    client = db.client()
    result = client.execute(query)
    return result
```

This function takes an argument `query`, and the database connection logic is only defined once, regardless what query is given. This way you don't have to maintain the same business logic in multiple places anymore. To use this function, reference it in your DAG script:

```python {6-10} wrap theme={null}
import datetime

from airflow import DAG
from airflow.operators.python import PythonOperator

def query_db(query):
    # Pseudocode:
    client = db.client()
    result = client.execute(query)
    return result

with DAG(dag_id="example", schedule=None, start_date=datetime.datetime(2023, 1, 1)):
    PythonOperator(task_id="get_locations", python_callable=query_db, op_kwargs={"query": "SELECT store_id, city FROM stores"})
    PythonOperator(task_id="get_purchases", python_callable=query_db, op_kwargs={"query": "SELECT customer_id, string_agg(store_id, ',') FROM customers GROUP BY customer_id"})
```

This solution is technically simple; you define one single function and reference that function in multiple Airflow tasks, in the same script. This improves the code because a change to the database connection logic requires only a single change instead of two. However, this solution is limited to a single script. You can't reuse the function in another script (without importing). Let's look at a solution in the next section.

## Shared Python code in `/include` folder

To reuse a piece of code across multiple scripts it needs to be accessible in a shared location. The Astro Runtime Docker image provides a convenient mechanism for that, the `/include` folder:

You can store the function in a separate file, for example `/include/db.py`:

```python wrap theme={null}
def query_db(query):
    # Pseudocode:
    client = db.client()
    result = client.execute(query)
    return result
```

And then import the function from your DAG:

```python {6} wrap theme={null}
import datetime

from airflow import DAG
from airflow.operators.python import PythonOperator

from include.db import query_db

with DAG(dag_id="example", schedule=None, start_date=datetime.datetime(2023, 1, 1)):
    PythonOperator(task_id="get_locations", python_callable=query_db, op_kwargs={"query": "SELECT store_id, city FROM stores"})
    PythonOperator(task_id="get_purchases", python_callable=query_db, op_kwargs={"query": "SELECT customer_id, string_agg(store_id, ',') FROM customers GROUP BY customer_id"})
```

The benefit of this solution is that the `query_db` function can be imported from multiple scripts (within the same Git repository).

## Shared Python code in Python package in separate project

In some cases, you might have code that needs to be shared across different Airflow deployments. For example, if you're onboarding multiple teams to the Astronomer platform and each team has their own code repository. This means you can't reuse the code in the `/include` folder, because it resides in a different Git repository.

To reuse code over multiple projects, you need to store it in a separate Git repository which can be reused by multiple projects. The best way to do this is to create your own Python package from the repository you want to be available to multiple projects. This takes a bit more work to set up, but enables multiple teams using multiple Git repositories to maintain a single source of code. You can see an example Python package in [this repository](https://github.com/astronomer/custom-package-demo).

The number of options for developing, building, and releasing a Python package are limitless and this guide only provides general guidance. See [**Structuring your project**](https://docs.python-guide.org/writing/structure) and [**Packaging Python projects**](https://packaging.python.org/en/latest/tutorials/packaging-projects) for more information on Python packaging.

Setting up a custom Python package requires roughly the following steps:

1. Create a separate Git repository for your shared code.
2. Write a `pyproject.toml` file. This is a configuration file which contains the build requirements of your Python project. You can find an example [here](https://github.com/astronomer/custom-package-demo/blob/main/pyproject.toml).
3. Create a folder for your code, for example `my_company_airflow`.
4. Create a folder for tests, for example `tests`.
5. Create a CI/CD pipeline to test, build, and release your package. You can see an example GitHub Actions workflow in the [custom package demo](https://github.com/astronomer/custom-package-demo/tree/main/.github/workflows).
6. Ensure your setup works correctly by building and releasing a first version of the package.
7. Validate the package by installing it in a project using the `requirements.txt` file.

After completing the preceding steps, you can now add shared code to the Python package so that other projects can use it. The code example must be added in a module in your Python package, for example `my_company_airflow/db.py`:

```python wrap theme={null}
def query_db(query):
    # Pseudocode:
    client = db.client()
    result = client.execute(query)
    return result
```

After installing the package, you can then import the function in your DAGs as:

```python {6} wrap theme={null}
import datetime

from airflow import DAG
from airflow.operators.python import PythonOperator

from my_company_airflow.db import query_db

with DAG(dag_id="example", schedule=None, start_date=datetime.datetime(2023, 1, 1)):
    PythonOperator(task_id="get_locations", python_callable=query_db, op_kwargs={"query": "SELECT store_id, city FROM stores"})
    PythonOperator(task_id="get_purchases", python_callable=query_db, op_kwargs={"query": "SELECT customer_id, string_agg(store_id, ',') FROM customers GROUP BY customer_id"})
```

The previous steps describe roughly how to set up a Python package, but your process might differ depending on your setup and how your organization manages code. In general, these are some key pointers and considerations for setting up a custom Python package:

* Think about how you distribute the Python package. Do you require/have an internal repository for storing Python packages such as [Artifactory](https://jfrog.com/artifactory) or [devpi](https://www.devpi.net)?
* Determine who is responsible for maintaining the shared Git repository.
* Set developments standards from the beginning, such as Flake8 linting and Black formatting.
* Ensure the end-to-end CI/CD pipeline works first, then start developing application code.

## Plan for the future

This guide demonstrates several solutions for sharing code, from code in a single file to code across multiple Git repositories. If you're currently only deploying from a single Git repository to the Astronomer platform, but plan for multiple teams in the future, we advise you start with shared code in a separate Git repository. Setting standards and best practices from the beginning is easier than introducing changes in hindsight.


# Airflow task groups
Source: https://astronomer.io/docs/learn/task-groups

Follow Astronomer’s step-by-step guide to use task groups for organizing tasks within the grid view of the Airflow user interface.

Airflow [task groups](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#taskgroups) are a tool to organize tasks into groups within your DAGs. Using task groups allows you to:

* Organize complicated DAGs, visually grouping tasks that belong together in the Airflow UI **Grid View**.
* Apply `default_args` to sets of tasks, instead of at the DAG level using [DAG parameters](/docs/learn/dags#dag-level-parameters).
* [Dynamically map](/docs/learn/dynamic-tasks) over groups of tasks, enabling complex dynamic patterns.
* Turn task patterns into modules that can be reused across DAGs or Airflow instances.

In this guide, you'll learn how to create and use task groups in your DAGs. You can find many example DAGs using task groups on the [Astronomer GitHub](https://github.com/astronomer/webinar-task-groups).

<Frame>
  <img alt="Task group intro gif" />
</Frame>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).

## When to use task groups

Task groups are most often used to visually organize complicated DAGs. For example, you might use task groups:

* In big ELT/ETL DAGs, where you have a task group per table or schema.
* In MLOps DAGs, where you have a task group per model being trained.
* In DAGs owned by several teams, where you have task groups to visually separate the tasks that belong to each team. Although in this case, it might be better to separate the DAG into multiple DAGs and use [Assets](/docs/learn/airflow-datasets) to connect them.
* When you are using the same patterns of tasks in multiple DAGs and want to create a reusable module.
* When you have an input of unknown length, for example an unknown number of files in a directory. You can use task groups to [dynamically map](#generate-task-groups-dynamically-at-runtime) over the input and create a task group performing sets of actions for each file. This is the only way to dynamically map sequential tasks in Airflow.

## Define task groups

There are two ways to define task groups in your DAGs:

* Use the `TaskGroup` class to create a task group context.
* Use the `@task_group` decorator on a Python function.

In most cases, it is a matter of personal preference which method you use. The only exception is when you want to [dynamically map](/docs/learn/dynamic-tasks) over a task group; this is possible only when using `@task_group`.

The following code shows how to instantiate a simple task group containing two sequential tasks. You can use dependency operators (`<<` and `>>`) both within and between task groups in the same way that you can with individual tasks.

<details>
  <summary>Decorator</summary>

  ```python wrap theme={null}
  # from airflow.decorators import task_group

  t0 = EmptyOperator(task_id='start')

  # Start task group definition
  @task_group(group_id='my_task_group')
  def tg1():
      t1 = EmptyOperator(task_id='task_1')
      t2 = EmptyOperator(task_id='task_2')

      t1 >> t2
  # End task group definition

  t3 = EmptyOperator(task_id='end')

  # Set task group's (tg1) dependencies
  t0 >> tg1() >> t3
  ```
</details>

<details>
  <summary>Context</summary>

  ```python wrap theme={null}
  # from airflow.utils.task_group import TaskGroup

  t0 = EmptyOperator(task_id='start')

  # Start task group definition
  with TaskGroup(group_id='my_task_group') as tg1:
      t1 = EmptyOperator(task_id='task_1')
      t2 = EmptyOperator(task_id='task_2')

      t1 >> t2
  # End task group definition

  t3 = EmptyOperator(task_id='end')

  # Set task group's (tg1) dependencies
  t0 >> tg1 >> t3
  ```
</details>

Task groups are shown in both the Grid and the Graph of your DAG:

<Frame>
  <img alt="Task groups simple example - grid" />
</Frame>

<Frame>
  <img alt="Task groups simple example - graph" />
</Frame>

## Task group parameters

You can use parameters to customize individual task groups. The two most important parameters are the `group_id` which determines the name of your task group, as well as the `default_args` which will be passed to all tasks in the task group. The following examples show task groups with some commonly configured parameters:

<details>
  <summary>Decorator</summary>

  ```python wrap theme={null}
  @task_group(
      group_id="task_group_1",
      default_args={"conn_id": "postgres_default"},
      tooltip="This task group is very important!",
      prefix_group_id=True,
      # parent_group=None,
      # dag=None,
  )
  def tg1():
      t1 = EmptyOperator(task_id="t1")

  tg1()
  ```
</details>

<details>
  <summary>Context</summary>

  ```python wrap theme={null}
  with TaskGroup(
      group_id="task_group_2",
      default_args={"conn_id": "postgres_default"},
      tooltip="This task group is also very important!",
      prefix_group_id=True,
      # parent_group=None,
      # dag=None,
      # add_suffix_on_collision=True, # resolves group_id collisions by adding a suffix
  ) as tg2:
      t1 = EmptyOperator(task_id="t1")
  ```
</details>

## `task_id` in task groups

When your task is within a task group, your callable `task_id` will be `group_id.task_id`. This ensures the `task_id` is unique across the DAG. It is important that you use this format when referring to specific tasks when working with [XComs](/docs/learn/airflow-passing-data-between-tasks) or [branching](/docs/learn/airflow-branch-operator). You can disable this behavior by setting the [task group parameter](#task-group-parameters) `prefix_group_id=False`.

For example, the `task_1` task in the following DAG has a `task_id` of `my_outer_task_group.my_inner_task_group.task_1`.

<details>
  <summary>Decorator</summary>

  ```python wrap theme={null}
  @task_group(group_id="my_outer_task_group")
  def my_outer_task_group():
      @task_group(group_id="my_inner_task_group")
      def my_inner_task_group():
          EmptyOperator(task_id="task_1")

      my_inner_task_group()

  my_outer_task_group()
  ```
</details>

<details>
  <summary>Context</summary>

  ```python wrap theme={null}
  with TaskGroup(group_id="my_outer_task_group") as tg1:
      with TaskGroup(group_id="my_inner_task_group") as tg2:
          EmptyOperator(task_id="task_1")
  ```
</details>

## Pass data through task groups

When you use the `@task_group` decorator, you can pass data through the task group just like with regular `@task` decorators:

```python expandable wrap theme={null}
from airflow.decorators import dag, task, task_group
from pendulum import datetime
import json


@dag(start_date=datetime(2023, 8, 1), schedule=None, catchup=False)
def task_group_example():
    @task
    def extract_data():
        data_string = '{"1001": 301.27, "1002": 433.21, "1003": 502.22}'
        order_data_dict = json.loads(data_string)
        return order_data_dict

    @task
    def transform_sum(order_data_dict: dict):
        total_order_value = 0
        for value in order_data_dict.values():
            total_order_value += value

        return {"total_order_value": total_order_value}

    @task
    def transform_avg(order_data_dict: dict):
        total_order_value = 0
        for value in order_data_dict.values():
            total_order_value += value
            avg_order_value = total_order_value / len(order_data_dict)

        return {"avg_order_value": avg_order_value}

    @task_group
    def transform_values(order_data_dict):
        return {
            "avg": transform_avg(order_data_dict),
            "total": transform_sum(order_data_dict),
        }

    @task
    def load(order_values: dict):
        print(
            f"""Total order value is: {order_values['total']['total_order_value']:.2f} 
            and average order value is: {order_values['avg']['avg_order_value']:.2f}"""
        )

    load(transform_values(extract_data()))


task_group_example()
```

The resulting DAG is shown in the following image:

<Frame>
  <img alt="Decorated task group" />
</Frame>

There are a few things to consider when passing information into and out of task groups:

* If downstream tasks require the output of tasks that are in the task group decorator, then the task group function must return a result. In the previous example, a dictionary with two values was returned, one from each of the tasks in the task group, that are then passed to the downstream `load()` task.
* If your task group function returns an output that another task takes as an input, Airflow can infer the task group and task dependency with the TaskFlow API. If your task group function's output isn't used as a task input, you must use the bit-shift operators (`<<` or `>>`) to define downstream dependencies to the task group.

## Generate task groups dynamically at runtime

You can use [dynamic task mapping](/docs/learn/dynamic-tasks) with the `@task_group` decorator to dynamically map over task groups. The following DAG shows how you can dynamically map over a task group with different inputs for a given parameter.

```python expandable wrap theme={null}
from airflow.decorators import dag, task_group, task
from pendulum import datetime


@dag(
    start_date=datetime(2022, 12, 1),
    schedule=None,
    catchup=False,
)
def task_group_mapping_example():
    # creating a task group using the decorator with the dynamic input my_num
    @task_group(group_id="group1")
    def tg1(my_num):
        @task
        def print_num(num):
            return num

        @task
        def add_42(num):
            return num + 42

        print_num(my_num) >> add_42(my_num)

    # a downstream task to print out resulting XComs
    @task
    def pull_xcom(**context):
        pulled_xcom = context["ti"].xcom_pull(
            # reference a task in a task group with task_group_id.task_id
            task_ids=["group1.add_42"],
            # only pull Xcom from specific mapped task group instances (2.5 feature)
            map_indexes=[2, 3],
            key="return_value",
        )

        # will print out a list of results from map index 2 and 3 of the add_42 task
        print(pulled_xcom)

    # creating 6 mapped task group instances of the task group group1 (2.5 feature)
    tg1_object = tg1.expand(my_num=[19, 23, 42, 8, 7, 108])

    # setting dependencies
    tg1_object >> pull_xcom()


task_group_mapping_example()
```

This DAG dynamically maps over the task group `group1` with different inputs for the `my_num` parameter. 6 mapped task group instances are created, one for each input. Within each mapped task group instance two tasks will run using that instances' value for `my_num` as an input. The `pull_xcom()` task downstream of the dynamically mapped task group shows how to access a specific [XCom](/docs/learn/airflow-passing-data-between-tasks) value from a list of mapped task group instances (`map_indexes`).

For more information on dynamic task mapping, including how to map over multiple parameters, see [Dynamic Tasks](/docs/learn/dynamic-tasks).

## Order task groups

By default, using a loop to generate your task groups will put them in parallel. If your task groups are dependent on elements of another task group, you'll want to run them sequentially. For example, when loading tables with foreign keys, your primary table records need to exist before you can load your foreign table.

In the following example, the third task group generated in the loop has a foreign key constraint on both previously generated task groups (first and second iteration of the loop), so you'll want to process it last. To do this, you'll create an empty list and append your task group objects as they are generated. Using this list, you can reference the task groups and define their dependencies to each other:

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  groups = []
  for g_id in range(1,4):
      tg_id = f"group{g_id}"

      @task_group(group_id=tg_id)
      def tg1():
          t1 = EmptyOperator(task_id="task1")
          t2 = EmptyOperator(task_id="task2")

          t1 >> t2

          if tg_id == "group1":
              t3 = EmptyOperator(task_id="task3")
              t1 >> t3

      groups.append(tg1())

  [groups[0] , groups[1]] >> groups[2]
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  groups = []
  for g_id in range(1,4):
      tg_id = f"group{g_id}"
      with TaskGroup(group_id=tg_id) as tg1:
          t1 = EmptyOperator(task_id="task1")
          t2 = EmptyOperator(task_id="task2")

          t1 >> t2

          if tg_id == "group1":
              t3 = EmptyOperator(task_id="task3")
              t1 >> t3

          groups.append(tg1)

  [groups[0] , groups[1]] >> groups[2]
  ```
</details>

The following image shows how these task groups appear in the Airflow UI:

<Frame>
  <img alt="Task group Dependencies" />
</Frame>

This example also shows how to add an additional task to `group1` based on your `group_id`, Even when you're creating task groups in a loop to take advantage of patterns, you can still introduce variations to the pattern while avoiding code redundancies.

## Nest task groups

For additional complexity, you can nest task groups by defining a task group indented within another task group. There is no limit to how many levels of nesting you can have.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  groups = []
  for g_id in range(1,3):
      @task_group(group_id=f"group{g_id}")
      def tg1():
          t1 = EmptyOperator(task_id="task1")
          t2 = EmptyOperator(task_id="task2")

          sub_groups = []
          for s_id in range(1,3):
              @task_group(group_id=f"sub_group{s_id}")
              def tg2():
                  st1 = EmptyOperator(task_id="task1")
                  st2 = EmptyOperator(task_id="task2")

                  st1 >> st2
              sub_groups.append(tg2())

          t1 >> sub_groups >> t2
      groups.append(tg1())

  groups[0] >> groups[1]
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  groups = []
  for g_id in range(1,3):
      with TaskGroup(group_id=f"group{g_id}") as tg1:
          t1 = EmptyOperator(task_id="task1")
          t2 = EmptyOperator(task_id="task2")

          sub_groups = []
          for s_id in range(1,3):
              with TaskGroup(group_id=f"sub_group{s_id}") as tg2:
                  st1 = EmptyOperator(task_id="task1")
                  st2 = EmptyOperator(task_id="task2")

                  st1 >> st2
                  sub_groups.append(tg2)

          t1 >> sub_groups >> t2
          groups.append(tg1)

  groups[0] >> groups[1]
  ```
</details>

The following image shows the expanded view of the nested task groups in the Airflow UI:

<Frame>
  <img alt="Nested task groups" />
</Frame>

## Custom task group classes

If you use the same patterns of tasks in several DAGs or Airflow instances, it may be useful to create a custom task group class module. To do so, you need to inherit from the `TaskGroup` class and then define your tasks within that custom class. You also need to use `self` to assign the task to the task group. Other than that, the task definitions will be the same as if you were defining them in a DAG file.

```python wrap theme={null}
from airflow.utils.task_group import TaskGroup
from airflow.decorators import task


class MyCustomMathTaskGroup(TaskGroup):
    """A task group summing two numbers and multiplying the result with 23."""

    # defining defaults of input arguments num1 and num2
    def __init__(self, group_id, num1=0, num2=0, tooltip="Math!", **kwargs):
        """Instantiate a MyCustomMathTaskGroup."""
        super().__init__(group_id=group_id, tooltip=tooltip, **kwargs)

        # assign the task to the task group by using `self`
        @task(task_group=self)
        def task_1(num1, num2):
            """Adds two numbers."""
            return num1 + num2

        @task(task_group=self)
        def task_2(num):
            """Multiplies a number by 23."""
            return num * 23

        # define dependencies
        task_2(task_1(num1, num2))
```

In the DAG, you import your custom TaskGroup class and instantiate it with the values for your custom arguments:

```python wrap theme={null}
from airflow.decorators import dag, task
from pendulum import datetime
from include.custom_task_group import MyCustomMathTaskGroup


@dag(
    start_date=datetime(2023, 8, 1),
    schedule=None,
    catchup=False,
    tags=["@task_group", "task_group"],
)
def custom_tg():
    @task
    def get_num_1():
        return 5

    tg1 = MyCustomMathTaskGroup(group_id="my_task_group", num1=get_num_1(), num2=19)

    @task
    def downstream_task():
        return "hello"

    tg1 >> downstream_task()


custom_tg()
```

The resulting image shows the custom templated task group which can now be reused in other DAGs with different inputs for `num1` and `num2`.

<Frame>
  <img alt="Custom task group" />
</Frame>


# Use Airflow templates
Source: https://astronomer.io/docs/learn/templating

Learn about Jinja templating in Apache Airflow and see examples of how to pass dynamic information into task instances at runtime.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

Templating allows you to pass dynamic information into task instances at runtime. For example, you can run the following command to print the day of the week every time you run a task:

```python wrap theme={null}
BashOperator(
    task_id="print_day_of_week",
    bash_command="echo Today is {{ execution_date.format('dddd') }}",
)
```

In this example, the value in the double curly braces `{{ }}` is the templated code that is evaluated at runtime. If you execute this code on a Wednesday, the `BashOperator`  prints `Today is Wednesday`. Templates have numerous applications. For example, you can use templating to create a new directory named after a task's execution date for storing daily data (`/data/path/20210824`). Alternatively, you can select a specific partition (`/data/path/yyyy=2021/mm=08/dd=24`) so that only the relevant data for a given execution date is scanned.

Airflow leverages [Jinja](https://jinja.palletsprojects.com), a Python templating framework, as its templating engine. In this guide, you'll learn the following:

* How to apply Jinja templates in your code.
* Which variables and functions are available when templating.
* Which operator fields can be templated and which can't.
* How to validate templates.
* How to apply custom variables and functions when templating.
* How to render templates to strings and native Python code.

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Astronomer Academy: [Airflow: Templating](https://academy.astronomer.io/astro-runtime-templating) module.
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Jinja templating. See [Jinja basics](https://jinja.palletsprojects.com/en/3.1.x/api/#basics).

## Template variables in Airflow

Templating in Airflow works the same as Jinja templating in Python. You enclose the code you want evaluated between double curly braces, and the expression is evaluated at runtime.

Some of the most commonly used Airflow variables that you can use in templates are:

* `{{ ds }}`: The DAG Run’s logical date as `YYYY-MM-DD`.
* `{{ ds_nodash }}`: The DAG run’s logical date as `YYYYMMDD`.
* `{{ data_interval_start }}`: The start of the data interval.
* `{{ data_interval_end }}`: The end of the data interval.

<Tip>
  To use a Jinja template in a Python f-string, add extra braces around the Jinja template. For example, `name_string = f"my name is {{{{ var.value.get('var_name') }}}}"`
</Tip>

For a complete list of the available variables, see the Airflow [Templates reference](https://airflow.apache.org/docs/apache-airflow/stable/macros-ref.html#default-variables).

In Airflow 2.10+, it is possible to pass a Python callable to templateable fields instead of a Jinja template, see [Use a python callable for template fields](#use-a-python-callable-for-template-fields).

## Templateable fields and scripts

Templates can't be applied to all arguments of an operator. Two attributes in the `BaseOperator` define where you can use templated values:

* `template_fields`: Defines which operator arguments can use templated values.
* `template_ext`: Defines which file extensions can use templated values.

The following example shows a simplified version of the `BashOperator`:

```python wrap theme={null}
class BashOperator(BaseOperator):
    template_fields = ('bash_command', 'env')  # defines which fields are templateable
    template_ext = ('.sh', '.bash')  # defines which file extensions are templateable

    def __init__(
        self,
        *,
        bash_command,
        env: None,
        output_encoding: 'utf-8',
        **kwargs,
    ):
        super().__init__(**kwargs)
        self.bash_command = bash_command  # templateable (can also give path to .sh or .bash script)
        self.env = env  # templateable
        self.output_encoding = output_encoding  # not templateable
```

The `template_fields` attribute holds a list of attributes that can use templated values. You can also find this list in the Airflow UI as shown in the following image:

<Frame>
  <img alt="Rendered Template view" />
</Frame>

`template_ext` contains a list of file extensions that can be read and templated at runtime. For example, instead of providing a Bash command to `bash_command`, you could provide a `.sh` script that contains a templated value:

```python wrap theme={null}
run_this = BashOperator(
    task_id="run_this",
    bash_command="script.sh",  # .sh extension can be read and templated
)
```

The `BashOperator` takes the contents of the following script, templates it, and executes it:

```bash wrap theme={null}
# script.sh
echo "Today is {{ execution_date.format('dddd') }}"
```

Templating from files speeds development because an integrated development environment (IDE) can apply language-specific syntax highlighting on the script. This wouldn't be possible if your script is defined as a big string of Airflow code.

By default, Airflow searches for the location of your scripts relative to the directory the DAG file is defined in. If your DAG is stored in `/path/to/dag.py` and your script is stored in `/path/to/scripts/script.sh`, update the value of `bash_command` in the previous example to `scripts/script.sh`.

Alternatively, you can set a base path for templates at the DAG-level with the `template_searchpath` argument. For example, the following DAG would look for `script.sh` at `/tmp/script.sh`:

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  @dag(..., template_searchpath="/tmp")
  def my_dag():
      run_this = BashOperator(task_id="run_this", bash_command="script.sh")
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  with DAG(..., template_searchpath="/tmp") as dag:
      run_this = BashOperator(task_id="run_this", bash_command="script.sh")
  ```
</details>

### Templating additional fields

If you need to template a field that isn't listed in the operator's `template_fields`, you can either set the `template_fields` attribute on a task or create a custom operator.
The following examples demonstrate how to use each method to template the `cwd` field of the `BashOperator`.

<details>
  <summary>Task</summary>

  After defining a task and assigning it to a Python variable, you can modify its `template_fields` attribute. This allows you to enable Jinja templating for any field that isn't templated by default.

  This method is preferable when you need to template a field only once.

  ```python wrap theme={null}
      from airflow.decorators import dag
      from airflow.operators.bash import BashOperator
      from airflow.utils.dates import days_ago

      @dag(schedule=None, start_date=days_ago(1))
      def templating_dag():
          bash_task = BashOperator(
              task_id="set_template_field",
              bash_command="script.sh",
              cwd="/usr/local/airflow/{{ ds }}",
          )
          bash_task.template_fields = ("bash_command", "env", "cwd")

      templating_dag()
  ```
</details>

<details>
  <summary>Operator</summary>

  You can create a custom operator with additional templated fields by subclassing the desired operator class and adding your desired field to the `template_fields` argument.
  In this example, `TemplatedBashOperator` is a new operator that inherits the behavior of `BashOperator` and allows Jinja templating of the `cwd` field.

  This method is preferred if you need to template a field repeatedly.

  For existing projects, naming your custom operator the same as the existing one simplifies refactoring by allowing you to only modify imports, minimizing the required code changes.

  ```python wrap theme={null}
      from airflow.decorators import dag
      from airflow.operators.bash import BashOperator
      from airflow.utils.dates import days_ago
      from typing import TYPE_CHECKING

      if TYPE_CHECKING:
          from collections.abc import Sequence

      class TemplatedBashOperator(BashOperator):
          template_fields: Sequence[str] = ("bash_command", "env", "cwd")

      @dag(schedule=None, start_date=days_ago(1))
      def templating_dag():
          bash_task = TemplatedBashOperator(
              task_id="custom_operator",
              bash_command="script.sh",
              cwd="/usr/local/airflow/{{ ds }}",
          )

      templating_dag()
  ```
</details>

### Disable templating

As of Airflow 2.8 it is possible to use a wrapper class to disable templating for the input to a templatable field without needing to modify the operator itself. This is useful when you want to pass a string that contains Jinja syntax to an operator without it being rendered. For example, you may want to pass a Jinja template to a `BashOperator` that won't be rendered. This can be achieved by wrapping the string into the `literal` function:

```python wrap theme={null}
from airflow.utils.template import literal

BashOperator(
    task_id="use_literal_wrapper_to_ignore_jinja_template",
    bash_command=literal("echo {{ params.the_best_number }}"),
)
```

The code above will print `{{ params.the_best_number }}` to the logs instead of showing the rendered value of `params.the_best_number`.

## Use a Python callable for template fields

In Airflow 2.10+ it is possible to pass a Python callable to templateable fields. This is especially useful when the parameter value is created using complex operations that might not be possible or are hard to read in Jinja.

The example below shows a `TriggerDagRunOperator` for which the `conf` parameter is generated based on a value in a JSON file by using a Python callable instead of a Jinja template. Note that the two keyword arguments `context` and `jinja_env` are mandatory to define in the provided callable.

```python wrap theme={null}
# from airflow.operators.trigger_dagrun import TriggerDagRunOperator

def build_conf(context, jinja_env):  # the two kwargs are mandatory
    import json

    with open("include/configuration.json", "r") as file:
        data = json.load(file)
    value = data.get("time_value", None)
    return {"sleep_time": value}

tdro = TriggerDagRunOperator(
    task_id="tdro",
    trigger_dag_id="tdro_downstream",
    conf=build_conf,
)

callable_template_custom()

```

For more information, see [Jinja Templating](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html#jinja-templating).

## Validate templates

The output of templates can be checked in both the Airflow UI and Airflow CLI. One advantage of the Airflow CLI is that you don't need to run any tasks before seeing the result.

The Airflow CLI command `airflow tasks render` renders all templateable attributes of a given task. Given a `dag_id`, `task_id`, and random `execution_date`, the command output is similar to the following example:

```bash wrap theme={null}
$ airflow tasks render example_dag run_this 2021-01-01

# ----------------------------------------------------------
# property: bash_command
# ----------------------------------------------------------
echo "Today is Friday"

# ----------------------------------------------------------
# property: env
# ----------------------------------------------------------
None
```

For this command to work, Airflow needs access to a metadata database. To set up a local SQLite database, run the following commands:

```bash wrap theme={null}
cd <your-project-directory>
export AIRFLOW_HOME=$(pwd)
airflow db migrate  # generates airflow.db, airflow.cfg, and webserver_config.py in your project dir
                    # note that in Airflow versions pre-2.7 you'll need to use `airflow db init` instead

# airflow tasks render [dag_id] [task_id] [execution_date]
```

If you use the Astro CLI, a Postgres metadata database is automatically configured for you after running `astro dev start` in your project directory. From here, you can run `astro dev run tasks render <parameters>` to test your templated values.

For most templates, this is sufficient. However, if an external system such as a variable in your production Airflow metadata database is reached by the templating logic, you must have connectivity to it.

To view the result of templated attributes after running a task in the Airflow UI, click a task and then click **Rendered** as shown in the following image:

<Frame>
  <img alt="Rendered button in the task instance popup" />
</Frame>

The Rendered Template view and the output of the templated attributes are shown in the following image:

<Frame>
  <img alt="Rendered Template view" />
</Frame>

## Macros: using custom functions and variables in templates

As discussed previously, there are several variables available during templating. A Jinja environment and Airflow runtime are different. You can view a Jinja environment as a very stripped-down Python environment. That, among other things, means modules can't be imported. For example, this command won't work in a Jinja template:

```python wrap theme={null}
from datetime import datetime

BashOperator(
    task_id="print_now",
    # raises jinja2.exceptions.UndefinedError: 'datetime' is undefined
    bash_command="echo It is currently {{ datetime.now() }}",
)
```

However, it is possible to inject functions into your Jinja environment. In Airflow, several standard Python modules are injected by default for templating, under the name macros. For example, the previous code example can be updated to use `macros.datetime`:

```python wrap theme={null}
BashOperator(
    task_id="print_now",
    # It is currently 2021-08-30 13:51:55.820299
    bash_command="echo It is currently {{ macros.datetime.now() }}",  
)
```

Airflow includes some pre-injected functions out of the box for you to use in your templates. See [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#macros) for a list of available functions. You can also load information in JSON format using `"{{ macros.json.loads(...) }}"` and information in YAML format using `"{{ macros.yaml.safe_load(...) }}"`.
Besides pre-injected functions, you can also use self-defined variables and functions in your templates. Airflow provides a convenient way to inject these into the Jinja environment. In the following example, a function is added to the DAG to print the number of days since May 1st, 2015:

```python wrap theme={null}
def days_to_now(starting_date):
    return (datetime.now() - starting_date).days
```

To use this inside a Jinja template, you can pass a dict to `user_defined_macros` in the DAG. For example:

```python wrap theme={null}
def days_to_now(starting_date):
    return (datetime.now() - starting_date).days

@dag(
    start_date=datetime(2021, 1, 1),
    schedule=None,
    user_defined_macros={
        "starting_date": datetime(2015, 5, 1),  # Macro can be a variable
        "days_to_now": days_to_now,  # Macro can also be a function
    },
)
def demo_template():
    print_days = BashOperator(
        task_id="print_days",
        # Call user defined macros
        bash_command="echo Days since {{ starting_date }} is {{ days_to_now(starting_date) }}",  
    )
    # Days since 2015-05-01 00:00:00 is 2313

demo_template():

```

It's also possible to inject functions as Jinja [filters](https://jinja.palletsprojects.com/en/3.0.x/api/#jinja2.Environment.filters) using `user_defined_filters`. You can use filters as pipe-operations. The following example completes the same work as the previous example, only this time filters are used:

```python wrap theme={null}
@dag(
    start_date=datetime(2021, 1, 1),
    schedule=None,
    # Set user_defined_filters to use function as pipe-operation
    user_defined_filters={"days_to_now": days_to_now},  
    user_defined_macros={"starting_date": datetime(2015, 5, 1)},
)
def bash_script_template():
    print_days = BashOperator(
        task_id="print_days",
        # Pipe value to function
        bash_command="echo Days since {{ starting_date }} is {{ starting_date | days_to_now }}", 
    )
    # Days since 2015-05-01 00:00:00 is 2313

bash_script_template()

```

Functions injected with `user_defined_filters` and `user_defined_macros` are both usable in the Jinja environment. While they achieve the same result, Astronomer recommends using filters when you need to import multiple custom functions because the filter formatting improves the readability of your code. You can see this when comparing the two techniques side-to-side:

```python wrap theme={null}
"{{ name | striptags | title }}"  # chained filters are read naturally from left to right
"{{ title(striptags(name)) }}"  # multiple functions are more difficult to interpret because reading right to left
```

If you want to use a function to generate input at the top-level of the DAG, for example for a value in the DAG's `default_args`, you can register a custom macro. Defining a function as a macro has the advantage that it is only parsed at runtime, not every time the DAG file is parsed. This pattern follows the best practice of [avoiding top-level code in your DAG](/docs/learn/dag-best-practices#avoid-top-level-code-in-your-dag-file).

To register a custom macro, you need to define it as an [Airflow plugin](/docs/learn/using-airflow-plugins). For example, if you add the following code to a file in the `plugins` directory:

```python wrap theme={null}
from airflow.plugins_manager import AirflowPlugin

def get_acl():
    return 'helooo!'

class TestPlugin(AirflowPlugin):
    name = 'test_macro'
    macros = [get_acl]
```

Then, you can use the `get_acl` macro in the `default_args` by accessing it in a Jinja template.

```python wrap theme={null}

default_args = {
    'owner': 'astro',
    'access_control_list': "{{ macros.test_macro.get_acl() }}",
}

```

## Render native Python code

By default, Jinja templates always render to Python strings. Sometimes it's desirable to render templates to native Python code. When the code you're calling doesn't work with strings, it can cause issues. For example:

```python wrap theme={null}
def sum_numbers(*args):
    total = 0
    for val in args:
        total += val
    return total

sum_numbers(1, 2, 3)
# returns 6
sum_numbers("1", "2", "3")
# TypeError: unsupported operand type(s) for +=: 'int' and 'str'
```

Consider a scenario where you're passing a list of values to this function by triggering a DAG with a config that holds some numbers:

```python wrap theme={null}
@dag(
    start_date=datetime.datetime(2021, 1, 1),
    schedule=None,
    catchup=False
)
def failing_template():
    PythonOperator(
        task_id="sumnumbers",
        python_callable=sum_numbers,
        op_args="{{ dag_run.conf['numbers'] }}",
    )

failing_template()

```

You would trigger the DAG with the following JSON to the DAG run configuration:

```json wrap theme={null}
{"numbers": [1,2,3]}
```

The rendered value is a string. Since the `sum_numbers` function unpacks the given string, it ends up trying to add up every character in the string:

```python wrap theme={null}
('[', '1', ',', ' ', '2', ',', ' ', '3', ']')
```

This rendered string won't work, so you must tell Jinja to return a native Python list instead of a string. Jinja supports this with Environments. The [default Jinja environment](https://jinja.palletsprojects.com/en/3.0.x/api/#jinja2.Environment) outputs strings, but you can configure a [NativeEnvironment](https://jinja.palletsprojects.com/en/3.0.x/nativetypes/#jinja2.nativetypes.NativeEnvironment) to render templates as native Python code with the `render_template_as_native_obj` argument on the Dag class. The parameter can be overridden at the task-level.

```python wrap theme={null}
def sum_numbers(*args):
    total = 0
    for val in args:
        total += val
    return total

@dag(
    dag_id="native_templating",
    start_date=datetime.datetime(2021, 1, 1),
    schedule=None,
    # Render templates using Jinja NativeEnvironment
    render_template_as_native_obj=True,
)
def native_templating()
    sumnumbers = PythonOperator(
        task_id="sumnumbers",
        python_callable=sum_numbers,
        op_args="{{ dag_run.conf['numbers'] }}",
    )

native_templating()

```

Passing the same JSON configuration `{"numbers": [1,2,3]}` now renders a list of integers which the `sum_numbers` function processes correctly:

```text wrap theme={null}
[2021-08-26 11:53:12,872] {python.py:151} INFO - Done. Returned value was: 6
```

The Jinja environment must be configured on the DAG-level. This means that all tasks in a DAG render either using the default Jinja environment or using the NativeEnvironment.


# Test Airflow DAGs
Source: https://astronomer.io/docs/learn/testing-airflow

Learn about testing Airflow DAGs and gain insight into various types of tests — validation testing, unit testing, and data and pipeline integrity testing.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

Effectively testing DAGs requires an understanding of their structure and their relationship to other code and data in your environment. In this guide, you'll learn about various types of DAG validation testing, unit testing, and where to find further information on data quality checks.

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Webinar: [How to easily test your Airflow DAGs with the new `dag.test`() function](https://www.astronomer.io/events/webinars/how-to-easily-test-your-airflow-dags-with-the-new-dag-test-function/).
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Python testing basics. See [Getting Started with Testing in Python](https://realpython.com/python-testing/).
* At least one Python test runner. This guide mostly uses [`pytest`](https://docs.pytest.org/en/stable/index.html), but you can use others including [`nose2`](https://docs.nose2.io/en/latest/getting_started.html) and [`unittest`](https://docs.python.org/3/library/unittest.html).
* CI/CD for Python scripts. See [Continuous Integration with Python: An Introduction](https://realpython.com/python-continuous-integration/).
* Basic Airflow and [Astro CLI](/docs/cli/v1.43/install-cli) concepts. See [Get started with Airflow](/docs/learn/get-started-with-airflow).

## Write DAG validation tests

DAG validation tests ensure that your DAGs fulfill a list of criteria. Using validation tests can help you:

* Develop DAGs without access to a local Airflow environment.
* Ensure that custom DAG requirements are systematically checked and fulfilled.
* Test DAGs automatically in a CI/CD pipeline.
* Enable power users to test DAGs from the CLI.

At a minimum, you should run DAG validation tests to check for [import errors](#check-for-import-errors). Additional tests can check things like custom logic, ensuring that `catchup` is set to False for every DAG in your Airflow instance, or making sure only `tags` from a defined list are used in the DAGs.

DAG validation tests apply to all DAGs in your Airflow environment, so you only need to create one test suite.

### Common DAG validation tests

This section covers the most common types of DAG validation tests with full code examples.

#### Check for import errors

The most common DAG validation test is to check for import errors. Checking for import errors through a validation test is faster than starting your Airflow environment and checking for errors in the Airflow UI. In the following test, `get_import_errors` checks the `.import_errors` attribute of the current [`DagBag`](https://airflow.apache.org/docs/apache-airflow/stable/_api/airflow/models/dagbag/index.html#).

```python wrap theme={null}
import os
import pytest
from airflow.models import DagBag

def get_import_errors():
    """
    Generate a tuple for import errors in the dag bag
    """

    dag_bag = DagBag(include_examples=False)

    def strip_path_prefix(path):
        return os.path.relpath(path, os.environ.get("AIRFLOW_HOME"))

    # prepend "(None,None)" to ensure that a test object is always created even if it's a no op.
    return [(None, None)] + [
        (strip_path_prefix(k), v.strip()) for k, v in dag_bag.import_errors.items()
    ]

@pytest.mark.parametrize(
    "rel_path,rv", get_import_errors(), ids=[x[0] for x in get_import_errors()]
)
def test_file_imports(rel_path, rv):
    """Test for import errors on a file"""
    if rel_path and rv:
        raise Exception(f"{rel_path} failed to import with message \n {rv}")
```

#### Check for custom code requirements

Airflow DAGs support many types of custom plugins and code. It is common for data engineering teams to define best practices and custom rules around how their DAGs should be written and create DAG validation tests to ensure those standards are met.

The code snippet below includes a test which checks that all DAGs have their `tags` parameter set to one or more of the `APPROVED_TAGS`.

```python wrap theme={null}
import os
import pytest
from airflow.models import DagBag

def get_dags():
    """
    Generate a tuple of dag_id, <DAG objects> in the DagBag
    """

    dag_bag = DagBag(include_examples=False)

    def strip_path_prefix(path):
        return os.path.relpath(path, os.environ.get("AIRFLOW_HOME"))

    return [(k, v, strip_path_prefix(v.fileloc)) for k, v in dag_bag.dags.items()]

APPROVED_TAGS = {"customer_success", "op_analytics", "product"}

@pytest.mark.parametrize(
    "dag_id,dag,fileloc", get_dags(), ids=[x[2] for x in get_dags()]
)
def test_dag_tags(dag_id, dag, fileloc):
    """
    test if a DAG is tagged and if those TAGs are in the approved list
    """
    assert dag.tags, f"{dag_id} in {fileloc} has no tags"
    if APPROVED_TAGS:
        assert not set(dag.tags) - APPROVED_TAGS
```

<Tip>
  You can view the attributes and methods available for the `dag` model in the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#tag/DAG).
</Tip>

You can also set requirements at the task level by accessing the `tasks` attribute within the `dag` model, which contains a list of all task objects of a DAG. The test below checks that all DAGs contain at least one task and all tasks use `trigger_rule="all_success"`.

```python wrap theme={null}
@pytest.mark.parametrize(
    "dag_id,dag,fileloc", get_dags(), ids=[x[2] for x in get_dags()]
)
def test_dag_tags(dag_id, dag, fileloc):
    """
    test if all DAGs contain a task and all tasks use the trigger_rule all_success
    """
    assert dag.tasks, f"{dag_id} in {fileloc} has no tasks"
    for task in dag.tasks:
        t_rule = task.trigger_rule
        assert (
            t_rule == "all_success"
        ), f"{task} in {dag_id} has the trigger rule {t_rule}"
```

## Implement DAG validation tests

Airflow offers different ways to run DAG validation tests using any Python test runner. This section gives an overview of the most common implementation methods. If you are new to testing Airflow DAGs, you can quickly get started by using Astro CLI commands.

### Airflow CLI

The Airflow CLI offers two commands related to local testing:

* [`airflow dags test`](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#test): Given a DAG ID and execution date, this command writes the results of a single DAG run to the metadata database. This command is useful for testing DAGs by creating manual DAG runs from the command line. In Airflow 2.10+ it is possible to skip tasks based on their task id with `--mark-success-pattern` flag and run them with their configured executor(s) with `--use-executor`.
* [`airflow tasks test`](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#test_repeat1): This command tests one specific task instance without checking for dependencies or recording the outcome in the metadata database.

With the Astro CLI, you can run all Airflow CLI commands using [`astro dev run`](/docs/cli/v1.43/astro-dev-run). For example, to run `airflow dags test` on the DAG `my_dag` for the execution date of `2023-01-29` run:

```sh wrap theme={null}
astro dev run dags test my_dag '2023-01-29'
```

### The Astro CLI

The Astro CLI includes a suite of commands to help simplify common testing workflows. See [Test your Astro project locally](/docs/cli/v1.43/test-your-astro-project-locally).

### Test DAGs in a CI/CD pipeline

You can use CI/CD tools to test and deploy your Airflow code. By installing the Astro CLI into your CI/CD process, you can test your DAGs before deploying them to a production environment. See [set up CI/CD](/docs/astro/set-up-ci-cd) for example implementations.

<Info>
  Astronomer customers can use the Astro GitHub integration, which allows you to automatically deploy code from a GitHub repository to an Astro deployment, viewing Git metadata in the Astro UI. See [Deploy code with the Astro GitHub integration](/docs/astro/deploy-github-integration).
</Info>

## Add test data or files for local testing

Use the `include` folder of your Astro project to store files for testing locally, such as test data or a dbt project file. The files in your `include` folder are included in your deploys to Astro, but they aren't parsed by Airflow. Therefore, you don't need to specify them in `.airflowignore` to prevent parsing.

If you're running Airflow locally, apply your changes by refreshing the Airflow UI.

## Debug interactively with `dag.test()`

The `dag.test()` method allows you to run all tasks in a DAG within a single serialized Python process, without running the Airflow scheduler. The `dag.test()` method lets you iterate faster and use IDE debugging tools when developing DAGs.

This functionality replaces the deprecated DebugExecutor. Learn more in the [Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/debug.html).

### Prerequisites

Ensure that your testing environment has:

* [Airflow 2.5.0](https://airflow.apache.org/docs/apache-airflow/stable/start.html) or later. You can check your version by running `airflow version`.
* All provider packages that your DAG uses.
* An initialized [Airflow metadata database](/docs/learn/airflow-database), if your DAG uses elements of the metadata database like XCom. The Airflow metadata database is created when Airflow is first run in an environment. You can check that it exists with `airflow db check` and initialize a new database with `airflow db migrate` (`airflow db init` in Airflow versions pre-2.7).

You may want to install these requirements and test your DAGs in a [virtualenv](https://virtualenv.pypa.io/en/latest/) to avoid dependency conflicts in your local environment.

### Setup

To use `dag.test()`, you only need to add a few lines of code to the end of your DAG file. If you are using a traditional DAG context, call `dag.test()` after your DAG declaration. If you are using the `@dag` decorator, assign your DAG function to a new object and call the method on that object.

<details>
  <summary>Traditional</summary>

  ```python {14-15} wrap theme={null}
  from airflow.models.dag import DAG
  from pendulum import datetime
  from airflow.operators.empty import EmptyOperator

  with DAG(
      dag_id="simple_classic_dag",
      start_date=datetime(2023, 1, 1),
      schedule="@daily",
      catchup=False,
  ) as dag:  # assigning the context to an object is mandatory for using dag.test()

      t1 = EmptyOperator(task_id="t1")

  if __name__ == "__main__":
      dag.test()
  ```
</details>

<details>
  <summary>Decorator</summary>

  ```python {18-19} wrap theme={null}
  from airflow.decorators import dag
  from pendulum import datetime
  from airflow.operators.empty import EmptyOperator

  @dag(
      start_date=datetime(2023, 1, 1),
      schedule="@daily",
      catchup=False,
  )
  def my_dag():

      t1 = EmptyOperator(task_id="t1")

  dag_object = my_dag()

  if __name__ == "__main__":
      dag_object.test()
  ```
</details>

You can run the `.test()` method with popular debugging tools such as:

* [VSCode](https://code.visualstudio.com/docs/editor/debugging).
* [PyCharm](https://www.jetbrains.com/help/pycharm/debugging-your-first-python-application.html).
* Tools like [The Python Debugger](https://docs.python.org/3/library/pdb.html) and the built-in [`breakpoint()`](https://docs.python.org/3/library/functions.html#breakpoint) function. These allow you to run `dag.test()` from the command line by running `python <path-to-dag-file>`.

### Use `dag.test()` with the Astro CLI

If you use the Astro CLI exclusively and don't have the `airflow` package installed locally, you can still debug using `dag.test()` by running `astro dev start`, entering the scheduler container with `astro dev bash -s`, and executing `python <path-to-dag-file>` from within the container. Unlike using the base `airflow` package, this testing method requires starting up a complete Airflow environment.

### Use variables and connections in `dag.test`()

To debug your DAGs in a more realistic environment, you can pass the following Airflow environment configurations to `dag.test()`:

* `execution_date` passed as a `pendulum.datetime` object.
* [Airflow connections](/docs/learn/connections) passed as a `.yaml` file.
* Airflow variables passed as a `.yaml` file.
* DAG configuration passed as a dictionary.

This is useful for testing your DAG for different dates or with different connections and configurations. The following code snippet shows the syntax for passing various parameters to `dag.test()`:

```python wrap theme={null}
from pendulum import datetime 

if __name__ == "__main__":
    conn_path = "connections.yaml"
    variables_path = "variables.yaml"
    my_conf_var = 23

    dag.test(
        execution_date=datetime(2023, 1, 29),
        conn_file_path=conn_path,
        variable_file_path=variables_path,
        run_conf={"my_conf_var": my_conf_var},
    )
```

The `connections.yaml` file should list connections with their properties as shown in the following example:

```yaml wrap theme={null}
my_aws_conn:
  conn_type: amazon
  login: <your-AWS-key>
  password: <your-AWS-secret>
  conn_id: my_aws_conn
```

Variables in a `variables.yaml` need to be listed with their `key` and `value`:

```yaml wrap theme={null}
my_variable:
  key: my_variable
  value: 42
```

<Info>
  By default, `dag.test()` runs tasks without an executor. In Airflow 2.10+ it is possible to run the tasks with their configured executors by setting `use_executor` to `True` in `dag.test()`.
</Info>

### Skip tasks when using `dag.test`()

Airflow 2.10 added the possibility to skip the execution of tasks whose ids are matching a provided [regex](https://en.wikipedia.org/wiki/Regular_expression) pattern when using `dag.test`(). This is particularly useful when you have sensors in a DAG that you'd like to bypass when testing.

```python wrap theme={null}
if __name__ == "__main__":

    dag.test(
        # new in Airflow 2.10
        mark_success_pattern="sensor.*",  # regex of task ids to be auto-marked as successful
    )
```

## Unit testing

[Unit testing](https://en.wikipedia.org/wiki/Unit_testing) is a software testing method where small chunks of source code are tested individually to ensure they function correctly. The objective is to isolate testable logic inside of small, well-named functions. For example:

```python wrap theme={null}
def test_function_returns_5():
    assert my_function(input) == 5
```

In the context of Airflow, you can write unit tests for any part of your DAG, but they are most frequently applied to hooks and operators. All Airflow hooks, operators, and provider packages must pass unit testing before code can be merged into the project. For an example of unit testing, see [AWS `S3Hook`](https://airflow.apache.org/registry/providers/amazon#amazon-s3-S3Hook) and the associated [unit tests](https://github.com/apache/airflow/tree/main/providers/amazon/tests/system/amazon/aws).

If you are using custom hooks or operators, Astronomer recommends using unit tests to check the logic and functionality. In the following example, a custom operator checks if a number is even:

```python wrap theme={null}
from airflow.models import BaseOperator

class EvenNumberCheckOperator(BaseOperator):
    def __init__(self, my_operator_param, *args, **kwargs):
        self.operator_param = my_operator_param
        super().__init__(*args, **kwargs)

    def execute(self, context):
        if self.operator_param % 2:
            return True
        else:
            return False

```

You then write a `test_evencheckoperator.py` file with unit tests similar to the following example:

```python expandable wrap theme={null}
import unittest
from datetime import datetime
from airflow.models.dag import DAG
from airflow.models import TaskInstance

DEFAULT_DATE = datetime(2021, 1, 1)

class EvenNumberCheckOperator(unittest.TestCase):
    def setUp(self):
        super().setUp()
        self.dag = DAG(
            "test_dag", default_args={"owner": "airflow", "start_date": DEFAULT_DATE}
        )
        self.even = 10
        self.odd = 11

    def test_even(self):
        """Tests that the EvenNumberCheckOperator returns True for 10."""
        task = EvenNumberCheckOperator(
            my_operator_param=self.even, task_id="even", dag=self.dag
        )
        ti = TaskInstance(task=task, execution_date=datetime.now())
        result = task.execute(ti.get_template_context())
        assert result is True

    def test_odd(self):
        """Tests that the EvenNumberCheckOperator returns False for 11."""
        task = EvenNumberCheckOperator(
            my_operator_param=self.odd, task_id="odd", dag=self.dag
        )
        ti = TaskInstance(task=task, execution_date=datetime.now())
        result = task.execute(ti.get_template_context())
        assert result is False

```

If your DAGs contain `PythonOperators` that execute your own Python functions, it is recommended that you write unit tests for those functions as well.

The most common way to implement unit tests in production is to automate them as part of your CI/CD process. Your CI tool executes the tests and stops the deployment process when errors occur.

### Use mocking in tests

Mocking is the imitation of an external system, dataset, or other object. For example, you might use mocking with an Airflow unit test if you are testing a connection, but don't have access to the metadata database. Mocking could also be used when you need to test an operator that executes an external service through an API endpoint, but you don't want to wait for that service to run a simple test.

Many [Airflow tests](https://github.com/apache/airflow/tree/main/airflow-core/tests) use mocking. The blog [Testing and debugging Apache Airflow](https://godatadriven.com/blog/testing-and-debugging-apache-airflow/) discusses Airflow mocking and it might help you get started.

## Data quality checks

Testing your DAG ensures that your code fulfills your requirements. But even if your code is perfect, data quality issues can break or negatively affect your pipelines. Airflow, being at the center of the modern data engineering stack, is the ideal tool for checking data quality.

Data quality checks differ from code-related testing because the data isn't static like your DAG code. It is best practice to incorporate data quality checks into your DAGs and use [Airflow dependencies](/docs/learn/managing-dependencies) and [branching](/docs/learn/airflow-branch-operator) to handle what should happen in the event of a data quality issue, from halting the pipeline to [sending notifications](/docs/learn/error-notifications-in-airflow) to data quality stakeholders.

There are many ways you can integrate data quality checks into your DAG:

* [SQL check operators](/docs/learn/airflow-sql-data-quality): Airflow-native operators that run highly customizable data quality checks on a wide variety of relational databases.
* [Great Expectations](/docs/learn/airflow-great-expectations): A data quality testing suite with an [Airflow provider](https://airflow.apache.org/registry/providers/great-expectations) offering the ability to define data quality checks in JSON to run on relational databases, Spark and pandas DataFrames.
* [Soda Core](/docs/learn/soda-data-quality): A framework to check data quality using YAML configuration to define data quality checks to run on relational databases and Spark dataframes.

Data quality checks work better at scale if you design your DAGs to load or process data incrementally. To learn more about incremental loading, see [DAG Writing Best Practices in Apache Airflow](/docs/learn/dag-best-practices). Processing smaller, incremental chunks of data in each DAG Run ensures that any data quality issues have a limited effect.

Learn more about how to approach data quality within Airflow:

* [Data quality and Airflow guide](/docs/learn/data-quality)
* [How to Keep Data Quality in Check with Airflow](https://www.astronomer.io/blog/how-to-keep-data-quality-in-check-with-airflow/)
* [Get Improved Data Quality Checks in Airflow with the Updated Great Expectations Operator](https://www.astronomer.io/blog/improved-data-quality-checks-in-airflow-with-great-expectations-operator/)


# Airflow plugins
Source: https://astronomer.io/docs/learn/using-airflow-plugins

How to use Airflow plugins.

[Airflow plugins](https://airflow.apache.org/docs/apache-airflow/stable/plugins.html) are external features that can be added to customize your Airflow installation, including the [Airflow UI](/docs/learn/airflow-ui). Airflow 3 added comprehensive plugin support in version 3.1 with a plugin manager interface that allows you to add many different components to a plugin, from custom macros, to FastAPI endpoints, to React apps.

In this guide, you'll learn about when you might want to use plugins and how to create them, including examples for popular types of plugins.

<Note>
  Airflow 2 supported Flask AppBuilder views, Flask AppBuilder menu items, and Flask Blueprints in plugins, which have been deprecated in Airflow 3. All new plugins in Airflow 3 should use [External views](#external-views), [React apps](#react-apps), [FastAPI apps](#fastapi-apps), and [FastAPI middlewares](#middlewares) instead.

  If you are looking to use legacy FAB-based plugin in Airflow 3, see the [Upgrading Guide in the FAB provider documentation](https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/upgrading.html).
</Note>

<Tip>
  In Airflow 3.2+ you can change the colors and CSS stylings of the UI with the [AIRFLOW\_\_API\_\_THEME](http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/stable/configurations-ref.html#theme) configuration.
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Airflow core components. See [Airflow's components](/docs/learn/airflow-components).
* Basics of [FastAPI](https://fastapi.tiangolo.com/).
* Basics of [JavaScript](https://developer.mozilla.org/en-US/docs/Web/JavaScript) and [React](https://react.dev/).

## When to use plugins

Plugins offer a flexible way to build on top of Airflow. While most plugins are written to extend the Airflow UI, you can also add other functionality to your Airflow instance like a FastAPI app. Some examples of when you might want to use plugins include:

* Adding a button to the **Home** view to trigger a custom action, for example to pause and unpause all dags.
* Adding middleware to the Airflow API to log or modify requests and responses.
* Adding a custom button to the task instance **Details** view that links to files or logs in external data tools relevant to the task.
* Creating additional API endpoints for your Airflow instance, for example to run a specific set of dags.
* Adding a custom dashboard displaying information related to your data pipelines on a new page in the Airflow UI, for example showing the status of your most critical tasks.

## Plugin interface

The plugin interface is defined by the `AirflowPlugin` class and allows you to add components to your plugin. A plugin can consist of one or more components; for example you can add a React app, a FastAPI app, and several custom macros in the same plugin. You can add as many plugins as you want to your Airflow instance.

To register a plugin, place it in a Python file in the `plugins` folder of your Airflow instance. Astronomer recommends keeping each plugin in a separate file.

The code snippet below shows the `my_plugin` plugin. It is instantiated by inheriting from the `AirflowPlugin` class and adding components to the plugin. Currently the plugin doesn't have any components, so it doesn't do anything. You can learn more about the different components in the [Plugin components](#plugin-components) section.

Note that when developing plugins you'll need to restart the Airflow API server to see the changes you make to the plugin. You can set `AIRFLOW__CORE__LAZY_LOAD_PLUGINS=False` in your `airflow.cfg` file to reload plugins automatically, however, changes won't be reflected in new running tasks until after the scheduler is restarted.

```python wrap theme={null}
from airflow.plugins_manager import AirflowPlugin

class MyPlugin(AirflowPlugin):
    name = "my_plugin"

    external_views = []
    react_apps = []
    macros = []
    fastapi_apps = []
    fastapi_root_middlewares = []
    global_operator_extra_links = []
    operator_extra_links = []
    timetables = []
    listeners = []

    def on_load(*args, **kwargs):
        pass

```

## Verify loaded plugins

To see all currently loaded plugins, and to verify whether your plugin has been loaded, open the Plugins page from the Admin menu on the left navigation bar.

<Frame>
  <img alt="Plugins view" />
</Frame>

## Plugin components

This section contains examples for each of the different components that can be added to a plugin. The available components are:

* [External views](#external-views): Additional Airflow UI views in different locations.
* [React apps](#react-apps): Embedding a React app in the Airflow UI.
* [Macros](#macros): Pre-defined functions that can be used in Jinja [templates](/docs/learn/templating) in templatable fields of your operators.
* [FastAPI apps](#fastapi-apps): Additional API endpoints for your Airflow instance.
* [Middlewares](#middlewares): Middleware for Airflow API.
* [Operator extra links](#operator-extra-links): Buttons for your operators that often link to external systems.
* Timetables: Additional timetables for your dags. See [timetables in the Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timetable.html).
* Listeners: Listeners for your Airflow instance that execute when certain events occur. See [listeners in the Airflow documentation](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/listeners.html).

### External views

You can add additional views to your Airflow instance in different [locations](#locations). You can point the view to an existing Website to embed it as an iframe or to a new one you create in a [FastAPI app](#fastapi-apps) that is registered alongside the external view in the same plugin. The example below shows how to add a view that embeds Wikipedia as an iframe.

```python wrap theme={null}
from airflow.plugins_manager import AirflowPlugin


class WikipediaExternalViewPlugin(AirflowPlugin):
    name = "wikipedia_external_view"

    external_views = [
        {
            "name": "📖 Wikipedia Search",
            "href": "https://en.wikipedia.org/wiki/Main_Page",
            "destination": "dag",
            "url_route": "wikipedia_search"
        }
    ]
```

In the Airflow UI, you can see the external view button in the **Details** view of the dag.

<Frame>
  <img alt="External view" />
</Frame>

### React apps

If you want to embed a more complex application in the Airflow UI, you can use a [React app](#react-apps). React apps can be added in the same locations as [External views](#external-views).

```python expandable wrap theme={null}
from pathlib import Path
from airflow.plugins_manager import AirflowPlugin
from fastapi import FastAPI
from fastapi.responses import FileResponse, HTMLResponse

PLUGIN_DIR = Path(__file__).parent
app = FastAPI(title="Simple React App", version="1.0.0")


@app.get("/my-app.js")
async def serve_react_component():
    js_file_path = PLUGIN_DIR / "my-app.js"
    return FileResponse(
        path=str(js_file_path),
        media_type="application/javascript",
        filename="my-app.js",
    )



@app.get("/")
async def root():
    return {
        "message": "🌟 Simple React App Plugin",
        "type": "react_app",
        "component_url": "/simple-react-app/my-app.js",
        "description": "Embeds a React component directly in Airflow UI",
    }


class SimpleReactAppPlugin(AirflowPlugin):

    name = "simple_react_app"

    fastapi_apps = [
        {"app": app, "url_prefix": "/simple-react-app", "name": "Simple React App"}
    ]

    react_apps = [
        {
            "name": "React Example Plugin",
            "bundle_url": "/simple-react-app/my-app.js",
            "destination": "nav",
            "category": "browse",
            "url_route": "simple-react-app",
        }
    ]
```

After this you can add your React app in the `my-app.js` file and access it in the Airflow UI at `http://localhost:8080/simple-react-app/`.

<Frame>
  <img alt="React app" />
</Frame>

You can put your React app inside of a few selected existing pages in the Airflow UI. The supported views are `dashboard`, `dag_overview` and `task_overview`. To position the element in the existing page use the CSS [`order` rule](https://www.w3schools.com/cssref/css3_pr_order.php) which will determine the flex order.

Note that you need to make the plugin available as a global variable in the JavaScript file. The example below shows how to do this for the `DAGToggleWidget` plugin.

```javascript wrap theme={null}
globalThis['DAG Toggle Widget'] = DAGToggleWidget; // Matching the plugin name
globalThis.AirflowPlugin = DAGToggleWidget; // Fallback that Airflow looks for
```

<Note>
  The React app integration is experimental and interfaces might change in future versions. Particularly, dependency and state interactions between the UI and plugins may need to be refactored for more complex plugin apps.
</Note>

### Macros

A macro is a pre-defined function that can be used in Jinja [templates](/docs/learn/templating) in templatable fields of your operators.

```python wrap theme={null}
from airflow.plugins_manager import AirflowPlugin
from datetime import datetime


def month_start(ds):
    d = datetime.strptime(ds, "%Y-%m-%d")
    return d.replace(day=1).strftime("%Y-%m-%d")


class AirflowTestPlugin(AirflowPlugin):
    name = "macro_plugin_example"
    macros = [month_start]
```

In your dag your can use the macro in any [templateable field](/docs/learn/templating).

```python wrap theme={null}
from airflow.providers.standard.operators.bash import BashOperator

BashOperator(
    task_id="print_hello",
    bash_command="echo {{ macros.macro_plugin_example.month_start(ds) }}"
)
```

### FastAPI apps

You can use a [FastAPI](https://fastapi.tiangolo.com/) app in your plugin to add endpoints to your Airflow instance through which you can interact with it, in addition to the public Airflow API.

```python wrap theme={null}
from airflow.plugins_manager import AirflowPlugin
from fastapi import FastAPI

app = FastAPI(title="Hello World FastAPI App", version="1.0.0")


@app.get("/hello")
async def hello_world():
    return {"message": "Hello from Airflow!"}


class FastAPIAppPlugin(AirflowPlugin):
    name = "hello_fastapi_app"

    fastapi_apps = [
        {"app": app, "url_prefix": "/hello-app", "name": "Hello World FastAPI App"}
    ]
```

After adding the plugin to your Airflow instance, you can access the endpoints of the FastAPI app at `http://localhost:8080/<url_prefix>`. In the example above, you can send a GET request to `http://localhost:8080/hello-app/hello` to get the response "Hello from Airflow!".

```bash wrap theme={null}
curl http://localhost:8080/hello-app/hello
```

<Note>
  **FastAPI endpoints on Astro**

  For calling FastAPI plugin endpoint on Astro, for example when combining a FastAPI app with an `external_views` nav link, use a **relative path** (no leading `/`) in the `href`. On Astro, the API server is reachable at a different base URL than in other environments, and a relative path ensures that URIs are resolved correctly.

  ```python wrap theme={null}
  from airflow.plugins_manager import AirflowPlugin
  from fastapi import FastAPI

  app = FastAPI()

  @app.get("/dashboard")
  async def dashboard():
      return {"message": "Hello!"}

  class MyDashboardPlugin(AirflowPlugin):
      name = "my_dashboard"
      fastapi_apps = [
          {"app": app, "url_prefix": "/my-dashboard", "name": "My Dashboard"}
      ]
      external_views = [{
          "name": "My Dashboard",
          "href": "my-dashboard/dashboard",  # relative path, no leading /
          "destination": "nav",
          "url_route": "my-dashboard",
      }]
  ```

  When serving static files using FastAPI's `StaticFiles`, always use **relative paths** in your HTML templates and Python code. Absolute paths like `/my-app/static/style.css` break on Astro because the base URL differs from local development.

  ```html wrap theme={null}
  <!-- Correct: relative path (works everywhere) -->
  <link rel="stylesheet" href="static/style.css">

  <!-- Wrong: absolute path (breaks on Astro) -->
  <link rel="stylesheet" href="/my-app/static/style.css">
  ```
</Note>

<Note>
  API documentation is automatically generated:

  * [OpenAPI JSON schema](https://swagger.io/specification) under `https://<api_server_url>/<url_prefix>/openapi.json`
  * [Swagger UI](https://swagger.io) under `https://<api_server_url>/<url_prefix>/docs`
  * [Redoc](https://github.com/Redocly/redoc) under `https://<api_server_url>/<url_prefix>/redoc`
</Note>

Note that this endpoint isn't protected by the Airflow API, so you need to set up authentication for it on your own. See [the FastAPI docs](https://fastapi.tiangolo.com/tutorial/security/) for more information.

### Middlewares

You can add middleware to the Airflow API server to modify requests and responses to **all** its APIs. This includes the REST API, and also the API serving the Airflow UI or any FastAPI app you might have added.

```python wrap theme={null}
from typing import Callable
from airflow.plugins_manager import AirflowPlugin
from fastapi import Request, Response
from starlette.middleware.base import BaseHTTPMiddleware


class HelloWorldLoggingMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next: Callable) -> Response:
        print(f"🌐 Hello from middleware! Request: {request.method} {request.url}")
        response = await call_next(request)
        response.headers["X-Hello-Middleware"] = "Hello from Airflow middleware!"

        return response


class HelloWorldMiddlewarePlugin(AirflowPlugin):
    name = "hello_middleware"

    fastapi_root_middlewares = [
        {
            "middleware": HelloWorldLoggingMiddleware,
            "args": [],
            "kwargs": {},
            "name": "Hello World Logging Middleware"
        }
    ]
```

This simple example just adds a print statement to the console when a request is made to the Airflow API, and adds a custom header to the response.

```text wrap theme={null}
2025-09-12T13:51:49.146084000+02:00🌐 Hello from middleware! Request: GET http://localhost:8080/api/v2/hitlDetails/?dag_id=plugin_dag&dag_run_id=manual__2025-09-12T09%3A35%3A49.540843%2B00%3A00&task_id=search_task
```

Remember that middleware is applied to all requests to any API served by the Airflow API server. If you'd like to be selective about which requests to modify, you need to implement the logic in the middleware to only execute the middleware for the specific requests you want to modify.

### Operator extra links

An operator extra link is a button that can be added to the **Details** view of a task instance of any operator. Operator extra links can be implemented in two ways:

* `global_operator_extra_links`: A button that will be added to the **Details** view of every task instance of every operator.
* `operator_extra_links`: A button that will be added to the **Details** view of all task instances of a specific operator.

Global and specific operator extra links are added separately in Airflow plugins. The example below shows how to add a global operator extra link that links to the Google search results of the value returned by a task instance.

```python wrap theme={null}
from airflow.plugins_manager import AirflowPlugin
from airflow.sdk.bases.operatorlink import BaseOperatorLink
from airflow.models import XCom
from urllib.parse import quote_plus
from typing import Optional, Dict, Any


class GoogleSearchXComLink(BaseOperatorLink):
    name = "🔍 Search Return Value in Google"

    def get_link(self, operator, *, ti_key, **context) -> str:

        xcom_value = XCom.get_value(ti_key=ti_key, key="return_value")

        search_term = str(xcom_value)
        encoded_search = quote_plus(search_term)

        return f"https://www.google.com/search?q={encoded_search}"


class OperatorExtraLinkPlugin(AirflowPlugin):
    name = "operator_extra_link"
    operator_extra_links = [GoogleSearchXComLink()]
```

To use this operator extra link you need to add it to one of your operators, essentially creating a [custom operator](/docs/learn/airflow-importing-custom-hooks-operators). The code snippet below shows the `SearchBashOperator` that subclasses the `BashOperator` and adds the `GoogleSearchXComLink` to it.

```python wrap theme={null}
class SearchBashOperator(BashOperator):
    operator_extra_links = [GoogleSearchXComLink()]

search_task = SearchBashOperator(
    task_id="search_task",
    bash_command="echo 'Cute animal picture'",
)
```

In the Airflow UI, you can see the operator extra link button in the **Details** view of the task instance.

<Frame>
  <img alt="Operator extra link" />
</Frame>

## Locations

UI plugins like [React apps](#react-apps) and [External views](#external-views) can be added to the Airflow UI in different locations.

Plugins set to `destination":"nav"` will be added in the navigation bar to the left. You also need to specify the `category` to the plugin, for example the `browse` category will add the plugin to the **Browse** menu.

<Frame>
  <img alt="Navigation bar" />
</Frame>

In Airflow 3.3+ you can use `"destination":"nav"` together with `"nav_top_level":True` to add your plugin as a top level item in the Airflow navigation bar.

The `dag` destination will add the plugin in an additional tab on every dag page.

<Frame>
  <img alt="DAG tab" />
</Frame>

Any other locations can be chosen by using the `base` destination.

<Frame>
  <img alt="Base destination" />
</Frame>

Similarly, the `dag_run`, `task`, and `task_instance` destinations will add the plugin in an additional tab on every dag run, task, and task instance page respectively.

In the case of React apps you can also embed them in existing pages in the Airflow UI, the supported locations are `dashboard`, `dag_overview` and `task_overview`. The example below shows a button that pauses and unpauses all dags in the Airflow instance, embedded in the **Home** (dashboard) page.

<Frame>
  <img alt="React app locations" />
</Frame>


# Airflow hooks
Source: https://astronomer.io/docs/learn/what-is-a-hook

Learn about hooks and how they should be used in Apache Airflow. See an example of implementing two different hooks in a DAG.

A hook is an abstraction of a specific API that allows Airflow to interact with an external system. Hooks are built into many operators, but they can also be used directly in DAG code.

In this guide, you'll learn about using hooks in Airflow and when you should use them directly in DAG code. You'll also implement two different hooks in a DAG.

Over 300 hooks are available in the [Airflow Registry](https://airflow.apache.org/registry). If a hook isn't available for your use case, you can write your own and share it with the community.

<Info>
  See the [Custom hooks and operators](/docs/learn/airflow-importing-custom-hooks-operators) guide for more information about writing custom hooks and operators.
</Info>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Basic Python. See the [Python Documentation](https://docs.python.org/3/tutorial/index.html).

## Hook basics

Hooks wrap around APIs and provide methods to interact with different external systems. Hooks standardize how Astronomer interacts with external systems and using them makes your DAG code cleaner, easier to read, and less prone to errors.

To use a hook, you typically only need a connection ID to connect with an external system. For more information about setting up connections, see [Manage your connections in Apache Airflow](/docs/learn/connections).

All hooks inherit from the [`BaseHook` class](https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/hooks/base.py), which contains the logic to set up an external connection with a connection ID. On top of making the connection to an external system, individual hooks can contain additional methods to perform various actions within the external system. These methods might rely on different Python libraries for these interactions. For example, the [`S3Hook`](https://airflow.apache.org/registry/providers/amazon#amazon-s3-S3Hook) relies on the [`boto3`](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) library to manage its Amazon S3 connection.

The `S3Hook` contains [over 20 methods](https://github.com/apache/airflow/blob/main/providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py) to interact with Amazon S3 buckets. The following are some of the methods that are included with `S3Hook`:

* `check_for_bucket`: Checks if a bucket with a specific name exists.
* `list_prefixes`: Lists prefixes in a bucket according to specified parameters.
* `list_keys`: Lists keys in a bucket according to specified parameters.
* `load_file`: Loads a local file to Amazon S3.
* `download_file`: Downloads a file from the Amazon S3 location to the local file system.

The following example shows how to use a hook in a DAG.

<details>
  <summary>TaskFlow</summary>

  ```python wrap theme={null}
  from airflow.sdk import dag, task

  @dag
  def my_dag():
      @task
      def my_task():
          from airflow.providers.amazon.aws.hooks.s3 import S3Hook

          s3_hook = S3Hook(aws_conn_id="my_aws_conn")
          # use hook methods here

      my_task()

  my_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python wrap theme={null}
  from airflow.sdk import DAG, task
  from airflow.providers.standard.operators.python import PythonOperator

  def _my_task():
      from airflow.providers.amazon.aws.hooks.s3 import S3Hook

      s3_hook = S3Hook(aws_conn_id="my_aws_conn")
      # use hook methods here

  with DAG(dag_id="my_dag"):

      my_task = PythonOperator(task_id="my_task", python_callable=_my_task)

  ```
</details>

## When to use hooks

Since hooks are the building blocks of operators, their use in Airflow is often abstracted away from the DAG author. However, there are some cases when you should use hooks directly in a Python function in your DAG. The following are some general guidelines for using hooks in Airflow:

* Hooks should always be used over manual API interaction to connect to external systems. It is common to use hooks in [Airflow decorated functions](/docs/learn/airflow-decorators), like when using `@task`, and in DAGs defined using the [`@asset` decorator](/docs/learn/airflow-datasets).
* If you write a custom operator to interact with an external system, it should use a hook.
* When an operator with built-in hooks exists for your specific use case, you should use the operator instead of manually setting up a hook.
* If you regularly need to connect to an API and a hook isn't available, write your own hook and share it with the community.

## Example implementation

The following example shows how you can use the hooks ([`S3Hook`](https://airflow.apache.org/registry/providers/amazon#amazon-s3-S3Hook) and [`SlackHook`](https://airflow.apache.org/registry/providers/slack#slack-slack-SlackHook)) to retrieve values from files in an Amazon S3 bucket, run a check on them, post the result of the check on Slack, and then log the response of the Slack API.

For this use case, you'll use hooks directly in your Python functions because none of the existing Amazon S3 operators can read data from multiple files within an Amazon S3 bucket. Also, none of the existing Slack operators can return the response of a Slack API call, which you might want to log for monitoring purposes.

The source code for the hooks used in this example can be found in the following locations:

* [`S3Hook` source code](https://github.com/apache/airflow/blob/main/providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py)
* [`SlackHook` source code](https://github.com/apache/airflow/blob/main/providers/slack/src/airflow/providers/slack/hooks/slack.py)

### Prerequisites

Before running the example DAG, make sure you have the necessary Airflow providers installed. If you are using the Astro CLI, add the following packages to your `requirements.txt` file:

```text wrap theme={null}
apache-airflow-providers-amazon
apache-airflow-providers-slack
```

### Create the connections

1. In the Airflow UI, go to **Admin** > **Connections** and click **+ Add Connection** button to define a new connection.
2. In the **Connection ID** field, enter a unique name for the connection.
3. In the **Connection Type** list, select **AWS** as the connection type for the Amazon S3 bucket. If the **AWS** connection type isn't available, make sure you installed the provider correctly.
4. Enter your AWS access key ID in the **Login** field.
5. Enter your AWS secret access key in the **Password** field. To retrieve your AWS access key ID and AWS secret access key, see [AWS Account and Access Keys](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html).
6. Click **Save**.
7. Repeat steps 1 to 6 to create a new connection for Slack. Select **slack** as the connection type and enter your [Bot User OAuth Token](https://api.slack.com/authentication/oauth-v2) in the **Password** field. To obtain the token, go to **Features** > **OAuth & Permissions**  on `api.slack.com/apps`.

### Run the example DAG

The following example DAG uses [Airflow decorators](/docs/learn/airflow-decorators) to define tasks and [XCom](/docs/learn/airflow-passing-data-between-tasks) to pass information between tasks. The name of the Amazon S3 bucket and the names of the files that the first task reads are stored as environment variables.

The following example DAG completes the following steps:

* A Python task with a manually implemented `S3Hook` reads three specific keys from Amazon S3 with the `read_key` method and then returns a dictionary with the file contents converted to integers.
* A second Python task completes a simple sum check using the results from the first task.
* The `SlackHook` `call` method posts the sum check results to a Slack channel and returns the response from the Slack API.

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  # importing necessary packages
  from datetime import datetime
  from airflow.decorators import dag, task
  from airflow.providers.slack.hooks.slack import SlackHook
  from airflow.providers.amazon.aws.hooks.s3 import S3Hook

  # set bucket name and file names
  S3BUCKET_NAME = "myhooktutorial"
  S3_EXAMPLE_FILE_NAME_1 = "file1.txt"
  S3_EXAMPLE_FILE_NAME_2 = "file2.txt"
  S3_EXAMPLE_FILE_NAME_3 = "file3.txt"


  # task to read 3 keys from your S3 bucket
  @task
  def read_keys_from_s3():
      s3_hook = S3Hook(aws_conn_id="aws_conn")
      response_file_1 = s3_hook.read_key(
          key=S3_EXAMPLE_FILE_NAME_1, bucket_name=S3BUCKET_NAME
      )
      response_file_2 = s3_hook.read_key(
          key=S3_EXAMPLE_FILE_NAME_2, bucket_name=S3BUCKET_NAME
      )
      response_file_3 = s3_hook.read_key(
          key=S3_EXAMPLE_FILE_NAME_3, bucket_name=S3BUCKET_NAME
      )

      response = {
          "num1": int(response_file_1),
          "num2": int(response_file_2),
          "num3": int(response_file_3),
      }

      return response


  # task running a check on the data retrieved from your S3 bucket
  @task
  def run_sum_check(response):
      if response["num1"] + response["num2"] == response["num3"]:
          return (True, response["num3"])
      return (False, response["num3"])


  # task posting to slack depending on the outcome of the above check
  # and returning the server response
  @task
  def post_to_slack(sum_check_result):
      slack_hook = SlackHook(slack_conn_id="hook_tutorial_slack_conn")

      if sum_check_result[0] is True:
          server_response = slack_hook.call(
              api_method="chat.postMessage",
              json={
                  "channel": "#test-airflow",
                  "text": f"""All is well in your bucket!
                          Correct sum: {sum_check_result[1]}!""",
              },
          )
      else:
          server_response = slack_hook.call(
              api_method="chat.postMessage",
              json={
                  "channel": "#test-airflow",
                  "text": f"""A test on your bucket contents failed!
                          Target sum not reached: {sum_check_result[1]}""",
              },
          )

      # return the response of the API call (for logging or use downstream)
      return server_response


  # implementing the DAG
  @dag(
      dag_id="hook_tutorial",
      start_date=datetime(2022, 5, 20),
      schedule="@daily",
      catchup=False,
  )
  def hook_tutorial():
      # the dependencies are automatically set by XCom
      response = read_keys_from_s3()
      sum_check_result = run_sum_check(response)
      post_to_slack(sum_check_result)


  hook_tutorial()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  # importing necessary packages
  from datetime import datetime
  from airflow import DAG
  from airflow.operators.python import PythonOperator
  from airflow.providers.slack.hooks.slack import SlackHook
  from airflow.providers.amazon.aws.hooks.s3 import S3Hook

  # set bucket name and file names
  S3BUCKET_NAME = "myhooktutorial"
  S3_EXAMPLE_FILE_NAME_1 = "file1.txt"
  S3_EXAMPLE_FILE_NAME_2 = "file2.txt"
  S3_EXAMPLE_FILE_NAME_3 = "file3.txt"


  # function to read 3 keys from your S3 bucket
  def read_keys_from_s3_function():
      s3_hook = S3Hook(aws_conn_id="aws_conn")
      response_file_1 = s3_hook.read_key(
          key=S3_EXAMPLE_FILE_NAME_1, bucket_name=S3BUCKET_NAME
      )
      response_file_2 = s3_hook.read_key(
          key=S3_EXAMPLE_FILE_NAME_2, bucket_name=S3BUCKET_NAME
      )
      response_file_3 = s3_hook.read_key(
          key=S3_EXAMPLE_FILE_NAME_3, bucket_name=S3BUCKET_NAME
      )

      response = {
          "num1": int(response_file_1),
          "num2": int(response_file_2),
          "num3": int(response_file_3),
      }

      return response


  # function running a check on the data retrieved from your S3 bucket
  def run_sum_check_function(response):
      if response["num1"] + response["num2"] == response["num3"]:
          return (True, response["num3"])
      return (False, response["num3"])


  # function posting to slack depending on the outcome of the above check
  # and returning the server response
  def post_to_slack_function(sum_check_result):
      slack_hook = SlackHook(slack_conn_id="hook_tutorial_slack_conn")

      if sum_check_result[0] is True:
          server_response = slack_hook.call(
              api_method="chat.postMessage",
              json={
                  "channel": "#test-airflow",
                  "text": f"""All is well in your bucket!
                          Correct sum: {sum_check_result[1]}!""",
              },
          )
      else:
          server_response = slack_hook.call(
              api_method="chat.postMessage",
              json={
                  "channel": "#test-airflow",
                  "text": f"""A test on your bucket contents failed!
                          Target sum not reached: {sum_check_result[1]}""",
              },
          )

      # return the response of the API call (for logging or use downstream)
      return server_response


  # implementing the DAG
  with DAG(
      dag_id="hook_tutorial",
      start_date=datetime(2022, 5, 20),
      schedule="@daily",
      catchup=False,
      # Render templates using Jinja NativeEnvironment
      render_template_as_native_obj=True,
  ):
      read_keys_form_s3 = PythonOperator(
          task_id="read_keys_form_s3", python_callable=read_keys_from_s3_function
      )

      run_sum_check = PythonOperator(
          task_id="run_sum_check",
          python_callable=run_sum_check_function,
          op_kwargs={
              "response": "{{ ti.xcom_pull(task_ids='read_keys_form_s3', \
                  key='return_value') }}"
          },
      )

      post_to_slack = PythonOperator(
          task_id="post_to_slack",
          python_callable=post_to_slack_function,
          op_kwargs={
              "sum_check_result": "{{ ti.xcom_pull(task_ids='run_sum_check', \
                  key='return_value') }}"
          },
      )

      # the dependencies are automatically set by XCom
      read_keys_form_s3 >> run_sum_check >> post_to_slack
  ```
</details>


# Airflow sensors
Source: https://astronomer.io/docs/learn/what-is-a-sensor

Get an overview of Airflow sensors and learn best practices for implementing sensors in production.

[Apache Airflow sensors](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/sensors.html) are a special kind of operator that are designed to wait for something to happen. When sensors run, they check to see if a certain condition is met before they are marked successful and let their downstream tasks execute.

In this guide, you'll learn how sensors are used in Airflow, best practices for implementing sensors in production, and how to use deferrable versions of sensors.

<Tip>
  Sensors are used to wait for a condition to be met before executing downstream tasks. Many sensors come with a deferrable mode, which allows them to release their worker slot while waiting for the condition to be met, increasing the efficiency of your DAGs. See [Deferrable operators](/docs/learn/deferrable-operators) for more information.

  If you want a DAG to run based on messages in a messaging queue, consider using [event-driven scheduling](/docs/learn/airflow-event-driven-scheduling) instead of sensors.
</Tip>

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Basic Python. See the [Python Documentation](https://docs.python.org/3/tutorial/index.html).

## Sensor basics

Sensors are a type of operator that checks if a condition is met at a specific interval. If the condition is met, the task is marked successful and the DAG can move to downstream tasks. If the condition isn't met, the sensor waits for another interval before checking again.

All sensors inherit from the [`BaseSensorOperator`](https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/bases/sensor.py) and have the following parameters:

* `mode`: How the sensor operates. There are two types of modes:
  * `poke`: This is the default mode. When using `poke`, the sensor occupies a worker slot for the entire execution time and sleeps between pokes. This mode is best if you expect a short runtime for the sensor.
  * `reschedule`: When using this mode, if the criteria isn't met then the sensor releases its worker slot and reschedules the next check for a later time. This mode is best if you expect a long runtime for the sensor, because it is less resource intensive and frees up workers for other tasks.
* `poke_interval`: When using `poke` mode, this is the time in seconds that the sensor waits before checking the condition again. The default is 60 seconds.
* `exponential_backoff`: When set to `True`, this setting creates exponentially longer wait times between pokes in `poke` mode.
* `timeout`: The maximum amount of time in seconds that the sensor checks the condition. If the condition isn't met within the specified period, the task fails.
* `soft_fail`: If set to `True`, the task is marked as skipped if the condition isn't met by the `timeout`.

Different types of sensors have different implementation details.

### Commonly used sensors

Many Airflow provider packages contain sensors that wait for various criteria in different source systems. The following are some of the most commonly used sensors:

* [`@task.sensor` decorator](https://airflow.apache.org/docs/apache-airflow/stable/tutorial/taskflow.html#using-the-taskflow-api-with-sensor-operators): Allows you to turn any Python function that returns a PokeReturnValue into an instance of the [`BaseSensorOperator`](https://airflow.apache.org/docs/task-sdk/stable/api.html#airflow.sdk.BaseSensorOperator) class. This way of creating a sensor is useful when checking for complex logic or if you are connecting to a tool using an API that has no specific sensor available.
* [`S3KeySensor`](https://airflow.apache.org/registry/providers/amazon#amazon-s3-S3KeySensor): Waits for a key (file) to appear in an Amazon S3 bucket. This sensor is useful if you want your DAG to process files from Amazon S3 as they arrive.
* [`DateTimeSensor`](https://airflow.apache.org/registry/providers/standard#standard-date_time-DateTimeSensor): Waits for a specified date and time. This sensor is useful if you want different tasks within the same DAG to run at different times.
* [`ExternalTaskSensor`](https://airflow.apache.org/registry/providers/standard#standard-external_task-ExternalTaskSensor): Waits for an Airflow task to be completed. This sensor is useful if you want to implement [cross-DAG dependencies](/docs/learn/cross-dag-dependencies) in the same Airflow environment.
* [`HttpSensor`](https://airflow.apache.org/registry/providers/http#http-http-HttpSensor): Waits for an API to be available. This sensor is useful if you want to ensure your API requests are successful.
* [`SqlSensor`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SqlSensor): Waits for data to be present in a SQL table. This sensor is useful if you want your DAG to process data as it arrives in your database.

To review the available Airflow sensors, go to the [Airflow Registry](https://airflow.apache.org/registry).

### Example implementation

The following example DAG shows how you might use the `SqlSensor` sensor:

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  from airflow.decorators import task, dag
  from airflow.providers.common.sql.sensors.sql import SqlSensor

  from typing import Dict
  from pendulum import datetime


  def _success_criteria(record):
      return record


  def _failure_criteria(record):
      return True if not record else False


  @dag(
      description="DAG in charge of processing partner data",
      start_date=datetime(2021, 1, 1),
      schedule="@daily",
      catchup=False,
  )
  def partner():
      waiting_for_partner = SqlSensor(
          task_id="waiting_for_partner",
          conn_id="postgres",
          sql="sql/CHECK_PARTNER.sql",
          parameters={"name": "partner_a"},
          success=_success_criteria,
          failure=_failure_criteria,
          fail_on_empty=False,
          poke_interval=20,
          mode="reschedule",
          timeout=60 * 5,
      )

      @task
      def validation() -> Dict[str, str]:
          return {"partner_name": "partner_a", "partner_validation": True}

      @task
      def storing():
          print("storing")

      waiting_for_partner >> validation() >> storing()


  partner()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  from airflow import DAG
  from airflow.operators.python import PythonOperator
  from airflow.providers.common.sql.sensors.sql import SqlSensor

  from typing import Dict
  from pendulum import datetime


  def _success_criteria(record):
      return record


  def _failure_criteria(record):
      return True if not record else False


  with DAG(
      dag_id="partner",
      description="DAG in charge of processing partner data",
      start_date=datetime(2021, 1, 1),
      schedule="@daily",
      catchup=False,
  ):
      waiting_for_partner = SqlSensor(
          task_id="waiting_for_partner",
          conn_id="postgres",
          sql="sql/CHECK_PARTNER.sql",
          parameters={"name": "partner_a"},
          success=_success_criteria,
          failure=_failure_criteria,
          fail_on_empty=False,
          poke_interval=20,
          mode="reschedule",
          timeout=60 * 5,
      )

      def validation_function() -> Dict[str, str]:
          return {"partner_name": "partner_a", "partner_validation": True}

      validation = PythonOperator(
          task_id="validation", python_callable=validation_function
      )

      def storing_function():
          print("storing")

      storing = PythonOperator(task_id="storing", python_callable=storing_function)

      waiting_for_partner >> validation >> storing
  ```
</details>

This DAG waits for data to be available in a Postgres database before running validation and storing tasks. The `SqlSensor` runs a SQL query and is marked successful when that query returns data. Specifically, when the result isn't in the set (0, '0', '', None). The `SqlSensor` task in the example DAG (`waiting_for_partner`) runs the `CHECK_PARTNER.sql` script every 20 seconds (the `poke_interval`) until the data is returned. The `mode` is set to `reschedule`, meaning between each 20 second interval the task won't take a worker slot. The `timeout` is set to 5 minutes, and the task fails if the data doesn't arrive within that time. When the `SqlSensor` criteria is met, the DAG moves to the downstream tasks.

## Sensor decorator / `PythonSensor`

If no sensor exists for your use case, you can create your own using either the `@task.sensor` decorator or the [`PythonSensor`](https://airflow.apache.org/registry/providers/standard#standard-python-PythonSensor). The `@task.sensor` decorator returns a `PokeReturnValue` as an instance of the `BaseSensorOperator`. The `PythonSensor` takes a `python_callable` that returns `True` or `False`.

The following DAG shows how to use either the sensor decorator or the `PythonSensor` to create the same custom sensor:

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  """
  ### Create a custom sensor using the @task.sensor decorator

  This DAG showcases how to create a custom sensor using the @task.sensor decorator
  to check the availability of an API.
  """

  from airflow.decorators import dag, task
  from pendulum import datetime
  import requests

  # importing the PokeReturnValue
  from airflow.sensors.base import PokeReturnValue


  @dag(start_date=datetime(2022, 12, 1), schedule="@daily", catchup=False)
  def sensor_decorator():
      # supply inputs to the BaseSensorOperator parameters in the decorator
      @task.sensor(poke_interval=30, timeout=3600, mode="poke")
      def check_dog_availability() -> PokeReturnValue:
          r = requests.get("https://random.dog/woof.json")
          print(r.status_code)

          # set the condition to True if the API response was 200
          if r.status_code == 200:
              condition_met = True
              operator_return_value = r.json()
          else:
              condition_met = False
              operator_return_value = None
              print(f"Woof URL returned the status code {r.status_code}")

          # the function has to return a PokeReturnValue
          # if is_done = True the sensor will exit successfully, if
          # is_done=False, the sensor will either poke or be rescheduled
          return PokeReturnValue(is_done=condition_met, xcom_value=operator_return_value)

      # print the URL to the picture
      @task
      def print_dog_picture_url(url):
          print(url)

      print_dog_picture_url(check_dog_availability())


  sensor_decorator()
  ```

  Here, the `@task.sensor` decorates the `check_dog_availability()` function, which checks if a given API returns a 200 status code. If the API returns a 200 status code, the sensor task is marked as successful. If any other status code is returned, the sensor pokes again after the `poke_interval` has passed.

  The optional `xcom_value` parameter in `PokeReturnValue` defines what data will be pushed to [XCom](/docs/learn/airflow-passing-data-between-tasks) once `is_done=true`. You can use the data that was pushed to XCom in any downstream tasks.
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  """
  ### Create a custom sensor using the PythonSensor

  This DAG showcases how to create a custom sensor using the PythonSensor
  to check the availability of an API.
  """

  from airflow.decorators import dag, task
  from pendulum import datetime
  import requests
  from airflow.sensors.python import PythonSensor


  def check_dog_availability_func(**context):
      r = requests.get("https://random.dog/woof.json")
      print(r.status_code)

      # set the condition to True if the API response was 200
      if r.status_code == 200:
          operator_return_value = r.json()
          # pushing the link to the Dog picture to XCom
          context["ti"].xcom_push(key="return_value", value=operator_return_value)
          return True
      else:
          operator_return_value = None
          print(f"Woof URL returned the status code {r.status_code}")
          return False


  @dag(
      start_date=datetime(2022, 12, 1),
      schedule=None,
      catchup=False,
      tags=["sensor"],
  )
  def pythonsensor_example():
      # turn any Python function into a sensor
      check_dog_availability = PythonSensor(
          task_id="check_dog_availability",
          poke_interval=10,
          timeout=3600,
          mode="reschedule",
          python_callable=check_dog_availability_func,
      )

      # click the link in the logs for a cute picture :)
      @task
      def print_dog_picture_url(url):
          print(url)

      print_dog_picture_url(check_dog_availability.output)


  pythonsensor_example()
  ```

  Here, the `PythonSensor` uses the `check_dog_availability_func` to check if a given API returns a 200 status code. If the API returns a 200 status code, the API response is pushed to [XCom](/docs/learn/airflow-passing-data-between-tasks) and the function returns `True`, causing the sensor task to be marked as successful. If any other status code is returned the `check_dog_availability_func` returns `False` and the sensor pokes again after the `poke_interval` has passed.
</details>

## Sensor best practices

When using sensors, keep the following in mind to avoid potential performance issues:

* Always define a meaningful `timeout` parameter for your sensor. The default for this parameter is seven days, which is a long time for your sensor to be running. When you implement a sensor, consider your use case and how long you expect the sensor to wait and then define the sensor's timeout accurately.
* Whenever possible and especially for long-running sensors, use `deferrable` mode. If no deferrable mode is available, use the `reschedule` mode. Both of these options help your sensor to not constantly occupy a worker slot. This helps avoid deadlocks in Airflow where sensors take all of the available worker slots.
* If your `poke_interval` is very short (less than about 5 minutes), use the `poke` mode. Using `reschedule` mode in this case can overload your scheduler.
* Define a meaningful `poke_interval` based on your use case. There is no need for a task to check a condition every 60 seconds (the default) if you know the total amount of wait time will be 30 minutes.

## Sensor failure modes

When using sensors, there are different options to define its behavior in case of an exception raised within the sensor.

* `soft_fail=True`: If an exception is raised within the task, it is marked as skipped, affecting downstream tasks according to their defined [trigger rules](/docs/learn/airflow-trigger-rules).
* `silent_fail=True`: If an exception is raised in the poke method that is **not** one of: AirflowSensorTimeout, AirflowTaskTimeout, AirflowSkipException or AirflowFailException, the sensor will log the error but continue its execution.
* `never_fail=True`: If the poke method raises any exception, the sensor task is skipped. This parameter is mutually exclusive with `soft_fail`.

## Deferrable operators

[Deferrable operators](/docs/learn/deferrable-operators) (sometimes referred to as asynchronous operators) eliminate the problem of having any operator or sensor using a full worker slot for the entire time they run. Many operators have a `deferrable` parameter that can be set to `True` to make the operator deferrable. For the sensors where this parameter isn't available, deferrable versions exist in open source Airflow and in the [Astronomer Providers package](https://github.com/astronomer/astronomer-providers). Astronomer recommends using these in most cases to reduce resource costs.

For DAG authors, using deferrable sensors is no different from using regular sensors. All you need is to do is run a `triggerer` process in Airflow and either:

* Set the Airflow config `operators.default_deferrable` to `True` to set all sensors with a `deferrable` parameter to be deferrable by default.
* Set the `deferrable` parameter to `True` on individual sensor instances you want to run in deferrable mode.
* Replace the name of a sensor with its deferrable counterpart if no `deferrable` parameter is available.

For more details, see [Deferrable operators](/docs/learn/deferrable-operators).


# Airflow operators
Source: https://astronomer.io/docs/learn/what-is-an-operator

Learn the basics of operators, which are the building blocks of Airflow DAGs.

Operators are one of the building blocks of Airflow DAGs. There are many different types of operators available in Airflow. The `PythonOperator` can execute any Python function, and is functionally equivalent to using the `@task` decorator, while other operators contain pre-created logic to perform a specific task, such as executing a Bash script (`BashOperator`) or running a SQL query in a relational database (`SQLExecuteQueryOperator`). Operators are used alongside other building blocks, such as [decorators](/docs/learn/airflow-decorators) and [hooks](/docs/learn/what-is-a-hook), to create tasks in a DAG written with the task-oriented approach. Operators classes can be imported from Airflow provider packages.

In this guide, you'll learn the basics of using operators in Airflow.

To view a list of available operators available in different Airflow provider packages, go to the [Airflow Registry](https://airflow.apache.org/registry).

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Basic Python. See the [Python Documentation](https://docs.python.org/3/tutorial/index.html).

## Operator basics

Operators are Python classes that encapsulate logic to do a unit of work. They can be viewed as a wrapper around each unit of work that defines the actions that will be completed and abstract the majority of code you would typically need to write. When you create an instance of an operator in a DAG and provide it with its required parameters, it becomes a task.

A base set of operators is contained in the [Airflow standard provider](https://airflow.apache.org/docs/apache-airflow-providers-standard/stable/index.html) package, which is pre-installed when using the Astro CLI. Other operators are contained in specialized provider packages, often centered around a specific technology or service. For example, the [Airflow Snowflake Provider](https://airflow.apache.org/registry/providers/snowflake/) package contains operators for interacting with Snowflake, while the [Airflow Google provider](https://airflow.apache.org/registry/providers/google/) package contains operators for interacting with Google Cloud services. There are also several packages that contain operators that can be used with a set of services:

* [Common SQL](https://airflow.apache.org/registry/providers/common-sql/)
* [Common IO](https://airflow.apache.org/registry/providers/common-io/)
* [Common Messaging](https://airflow.apache.org/docs/apache-airflow-providers-common-messaging/stable/index.html)

## Operator examples

Following are some of the most frequently used Airflow operators. Note that only a few of the possible parameters are shown, refer to the [Airflow Registry](https://airflow.apache.org/registry/) for a full list of parameters for each operator.

* [`PythonOperator`](https://airflow.apache.org/registry/providers/standard#standard-python-PythonOperator): Executes a Python function. It is functionally equivalent to using the `@task` decorator. See, [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).

  ```python wrap theme={null}
  from airflow.providers.standard.operators.python import PythonOperator

  def _my_python_function():
      print("Hello world!")

  my_task = PythonOperator(
      task_id="my_task",
      python_callable=_my_python_function,
  )
  ```

* [`BashOperator`](https://airflow.apache.org/registry/providers/standard#standard-bash-BashOperator): Executes a bash script. See also the [Using the `BashOperator`](/docs/learn/bashoperator) guide.

  ```python wrap theme={null}
  from airflow.providers.standard.operators.bash import BashOperator

  my_task = BashOperator(
      task_id="my_task",
      bash_command="echo 'Hello world!'",
  )
  ```

* [`KubernetesPodOperator`](https://airflow.apache.org/registry/providers/cncf-kubernetes#cncf-kubernetes-pod-KubernetesPodOperator): Executes a task defined as a Docker image in a Kubernetes Pod. See, [Use the `KubernetesPodOperator`](/docs/learn/kubepod-operator).

  ```python wrap theme={null}
  from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

  my_task = KubernetesPodOperator(
      task_id="my_task",
      kubernetes_conn_id="<my-kubernetes-connection>",
      name="<my-pod-name>",
      namespace="<my-namespace>",
      image="python:3.12-slim", # Docker image to run
      cmds=["python", "-c"], # Command to run in the container
      arguments=["print('Hello world!')"], # Arguments to the command
  )
  ```

* [`SQLExecuteQueryOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLExecuteQueryOperator): Executes a SQL query against a relational database.

  ```python wrap theme={null}
  from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator

  my_task = SQLExecuteQueryOperator(
      task_id="my_task",
      sql="SELECT * FROM my_table",
      database="<my-database>",
      conn_id="<my-connection>",
  )
  ```

* [`EmptyOperator`](https://airflow.apache.org/registry/providers/standard#standard-empty-EmptyOperator): A no-op operator that does nothing. This is useful for creating placeholder tasks in a DAG.

  ```python wrap theme={null}
  from airflow.providers.standard.operators.empty import EmptyOperator

  my_task = EmptyOperator(task_id="my_task")
  ```

All operators inherit from the abstract [`BaseOperator` class](https://github.com/apache/airflow/blob/main/task-sdk/src/airflow/sdk/bases/operator.py), which contains the logic to execute the work of the operator within the context of a DAG.

Arguments of the `BaseOperator` class can be passed to all operators. The most common arguments are:

* `task_id`: A unique identifier for the task. This is required for all operators.
* `retries`: The number of times to retry the task if it fails. This is optional and defaults to 0. See [Rerun Airflow DAGs and tasks](/docs/learn/rerunning-dags#automatically-retry-tasks).
* `pool`: The name of the pool to use for the task. This is optional and defaults to None. See [Airflow pools](/docs/learn/airflow-pools).
* `execution_timeout`: The maximum time to wait for the task to complete. This is optional and defaults to None. It is a good practice to set this value to prevent tasks from running indefinitely.

You can set these arguments and other `BaseOperator` arguments (other than `task_id` which needs to be unique per operator) at the DAG level for all tasks in a DAG. By using the `default_args` dictionary. You can override these values for individual tasks by setting the same arguments in the task definition.

```python expandable wrap theme={null}
import hashlib
import json

from airflow.exceptions import AirflowException
from airflow.decorators import dag, task
from airflow.models import Variable
from airflow.models.baseoperator import chain
from airflow.operators.empty import EmptyOperator
from airflow.utils.dates import datetime
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
from airflow.providers.amazon.aws.transfers.local_to_s3 import (
    LocalFilesystemToS3Operator,
)
from airflow.providers.amazon.aws.transfers.s3_to_redshift import S3ToRedshiftOperator
from airflow.providers.postgres.operators.postgres import PostgresOperator
from airflow.operators.sql import SQLCheckOperator
from airflow.utils.task_group import TaskGroup


# The file(s) to upload shouldn't be hardcoded in a production setting,
# this is just for demo purposes.
CSV_FILE_NAME = "forestfires.csv"
CSV_FILE_PATH = f"include/sample_data/forestfire_data/{CSV_FILE_NAME}"


@dag(
    "simple_redshift_3",
    start_date=datetime(2021, 7, 7),
    description="""A sample Airflow DAG to load data from csv files to S3
                 and then Redshift, with data integrity and quality checks.""",
    schedule=None,
    template_searchpath="/usr/local/airflow/include/sql/redshift_examples/",
    catchup=False,
)
def simple_redshift_3():
    """
    Before running the DAG, set the following in an Airflow
    or Environment Variable:
    - key: aws_configs
    - value: { "s3_bucket": [bucket_name], "s3_key_prefix": [key_prefix],
             "redshift_table": [table_name]}
    Fully replacing [bucket_name], [key_prefix], and [table_name].
    """

    upload_file = LocalFilesystemToS3Operator(
        task_id="upload_to_s3",
        filename=CSV_FILE_PATH,
        dest_key="{{ var.json.aws_configs.s3_key_prefix }}/" + CSV_FILE_PATH,
        dest_bucket="{{ var.json.aws_configs.s3_bucket }}",
        aws_conn_id="aws_default",
        replace=True,
    )

    @task
    def validate_etag():
        """
        #### Validation task
        Check the destination ETag against the local MD5 hash to ensure
        the file was uploaded without errors.
        """
        s3 = S3Hook()
        aws_configs = Variable.get("aws_configs", deserialize_json=True)
        obj = s3.get_key(
            key=f"{aws_configs.get('s3_key_prefix')}/{CSV_FILE_PATH}",
            bucket_name=aws_configs.get("s3_bucket"),
        )
        obj_etag = obj.e_tag.strip('"')
        # Change `CSV_FILE_PATH` to `CSV_CORRUPT_FILE_PATH` for the "sad path".
        file_hash = hashlib.md5(open(CSV_FILE_PATH).read().encode("utf-8")).hexdigest()
        if obj_etag != file_hash:
            raise AirflowException(
                """Upload Error: Object ETag in S3 did not match
                hash of local file."""
            )

    # Tasks that were created using decorators have to be called to be used
    validate_file = validate_etag()

    # --- Create Redshift Table --- #
    create_redshift_table = PostgresOperator(
        task_id="create_table",
        sql="create_redshift_forestfire_table.sql",
        postgres_conn_id="redshift_default",
    )

    # --- Second load task --- #
    load_to_redshift = S3ToRedshiftOperator(
        task_id="load_to_redshift",
        s3_bucket="{{ var.json.aws_configs.s3_bucket }}",
        s3_key="{{ var.json.aws_configs.s3_key_prefix }}" + f"/{CSV_FILE_PATH}",
        schema="PUBLIC",
        table="{{ var.json.aws_configs.redshift_table }}",
        copy_options=["csv"],
    )

    # --- Redshift row validation task --- #
    validate_redshift = SQLCheckOperator(
        task_id="validate_redshift",
        conn_id="redshift_default",
        sql="validate_redshift_forestfire_load.sql",
        params={"filename": CSV_FILE_NAME},
    )

    # --- Row-level data quality check --- #
    with open("include/validation/forestfire_validation.json") as ffv:
        with TaskGroup(group_id="row_quality_checks") as quality_check_group:
            ffv_json = json.load(ffv)
            for id, values in ffv_json.items():
                values["id"] = id
                SQLCheckOperator(
                    task_id=f"forestfire_row_quality_check_{id}",
                    conn_id="redshift_default",
                    sql="row_quality_redshift_forestfire_check.sql",
                    params=values,
                )

    # --- Drop Redshift table --- #
    drop_redshift_table = PostgresOperator(
        task_id="drop_table",
        sql="drop_redshift_forestfire_table.sql",
        postgres_conn_id="redshift_default",
    )

    begin = EmptyOperator(task_id="begin")
    end = EmptyOperator(task_id="end")

    # --- Define task dependencies --- #
    chain(
        begin,
        upload_file,
        validate_file,
        create_redshift_table,
        load_to_redshift,
        validate_redshift,
        quality_check_group,
        drop_redshift_table,
        end,
    )


simple_redshift_3()
```

## Best practices

Operators typically only require a few parameters. Keep the following considerations in mind when using Airflow operators:

* The [Airflow Registry](https://airflow.apache.org/registry) is the best resource for learning what operators are available and how they are used.
* The [Airflow standard provider](https://airflow.apache.org/docs/apache-airflow-providers-standard/stable/index.html) package includes basic operators such as the `PythonOperator` and `BashOperator`. These operators are automatically available in your Airflow environment if you are using the Astro CLI. All other operators are part of provider packages, some which you must install separately, depending on what type of Airflow distribution you are using.
* You can combine operators and [decorators](/docs/learn/airflow-decorators) freely in the same DAG. Many users choose to use the `@task` decorator for most of their tasks, and add operators for tasks where a specialized operator exists for their use case. The example above shows a DAG with one operator (`BashOperator`) and one `@task` decorated task.
* If an operator exists for your specific use case, you should use it instead of your own Python functions or [hooks](/docs/learn/what-is-a-hook). This makes your DAGs easier to read and maintain.
* If an operator doesn't exist for your use case, you can either use custom Python code in an `@task` decorated task or `PythonOperator` or extend an operator to meet your needs. For more information about customizing operators, see [Custom hooks and operators](/docs/learn/airflow-importing-custom-hooks-operators).
* [Sensors](/docs/learn/what-is-a-sensor) are a type of operator that waits for something to happen. They can be used to detect events in systems outside of Airflow.
* [Deferrable Operators](/docs/learn/deferrable-operators) are a type of operator that releases their worker slot while waiting for their work to be completed. This can result in cost savings and greater scalability. Astronomer recommends using deferrable operators whenever one exists for your use case and your task takes longer than a minute. A lot of operators that potentially need to wait for something have a deferrable mode which you can enable by setting their `deferrable` parameter to `True`.
* Any operator that interacts with a service external to Airflow typically requires a connection so that Airflow can authenticate to that external system. For more information about setting up connections, see [Managing your connections in Apache Airflow](/docs/learn/connections) or in the examples to follow.


# Integrate OpenLineage and Airflow
Source: https://astronomer.io/docs/learn/airflow-openlineage

Learn about OpenLineage concepts and benefits of integrating with Airflow.

[Data lineage](https://en.wikipedia.org/wiki/Data_lineage) is the concept of tracking and visualizing data from its origin to wherever it flows and is consumed downstream. Lineage is growing in importance as companies must rely on increasingly complex data ecosystems to make business-critical decisions. Data lineage can help with everything from understanding your data sources, to troubleshooting job failures, to managing PII, to ensuring compliance with data regulations.

As a one-stop-shop orchestrator for an organization’s data pipelines, Apache Airflow is an ideal platform for integrating data lineage to understand the movement of and interactions within your data.

In this guide, you’ll learn about core data lineage concepts and understand how lineage works with Airflow.

<Tip>
  Astro Observe offers robust support for extracting and visualizing data lineage. To learn more, see [Astro Observe overview](/docs/astro/astro-observe).
</Tip>

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Webinar: [Introduction to Observability for Data Pipelines](https://www.astronomer.io/events/webinars/introduction-to-observability-for-data-pipelines-video/).
  * Webinar: [How to build reliable data products with Astro](https://www.astronomer.io/events/webinars/how-to-build-reliable-data-products-with-astro-video/).
  * Webinar: [OpenLineage and Airflow: A Deeper Dive](https://www.astronomer.io/events/webinars/openlineage-and-airflow-deeper-dive/).
  * Guide: [Leveraging data products for health and performance benefits](/docs/learn/data-products).
</Tip>

## Assumed knowledge

To get the most out of this guide, make sure you have an understanding of:

* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).

## What is data lineage?

Data lineage represents the complex set of relationships that exist among datasets within an ecosystem. Typically, a lineage solution has three basic elements:

* **Lineage metadata**, which describes your datasets (tables in Snowflake or S3 buckets, for example) and jobs (tasks in your DAG, for example).
* **A lineage backend**, which stores and processes lineage metadata.
* **A lineage frontend**, which allows you to view and interact with your lineage metadata — for example, a visual graph of jobs, datasets and columns that shows how they are connected.

If you want to read more about the concept of data lineage and its value, see: [What is data lineage and why does it matter?](https://www.astronomer.io/blog/what-is-data-lineage).

Visually, your data lineage graph might look like this lineage graph in Astro Observe:

<Frame>
  <img alt="Lineage Graph" />
</Frame>

If you are using data lineage, your solution most likely has a lineage backend that collects and stores lineage metadata and a frontend that visualizes the metadata. There are proprietary tools, including Astro Observe, that provide these services, and there are open-source options that can be integrated with Airflow: namely [OpenLineage](https://openlineage.io), a lineage specification and collection API for getting lineage from tools in your pipelines, and [Marquez](https://marquezproject.github.io/marquez/), a tool for storing and visualizing lineage that includes a metadata repository, query API, and graphical UI.

### OpenLineage

[OpenLineage](https://openlineage.io/) is the open-source industry standard framework for data lineage. It standardizes the definition of data lineage, the metadata that makes up lineage metadata, and the approach to collecting lineage metadata from external systems. In other words, it defines a [formalized specification](https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md) for all the core concepts related to data lineage.

The purpose of an open standard for lineage is to create a more cohesive governance and monitoring experience across the industry and reduce duplicated work for stakeholders. It allows for a simpler, more consistent experience when integrating lineage from many different tools, similarly to how Airflow providers reduce the work of dag authoring by providing standardized modules for integrating Airflow with other tools.

In Airflow, OpenLineage implementations use two main components, each serving a distinct function:

* **OpenLineage Airflow Provider (`apache-airflow-providers-openlineage`)**
  This library functions as an adapter, integrating OpenLineage with Airflow. It hooks into Airflow's internal mechanisms to capture metadata about dag and task execution. As a plugin, it is version-coupled with Airflow.

* **OpenLineage Client (`openlineage-python`)**
  This is the core library responsible for building and sending OpenLineage events to backends such as Astro Observe. The client is updated and configured independently of Airflow, allowing users on all Airflow versions to upgrade this package at any time and take advantage of the latest improvements, fixes, or features.

### Get started on Astro

[Astro Observe](/docs/astro/astro-observe) offers the easiest path to reliable lineage from Airflow using OpenLineage. You can use Astro Observe whether you use open-source Airflow, Astro, or another managed service for Airflow.

### Core lineage concepts

The following terms are used frequently when discussing data lineage in general and OpenLineage in particular:

* **Integration**: a means of gathering lineage metadata from a source system such as a scheduler or data platform. For example, the OpenLineage Airflow Provider allows lineage metadata to be collected from Airflow DAGs. Airflow operators automatically gather lineage metadata from the source system every time a DAG runs, preparing and transmitting OpenLineage events to a lineage backend.
* **Job**: a process that consumes or produces datasets. Jobs can be viewed on your lineage graph. In Airflow, an OpenLineage job corresponds to a task in your DAG or the DAG itself. Note that only [supported operators](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/supported_classes.html) will have input/output metadata; other tasks in your DAG may show up as orphans on the lineage graph if those dataset metadata is missing. On Astro, jobs appear as nodes on your lineage graphs in the [lineage UI](/docs/astro/create-data-products).
* **Dataset**: a representation of a set of data in your lineage metadata and graph. For example, it might correspond to a table in your database or a set of data on which you run a Great Expectations check. Typically, a dataset is registered as part of your lineage metadata when a job writing to the dataset is completed (for example, data is inserted into a table).
* **Run**: an instance of a job in which lineage metadata is generated. An OpenLineage run is generated with each DAG and task run, for example.
* **Facet**: a piece of lineage metadata about a job, dataset, or run (for example, you might hear “job facet” in reference to a piece of metadata attached to a job).

Read more about each term in [the OpenLineage Object Model documentation](https://openlineage.io/docs/spec/object-model).

## Why OpenLineage with Airflow?

Using OpenLineage with Airflow allows you to have more insight into the operation and structure of complex data ecosystems and supports better data governance. Airflow is a natural place to integrate data lineage because it is often used as a one-stop-shop orchestrator that touches data across many parts of an organization.

OpenLineage with Airflow provides the following capabilities:

* Quickly find the **root cause** of task failures by identifying issues in upstream datasets (for example, if an upstream job outside Airflow failed to populate a key dataset).
* Easily see the **affected area** of any job failures or changes to data by visualizing the relationship between jobs and datasets, including column-level lineage for some operators.
* Identify where **sensitive data** is used in jobs across an organization.

These capabilities translate into real-world benefits by:

* Making recovery from complex failures faster. The faster you can identify the problem and the affected area, the easier it is to find a solution and prevent erroneous decisions based on bad data.
* Making it easier for teams to work together across an organization. Visualizing the full scope of where an asset is used reduces “sleuthing” time.
* Helping ensure compliance with data regulations by fully understanding where data is used in an organization.

## Lineage on Astro

For Airflow users leveraging [Astro Observe](/docs/astro/astro-observe), data lineage is built-in. Viewing the graph in Astro Observe helps you troubleshoot issues with your data pipelines and understand the movement of data within a [Data Product](/docs/astro/create-data-products). For help getting started with Astro Observe, see: [Astro Observe overview](/docs/astro/astro-observe).

## Lineage with open-source tools

If you want to build your own lineage stack using open-source tools, Astronomer recommends the official [OpenLineage Airflow Provider](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/index.html) for producing lineage. The provider supports many operators, and more are added regularly. You can find a list of currently supported operators in the [Provider documentation](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/supported_classes.html). You won't have to modify your DAGs to start emitting lineage information, but some basic configuration is necessary, including installing a package and setting up a transport. For more details about configuring the Provider, see: [Using OpenLineage integration](https://airflow.apache.org/docs/apache-airflow-providers-openlineage/stable/guides/user.html).

Starting with Apache Airflow version 2.10, the OpenLineage Airflow Provider automatically collects lineage from supported [hooks](/docs/learn/what-is-a-hook). Hook-based lineage enables lineage collection from custom operators, `PythonOperator`, and DAGs using the [TaskFlow API](/docs/learn/airflow-decorators).

To consume and visualize data lineage from Airflow using open-source tools, Astronomer recommends running OpenLineage with [Marquez](https://marquezproject.github.io/marquez/) as your lineage metadata repository, query API (backend), and UI (frontend). See the [Integrate OpenLineage and Airflow locally with Marquez](/docs/learn/marquez) tutorial to get started. For a configuration-free option for demo purposes, you can explore [Marquez on GitPod](https://gitpod.io/#https://github.com/MarquezProject/marquez).


# Orchestrate OpenSearch operations with Apache Airflow
Source: https://astronomer.io/docs/learn/airflow-opensearch

Learn how to integrate OpenSearch and Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[OpenSearch](https://opensearch.org/) is an open source distributed search and analytics engine based on [Apache Lucene](https://lucene.apache.org/). It offers advanced search capabilities on large bodies of text alongside powerful machine learning plugins. The [OpenSearch Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-opensearch/stable/index.html) offers modules to easily integrate OpenSearch with Airflow.

In this tutorial you'll use Airflow to create an index in OpenSearch, ingest the lyrics of the musical [Hamilton](https://hamiltonmusical.com/new-york/) into the index, and run a search query on the index to see which character most often sings a specific word.

## Why use Airflow with OpenSearch?

OpenSearch allows you to perform complex search queries on indexed text documents. Additionally, the tool comes with a variety of plugins for use cases such as security analytics, semantic search, and neural search.

Integrating OpenSearch with Airflow allows you to:

* Use Airflow's [data-driven scheduling](/docs/learn/airflow-datasets) to run operations involving documents stored in OpenSearch based on upstream events in your data ecosystem, such as when a new model is trained or a new dataset is available.
* Run dynamic queries based on upstream events in your data ecosystem or user input via [Airflow params](/docs/learn/airflow-params) on documents and vectors stored in OpenSearch to retrieve relevant objects.
* Add Airflow features like [retries](/docs/learn/rerunning-dags#automatically-retry-tasks) and [alerts](/docs/learn/error-notifications-in-airflow) to your OpenSearch operations.

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of OpenSearch. See the [OpenSearch documentation](https://opensearch.org/docs/latest/about/).
* Vector embeddings. See [Using OpenSearch as a Vector Database](https://opensearch.org/platform/search/vector-database.html).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow decorators. [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).
* Airflow connections. See [Managing your Connections in Apache Airflow](/docs/learn/connections).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).

This tutorial uses a local OpenSearch instance created as a [Docker container](https://hub.docker.com/r/opensearchproject/opensearch). You don't need to install OpenSearch on your machine.

<Info>
  The example code from this tutorial is also available on [GitHub](https://github.com/astronomer/airflow-opensearch-tutorial).
</Info>

## Step 1: Configure your Astro project

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-opensearch-tutorial && cd astro-opensearch-tutorial
   $ astro dev init
   ```

2. Add the following two lines to your Astro project `requirements.txt` file to install the [OpenSearch Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-opensearch/stable/index.html) and the [pandas](https://pandas.pydata.org/) package in your Astro project:

   ```text wrap theme={null}
   apache-airflow-providers-opensearch==1.0.0
   pandas==1.5.3
   ```

3. This tutorial uses a local OpenSearch instance running in a Docker container. To run an OpenSearch container as part of your Airflow environment, create a new file in your Astro project root directory called `docker-compose.override.yml` and copy and paste the following into it:

   ```yaml expandable wrap theme={null}
   version: '3.1'
   services:
     opensearch:
       image: opensearchproject/opensearch:2
       ports:
         - "9200:9200"  # OpenSearch REST API
         - "9300:9300"  # OpenSearch Node-to-Node communication
       environment:
         - discovery.type=single-node
         - plugins.security.ssl.http.enabled=false
       volumes:
         - opensearch-data:/usr/share/opensearch/data
       networks:
         - airflow
   # Airflow containers
     scheduler:
       networks:
         - airflow
     webserver:
       networks:
         - airflow
     triggerer:
       networks:
         - airflow
     postgres:
       networks:
         - airflow

   # volume for OpenSearch
   volumes:
     opensearch-data:
   ```

4. Add the following configuration to your `.env` file to create an [Airflow connection](/docs/learn/connections) between Airflow and your OpenSearch instance. If you already have a cloud-based OpenSearch instance, you can connect to that instead of the local instance by adjusting the values in the connection.

   ```text wrap theme={null}
   AIRFLOW_CONN_OPENSEARCH_DEFAULT='{
       "conn_type": "opensearch",
       "host": "opensearch",
       "port": 9200,
       "login": "admin",
       "password": "admin"
   }'
   ```

## Step 2: Add your data

The DAG in this tutorial uses a [Kaggle](https://www.kaggle.com/datasets/lbalter/hamilton-lyrics) dataset that contains the lyrics of the musical [Hamilton](https://hamiltonmusical.com/new-york/).

1. Download the [`hamilton_lyrics`.csv](https://github.com/astronomer/airflow-opensearch-tutorial/blob/main/include/hamilton_lyrics.csv) from Astronomer's GitHub.

2. Save the file in your Astro project `include` folder.

## Step 3: Create your DAG

1. In your `dags` folder, create a file called `search_hamilton.py`.

2. Copy the following code into the file.

   ```python expandable wrap theme={null}
   """
   ## Use the OpenSearch provider to ingest and search Hamilton lyrics

   This DAG uses the OpenSearch provider to create an index in OpenSearch,
   ingest Hamilton lyrics into the index, and search for which character and which song
   mention a keyword the most.
   """

   from airflow.decorators import dag, task
   from airflow.models.baseoperator import chain
   from airflow.operators.empty import EmptyOperator
   from airflow.providers.opensearch.operators.opensearch import (
       OpenSearchAddDocumentOperator,
       OpenSearchCreateIndexOperator,
       OpenSearchQueryOperator,
   )
   from airflow.providers.opensearch.hooks.opensearch import OpenSearchHook
   from pendulum import datetime
   import csv
   import uuid
   import pandas as pd

   OPENSEARCH_INDEX_NAME = "hamilton_lyrics"
   OPENSEARCH_CONN_ID = "opensearch_default"
   LYRICS_CSV_PATH = "include/hamilton_lyrics.csv"
   KEYWORD_TO_SEARCH = "write"


   @dag(
       start_date=datetime(2023, 10, 18),
       schedule=None,
       catchup=False,
   )
   def search_hamilton():
       @task.branch
       def check_if_index_exists(index_name: str, conn_id: str) -> str:
           client = OpenSearchHook(open_search_conn_id=conn_id, log_query=True).client
           is_index_exist = client.indices.exists(index_name)
           if is_index_exist:
               return "index_exists"
           return "create_index"

       create_index = OpenSearchCreateIndexOperator(
           task_id="create_index",
           opensearch_conn_id=OPENSEARCH_CONN_ID,
           index_name=OPENSEARCH_INDEX_NAME,
           index_body={
               "settings": {"index": {"number_of_shards": 1}},
               "mappings": {
                   "properties": {
                       "title": {"type": "keyword"},
                       "speaker": {
                           "type": "keyword",
                       },
                       "lines": {"type": "text"},
                   }
               },
           },
       )

       index_exists = EmptyOperator(task_id="index_exists")

       @task
       def csv_to_dict_list(csv_file_path: str) -> list:
           with open(csv_file_path, mode="r", encoding="utf-8") as file:
               reader = csv.DictReader(file)
               list_of_hamilton_lines = list(reader)

           list_of_kwargs = []
           for line in list_of_hamilton_lines:
               unique_line_id = uuid.uuid5(
                   name=" ".join([line["title"], line["speaker"], line["lines"]]),
                   namespace=uuid.NAMESPACE_DNS,
               )
               kwargs = {"doc_id": str(unique_line_id), "document": line}

               list_of_kwargs.append(kwargs)

           return list_of_kwargs

       list_of_document_kwargs = csv_to_dict_list(csv_file_path=LYRICS_CSV_PATH)

       add_lines_as_documents = OpenSearchAddDocumentOperator.partial(
           task_id="add_lines_as_documents",
           opensearch_conn_id=OPENSEARCH_CONN_ID,
           trigger_rule="none_failed",
           index_name=OPENSEARCH_INDEX_NAME,
       ).expand_kwargs(list_of_document_kwargs)

       search_for_keyword = OpenSearchQueryOperator(
           task_id=f"search_for_{KEYWORD_TO_SEARCH}",
           opensearch_conn_id=OPENSEARCH_CONN_ID,
           index_name=OPENSEARCH_INDEX_NAME,
           query={
               "size": 0,
               "query": {
                   "match": {"lines": {"query": KEYWORD_TO_SEARCH, "fuzziness": "AUTO"}}
               },
               "aggs": {
                   "most_mentions_person": {"terms": {"field": "speaker"}},
                   "most_mentions_song": {"terms": {"field": "title"}},
               },
           },
       )

       @task
       def print_query_result(query_result: dict, keyword: str) -> None:
           results_most_mentions_person = query_result["aggregations"][
               "most_mentions_person"
           ]["buckets"]
           results_most_mentions_song = query_result["aggregations"]["most_mentions_song"][
               "buckets"
           ]

           df_person = pd.DataFrame(results_most_mentions_person)
           df_person.columns = ["Character", f"Number of lines that include '{keyword}'"]
           df_song = pd.DataFrame(results_most_mentions_song)
           df_song.columns = ["Song", f"Number of lines that include '{keyword}'"]

           print(
               f"\n Top 3 Hamilton characters that mention '{keyword}' the most:\n ",
               df_person.head(3).to_string(index=False),
           )
           print(
               f"\n Top 3 Hamilton songs that mention '{keyword}' the most:\n ",
               df_song.head(3).to_string(index=False),
           )

       chain(
           check_if_index_exists(
               index_name=OPENSEARCH_INDEX_NAME, conn_id=OPENSEARCH_CONN_ID
           ),
           [create_index, index_exists],
           add_lines_as_documents,
       )
       chain(
           list_of_document_kwargs,
           add_lines_as_documents,
           search_for_keyword,
           print_query_result(
               query_result=search_for_keyword.output,
               keyword=KEYWORD_TO_SEARCH,
           ),
       )


   search_hamilton()
   ```

This DAG consists of seven tasks to make a simple ML orchestration pipeline.

* The `check_if_index_exists` task uses a [`@task.branch`](/docs/learn/airflow-branch-operator#@task-branch-branchpythonoperator) decorator to check if the index `OPENSEARCH_INDEX_NAME` exists in your OpenSearch instance. If it doesn't exist, the task returns the string `create_index` causing the downstream `create_index` task to run. If the index exists, the task causes the empty `index_exists` task to run instead.
* The `create_index` task defined with the [`OpenSearchCreateIndexOperator`](https://airflow.apache.org/registry/providers/opensearch#opensearch-opensearch-OpenSearchCreateIndexOperator) creates the index `OPENSEARCH_INDEX_NAME` in your OpenSearch instance with the three properties `title`, `speaker` and `lines`.
* The `csv_to_dict_list` task uses the [`@task`](/docs/learn/airflow-decorators) decorator to ingest the lyrics of the musical Hamilton from the `hamilton_lyrics.csv` file into a list of Python dictionaries. Each dictionary represents a line of the musical and will be one document in the OpenSearch index.
* The `add_lines_as_documents` task is a [dynamically mapped task](/docs/learn/dynamic-tasks) using the [`OpenSearchAddDocumentOperator`](https://airflow.apache.org/registry/providers/opensearch#opensearch-opensearch-OpenSearchAddDocumentOperator) to create one mapped task instance for each document to ingest.
* The `search_for_keyword` task is defined with the [`OpenSearchQueryOperator`](https://airflow.apache.org/registry/providers/opensearch#opensearch-opensearch-OpenSearchQueryOperator) and performs a [fuzzy query](https://opensearch.org/docs/latest/query-dsl/term/fuzzy/) on the OpenSearch index to find the character and song that mention the `KEYWORD_TO_SEARCH` the most.
* The `print_query_result` prints the query results to the task logs.

<Frame>
  <img alt="Screenshot of the Airflow UI showing the successful completion of the search_hamilton DAG in the Grid view with the Graph tab selected." />
</Frame>

<Tip>
  For information on more advanced search techniques in OpenSearch, see the [OpenSearch documentation](https://opensearch.org/docs/latest/).
</Tip>

## Step 4: Run your DAG

1. Run `astro dev start` in your Astro project to start Airflow and open the Airflow UI at `localhost:8080`.

2. In the Airflow UI, run the `search_hamilton` DAG by clicking the **Play** button. By default the DAG will search the lyrics for the word `write`, but you can change the search term by updating the `KEYWORD_TO_SEARCH` variable in your DAG file.

3. View your song results in the task logs of the `print_query_result` task:

   ```text wrap theme={null}
   [2023-11-22, 14:01:58 UTC] {logging_mixin.py:154} INFO - 
    Top 3 Hamilton characters that mention 'write' the most:
     Character  Number of lines that include 'write'
     HAMILTON                                    15
        ELIZA                                     8
         BURR                                     4
   [2023-11-22, 14:01:58 UTC] {logging_mixin.py:154} INFO - 
    Top 3 Hamilton songs that mention 'write' the most:
        Song  Number of lines that include 'write'
     Non-Stop                                    11
    Hurricane                                    10
         Burn                                     3
   ```

4. (Optional) Listen to the [song](https://open.spotify.com/track/7qfoq1JFKBUEIvhqOHzuqX?si=49a2e7c259ad43e2) that mentions your keyword the most.

## Conclusion

Congratulations! You used Airflow and OpenSearch to analyze the lyrics of Hamilton! You can now use Airflow to orchestrate OpenSearch operations in your own machine learning pipelines. History has its eyes on you.


# Orchestrate pgvector operations with Apache Airflow
Source: https://astronomer.io/docs/learn/airflow-pgvector

Learn how to integrate pgvector and Airflow.

[Pgvector](https://github.com/pgvector/pgvector) is an open source extension for PostgreSQL databases that adds the possibility to store and query high-dimensional object embeddings. The [pgvector Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-pgvector/stable/index.html) offers modules to easily integrate pgvector with Airflow.

In this tutorial, you use Airflow to orchestrate the embedding of book descriptions with the OpenAI API, ingest the embeddings into a PostgreSQL database with pgvector installed, and query the database for books that match a user-provided mood.

## Why use Airflow with pgvector?

Pgvector allows you to store objects alongside their vector embeddings and to query these objects based on their similarity. Vector embeddings are key components of many modern machine learning models such as [LLMs](https://en.wikipedia.org/wiki/Large_language_model) or [ResNet](https://arxiv.org/abs/1512.03385).

Integrating PostgreSQL with pgvector and Airflow into one end-to-end machine learning pipeline allows you to:

* Use Airflow's [data-driven scheduling](/docs/learn/airflow-datasets) to run operations involving vectors stored in PostgreSQL based on upstream events in your data ecosystem, such as when a new model is trained or a new dataset is available.
* Run dynamic queries based on upstream events in your data ecosystem or user input via [Airflow params](/docs/learn/airflow-params) on vectors stored in PostgreSQL to retrieve similar objects.
* Add Airflow features like [retries](/docs/learn/rerunning-dags#automatically-retry-tasks) and [alerts](/docs/learn/error-notifications-in-airflow) to your pgvector operations.
* Check your vector database for the existence of a unique key before running potentially costly embedding operations on your data.

## Time to complete

This tutorial takes approximately 30 minutes to complete (reading your suggested book not included).

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of pgvector. See the [README of the pgvector repository](https://github.com/pgvector/pgvector/blob/master/README).
* Basic SQL. See [SQL Tutorial](https://www.w3schools.com/sql/sql_intro.asp).
* Vector embeddings. See [Vector Embeddings](https://tembo.io/blog/pgvector-and-embedding-solutions-with-postgres/).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow decorators. [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).
* Airflow connections. See [Managing your Connections in Apache Airflow](/docs/learn/connections).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).
* An OpenAI API key of at least [tier 1](https://platform.openai.com/docs/guides/rate-limits/usage-tiers) if you want to use OpenAI for vectorization. If you don't want to use OpenAI, you can adapt the `create_embeddings` function at the start of the DAG to use a different vectorizer.

This tutorial uses a local PostgreSQL database created as a Docker container. [The image](https://hub.docker.com/r/ankane/pgvector) comes with pgvector preinstalled.

<Info>
  The example code from this tutorial is also available on [GitHub](https://github.com/astronomer/airflow-pgvector-tutorial).
</Info>

## Step 1: Configure your Astro project

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-pgvector-tutorial && cd astro-pgvector-tutorial
   $ astro dev init
   ```

2. Add the following two packages to your `requirements.txt` file to install the [pgvector Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-pgvector/stable/index.html) and the [OpenAI Python client](https://platform.openai.com/docs/libraries) in your Astro project:

   ```text wrap theme={null}
   apache-airflow-providers-pgvector==1.0.0
   openai==1.3.2
   ```

3. This tutorial uses a local PostgreSQL database running in a Docker container. To add a second PostgreSQL container to your Astro project, create a new file in your project's root directory called `docker-compose.override.yml` and add the following. The `ankane/pgvector` image builds a PostgreSQL database with pgvector preinstalled.

   ```yaml wrap theme={null}
   services:
     postgres_pgvector:
       image: ankane/pgvector
       volumes:
         - ${PWD}/include/postgres:/var/lib/postgresql/data
         - ${PWD}/include:/include
       networks:
         - airflow
       ports:
         - 5433:5432
       environment:
         - POSTGRES_USER=postgres
         - POSTGRES_PASSWORD=postgres
   # Airflow containers
     scheduler:
       networks:
         - airflow
     api-server:
       networks:
         - airflow
     triggerer:
       networks:
         - airflow
     postgres:
       networks:
         - airflow
   ```

4. To create an [Airflow connection](/docs/learn/connections) to the PostgreSQL database, add the following to your `.env` file. If you are using the OpenAI API for embeddings you will need to update the `OPENAI_API_KEY` environment variable.

   ```text wrap theme={null}
   AIRFLOW_CONN_POSTGRES_DEFAULT='{
       "conn_type": "postgres",
       "login": "postgres",
       "password": "postgres",
       "host": "host.docker.internal",
       "port": 5433,
       "schema": "postgres"
   }'
   OPENAI_API_KEY="<your-openai-api-key>"
   ```

## Step 2: Add your data

The DAG in this tutorial runs a query on vectorized book descriptions from [Goodreads](https://www.goodreads.com/), but you can adjust the DAG to use any data you want.

1. Create a new file called `book_data.txt` in the `include` directory.

2. Copy the book description from the [`book_data.txt`](https://github.com/astronomer/airflow-pgvector-tutorial/blob/main/include/book_data.txt) file in Astronomer's GitHub for a list of great books.

<Tip>
  If you want to add your own books make sure the data is in the following format:

  ```text wrap theme={null}
  <index integer> ::: <title> (<year of publication>) ::: <author> ::: <description>
  ```

  One book corresponds to one line in the file.
</Tip>

## Step 3: Create your DAG

1. In your `dags` folder, create a file called `query_book_vectors.py`.

2. Copy the following code into the file. If you want to use a vectorizer other than OpenAI, make sure to adjust both the `create_embeddings` function at the start of the DAG and provide the correct `MODEL_VECTOR_LENGTH`.

   ```python expandable wrap theme={null}
   """
   ## Vectorize book descriptions with OpenAI and store them in Postgres with pgvector

   This DAG shows how to use the OpenAI API 1.0+ to vectorize book descriptions and 
   store them in Postgres with the pgvector extension.
   It will also help you pick your next book to read based on a mood you describe.

   You will need to set the following environment variables:
   - `AIRFLOW_CONN_POSTGRES_DEFAULT`: an Airflow connection to your Postgres database
       that has pgvector installed
   - `OPENAI_API_KEY`: your OpenAI API key
   """

   from airflow.sdk import dag, task
   from airflow.models.baseoperator import chain
   from airflow.models.param import Param
   from airflow.providers.pgvector.operators.pgvector import PgVectorIngestOperator
   from airflow.providers.postgres.operators.postgres import PostgresOperator
   from airflow.exceptions import AirflowSkipException
   from pendulum import datetime
   from openai import OpenAI
   import uuid
   import re
   import os

   POSTGRES_CONN_ID = "postgres_default"
   TEXT_FILE_PATH = "include/book_data.txt"
   TABLE_NAME = "Book"
   OPENAI_MODEL = "text-embedding-ada-002"
   MODEL_VECTOR_LENGTH = 1536


   def create_embeddings(text: str, model: str):
       """Create embeddings for a text with the OpenAI API."""
       client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
       response = client.embeddings.create(input=text, model=model)
       embeddings = response.data[0].embedding

       return embeddings


   @dag(
       start_date=datetime(2025, 8, 1),
       schedule=None,
       tags=["pgvector"],
       params={
           "book_mood": Param(
               "A philosophical book about consciousness.",
               type="string",
               description="Describe the kind of book you want to read.",
           ),
       },
   )
   def query_book_vectors():
       enable_vector_extension_if_not_exists = PostgresOperator(
           task_id="enable_vector_extension_if_not_exists",
           postgres_conn_id=POSTGRES_CONN_ID,
           sql="CREATE EXTENSION IF NOT EXISTS vector;",
       )

       create_table_if_not_exists = PostgresOperator(
           task_id="create_table_if_not_exists",
           postgres_conn_id=POSTGRES_CONN_ID,
           sql=f"""
           CREATE TABLE IF NOT EXISTS {TABLE_NAME} (          
               book_id UUID PRIMARY KEY,
               title TEXT,
               year INTEGER,
               author TEXT,
               description TEXT,
               vector VECTOR(%(vector_length)s)
           );
           """,
           parameters={"vector_length": MODEL_VECTOR_LENGTH},
       )

       get_already_imported_book_ids = PostgresOperator(
           task_id="get_already_imported_book_ids",
           postgres_conn_id=POSTGRES_CONN_ID,
           sql=f"""
           SELECT book_id
           FROM {TABLE_NAME};
           """,
       )

       @task
       def import_book_data(text_file_path: str, table_name: str) -> list:
           "Read the text file and create a list of dicts from the book information."
           with open(text_file_path, "r") as f:
               lines = f.readlines()

               num_skipped_lines = 0
               list_of_params = []
               for line in lines:
                   parts = line.split(":::")
                   title_year = parts[1].strip()
                   match = re.match(r"(.+) \((\d{4})\)", title_year)
                   try:
                       title, year = match.groups()
                       year = int(year)
                   # skip malformed lines
                   except:
                       num_skipped_lines += 1
                       continue

                   author = parts[2].strip()
                   description = parts[3].strip()

                   list_of_params.append(
                       {
                           "book_id": str(
                               uuid.uuid5(
                                   name=" ".join([title, str(year), author, description]),
                                   namespace=uuid.NAMESPACE_DNS,
                               )
                           ),
                           "title": title,
                           "year": year,
                           "author": author,
                           "description": description,
                       }
                   )

               print(
                   f"Created a list with {len(list_of_params)} elements "
                   " while skipping {num_skipped_lines} lines."
               )
               return list_of_params

       @task
       def create_embeddings_book_data(
           book_data: dict, model: str, already_imported_books: list
       ) -> dict:
           "Create embeddings for a book description and add them to the book data."
           already_imported_books_ids = [x[0] for x in already_imported_books]
           if book_data["book_id"] in already_imported_books_ids:
               raise AirflowSkipException("Book already imported.")
           embeddings = create_embeddings(text=book_data["description"], model=model)
           book_data["vector"] = embeddings
           return book_data

       @task
       def create_embeddings_query(model: str, **context) -> list:
           "Create embeddings for the user provided book mood."
           query = context["params"]["book_mood"]
           embeddings = create_embeddings(text=query, model=model)
           return embeddings

       book_data = import_book_data(text_file_path=TEXT_FILE_PATH, table_name=TABLE_NAME)
       book_embeddings = create_embeddings_book_data.partial(
           model=OPENAI_MODEL,
           already_imported_books=get_already_imported_book_ids.output,
       ).expand(book_data=book_data)
       query_vector = create_embeddings_query(model=OPENAI_MODEL)

       import_embeddings_to_pgvector = PgVectorIngestOperator.partial(
           task_id="import_embeddings_to_pgvector",
           trigger_rule="none_failed",
           conn_id=POSTGRES_CONN_ID,
           sql=(
               f"INSERT INTO {TABLE_NAME} "
               "(book_id, title, year, author, description, vector) "
               "VALUES (%(book_id)s, %(title)s, %(year)s, "
               "%(author)s, %(description)s, %(vector)s) "
               "ON CONFLICT (book_id) DO NOTHING;"
           ),
       ).expand(parameters=book_embeddings)

       get_a_book_suggestion = PostgresOperator(
           task_id="get_a_book_suggestion",
           postgres_conn_id=POSTGRES_CONN_ID,
           trigger_rule="none_failed",
           sql=f"""
               SELECT title, year, author, description
               FROM {TABLE_NAME}
               ORDER BY vector <-> CAST(%(query_vector)s AS VECTOR)
               LIMIT 1;
           """,
           parameters={"query_vector": query_vector},
       )

       @task
       def print_suggestion(query_result, **context):
           "Print the book suggestion."
           query = context["params"]["book_mood"]
           book_title = query_result[0][0]
           book_year = query_result[0][1]
           book_author = query_result[0][2]
           book_description = query_result[0][3]
           print(f"Book suggestion for '{query}':")
           print(
               f"You should read {book_title} by {book_author}, published in {book_year}!"
           )
           print(f"Goodreads describes the book as: {book_description}")

       chain(
           enable_vector_extension_if_not_exists,
           create_table_if_not_exists,
           get_already_imported_book_ids,
           import_embeddings_to_pgvector,
           get_a_book_suggestion,
           print_suggestion(query_result=get_a_book_suggestion.output),
       )

       chain(query_vector, get_a_book_suggestion)
       chain(get_already_imported_book_ids, book_embeddings)


   query_book_vectors()
   ```

   This DAG consists of nine tasks to make a simple ML orchestration pipeline.

   * The `enable_vector_extension_if_not_exists` task uses a [`PostgresOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLExecuteQueryOperator) to enable the pgvector extension in the PostgreSQL database.
   * The `create_table_if_not_exists` task creates the `Book` table in PostgreSQL. Note the `VECTOR()` datatype used for the `vector` column. This datatype is added to PostgreSQL by the pgvector extension and needs to be defined with the vector length of the vectorizer you use as an argument. This example uses the OpenAI's `text-embedding-ada-002` to create 1536-dimensional vectors, so we define the columns with the type `VECTOR(1536)` using parameterized SQL.
   * The `get_already_imported_book_ids` task queries the `Book` table to return all `book_id` values of books that were already stored with their vectors in previous DAG runs.
   * The `import_book_data` task uses the [`@task` decorator](/docs/learn/airflow-decorators) to read the book data from the `book_data.txt` file and return it as a list of dictionaries with keys corresponding to the columns of the `Book` table.
   * The `create_embeddings_book_data` task is [dynamically mapped](/docs/learn/dynamic-tasks) over the list of dictionaries returned by the `import_book_data` task to parallelize vector embedding of all book descriptions that haven't been added to the `Book` table in previous DAG runs. The `create_embeddings` function defines how the embeddings are computed and can be modified to use other embedding models. If all books in the list have already been added to the `Book` table, then all mapped task instances are skipped.
   * The `create_embeddings_query` task applies the same `create_embeddings` function to the desired book mood the user provided via [Airflow params](/docs/learn/airflow-params).
   * The `import_embeddings_to_pgvector` task uses the [`PgVectorIngestOperator`](https://airflow.apache.org/registry/providers/pgvector#pgvector-pgvector-PgVectorIngestOperator) to insert the book data including the embedding vectors into the PostgreSQL database. This task is dynamically mapped to import the embeddings from one book at a time. The dynamically mapped task instances of books that have already been imported in previous DAG runs are skipped.
   * The `get_a_book_suggestion` task queries the PostgreSQL database for the book that is most similar to the user-provided mood using nearest neighbor search. Note how the vector of the user-provided book mood (`query_vector`) is cast to the `VECTOR` datatype before similarity search: `ORDER BY vector <-> CAST(%(query_vector)s AS VECTOR)`.
   * The `print_book_suggestion` task prints the book suggestion to the task logs.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the successful completion of the query_book_vectors DAG in the Grid view with the Graph tab selected." />
   </Frame>

<Tip>
  For information on more advanced search techniques in pgvector, see the [pgvector README](https://github.com/pgvector/pgvector/blob/master/README).
</Tip>

## Step 4: Run your DAG

1. Run `astro dev start` in your Astro project to start Airflow and open the Airflow UI at `localhost:8080`.

2. In the Airflow UI, run the `query_book_vectors` DAG by clicking the **Play** button. Then, provide the [Airflow param](/docs/learn/airflow-params) for the desired `book_mood`.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the input form for the book_mood param." />
   </Frame>

3. View your book suggestion in the task logs of the `print_book_suggestion` task:

   ```text wrap theme={null}
   [2025-08-27, 09:45:54] INFO - Book suggestion for 'A philosophical book about consciousness.':: chan="stdout": source="task"
   [2025-08-27, 09:45:54] INFO - You should read The Idea of the World by Bernardo Kastrup, published in 2019!: chan="stdout": source="task"
   [2025-08-27, 09:45:54] INFO - Goodreads describes the book as: A rigorous case for the primacy of mind in nature, from philosophy to neuroscience, psychology and physics. The Idea of the World offers a grounded alternative to the frenzy of unrestrained abstractions and unexamined assumptions in philosophy and science today. [...]
   ```

## Step 5: (Optional) Fetch and read the book

1. Go to the website of your local library and search for the book. If it is available, order it and wait for it to arrive. You will likely need a library card to check out the book.
2. Make sure to prepare an adequate amount of tea for your reading session. Astronomer recommends [Earl Grey](https://en.wikipedia.org/wiki/Earl_Grey_tea), but you can use any tea you like.
3. Enjoy your book!

## Conclusion

Congratulations! You used Airflow and pgvector to get a book suggestion! You can now use Airflow to orchestrate pgvector operations in your own machine learning pipelines. Additionally, you remembered the satisfaction and joy of spending hours reading a good book and supported your local library.


# Orchestrate Pinecone operations with Apache Airflow
Source: https://astronomer.io/docs/learn/airflow-pinecone

Learn how to integrate Pinecone and Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Pinecone](https://www.pinecone.io/) is a proprietary vector database platform designed for handling large-scale vector based AI applications. The [Pinecone Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-pinecone/stable/index.html) offers modules to easily integrate Pinecone with Airflow.

In this tutorial you'll use Airflow to create vector embeddings of series descriptions, create an index in your Pinecone project, ingest the vector embeddings into that index, and query Pinecone to get a suggestion for your next binge-watchable series based on your current mood.

## Why use Airflow with Pinecone?

Integrating Pinecone with Airflow provides a robust solution for managing large-scale vector search workflows in your AI applications. Pinecone specializes in efficient vector storage and similarity search, which is essential for leveraging advanced models like language transformers or deep neural networks.

By combining Pinecone with Airflow, you can:

* Use Airflow's [data-driven scheduling](/docs/learn/airflow-datasets) to run operations in Pinecone based on upstream events in your data ecosystem, such as when a new model is trained or a new dataset is available.
* Run dynamic queries with [dynamic task mapping](/docs/learn/dynamic-tasks), for example to parallelize vector ingestion or search operations to improve performance.
* Add Airflow features like [retries](/docs/learn/rerunning-dags#automatically-retry-tasks) and [alerts](/docs/learn/error-notifications-in-airflow) to your Pinecone operations. Retries protect your MLOps pipelines from transient failures, and alerts notify you of events like task failures or missed service level agreements (SLAs).

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of Pinecone. See [Pinecone Introduction](https://docs.pinecone.io/docs/overview).
* The basics of vector embeddings. See [Vector Embeddings for Developers: The Basics](https://www.pinecone.io/learn/vector-embeddings-for-developers/).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow decorators. See [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow hooks. See [Hooks 101](/docs/learn/what-is-a-hook).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).
* A [Pinecone account](https://app.pinecone.io/?sessionType=signup) with an [API key](https://docs.pinecone.io/docs/authentication). You can use a free tier account for this tutorial.
* An OpenAI API key of at least [tier 1](https://platform.openai.com/docs/guides/rate-limits/usage-tiers) if you want to use OpenAI for vectorization. If you don't want to use OpenAI you can adapt the `create_embeddings` function at the start of the DAG to use a different vectorizer. Note that you will likely need to adjust the `EMBEDDING_MODEL_DIMENSIONS` parameter in the DAG if you use a different vectorizer.

<Info>
  The example code from this tutorial is also available on [GitHub](https://github.com/astronomer/airflow-pinecone-tutorial).
</Info>

## Step 1: Configure your Astro project

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-pinecone-tutorial && cd astro-pinecone-tutorial
   $ astro dev init
   ```

2. Add the following two lines to your `requirements.txt` file to install the [Pinecone Airflow Provider](https://airflow.apache.org/docs/apache-airflow-providers-pinecone/stable/index.html) and [OpenAI Python client](https://platform.openai.com/docs/libraries) in your Astro project:

   ```text wrap theme={null}
   apache-airflow-providers-pinecone==1.0.0
   openai==1.3.2
   ```

3. Add the following environment variables to your Astro project `.env` file. These variables store the configuration for an [Airflow connection](/docs/learn/connections) to your Pinecone account and allow you to use the OpenAI API. Provide your own values for `<your-pinecone-environment>` (for example `gcp-starter`), `<your-pinecone-api-key>` and `<your-openai-api-key>`:

   ```text wrap theme={null}
   AIRFLOW_CONN_PINECONE_DEFAULT='{
       "conn_type": "pinecone",
       "login": "<your-pinecone-environment>",
       "password": "<your-pinecone-api-key>"
   }'
   OPENAI_API_KEY="<your-openai-api-key>"
   ```

## Step 2: Add your data

The DAG in this tutorial runs a query on vectorized series descriptions, which were mostly retrieved from [IMDB](https://www.imdb.com/) with added domain expert inputs.

1. In your Astro project `include` directory, create a file called `series_data.txt`.
2. Copy and paste the following text into the file:

   ```text wrap theme={null}
   1 ::: Star Trek: Discovery (2017) ::: sci-fi ::: Ten years before Kirk, Spock, and the Enterprise, the USS Discovery discovers new worlds and lifeforms using a new innovative mushroom based propulsion system. 
   2 ::: Feel Good (2020) ::: romance ::: The series follows recovering addict and comedian Mae, who is trying to control the addictive behaviors and intense romanticism that permeate every facet of their life.
   3 ::: For All Mankind (2019) ::: sci-fi ::: The series dramatizes an alternate history depicting "what would have happened if the global space race had never ended" after the Soviet Union succeeds in the first crewed Moon landing ahead of the United States.
   4 ::: The Legend of Korra (2012) ::: anime ::: Avatar Korra fights to keep Republic City safe from the evil forces of both the physical and spiritual worlds.
   5 ::: Mindhunter (2017) ::: crime ::: In the late 1970s, two FBI agents broaden the realm of criminal science by investigating the psychology behind murder and end up getting too close to real-life monsters.
   6 ::: The Umbrella Academy (2019) ::: adventure ::: A family of former child heroes, now grown apart, must reunite to continue to protect the world.
   7 ::: Star Trek: Picard (2020) ::: sci-fi ::: Follow-up series to Star Trek: The Next Generation (1987) and Star Trek: Nemesis (2002) that centers on Jean-Luc Picard in the next chapter of his life.
   8 ::: Invasion (2021) ::: sci-fi ::: Earth is visited by an alien species that threatens humanity's existence. Events unfold in real time through the eyes of five ordinary people across the globe as they struggle to make sense of the chaos unraveling around them.
   ```

## Step 3: Create your DAG

1. In your Astro project `dags` folder, create a file called `query_series_vectors.py`.

2. Copy the following code into the file:

   ```python expandable wrap theme={null}
   """
   ## Use the Pinecone Airflow Provider to generate and query vectors for series descriptions

   This DAG runs a simple MLOps pipeline that uses the Pinecone Airflow Provider to import
   series descriptions, generate vectors for them, and query the vectors for series based on
   a user-provided mood.
   """

   from airflow.decorators import dag, task
   from airflow.models.param import Param
   from airflow.models.baseoperator import chain
   from airflow.providers.pinecone.operators.pinecone import PineconeIngestOperator
   from airflow.providers.pinecone.hooks.pinecone import PineconeHook
   from pendulum import datetime
   from openai import OpenAI
   import uuid
   import re
   import os

   PINECONE_INDEX_NAME = "series-to-watch"
   DATA_FILE_PATH = "include/series_data.txt"
   PINECONE_CONN_ID = "pinecone_default"
   EMBEDDING_MODEL = "text-embedding-ada-002"
   EMBEDDING_MODEL_DIMENSIONS = 1536


   def generate_uuid5(identifier: list) -> str:
       "Create a UUID5 from a list of strings and return the uuid as a string."
       name = "/".join([str(i) for i in identifier])
       namespace = uuid.NAMESPACE_DNS
       uuid_obj = uuid.uuid5(namespace=namespace, name=name)
       return str(uuid_obj)


   def create_embeddings(text: str, model: str) -> list:
       """Create embeddings for a text with the OpenAI API."""
       client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
       response = client.embeddings.create(input=text, model=model)
       embeddings = response.data[0].embedding

       return embeddings


   @dag(
       start_date=datetime(2023, 10, 18),
       schedule=None,
       catchup=False,
       tags=["Pinecone"],
       params={"series_mood": Param("A series about astronauts.", type="string")},
   )
   def query_series_vectors():
       @task
       def import_data_func(text_file_path: str) -> list:
           "Import data from a text file and return it as a list of dicts."
           with open(text_file_path, "r") as f:
               lines = f.readlines()
               num_skipped_lines = 0
               descriptions = []
               data = []
               for line in lines:
                   parts = line.split(":::")
                   title_year = parts[1].strip()
                   match = re.match(r"(.+) \((\d{4})\)", title_year)
                   try:
                       title, year = match.groups()
                       year = int(year)
                   except:
                       num_skipped_lines += 1
                       continue

                   genre = parts[2].strip()
                   description = parts[3].strip()
                   descriptions.append(description)
                   data.append(
                       {
                           "id": generate_uuid5(
                               identifier=[title, year, genre, description]
                           ),  # an `id` property is required for Pinecone
                           "metadata": {
                               "title": title,
                               "year": year,
                               "genre": genre,
                               "description": description,  # this is the text we'll embed
                           },
                       }
                   )

           return data

       series_data = import_data_func(text_file_path=DATA_FILE_PATH)

       @task
       def vectorize_series_data(series_data: dict, model: str) -> dict:
           "Create embeddings for the series descriptions."
           response = create_embeddings(
               text=series_data["metadata"]["description"], model=model
           )

           series_data["values"] = response

           return series_data

       vectorized_data = vectorize_series_data.partial(model=EMBEDDING_MODEL).expand(
           series_data=series_data
       )

       @task
       def vectorize_user_mood(model: str, **context) -> list:
           "Create embeddings for the user mood."
           user_mood = context["params"]["series_mood"]
           response = create_embeddings(text=user_mood, model=model)

           return response

       @task
       def create_index_if_not_exists(
           index_name: str, vector_size: int, pinecone_conn_id: str
       ) -> None:
           "Create a Pinecone index of the provided name if it doesn't already exist."
           hook = PineconeHook(conn_id=pinecone_conn_id)
           existing_indexes = hook.list_indexes()
           if index_name not in existing_indexes:
               newindex = hook.create_index(index_name=index_name, dimension=vector_size)
               return newindex
           else:
               print(f"Index {index_name} already exists")

       create_index_if_not_exists_obj = create_index_if_not_exists(
           vector_size=EMBEDDING_MODEL_DIMENSIONS,
           index_name=PINECONE_INDEX_NAME,
           pinecone_conn_id=PINECONE_CONN_ID,
       )

       pinecone_vector_ingest = PineconeIngestOperator(
           task_id="pinecone_vector_ingest",
           conn_id=PINECONE_CONN_ID,
           index_name=PINECONE_INDEX_NAME,
           input_vectors=vectorized_data,
       )

       @task
       def query_pinecone(
           index_name: str,
           pinecone_conn_id: str,
           vectorized_user_mood: list,
       ) -> None:
           "Query the Pinecone index with the user mood and print the top result."
           hook = PineconeHook(conn_id=pinecone_conn_id)

           query_response = hook.query_vector(
               index_name=index_name,
               top_k=1,
               include_values=True,
               include_metadata=True,
               vector=vectorized_user_mood,
           )

           print("You should watch: " + query_response["matches"][0]["metadata"]["title"])
           print("Description: " + query_response["matches"][0]["metadata"]["description"])

       query_pinecone_obj = query_pinecone(
           index_name=PINECONE_INDEX_NAME,
           pinecone_conn_id=PINECONE_CONN_ID,
           vectorized_user_mood=vectorize_user_mood(model=EMBEDDING_MODEL),
       )

       chain(
           create_index_if_not_exists_obj,
           pinecone_vector_ingest,
           query_pinecone_obj,
       )


   query_series_vectors()
   ```

   This DAG consists of six tasks to make a simple ML orchestration pipeline.

   * The `import_data_func` task defined with the [`@task` decorator](/docs/learn/airflow-decorators) reads the data from the `series_data.txt` file and returns a list of dictionaries containing the series title, year, genre, and description. Note that the task will create a UUID for each series using the `create_uuid` function and add it to the `id` key. Having a unique ID for each series is required for the Pinecone ingestion task.
   * The `vectorize_series_data` task is a [dynamic task](/docs/learn/dynamic-tasks) that creates one mapped task instance for each series in the list returned by the `import_data_func` task. The task uses the `create_embeddings` function to generate vector embeddings for each series' description. Note that if you want to use a different vectorizer than OpenAI's `text-embedding-ada-002` you can adjust this function to return your preferred vectors and set the `EMBEDDING_MODEL_DIMENSIONS` parameter in the DAG to the vector size of your model.
   * The `vectorize_user_mood` task calls the `create_embeddings` function to generate vector embeddings for the mood the user can provide as an [Airflow param](/docs/learn/airflow-params).
   * The `create_index_if_not_exists` task uses the [`PineconeHook`](https://airflow.apache.org/registry/providers/pinecone#pinecone-pinecone-PineconeHook) to connect to your Pinecone instance and retrieve the current list of indexes in your Pinecone environment. If no index of the name `PINECONE_INDEX_NAME` exists yet, the task will create it. Note that with a free tier Pinecone account you can only have one index.
   * The `pinecone_vector_ingest` task uses the [`PineconeIngestOperator`](https://airflow.apache.org/registry/providers/pinecone#pinecone-pinecone-PineconeIngestOperator) to ingest the vectorized series data into the index created by the `create_index_if_not_exists` task.
   * The `query_pinecone` task performs a vector search in Pinecone to get the series most closely matching the user-provided mood and prints the result to the task logs.

   <Frame>
     <img alt="A screenshot from the Airflow UI's Grid view with the Graph tab selected showing a successful run of the query_series_vectors DAG." />
   </Frame>

## Step 4: Run your DAG

1. Open your Astro project, then run `astro dev start` to run Airflow locally.

2. Open the Airflow UI at `localhost:8080`, then run the `query_series_vectors` DAG by clicking the **Play** button. Provide your input to the [Airflow param](/docs/learn/airflow-params) for `series_mood`.

   <Frame>
     <img alt="A screenshot of the Trigger DAG view in the Airflow UI showing the mood A series about Astronauts being provided to the series_mood param." />
   </Frame>

3. View your series suggestion in the task logs of the `query_pinecone` task:

   ```text wrap theme={null}
   [2023-11-20, 14:03:48 UTC] {logging_mixin.py:154} INFO - You should watch: For All Mankind
   [2023-11-20, 14:03:48 UTC] {logging_mixin.py:154} INFO - Description: The series dramatizes an alternate history depicting "what would have happened if the global space race had never ended" after the Soviet Union succeeds in the first crewed Moon landing ahead of the United States.
   ```

<Tip>
  When watching `For All Mankind`, make sure to have a tab with [Wikipedia](https://en.wikipedia.org) open to compare the alternate timeline with ours and remember, flying spacecraft isn't like driving a car. It doesn't just go where you point it.
</Tip>

## Conclusion

Congrats! You've successfully integrated Airflow and Pinecone! You can now use this tutorial as a starting point to build your own AI applications with Airflow and Pinecone.


# Orchestrate semantic querying in Qdrant with Airflow
Source: https://astronomer.io/docs/learn/airflow-qdrant

Learn how to integrate Qdrant and Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Qdrant](https://qdrant.tech/) is an open-source vector database and similarity search engine designed for AI applications.

In this tutorial, you'll use the [Qdrant Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-qdrant/stable/index.html) to write a DAG that generates embeddings in parallel and performs semantic retrieval based on user input.

Airflow provides useful operational and orchestration when running operations in Qdrant based on data events or building parallel tasks for generating vector embeddings. By using Airflow, you can set up monitoring and alerts for your pipelines for full observability.

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Prerequisites

* A running Qdrant instance. A [free instance](https://cloud.qdrant.io) is available.
* The Astro CLI. See [Install the Astro CLI](/docs/cli/v1.43/install-cli).
* A [HuggingFace token](https://huggingface.co/docs/hub/en/security-tokens) to generate embeddings.

## Step 1: Set up the project

1. Create a new Astro project:

   ```bash wrap theme={null}
   mkdir qdrant-airflow-tutorial && cd qdrant-airflow-tutorial
   astro dev init
   ```

2. To use Qdrant in Airflow, install the Qdrant Airflow provider by adding the following to your `requirements.txt` file:

   ```text wrap theme={null}
   apache-airflow-providers-qdrant==1.1.0
   ```

## Step 2: Configure credentials

Add the following code to your `.env` file to create Airflow connections between Airflow and HuggingFace and Qdrant. Make sure to update the sample code with your HuggingFace access token and Qdrant instance details.

```text wrap theme={null}
HUGGINGFACE_TOKEN="<YOUR_HUGGINGFACE_ACCESS_TOKEN>"
AIRFLOW_CONN_QDRANT_DEFAULT='{
    "conn_type": "qdrant",
    "host": "xyz-example.eu-central.aws.cloud.qdrant.io:6333",
    "password": "<YOUR_QDRANT_API_KEY>"
}'
```

## Step 3: Add your data

Paste the following sample data into a file called `books.txt` in your `include` directory.

```text wrap theme={null}
1 | To Kill a Mockingbird (1960) | fiction | Harper Lee's Pulitzer Prize-winning novel explores racial injustice and moral growth through the eyes of young Scout Finch in the Deep South.
2 | Harry Potter and the Sorcerer's Stone (1997) | fantasy | J.K. Rowling's magical tale follows Harry Potter as he discovers his wizarding heritage and attends Hogwarts School of Witchcraft and Wizardry.
3 | The Great Gatsby (1925) | fiction | F. Scott Fitzgerald's classic novel delves into the glitz, glamour, and moral decay of the Jazz Age through the eyes of narrator Nick Carraway and his enigmatic neighbour, Jay Gatsby.
4 | 1984 (1949) | dystopian | George Orwell's dystopian masterpiece paints a chilling picture of a totalitarian society where individuality is suppressed and the truth is manipulated by a powerful regime.
5 | The Catcher in the Rye (1951) | fiction | J.D. Salinger's iconic novel follows disillusioned teenager Holden Caulfield as he navigates the complexities of adulthood and society's expectations in post-World War II America.
6 | Pride and Prejudice (1813) | romance | Jane Austen's beloved novel revolves around the lively and independent Elizabeth Bennet as she navigates love, class, and societal expectations in Regency-era England.
7 | The Hobbit (1937) | fantasy | J.R.R. Tolkien's adventure follows Bilbo Baggins, a hobbit who embarks on a quest with a group of dwarves to reclaim their homeland from the dragon Smaug.
8 | The Lord of the Rings (1954-1955) | fantasy | J.R.R. Tolkien's epic fantasy trilogy follows the journey of Frodo Baggins to destroy the One Ring and defeat the Dark Lord Sauron in the land of Middle-earth.
9 | The Alchemist (1988) | fiction | Paulo Coelho's philosophical novel follows Santiago, an Andalusian shepherd boy, on a journey of self-discovery and spiritual awakening as he searches for a hidden treasure.
10 | The Da Vinci Code (2003) | mystery/thriller | Dan Brown's gripping thriller follows symbologist Robert Langdon as he unravels clues hidden in art and history while trying to solve a murder mystery with far-reaching implications.
```

## Step 4: Create your DAG

1. In your `dags` folder, create a file called `books_recommend.py`.

2. Copy the code from the [`recommend_books_dag.py`](https://github.com/astronomer/docs/blob/main/code-samples/dags/airflow-qdrant/recommend_books_dag.py) file into your `books_recommend.py` file.

This Qdrant demo DAG consists of six tasks that generate embeddings in parallel for the data corpus and perform semantic retrieval based on user input.

* `import_books`: This task reads a text file containing information about the books (such as title, genre, and description) and then returns the data as a list of dictionaries.

* `init_collection`: This task initializes a collection in the Qdrant database, where you store the vector representations of the book descriptions. The `recreate_collection()` function deletes a collection first if it already exists. Trying to create a collection that already exists throws an error.

* `embed_description`: This is a dynamic task that creates one mapped task instance for each book in the list. The task uses the `embed` function to generate vector embeddings for each description. To use a different embedding model, you can adjust the `EMBEDDING_MODEL_ID` and `EMBEDDING_DIMENSION` values.

* `embed_user_preference`: This task takes a user’s input and converts it into a vector using the same pre-trained model used for the book descriptions.

* `qdrant_vector_ingest`: This task ingests the book data into the Qdrant collection using the `QdrantIngestOperator`, associating each book description with its corresponding vector embeddings.

* `search_qdrant`: Finally, this task performs a search in the Qdrant database using the vectorized user preference. It finds the most relevant book in the collection based on vector similarity.

  <Frame>
    <img alt="Screenshot of the graph view for the Qdrant demo DAG, with the each of the six steps shown." />
  </Frame>

## Step 5: Run your DAG

1. Run `astro dev start` in your Astro project to start Airflow and open the Airflow UI at `localhost:8080`.

2. In the Airflow UI, run the `books_recommend` DAG by clicking the play button. You'll be asked for input about your book preference.

   <Frame>
     <img alt="Qdrant reference input shows an example of the UI where Airflow prompts you to enter your book preference." />
   </Frame>

3. View the output of your search in the logs of the `search_qdrant` task.

   <Frame>
     <img alt="Screenshot of the Qdrant output example, that shows the title and description of the book recommendation." />
   </Frame>


# Orchestrate Ray jobs with Apache Airflow®
Source: https://astronomer.io/docs/learn/airflow-ray

Learn how to use the Ray provider package to orchestrate Ray jobs with Apache Airflow®.

[Ray](https://www.ray.io/) is an open-source framework for scaling Python applications, particularly for machine learning and AI workloads where it provides the layer for parallel processing and distributed computing. Many large language models (LLMs) are trained using Ray, including [OpenAI's GPT](https://platform.openai.com/docs/models) models.

The [Ray provider package](https://github.com/astronomer/astro-provider-ray) for [Apache Airflow®](https://airflow.apache.org/) allows you to interact with Ray from your Airflow Dags. This tutorial demonstrates how to use the Ray provider package to orchestrate a simple Ray job with Airflow in an existing Ray cluster. For more in-depth information, see the [Ray provider documentation](https://astronomer.github.io/astro-provider-ray/index.html).

For instructions on how to run Ray jobs on the [Anyscale](https://www.anyscale.com/) platform with Airflow, see the [Orchestrate Ray jobs on Anyscale with Apache Airflow®](/docs/learn/airflow-anyscale) tutorial.

<Tip>
  This tutorial shows a simple implementation of the Ray provider package. For a more complex example, see the [Processing User Feedback: an LLM-fine-tuning reference architecture with Ray on Anyscale](/docs/learn/reference-architecture-fine-tuning-anyscale) reference architecture.
</Tip>

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* Ray basics. See the [Getting Started section of the Ray documentation](https://docs.ray.io/en/latest/ray-overview/getting-started.html).
* Airflow decorators. See [Airflow decorators](/docs/learn/airflow-decorators).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).
* Optional: A pre-existing Ray cluster. This tutorial shows how to spin up a local Ray cluster using Docker. To connect to your existing Ray cluster, modify the connection defined in [Step 2](#step-2-configure-a-ray-connection).

<Tip>
  The Ray provider package can also create a Ray cluster for you in an existing Kubernetes cluster. For more information, see the [Ray provider package documentation](https://astronomer.github.io/astro-provider-ray/getting_started/setup.html). Note that you need a Kubernetes cluster with a pre-configured LoadBalancer service to use the Ray provider package.
</Tip>

## Step 1: Configure your Astro project

Use the Astro CLI to create and run an Airflow project on your local machine.

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-ray-tutorial && cd astro-ray-tutorial
   $ astro dev init
   ```

2. In the `requirements.txt` file, add the [Ray provider](https://github.com/astronomer/astro-provider-ray).

   ```text wrap theme={null}
   astro-provider-ray==0.3.1
   ```

3. (Optional). If you don't have a pre-existing Ray cluster, you can spin up a local Ray cluster alongside your local Astro project by using a `docker-compose.override.yml` file. Create a new file in your project's root directory called `docker-compose.override.yml` and add the following:

   ```yaml expandable wrap theme={null}
   services:

     ray-head:
       image: rayproject/ray:latest
       container_name: ray-head
       command: >
         ray start
         --head
         --dashboard-host=0.0.0.0
         --dashboard-port=8265
         --ray-client-server-port=10001
         --port=6379
         --num-cpus=4
        --block
       ports:
         - "8265:8265"  # Ray dashboard
         - "10001:10001"  # Ray client server
         - "6379:6379"  # Ray Redis
       networks:
         - airflow
       environment:
         - RAY_GRAFANA_HOST=http://grafana:3000
         - RAY_PROMETHEUS_HOST=http://prometheus:9090
       healthcheck:
         test: ["CMD", "ray", "status"]
         interval: 30s
         timeout: 10s
         retries: 5
         start_period: 30s
       restart: unless-stopped

   networks:
     airflow:
   ```

4. In your `.env` file, specify your Ray cluster address. Modify this address if you are using a pre-existing Ray cluster.

   ```text wrap theme={null}
   RAY_ADDRESS=http://ray-head:8265
   ```

5. Run the following command to start your Astro project:

   ```sh wrap theme={null}
   astro dev start
   ```

## Step 2: Configure a Ray connection

<Info>
  For Astro customers, Astronomer recommends using the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager) to store connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See [Manage Astro connections in branch-based deploy workflows](/docs/astro/best-practices/connections-branch-deploys).
</Info>

1. In the Airflow UI, go to **Admin** -> **Connections** and click **+**.

2. Create a new connection and choose the `Ray` connection type. If you used the `docker-compose.override.yml` file to spin up a local Ray cluster, use the information below. If you are connecting to your existing Ray cluster, you need to modify your values accordingly.

   * Connection ID: `ray_conn`
   * Host: `ray-head`
   * Port: `8265`
   * Extra Fields:
     * `ray_dashboard_url`: `"http://ray-head:8265"`
     * `disable_job_log_to_stdout`: `false`

3. Click **Save**.

<Info>
  If you are connecting to a Ray cluster running on a cloud provider, you need to provide the `.kubeconfig` file of the Kubernetes cluster where the Ray cluster is running as `Kube config (JSON format)`, as well as valid Cloud credentials as environment variables.
</Info>

## Step 3: Write a Dag to orchestrate Ray jobs

1. Create a new file in your `dags` directory called `ray_tutorial.py`.

2. Copy and paste the code below into the file:

<details>
  <summary>TaskFlow</summary>

  ```python expandable wrap theme={null}
  """
  ## Ray Tutorial

  This tutorial demonstrates how to use the Ray provider in Airflow to parallelize
  a task using Ray.
  """

  from airflow.sdk import dag, task
  from ray_provider.decorators import ray

  CONN_ID = "ray_conn"
  RAY_TASK_CONFIG = {
      "conn_id": CONN_ID,
      "num_cpus": 1,
      "num_gpus": 0,
      "memory": 0,
      "poll_interval": 5,
  }


  @dag(doc_md=__doc__)
  def ray_example_dag():

      @task
      def generate_data() -> list:
          """
          Generate sample data
          Returns:
              list: List of integers
          """
          import random

          return [random.randint(1, 100) for _ in range(10)]

      # use the @ray.task decorator to parallelize the task
      @ray.task(config=RAY_TASK_CONFIG)
      def get_mean_squared_value(data: list) -> float:
          """
          Get the mean squared value from a list of integers
          Args:
              data (list): List of integers
          Returns:
              float: Mean value of the list
          """
          import numpy as np
          import ray

          @ray.remote
          def square(x: int) -> int:
              """
              Square a number
              Args:
                  x (int): Number to square
              Returns:
                  int: Squared number
              """
              return x**2

          ray.init()
          data = np.array(data)
          futures = [square.remote(x) for x in data]
          results = ray.get(futures)
          mean = np.mean(results)
          print(f"Mean squared value: {mean}")

      data = generate_data()
      get_mean_squared_value(data)


  ray_example_dag()
  ```
</details>

<details>
  <summary>Traditional</summary>

  ```python expandable wrap theme={null}
  """
  ## Ray Tutorial

  This tutorial demonstrates how to use the Ray provider in Airflow to
  parallelize a task using Ray.
  """

  from airflow.sdk import dag, chain
  from airflow.providers.standard.operators.python import PythonOperator
  from ray_provider.operators import SubmitRayJob
  from pathlib import Path

  CONN_ID = "ray_conn"
  FOLDER_PATH = Path(__file__).parent
  RAY_RUNTIME_ENV = {"working_dir": str(FOLDER_PATH)}


  def _generate_data() -> list:
      """
      Generate sample data
      Returns:
          list: List of integers
      """
      import random

      return [random.randint(1, 100) for _ in range(10)]


  @dag(doc_md=__doc__)
  def ray_tutorial():

      data = PythonOperator(
          task_id="generate_data",
          python_callable=_generate_data,
      )

      get_mean_squared_value = SubmitRayJob(
          task_id="SubmitRayJob",
          conn_id=CONN_ID,
          entrypoint="python ray_script.py {{ ti.xcom_pull(task_ids='generate_data') | join(' ') }}",
          runtime_env=RAY_RUNTIME_ENV,
          num_cpus=1,
          num_gpus=0,
          memory=0,
          resources={},
          xcom_task_key="SubmitRayJob.dashboard",
          fetch_logs=True,
          wait_for_completion=True,
          job_timeout_seconds=600,
          poll_interval=5,
      )

      chain(data, get_mean_squared_value)


  ray_tutorial()
  ```
</details>

This is a simple Dag comprised of two tasks:

* The `generate_data` task randomly generates a list of 10 integers.
* The `get_mean_squared_value` task submits a Ray job on Anyscale to calculate the mean squared value of the list of integers.

3. (Optional). If you are using the traditional syntax with the SubmitRayJob operator, you need to provide the Python code to run in the Ray job as a script. Create a new file in your `dags` directory called `ray_script.py` and add the following code:

   ```python wrap theme={null}
   # ray_script.py
   import numpy as np
   import ray
   import argparse

   @ray.remote
   def square(x):
       return x**2

   def main(data):
       ray.init()
       data = np.array(data)
       futures = [square.remote(x) for x in data]
       results = ray.get(futures)
       mean = np.mean(results)
       print(f"Mean of this population is {mean}")
       return mean

   if __name__ == "__main__":
       parser = argparse.ArgumentParser(description="Process some integers.")
       parser.add_argument('data', nargs='+', type=float, help='List of numbers to process')
       args = parser.parse_args()

       data = args.data
       main(data)

   ```

## Step 4: Run the Dag

1. In the Airflow UI, click the play button to manually run your Dag.

2. After the Dag runs successfully, check go to your Ray dashboard to see the job submitted by Airflow.

   <Frame>
     <img alt="Ray dashboard showing a Job completed successfully." />
   </Frame>

## Conclusion

Congratulations! You've run a Ray job using Apache Airflow. You can now use the Ray provider package to orchestrate more complex Ray jobs, see [Processing User Feedback: an LLM-fine-tuning reference architecture with Ray on Anyscale](/docs/learn/reference-architecture-fine-tuning-anyscale) for an example.


# Orchestrate Redshift operations with Airflow
Source: https://astronomer.io/docs/learn/airflow-redshift

Orchestrate Redshift queries from your Airflow DAGs.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

Amazon Redshift is a fully managed cloud data warehouse. It has become the most popular cloud data warehouse in part because of its ability to analyze exabytes of data and run complex analytical queries.

Developing a dimensional data mart in Redshift requires automation and orchestration for repeated queries, data quality checks, and overall cluster operations. This makes Airflow the perfect orchestrator to pair with Redshift. With Airflow, you can orchestrate each step of your Redshift pipeline, integrate with services that clean your data, and store and publish your results using SQL and Python code.

In this tutorial, you'll learn about the Redshift modules that are available in the [AWS Airflow provider package](https://airflow.apache.org/registry/providers/amazon). You'll also complete sample implementations that execute SQL in a Redshift cluster, pause and resume a Redshift cluster, and transfer data between Amazon S3 and a Redshift cluster.

All code in this tutorial is located in the [GitHub repository](https://github.com/astronomer/cs-tutorial-redshift).

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of [Amazon Redshift](https://aws.amazon.com/redshift/getting-started/?nc=sn\&loc=4\&dn=1).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow connections. See [Managing your Connections in Apache Airflow](/docs/learn/connections).

## Setup

To use Redshift operators in Airflow, you first need to install the Redshift provider package and create a connection to your Redshift cluster.

1. If you are working with the [Astro CLI](/docs/cli/v1.43/install-cli), add `apache-airflow-providers-amazon` to the `requirements.txt` file of your Astro project. Otherwise, run `pip install apache-airflow-providers-amazon`.

2. Connect your Airflow instance to Redshift. The most common way of doing this is by configuring an Airflow connection. In the Airflow UI, go to **Admin** > **Connections** and add the following connections:

   * `redshift_default`: The default connection that Airflow Redshift modules use. If you use a name other than `redshift_default` for this connection, you'll need to specify it in the modules that require a Redshift connection. Use the following parameters for your new connection (all other fields can be left blank):

   ```yaml wrap theme={null}
   Connection ID: redshift_default
   Connection Type: Amazon Redshift
   Host: <your-redshift-endpoint> (for example, redshift-cluster-1.123456789.us-west-1.redshift.amazonaws.com)
   Schema: <your-redshift-database> (for example, dev, test, prod, etc.)
   Login: <your-redshift-username> (for example, awsuser)
   Password: <your-redshift-password>
   Port: <your-redshift-port> (for example, 5439)
   ```

   * `aws_default`: The default connection that other Airflow AWS modules use. For the examples in this guide, you will need this connection for Airflow to communicate with Amazon S3. If you use a name other than `aws_default` for this connection, you'll need to specify it in the modules that require an AWS connection. Use the following parameters for your new connection (all other fields can be left blank):

   ```yaml wrap theme={null}
   Connection ID: aws_default
   Connection Type: Amazon Web Services
   Extra: {
     "aws_access_key_id": "<your-access-key-id>", 
     "aws_secret_access_key": "<your-secret-access-key>", 
     "region_name": "<your-region-name>"
   }
   ```

3. Configure the following in your Redshift cluster:

   * [Allow inbound traffic](https://docs.aws.amazon.com/redshift/latest/gsg/rs-gsg-authorize-cluster-access.html) from the IP Address where Airflow is running
   * Give `aws_default` the following permissions on AWS:
     * Read/write permissions for a pre-configured S3 bucket
     * Permission to interact with the Redshift cluster, specifically:
       * `redshift:DescribeClusters`
       * `redshift:PauseCluster`
       * `redshift:ResumeCluster`

   If your Airflow instance is running in the same AWS VPC as your Redshift cluster, you may have other authentication options available. To authenticate to Redshift using IAM Authentication or Okta, see the [Apache Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/redshift.html).

### Sample data

The following examples use the sample database (TICKIT) provided by AWS. For more details on the underlying data, see [Sample database](https://docs.aws.amazon.com/redshift/latest/dg/c_sampledb.html).

## Use the `RedshiftSQLOperator`

After you've implemented a connection to Redshift, you can start using the [`RedshiftSQLOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLExecuteQueryOperator). The `RedshiftSQLOperator` is used to run one or multiple SQL statements against a Redshift cluster. Use cases for this operator include:

* Creating data schema models in your data warehouse.
* Creating fact or dimension tables for various data models.
* Performing data transformations or data cleaning.

The following DAG shows how you can use the `RedshiftSQLOperator` to run a `.sql` query against a Redshift schema:

```python wrap theme={null}
from datetime import datetime
from airflow.models import DAG
from airflow.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperator

with DAG(
    dag_id=f"example_dag_redshift",
    schedule="@daily",
    start_date=datetime(2023, 1, 1),
    max_active_runs=1,
    template_searchpath='/usr/local/airflow/include/example_dag_redshift',
    catchup=False
) as dag:

    t = RedshiftSQLOperator(
        task_id='fct_listing',
        sql='/sql/fct_listing.sql',
        params={
            "schema": "fct",
            "table": "listing"
        }
    )
```

Notice the value for `template_searchpath` in the `DAG()` configuration. This value indicates that the DAG looks for the `.sql` file at `/usr/local/airflow/include/example_dag_redshift/sql/fct_listing.sql`. For this example, you'll use the following SQL file:

```sql wrap theme={null}
begin;
create table if not exists {{ params.schema }}.{{ params.table }} (
  date_key date,
  total_sellers int,
  total_events int,
  total_tickets int,
  total_revenue double precision
) sortkey(date_key)
;

delete from {{ params.schema }}.{{ params.table }} where date_key = '{{ ds }}';

insert into {{ params.schema }}.{{ params.table }}
  select
    listtime::date as date_key,
    count(distinct sellerid) as total_sellers,
    count(distinct eventid) as total_events,
    sum(numtickets) as total_tickets,
    sum(totalprice) as total_revenue
  from tickit.listing
  where listtime::date = '{{ ds }}'
  group by date_key
;
end;
```

In this SQL query, there are multiple templated parameters: `{{ params.schema }}`, `{{ params.table }}`, and `{{ ds }}`. Based on the task definition, `{{ params.schema }}` is set as `fct` and `{{ params.table }}` is set as `listing`. These values are injected into the SQL query at runtime. The `{{ ds }}` variable is a built-in [Airflow Jinja Template Variable](https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html) that returns the DAG run's logical date in the format YYYY-MM-DD. Using templated variables makes your SQL code reusable and aligned DAG writing best practices (particularly in relation to [idempotency](/docs/learn/dag-best-practices#review-idempotency)).

## Use the `S3ToRedshiftOperator`

The [`S3ToRedshiftOperator`](https://airflow.apache.org/registry/providers/amazon#amazon-s3_to_redshift-S3ToRedshiftOperator) executes a
`COPY` command to load files from S3 to Redshift. The following example DAG demonstrates how to use this operator:

```python wrap theme={null}
from datetime import datetime
from airflow.models import DAG
from airflow.providers.amazon.aws.transfers.s3_to_redshift import S3ToRedshiftOperator

with DAG(
    dag_id=f"example_dag_redshift",
    schedule="@daily",
    start_date=datetime(2023, 1, 1),
    max_active_runs=1,
    template_searchpath='/usr/local/airflow/include/example_dag_redshift',
    catchup=False
) as dag:

    s3_to_redshift = S3ToRedshiftOperator(
        task_id='s3_to_redshift',
        schema='fct',
        table='from_redshift',
        s3_bucket='airflow-redshift-demo',
        s3_key='fct/from_redshift',
        redshift_conn_id='redshift_default',
        aws_conn_id='aws_default',
        copy_options=[
            "DELIMITER AS ','"
        ],
        method='REPLACE'
    )
```

This DAG copies the S3 blob `s3://airflow-redshift-demo/fct/from_redshift` into the table `fct.from_redshift` on the Redshift cluster. With this operator, you can pass all the same copy options that exist in AWS with the `copy_options` parameter. For more information about the `COPY` command, see [Data format parameters](https://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html#r_COPY-syntax-overview-data-format). In this example, a copy option is used to change the delimiter for the blob from a pipe character to a comma.

## Use the `RedshiftToS3Operator`

The [`RedshiftToS3Operator`](https://airflow.apache.org/registry/providers/amazon#amazon-redshift_to_s3-RedshiftToS3Operator) executes an `UNLOAD` command to Amazon S3 as a CSV file with headers. `UNLOAD` automatically creates encrypted files using Amazon S3 server-side encryption (SSE). There are numerous use cases for using the `UNLOAD` command. Some of the more common use cases include:

* Archiving old data that is no longer needed in your Redshift cluster.
* Sharing the results of query data without granting access to Redshift.
* Saving the result of query data into Amazon S3 for analysis with BI tools or use in an ML pipeline.

The following DAG shows an example implementation:

```python wrap theme={null}
from datetime import datetime
from airflow.models import DAG
from airflow.providers.amazon.aws.transfers.redshift_to_s3 import RedshiftToS3Operator

with DAG(
    dag_id=f"example_dag_redshift",
    schedule="@daily",
    start_date=datetime(2023, 1, 1),
    max_active_runs=1,
    template_searchpath='/usr/local/airflow/include/example_dag_redshift',
    catchup=False
) as dag:

    redshift_to_s3 = RedshiftToS3Operator(
        task_id='fct_listing_to_s3',
        s3_bucket='airflow-redshift-demo',
        s3_key='fct/listing/{{ ds }}_',
        schema='fct',
        table='listing',
        redshift_conn_id='redshift_default',
        aws_conn_id='aws_default',
        table_as_file_name=False,
        unload_options=[
            "DELIMITER AS ','",
            "FORMAT AS CSV",
            "ALLOWOVERWRITE",
            "PARALLEL OFF",
            "HEADER"
        ]
    )
```

This DAG copies the `fct.listing` table from a Redshift cluster to the Amazon S3 blob `s3:://airflow-redshift-demo/fct/listing/YYYY-MM-DD_` (where YYYY-MM-DD is the logical date for the DAG run). With this operator, you can pass all the same unload options that exist in AWS. For more information on the `UNLOAD` command, see the AWS [UNLOAD documentation](https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html).

In this example, based on the parameters set in the DAG, the delimiter for the blob has been specified as a comma and the format of the blob has been specified as a CSV. Any existing files are overwritten, data isn't written in parallel across multiple files, and a header line containing column names is included at the top of the file.

## Pause and resume a Redshift cluster from Airflow

Amazon Redshift supports the ability to pause and resume a cluster, allowing customers to suspend on-demand billing when the cluster isn't being used. For more information, see [Amazon Redshift launches pause and resume](https://aws.amazon.com/about-aws/whats-new/2020/03/amazon-redshift-launches-pause-resume/#:~:text=Amazon%20Redshift%20now%20supports%20the,suspended%20when%20not%20in%20use.).

You may want your Airflow DAG to pause and unpause a Redshift cluster at times when it isn't being queried or used. Additionally, you may want to pause your Redshift cluster at the end of your Airflow pipeline, or resume your Redshift cluster at the beginning of your Airflow pipeline. There are currently three Airflow modules available to accomplish this:

* The [`RedshiftPauseClusterOperator`](https://airflow.apache.org/registry/providers/amazon#amazon-redshift_cluster-RedshiftPauseClusterOperator) can be used to pause an AWS Redshift Cluster.
* The [`RedshiftResumeClusterOperator`](https://airflow.apache.org/registry/providers/amazon#amazon-redshift_cluster-RedshiftResumeClusterOperator) can be used to resume a paused AWS Redshift Cluster.
* The [`RedshiftClusterSensor`](https://airflow.apache.org/registry/providers/amazon#amazon-redshift_cluster-RedshiftClusterSensor) can be used to wait for a Redshift cluster to reach a specific status, For example, available after it has been unpaused.

If no operations queried your Redshift cluster after your last ETL job, you can use the `RedshiftPauseClusterOperator` to pause your Redshift cluster, which would lower your AWS bill. On the first ETL job of the day, you could add the `RedshiftResumeClusterOperator` at the beginning of your DAG to send the request to AWS to unpause it. Following that task, you could use a `RedshiftClusterSensor` to ensure the cluster is fully available before running the remainder of your DAG.

The following DAG shows how to pause and unpause a Redshift cluster using the available operators. As the sensor is implemented at the end, the DAG is identified as successful when the cluster state is `Available`.

```python expandable wrap theme={null}
from datetime import datetime
from airflow.models import DAG
from airflow.providers.amazon.aws.operators.redshift_cluster import RedshiftPauseClusterOperator
from airflow.providers.amazon.aws.operators.redshift_cluster import RedshiftResumeClusterOperator
from airflow.providers.amazon.aws.sensors.redshift_cluster import RedshiftClusterSensor

with DAG(
    dag_id=f"example_dag_redshift",
    schedule="@daily",
    start_date=datetime(2023, 1, 1),
    max_active_runs=1,
    template_searchpath='/usr/local/airflow/include/example_dag_redshift',
    catchup=False
) as dag:

    pause_redshift = RedshiftPauseClusterOperator(
        task_id='pause_redshift',
        cluster_identifier='astronomer-success-redshift',
        aws_conn_id='aws_default'
    )

    resume_redshift = RedshiftResumeClusterOperator(
      task_id='resume_redshift',
      cluster_identifier='astronomer-success-redshift',
      aws_conn_id='aws_default'
    )

    cluster_sensor = RedshiftClusterSensor(
        task_id='wait_for_cluster',
        cluster_identifier='astronomer-success-redshift',
        target_status='available',
        aws_conn_id='aws_default'
    )

    pause_redshift >> resume_redshift >> cluster_sensor
```


# Train a machine learning model with SageMaker and Airflow
Source: https://astronomer.io/docs/learn/airflow-sagemaker

Follow a step-by-step tutorial for using Airflow to orchestrate the training and testing of a SageMaker model.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Amazon SageMaker](https://aws.amazon.com/sagemaker/) is a comprehensive AWS machine learning (ML) service that is frequently used by data scientists to develop and deploy ML models at scale. With Airflow, you can orchestrate every step of your SageMaker pipeline, integrate with services that clean your data, and store and publish your results using only Python code.

This tutorial demonstrates how to orchestrate a full ML pipeline including creating, training, and testing a new SageMaker model. This use case is relevant if you want to automate the model training, testing, and deployment components of your ML pipeline.

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Webinar: [Batch Inference with Airflow and SageMaker](https://www.astronomer.io/events/webinars/batch-inference-with-airflow-and-sagemaker/).
</Tip>

## Time to complete

This tutorial takes approximately 60 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of [Amazon S3](https://aws.amazon.com/s3/getting-started/) and [Amazon SageMaker](https://aws.amazon.com/sagemaker/getting-started/).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Airflow connections. See [Managing your Connections in Apache Airflow](/docs/learn/connections).

## Prerequisites

To complete this tutorial, you need:

* An AWS account with:

  * Access to an S3 [storage bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/GetStartedWithS3.html). If you don't already have an account, Amazon offers 5 GB of free storage in S3 for 12 months. This should be more than enough for this tutorial.
  * Access to [AWS SageMaker](https://aws.amazon.com/sagemaker/). If you don't already use SageMaker, Amazon offers a free tier for the first month.

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).

## Step 1: Create a role to access SageMaker

For this tutorial, you will need to access SageMaker from your Airflow environment. There are multiple ways to do this, but for this tutorial you will create an AWS role that can access SageMaker and create temporary credentials for that role.

<Info>
  If you are uncertain which method you use to connect to AWS, contact your AWS Administrator. These steps may not fit your desired authentication mechanism.
</Info>

1. From the AWS web console, go to **IAM** service page and create a new execution role for SageMaker. See [Create execution roles](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html#sagemaker-roles-create-execution-role) in the AWS documentation.

2. Add your AWS user and `sagemaker.amazonaws.com` as a trusted entity to the role you just created. See [Editing the trust relationship for an existing role](https://docs.aws.amazon.com/directoryservice/latest/admin-guide/edit_trust.html). Note that this step might not be necessary depending on how IAM roles are managed for your organization's AWS account.

3. Using the ARN of the role you just created, generate temporary security credentials for your role. There are multiple methods for generating the credentials. For detailed instructions, see [Using temporary credentials with AWS resources](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html). You will need the access key ID, secret access key, and session token in Step 5.

## Step 2: Create an S3 bucket

Create an S3 bucket in the `us-east-2` region that will be used for storing data and model training results. Make sure that your bucket is accessible by the role you created in Step 1. See [Creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html).

Note that if you need to create a bucket in a different region than `us-east-2`, you will need to modify the region specified in the environment variables created in Step 3 and in the DAG code in Step 6.

## Step 3: Configure your Astro project

Now that you have your AWS resources configured, you can move on to Airflow setup. Using the Astro CLI:

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-sagemaker-tutorial && cd astro-sagemaker-tutorial
   $ astro dev init
   ```

2. Add the following line to the `requirements.txt` file of your Astro project:

   ```text wrap theme={null}
   apache-airflow-providers-amazon>=5.1.0
   astronomer-providers[amazon]>=1.11.0
   ```

   This installs the AWS provider package which contains relevant S3 and SageMaker modules. It also installs the `astronomer-providers` package, which contains [deferrable](/docs/learn/deferrable-operators) versions of some SageMaker modules that can help you save costs for long-running jobs. If you use an older version of the AWS provider, some of the DAG configuration in later steps may need to be modified. Deferrable SageMaker operators aren't available in older versions of the `astronomer-providers` package.

3. Add the following environment variables to the `.env` file of your project:

   ```text wrap theme={null}
   AIRFLOW__CORE__ENABLE_XCOM_PICKLING=True
   AWS_DEFAULT_REGION=us-east-2
   ```

   These variables ensure that all SageMaker operators will work in your Airflow environment. Some require XCom pickling to be turned on in order to work because they return objects that aren't JSON serializable.

4. Run the following command to start your project in a local environment:

   ```sh wrap theme={null}
   astro dev start
   ```

## Step 4: Add Airflow Variables

Add two Airflow variables that will be used by your DAG. In the Airflow UI, go to **Admin** -> **Variables**.

1. Add a variable with the ARN of the role you created in Step 1.

   * **Key**: `role`
   * **Val**: `<your-role-arn>`

2. Add a variable with the name of the S3 bucket you created in Step 2.

   * **Key**: `s3_bucket`
   * **Val**: `<your-s3-bucket-name>`

## Step 5: Add an Airflow connection to SageMaker

Add a connection that Airflow will use to connect to SageMaker and S3. In the Airflow UI, go to **Admin** -> **Connections**.

Create a new connection named `aws-sagemaker` and choose the `Amazon Web Services` connection type. Fill in the **AWS Access Key ID** and **AWS Secret Access Key** with the access key ID and secret access key you generated in Step 1.

In the **Extra** field, provide your AWS session token generated in Step 1 using the following format:

```text wrap theme={null}
{
    "aws_session_token": "<your-session-token>"
}
```

<Danger>
  Your AWS credentials will only last one day using this authentication method. If you return to this step after your credentials have expired, you will need to edit the details in the connection with updated credentials.
</Danger>

Your connection should look like this:

<Frame>
  <img alt="SageMaker Connection" />
</Frame>

<Info>
  As mentioned in Step 1, there are multiple ways of connecting Airflow to AWS resources. If you are using a method other than the one described in this tutorial, such as having your Airflow environment assume an IAM role, you may need to update your connection accordingly.
</Info>

## Step 6: Create your DAG

In your Astro project `dags/` folder, create a new file called `sagemaker_pipeline.py`. Paste the following code into the file:

```python expandable wrap theme={null}
"""
This DAG shows an example implementation of machine learning model orchestration using Airflow
and AWS SageMaker. Using the AWS provider's SageMaker operators, Airflow orchestrates getting data
from an API endpoint and pre-processing it (task-decorated function), training the model (SageMakerTrainingOperatorAsync),
creating the model with the training results (SageMakerModelOperator), and testing the model using
a batch transform job (SageMakerTransformOperatorAsync).

The example use case shown here is using a built-in SageMaker K-nearest neighbors algorithm to make
predictions on the Iris dataset. To use the DAG, add Airflow variables for `role` (Role ARN to execute SageMaker jobs)
then fill in the information directly below with the target AWS S3 locations, and model and training job names.
"""
import textwrap
from airflow.models.dag import DAG
from airflow.decorators import task
from airflow.providers.amazon.aws.operators.sagemaker import SageMakerModelOperator
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
from astronomer.providers.amazon.aws.operators.sagemaker import (
    SageMakerTrainingOperatorAsync,
    SageMakerTransformOperatorAsync
)
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
from sagemaker import image_uris

from datetime import datetime
import requests
import io
import pandas as pd
import numpy as np
import os

# Define variables used in configs
data_url = "https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data"  # URL for Iris data API
date = "{{ ts_nodash }}"  # Date for transform job name

input_s3_key = 'iris/processed-input-data'  # Train and test data S3 path
output_s3_key = 'iris/results'  # S3 path for output data
model_name = f"Iris-KNN-{date}"  # Name of model to create
training_job_name = f'train-iris-{date}'  # Name of training job
region = "us-east-2"

with DAG('sagemaker_pipeline',
         start_date=datetime(2022, 1, 1),
         max_active_runs=1,
         schedule=None,
         default_args={'retries': 0, },
         catchup=False,
         doc_md=__doc__
) as dag:
    @task
    def data_prep(data_url, s3_bucket, input_s3_key, aws_conn_id):
        """
        Grabs the Iris dataset from API, splits into train/test splits, and saves CSV's to S3 using S3 Hook
        """
        # Get data from API
        iris_response = requests.get(data_url).content
        columns = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species']
        iris = pd.read_csv(io.StringIO(iris_response.decode('utf-8')), names=columns)

        # Process data
        iris['species'] = iris['species'].replace({'Iris-virginica': 0, 'Iris-versicolor': 1, 'Iris-setosa': 2})
        iris = iris[['species', 'sepal_length', 'sepal_width', 'petal_length', 'petal_width']]

        # Split into test and train data
        iris_train, iris_test = np.split(iris.sample(frac=1, random_state=np.random.RandomState()),
                                         [int(0.7 * len(iris))])
        iris_test.drop(['species'], axis=1, inplace=True)

        # Save files to S3
        iris_train.to_csv('iris_train.csv', index=False, header=False)
        iris_test.to_csv('iris_test.csv', index=False, header=False)
        s3_hook = S3Hook(aws_conn_id=aws_conn_id)
        s3_hook.load_file('iris_train.csv', f'{input_s3_key}/train.csv', bucket_name=s3_bucket, replace=True)
        s3_hook.load_file('iris_test.csv', f'{input_s3_key}/test.csv', bucket_name=s3_bucket, replace=True)

        # cleanup
        os.remove('iris_train.csv')
        os.remove('iris_test.csv')

    data_prep = data_prep(data_url, "{{ var.value.get('s3_bucket') }}", input_s3_key)

    train_model = SageMakerTrainingOperatorAsync(
        task_id='train_model',
        doc_md=textwrap.dedent("""
        Train the KNN algorithm on the data using the `SageMakerTrainingOperatorAsync`. We use a deferrable version of this operator to save resources on potentially long-running training jobs. The configuration for this operator requires:
        - Information about the algorithm being used.
        - Any required hyper parameters.
        - The input data configuration.
        - The output data configuration.
        - Resource specifications for the machine running the training job.
        - The Role ARN for execution.
        For more information about submitting a training job, check out the [API documentation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html).
        """),
        aws_conn_id='aws-sagemaker',
        config={
            "AlgorithmSpecification": {
                "TrainingImage": image_uris.retrieve(framework='knn',region=region),
                "TrainingInputMode": "File"
            },
            "HyperParameters": {
                "predictor_type": "classifier",
                "feature_dim": "4",
                "k": "3",
                "sample_size": "150"
            },
            "InputDataConfig": [
                {"ChannelName": "train",
                 "DataSource": {
                     "S3DataSource": {
                         "S3DataType": "S3Prefix",
                         "S3Uri": f"s3://{bucket}/{input_s3_key}/train.csv"
                     }
                 },
                 "ContentType": "text/csv",
                 "InputMode": "File"
                 }
            ],
            "OutputDataConfig": {
                "S3OutputPath": f"s3://{bucket}/{output_s3_key}"
            },
            "ResourceConfig": {
                "InstanceCount": 1,
                "InstanceType": "ml.m5.large",
                "VolumeSizeInGB": 1
            },
            # We are using a Jinja Template to fetch the Role name dynamically at runtime via looking up the Airflow Variable
            "RoleArn": "{{ var.value.get('role') }}",
            "StoppingCondition": {
                "MaxRuntimeInSeconds": 6000
            },
            "TrainingJobName": training_job_name
        },
        wait_for_completion=True
    )

    create_model = SageMakerModelOperator(
        task_id='create_model',
        aws_conn_id='aws-sagemaker',
        doc_md=textwrap.dedent("""
        Create a SageMaker model based on the training results using the `SageMakerModelOperator`. This step creates a model artifact in SageMaker that can be called on demand to provide inferences. The configuration for this operator requires:
        - A name for the model.
        - The Role ARN for execution.
        - The image containing the algorithm (in this case the pre-built SageMaker image for KNN).
        - The S3 path to the model training artifact.
        For more information on creating a model, check out the API documentation [here](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html).
        """),
        config={
            # We are using a Jinja Template to fetch the Role name dynamically at runtime via looking up the Airflow Variable
            "ExecutionRoleArn": "{{ var.value.get('role') }}",
            "ModelName": model_name,
            "PrimaryContainer": {
                "Mode": "SingleModel",
                "Image": image_uris.retrieve(framework='knn',region=region),
                # We are using a Jinja Template to pull the XCom output of the 'train_model' task to use as input for this task
                "ModelDataUrl": "{{ ti.xcom_pull(task_ids='train_model')['Training']['ModelArtifacts']['S3ModelArtifacts'] }}"
            },
        }
    )

    test_model = SageMakerTransformOperatorAsync(
        task_id='test_model',
        doc_md=textwrap.dedent("""
        Evaluate the model on the test data created in task 1 using the `SageMakerTransformOperatorAsync`. This step runs a batch transform to get inferences on the test data from the model created in task 3. The DAG uses a deferrable version of `SageMakerTransformOperator` to save resources on potentially long-running transform jobs. The configuration for this operator requires:
        - Information about the input data source.
        - The output results path.
        - Resource specifications for the machine running the training job.
        - The name of the model.
        For more information on submitting a batch transform job, check out the [API documentation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTransformJob.html).
        """),
        aws_conn_id='aws-sagemaker',
        config={
            "TransformJobName": f"test-knn-{date}",
            "TransformInput": {
                "DataSource": {
                    "S3DataSource": {
                        "S3DataType": "S3Prefix",
                        "S3Uri": f"s3://{bucket}/{input_s3_key}/test.csv"
                    }
                },
                "SplitType": "Line",
                "ContentType": "text/csv",
            },
            "TransformOutput": {
                "S3OutputPath": f"s3://{bucket}/{output_s3_key}"
            },
            "TransformResources": {
                "InstanceCount": 1,
                "InstanceType": "ml.m5.large"
            },
            "ModelName": model_name
        },
    )

    data_prep >> train_model >> create_model >> test_model
```

The graph view of the DAG should look similar to this:

<Frame>
  <img alt="SageMaker DAG graph" />
</Frame>

This DAG uses the [@task decorator](/docs/learn/airflow-decorators) and several SageMaker operators to get data from an API endpoint, train a model, create a model in SageMaker, and test the model. See [How it works](#how-it-works) for more information about each task in the DAG and how the different SageMaker operators work.

## Step 7: Run your DAG to train the model

Go to the Airflow UI, unpause your `sagemaker_pipeline` DAG, and trigger it to train, create and test the model in SageMaker. Note that the `train_model` and `test_model` tasks may take up to 10 minutes each to complete.

In the SageMaker console, you should see your completed training job.

<Frame>
  <img alt="SageMaker training job" />
</Frame>

In your S3 bucket, you will have a folder called `processed-input-data/` containing the processed data from the `data_prep` task, and a folder called `results/` that contains a `model.tar.gz` file with the trained model and a CSV file with the output of the model testing.

<Frame>
  <img alt="SageMaker results" />
</Frame>

## How it works

This example DAG acquires and pre-processes data, trains a model, creates a model from the training results, and evaluates the model on test data using a batch transform job. This example uses the [Iris dataset](https://archive.ics.uci.edu/ml/datasets/iris), and trains a built-in SageMaker K-Nearest Neighbors (KNN) model. The general steps in the DAG are:

1. Using a `PythonOperator`, grab the data from the API, complete some pre-processing so the data is compliant with KNN requirements, split into train and test sets, and save them to S3 using the `S3Hook`.

2. Train the KNN algorithm on the data using the `SageMakerTrainingOperatorAsync`. We use a deferrable version of this operator to save resources on potentially long-running training jobs. The configuration for this operator requires:

   * Information about the algorithm being used.
   * Any required hyper parameters.
   * The input data configuration.
   * The output data configuration.
   * Resource specifications for the machine running the training job.
   * The Role ARN for execution.

   For more information about submitting a training job, check out the [API documentation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html).

3. Create a SageMaker model based on the training results using the `SageMakerModelOperator`. This step creates a model artifact in SageMaker that can be called on demand to provide inferences. The configuration for this operator requires:

   * A name for the model.
   * The Role ARN for execution.
   * The image containing the algorithm (in this case the pre-built SageMaker image for KNN).
   * The S3 path to the model training artifact.

   For more information on creating a model, check out the API documentation [here](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html).

4. Evaluate the model on the test data created in task 1 using the `SageMakerTransformOperatorAsync`. This step runs a batch transform to get inferences on the test data from the model created in task 3. The DAG uses a deferrable version of `SageMakerTransformOperator` to save resources on potentially long-running transform jobs. The configuration for this operator requires:

   * Information about the input data source.
   * The output results path.
   * Resource specifications for the machine running the training job.
   * The name of the model.

   For more information on submitting a batch transform job, check out the [API documentation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTransformJob.html).

<Info>
  To clone the entire repository used to create this tutorial as well as an additional DAG that performs batch inference using an existing SageMaker model, check out the [Airflow Registry](https://airflow.apache.org/registry/providers/amazon#amazon-sagemaker-SageMakerPipelineTrigger).
</Info>

In addition to the modules shown in this tutorial, the [AWS provider](https://airflow.apache.org/registry/providers/amazon) contains multiple other SageMaker operators and sensors that are built on the [SageMaker API](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_Operations_Amazon_SageMaker_Service.html) and cover a wide range of SageMaker features. In general, documentation on what should be included in the configuration of each operator can be found in the corresponding Actions section of the [API documentation](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_Operations.html).

## Conclusion

In this tutorial, you learned how to use Airflow with SageMaker to automate an end-to-end ML pipeline. A natural next step would be to deploy this model to a SageMaker endpoint using the `SageMakerEndpointConfigOperator` and `SageMakerEndpointOperator`, which provisions resources to host the model. In general, the SageMaker modules in the [AWS provider](https://airflow.apache.org/registry/providers/amazon) allow for many possibilities when using Airflow to orchestrate ML pipelines.


# Orchestrate Snowflake Queries with Airflow
Source: https://astronomer.io/docs/learn/airflow-snowflake

Get enhanced observability and compute savings while orchestrating Snowflake jobs from your Airflow DAGs.

<Tip>
  The key information from this and other Snowflake guides is available as an [Astronomer Cheat Sheet](https://www.astronomer.io/ebooks/airflow-snowflake-cheatsheet/?utm_source=website\&utm_medium=learn-guides\&utm_campaign=snowflake-tutorial).
</Tip>

[Snowflake](https://www.snowflake.com/) is one of the most commonly used data warehouses, and orchestrating Snowflake queries as part of a data pipeline is one of the most common Airflow use cases. Two Airflow provider packages, the [Snowflake Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/index.html) and the [Common SQL provider](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/stable/index.html) contain hooks and operators that make it easy to interact with Snowflake from Airflow.

This tutorial covers an example of executing Snowflake operations with Airflow, including:

* Setting up a connection to Snowflake in Airflow.
* Executing individual SQL statements using the [`SQLExecuteQueryOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLExecuteQueryOperator).
* Executing multiple SQL statements using the [`SnowflakeSqlApiOperator`](https://airflow.apache.org/registry/providers/snowflake#snowflake-snowflake-SnowflakeSqlApiOperator).
* Running data quality checks using the [`SQLColumnCheckOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLColumnCheckOperator).

Additionally, [More on the Airflow Snowflake integration](#more-on-the-airflow-snowflake-integration) offers further information on general best practices and considerations when interacting with Snowflake from Airflow.

## Time to complete

This tutorial takes approximately 20 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* Snowflake basics. See [Introduction to Snowflake](https://docs.snowflake.com/en/user-guide-intro.html).
* Airflow operators. See [Airflow operators](/docs/learn/what-is-an-operator).
* SQL basics. See the [W3 SQL tutorial](https://www.w3schools.com/sql/).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).
* A Snowflake account. A [30-day free trial](https://trial.snowflake.com/?owner=SPN-PID-365384) is available. You need to have at least one database, one schema and one warehouse set up in your Snowflake account as well as a user with the necessary permissions to create tables and run queries in the schema.

## Step 1: Configure your Astro project

Use the Astro CLI to create and run an Airflow project on your local machine.

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-snowflake-tutorial && cd astro-snowflake-tutorial
   $ astro dev init
   ```

2. In the `requirements.txt` file, add the [Snowflake Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/index.html) and the [Common SQL provider](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/stable/index.html).

   ```txt wrap theme={null}
   apache-airflow-providers-snowflake==6.4.0
   apache-airflow-providers-common-sql==1.27.2
   ```

3. Run the following command to start your Airflow project:

   ```sh wrap theme={null}
   astro dev start
   ```

## Step 2: Configure a Snowflake connection

There are different options to [authenticate to Snowflake](/docs/learn/connections/snowflake). The `SnowflakeAPIOperator` used in this tutorial requires you to use key-pair authentication, which is the preferred method. This method requires you to generate a public/private key pair, add the public key to your role in Snowflake, and use the private key in your [Airflow connection](/docs/learn/connections).

<Info>
  For Astro customers, Astronomer recommends taking advantage of the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager) to store connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See [Manage Astro connections in branch-based deploy workflows](/docs/astro/best-practices/connections-branch-deploys).
</Info>

1. In your terminal, run the following command to [generate a private RSA key using OpenSSL](https://docs.openssl.org/master/man1/openssl-genrsa/). Note that while there are other options to generate a key pair, Snowflake has [specific requirements for the key format](https://docs.snowflake.com/en/user-guide/key-pair-auth) and may not accept keys generated with other tools. Make sure to write down the key passphrase as you will need it later.

   ```bash wrap theme={null}
   openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 des3 -inform PEM -out rsa_key.p8
   ```

2. Generate the associated public key using the following command:

   ```bash wrap theme={null}
   openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
   ```

3. Format the private key. Version 6.3.0+ of the Airflow Snowflake provider requires the private key to be base64 encoded. You can create a base64 encoded key with the following script:

   ```python wrap theme={null}
   import base64

   with open("path/to/rsa_key.p8", "rb") as key_file:
       private_key_content = base64.b64encode(key_file.read()).decode("utf-8")
       print(private_key_content)
   ```

<Note>
  If you're on version 6.2.2 or older of the Airflow Snowflake provider, you need to provide the private key without any coding conversions but with newlines encoded as `\n`. You can use the script below to format the key correctly:

  ```python wrap theme={null}
  def format_private_key(private_key_path):
      with open(private_key_path, 'r') as key_file:
          private_key = key_file.read()
      return private_key.replace('\n', '\\n')

  formatted_key = format_private_key('rsa_key.pem')
  print(formatted_key)
  ```
</Note>

4. In the Snowflake UI, run the following SQL command to add the **public** key to your [user](https://docs.snowflake.com/en/user-guide/admin-user-management). You can paste the **public** key directly from the `rsa_key.pub` file without needing to modify it.

   ```sql wrap theme={null}
   ALTER USER <your user> SET RSA_PUBLIC_KEY='<your public key>';
   ```

5. In the Airflow UI, go to **Admin** -> **Connections** and click **+** to create a new connection. Choose the `Snowflake` connection type and enter the following information:

   * **Connection ID**: `snowflake_conn`
   * **[Schema](https://docs.snowflake.com/en/sql-reference/sql/create-schema.html)**: Your Snowflake schema. The example DAG uses `DEMO_SCHEMA`.
   * **Login**: Your Snowflake [user](https://docs.snowflake.com/en/sql-reference/sql/create-user) name. Make sure to capitalize the user name as the `SnowflakeAPIOperator` requires it.
   * **Password**: Your private key passphrase.
   * **Extra**: Enter the following JSON object with your own Snowflake [account identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier), [database](https://docs.snowflake.com/en/sql-reference/sql/create-database), your [role](https://docs.snowflake.com/en/sql-reference/sql/create-role) in properly capitalized format, and your [warehouse](https://docs.snowflake.com/en/sql-reference/sql/create-warehouse).

     ```json wrap theme={null}
     {
         "account": "<your account id in the form of abc12345>",
         "warehouse": "<your warehouse>",
         "database": "DEMO_DB",
         "region": "<your region>",
         "role": "<your role in capitalized format>",
         "private_key_content": "LS0..<key>..C0="
     }
     ```

<Tip>
  When using JSON format to set your connection, use the following parameters:

  ```json wrap theme={null}
  AIRFLOW_CONN_SNOWFLAKE_DEFAULT='{
      "conn_type":"snowflake",
      "login":"<your user, properly capitalized>",
      "password":"<your private key passphrase>",
      "schema":"DEMO_SCHEMA",
      "extra":{
          "account":"<your account id in the form of abc12345",
          "warehouse":"<your warehouse>",
          "database":"DEMO_DB",
          "region":"<your region>",
          "role":"<your role, properly capitalized>",
          "private_key_content":"LS0..<key>..C0="
      }
  }'
  ```
</Tip>

## Step 3: Add your SQL statements

The DAG you will create in Step 4 runs multiple SQL statements against your Snowflake data warehouse. While it is possible to add SQL statements directly in your DAG file it is common practice to store them in separate files. When initializing your Astro project with the Astro CLI, an `include` folder was created. The contents of this folder will automatically be mounted into the Dockerfile, which makes it the standard location in which supporting files are stored.

1. Create a folder called `sql` in your `include` folder.

2. Create a new file in `include/sql` called `insert_data.sql` and copy the following code:

   ```sql wrap theme={null}
   INSERT INTO {{ params.db_name }}.{{ params.schema_name }}.{{ params.table_name }} (ID, NAME)
   VALUES
       (1, 'Avery');
   ```

   This file contains one SQL statement that inserts a row into a table. The database, schema, and table names are parameterized so that you can pass them to the operator at runtime.

3. The `SnowflakeSqlApiOperator` can run multiple SQL statements in a single task. Create a new file in `include/sql` called `multiple_statements_query.sql` and copy the following code:

   ```sql wrap theme={null}
   INSERT INTO {{ params.db_name }}.{{ params.schema_name }}.{{ params.table_name }} (ID, NAME)
   VALUES
       (2, 'Peanut'),
       (3, 'Butter');

   INSERT INTO {{ params.db_name }}.{{ params.schema_name }}.{{ params.table_name }} (ID, NAME)
   VALUES
       (4, 'Vega'),
       (5, 'Harper');
   ```

   This file contains two SQL statements that insert multiple rows into a table.

<Tip>
  When running SQL statements from Airflow operators, you can store the SQL code in individual SQL files, in a combined SQL file, or as strings in a Python module. Astronomer recommends storing lengthy SQL statements in a dedicated file to keep your DAG files clean and readable.
</Tip>

## Step 4: Write a Snowflake DAG

1. Create a new file in your `dags` directory called `my_snowflake_dag.py`.

2. Copy and paste the code below into the file:

   ```python expandable wrap theme={null}
   """
   ### Snowflake Tutorial DAG

   This DAG demonstrates how to use the SQLExecuteQueryOperator,
   SnowflakeSqlApiOperator and SQLColumnCheckOperator to interact with Snowflake.
   """

   from airflow.decorators import dag
   from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
   from airflow.providers.common.sql.operators.sql import SQLColumnCheckOperator
   from airflow.providers.snowflake.operators.snowflake import SnowflakeSqlApiOperator
   from airflow.models.baseoperator import chain
   from pendulum import datetime, duration
   import os

   _SNOWFLAKE_CONN_ID = "snowflake_conn"
   _SNOWFLAKE_DB = "DEMO_DB"
   _SNOWFLAKE_SCHEMA = "DEMO_SCHEMA"
   _SNOWFLAKE_TABLE = "DEMO_TABLE"


   @dag(
       dag_display_name="Snowflake Tutorial DAG ❄️",
       start_date=datetime(2024, 9, 1),
       schedule=None,
       catchup=False,
       default_args={"owner": "airflow", "retries": 1, "retry_delay": duration(seconds=5)},
       doc_md=__doc__,
       tags=["tutorial"],
       template_searchpath=[
           os.path.join(os.path.dirname(os.path.abspath(__file__)), "../include/sql")
       ],  # path to the SQL templates
   )
   def my_snowflake_dag():

       # you can execute SQL queries directly using the SQLExecuteQueryOperator
       create_or_replace_table = SQLExecuteQueryOperator(
           task_id="create_or_replace_table",
           conn_id=_SNOWFLAKE_CONN_ID,
           database="DEMO_DB",
           sql=f"""
               CREATE OR REPLACE TABLE {_SNOWFLAKE_SCHEMA}.{_SNOWFLAKE_TABLE} (
                   ID INT,
                   NAME VARCHAR
               )
           """,
       )

       # you can also execute SQL queries from a file, make sure to add the path to the template_searchpath
       insert_data = SQLExecuteQueryOperator(
           task_id="insert_data",
           conn_id=_SNOWFLAKE_CONN_ID,
           database="DEMO_DB",
           sql="insert_data.sql",
           params={
               "db_name": _SNOWFLAKE_DB,
               "schema_name": _SNOWFLAKE_SCHEMA,
               "table_name": _SNOWFLAKE_TABLE,
           },
       )

       # you can also execute multiple SQL statements using the SnowflakeSqlApiOperator
       # make sure to set the statement_count parameter to the number of statements in the SQL file
       # and that your connection details are in their proper capitalized form!
       insert_data_multiple_statements = SnowflakeSqlApiOperator(
           task_id="insert_data_multiple_statements",
           snowflake_conn_id=_SNOWFLAKE_CONN_ID,
           sql="multiple_statements_query.sql",
           database=_SNOWFLAKE_DB,
           schema=_SNOWFLAKE_SCHEMA,
           params={
               "db_name": _SNOWFLAKE_DB,
               "schema_name": _SNOWFLAKE_SCHEMA,
               "table_name": _SNOWFLAKE_TABLE,
           },
           statement_count=2,  # needs to match the number of statements in the SQL file
           autocommit=True,
       )

       # use SQLCheck operators to check the quality of your data
       data_quality_check = SQLColumnCheckOperator(
           task_id="data_quality_check",
           conn_id=_SNOWFLAKE_CONN_ID,
           database=_SNOWFLAKE_DB,
           table=f"{_SNOWFLAKE_SCHEMA}.{_SNOWFLAKE_TABLE}",
           column_mapping={
               "ID": {"null_check": {"equal_to": 0}, "distinct_check": {"geq_to": 3}}
           },
       )

       chain(
           create_or_replace_table,
           insert_data,
           insert_data_multiple_statements,
           data_quality_check,
       )


   my_snowflake_dag()
   ```

   The DAG completes the following steps:

   * Uses the [`SQLExecuteQueryOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLExecuteQueryOperator) to run an in-line SQL statement that creates a table in Snowflake.
   * Uses the `SQLExecuteQueryOperator` to run an SQL file containing a singular SQL statement that inserts data into the table.
   * Uses the [`SnowflakeSqlApiOperator`](https://airflow.apache.org/registry/providers/snowflake#snowflake-snowflake-SnowflakeSqlApiOperator) to run an SQL file containing multiple SQL statements that insert data into the table. The operator is set to run in [deferrable](/docs/learn/deferrable-operators) mode.
   * Uses the [`SQLColumnCheckOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLColumnCheckOperator) to run a data quality check on the table checking that there are no NULL values in the `ID` column and that it contains at least 3 distinct values. To learn more about SQL check operators, see [Run data quality checks using SQL check operators](/docs/learn/airflow-sql-data-quality).

   The `chain()` method at the end of the DAG sets the [dependencies](/docs/learn/managing-dependencies). This method is commonly used over bitshift operators (`>>`) to make it easier to read dependencies between many tasks.

## Step 5: Run the DAG

1. In the Airflow UI, click the play button to manually run your DAG.

2. Open the [logs](/docs/learn/logging) for the `data_quality_check` task to see the results of the data quality check, confirming that the table was created and populated correctly.

   ```text wrap theme={null}
   [2025-07-03, 16:03:09 UTC] {sql.py:469} INFO - Record: [('ID', 'null_check', 0), ('ID', 'distinct_check', 5)]
   [2025-07-03, 16:03:09 UTC] {sql.py:492} INFO - All tests have passed
   ```

## More on the Airflow Snowflake integration

This section provides additional information on orchestrating actions in Snowflake with Airflow.

### Snowflake operators and hooks

Several open source packages contain operators used to orchestrate Snowflake in Airflow.

The [Common SQL provider package](https://airflow.apache.org/registry/providers/common-sql) contains operators that you can use with several SQL databases, including Snowflake:

* [`SQLExecuteQueryOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLExecuteQueryOperator): Executes a single SQL statement. This operator replaces the deprecated `SnowflakeOperator`.
* [`SQLColumnCheckOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLColumnCheckOperator): Performs a data quality check against columns of a given table. See [Run data quality checks using SQL check operators](/docs/learn/airflow-sql-data-quality).
* [`SQLTableCheckOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLTableCheckOperator): Performs a data quality check against a given table.

The [Snowflake provider package](https://airflow.apache.org/registry/providers/snowflake/) contains:

* [`SnowflakeSqlApiOperator`](https://airflow.apache.org/registry/providers/snowflake#snowflake-snowflake-SnowflakeSqlApiOperator): Executes multiple SQL statements in a single task. Note that this operator uses the Snowflake SQL API, which requires connection parameters such as the role and user to be properly capitalized. The operator can be set to be [deferrable](/docs/learn/deferrable-operators) using `deferrable=True`.
* [`CopyFromExternalStageToSnowflakeOperator`](https://airflow.apache.org/registry/providers/snowflake#snowflake-copy_into_snowflake-CopyFromExternalStageToSnowflakeOperator): Copies data from an external stage to a Snowflake table. Note that the `prefix` parameter will be added to the full stage path defined in Snowflake.
* [`SnowflakeHook`](https://airflow.apache.org/registry/providers/snowflake#snowflake-snowflake-SnowflakeHook): A client to interact with Snowflake which is commonly used when building custom operators interacting with Snowflake.

### Best practices and considerations

The following are some best practices and considerations to keep in mind when orchestrating Snowflake queries from Airflow:

* To reduce costs and improve the scalability of your Airflow environment, consider using the `SnowflakeSqlApiOperator` in [deferrable](/docs/learn/deferrable-operators) mode for long running queries.
* Set your default Snowflake query specifications such as Warehouse, Role, Schema, and so on in the Airflow connection. Then overwrite those parameters for specific tasks as necessary in your operator definitions. This is cleaner and easier to read than adding `USE Warehouse XYZ;` statements within your queries. If you are an Astro customer, use the [Astro Environment Manager](/docs/astro/create-and-link-connections) to define your base connection and add overrides for specific deployments and tasks.
* Pay attention to which Snowflake compute resources your tasks are using, as overtaxing your assigned resources can cause slowdowns in your Airflow tasks. It is generally recommended to have different warehouses devoted to your different Airflow environments to ensure DAG development and testing doesn't interfere with DAGs running in production. If you want to optimize your Snowflake usage, consider using [SnowPatrol](/docs/learn/reference-architecture-snowpatrol) to detect anomalies in your Snowflake spend.
* Make use of [Snowflake stages](https://docs.snowflake.com/en/sql-reference/sql/create-stage.html) together with the [`CopyFromExternalStageToSnowflakeOperator`](https://airflow.apache.org/registry/providers/snowflake#snowflake-copy_into_snowflake-CopyFromExternalStageToSnowflakeOperator) when loading large amounts data from an external system using Airflow.

## Conclusion

Congratulations! You've connected Airflow to Snowflake and executed Snowflake queries from your Airflow DAGs. You've also learned about best practices and considerations when orchestrating Snowflake queries from Airflow.


# Orchestrate Snowpark Machine Learning Workflows with Apache Airflow
Source: https://astronomer.io/docs/learn/airflow-snowpark

Learn how to integrate Snowpark and Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Snowpark](https://www.snowflake.com/en/data-cloud/snowpark/) is the set of runtimes and libraries that securely deploy and process Python and other programming code in [Snowflake](https://www.snowflake.com/en/). This includes [Snowpark ML](https://docs.snowflake.com/en/developer-guide/snowpark-ml/index), the Python library and underlying infrastructure for end-to-end ML workflows in Snowflake. Snowpark ML has 2 components: [Snowpark ML Modeling](https://docs.snowflake.com/en/developer-guide/snowpark-ml/snowpark-ml-modeling) for model development, and Snowpark ML Operations including the [Snowpark Model Registry](https://docs.snowflake.com/en/developer-guide/snowpark-ml/snowpark-ml-mlops-model-registry), for model deployment and management.

In this tutorial, you'll learn how to:

* Create a [custom XCom backend](/docs/learn/custom-xcom-backend-strategies) in Snowflake.
* Create and use the Snowpark Model Registry in Snowflake.
* Use Airflow decorators to run code in Snowpark, both in a pre-built and custom virtual environment.
* Run a [Logistic Regression model](https://mlu-explain.github.io/logistic-regression/) on a synthetic dataset to predict skiers' afternoon beverage choice.

<Warning>
  The provider used in this tutorial is currently in beta and both its contents and decorators are subject to change. After the official release, this tutorial will be updated.
</Warning>

<Frame>
  <img alt="A plot showing a confusion matrix and ROC curve with a high AUC for hot_chocolate, a medium AUC for snow_mocha and tea, and low predictive power for wine and coffee." />
</Frame>

## Why use Airflow with Snowpark?

Snowpark allows you to use Python to perform transformations and machine learning operations on data stored in Snowflake.

Integrating Snowpark for Python with Airflow offers the benefits of:

* Running machine learning models directly in Snowflake, without having to move data out of Snowflake.
* Expressing data transformations in Snowflake in Python instead of SQL.
* Storing and versioning your machine learning models using the Snowpark Model Registry inside Snowflake.
* Using Snowpark's compute resources instead of your Airflow cluster resources for machine learning.
* Using Airflow for Snowpark Python orchestration to enable automation, auditing, logging, retry, and complex triggering for powerful workflows.

The Snowpark provider for Airflow simplifies interacting with Snowpark by:

* Connecting to Snowflake using an [Airflow connection](/docs/learn/connections/snowflake), removing the need to directly pass credentials in your DAG.
* Automatically instantiating a Snowpark session.
* Automatically serializing and deserializing Snowpark dataframes passed using [Airflow XCom](/docs/learn/airflow-passing-data-between-tasks).
* Integrating with [OpenLineage](/docs/learn/airflow-openlineage).
* Providing a pre-built custom XCom backend for Snowflake.

Additionally, this tutorial shows how to use Snowflake as a custom XCom backend. This is especially useful for organizations with strict compliance requirements who want to keep all their data in Snowflake, but still leverage [Airflow XCom](/docs/learn/airflow-passing-data-between-tasks) to pass data between tasks.

## Time to complete

This tutorial takes approximately 45 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of Snowflake and Snowpark. See [Introduction to Snowflake](https://docs.snowflake.com/en/user-guide-intro.html) and the [Snowpark API documentation](https://docs.snowflake.com/en/developer-guide/snowpark/index).
* Airflow decorators. See [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).
* Airflow connections. See [Managing your Connections in Apache Airflow](/docs/learn/connections).
* Setup/ teardown tasks in Airflow. See [Use setup and teardown tasks in Airflow](/docs/learn/airflow-setup-teardown).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).

* A Snowflake account. A [30-day free trial](https://trial.snowflake.com/?owner=SPN-PID-365384) is available. You need to have at least one database and one schema created to store the data and models used in this tutorial.

* (Optional) This tutorial includes instructions on how to use the Snowflake [custom XCom backend](/docs/learn/custom-xcom-backend-strategies) included in the provider. If you want to use this custom XCom backend you will need to either:
  * Run the DAG using a Snowflake account with `ACCOUNTADMIN` privileges to allow the DAG's first task to create the required database, schema, stage and table. See [Step 3.3](#step-3-create-your-dag) for more instructions. The free trial account has the required privileges.
  * Ask your Snowflake administrator to:

    * Provide you with the name of an existing database, schema, and stage. You need to use these names in [Step 1.8](#step-1-configure-your-astro-project) for the `AIRFLOW__CORE__XCOM_SNOWFLAKE_TABLE` and `AIRFLOW__CORE__XCOM_SNOWFLAKE_STAGE` environment variables.
    * Create an `XCOM_TABLE` with the following schema:

    ```sql wrap theme={null}
    dag_id varchar NOT NULL, 
    task_id varchar NOT NULL, 
    run_id varchar NOT NULL,
    multi_index integer NOT NULL,
    key varchar NOT NULL,
    value_type varchar NOT NULL,
    value varchar NOT NULL
    ```

<Info>
  The example code from this tutorial is also available on [GitHub](https://github.com/astronomer/airflow-snowpark-tutorial).
</Info>

## Step 1: Configure your Astro project

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-snowpark-tutorial && cd astro-snowpark-tutorial
   $ astro dev init
   ```

2. Create a new file in your Astro project's root directory called `requirements-snowpark.txt`. This file contains all Python packages that you install in your reusable Snowpark environment.

   ```text wrap theme={null}
   psycopg2-binary
   snowflake_snowpark_python[pandas]>=1.11.1
   git+https://github.com/astronomer/astro-provider-snowflake.git
   virtualenv
   ```

3. Change the content of the `Dockerfile` of your Astro project to the following, which creates a virtual environment by using the [Astro venv buildkit](https://github.com/astronomer/astro-provider-venv). The requirements added in the previous step are installed in that virtual environment. This tutorial includes Snowpark Python tasks that are running in virtual environments, which is a common pattern in production to simplify dependency management. This Dockerfile creates a virtual environment called `snowpark` with the Python version 3.8 and the packages specified in `requirements-snowpark.txt`.

   ```dockerfile wrap theme={null}
   # syntax=quay.io/astronomer/airflow-extensions:latest

   FROM quay.io/astronomer/astro-runtime:10.2.0

   # Create the virtual environment
   PYENV 3.8 snowpark requirements-snowpark.txt

   # Install packages into the virtual environment
   COPY requirements-snowpark.txt /tmp
   RUN python3.8 -m pip install -r /tmp/requirements-snowpark.txt
   ```

4. Add the following package to your `packages.txt` file:

   ```text wrap theme={null}
   build-essential
   git
   ```

5. Add the following packages to your `requirements.txt` file. The Astro Snowflake provider is installed from the `whl` file.

   ```text wrap theme={null}
   apache-airflow-providers-snowflake==5.2.0
   apache-airflow-providers-amazon==8.15.0
   snowflake-snowpark-python[pandas]==1.11.1
   snowflake-ml-python==1.1.2
   matplotlib==3.8.1
   git+https://github.com/astronomer/astro-provider-snowflake.git
   ```

<Warning>
  The Astro Snowflake provider is currently in beta. Classes from this provider might be subject to change and will be included in the [Snowflake provider](https://airflow.apache.org/registry/providers/snowflake/) in a future release.
</Warning>

7. To create an [Airflow connection](/docs/learn/connections) to Snowflake and [allow serialization of Astro Python SDK objects](https://astro-sdk-python.readthedocs.io/en/stable/guides/xcom_backend.html#airflow-s-xcom-backend), add the following to your `.env` file. Make sure to enter your own Snowflake credentials as well as the name of an existing database and schema.

   ```text wrap theme={null}
   AIRFLOW__CORE__ALLOWED_DESERIALIZATION_CLASSES=airflow\.* astro\.*
   AIRFLOW_CONN_SNOWFLAKE_DEFAULT='{
       "conn_type":"snowflake",
       "login":"<username>",
       "password":"<password>",
       "schema":"MY_SKI_DATA_SCHEMA",
       "extra":
           {
               "account":"<account>",
               "warehouse":"<warehouse>",
               "database":"MY_SKI_DATA_DATABASE",
               "region":"<region>",
               "role":"<role>",
               "authenticator":"snowflake",
               "session_parameters":null,
               "application":"AIRFLOW"
           }
       }'
   ```

<Info>
  For more information on creating a Snowflake connection, see [Create a Snowflake connection in Airflow](/docs/learn/connections/snowflake).
</Info>

8. (Optional) If you want to use a Snowflake custom XCom backend, add the following additional variables to your `.env`. Replace the values with the name of your own database, schema, table, and stage if you aren't using the suggested values.

   ```text wrap theme={null}
   AIRFLOW__CORE__XCOM_BACKEND=snowpark_provider.xcom_backends.snowflake.SnowflakeXComBackend
   AIRFLOW__CORE__XCOM_SNOWFLAKE_TABLE='AIRFLOW_XCOM_DB.AIRFLOW_XCOM_SCHEMA.XCOM_TABLE'
   AIRFLOW__CORE__XCOM_SNOWFLAKE_STAGE='AIRFLOW_XCOM_DB.AIRFLOW_XCOM_SCHEMA.XCOM_STAGE'
   AIRFLOW__CORE__XCOM_SNOWFLAKE_CONN_NAME='snowflake_default'
   ```

## Step 2: Add your data

The DAG in this tutorial runs a classification model on synthetic data to predict which afternoon beverage a skier will choose based on attributes like ski color, ski resort, and amount of new snow. The data is generated using [this script](https://github.com/astronomer/airflow-snowpark-tutorial/blob/main/include/data/create_ski_dataset.py).

1. Create a new directory in your Astro project's `include` directory called `data`.

2. Download the dataset from [Astronomer's GitHub](https://github.com/astronomer/learn-tutorials-data/blob/main/ski_dataset.csv) and save it in `include/data`.

## Step 3: Create your DAG

1. In your `dags` folder, create a file called `airflow_with_snowpark_tutorial.py`.

2. Copy the following code into the file. Make sure to provide your Snowflake database and schema names to `MY_SNOWFLAKE_DATABASE` and `MY_SNOWFLAKE_SCHEMA`.

   ```python expandable wrap theme={null}
   """
   ### Orchestrate data transformation and model training in Snowflake using Snowpark

   This DAG shows how to use specialized decorators to run Snowpark code in Airflow.
   Note that it uses the Airflow 2.7 feature of setup/ teardown tasks to create
   and clean up a Snowflake custom XCom backend.
   If you want to use regular XCom set
   `SETUP_TEARDOWN_SNOWFLAKE_CUSTOM_XCOM_BACKEND` to `False`.
   """

   from datetime import datetime
   from airflow.decorators import dag, task
   from astro import sql as aql
   from astro.files import File
   from astro.sql.table import Table
   from airflow.models.baseoperator import chain


   # toggle to True if you are using the Snowflake XCOM backend and want to
   # use setup/ teardown tasks to create all necessary objects and clean up the XCOM table
   # after the DAG has run
   SETUP_TEARDOWN_SNOWFLAKE_CUSTOM_XCOM_BACKEND = False
   # provide your Snowflake XCOM database, schema, stage and table names
   MY_SNOWFLAKE_XCOM_DATABASE = "SNOWPARK_XCOM_DB"
   MY_SNOWFLAKE_XCOM_SCHEMA = "SNOWPARK_XCOM_SCHEMA"
   MY_SNOWFLAKE_XCOM_STAGE = "XCOM_STAGE"
   MY_SNOWFLAKE_XCOM_TABLE = "XCOM_TABLE"

   # provide your Snowflake database name, schema name, connection ID
   # and path to the Snowpark environment binary
   MY_SNOWFLAKE_DATABASE = "MY_SKI_DATA_DATABASE"  # an existing database
   MY_SNOWFLAKE_SCHEMA = "MY_SKI_DATA_SCHEMA"  # an existing schema
   MY_SNOWFLAKE_TABLE = "MY_SKI_DATA_TABLE"
   SNOWFLAKE_CONN_ID = "snowflake_default"
   SNOWPARK_BIN = "/home/astro/.venv/snowpark/bin/python"

   # while this tutorial will run with the default Snowflake warehouse, larger
   # datasets may require a Snowpark optimized warehouse. Set the following toggle to true to
   # use such a warehouse and provide your Snowpark and regular warehouses' names.
   USE_SNOWPARK_WAREHOUSE = False
   MY_SNOWPARK_WAREHOUSE = "SNOWPARK_WH"
   MY_SNOWFLAKE_REGULAR_WAREHOUSE = "HUMANS"


   @dag(
       start_date=datetime(2023, 9, 1),
       schedule=None,
       catchup=False,
   )
   def airflow_with_snowpark_tutorial():
       if SETUP_TEARDOWN_SNOWFLAKE_CUSTOM_XCOM_BACKEND:

           @task.snowpark_python(
               snowflake_conn_id=SNOWFLAKE_CONN_ID,
           )
           def create_snowflake_objects(
               snowflake_xcom_database,
               snowflake_xcom_schema,
               snowflake_xcom_table,
               snowflake_xcom_table_stage,
               use_snowpark_warehouse=False,
               snowpark_warehouse=None,
           ):
               from snowflake.snowpark.exceptions import SnowparkSQLException

               try:
                   snowpark_session.sql(
                       f"""CREATE DATABASE IF NOT EXISTS
                           {snowflake_xcom_database};
                       """
                   ).collect()
                   print(f"Created database {snowflake_xcom_database}.")

                   snowpark_session.sql(
                       f"""CREATE SCHEMA IF NOT EXISTS 
                           {snowflake_xcom_database}.
                           {snowflake_xcom_schema};
                       """
                   ).collect()
                   print(f"Created schema {snowflake_xcom_schema}.")

                   if use_snowpark_warehouse:
                       snowpark_session.sql(
                           f"""CREATE WAREHOUSE IF NOT EXISTS 
                           {snowpark_warehouse} 
                               WITH 
                                   WAREHOUSE_SIZE = 'MEDIUM'
                                   WAREHOUSE_TYPE = 'SNOWPARK-OPTIMIZED';
                           """
                       ).collect()
                       print(f"Created warehouse {snowpark_warehouse}.")
               except SnowparkSQLException as e:
                   print(e)
                   print(
                       f"""You do not have the necessary privileges to create objects in Snowflake.
                       If they do not exist already, please contact your Snowflake administrator
                       to create the following objects for you: 
                           - DATABASE: {snowflake_xcom_database},
                           - SCHEMA: {snowflake_xcom_schema},
                           - WAREHOUSE: {snowpark_warehouse} (if you want to use a Snowpark warehouse) 
                       """
                   )

               snowpark_session.sql(
                   f"""CREATE TABLE IF NOT EXISTS
                       {snowflake_xcom_database}.
                       {snowflake_xcom_schema}.
                       {snowflake_xcom_table}
                           ( 
                               dag_id varchar NOT NULL, 
                               task_id varchar NOT NULL, 
                               run_id varchar NOT NULL,
                               multi_index integer NOT NULL,
                               key varchar NOT NULL,
                               value_type varchar NOT NULL,
                               value varchar NOT NULL
                           ); 
                   """
               ).collect()
               print(f"Table {snowflake_xcom_table} is ready!")

               snowpark_session.sql(
                   f"""CREATE STAGE IF NOT EXISTS
                       {snowflake_xcom_database}.\
                       {snowflake_xcom_schema}.\
                       {snowflake_xcom_table_stage} 
                           DIRECTORY = (ENABLE = TRUE)
                           ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');
                       """
               ).collect()
               print(f"Stage {snowflake_xcom_table_stage} is ready!")

           create_snowflake_objects_obj = create_snowflake_objects(
               snowflake_xcom_database=MY_SNOWFLAKE_XCOM_DATABASE,
               snowflake_xcom_schema=MY_SNOWFLAKE_XCOM_SCHEMA,
               snowflake_xcom_table=MY_SNOWFLAKE_XCOM_TABLE,
               snowflake_xcom_table_stage=MY_SNOWFLAKE_XCOM_STAGE,
               use_snowpark_warehouse=USE_SNOWPARK_WAREHOUSE,
               snowpark_warehouse=MY_SNOWPARK_WAREHOUSE,
           )

       # use the Astro Python SDK to load data from a CSV file into Snowflake
       load_file_obj = aql.load_file(
           task_id="load_file",
           input_file=File("include/data/ski_dataset.csv"),
           output_table=Table(
               metadata={
                   "database": MY_SNOWFLAKE_DATABASE,
                   "schema": MY_SNOWFLAKE_SCHEMA,
               },
               conn_id=SNOWFLAKE_CONN_ID,
               name=MY_SNOWFLAKE_TABLE,
           ),
           if_exists="replace",
       )

       # create a model registry in Snowflake
       @task.snowpark_python(
           snowflake_conn_id=SNOWFLAKE_CONN_ID,
       )
       def create_model_registry(demo_database, demo_schema):
           from snowflake.ml.registry import model_registry

           model_registry.create_model_registry(
               session=snowpark_session,
               database_name=demo_database,
               schema_name=demo_schema,
           )

       # Tasks using the @task.snowpark_python decorator run in
       # the regular Snowpark Python environment
       @task.snowpark_python(
           snowflake_conn_id=SNOWFLAKE_CONN_ID,
       )
       def transform_table_step_one(df):
           from snowflake.snowpark.functions import col
           import pandas as pd
           import re

           pattern = r"table=([^&]+)&schema=([^&]+)&database=([^&]+)"
           match = re.search(pattern, df.uri)

           formatted_result = f"{match.group(3)}.{match.group(2)}.{match.group(1)}"

           df_snowpark = snowpark_session.table(formatted_result)

           filtered_data = df_snowpark.filter(
               (col("AFTERNOON_BEVERAGE") == "coffee")
               | (col("AFTERNOON_BEVERAGE") == "tea")
               | (col("AFTERNOON_BEVERAGE") == "snow_mocha")
               | (col("AFTERNOON_BEVERAGE") == "hot_chocolate")
               | (col("AFTERNOON_BEVERAGE") == "wine")
           ).collect()
           filtered_df = pd.DataFrame(filtered_data, columns=df_snowpark.columns)

           return filtered_df

       # Tasks using the @task.snowpark_ext_python decorator can use an
       # existing python environment
       @task.snowpark_ext_python(snowflake_conn_id=SNOWFLAKE_CONN_ID, python=SNOWPARK_BIN)
       def transform_table_step_two(df):
           df_serious_skiers = df[df["HOURS_SKIED"] >= 1]

           return df_serious_skiers

       # Tasks using the @task.snowpark_virtualenv decorator run in a virtual
       # environment created on the spot using the requirements specified
       @task.snowpark_virtualenv(
           snowflake_conn_id=SNOWFLAKE_CONN_ID,
           requirements=["pandas", "scikit-learn"],
       )
       def train_beverage_classifier(
           df,
           database_name,
           schema_name,
           use_snowpark_warehouse=False,
           snowpark_warehouse=None,
           snowflake_regular_warehouse=None,
       ):
           from sklearn.model_selection import train_test_split
           import pandas as pd
           from snowflake.ml.registry import model_registry
           from snowflake.ml.modeling.linear_model import LogisticRegression
           from uuid import uuid4
           from snowflake.ml.modeling.preprocessing import OneHotEncoder, StandardScaler

           registry = model_registry.ModelRegistry(
               session=snowpark_session,
               database_name=database_name,
               schema_name=schema_name,
           )

           df.columns = [str(col).replace("'", "").replace('"', "") for col in df.columns]
           X = df.drop(columns=["AFTERNOON_BEVERAGE", "SKIER_ID"])
           y = df["AFTERNOON_BEVERAGE"]

           X_train, X_test, y_train, y_test = train_test_split(
               X, y, test_size=0.2, random_state=42
           )

           train_data = pd.concat([X_train, y_train], axis=1)
           test_data = pd.concat([X_test, y_test], axis=1)

           categorical_features = ["RESORT", "SKI_COLOR", "JACKET_COLOR", "HAD_LUNCH"]
           numeric_features = ["HOURS_SKIED", "SNOW_QUALITY", "CM_OF_NEW_SNOW"]
           label_col = ["AFTERNOON_BEVERAGE"]

           scaler = StandardScaler(
               input_cols=numeric_features,
               output_cols=numeric_features,
               drop_input_cols=True,
           )
           scaler.fit(train_data)
           train_data_scaled = scaler.transform(train_data)
           test_data_scaled = scaler.transform(test_data)

           one_hot_encoder = OneHotEncoder(
               input_cols=categorical_features,
               output_cols=categorical_features,
               drop_input_cols=True,
           )
           one_hot_encoder.fit(train_data_scaled)
           train_data_scaled_encoded = one_hot_encoder.transform(train_data_scaled)
           test_data_scaled_encoded = one_hot_encoder.transform(test_data_scaled)

           feature_cols = train_data_scaled_encoded.drop(
               columns=["AFTERNOON_BEVERAGE"]
           ).columns

           classifier = LogisticRegression(
               max_iter=10000, input_cols=feature_cols, label_cols=label_col
           )

           feature_cols = [str(col).replace('"', "") for col in feature_cols]

           if use_snowpark_warehouse:
               snowpark_session.use_warehouse(snowpark_warehouse)

           classifier.fit(train_data_scaled_encoded)
           score = classifier.score(test_data_scaled_encoded)
           print(f"Accuracy: {score:.4f}")

           y_pred = classifier.predict(test_data_scaled_encoded)
           y_pred_proba = classifier.predict_proba(test_data_scaled_encoded)

           # register the Snowpark model in the Snowflake model registry
           registry.log_model(
               model=classifier,
               model_version=uuid4().urn,
               model_name="Ski Beverage Classifier",
               tags={"stage": "dev", "model_type": "LogisticRegression"},
           )

           if use_snowpark_warehouse:
               snowpark_session.use_warehouse(snowflake_regular_warehouse)
               snowpark_session.sql(
                   f"""ALTER WAREHOUSE
                   {snowpark_warehouse}
                   SUSPEND;"""
               ).collect()

           y_pred_proba.columns = [
               str(col).replace('"', "") for col in y_pred_proba.columns
           ]
           y_pred.columns = [str(col).replace('"', "") for col in y_pred.columns]

           prediction_results = pd.concat(
               [
                   y_pred_proba[
                       [
                           "PREDICT_PROBA_snow_mocha",
                           "PREDICT_PROBA_tea",
                           "PREDICT_PROBA_coffee",
                           "PREDICT_PROBA_hot_chocolate",
                           "PREDICT_PROBA_wine",
                       ]
                   ],
                   y_pred[["OUTPUT_AFTERNOON_BEVERAGE"]],
                   y_test,
               ],
               axis=1,
           )

           classes = classifier.to_sklearn().classes_
           classes_df = pd.DataFrame(classes)

           # convert to string column names for parquet serialization
           prediction_results.columns = [
               str(col).replace("'", "").replace('"', "")
               for col in prediction_results.columns
           ]
           classes_df.columns = ["classes"]

           return {
               "prediction_results": prediction_results,
               "classes": classes_df,
           }

       # using a regular Airflow task to plot the results
       @task
       def plot_results(prediction_results):
           import matplotlib.pyplot as plt
           from sklearn.metrics import roc_curve, auc, ConfusionMatrixDisplay
           from sklearn.preprocessing import label_binarize

           y_pred = prediction_results["prediction_results"]["OUTPUT_AFTERNOON_BEVERAGE"]
           y_test = prediction_results["prediction_results"]["AFTERNOON_BEVERAGE"]
           y_proba = prediction_results["prediction_results"][
               [
                   "PREDICT_PROBA_coffee",
                   "PREDICT_PROBA_hot_chocolate",
                   "PREDICT_PROBA_snow_mocha",
                   "PREDICT_PROBA_tea",
                   "PREDICT_PROBA_wine",
               ]
           ]
           y_score = y_proba.to_numpy()
           classes = prediction_results["classes"].iloc[:, 0].values
           y_test_bin = label_binarize(y_test, classes=classes)

           fig, ax = plt.subplots(1, 2, figsize=(15, 6))

           ConfusionMatrixDisplay.from_predictions(y_test, y_pred, ax=ax[0], cmap="Blues")
           ax[0].set_title(f"Confusion Matrix")

           fpr = dict()
           tpr = dict()
           roc_auc = dict()

           for i, cls in enumerate(classes):
               fpr[cls], tpr[cls], _ = roc_curve(y_test_bin[:, i], y_score[:, i])
               roc_auc[cls] = auc(fpr[cls], tpr[cls])

               ax[1].plot(
                   fpr[cls],
                   tpr[cls],
                   label=f"ROC curve (area = {roc_auc[cls]:.2f}) for {cls}",
               )

           ax[1].plot([0, 1], [0, 1], "k--")
           ax[1].set_xlim([0.0, 1.0])
           ax[1].set_ylim([0.0, 1.05])
           ax[1].set_xlabel("False Positive Rate")
           ax[1].set_ylabel("True Positive Rate")
           ax[1].set_title(f"ROC Curve")
           ax[1].legend(loc="lower right")

           fig.suptitle("Predicting afternoon beverage based on skiing data")

           plt.tight_layout()
           plt.savefig(f"include/metrics.png")

       if SETUP_TEARDOWN_SNOWFLAKE_CUSTOM_XCOM_BACKEND:
           # clean up the XCOM table
           @task.snowpark_ext_python(
               snowflake_conn_id=SNOWFLAKE_CONN_ID,
               python="/home/astro/.venv/snowpark/bin/python",
           )
           def cleanup_xcom_table(
               snowflake_xcom_database,
               snowflake_xcom_schema,
               snowflake_xcom_table,
               snowflake_xcom_stage,
           ):
               snowpark_session.database = snowflake_xcom_database
               snowpark_session.schema = snowflake_xcom_schema

               snowpark_session.sql(
                   f"""DROP TABLE IF EXISTS 
                   {snowflake_xcom_database}.
                   {snowflake_xcom_schema}.
                   {snowflake_xcom_table};"""
               ).collect()

               snowpark_session.sql(
                   f"""DROP STAGE IF EXISTS                
                   {snowflake_xcom_database}.
                   {snowflake_xcom_schema}.
                   {snowflake_xcom_stage};"""
               ).collect()

           cleanup_xcom_table_obj = cleanup_xcom_table(
               snowflake_xcom_database=MY_SNOWFLAKE_XCOM_DATABASE,
               snowflake_xcom_schema=MY_SNOWFLAKE_XCOM_SCHEMA,
               snowflake_xcom_table=MY_SNOWFLAKE_XCOM_TABLE,
               snowflake_xcom_stage=MY_SNOWFLAKE_XCOM_STAGE,
           )

       # set dependencies
       create_model_registry_obj = create_model_registry(
           demo_database=MY_SNOWFLAKE_DATABASE, demo_schema=MY_SNOWFLAKE_SCHEMA
       )

       train_beverage_classifier_obj = train_beverage_classifier(
           transform_table_step_two(transform_table_step_one(load_file_obj)),
           database_name=MY_SNOWFLAKE_DATABASE,
           schema_name=MY_SNOWFLAKE_SCHEMA,
           use_snowpark_warehouse=USE_SNOWPARK_WAREHOUSE,
           snowpark_warehouse=MY_SNOWPARK_WAREHOUSE,
           snowflake_regular_warehouse=MY_SNOWFLAKE_REGULAR_WAREHOUSE,
       )

       chain(create_model_registry_obj, train_beverage_classifier_obj)

       plot_results_obj = plot_results(train_beverage_classifier_obj)

       if SETUP_TEARDOWN_SNOWFLAKE_CUSTOM_XCOM_BACKEND:
           chain(create_snowflake_objects_obj, load_file_obj)
           chain(
               plot_results_obj,
               cleanup_xcom_table_obj.as_teardown(setups=create_snowflake_objects_obj),
           )


   airflow_with_snowpark_tutorial()
   ```

   This DAG consists of eight tasks in a simple ML orchestration pipeline.

   * (Optional) `create_snowflake_objects`: Creates the Snowflake objects required for the Snowflake custom XCom backend. This task uses the `@task.snowflake_python` decorator to run code within Snowpark, automatically instantiating a Snowpark session called `snowpark_session` from the connection ID provided to the `snowflake_conn_id` parameter. This task is a [setup task](/docs/learn/airflow-setup-teardown) and is only shown in the DAG graph if you set `SETUP_TEARDOWN_SNOWFLAKE_CUSTOM_XCOM_BACKEND` to `True`. See also Step 3.3.

   * `load_file`: Loads the data from the `ski_dataset.csv` file into the Snowflake table `MY_SNOWFLAKE_TABLE` using the [`load_file` operator](https://astro-sdk-python.readthedocs.io/en/stable/astro/sql/operators/load_file.html) from the Astro Python SDK.

   * `create_model_registry`: Creates a model registry in Snowpark using the [Snowpark ML package](https://docs.snowflake.com/en/developer-guide/snowpark-ml/index). Since the task is defined by the `@task.snowflake_python` decorator, the snowpark session is automatically instantiated from provided connection ID.

   * `transform_table_step_one`: Transforms the data in the Snowflake table using Snowpark syntax to filter to only include rows of skiers that ordered the beverages we are interested in. Computation of this task runs within Snowpark. The resulting table is written to [XCom](/docs/learn/airflow-passing-data-between-tasks) as a pandas DataFrame.

   * `transform_table_step_two`: Transforms the pandas DataFrame created by the upstream task to filter only for serious skiers (those who skied at least one hour that day).
     This task uses the `@task.snowpark_ext_python` decorator, running the code in the Snowpark virtual environment created in Step 1. The binary provided to the `python` parameter of the decorator determines which virtual environment to run a task in. The `@task.snowpark_ext_python` decorator works analogously to the [@task.`external_python` decorator](/docs/learn/airflow-isolated-environments), except the code is executed within Snowpark's compute.

   * `train_beverage_classifier`: Trains a [Snowpark Logistic Regression model](https://docs.snowflake.com/en/developer-guide/snowpark-ml/reference/latest/api/modeling/snowflake.ml.modeling.linear_model.LogisticRegression) on the dataset, saves the model to the model registry, and creates predictions from a test dataset. This task uses the `@task.snowpark_virtualenv` decorator to run the code in a newly created virtual environment within Snowpark's compute. The `requirements` parameter of the decorator specifies the packages to install in the virtual environment. The model predictions are saved to XCom as a pandas DataFrame.

   * `plot_metrics`: Creates a plot of the model performance metrics and saves it to the `include` directory. This task runs in the Airflow environment using the `@task` decorator.

   * (Optional) `cleanup_xcom_table`: Cleans up the Snowflake custom XCom backend by dropping the `XCOM_TABLE` and `XCOM_STAGE`. This task is a [teardown task](/docs/learn/airflow-setup-teardown) and is only shown in the DAG graph if you set `SETUP_TEARDOWN_SNOWFLAKE_CUSTOM_XCOM_BACKEND` to `True`. See also Step 3.3.

3. (Optional) This DAG has two optional features you can enable.

   * If you want to use [setup/ teardown tasks](/docs/learn/airflow-setup-teardown) to create and clean up a Snowflake custom XCom backend for this DAG, set `SETUP_TEARDOWN_SNOWFLAKE_CUSTOM_XCOM_BACKEND` to `True`. This setting adds the `create_snowflake_objects` and `cleanup_xcom_table` tasks to your DAG and creates a setup/ teardown workflow. Note that your Snowflake account needs to have `ACCOUNTADMIN` privileges to perform the operations in the `create_snowflake_objects` task and you need to define the environment variables described in [Step 1.8](#step-1-configure-your-astro-project) to enable the custom XCom backend.

   * If you want to use a [Snowpark-optimized warehouse](https://docs.snowflake.com/en/user-guide/warehouses-snowpark-optimized) for model training, set the `USE_SNOWPARK_WH` variable to `True` and provide your warehouse names to `MY_SNOWPARK_WAREHOUSE` and `MY_SNOWFLAKE_REGULAR_WAREHOUSE`. If the `create_snowflake_objects` task is enabled, it creates the `MY_SNOWPARK_WAREHOUSE` warehouse. Otherwise, you need to create the warehouse manually before running the DAG.

<Info>
  While this tutorial DAG uses a small dataset where model training can be accomplished using the standard Snowflake warehouse, Astronomer recommends using a Snowpark-optimized warehouse for model training in production.
</Info>

## Step 4: Run your DAG

1. Run `astro dev start` in your Astro project to start up Airflow and open the Airflow UI at `localhost:8080`.

2. In the Airflow UI, run the `airflow_with_snowpark_tutorial` DAG by clicking the play button.

<details>
  <summary>Standard</summary>

  <Frame>
    <img alt="Screenshot of the Airflow UI showing the airflow_with_snowpark_tutorial DAG having completed successfully in the Grid view with the Graph tab selected." />
  </Frame>
</details>

<details>
  <summary>Setup Teardown</summary>

  <Frame>
    <img alt="Screenshot of the Airflow UI in the Grid view with the Graph tab selected, showing the successfully completed airflow_with_snowpark_tutorial DAG. This screenshot displays the version of the DAG where SETUP_TEARDOWN_SNOWFLAKE_CUSTOM_XCOM_BACKEND is set to true, creating an additional setup/ teardown workflow." />
  </Frame>
</details>

3. In the Snowflake UI, view the model registry to see the model that was created by the DAG. In a production context, you can pull a specific model from the registry to run predictions on new data.

   <Frame>
     <img alt="Screenshot of the Snowflake UI showing the model registry containing one model." />
   </Frame>

4. Navigate to your `include` directory to view the `metrics.png` image, which contains the model performance metrics shown at the start of this tutorial.

## Conclusion

Congratulations! You trained a classification model in Snowpark using Airflow. This pipeline shows the three main options to run code in Snowpark using Airflow decorators:

* `@task.snowpark_python` runs your code in a standard Snowpark environment. Use this decorator if you need to run code in Snowpark that doesn't require any additional packages that aren't preinstalled in a standard Snowpark environment. The corresponding traditional operator is the `SnowparkPythonOperator`.
* `@task.snowpark_ext_python` runs your code in a pre-existing virtual environment within Snowpark. Use this decorator when you want to reuse virtual environments in different tasks in the same Airflow instances, or your virtual environment takes a long time to build. The corresponding traditional operator is the `SnowparkExternalPythonOperator`.
* `@task.snowpark_virtualenv` runs your code in a virtual environment in Snowpark that is created at runtime for that specific task. Use this decorator when you want to tailor a virtual environment to a task and don't need to reuse it. The corresponding traditional operator is the `SnowparkVirtualenvOperator`.

Corresponding traditional operators are available:

* [`SnowparkPythonOperator`](https://registry.astronomer.io/providers/astro-provider-snowflake/versions/latest/modules/SnowparkPythonOperator), which you can import using `from snowpark_provider.operators.snowpark import SnowparkPythonOperator`.
* [`SnowparkExternalPythonOperator`](https://airflow.apache.org/registry/providers/standard#standard-python-ExternalPythonOperator), available using `from snowpark_provider.operators.snowpark import SnowparkExternalPythonOperator`.
* [`SnowparkVirtualenvOperator`](https://registry.astronomer.io/providers/astro-provider-snowflake/versions/latest/modules/SnowparkVirtualenvOperator), with the import `from snowpark_provider.operators.snowpark import SnowparkVirtualenvOperator`.


# Orchestrate Weaviate operations with Apache Airflow
Source: https://astronomer.io/docs/learn/airflow-weaviate

Learn how to integrate Weaviate and Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Weaviate](https://weaviate.io/developers/weaviate) is an open source vector database, which store high-dimensional embeddings of objects like text, images, audio or video. The [Weaviate Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-weaviate/stable/index.html) offers modules to easily integrate Weaviate with Airflow.

In this tutorial you'll use Airflow to ingest movie descriptions into Weaviate, use Weaviate's automatic vectorization to create vectors for the descriptions, and query Weaviate for movies that are thematically close to user-provided concepts.

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Webinar: [Modern Infrastructure for World Class AI Applications](https://www.astronomer.io/events/webinars/modern-infrastructure-for-world-class-ai-applications-video/).
</Tip>

## Why use Airflow with Weaviate?

Weaviate allows you to store objects alongside their vector embeddings and to query these objects based on their similarity. Vector embeddings are key components of many modern machine learning models such as [LLMs](https://en.wikipedia.org/wiki/Large_language_model) or [ResNet](https://arxiv.org/abs/1512.03385).

Integrating Weaviate with Airflow into one end-to-end machine learning pipeline allows you to:

* Use Airflow's [data-driven scheduling](/docs/learn/airflow-datasets) to run operations on Weaviate based on upstream events in your data ecosystem, such as when a new model is trained or a new dataset is available.
* Run dynamic queries based on upstream events in your data ecosystem or user input via [Airflow params](/docs/learn/airflow-params) against Weaviate to retrieve objects with similar vectors.
* Add Airflow features like [retries](/docs/learn/rerunning-dags#automatically-retry-tasks) and [alerts](/docs/learn/error-notifications-in-airflow) to your Weaviate operations.

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* The basics of Weaviate. See [Weaviate Introduction](https://weaviate.io/developers/weaviate).
* Airflow fundamentals, such as writing DAGs and defining tasks. See [Get started with Apache Airflow](/docs/learn/get-started-with-airflow).
* Airflow decorators. [Introduction to the TaskFlow API and Airflow decorators](/docs/learn/airflow-decorators).
* Airflow hooks. See [Hooks 101](/docs/learn/what-is-a-hook).
* Airflow connections. See [Managing your Connections in Apache Airflow](/docs/learn/connections).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/get-started-cli).
* (Optional) An OpenAI API key of at least [tier 1](https://platform.openai.com/docs/guides/rate-limits/usage-tiers) if you want to use OpenAI for vectorization. The tutorial can be completed using local vectorization with `text2vec-transformers` if you don't have an OpenAI API key.

This tutorial uses a local Weaviate instance created as a Docker container. You do not need to install the Weaviate client locally.

<Info>
  The example code from this tutorial is also available on [GitHub](https://github.com/astronomer/airflow-weaviate-tutorial).
</Info>

## Step 1: Configure your Astro project

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-weaviate-tutorial && cd astro-weaviate-tutorial
   $ astro dev init
   ```

2. Add `build-essential` to your `packages.txt` file to be able to install the [Weaviate Airflow Provider](https://airflow.apache.org/docs/apache-airflow-providers-weaviate/stable/index.html).

   ```text wrap theme={null}
   build-essential
   ```

3. Add the following two packages to your `requirements.txt` file to install the [Weaviate Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers-weaviate/stable/index.html) and the [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) in your Astro project:

   ```text wrap theme={null}
   apache-airflow-providers-weaviate==2.0.0
   weaviate-client==4.7.1
   ```

4. This tutorial uses a local Weaviate instance and a [text2vec-transformer model](https://hub.docker.com/r/semitechnologies/transformers-inference/), with each running in a Docker container. To add additional containers to your Astro project, create a new file in your project's root directory called `docker-compose.override.yml` and add the following:

   ```yaml expandable wrap theme={null}
   version: '3.1'
   services:
     weaviate:
       image: cr.weaviate.io/semitechnologies/weaviate:1.25.6
       command: "--host 0.0.0.0 --port '8081' --scheme http"
       ports:
         - "8081:8081"
         - "50051:50051"
       volumes:
         - ./include/weaviate/backup:/var/lib/weaviate/backup
       environment:
         QUERY_DEFAULTS_LIMIT: 25
         AUTHENTICATION_APIKEY_ENABLED: 'true'
         AUTHENTICATION_APIKEY_ALLOWED_KEYS: 'readonlykey,adminkey'
         AUTHENTICATION_APIKEY_USERS: 'jane@doe.com,john@doe.com'
         PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
         DEFAULT_VECTORIZER_MODULE: 'text2vec-openai'
         ENABLE_MODULES: 'text2vec-openai, backup-filesystem, qna-openai, text2vec-transformers'
         BACKUP_FILESYSTEM_PATH: '/var/lib/weaviate/backup'
         CLUSTER_HOSTNAME: 'node1'
         TRANSFORMERS_INFERENCE_API: 'http://t2v-transformers:8080'
       networks:
         - airflow
     t2v-transformers:
       image: semitechnologies/transformers-inference:sentence-transformers-multi-qa-MiniLM-L6-cos-v1
       environment:
         ENABLE_CUDA: 0 # set to 1 to enable
       ports:
         - 8082:8080
       networks:
         - airflow
   ```

5. To create an [Airflow connection](/docs/learn/connections) to the local Weaviate instance, add the following environment variable to your `.env` file. You only need to provide an `X-OpenAI-Api-Key` if you plan on using the OpenAI API for vectorization. To create a connection to your Weaviate Cloud instance, refer to the commented connection version below.

   ```text wrap theme={null}
   ## Local Weaviate connection
   AIRFLOW_CONN_WEAVIATE_DEFAULT='{
       "conn_type":"weaviate",
       "host":"weaviate",
       "port":"8081",
       "extra":{
           "token":"adminkey",
           "additional_headers":{"X-Openai-Api-Key":"<YOUR OPENAI API KEY>"},
           "grpc_port":"50051",
           "grpc_host":"weaviate",
           "grpc_secure":"False",
           "http_secure":"False"
       }
   }'

   ## The Weaviate Cloud connection uses the following pattern:
   # AIRFLOW_CONN_WEAVIATE_DEFAULT='{
   #     "conn_type":"weaviate",
   #     "host":"<YOUR HOST>.gcp.weaviate.cloud",
   #     "port":"8081",
   #     "extra":{
   #         "token":"<YOUR WEAVIATE KEY>",
   #         "additional_headers":{"X-Openai-Api-Key":"<YOUR OPENAI API KEY>"},
   #         "grpc_port":"443",
   #         "grpc_host":"grpc-<YOUR HOST>.gcp.weaviate.cloud",
   #         "grpc_secure":"True",
   #         "http_secure":"True"
   #     }
   # }'
   ```

<Tip>
  See the Weaviate documentation on [environment variables](https://weaviate.io/developers/weaviate/config-refs/env-vars), [models](https://weaviate.io/developers/weaviate/model-providers), and [client instantiation](https://weaviate.io/developers/weaviate/client-libraries/python#instantiate-a-client) for more information on configuring a Weaviate instance and connection.
</Tip>

## Step 2: Add your data

The DAG in this tutorial runs a query on vectorized movie descriptions from [IMDB](https://www.imdb.com/). If you run the project locally, Astronomer recommends testing the pipeline with a small subset of the data. If you use a remote vectorizer like `text2vec-openai`, you can use larger parts of the [full dataset](https://github.com/astronomer/learn-tutorials-data/blob/main/movie_descriptions.txt).

Create a new file called `movie_data.txt` in the `include` directory, then copy and paste the following information:

```text wrap theme={null}
1 ::: Arrival (2016) ::: sci-fi ::: A linguist works with the military to communicate with alien lifeforms after twelve mysterious spacecraft appear around the world.
2 ::: Don't Look Up (2021) ::: drama ::: Two low-level astronomers must go on a giant media tour to warn humankind of an approaching comet that will destroy planet Earth.
3 ::: Primer (2004) ::: sci-fi ::: Four friends/fledgling entrepreneurs, knowing that there's something bigger and more innovative than the different error-checking devices they've built, wrestle over their new invention.
4 ::: Serenity (2005) ::: sci-fi ::: The crew of the ship Serenity try to evade an assassin sent to recapture telepath River.
5 ::: Upstream Colour (2013) ::: romance ::: A man and woman are drawn together, entangled in the life cycle of an ageless organism. Identity becomes an illusion as they struggle to assemble the loose fragments of wrecked lives.
6 ::: The Matrix (1999) ::: sci-fi ::: When a beautiful stranger leads computer hacker Neo to a forbidding underworld, he discovers the shocking truth--the life he knows is the elaborate deception of an evil cyber-intelligence.
7 ::: Inception (2010) ::: sci-fi ::: A thief who steals corporate secrets through the use of dream-sharing technology is given the inverse task of planting an idea into the mind of a C.E.O., but his tragic past may doom the project and his team to disaster.
```

## Step 3: Create your DAG

1. In your `dags` folder, create a file called `query_movie_vectors.py`.

2. Copy the following code into the file. If you want to use `text2vec-openai` for vectorization, change the `VECTORIZER` variable to `text2vec-openai` and make sure you provide an OpenAI API key in the `AIRFLOW_CONN_WEAVIATE_DEFAULT` in your `.env` file.

   ```python expandable wrap theme={null}
   """
   ## Use the Airflow Weaviate Provider to generate and query vectors for movie descriptions

   This DAG runs a simple MLOps pipeline that uses the Weaviate Provider to import
   movie descriptions, generate vectors for them, and query the vectors for movies based on
   concept descriptions.
   """

   from airflow.decorators import dag, task
   from airflow.models.param import Param
   from airflow.operators.empty import EmptyOperator
   from airflow.models.baseoperator import chain
   from airflow.providers.weaviate.hooks.weaviate import WeaviateHook
   from airflow.providers.weaviate.operators.weaviate import WeaviateIngestOperator
   from weaviate.util import generate_uuid5
   import weaviate.classes.config as wvcc
   from pendulum import datetime
   import logging
   import re

   t_log = logging.getLogger("airflow.task")

   WEAVIATE_USER_CONN_ID = "weaviate_default"
   TEXT_FILE_PATH = "include/movie_data.txt"
   # the base collection name is used to create a unique collection name for the vectorizer
   # note that it is best practice to capitalize the first letter of the collection name
   COLLECTION_NAME = "Movie"

   # set the vectorizer to text2vec-openai if you want to use the openai model
   # note that using the OpenAI vectorizer requires a valid API key in the
   # AIRFLOW_CONN_WEAVIATE_DEFAULT connection.
   # If you want to use a different vectorizer model
   # (https://weaviate.io/developers/weaviate/model-providers)
   # make sure to also add it to the weaviate configuration's `ENABLE_MODULES` list
   # for example in the docker-compose.override.yml file
   VECTORIZER = wvcc.Configure.Vectorizer.text2vec_transformers()
   # VECTORIZER = wvcc.Configure.Vectorizer.text2vec_openai(model="ada")


   @dag(
       start_date=datetime(2023, 9, 1),
       schedule=None,
       catchup=False,
       tags=["weaviate"],
       params={
           "movie_concepts": Param(
               ["innovation", "friends"],
               type="array",
               description=(
                   "What kind of movie do you want to watch today?"
                   + " Add one concept per line."
               ),
           ),
       },
   )
   def query_movie_vectors():
       @task.branch
       def check_for_collection(conn_id: str, collection_name: str) -> bool:
           "Check if the provided collection already exists and decide on the next step."
           # connect to Weaviate using the Airflow connection `conn_id`
           hook = WeaviateHook(conn_id)

           # check if the collection exists in the Weaviate database
           collection = hook.get_conn().collections.exists(collection_name)

           if collection:
               t_log.info(f"Collection {collection_name} already exists.")
               return "collection_exists"
           else:
               t_log.info(f"collection {collection_name} does not exist yet.")
               return "create_collection"

       @task
       def create_collection(conn_id: str, collection_name: str, vectorizer: str):
           "Create a collection with the provided name and vectorizer."
           hook = WeaviateHook(conn_id)

           hook.create_collection(name=collection_name, vectorizer_config=vectorizer)

       collection_exists = EmptyOperator(task_id="collection_exists")

       def import_data_func(text_file_path: str, collection_name: str):
           "Read the text file and create a list of dicts for ingestion to Weaviate."
           with open(text_file_path, "r") as f:
               lines = f.readlines()

               num_skipped_lines = 0
               data = []
               for line in lines:
                   parts = line.split(":::")
                   title_year = parts[1].strip()
                   match = re.match(r"(.+) \((\d{4})\)", title_year)
                   try:
                       title, year = match.groups()
                       year = int(year)
                   # skip malformed lines
                   except:
                       num_skipped_lines += 1
                       continue

                   genre = parts[2].strip()
                   description = parts[3].strip()

                   data.append(
                       {
                           "movie_id": generate_uuid5(
                               identifier=[title, year, genre, description],
                               namespace=collection_name,
                           ),
                           "title": title,
                           "year": year,
                           "genre": genre,
                           "description": description,
                       }
                   )

               print(
                   f"Created a list with {len(data)} elements while skipping {num_skipped_lines} lines."
               )
               return data

       import_data = WeaviateIngestOperator(
           task_id="import_data",
           conn_id=WEAVIATE_USER_CONN_ID,
           collection_name=COLLECTION_NAME,
           input_json=import_data_func(
               text_file_path=TEXT_FILE_PATH, collection_name=COLLECTION_NAME
           ),
           trigger_rule="none_failed",
       )

       @task
       def query_embeddings(weaviate_conn_id: str, collection_name: str, **context):
           "Query the Weaviate instance for movies based on the provided concepts."
           hook = WeaviateHook(weaviate_conn_id)
           movie_concepts = context["params"]["movie_concepts"]

           my_movie_collection = hook.get_collection(collection_name)

           movie = my_movie_collection.query.near_text(
               query=movie_concepts,
               return_properties=["title", "year", "genre", "description"],
               limit=1,
           )

           movie_title = movie.objects[0].properties["title"]
           movie_year = movie.objects[0].properties["year"]
           movie_genre = movie.objects[0].properties["genre"]
           movie_description = movie.objects[0].properties["description"]

           t_log.info(f"You should watch {movie_title}!")
           t_log.info(
               f"It was filmed in {int(movie_year)} and belongs to the {movie_genre} genre."
           )
           t_log.info(f"Description: {movie_description}")

       chain(
           check_for_collection(
               conn_id=WEAVIATE_USER_CONN_ID, collection_name=COLLECTION_NAME
           ),
           [
               create_collection(
                   conn_id=WEAVIATE_USER_CONN_ID,
                   collection_name=COLLECTION_NAME,
                   vectorizer=VECTORIZER,
               ),
               collection_exists,
           ],
           import_data,
           query_embeddings(
               weaviate_conn_id=WEAVIATE_USER_CONN_ID, collection_name=COLLECTION_NAME
           ),
       )


   query_movie_vectors()
   ```

   This DAG consists of five tasks to make a simple ML orchestration pipeline.

   * The `check_for_collection` task uses the [`WeaviateHook`](https://airflow.apache.org/docs/apache-airflow-providers-weaviate/stable/_api/airflow/providers/weaviate/hooks/weaviate/index.html) to check if a collection of the name `COLLECTION_NAME` already exists in your Weaviate instance. The task is defined using the [`@task.branch` decorator](/docs/learn/airflow-branch-operator#@task-branch-branchpythonoperator) and returns the id of the task to run next based on whether the collection of interest exists. If the collection exists, the DAG runs the empty `collection_exists` task. If the collection doesn't exist, the DAG runs the `create_collection` task.
   * The `create_collection` task uses the `WeaviateHook` to create a collection with the `COLLECTION_NAME` and specified `VECTORIZER` in your Weaviate instance.
   * The `import_data` task is defined using the [`WeaviateIngestOperator`](https://airflow.apache.org/docs/apache-airflow-providers-weaviate/stable/operators/weaviate.html) and ingests the data into Weaviate. You can run any Python code on the data before ingesting it into Weaviate by providing a callable to the `input_json` parameter. This makes it possible to create your own embeddings or complete other transformations before ingesting the data. In this example we use automatic schema inference and vector creation by Weaviate.
   * The `query_embeddings` task uses the `WeaviateHook` to connect to the Weaviate instance and run a query. The query returns the most similar movie to the concepts provided by the user when running the DAG in the next step.

## Step 4: Run your DAG

1. Run `astro dev start` in your Astro project to start Airflow and open the Airflow UI at `localhost:8080`.

2. In the Airflow UI, run the `query_movie_vectors` DAG by clicking the play button. Then, provide [Airflow params](/docs/learn/airflow-params) for `movie_concepts`.

   Note that if you are running the project locally on a larger dataset, the `import_data` task might take a longer time to complete because Weaviate generates the vector embeddings in this task.

   <Frame>
     <img alt="Screenshot of the Airflow UI showing the successful completion of the query_movie_vectors DAG in the Grid view with the Graph tab selected. Since this was the first run of the DAG, the schema had to be newly created. The schema creation was enabled when the branching task branch_create_schema selected the downstream create_schema task to run." />
   </Frame>

3. View your movie suggestion in the task logs of the `query_embeddings` task:

   ```text wrap theme={null}
   [2024-08-15, 13:34:10 UTC] {query_movie_vectors.py:155} INFO - You should watch Primer!
   [2024-08-15, 13:34:10 UTC] {query_movie_vectors.py:156} INFO - It was filmed in 2004 and belongs to the sci-fi genre.
   [2024-08-15, 13:34:10 UTC] {query_movie_vectors.py:159} INFO - Description: Four friends/fledgling entrepreneurs, knowing that there's something bigger and more innovative than the different error-checking devices they've built, wrestle over their new invention.
   ```

## Conclusion

Congratulations! You used Airflow and Weaviate to get your next movie suggestion!


# Manage your ML models with Weights and Biases and Airflow
Source: https://astronomer.io/docs/learn/airflow-weights-and-biases

Learn how to use Airflow and Weights and Biases to manage and visualize your ML model lifecycle.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Weights and Biases](https://wandb.ai/site) (W\&B) is a machine learning platform for model management that includes features like experiment tracking, dataset versioning, and model performance evaluation and visualization. Using W\&B with Airflow gives you a powerful ML orchestration stack with first-class features for building, training, and managing your models.

In this tutorial, you'll learn how to create an Airflow DAG that completes feature engineering, model training, and predictions with the Astro Python SDK and scikit-learn, and registers the model with W\&B for evaluation and visualization.

<Info>
  This tutorial was developed in partnership with Weights and Biases. For resources on implementing other use cases with W\&B, see [Tutorials](https://wandb.ai/site/tutorials).
</Info>

## Time to complete

This tutorial takes approximately one hour to complete.

## Assumed knowledge

To get the most out of this tutorial, you should be familiar with:

* Airflow operators. See [Operators 101](/docs/learn/what-is-an-operator).
* Weights and Biases. See [What is Weights and Biases?](https://docs.wandb.ai/?_gl=1*i7pmr7*_ga*MTI3ODk4OTUzNy4xNjc5Njc2MzE5*_ga_JH1SJHJQXJ*MTY4MTI0ODQ2OS42LjEuMTY4MTI0ODUxMS4xOC4wLjA.).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* A [Weights and Biases](https://wandb.ai/site) account. Personal accounts are available for free.

## Quickstart

If you have a GitHub account, you can get started quickly by cloning the demo repository. For more detailed instructions for setting up the project, start with [Step 1](#step-1-configure-your-astro-project).

1. Clone the demo repository:

   ```sh wrap theme={null}
   git clone https://github.com/astronomer/airflow-wandb-demo
   cd airflow-wandb-demo
   ```

2. Update the `.env` file with your `WANDB_API_KEY`.

3. Start Airflow by running:

   ```sh wrap theme={null}
   astro dev start
   ```

4. Continue with [Step 7](#step-7-run-your-dag-and-view-results) below.

## Step 1: Configure your Astro project

Use the Astro CLI to create and run an Airflow project locally.

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-wandb-tutorial && cd astro-wandb-tutorial
   $ astro dev init
   ```

2. Add the following line to the `requirements.txt` file of your Astro project:

   ```text wrap theme={null}
   astro-sdk-python[postgres]==1.5.3
   wandb==0.14.0
   pandas==1.5.3
   numpy==1.24.2
   scikit-learn==1.2.2
   ```

   This installs the packages needed to transform the data and run feature engineering, model training, and predictions.

## Step 2: Prepare the data

This tutorial creates a model that classifies churn risk based on customer data.

1. Create a subfolder called `data` in your Astro project `include` folder.
2. Download the demo CSV files from [this GitHub directory](https://github.com/astronomer/airflow-wandb-demo/tree/main/include/data).
3. Save the downloaded CSV files in the `include/data` folder. You should have 5 files in total.

## Step 3: Create your SQL transformation scripts

Before feature engineering and training, the data needs to be transformed. This tutorial uses the Astro Python SDK `transform_file` function to complete several transformations using SQL.

1. Create a file in your `include` folder called `customer_churn_month.sql` and copy the following code into the file.

   ```sql expandable wrap theme={null}
   with subscription_periods as (
       select subscription_id, 
           customer_id, 
           cast(start_date as date) as start_date, 
           cast(end_date as date) as end_date, 
           monthly_amount 
           from {{subscription_periods}}
   ),

   months as (
       select cast(date_month as date) as date_month from {{util_months}}
   ),

   customers as (
       select
           customer_id,
           date_trunc('month', min(start_date)) as date_month_start,
           date_trunc('month', max(end_date)) as date_month_end
       from subscription_periods
       group by 1
   ),

   customer_months as (
       select
           customers.customer_id,
           months.date_month
       from customers
       inner join months
           on  months.date_month >= customers.date_month_start
           and months.date_month < customers.date_month_end
   ),

   joined as (
       select
           customer_months.date_month,
           customer_months.customer_id,
           coalesce(subscription_periods.monthly_amount, 0) as mrr
       from customer_months
       left join subscription_periods
           on customer_months.customer_id = subscription_periods.customer_id
           and customer_months.date_month >= subscription_periods.start_date
           and (customer_months.date_month < subscription_periods.end_date
                   or subscription_periods.end_date is null)
   ),

   customer_revenue_by_month as (
       select
           date_month,
           customer_id,
           mrr,
           mrr > 0 as is_active,
           min(case when mrr > 0 then date_month end) over (
               partition by customer_id
           ) as first_active_month,

           max(case when mrr > 0 then date_month end) over (
               partition by customer_id
           ) as last_active_month,

           case
           when min(case when mrr > 0 then date_month end) over (
               partition by customer_id
           ) = date_month then true
           else false end as is_first_month,
           case
           when max(case when mrr > 0 then date_month end) over (
               partition by customer_id
           ) = date_month then true
           else false end as is_last_month
       from joined
   ),

   joined1 as (

       select
           date_month + interval '1 month' as date_month,
           customer_id,
           0::float as mrr,
           false as is_active,
           first_active_month,
           last_active_month,
           false as is_first_month,
           false as is_last_month

       from customer_revenue_by_month

       where is_last_month

   )

   select * from joined1;
   ```

2. Create another file in your `include` folder called `customers.sql` and copy the following code into the file.

   ```sql expandable wrap theme={null}
   with
   customers as (

       select *
       from {{customers_table}}

   ),

   orders as (

       select *
       from {{orders_table}}

   ),

   payments as (

       select *
       from {{payments_table}}

   ),

   customer_orders as (

       select
       customer_id,
       cast(min(order_date) as date) as first_order,
       cast(max(order_date) as date) as most_recent_order,
       count(order_id) as number_of_orders
       from orders

       group by customer_id

   ),

   customer_payments as (

       select
       orders.customer_id,
       sum(amount / 100) as total_amount

       from payments

       left join orders on payments.order_id = orders.order_id

       group by orders.customer_id

   ),

   final as (

       select
       customers.customer_id,
       customers.first_name,
       customers.last_name,
       customer_orders.first_order,
       customer_orders.most_recent_order,
       customer_orders.number_of_orders,
       customer_payments.total_amount as customer_lifetime_value

       from customers

       left join customer_orders on customers.customer_id = customer_orders.customer_id

       left join customer_payments on customers.customer_id = customer_payments.customer_id

   )

   select
   *
   from final
   ```

## Step 4: Create a W\&B API key

In your W\&B account, create an API key that you will use to connect Airflow to W\&B. You can create a key by going to the [Authorize](https://wandb.ai/authorize) page or your user settings.

## Step 5: Set up your connections and environment variables

You'll use environment variables to create Airflow connections to Snowflake and W\&B, as well as to configure the Astro Python SDK.

1. Open the `.env` file in your Astro project and paste the following code.

   ```text wrap theme={null}
   WANDB_API_KEY='<your-wandb-api-key>'
   AIRFLOW_CONN_POSTGRES_DEFAULT='postgresql://postgres:postgres@host.docker.internal:5432/postgres?options=-csearch_path%3Dtmp_astro'
   ```

2. Replace `<your-wandb-api-key>` with the API key you created in [Step 4](#step-4-create-a-w\&b-api-key). No changes are needed for the `AIRFLOW_CONN_POSTGRES_DEFAULT` environment variable.

## Step 6: Create your DAG

1. Create a file in your Astro project `dags` folder called `customer_analytics.py` and copy the following code into the file:

   ```python expandable wrap theme={null}
   from datetime import datetime
   import os

   from astro import sql as aql
   from astro.files import File
   from astro.sql.table import Table
   from airflow.decorators import dag, task_group

   import pandas as pd
   import numpy as np
   from sklearn.model_selection import train_test_split
   from sklearn.ensemble import RandomForestClassifier
   import tempfile
   import pickle
   from pathlib import Path

   import wandb
   from wandb.sklearn import plot_precision_recall, plot_feature_importances
   from wandb.sklearn import plot_class_proportions, plot_learning_curve, plot_roc

   _POSTGRES_CONN = "postgres_default"
   wandb_project = "demo"
   wandb_team = "astro-demos"
   local_data_dir = "include/data"
   sources = ["subscription_periods", "util_months", "customers", "orders", "payments"]


   @dag(schedule=None, start_date=datetime(2023, 1, 1), catchup=False)
   def customer_analytics():
       @task_group()
       def extract_and_load(sources: list) -> dict:
           for source in sources:
               aql.load_file(
                   task_id=f"load_{source}",
                   input_file=File(f"{local_data_dir}/{source}.csv"),
                   output_table=Table(
                       name=f"STG_{source.upper()}", conn_id=_POSTGRES_CONN
                   ),
                   if_exists="replace",
               )

       @task_group()
       def transform():
           aql.transform_file(
               task_id="transform_churn",
               file_path=f"{Path(__file__).parent.as_posix()}/../include/customer_churn_month.sql",
               parameters={
                   "subscription_periods": Table(
                       name="STG_SUBSCRIPTION_PERIODS", conn_id=_POSTGRES_CONN
                   ),
                   "util_months": Table(name="STG_UTIL_MONTHS", conn_id=_POSTGRES_CONN),
               },
               op_kwargs={
                   "output_table": Table(
                       name="CUSTOMER_CHURN_MONTH", conn_id=_POSTGRES_CONN
                   )
               },
           )

           aql.transform_file(
               task_id="transform_customers",
               file_path=f"{Path(__file__).parent.as_posix()}/../include/customers.sql",
               parameters={
                   "customers_table": Table(name="STG_CUSTOMERS", conn_id=_POSTGRES_CONN),
                   "orders_table": Table(name="STG_ORDERS", conn_id=_POSTGRES_CONN),
                   "payments_table": Table(name="STG_PAYMENTS", conn_id=_POSTGRES_CONN),
               },
               op_kwargs={"output_table": Table(name="CUSTOMERS", conn_id=_POSTGRES_CONN)},
           )

       @aql.dataframe()
       def features(customer_df: pd.DataFrame, churned_df: pd.DataFrame) -> pd.DataFrame:
           customer_df["customer_id"] = customer_df["customer_id"].apply(str)
           customer_df.set_index("customer_id", inplace=True)

           churned_df["customer_id"] = churned_df["customer_id"].apply(str)
           churned_df.set_index("customer_id", inplace=True)
           churned_df["is_active"] = churned_df["is_active"].astype(int).replace(0, 1)

           df = (
               customer_df[["number_of_orders", "customer_lifetime_value"]]
               .join(churned_df[["is_active"]], how="left")
               .fillna(0)
               .reset_index()
           )  # inplace=True)

           return df

       @aql.dataframe()
       def train(df: pd.DataFrame) -> dict:
           features = ["number_of_orders", "customer_lifetime_value"]
           target = ["is_active"]

           test_size = 0.3
           X_train, X_test, y_train, y_test = train_test_split(
               df[features], df[target], test_size=test_size, random_state=1883
           )
           X_train = np.array(X_train.values.tolist())
           y_train = np.array(y_train.values.tolist()).reshape(
               len(y_train),
           )
           y_train = y_train.reshape(
               len(y_train),
           )
           X_test = np.array(X_test.values.tolist())
           y_test = np.array(y_test.values.tolist())
           y_test = y_test.reshape(
               len(y_test),
           )

           model = RandomForestClassifier()
           _ = model.fit(X_train, y_train)
           model_params = model.get_params()

           y_pred = model.predict(X_test)
           y_probas = model.predict_proba(X_test)
           importances = model.feature_importances_
           indices = np.argsort(importances)[::-1]

           wandb.login()
           run = wandb.init(
               project=wandb_project,
               config=model_params,
               entity=wandb_team,
               group="wandb-demo",
               name="jaffle_churn",
               dir="include",
               mode="online",
           )

           wandb.config.update(
               {"test_size": test_size, "train_len": len(X_train), "test_len": len(X_test)}
           )
           plot_class_proportions(y_train, y_test, ["not_churned", "churned"])
           plot_learning_curve(model, X_train, y_train)
           plot_roc(y_test, y_probas, ["not_churned", "churned"])
           plot_precision_recall(y_test, y_probas, ["not_churned", "churned"])
           plot_feature_importances(model)

           model_artifact_name = "churn_classifier"

           with tempfile.NamedTemporaryFile(delete=False) as tf:
               pickle.dump(model, tf)
               tf.close()
               artifact = wandb.Artifact(model_artifact_name, type="model")
               artifact.add_file(local_path=tf.name, name=model_artifact_name)
               wandb.log_artifact(artifact)
               os.remove(tf.name)

           wandb.finish()

           return {"run_id": run.id, "artifact_name": model_artifact_name}

       @aql.dataframe()
       def predict(model_info: dict, customer_df: pd.DataFrame) -> pd.DataFrame:
           wandb.login()
           run = wandb.init(
               project=wandb_project,
               entity=wandb_team,
               group="wandb-demo",
               name="jaffle_churn",
               dir="include",
               resume="must",
               id=model_info["run_id"],
           )

           customer_df.fillna(0, inplace=True)

           features = ["number_of_orders", "customer_lifetime_value"]

           artifact = run.use_artifact(
               f"{model_info['artifact_name']}:latest", type="model"
           )

           with tempfile.TemporaryDirectory() as td:
               with open(artifact.file(td), "rb") as mf:
                   model = pickle.load(mf)
                   customer_df["PRED"] = model.predict_proba(
                       np.array(customer_df[features].values.tolist())
                   )[:, 0]

           wandb.finish()

           customer_df.reset_index(inplace=True)

           return customer_df

       _extract_and_load = extract_and_load(sources)

       _transformed = transform()

       _features = features(
           customer_df=Table(name="customers", conn_id=_POSTGRES_CONN),
           churned_df=Table(name="customer_churn_month", conn_id=_POSTGRES_CONN),
       )

       _model_info = train(df=_features)

       _predict_churn = predict(
           model_info=_model_info,
           customer_df=Table(name="customers", conn_id=_POSTGRES_CONN),
           output_table=Table(name=f"pred_churn", conn_id=_POSTGRES_CONN),
       )

       _extract_and_load >> _transformed >> _features


   customer_analytics()
   ```

   This DAG completes the following steps:

   * The `extract_and_load` task group contains one task for each CSV in your `include/data` folder that uses the Astro Python SDK `load_file` function to load the data to Postgres.
   * The `transform` task group contains two tasks that transform the data using the Astro Python SDK `transform_file` function and the SQL scripts in your `include` folder.
   * The `features` task is a Python function implemented with the Astro Python SDK `@dataframe` decorator that uses Pandas to create the features needed for the model.
   * The `train` task is a Python function implemented with the Astro Python SDK `@dataframe` decorator that uses scikit-learn to train a Random Forest classifier model and push the results to W\&B.
   * The `predict` task pulls the model from W\&B in order to make predictions and stores them in Postgres.

2. Run the following command to start your project in a local environment:

   ```sh wrap theme={null}
   astro dev start
   ```

## Step 7: Run your DAG and view results

1. Open the [Airflow UI](http://localhost:8080), unpause the `customer_analytics` DAG, and trigger the DAG.

2. The logs in the `train` and `predict` tasks will contain a link to your W\&B project which shows plotted results from the training and prediction.

   <Frame>
     <img alt="wandb task logs" />
   </Frame>

   Go to one of the links to view the results in W\&B.

   <Frame>
     <img alt="wandb results" />
   </Frame>


# Get started with Astro Observe
Source: https://astronomer.io/docs/learn/astro-observe-quickstart

Learn how to get started with Astro Observe

<Info>
  This page has not yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Astro Observe](https://www.astronomer.io/product/observe/) is Astronomer's comprehensive observability solution for [Apache Airflow®](https://airflow.apache.org/). It allows you to define [data products](/docs/learn/data-products) consisting of Airflow pipelines and [datasets](/docs/learn/airflow-datasets) across your deployments. You can set [Service Level Agreements (SLAs)](/docs/learn/using-slas) on these data products to ensure that your data is delivered on time and is fresh. Easy to set up alerts notify you when SLAs aren't met.

After you complete this tutorial, you'll be able to use Astro Observe to:

* Create a data product.
* Set a timeliness SLA on a data product.
* Set a freshness SLA on a data product.
* Create an alert in case of an SLA not being met.

<Tip>
  **Other ways to learn**

  There are multiple resources for learning about this topic. See also:

  * Webinar: [Introducing Astro Observe: Pipeline-level Observability](https://www.astronomer.io/events/webinars/introducing-astro-observe-pipeline-level-observability-video/).
</Tip>

## Time to complete

This quickstart takes approximately 45 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* Basic Airflow concepts. See [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* Basic observability concepts like data products and SLAs.
* Basics knowledge of how to navigate the [Astro Observe UI](/docs/astro/astro-observe).

## Prerequisites

* An Astro account with access to Astro Observe. Astro customers can [request access to Astro Observe](https://www.astronomer.io/product/observe/request).
* A GitHub account with permission to [authorize GitHub Apps](https://docs.github.com/en/apps/using-github-apps/authorizing-github-apps). If you don't have a GitHub account, you can create one for free on the [GitHub website](https://github.com/signup).

## Step 1: Set up your Astro deployment

1. Sign in to your Astro account.
2. Click **+ Deployment** to [create a new deployment](/docs/astro/create-deployment) using default settings.

<Frame>
  <img alt="Create a new deployment in the Astro UI with the button in the top right corner." />
</Frame>

3. [Fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) the [Observe Quickstart GitHub repository](https://github.com/astronomer/observe-quickstart) to your GitHub account.

   <Frame>
     <img alt="Fork the GitHub repository using the Fork button at the top right of the screen." />
   </Frame>

4. In the Astro UI, go to your newly created deployment and click **Configure** in the **BRANCH MAPPING** section to map the deployment to a branch in your forked repository, see [Deploy code with the Astro GitHub integration](/docs/astro/deploy-github-integration)

   <Frame>
     <img alt="Map the deployment to a branch in your forked repository." />
   </Frame>

5. Click **Connect Repository** and go through the steps in the UI to connect your forked repository to the deployment. Note that your GitHub Account needs to have the necessary permissions to [authorize GitHub Apps](https://docs.github.com/en/apps/using-github-apps/authorizing-github-apps). Make sure to select **Use an existing repository** and select your forked repository in the dropdown. You don't need to provide an **Astro project path**, since the project is at the root of the repository. Click **Connect Repository** to connect the repository to your Astro organization.

   <Frame>
     <img alt="Connect the forked repository to the deployment." />
   </Frame>

6. Select the `main` branch as the **Branch** and map it to your empty Deployment. Click **Update Mappings** to save the changes.

   <Frame>
     <img alt="Map the main branch to the new Deployment." />
   </Frame>

7. To deploy the code currently on the main branch of the mapped repository, click the **...** button in the top right corner, and then click **Trigger Git Deploy...**. Any subsequent commits to the mapped branch will automatically trigger code deployment. You can verify that deploying was successful through the **Deploy History** of your Astro Deployment, which should show 2 Deploys with a green checkmark as their status. This process may take a few minutes.

   <Frame>
     <img alt="Trigger Git Deploy... option in the Deployment menu." />
   </Frame>

## Step 2: Unpause your DAGs

1. After the code has been deployed, click **Open Airflow** to open the Airflow UI.

   <Frame>
     <img alt="Open the Airflow UI." />
   </Frame>

2. Unpause all 3 DAGs in the Airflow UI by clicking on the play button next to each DAG. They will start running according to their schedules.

The three DAGs form a pipeline monitoring air quality data:

* `aq_etl`: This DAG runs once every minute to fetch the latest measurement from an air quality sensor. It will occasionally fail, mocking the sensor not always being available.
* `send_aq_alerts`: This DAG runs once every 10 minutes to check if the current air quality determined by the average of the measurements of the last 20 minutes is below a certain threshold. If it is, the DAG sends an alert.
* `create_aq_report`: This DAG runs at the top of every hour to create a report of air quality measurements. It reports average measurements for each of the last 24 hours.

## Step 3: Create your first data product

After all DAGs have at least run once, you can use Astro Observe to [create a data product](/docs/astro/create-data-products) to represent the output of these DAGs.

1. In the Astro UI, click **Observe** in the sidebar and then click **+ Data Product**.

   <Frame>
     <img alt="Create a new data product in the Astro UI." />
   </Frame>

2. Give your data product a name (we used `Air Quality Report`) and select a **User** as an owner. Then search **All Assets** for the `send_aq_report` task in the `create_aq_report` DAG. This task creates a report of air quality measurements. Click the plus sign to add the task to the data product. Astro Observe will automatically determine upstream dependencies for the task and build a data product graph. This even works across Astro Deployments! Click **Create Data Product** to save the data product.

   <Frame>
     <img alt="Add the send_aq_report task to the data product." />
   </Frame>

You can now see the data product in the Astro UI, monitoring the `send_aq_report` task and its upstream dependencies. Next, you'll create a timeliness SLA for this data product.

## Step 4: Set a timeliness SLA on your data product

The `create_aq_report` DAG runs once per hour. Once a day, you are required to send the latest air quality report to an external agency for compliance purposes. It is very important that this report is delivered on time, meaning you need to be sure that the `send_aq_report` task has successfully completed in the hour before the report is due.
This is where an SLA on a data product comes in handy.

1. Click the **Overview** tab of the `Air Quality Report` data product and then click **+ SLA**.

   <Frame>
     <img alt="Add an SLA to the data product." />
   </Frame>

2. Define a new SLA with the following values:

   * **Name**: `Air Quality Compliance Report`
   * **Description**: `Ensure that the air quality report is delivered on time.`
   * **SLA Type**: **Timeliness**
   * **Days of the week (UTC)**: Click **Select All**.
   * **Verification Time (UTC)**: Pick a time that is close to your current time, for example 10 minutes in the future.
   * **Lookback Period**: `1 hour` (since you want a latest report that isn't older than 1 hour)

   <Frame>
     <img alt="Define the SLA for the data product." />
   </Frame>

3. Click **Create SLA** to save the SLA.

## Step 5: Create an alert on your SLA

After creating the SLA, you can add an alert to it. This alert will notify you if the SLA isn't met — that is, if the **Verification Time** is reached and the `send_aq_report` task hasn't been successfully completed in the last hour.

1. Click the **Alerts** tab and **+ Alert** to add alerts to your SLAs.

   <Frame>
     <img alt="Add an alert to the SLA." />
   </Frame>

2. Select the following values for the alert, leaving all other fields at their default values:

   * **Type**: **Data Product SLA Violation**
   * **Severity**: **Critical**

   If this is your first time creating an alert, you need to create a new notification channel. Click **+ Notification Channel**, give it a name, and select **Email** and add your email address using the blue plus sign. Click **Create Notification Channel** to save the notification channel. After creating the notification channel, you can select it in the alert form. Click **Create Alert** to save the alert.

   <Frame>
     <img alt="Define the alert for the SLA." />
   </Frame>

## Step 6: Create a second data product

Awesome! You will now be immediately informed if the air quality report isn't delivered on time. But there is a second aspect to this pipeline that you want to monitor: that the air quality measurements happen frequently enough, that is, that the data is fresh enough. For this, create a second data product that focuses on the `load_aq_data` task in the `aq_etl` DAG.

1. In the Astro UI, click **+ Data Product**.
2. Give your data product a name (we used `Air Quality Measurements`), select a **User** as an owner, and then search **All Assets** for the `load_aq_data` task in the `aq_etl` DAG. This task updates the air quality measurements table. Click the plus sign to add the task to the data product.
3. Click **Create Data Product** to save the data product.

## Step 7: Set a freshness SLA with an alert

The `aq_etl` DAG runs once per minute. Occasionally the air quality sensor is offline. This is expected, but you want to be sure that the latest air quality measurement is never older than 1 hour.

1. Click the **Overview** tab of the `Air Quality Measurements` data product and then click **+ SLA**.

2. Define a new SLA with the following values:

   * **Name**: `Air Quality Freshness`
   * **Description**: `Ensure that the air quality measurements are updated at least once every Hour.`
   * **SLA Type**: **Freshness**
   * **Freshness Policy**: `1 hour`

3. Click **Create SLA** to save the SLA.

4. Click the **Alerts** tab and **+ Alert** to add an alert to the SLA and add another Critical alert with the same notification channel as before.

5. Click **Create Alert** to save the alert.

## Step 8: (Optional) Test the SLA alert

To test the SLAs, you can manually trigger the `aq_etl` with the option of delaying the `get_aq_data` task for two hours. This will cause the `load_aq_data` task to not be updated for two hours, which will cause the freshness SLA to be violated. No new DAG runs are scheduled, because the DAG is set to only allow one concurrent run with the `max_active_runs` parameter.

1. In the Airflow UI, click the play button next to the `aq_etl` DAG to open the form for manually triggering the DAG.

2. In the form, toggle the `simulate_api_delay` button.

   <Frame>
     <img alt="Manually trigger the DAG with a delay." />
   </Frame>

3. Click **Trigger** to trigger a manual DAG run.

After 1 hour, you should receive an email notification that the freshness SLA has been violated.

<Frame>
  <img alt="Screenshot of an email notification of a violated SLA." />
</Frame>

Congrats! You have successfully set up Astro Observe to monitor several aspects of an air quality pipeline! Of course there is much more to Astro Observe, check out the [Astro Observe documentation](/docs/astro/astro-observe) for more information.


# Create a PostgreSQL connection in Airflow
Source: https://astronomer.io/docs/learn/connections/postgres

Learn how to create a PostgreSQL connection in Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Postgres](https://www.postgresql.org/) is a free and open source relational database system. Integrating Postgres with Airflow allows you to interact with your Postgres database, run queries, and load or export data from an Airflow DAG.

This guide provides the basic setup for creating a Postgres connection.

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* A locally running [Astro project](/docs/cli/v1.43/get-started-cli).
* A Postgres database running in the cloud or on-premises.
* [Permission](https://www.digitalocean.com/community/tutorials/how-to-use-roles-and-manage-grant-permissions-in-postgresql-on-a-vps-2) to access your Postgres database from your local Airflow environment.

## Get connection details

A connection from Airflow to Postgres requires the following information:

* Host (also known as the endpoint URL, server name, or instance ID based on your cloud provider)
* Port (default is 5432)
* Username
* Password
* Schema (default is `public`)

The method to retrieve these values varies based on which cloud provider you use to host Postgres. Refer to the following documents for more information about retrieving these values:

* AWS: Connect to Postgres running on [RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ConnectToPostgreSQLInstance.html)
* GCP: Connect to Postgres running on [ Cloud SQL](https://cloud.google.com/sql/docs/postgres/connect-instance-local-computer)
* Azure: Connect to Postgres running on an [Azure database](https://learn.microsoft.com/en-us/training/modules/create-connect-to-postgres/4-connect-develop-your-database)

For example, if you're running Postgres in a Relational Data Store (RDS) in AWS, complete the following steps to retrieve these values:

1. In your AWS console, select your region, then go to the RDS service and select your Postgres database.
2. Open the **Connectivity & security** tab and copy the **Endpoint** and **Port**.
3. Follow the AWS instructions to [create a user](https://www.postgresql.org/docs/8.0/sql-createuser.html) and [grant a role to the user](https://www.postgresql.org/docs/current/sql-grant.html) that Airflow will use to connect to Postgres. Copy the username and password.
4. (Optional) To use a specific schema, copy the name of the schema. If you skip this, Airflow uses the default schema `public`.

## Create your connection

<Info>
  Astro users can also create connections using the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager), which stores connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).
</Info>

1. Open your Astro project and add the following line to your `requirements.txt` file:

   ```text wrap theme={null}
   apache-airflow-providers-postgres
   ```

   This installs the Postgres provider package, which makes the Postgres connection type available in Airflow.

2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

3. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then choose **Postgres** as the connection type.

4. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

   * **Connection Id**: Enter a name for the connection.
   * **Host**: Enter your Postgres server's host/ endpoint URL/ server name/ instance ID.
   * **Schema**: Enter your schema name.
   * **Login**: Enter your username.
   * **Password**: Enter your password.
   * **Port**: Enter your Postgres server's **Port**.

5. Click **Test**. After the connection test succeeds, click **Save**.

   <Frame>
     <img alt="connection-postgres" />
   </Frame>

## How it works

Airflow uses the [psycopg2](https://pypi.org/project/psycopg2/) python library to connect to Postgres through the [`PostgresHook`](https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/_api/airflow/providers/postgres/hooks/postgres/index.html). You can also directly use the `PostgresHook` to create your own custom operators.

## See also

* [Apache Airflow Postgres provider package documentation](https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/index.html)
* Postgres modules in the [Airflow Registry](https://airflow.apache.org/registry/)
* [Import and export Airflow connections using Astro CLI](/docs/astro/import-export-connections-variables#using-the-astro-cli-local-environments-only)


# Create a Redshift connection in Airflow
Source: https://astronomer.io/docs/learn/connections/redshift

Learn how to create a Redshift connection in Airflow.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Amazon Redshift](https://aws.amazon.com/redshift/) is a data warehouse product from AWS. Integrating Redshift with Airflow allows you to automate, schedule and monitor a variety of tasks. These tasks include creating, deleting, and resuming a cluster; ingesting or exporting data to and from Redshift; and running SQL queries against Redshift.

This document covers two different methods to connect Airflow to Amazon Redshift:

* Using database (DB) user credentials
* Using IAM credentials
* Using IAM role

<Tip>
  If you're an Astro user, Astronomer recommends using workload identity to authorize to your Deployments to Redshift. This eliminates the need to specify secrets in your Airflow connections or copying credentials file to your Airflow project. See [Authorize Deployments to your cloud](/docs/astro/authorize-deployments-to-your-cloud).
</Tip>

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* A locally running [Astro project](/docs/cli/v1.43/get-started-cli).
* Permissions to access to your Redshift cluster. See [Using IAM authentication](https://docs.aws.amazon.com/redshift/latest/mgmt/generating-user-credentials.html) and [Authorizing Amazon Redshift to access other AWS services](https://docs.aws.amazon.com/redshift/latest/mgmt/authorizing-redshift-service.html).

## Get connection details

<details>
  <summary>DB Credentials</summary>

  DB user credentials can be used to establish a connection to an Amazon Redshift cluster. While straightforward to use, this approach lacks the strong security and user access controls provided by identity and access management (IAM). Connecting this way requires the following information:

  * Cluster identifier
  * Database name
  * Port
  * User
  * Password

  Complete the following steps to retrieve these values:

  1. In your AWS console, select the region that contains your Redshift cluster, open the Redshift cluster dashboard, then open your cluster.

  2. From the **General information** section, copy the **Cluster identifier** and **Endpoint**.

  3. Open the **Properties** tab and copy the **Database name** and **Port**.

  4. [Create a Redshift user](https://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_USER.html) and [grant a role](https://docs.aws.amazon.com/redshift/latest/dg/r_GRANT.html) so that Airflow can access Redshift through the user. Copy the username and password.
</details>

<details>
  <summary>IAM Credentials</summary>

  You can use IAM credentials to connect Airflow to Redshift. This approach lets you use IAM credentials and limits Airflow's permissions. The limitation of this method is that you must include an AWS credentials file in your Airflow project.

  This approach requires the following information:

  * Cluster identifier
  * Database name
  * Port
  * Region
  * IAM user
  * AWS credentials file

  Complete the following steps to retrieve these values:

  1. In your AWS console, select the region that contains your Redshift cluster, open the Redshift cluster dashboard, then open your cluster.

  2. Open the **General information** tab, then copy the **Cluster identifier** and **Endpoint**.

  3. Open the **Properties** tab and copy the **Database name** and **Port**.

  4. Open your IAM dashboard, go to **Users** and select your user. Then, go to **Permissions** and follow the [AWS documentation](https://docs.aws.amazon.com/redshift/latest/mgmt/redshift-iam-access-control-identity-based.html) to ensure that the IAM user is authorized to connect to Redshift and perform SQL operations.

  5. [Generate a new access key ID and secret access key](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html).
</details>

<details>
  <summary>IAM Role</summary>

  You can use AWS's [Assume Role](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) method to automatically generate temporary credentials to connect to Redshift. This is useful to grant temporary access to limited-privilege IAM users or roles without storing any credentials on disk. Creating the connection requires the following information:

  * Cluster identifier
  * Database name
  * Port
  * Region
  * IAM role ARN

  Complete the following steps to retrieve these values:

  1. In your AWS console, select the region that contains your Redshift cluster, open the Redshift cluster dashboard, then open your cluster.

  2. Open the **General information** tab, then copy the **Cluster identifier** and **Endpoint**.

  3. Open the **Properties** tab and copy the **Database name** and **Port**.

  4. Open your IAM dashboard, and [follow the AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions_create-policies.html) to create an IAM role and attach an IAM Policy to access the required services, for example AWS Redshift.

  5. Edit the trust relationship of the role created in Step 4 to add a trust policy that allows the IAM role to assume your new role.

     ```json wrap theme={null}

     {
         "Version": "2012-10-17",
         "Statement": [
             {
                 "Effect": "Allow",
                 "Principal": {
                     "AWS": "arn:aws:iam::<your-aws-account>:role/<your-role-name>"
                 },
                 "Action": "sts:AssumeRole"
             }
         ]
     }

     ```

  6. Copy the **ARN** of the role.
</details>

## Create your connection

<Info>
  Astro users can also create connections using the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager), which stores connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).
</Info>

<details>
  <summary>DB Credentials</summary>

  1. Open your Astro project and add the following line to your `requirements.txt` file:

     ```text wrap theme={null}
     apache-airflow-providers-amazon
     ```

     This will install the Amazon provider package, which makes the Amazon Redshift connection type available in Airflow.

  2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

  3. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then select the connection type as **Amazon Redshift**.

  4. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

     * **Connection Id**: Enter a name for the connection.
     * **Host**: Enter the cluster **Endpoint**.
     * **Database**: Enter the **Database name**.
     * **User**: Enter the DB user username.
     * **Password**: Enter the DB user password.
     * **Port**: Enter the **Port**.

  5. Click **Test**. After the connection test succeeds, click **Save**.

     <Frame>
       <img alt="aws-connection-db-creds" />
     </Frame>
</details>

<details>
  <summary>IAM Credentials</summary>

  1. Open your Astro project and add the following line to your `requirements.txt` file:

     ```text wrap theme={null}
     apache-airflow-providers-amazon
     ```

     This will install the Amazon provider package, which makes the Amazon Redshift connection type available in Airflow.

  2. Copy the `aws` credentials file to the `include` directory of your Astro project. It should have the following format:

     ```yaml wrap theme={null}
     # ~/.aws/credentials
     [<your-profile-name>]
     aws_access_key_id="your_aws_access_key_id"
     aws_secret_access_key="your_aws_secret_access_key"
     ```

  3. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

  4. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then select the connection type as **Amazon Redshift**.

  5. Enter a name for the connection in the **Connection Id** field.

  6. Copy the following JSON template into the **Extra** field, then replace the placeholder values with the information you retrieved in [Get connection details](#get-connection-details).

     ```json wrap theme={null}

     {
         "iam": true, 
         "cluster_identifier": "<your-cluster-identifier>", 
         "port": 5439, 
         "region": "<your-region>",
         "db_user": "<your-user>", 
         "database": "<your-database>", 
         "profile": "<your-profile-name>"
     }

     ```

  7. Click **Test**. After the connection test succeeds, click **Save**.

     <Frame>
       <img alt="aws-connection-iam-creds" />
     </Frame>
</details>

<details>
  <summary>IAM Role</summary>

  1. Open your Astro project and add the following line to your `requirements.txt` file:

     ```text wrap theme={null}
     apache-airflow-providers-amazon
     ```

     This will install the Amazon provider package, which makes the Amazon Redshift connection type available in Airflow.

  2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

  3. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection, then select the connection type as **Amazon Redshift**.

  4. Complete the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

     * **Connection Id**: Enter a name for the connection.
     * **Host**: Enter the cluster **Endpoint**.
     * **Database**: Enter the **Database name**.
     * **Port**: Enter the **Port**.
     * **Extra**:
       ```json wrap theme={null}

           {
           "role_arn": "<your-role-arn>",
           "region_name": "<your-region>"
           }

       ```

  5. Click **Test**. After the connection test succeeds, click **Save**.
</details>

## How it works

Airflow uses the [Amazon Redshift Python Connector](https://docs.aws.amazon.com/redshift/latest/mgmt/python-configuration-options.html) to connect to Redshift through the [`RedshiftSQLHook`](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/hooks/redshift_sql/index.html).

## See also

* [Apache Airflow Amazon provider package documentation](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/redshift.html)
* Redshift modules in the [Airflow Registry](https://airflow.apache.org/registry/)
* [Import and export Airflow connections using Astro CLI](/docs/astro/import-export-connections-variables#using-the-astro-cli-local-environments-only)


# Create a Snowflake Connection in Airflow
Source: https://astronomer.io/docs/learn/connections/snowflake

Learn how to create a Snowflake connection in Airflow.

<Tip>
  The key information from this and other Snowflake guides is available as an [Astronomer Cheat Sheet](https://www.astronomer.io/ebooks/airflow-snowflake-cheatsheet/?utm_source=website\&utm_medium=learn-guides\&utm_campaign=snowflake-tutorial).
</Tip>

[Snowflake](https://www.snowflake.com/en/) is a cloud data warehouse where you can store and analyze your data. Integrating Snowflake with Airflow allows you to do all of the following and more from a DAG:

* Run SQL
* Monitor the status of SQL queries
* Run a Snowpark Python function
* Load and export data to/from Snowflake

This guide provides the basic setup for creating a Snowflake connection. For a complete integration tutorial, see [Orchestrate Snowflake Queries with Airflow](/docs/learn/airflow-snowflake).

## Prerequisites

* The [Astro CLI](/docs/cli/v1.43/overview).
* A locally running [Astro project](/docs/cli/v1.43/get-started-cli).
* A [Snowflake account](https://trial.snowflake.com/?owner=SPN-PID-365384).

## Key-pair authentication

A key-pair connection from Airflow to Snowflake requires the following information:

Base parameters:

* `conn_id`: A unique name for the connection.
* `conn_type`: `snowflake`. Note that you need to install the [`apache-airflow-providers-snowflake` provider package](https://airflow.apache.org/registry/providers/snowflake/) to use this connection type.
* `login`: The [user](https://docs.snowflake.com/en/sql-reference/sql/create-user) you authenticate with. Note that some operators require the user to be properly capitalized.
* `password`: The passphrase of the private key. If your private key isn't encrypted (not recommended), you can leave this field as an empty string.
* `schema`: The default [schema](https://docs.snowflake.com/en/sql-reference/sql/create-schema.html) for the connection, this can be overridden in the operator.

Parameters in the `extra` field:

* `account`: The [account identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier) from your Snowflake account URL in the format `abc12345`.
* `warehouse`: The default [warehouse](https://docs.snowflake.com/en/sql-reference/sql/create-warehouse) for this connection, this can be overridden in the operator.
* `database`: The default [database](https://docs.snowflake.com/en/sql-reference/sql/create-database) for the connection, this can be overridden in the operator.
* `region`: The region identifier from your Snowflake account URL in the format `us-west-2`. Note that for some regions, you might have to include the cloud provider identifier after the region name, see [the Snowflake documentation on account identifiers](https://docs.snowflake.com/en/user-guide/admin-account-identifier)
* `role`: The [role](https://docs.snowflake.com/en/sql-reference/sql/create-role) you want Airflow to have in Snowflake. Note that some operators require the user to be properly capitalized.
* `private_key_content`: The content of your private key file. For the Airflow Snowflake provider version 6.3.0+ the key needs to be base64 encoded.
* `private_key_file`: alternatively to `private_key_content`, you can provide the path to your private key file.

Optional:

* `authenticator`: `snowflake` (default). To connect using OAuth, set this parameter to `oauth`.
* `refresh_token`: The refresh token for OAuth authentication.
* `session_parameters`: A dictionary of [session parameters](https://docs.snowflake.com/en/user-guide/python-connector-example.html#setting-session-parameters) to set for the connection.
* `insecure_mode`: `false` (default). Set to `true` to disable [OCSP certificate checks](https://community.snowflake.com/s/article/How-to-turn-off-OCSP-checking-in-Snowflake-client-drivers).

See the template below for a private key connection in JSON format:

```json wrap theme={null}
AIRFLOW_CONN_SNOWFLAKE_DEFAULT='{
    "conn_type":"snowflake",
    "login":"<your user, properly capitalized>",
    "password":"<your private key passphrase>",
    "schema":"DEMO_SCHEMA",
    "extra":{
        "account":"<your account id in the form of abc12345",
        "warehouse":"<your warehouse>",
        "database":"DEMO_DB",
        "region":"<your region>",
        "role":"<your role, properly capitalized>",
        "private_key_content":"LS0..<key>..C0="
    }
}'
```

### Get connection details

Complete the following steps to retrieve the needed connection values:

<details>
  <summary>Snowsight</summary>

  1. Open [Snowsight](https://docs.snowflake.com/en/user-guide/ui-snowsight). Follow the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/ui-snowsight-gs#using-snowsight) to open the account selector at the end of the left nav. Hover over your account to see more details, then click the **Copy URL** icon to copy the account URL. The URL has a similar format to `https://<account-identifier>.<region>.snowflakecomputing.com/`. Copy `<account-identifier>` and `<region>` from the URL.

     <Frame>
       <img alt="Screenshot of the bottom of the left nav in Snowsight showing how to copy the account URL." />
     </Frame>

  <Info>
    When you copy your `region`, you might have to additionally copy the cloud provider identifier after the region name for some GCP and some AWS regions. For example, if your account URL is `https://ZS86751.europe-west4.gcp.snowflakecomputing.com`, then your `region` will be `europe-west4.gcp`. See [Account identifiers](https://docs.snowflake.com/en/user-guide/admin-account-identifier) to learn more about Snowflake's account types and their identifiers.
  </Info>

  2. Click the user menu at the beginning of the left sidebar and copy the role you want Airflow to have in Snowflake. You can click **Switch Role** to see all the available roles.

     <Frame>
       <img alt="Screenshot of the user menu in Snowsight showing how to copy the role." />
     </Frame>

  3. Copy the name of your **Warehouse**. To see all available warehouses, open a new **Worksheet** and open the [context selector menu](https://docs.snowflake.com/en/user-guide/ui-snowsight-worksheets#change-the-session-context-for-a-worksheet) in the content pane.

     <Frame>
       <img alt="Screenshot of the context selector menu in Snowsight showing how to copy the warehouse." />
     </Frame>
</details>

<details>
  <summary>Classic</summary>

  1. Open the [Snowflake classic console](https://docs.snowflake.com/en/user-guide/ui-using) and locate the URL for the page. The URL should be in the format `https://<account-identifier>.<region>.snowflakecomputing.com/`. Copy `<account-identifier>` and `<region>` from the URL.

  <Info>
    When you copy your `region`, you might have to additionally copy the cloud provider identifier after the region name for some GCP and some AWS regions. For example, if your account URL is `https://ZS86751.europe-west4.gcp.snowflakecomputing.com`, then your `region` will be `europe-west4.gcp`. See [Account identifiers](https://docs.snowflake.com/en/user-guide/admin-account-identifier) to learn more about Snowflake's account types and their identifiers.
  </Info>

  2. Click your account name in the top right corner and hover over **Switch Role** to see a list of all available roles. Copy your **Role**.

     <Frame>
       <img alt="Screenshot roles in Snowflake classic console." />
     </Frame>

  3. Copy your **Warehouse** from the **Warehouses** tab.

     <Frame>
       <img alt="Screenshot warehouses tab in Snowflake classic console." />
     </Frame>
</details>

4. Copy the names for your **Database** and **Schema**.

5. In your terminal run the following command to [generate a private RSA key using OpenSSL](https://docs.openssl.org/master/man1/openssl-genrsa/). Note that while there are other options to generate a key pair, Snowflake has [specific requirements for the key format](https://docs.snowflake.com/en/user-guide/key-pair-auth) and may not accept keys generated with other tools. Make sure to write down the key passphrase as you will need it later.

   ```bash wrap theme={null}
   openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 des3 -inform PEM -out rsa_key.p8
   ```

6. Generate the associated public key using the following command:

   ```bash wrap theme={null}
   openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
   ```

7. Format the private key. Version 6.3.0+ of the Airflow Snowflake provider requires the private key to be base64 encoded. You can create a base64 encoded key with the following script:

   ```python wrap theme={null}
   import base64

   with open("path/to/rsa_key.p8", "rb") as key_file:
       private_key_content = base64.b64encode(key_file.read()).decode("utf-8")
       print(private_key_content)
   ```

   <Note>
     If you're on version 6.2.2 or older of the Airflow Snowflake provider, you need to provide the private key without any coding conversions but with newlines encoded as `\n`. You can use the script below to format the key correctly:

     ```python wrap theme={null}
     def format_private_key(private_key_path):
         with open(private_key_path, 'r') as key_file:
             private_key = key_file.read()
         return private_key.replace('\n', '\\n')

     formatted_key = format_private_key('rsa_key.pem')
     print(formatted_key)
     ```
   </Note>

8. In the Snowflake UI, [create a new user](https://docs.snowflake.com/en/sql-reference/sql/create-user) that Airflow can use to access Snowflake. Copy the username and password.

9. Add the **public key** to the user you created in Snowflake. In the Snowflake UI, run the following command. You can paste the **public** key directly from the `rsa_key.pub` file without needing to modify it.

   ```sql wrap theme={null}
   ALTER USER <your user> SET RSA_PUBLIC_KEY='<your public key>';
   ```

## Create your connection

Airflow connections can be created using multiple methods, such as environment variables, the Airflow UI or the Airflow CLI. The following example shows how to create a Snowflake connection using the Airflow UI.

<Info>
  Astro users can also create connections using the [Astro Environment Manager](/docs/astro/manage-connections-variables#astro-environment-manager), which stores connections in an Astro-managed secrets backend. These connections can be shared across multiple deployed and local Airflow environments. See [Create Airflow connections in the Astro UI](/docs/astro/create-and-link-connections).
</Info>

1. Open your Astro project and add the following line to your `requirements.txt` file:

   ```text wrap theme={null}
   apache-airflow-providers-snowflake>=6.4.0
   ```

   This will install the Snowflake provider package, which makes the Snowflake connection type available in Airflow.

2. Run `astro dev restart` to restart your local Airflow environment and apply your changes in `requirements.txt`.

3. In the Airflow UI for your local Airflow environment, go to **Admin** > **Connections**. Click **+** to add a new connection.

4. Fill out the following connection fields using the information you retrieved from [Get connection details](#get-connection-details):

   * **Connection Id**: Enter a name for the connection.
   * **Connection Type**: Select `Snowflake`. If you don't see this option, make sure you've added the `apache-airflow-providers-snowflake` provider package to your `requirements.txt` file.
   * **Description**: (Optional) Enter a description for the connection.
   * **Schema**: Enter your default schema.
   * **Login**: Enter your user. Make sure it's properly capitalized.
   * **Password**: Enter your private key passphrase.
   * **Extra**: Enter the following JSON in the extra field and replace the values with your Snowflake connection details. Add any optional parameters as needed.

     ```json wrap theme={null}
     {
         "account": "<your account id in the form of abc12345>",
         "warehouse": "<your warehouse>",
         "database": "<your database>",
         "region": "<your region>",
         "role": "<your role in capitalized format>",
         "private_key_content": "LS0..<key>..C0="
     }
     ```

## How it works

Airflow uses the [Snowflake connector](https://github.com/snowflakedb/snowflake-connector-python) Python package to connect to Snowflake through the [`SnowflakeHook`](https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/_api/airflow/providers/snowflake/hooks/snowflake/index.html).

The [`SnowflakeSqlApiOperator`](https://airflow.apache.org/registry/providers/snowflake#snowflake-snowflake-SnowflakeSqlApiOperator) uses the [Snowflake SQL API](https://docs.snowflake.com/en/developer-guide/sql-api/index) through the [`SnowflakeSqlApiHook`](https://airflow.apache.org/registry/providers/snowflake#snowflake-snowflake_sql_api-SnowflakeSqlApiHook).

## See also

* [Snowflake Airflow provider package documentation](https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/connections/snowflake.html)
* [Orchestrate Snowflake Queries with Airflow](/docs/learn/airflow-snowflake) tutorial
* [Common SQL Airflow provider package documentation](https://airflow.apache.org/docs/apache-airflow-providers-common-sql/stable/index.html)
* [Import and export Airflow connections using Astro CLI](/docs/astro/import-export-connections-variables#using-the-astro-cli-local-environments-only)
* [See how Snowflake pipelines can run on Astro](https://www.astronomer.io/snowflake-demo/)


# How data products translate pipelines into business value
Source: https://astronomer.io/docs/learn/data-products

Understand data products and how observability transforms invisible data work into measurable business value.

## What is a data product?

A **data product** is a composition of assets that, taken together, deliver a result with business relevance. It captures the end-to-end data lifecycle, and all elements that are involved in creating the product. Dags, tasks, and tables can all be **assets** of data products.

At Astronomer, we see data products as more than just tables or dashboards. They represent the end-to-end data supply chain: from raw sources through transformations to the final deliverable (a report, dashboard, API, ML model, or analytical dataset). What makes something a data product is the **accountability, reliability, and business impact** that comes with it, not just the data itself.

<Info>
  **Note on terminology**

  In this guide, we use the term *data product* to describe the complete pipeline, with its input, intermediate processing steps and states and its output. Each element of a data product is referred to as an asset. This shouldn't be confused with Airflow-specific concepts like [assets for data-aware scheduling](/docs/learn/airflow-datasets). Think of it this way: a data product is the business concept; assets are an element of it and Airflow assets are one technical mechanism Airflow uses to implement data-aware orchestration within a data product.
</Info>

### Data products for managers

From a management perspective, a data product is a **measurable business asset with clear ROI and accountability**. It's something you can point to during budget discussions, assign ownership to, and track performance against defined SLAs. Data products make the invisible work of data teams visible by translating technical complexity into business outcomes.

### Data products for data engineers

From a data engineering perspective, a data product is **a collection of interdependent Dags, tasks, tables and other resources, for a business-critical output** and requires coordinated effort across multiple systems and teams to maintain. Data products represent the pipelines worth investing in, where you'll want proper observability, documentation, and SLA monitoring.

## Why data products matter for data teams

Data products power everything from analytics dashboards and ML models to customer-facing features like dynamic pricing, fraud detection, and personalized recommendations.

<Tip>
  **When they work, they're invisible. When they break, everyone notices.**
</Tip>

Data teams face a major challenge: **much of their most critical work happens behind the scenes**. While software engineers ship visible features that users interact with directly, data engineers build the data infrastructure that makes everything else possible. Systems act as a data source for other systems, or decisions are made based on reports powered by the data output of those processes. This can make it difficult to demonstrate value to management and stakeholders.

Like a manufacturing supply chain, that transforms raw materials into finished products, **data products rely on a complex web of dependencies** across software, systems, tools, and teams. Any failure can have direct impact on revenue, customer satisfaction, and regulatory compliance.

**Data products solve this visibility problem** by creating a common language between data teams and the business. Instead of explaining *we built a 12-stage ETL pipeline with incremental processing and data quality checks*, data teams can communicate *we built the Customer 360 data product that powers our marketing campaigns*. The business understands products. They understand when products break and when they deliver value.

<Frame>
  <img alt="Data product example: Customer 360" />
</Frame>

As data architectures grow in complexity, and start to span multiple teams, clouds, and tools, it becomes increasingly difficult to answer critical questions like:

* **What broke and why?** When a dashboard goes down, can you trace the failure back through 5 Dags, 3 dbt models, and 2 data sources?
* **Who owns this?** When data quality issues arise, do you know which team is responsible?
* **What's the business impact?** If this pipeline fails, does it delay a regulatory report, break customer-facing features, or just update a weekly internal dashboard?
* **What value does the data team provide?** When executives ask what the data team accomplished this quarter, can you point to concrete products rather than technical tasks?

**Data products solve these problems** by treating critical data assets as managed products with:

* **Clear ownership and accountability**: Teams know who has to react to which situation.
* **End-to-end lineage and observability**: You can trace data from source to destination.
* **Defined SLAs and quality standards**: Everyone knows what a *healthy* product looks like.
* **Proactive monitoring and alerting**: Issues are caught before they impact the business.
* **Business-aligned communication**: Stakeholders understand products, not pipelines.

## How to identify data products in your organization

Let's be honest: ***data product* can sound very abstract**. Many data engineers prefer more concrete terms when talking with each other: "*critical table*," "*business-critical pipeline*," or simply "*that table that wakes us up in the middle of the night when it breaks*." The terminology matters less than the **practices and accountability** behind it.

We use *data product* because it helps communicate with non-technical stakeholders. The analogy with physical products and supply chains maps naturally to data products and data pipelines, making it easier to explain concepts like ownership, SLAs, and quality control to business leaders. **More importantly, it helps data teams communicate their value in terms the business understands.**

Not every table or Dag deserves to be a data product, yet they might be part of one. Here are **three practical patterns** that signal you're looking at a data product candidate:

<CardGroup>
  <Card title="Business impact" icon="siren-on">
    If it breaks, someone gets paged and money is lost. Failure triggers incidents, impacts revenue, or causes customer complaints. Does this pipeline wake someone up when it fails at 3 AM?
  </Card>

  <Card title="Orchestration complexity" icon="diagram-project">
    Parent Dags triggering multiple child Dags, pipeline chaining across teams, or complex dependencies using Airflow assets. The orchestration complexity itself signals business criticality.
  </Card>

  <Card title="High-value consumption" icon="users">
    Critical tables used by multiple teams, feeding executive dashboards, powering ML models, or serving external partners. If this disappeared tomorrow, would executives notice? Would customers be impacted?
  </Card>
</CardGroup>

Once you've identified a potential data product using these patterns, you're looking at one or more outcomes of this product, like a table or model. The next step is identifying all the involved upstream assets that produce this outcome.

<Info>
  **Data product assets**

  An **asset** is any component within a data product's lifecycle, including source data (tables), intermediate transformations (Dags or tasks), files, or final outputs, that contributes to delivering the product's business value. Assets are the building blocks that, when combined together, create a complete data product.
</Info>

## Make data products observable

Once you've identified your data product and its assets, the next step is making it visible, observable, and ensuring data quality, timeliness, and freshness through automation.

### The observability gap

Here's the challenge: **orchestration alone isn't enough**.

In software development, once you understand requirements and build to spec, the product works predictably. In data projects, you often don't know about data quality, and concrete attributes like volume and velocity, until you've built the product and put real data in front of real users. This fundamental uncertainty affects how you scope work, communicate with stakeholders, and define success.

Orchestration coordinates the interactions and dependencies between source data, tools, compute resources, and teams. But **many orchestration tools offer only limited monitoring**. It's not enough to just detect that a task is running at high latency, or a Dag has failed. These may appear as isolated incidents, but **one delay often starts a cascade of errors** that quickly overwhelms the system and the people maintaining it.

Platform and data engineers spend their time **reacting to failures** rather than proactively managing data products. Issues are often not detected until the data product is being used (or is missing), by which time it's too late.

### Astro Observe

**[Astro Observe](https://www.astronomer.io/product/observe/)** delivers observability built directly into Astro, providing complete pipeline visibility from ingestion through transformation to delivery. Unlike traditional observability tools that stop at the warehouse, Astro Observe gives you end-to-end visibility.

With Astro Observe, you can:

* **Define data products** with clear ownership and SLAs.
* **Monitor pipeline health in real-time** with automatic lineage visualization.
* **Get proactive alerts** before failures impact the business.
* **Accelerate troubleshooting** with AI-powered insights and log summaries.
* **Track data quality** with built-in checks linked to upstream pipelines.
* **Attribute warehouse costs** to specific pipelines and data products.

### Create a data product in Astro Observe

[Defining a data product in Astro Observe](/docs/learn/astro-observe-quickstart) is straightforward. You identify the critical assets (Dags, tasks, tables) that together deliver business value, assign ownership, set SLAs, and start monitoring. **What makes this process particularly convenient in Observe is that it automatically infers upstream assets based on Airflow metadata just by running your Dags**.

<Frame>
  <img alt="Astro Observe: create data product" />
</Frame>

## Example: Compliance report

An online betting company must submit daily compliance reports to state gaming commissions detailing all wagers placed, payouts made, and suspicious betting patterns that could indicate problem gambling or match-fixing. Each state has different reporting requirements, formats, and submission deadlines, making this a complex challenge.

**Let's check for the three indicators described in the [previous section](#how-to-identify-data-products-in-your-organization) to see if the compliance report should be a data product:**

* **Business impact**: Missing a report triggers substantial fines and potential license suspension, directly halting revenue in that state and damaging the company's reputation with regulators.
* **Orchestration complexity**: Multiple Dags are involved, connected through Airflow assets, all leading to one, final Dag producing the report.
* **High-value consumption**: Serves gaming regulators across multiple states, internal compliance officers, responsible gaming monitors, fraud detection systems, and executive risk dashboards.

Creating a data product and adding all involved assets in Astro Observe enables visualization and monitoring of the flow of data through the distributed tasks that generate and modify the tables.

In Observe, you see a lineage graph that visualizes the path of the report's data from all sources through the Dags that extracted, transformed, and loaded the data into the data product.

Each node represents an asset with a unique identifier, the emitting system (Apache Airflow, Snowflake), and the length of time since the asset was last observed.

<Frame>
  <img alt="Compliance report data product lineage graph" />
</Frame>

Now that the data product is identified and configured, it can also be [monitored proactively](/docs/astro/observe-monitors#data-product-monitors).

## Summary and next steps

<CardGroup>
  <Card title="Communicate value, not just pipelines" icon="comments-dollar">
    Data products help data teams demonstrate ROI. Frame work as *maintaining 10 data products supporting \$50M in revenue* rather than technical tasks.
  </Card>

  <Card title="Data products drive business value" icon="bell-exclamation">
    If pipeline failure wakes someone up, triggers executive escalation, or stops revenue, it's a data product. Business impact is the go-to test for identifying what deserves product-level treatment.
  </Card>

  <Card title="Dependency webs indicate data products" icon="sitemap">
    When you see Dags triggering other Dags, Airflow assets coordinating dependencies across teams, or workflows spanning multiple domains, the orchestration complexity itself indicates that there is a data product.
  </Card>

  <Card title="Business context matters" icon="chart-network">
    Not every popular table is a data product. Ask: Who notices if it disappeared? Would customers be impacted? Would we lose money? Criticality and accountability matter more than usage alone.
  </Card>

  <Card title="Observability prevents cascading failures" icon="radar">
    One delay often cascades into multiple failures. Proactive monitoring with SLA tracking, lineage visualization, and proactive alerting catches issues before they impact business outcomes.
  </Card>

  <Card title="Ownership drives accountability" icon="clipboard-user">
    Clear ownership ensures someone is responsible for quality, uptime, SLAs, and evolution. It enables faster incident response, better stakeholder communication, and demonstrates who delivers value when products succeed.
  </Card>
</CardGroup>

**Want to see how this works in practice?** [Book a demo](https://www.astronomer.io/book-a-demo/) and explore how unified orchestration and observability can transform your data operations.


# Data quality and Airflow
Source: https://astronomer.io/docs/learn/data-quality

Check the quality of your data using Airflow, Astro Observe, and third-party frameworks.

Data quality is fundamental to trustworthy analytics and AI. Silent data issues, like missing records, schema changes, or duplicate entries, can go undetected until dashboards break or models fail. By then, the damage is done.

This guide covers Dag-level checks with SQL check operators, platform-level checks with Astro Observe, and third-party frameworks, giving you the tools to build a comprehensive data quality strategy.

<CardGroup>
  <Card title="Dag-level checks" icon="code" href="#dag-level-checks-with-airflow">
    SQL check operators that run as part of your pipeline, with full control over failure behavior and notifications.
  </Card>

  <Card title="Platform-level checks" icon="chart-line" href="#platform-level-checks-with-astro-observe">
    Astro Observe monitors that run independently of Dag execution, with table-level lineage and event-driven checks.
  </Card>

  <Card title="Third-party frameworks" icon="puzzle-piece" href="#third-party-frameworks">
    Great Expectations, dbt tests, and Soda Core for additional validation capabilities.
  </Card>
</CardGroup>

## Types of data quality challenges

* **Volume anomalies**: Unexpected spikes or drops in row counts.
* **Schema drift**: Column types changing without notice.
* **Completeness issues**: Null values in critical fields.
* **Duplicate records**: Compromising uniqueness constraints.
* **Business rule violations**: Negative amounts, invalid dates, orphaned records.

## Two approaches to data quality

Data quality checks can be implemented at two levels. You often need both: Dag-level checks catch issues before bad data propagates, while platform-level checks provide coverage even when Airflow itself has problems. This also includes checks when data lands, before any Dag processes potentially problematic data.

|                   | Dag-level checks                                      | Platform-level checks                                                     |
| ----------------- | ----------------------------------------------------- | ------------------------------------------------------------------------- |
| **Execution**     | Run as part of your pipeline                          | Run independently of Dag execution                                        |
| **Blocking**      | Can be blocking (see the following note)              | Non-blocking: alert without stopping pipelines                            |
| **Configuration** | Require Dag code changes and deployment               | Configure through an independent UI, no deployment                        |
| **Expertise**     | Require Dag authoring expertise                       | Require no Dag authoring expertise                                        |
| **Availability**  | Can't check data when there is a problem with Airflow | Monitor data even if there is a problem with Airflow                      |
| **Lineage**       | Dag-level lineage                                     | Table-level lineage                                                       |
| **Ideal for**     | Critical validations that demand pipeline actions     | Critical validations and ongoing monitoring that runs independent of Dags |

<Info>
  **Dag-level checks do not have to stop your pipeline**

  You have several options to control failure behavior:

  1. **Trigger rules**: Use trigger rules like `all_done` or `none_failed_min_one_success` on downstream tasks to continue despite failed checks. See [Airflow trigger rules](/docs/learn/airflow-trigger-rules).
  2. **Shell exit code handling**: When calling third-party frameworks through `@task.bash`, append `|| true` or `|| exit 0` to prevent non-zero exit codes from failing the task.
  3. **Skip instead of fail**: Set `skip_on_exit_code` in `@task.bash(...)` to mark tasks as skipped rather than failed.

  **Important**: Even when a task fails but the Dag succeeds (through trigger rules), task-level `on_failure_callback` still fires, ensuring you can get notified about check failures without blocking your pipeline.
</Info>

## Choose a tool

Which tool you choose is determined by the needs and preferences of your organization. Astronomer recommends using Dag-level checks with SQL check operators if you want to:

* Write checks without needing to set up software in addition to Airflow.
* Write checks as Python dictionaries and in SQL.
* Use any SQL statement that returns a single row of booleans as a data quality check.
* Implement many different downstream dependencies depending on the outcome of different checks.
* Have full observability of which checks failed from within Airflow task logs, including the full SQL statements of failed checks.

Astronomer recommends using Dag-level checks with a data validation framework such as Great Expectations or Soda in the following circumstances:

* You want to collect the results of your data quality checks in a central place.
* You prefer to write checks in JSON (Great Expectations) or YAML (Soda).
* Most or all of your checks can be implemented by the predefined checks in the solution of your choice.
* You want to abstract your data quality checks from the Dag code.

In both cases, Astronomer recommends running platform-level checks for Airflow-independent data quality checks across your business-critical data products.

## Dag-level checks with Airflow

### When to use each operator

| Use case                     | Recommended operator        |
| ---------------------------- | --------------------------- |
| Primary key validation       | `SQLColumnCheckOperator`    |
| Null/duplicate checks        | `SQLColumnCheckOperator`    |
| Value range validation       | `SQLColumnCheckOperator`    |
| Row count thresholds         | `SQLTableCheckOperator`     |
| Cross-column business rules  | `SQLTableCheckOperator`     |
| Aggregate validations        | `SQLTableCheckOperator`     |
| Compare to expected value    | `SQLValueCheckOperator`     |
| Compare to historical data   | `SQLIntervalCheckOperator`  |
| Min/max threshold validation | `SQLThresholdCheckOperator` |
| Complex multi-table queries  | `SQLCheckOperator`          |

### SQL check operators

To access the SQL check operators, install the [Common SQL provider](https://airflow.apache.org/registry/providers/common-sql/):

```text wrap theme={null}
apache-airflow-providers-common-sql
```

Import and use them within your Dag:

```python wrap theme={null}
from airflow.providers.common.sql.operators.sql import (
    SQLColumnCheckOperator,
    SQLTableCheckOperator,
    SQLCheckOperator,
    SQLValueCheckOperator,
    SQLIntervalCheckOperator,
    SQLThresholdCheckOperator,
)
```

<AccordionGroup>
  <Accordion title="SQLCheckOperator">
    The `SQLCheckOperator` is the most generic check operator. It runs any SQL query and evaluates the result, giving you enough freedom to cover complex business rules. The check fails if any returned value evaluates to `False` in Python (for example, `0`, `None`, empty string).

    ```python wrap theme={null}
    _check_no_orphaned_payments = SQLCheckOperator(
        task_id="check_no_orphaned_payments",
        conn_id=_DB_CONN_ID,
        sql="""
            SELECT COUNT(*) = 0
            FROM payments p
            LEFT JOIN bookings b ON p.booking_id = b.booking_id
            WHERE b.booking_id IS NULL
        """,
    )
    ```
  </Accordion>

  <Accordion title="SQLColumnCheckOperator">
    The `SQLColumnCheckOperator` validates individual columns using built-in check types. Define a `column_mapping` dictionary to run multiple checks in a single task.

    **Built-in check types:**

    | Check            | Description               |
    | ---------------- | ------------------------- |
    | `null_check`     | Count of NULL values      |
    | `unique_check`   | Count of duplicate values |
    | `distinct_check` | Count of unique values    |
    | `min`            | Minimum value in column   |
    | `max`            | Maximum value in column   |

    **Comparison options:**

    * `equal_to`, `greater_than`, `geq_to` (`>=`)
    * `less_than`, `leq_to` (`<=`)
    * `tolerance` (percentage threshold, as a fraction: `0.1` = 10%)

    ```python wrap theme={null}
    _check_columns = SQLColumnCheckOperator(
        task_id="check_non_promo_bookings_columns",
        conn_id=_DB_CONN_ID,
        table="bookings",
        partition_clause="promo_code IS NOT NULL",
        column_mapping={
            "booking_id": {
                "null_check": {"equal_to": 0},
                "unique_check": {"equal_to": 0},
            },
            "passengers": {
                "min": {"geq_to": 1},
                "max": {"leq_to": 10, "tolerance": 0.05},  # 5% tolerance
            },
        },
    )
    ```

    <Info>
      **partition\_clause**

      The optional `partition_clause` is an additional WHERE filter applied before the checks. It can be added at the operator level (partitions all checks), at the column level in the column mapping (partitions all checks for that column), or at the check level (partitions just that check).
    </Info>
  </Accordion>

  <Accordion title="SQLTableCheckOperator">
    The `SQLTableCheckOperator` runs custom SQL expressions against a given table that must evaluate to `true`. It is suited for business rules spanning multiple columns or requiring aggregations.

    ```python wrap theme={null}
    _check_report_business_rules = SQLTableCheckOperator(
        task_id="check_report_business_rules",
        conn_id=_DB_CONN_ID,
        table="daily_planet_report",
        checks={
            "net_fare_not_negative": {
                "check_statement": "total_net_fare_usd >= 0",
            },
            "discounts_leq_gross": {
                "check_statement": "total_discounts_usd <= total_gross_fare_usd",
            },
            "has_rows_for_today": {
                "check_statement": "COUNT(*) >= 1",
                "partition_clause": "report_date = '{{ ds }}'",
            },
        },
    )
    ```

    The `SQLTableCheckOperator` also supports an optional `partition_clause` on check level for an additional WHERE filter applied before the check.
  </Accordion>

  <Accordion title="SQLValueCheckOperator">
    Performs a simple value check by comparing a SQL result to an expected value (`pass_value`). The value can be of any type. For numerical values, you can set an additional tolerance percentage.

    ```python wrap theme={null}
    _check_planet_count = SQLValueCheckOperator(
        task_id="check_planet_count",
        conn_id=_DB_CONN_ID,
        sql="SELECT COUNT(*) FROM planets",
        pass_value=3,
        tolerance=0.1,  # 10% tolerance
    )
    ```
  </Accordion>

  <Accordion title="SQLIntervalCheckOperator">
    Verify that metrics defined as SQL expressions remain within tolerance compared to those from previous days (`days_back`). This utility helps track how values change over time and identify potential outliers.

    ```python wrap theme={null}
    _check_bookings_vs_last_week = SQLIntervalCheckOperator(
        task_id="check_bookings_vs_last_week",
        conn_id=_DB_CONN_ID,
        table="bookings",
        date_filter_column="CAST(booked_at AS DATE)",
        days_back=-7,
        ratio_formula="max_over_min",
        metrics_thresholds={"COUNT(*)": 3},  # max 3x deviation
    )
    ```

    <Info>
      **Defaults**

      The default for `days_back` is `-7`, and `ds` for the `date_filter_column`. Always set `date_filter_column` explicitly to your table's actual date column.
    </Info>
  </Accordion>

  <Accordion title="SQLThresholdCheckOperator">
    Performs a value check against a minimum and maximum threshold.

    ```python wrap theme={null}
    _check_avg_fare_in_range = SQLThresholdCheckOperator(
        task_id="check_avg_fare_in_range",
        conn_id=_DB_CONN_ID,
        sql="SELECT AVG(amount_usd) FROM payments",
        min_threshold=40000,
        max_threshold=80000,
    )
    ```

    <Info>
      **SQL expression thresholds**

      Thresholds can also be SQL expressions, not just numeric values. For example: `min_threshold="SELECT MIN(target_avg) FROM benchmarks"`.
    </Info>
  </Accordion>
</AccordionGroup>

### Notifications

Detecting data quality issues is only part of the story. The other part is raising awareness. To be notified when a data quality check fails, combine the `on_failure_callback` task parameter with [Airflow notifiers](/docs/learn/error-notifications-in-airflow).

**Slack example:**

To use the `SlackNotifier`, install the following package:

```text wrap theme={null}
apache-airflow-providers-slack
```

In the templated text field, you can access the table through `task.table` and the actual quality issue through `exception`.

````python wrap theme={null}
from airflow.providers.slack.notifications.slack import SlackNotifier

SQLTableCheckOperator(
    task_id="business_rule_checks",
    on_failure_callback=SlackNotifier(
        slack_conn_id="slack_default",
        text="""
            Data quality checks failed for table: `{{ task.table }}`!
            ```
            {{ exception }}
            ```
        """,
        channel="#data-alerts",
    ),
    ...
)
````

<Info>
  **SlackNotifier**

  The `SlackNotifier` requires a properly configured Slack connection. In this case, the connection ID is `slack_default`.
</Info>

<Info>
  **AppriseNotifier**

  The `AppriseNotifier` supports 100+ notification services (Slack, Email, PagerDuty, Teams, etc.) through a unified interface. Install it using `apache-airflow-providers-apprise`.
</Info>

### Check patterns

Astronomer recommends running **column checks first** (field-level validation), followed by **table checks** (business logic), and then proceed with downstream processing.

Additionally, use **task groups** to organize your data quality checks and use the `default_args` parameter to configure notifications for all checks at once.

```python wrap theme={null}
@task_group(
    default_args={
        "on_failure_callback": notify_dq_failure,
    },
)
def data_quality_checks():
    _column_checks = SQLColumnCheckOperator(...)
    _table_checks = SQLTableCheckOperator(...)
```

## Platform-level checks with Astro Observe

**Astro Observe** delivers pipeline-aware data observability purpose-built for Airflow, giving your team visibility into the health, quality, and performance of your business-critical data products across the modern data stack.

Astro Observe data quality helps you monitor tables to ensure data accuracy, completeness, and integrity across your pipelines. It allows you to track key metrics such as column null percentages, schema changes, and table row counts to detect anomalies or unexpected shifts in your data.

<Info>
  Astro Observe is only available to Astro customers.
</Info>

### Create a connection

Before connecting to Astro Observe, configure the necessary permissions for your data platform. Navigate to **Observe** → **Connections** to create a new connection.

<Frame>
  <img alt="Create a new connection in Astro Observe" />
</Frame>

### Asset catalog

Once connected, warehouse tables appear in the Asset Catalog alongside Airflow Dags, tasks, and datasets. Each table shows:

* **Table popularity score**: Helping you prioritize monitoring on frequently accessed tables.
* **Schema details**: Column names and types.
* **Lineage**: Upstream assets and downstream dependencies.

<Frame>
  <img alt="Asset Catalog with table popularity scores" />
</Frame>

### Table-level lineage

Data quality is also about context and understanding dependencies across Dag borders. **Table-level lineage in Astro Observe starts where Airflow stops.** It spans across Dags and shows upstream and downstream assets, including other Dags, tags, and tables from your warehouse.

With **impact analysis** for upstream and downstream assets, you gain platform-level data quality insights.

When a data quality alert fires, Astro Observe shows you which Airflow Dag and task wrote the failing table, what upstream dependencies fed into it, and which data products downstream are now at risk.

<Frame>
  <img alt="Table-level lineage in Astro Observe" />
</Frame>

### Monitors

Navigate to **Observe** → **Data Quality** → **Monitors** to see an overview of existing monitors, and to create new ones using **+ Monitor**.

Monitors are configurable data quality checks that run against your warehouse tables. Each monitor defines **what** to check, **when** to run, and **how** to alert. Astro Observe supports three scopes of monitors:

* **Table monitors**: Monitor data quality metrics for database tables.
* **Data product monitors**: Monitor data product health and pipeline failures.
* **Custom SQL**: Monitor custom data quality or business logic using SQL queries.

When a monitor's condition is breached, it creates an **alert** with a severity level and notifies you through your configured channels.

### Table monitors

Table monitors check data quality for one specific table and support the following types:

* **Row volume change**: Send an alert when any query causes a change in number of rows above the specified range.
* **Column null percentage**: Monitor the percentage of null values in specific columns.
* **Table schema change**: Track schema changes and detect modifications in column types or structures.

For each type, you can configure **conditions**, such as how large the difference in row count must be before triggering an alert or which columns to check for null values. In addition, you can select a **schedule** to define how frequently the conditions should be evaluated. You can also select **notification channels**. Observe supports Email, Slack, PagerDuty, Dag triggers, and Opsgenie out of the box.

<Frame>
  <img alt="Configure monitor conditions and notifications" />
</Frame>

### Data product monitors

A **[data product](/docs/learn/data-products)** is a composition of assets that, taken together, deliver a result with business relevance. It captures the end-to-end data lifecycle, and all elements that are involved in creating the product. Dags, tasks, and tables can all be assets of data products.

Data product monitors check data product health and pipeline failures. They send alerts when any upstream or final Dag in the data product fails.

### Custom SQL monitors

<Warning>
  Custom SQL monitors are currently available for Snowflake only.
</Warning>

Custom SQL monitors check data quality or business logic using SQL queries against your data connections. Write a SQL query that returns a numerical value you want to monitor. The query results are then used to configure conditions.

```sql wrap theme={null}
SELECT COUNT(*) AS inserted_row_count
FROM DEMO.CARMICHAEL_INDUSTRIES.TRANSACTIONS__SALE_ITEMS
WHERE SALE_ITEM_ID LIKE 'ITEM-%';
```

Custom SQL monitors can also run **when data lands in a table**, allowing downstream consumers to be notified of data quality issues immediately.

<Frame>
  <img alt="Custom SQL monitor with event-driven execution" />
</Frame>

### Alerts overview

In addition to proactive alerting, you can get an overview of alerts triggered by your monitors by navigating to **Observe** → **Data Quality**.

<Frame>
  <img alt="Data quality alerts overview in Astro Observe" />
</Frame>

## Third-party frameworks

### Great Expectations

The [airflow-provider-great-expectations](https://great-expectations.github.io/airflow-provider-great-expectations/latest/getting-started/) package provides operators for running Great Expectations validations directly in your Dags.

When deciding which operator fits your use case, consider:

1. **Where is your data?** In memory as a DataFrame, or in an external data source?
2. **Do you need to trigger actions?** Such as sending notifications or updating external systems based on validation results.
3. **What Data Context do you need?** Ephemeral for stateless validations, or persistent to track results over time.

| Scenario                                                      | Recommended operator           |
| ------------------------------------------------------------- | ------------------------------ |
| Data already in memory as Pandas or Spark DataFrame           | `GXValidateDataFrameOperator`  |
| Data in a database, warehouse, or file system                 | `GXValidateBatchOperator`      |
| Need to trigger Slack notifications, emails, or other actions | `GXValidateCheckpointOperator` |
| Want full GX Core features with ValidationDefinitions         | `GXValidateCheckpointOperator` |

See [Orchestrate Great Expectations with Airflow](/docs/learn/airflow-great-expectations) to learn how to use these operators in your Dag.

### dbt tests

If you use dbt with Airflow through [Cosmos](/docs/learn/airflow-dbt), use dbt's built-in testing framework:

* **Schema tests**: `unique`, `not_null`, `accepted_values`, `relationships`.
* **Custom tests**: SQL-based assertions for business-specific validation.

See [Orchestrate dbt Core with Airflow](/docs/learn/airflow-dbt) for integration patterns.

### Soda Core

Run [Soda](/docs/learn/soda-data-quality) checks using `@task.bash`:

```python wrap theme={null}
@task.bash
def soda_scan():
    return "soda scan -d snowflake -c soda_config.yml checks.yml"
```

## Quick reference

| Approach                       | Best for                                        | Trade-offs                  |
| ------------------------------ | ----------------------------------------------- | --------------------------- |
| `SQLColumnCheckOperator`       | Column-level validation (null, unique, min/max) | Requires code changes       |
| `SQLTableCheckOperator`        | Business rules, aggregations                    | Requires code changes       |
| `SQLValueCheckOperator`        | Compare to expected value with tolerance        | Simple comparisons only     |
| `SQLIntervalCheckOperator`     | Compare to historical data                      | Requires consistent history |
| `SQLThresholdCheckOperator`    | Min/max bounds validation                       | Simple bounds only          |
| `SQLCheckOperator`             | Complex multi-table queries                     | Most flexible, most verbose |
| Observe table monitors         | Common patterns, no code                        | Scheduled only              |
| Observe custom SQL             | Business rules, event-driven                    | Snowflake only (for now)    |
| `GXValidateDataFrameOperator`  | In-memory DataFrame validation                  | Requires data in memory     |
| `GXValidateBatchOperator`      | Database/file validation using BatchDefinition  | More setup than DataFrame   |
| `GXValidateCheckpointOperator` | Full GX features with actions                   | Most configuration required |
| dbt tests                      | Model-layer validation                          | Requires dbt                |
| Soda Core                      | Declarative YAML-based checks                   | Additional tool             |

## Conclusion

Data quality is a continuous practice that evolves with your data ecosystem. **Dag-level checks** give you precise control within your pipelines, catching issues before bad data propagates downstream.

**Platform-level checks with Astro Observe** extend that coverage beyond your Dags, monitoring data the moment it lands, tracing issues through table-level lineage, and providing the full context needed to assess impact and resolve problems fast.

**The most resilient data teams use both.** Dag-level SQL check operators for your critical validations, then layer on platform-level monitors in Astro Observe for broader coverage, event-driven checks, and the observability context that connects data quality to your entire DataOps ecosystem.


# Glossary
Source: https://astronomer.io/docs/learn/glossary

A list of key Apache Airflow, observability, and other data terms and definitions that can help you learn important concepts.

Use this glossary to quickly reference key terms, components, and concepts.

| Term                          | Definition                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Airflow connection            | An [Airflow connection](/docs/learn/connections) is a set of configurations and credentials that allows Airflow to connect with external tools.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Airflow UI                    | The [Airflow UI](/docs/learn/airflow-ui) is the primary visual interface for managing DAG and task runs. It contains pages for modifying, monitoring, and troubleshooting an Airflow environment.                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| Airflow variable              | An [Airflow variable](/docs/learn/airflow-variables) is a generic key value pair, such as an API key or file path, that's stored in the Airflow metadata database and that you can reference in a DAG.                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Apache Airflow                | Apache Airflow is an open source tool for programmatically authoring, scheduling, and monitoring data pipelines written in Python. Airflow is scalable, configurable, and the industry standard for managing workflows across your ecosystem.                                                                                                                                                                                                                                                                                                                                                                                                         |
| Data orchestration            | Data orchestration is the automated configuration, scheduling, and management of sequential, interdependent tasks involving data. The complexity of modern data pipelines is reflected in the architecture and feature sets of orchestrators, which shouldn't be confused with simple schedulers. In addition to scheduling, orchestration involves the handling of errors and dependencies between tasks, among other aspects of data pipeline management.                                                                                                                                                                                           |
| Data product                  | In the context of observability, a data product is a data asset monitored by an observability tool. For example, a warehouse table, data lake bucket, local database table, or local file containing business-critical customer data could be a data product.                                                                                                                                                                                                                                                                                                                                                                                         |
| Dataset                       | A [dataset](/docs/learn/airflow-datasets) is a logical grouping of data consumed or produced by tasks in an Airflow DAG. It can be a table, a file, a blob, or a dataframe. Datasets can be used to schedule DAGs with dataset-driven scheduling.                                                                                                                                                                                                                                                                                                                                                                                                          |
| Decorator                     | In Python, decorators are functions that take another function as an argument and extend the behavior of that function. In Airflow, [decorators](/docs/learn/airflow-decorators) provide a simpler way to define Airflow tasks and DAGs compared to traditional operators.                                                                                                                                                                                                                                                                                                                                                                                 |
| Deferrable operator           | A [deferrable operator](/docs/learn/deferrable-operators), also known as an async operator, is an operator that suspends itself while waiting for its condition to be met and resumes on receiving the job status. Tasks that use deferrable operators consume resources more efficiently than sensors because they don't occupy a worker slot when they are in a deferred state. Instead, deferred tasks use the triggerer to poll for job statuses.                                                                                                                                                                                                      |
| Docker image                  | An Airflow [Docker image](https://www.techtarget.com/searchitoperations/definition/Docker-image) is a template used to build containers with [Podman](https://docs.podman.io/en/latest/) or [Docker](https://www.docker.com/), which run Airflow components and execute DAG code. Both Apache Airflow and Astronomer distribute Docker images for Airflow with different build instructions and pre-installed packages.                                                                                                                                                                                                                               |
| Dynamic DAG                   | A [Dynamic DAG](/docs/learn/dynamically-generating-dags) is a DAG that is generated automatically when the scheduler parses the `dags` folder. You can dynamically create DAGs based on code in one or more Python files or by using tools like `gusty` or `dag-factory`.                                                                                                                                                                                                                                                                                                                                                                                  |
| Dynamic task                  | A [Dynamic task](/docs/learn/dynamic-tasks#dynamic-task-concepts) is a task instance that's generated at runtime based on a set of parameters in DAG code. Dynamic task mapping, the Airflow feature that creates dynamic tasks, allows users to create an arbitrary number of parallel tasks at runtime based on an input parameter.                                                                                                                                                                                                                                                                                                                      |
| Environment variable          | An environment variable is a key-value pair that can be used to define an Airflow environment configuration. You typically set environment variables in the `airflow.cfg` file.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| Executor                      | An [executor](/docs/learn/airflow-executors-explained) is a core process within the Airflow scheduler that is responsible for assigning scheduled tasks to a worker process that will complete the function of a task. Airflow supports multiple executors that differ based on the types of workers they use.                                                                                                                                                                                                                                                                                                                                             |
| Hook                          | A [hook](/docs/learn/what-is-a-hook) is an abstraction of a specific API that allows Airflow to interact with an external system. Hooks are built into many operators, but they can also be used directly in DAG code.                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| Jinja Template                | [Jinja templating](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html#jinja-templating) is a format that is used to pass dynamic information into task instances at runtime. A jinja templated value is enclosed in double curly braces.                                                                                                                                                                                                                                                                                                                                                                              |
| Lineage                       | In the context of observability, lineage refers to the collecting of data product metadata and task metadata in pipelines, typically in real time. Lineage consumers use this metadata to represent the movement of data through pipelines as a dynamic map of the relationships between tasks, data products, and systems. Lineage enables the visualization of the upstream and downstream dependencies of business-critical data products, making it easy to identify the tasks and data products that feed them. Lineage also facilitates SLA evaluation, alerting, tagging, and root-cause analysis of bottlenecks, data latency, and staleness. |
| Lineage graph                 | A lineage graph is a map of the relationships between tasks, data products, and systems in a data ecosystem. Lineage graphs enable the visualization of upstream and downstream dependencies, making it easy to trace the path taken by data in critical data products back to their originating tasks and data sources.                                                                                                                                                                                                                                                                                                                              |
| Notifier                      | A [notifier](/docs/learn/error-notifications-in-airflow#pre-built-notifiers) is a custom class that is pre-built into some provider packages and can be used to send notifications to tools like Slack or PagerDuty.                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| Observability                 | Observability refers to the insights gleaned from collecting, visualizing, and analyzing metadata about data products and tasks, typically at runtime. The metadata and analytics available from observability include upstream and downstream dependencies (jobs as well as data products), SLA evaluations, alerts, data freshness and quality metrics, data product ownership information, job duration, and job status.                                                                                                                                                                                                                           |
| Operator                      | [Operators](/docs/learn/what-is-an-operator) are the building blocks of Airflow DAGs. An operator contains the logic of how data is processed in a pipeline. Each task in a DAG is defined by instantiating an operator.                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| Provider                      | An [Airflow provider](https://airflow.apache.org/docs/apache-airflow-providers/) is a Python package that can be added to core Airflow to extend its functionality. A provider package typically contains modules such as operators, hooks, and sensors to interact with an external service. You can add providers to your Airflow environment by adding their package names to the `requirements.txt` file of your Astro project. For a list of available providers, see the [Airflow Registry](https://airflow.apache.org/registry/providers/).                                                                                                    |
| Scheduler                     | The scheduler is the Airflow component responsible for scheduling job and task instances. It is a multi-threaded Python process that determines what tasks need to be run, when they need to be run, and where they are run.                                                                                                                                                                                                                                                                                                                                                                                                                          |
| Sensor                        | An [Airflow Sensor](/docs/learn/what-is-a-sensor) is a special kind of operator that is designed to wait for something to happen. When sensors run, they check to see if a certain condition is met before they are marked successful and let their downstream tasks execute.                                                                                                                                                                                                                                                                                                                                                                              |
| Service Level Agreement (SLA) | SLAs are increasingly important tools organizations use to help ensure the efficient, timely, and reliable delivery of data. SLAs can specify: an expected timeframe in which data should be processed and made available; bounds for pipeline latency, uptime, errors, and throughput; recovery time and recovery point objectives; expected notification and resolution times; and more.                                                                                                                                                                                                                                                            |
| Service Level Indicator (SLI) | SLIs are quantifiable measures of data quality. Common SLIs of importance to both data teams and data consumers are data freshness and on-time delivery.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| Service Level Objective (SLO) | SLOs are performance targets for Service Level Indicators (SLIs) such as timely on-time delivery rates and freshness rates.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| Tag                           | In the context of data observability, tags are custom labels that data practitioners and admins can attach to data products and tasks to aid in metadata discovery across pipelines. A common use case for tags is monitoring PII access.                                                                                                                                                                                                                                                                                                                                                                                                             |
| Task                          | A task is the basic unit of execution in Airflow. Tasks are arranged into DAGs, and then have upstream and downstream dependencies set between them to express the order in which they should run.                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| Task dependency               | A task dependency is an instruction that defines whether a task must be completed either before or after another task in the same DAG. Task dependencies are defined in DAG code either explicitly with bitshift operators or implicitly with the TaskFlow API.                                                                                                                                                                                                                                                                                                                                                                                       |
| Task group                    | A [task group](/docs/learn/task-groups) is a way to visually organize a group of tasks in the Airflow UI. Task groups are defined in DAG code and render as groupings in the **Graph** view of the Airflow UI.                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| TaskFlow API                  | The [TaskFlow API](/docs/learn/airflow-decorators#how-to-use-the-taskflow-api) is a framework for using decorators to define DAGs and tasks. Compared to traditional operators, using the TaskFlow API simplifies the process for passing data between tasks and defining dependencies.                                                                                                                                                                                                                                                                                                                                                                    |
| Triggerer                     | The triggerer is an optional Airflow component responsible for running [deferrable operators](/docs/learn/deferrable-operators#terms-and-concepts) when they're in a deferred state.                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| Webserver                     | The webserver is the Airflow component that serves the Airflow UI. It is a Flask server running with Gunicorn.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| XCom                          | [XCom](/docs/learn/airflow-passing-data-between-tasks#xcom) is an Airflow feature that allows you to exchange task metadata or small amounts of data between tasks. XComs are defined by a key, value, and timestamp.                                                                                                                                                                                                                                                                                                                                                                                                                                      |


# Implementing proactive alerting
Source: https://astronomer.io/docs/learn/proactive-alerting

Learn about implementing proactive alerting as part of your observability solution.

Proactive alerting enables teams to stay ahead of data quality issues. You can set up an open-source approach using tools that integrate with Airflow or use Astro, which offers built-in support for data quality monitoring.

<Tip>
  **Other ways to learn**

  See also:

  * Blog post: [Data Products: It's not what you call them that matters. It's what you do with them](https://www.astronomer.io/blog/data-products-data-observability/).
</Tip>

## Why to use proactive alerting

The use of proactive alerting adds value by helping ensure:

* **Clear expectations for data quality**: You can use proactive alerts to establish and evaluate data products against expectations for quality metrics. For example, you can track the accuracy, completeness, consistency, and timeliness of the data in your pipelines.

* **Accountability**: By formalizing standards for data quality, proactive alerting establishes accountability. If the data don't meet the agreed-upon standards, the responsible party (whether it's an internal team or a third-party provider) can be held accountable for remediation.

* **Performance measurement**: Proactive alerts provide a framework for setting and evaluating assets against benchmarks and metrics that allow organizations to track performance over time, identify issues, and measure improvements.

* **Proactive management**: With proactive alerts in place, organizations can proactively manage data quality. Regular monitoring helps catch and address issues before they impact business processes or decision-making.

* **Risk mitigation**: High-quality data are essential for making informed decisions. Proactive alerting helps mitigate risks associated with poor data quality by ensuring that data meet required standards, reducing the likelihood of errors or misleading information.

* **Continuous improvement**: Proactive alerting often includes provisions for regular reviews and updates. This encourages continuous improvement in data quality practices and helps teams adapt to changing needs or new challenges.

* **Customer satisfaction**: For organizations that provide data as a service, proactive alerting helps build trust with clients by guaranteeing a certain level of data quality. This can lead to higher customer satisfaction and retention over time.

* **Legal and compliance requirements**: In regulated industries, maintaining high data quality is often a legal requirement. Proactive alerting helps ensure compliance with these regulations by formalizing data quality standards and monitoring processes.

* **Resource allocation**: Proactive alerting helps in resource planning and allocation. By defining expectations for quality, organizations can allocate resources more effectively to meet these standards and more easily handle any issues that arise.

In essence, proactive alerting helps create a structured approach to managing and ensuring data quality, which is critical for effective decision-making and operational efficiency.

## When to use proactive alerting

Proactive alerting is most useful when you need to intervene *before* a table isn't delivered on time or an update fails. When performance and governance aren't concerns, but you do need to know when a particular task or DAG fails, alerts may suffice. For this use case, Astronomer recommends Astro [alerts](/docs/astro/alerts), which don't require modification of DAG code. Airflow [notifications](/docs/learn/error-notifications-in-airflow) provide some of the same functionality.

By contrast, when you need to implement robust performance and data quality monitoring, a standardized approach at the organizational level, alerting on timeliness and freshness of business-critical [data products](/docs/learn/data-products) is recommended. In this case, Astronomer recommends using [Astro Observe](/docs/astro/astro-observe).

## Implement proactive alerting

Using [Astro Observe](/docs/astro/astro-observe), you can add proactive alerts to any of your [SLAs](/docs/learn/using-slas) to get notified when an SLA is at risk of being breached. There are two types of proactive alerts you can set up in Astro Observe.

### Data product proactive SLA alerts

These alerts send a notification when a data product asset is approaching its SLA deadline. A common use case is to configure a data product proactive alert on an asset (a task or dataset) that is upstream of the data product. This way you can catch a delay in your pipeline before the deadline of the data product is missed.

For example, you might have a dashboard that needs to be updated by 9am EST every day. You could set up a proactive SLA alert to notify the European team if the 3 most important tables feeding the dashboard haven't been updated by 7am EST, giving them 2 hours to investigate and fix the issue before the executive in the US looks at the dashboard.

### Data product proactive failure alerts

These alerts send a notification when a data product asset has failed. Being apprised of an upstream asset failure can help you fix the issue in time to be able to rerun the pipeline for the data product to meet its SLA.

### How to set up a proactive alert in Astro Observe

1. To create a proactive alert, in Astro Observe, navigate to the **Alerts** tab of your data product. Click **+ Alert** to set a new alert.

   <Frame>
     <img alt="Astro Observe Alerts tab screenshot" />
   </Frame>

2. Select the alert type from the dropdown menu and select the severity of the alert. Note that to be able to define a Data Product Proactive SLA, you need to have a timeliness [SLA](/docs/learn/using-slas) set up on the data product.

   <Frame>
     <img alt="Astro Observe Alerts dropdown screenshot" />
   </Frame>

3. You can define the alert conditions, specifically which assets to base the alert on if they either fail (Data Product Proactive Failure Alert) or aren't updated by their SLA deadline (Data Product Proactive SLA Alert).

   <Frame>
     <img alt="Astro Observe Alerts conditions screenshot" />
   </Frame>

4. Lastly, decide how you want to be notified. If this is your first time setting up a notification on Astro you might need to add a new notification channel by clicking **+ Notification Channel**.

You can now view and modify your new alert in the **Alerts** tab of your data product.

<Frame>
  <img alt="Astro Observe Alerts list screenshot" />
</Frame>


# AI-powered education operations with Apache Airflow®
Source: https://astronomer.io/docs/learn/reference-architecture-ai-education-operations

Learn how to build an event-driven, multi-agent AI system that automates ticket processing, customer reporting, and conversational data queries using Apache Airflow.

## Overview

Education teams managing certification programs juggle three workstreams: responding to student tickets (badge issuance, exam extensions, coupon validation, invoice generation), generating enrollment and certification reports for customers, and answering internal questions about education data. This architecture uses Airflow to orchestrate a multi-agent AI system that automates all three — processing Zendesk tickets through a human-in-the-loop pipeline, producing on-demand customer education reports from Snowflake, and letting employees query education data conversationally using Slack.

Airflow sits at the center of this system, coordinating six Dags that react to external events from Zendesk, Slack, and an SQS message queue. Rather than running on a fixed schedule, assets and asset watchers make the system event-driven: a Slack message or a new ticket triggers a Dag run within seconds, and cross-Dag asset dependencies chain the processing stages together without manual orchestration.

## Architecture

<Frame>
  <img alt="Architecture diagram for AI-powered education operations with Apache Airflow." />
</Frame>

This architecture consists of five main components:

* **Ticket ingestion**: A daily Dag fetches open Zendesk tickets and emits one asset event per ticket. A separate asset watcher monitors an SQS queue for Slack interaction payloads (**Approve**, **Regenerate**, **Skip**), triggering a Slack event handler Dag that either posts approved responses to Zendesk, re-enters the processing pipeline with reviewer feedback, or skips the ticket.
* **AI agents**: Multiple Pydantic AI agents powered by Claude handle each stage of the pipeline — categorization (mapping tickets to 11 categories with confidence scores), email disambiguation (resolving cross-account requests), coupon extraction and validation, name extraction for badge corrections, response generation (using an academy handbook as context), and status classification (solved, pending, or open).
* **Service handlers**: Deterministic handlers dispatch actions based on the ticket category — querying Snowflake for exam results, issuing Credly badges, extending Skilljar deadlines, generating PDF invoices, or merging student accounts.
* **Human review loop**: Every generated response is posted to Slack with **Approve**, **Regenerate**, and **Skip** buttons. Reviewer actions flow through SQS back into Airflow, where an event handler Dag branches on the action type: approve updates the Zendesk ticket directly, regenerate re-enters the processing pipeline with feedback to refine both categorization and response quality, and skip leaves the ticket for manual handling.
* **Data query interface**: A custom trigger watches a Slack channel for bot mentions. A routing Dag classifies each message — domain URLs trigger a group analytics Dag for customer enrollment metrics, while free-text questions trigger a data query Dag where an LLM-powered branch routes to either a Snowflake or Zendesk agent to answer the question in-thread.

### Airflow features

* [**Assets and asset watchers**](/docs/learn/airflow-datasets): Six assets connect six Dags into an event-driven graph. `fetch_and_process_tickets` emits `process_ticket_asset` to trigger processing; the Slack event handler emits `regenerate_ticket_asset` to re-trigger the same Dag with reviewer feedback as asset metadata; the Slack channel router emits `group_summary_asset` or `slack_data_query_asset` depending on message type. Two asset watchers bridge external systems: a `MessageQueueTrigger` polls SQS for Slack interaction payloads, and a custom `SlackChannelTrigger` polls Slack channels for bot mentions — both create Dag runs without polling from task code.
* [**Dynamic task mapping**](/docs/learn/dynamic-tasks): `process_a_ticket.expand(ticket_id=ticket_ids)` creates one mapped task instance per Zendesk ticket, with the count determined at runtime by the daily fetch query. The same pattern applies to group summaries, where `process_domain.expand_kwargs(domain_data)` maps one task group per customer domain.
* [**Branching**](/docs/learn/airflow-branch-operator): `@task.branch` routes ticket processing based on the trigger source — new ticket, regeneration with feedback, or manual trigger with a ticket ID parameter. The Slack event handler uses the same pattern to dispatch approve, regenerate, and skip actions to separate task paths.
* **LLM branching**: `@task.llm_branch` in the data query Dag uses Claude to decide whether a Slack question should be answered by querying Snowflake (education platform data) or Zendesk (support ticket data), routing to the appropriate downstream task without writing custom classification logic.
* [**Task groups**](/docs/learn/task-groups): The group summary Dag wraps per-domain processing — domain lookup, report building, onboarding metrics, and Slack posting — in a `@task_group` that maps dynamically over each customer domain.

## Considerations

* **Human-in-the-loop granularity**: This architecture routes every Zendesk ticket response through Slack review before posting. For high-confidence categories (badge issuance after a verified exam pass), you could skip review and post directly to Zendesk, reserving human review for low-confidence or sensitive categories like refunds.
* **Deterministic post-classification**: Once the AI categorizes a ticket, the service handlers that process it are entirely deterministic — issuing a badge, extending a deadline, or validating a coupon are predictable operations with known steps. There is no point in using AI when you already know what to do. AI handles the ambiguity (what does this ticket need?), and code handles the execution (do it). This keeps the pipeline fast, reliable, and auditable where it matters most.
* **Agent model selection**: The categorization and response agents use Claude Sonnet for a balance of speed and accuracy. If classification accuracy drops for edge cases, switching the categorizer to a more capable model while keeping the response agent on Sonnet preserves throughput where it matters.
* **Asset watcher polling frequency**: Both asset watchers use custom triggers — the SQS watcher polls every 5 seconds and the Slack channel watcher every 10 seconds. Lower intervals improve responsiveness but increase API calls. Tune these based on your ticket volume and SLA requirements.

## Next steps

* To learn more about connecting Airflow to external event sources, see [Assets and asset watchers](/docs/learn/airflow-datasets). For patterns around orchestrating AI agents with Airflow, see LLM branching.


# Context graphs for self-improving AI Agents with Apache Airflow®
Source: https://astronomer.io/docs/learn/reference-architecture-context-graph

Learn how to build a context graph architecture that captures business decision traces and makes them available to AI agents as institutional memory.

## Overview

AI agents can research, analyze, and recommend, but without context they don't know your organization's decision patterns and reasoning. A [*context graph*](https://foundationcapital.com/ideas/context-graphs-ais-trillion-dollar-opportunity) is the full trace of a business decision, including every input, step, incremental decision by AI Agents and Humans, as well as the reasoning behind each decision, final output, and feedback. That record becomes retrievable memory, so the next time the agent encounters a similar situation, it draws on your organization's actual decision history instead of its training data alone.

This reference architecture demonstrates the pattern using a customer communication drafting scenario. A sales professional requests a personalized draft for a specific customer. Airflow orchestrates six Dags that gather public and proprietary context, generate an AI draft, route it through human review, send the communication, and capture the full decision trace as a context graph for future use.

## Architecture

<Frame>
  <img alt="Context graph reference architecture diagram showing six Airflow Dags, three storage layers, and the flow from stakeholder request to context graph assembly." />
</Frame>

A stakeholder request from a sales professional for communication tailored to a specific customer triggers the pipeline through event-driven scheduling. Four Dags process the request, generate and review a communication, and assemble the context graph.

* **Context gathering Dag**: Retrieves existing context and memory from the context store, including relevant past decision traces. This gives the AI agent access to how similar customers were handled previously, as well as detailed information about the customer and market signals in their business domain.
* **Draft communication Dag**: Uses the gathered context to generate an AI draft proposal, then evaluates it with an AI-as-a-judge step. A human-in-the-loop step sends a message to the requesting stakeholder to ask for approval, edit suggestions and the reasoning behind any proposed changes. Depending on the outcome, the draft is either approved and the next Dag is triggered, or the draft is rejected and a second run of the draft communication Dag is triggered to propose an improved draft.
* **Email interaction Dag**: Sends the approved communication through the email client and captures the customer's response.
* **Context graph gathering Dag**: Assembles the complete context graph by collecting the context input, AI draft and reasoning, AI review, human decision and reasoning, final output, and customer response. Stores the assembled context graph in the context store for future retrieval by the AI agent.

The first Dag in the event-driven pipeline takes advantage of context that has been gathered by two batch Dags:

* **Public context processing Dag**: Extracts data from public sources including industry news feeds, social media mentions, and market signals. Transforms the data and loads it to object storage, after it passes a data quality check.
* **Proprietary data processing Dag**: Extracts proprietary customer data from product usage analytics, support interactions, and sales interactions. Follows the same extract, transform, data quality check, and load pattern as the public context Dag with the option to store the data in a relational database instead of object storage, depending on the data type.

The logos in the architecture diagram represent options for different data sources and storage layers, Airflow can connect to any tool that has an API, which means you can easily switch to different tools and vendors for any of the steps in the pipeline.

### Storage

The architecture uses three storage layers:

* **Object storage**: Raw and intermediate data in JSON, Markdown, and CSV formats.
* **Relational database**: Operational tables, reporting tables, dashboards, and aggregated relational data.
* **Context store**: Assembled context graphs stored as Markdown documents, vector embeddings, and graph-based storage for retrieval by the context gathering Dag.

### Airflow features

* [**Event-driven scheduling**](/docs/learn/airflow-event-driven-scheduling): The pipeline starts when a sales professional submits a stakeholder request. This event triggers the context gathering Dag.
* [**Assets and data-aware scheduling**](/docs/learn/airflow-datasets): The four inference Dags use a data-aware schedule to run as soon as all relevant upstream data is ready. For example, the draft communication Dag runs as soon as the context gathering Dag has determined which context is relevant to the customer request.
* [**Human-in-the-loop (HITL)**](/docs/learn/airflow-human-in-the-loop): The draft communication Dag use an Airflow HITL operator to pause execution and wait for a human reviewer to approve, edit, or reject the AI-generated draft. The reviewer provides their decision and reasoning through a frontend separate from the Airflow UI, for example Slack.
* [**Airflow AI SDK**](https://github.com/astronomer/airflow-ai-sdk): The AI draft proposal and AI-as-a-judge steps in the draft communication Dag use the `@task.agent` decorator of the Airflow AI SDK to run AI agents as Airflow tasks.
* [**Dynamic task mapping**](/docs/learn/dynamic-tasks): Extraction from multiple public sources is parallelized using dynamic task mapping, where the number of mapped tasks is determined at runtime based on the amount of source data that is available.
* [**Automatic retries**](/docs/learn/rerunning-dags#automatically-retry-tasks): Tasks that call external APIs for data extraction are configured to automatically retry after an adjustable delay to handle transient failures and rate limits.
* [**Airflow plugins**](/docs/learn/using-airflow-plugins): A [custom plugin](https://github.com/TJaniF/airflow-hitl-slack-plugin) provides sends the required actions from the human-in-the-loop operator to a Slack channel.

## Considerations

* **Decision trace granularity**: Capturing too little context makes the memory useless. Capturing too much makes retrieval noisy. Start with the inputs, the decision, and the reasoning, then iterate based on how well the agent's future suggestions improve.
* **Context store technology**: The right storage format depends on how much context each decision carries and how interconnected your decisions are. If decisions are relatively self-contained with a small number of inputs, vector embeddings with similarity search work well for retrieval. If decisions frequently involve many interdependent factors, or if you need to traverse relationships between past decisions, graph-based storage makes it easier to surface relevant precedents.
* **Human interaction interface**: The human-in-the-loop feature in Airflow does provide a way to respond to required actions in the Airflow UI, but in many cases the stakeholders who make the decision might not be familiar with, or don't even have access to the Airflow UI. In these situations you can use the [Airflow REST API](https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html) to send the required actions to a Slack channel or other messaging tool, so the stakeholder can respond directly from their preferred channel.

## Next steps

You can find a write-up of a similar architecture in the [Astronomer blog](https://www.astronomer.io/blog/how-to-build-a-decision-tracing-context-graph-with-apache-airflow/), with the full code on [GitHub](https://github.com/TJaniF/airflow-hitl-slack-plugin). This GitHub repository also contains the proof-of-concept implementation of the custom Airflow plugin that sends the required actions to a Slack channel.

To build your own context graph architecture we highly recommend first determining a common workflow in your organization and its main decision steps, then start gathering the context that is relevant for the decision programmatically. Often this means you'll want to add human-in-the-loop operators to the Dag, see [Human-in-the-loop workflows](/docs/learn/airflow-human-in-the-loop) for more details.


# ELT with BigQuery, dbt, and Apache Airflow® for eCommerce
Source: https://astronomer.io/docs/learn/reference-architecture-elt-bigquery-dbt

Learn how to build an end-to-end ELT pipeline with Apache Airflow®, BigQuery, and dbt Core.

## Overview

This reference architecture shows how to build an ELT pipeline that ingests eCommerce transaction data, loads it to [Google BigQuery](https://cloud.google.com/bigquery), transforms it through multiple layers using [dbt Core](https://github.com/dbt-labs/dbt-core), and reports on top customers through Slack. [Apache Airflow®](https://airflow.apache.org/) orchestrates the entire flow across three Dags that are chained together using data-aware scheduling.

<Frame>
  <img alt="Screenshot of a Slack message listing the cheese enthusiasts." />
</Frame>

The architecture demonstrates a common pattern in analytics engineering: extracting from an API, staging raw data in object storage, loading to a warehouse, and running layered transformations (staging, intermediate, mart) using dbt. You can adapt it by swapping the data source, adjusting the dbt models, or replacing the Slack reporting step.

## Architecture

<Frame>
  <img alt="BigQuery reference architecture diagram." />
</Frame>

This reference architecture consists of four main components:

* **Extraction**: An Airflow Dag calls the eCommerce store's API and writes the response data (customers, orders, products) to a [Google Cloud Storage (GCS)](https://cloud.google.com/storage) bucket. Each record type is extracted as a separate file, and only new or updated records are fetched on each run.
* **Loading**: A second Dag picks up the files from GCS and loads them into BigQuery using the BigQuery transfer service. Each record type is loaded into its own raw table.
* **Transformation**: Once the raw tables are populated, dbt Core runs a series of transformations orchestrated with [Astronomer Cosmos](https://astronomer.github.io/astronomer-cosmos/index.html). The models follow a staging, intermediate, and mart layer pattern to produce clean, aggregated reporting tables.
* **Reporting**: After transformations complete, a final task queries the mart tables for top customers and sends a summary to a Slack channel.

Data flows through the system in a clear sequence: API to GCS to BigQuery raw tables to dbt-transformed marts to Slack. Each Dag handles one phase and triggers the next through data-aware scheduling, so downstream work only starts when upstream data is ready.

### Airflow features

* [Astronomer Cosmos](/docs/learn/airflow-dbt): dbt Core models are rendered as individual Airflow tasks using Cosmos, giving operators full visibility into each dbt model's execution status directly in the Airflow UI rather than treating the entire dbt run as a single opaque task.
* [Dynamic task mapping](/docs/learn/dynamic-tasks): During extraction, one mapped task is created per record type (customers, orders, products). The number of files and their names are determined at runtime, so the Dag adapts automatically when new record types are added.
* [Data-aware scheduling](/docs/learn/airflow-datasets): The extraction Dag runs on a time-based schedule, but the loading and transformation Dags use assets to trigger only when the data they depend on has been updated. This eliminates idle polling and ensures each phase runs exactly when its input data is ready.
* [Task groups](/docs/learn/task-groups): The loading step groups related tasks (one per record type) into a task group, keeping the Dag graph readable even as the number of record types grows.
* [Airflow retries](/docs/learn/rerunning-dags#automatically-retry-tasks): All tasks that interact with external services (the eCommerce API, GCS, BigQuery) are configured to automatically retry after an adjustable delay to handle transient failures.
* [Custom XCom backend](/docs/learn/custom-xcom-backends-tutorial): Extracted records are passed between the extraction and loading Dags through XCom. Because the payloads can be large, XComs are stored in GCS using an Object Storage custom XCom backend instead of the Airflow metadata database.
* Modularization: SQL queries used in `BigQueryInsertJobOperator` tasks are stored in the `include` folder and imported into the Dag file. This separates orchestration logic from transformation logic and makes individual queries reusable across Dags.

### Astro features

This architecture includes a dbt Core project alongside the Airflow Dags. Astro customers have two options for [deploying dbt projects to Astro](/docs/astro/deploy-dbt-project):

* **Include dbt in the Astro project**: Add a `/dbt` folder to the Astro project and deploy everything together with `astro deploy`. This is the quickest option for small teams where dbt and Airflow code live in the same repository.
* **dbt Deploys**: Deploy dbt code independently from the Astro project image using `astro dbt deploy`. This decouples dbt iterations from Airflow deployments, which is useful when dbt code lives in a separate repository or when multiple teams need to update dbt models without redeploying the full Astro project.

<Frame>
  <img alt="Screenshot of the Astro UI showing the Deploy History for a Deployment with a dbt Deploys entry." />
</Frame>

Both options provide enhanced dbt observability in the Astro UI.

## Next steps

To build your own ELT pipeline with BigQuery, dbt Core, and Apache Airflow, explore the individual Learn guides linked in the Airflow features section for detailed implementation guidance on each pattern. Astronomer recommends deploying Airflow pipelines using a [free trial of Astro](https://www.astronomer.io/lp/signup/).


# ELT with Snowflake and Apache Airflow® for eCommerce
Source: https://astronomer.io/docs/learn/reference-architecture-elt-snowflake

Learn how to build an end-to-end ELT pipeline with Apache Airflow® and Snowflake.

## Overview

This reference architecture shows how to build an ELT pipeline that ingests eCommerce transaction data, loads it to [Snowflake](https://www.snowflake.com/), transforms it through multiple SQL layers, runs data quality checks at each stage, and displays the results in a [Streamlit](https://streamlit.io/) dashboard. [Apache Airflow®](https://airflow.apache.org/) orchestrates the entire flow across multiple Dags that are chained together using data-aware scheduling. A demo of the architecture is shown in the [Implementing reliable ETL and ELT pipelines with Airflow and Snowflake](https://www.astronomer.io/events/webinars/implementing-reliable-etl-elt-pipelines-with-airflow-and-snowflake-video/) webinar.

<Frame>
  <img alt="Screenshot of dashboard showing data about tea sales." />
</Frame>

The architecture demonstrates a pattern common in analytics teams: extracting from an API, staging raw files in object storage with a clear lifecycle (ingest, stage, archive), loading to a warehouse, running SQL transformations with built-in data quality gates, and surfacing the results in a dashboard. You can adapt it by swapping the data source, adjusting the SQL transformations, or replacing the Streamlit dashboard.

## Architecture

<Frame>
  <img alt="Snowflake reference architecture diagram." />
</Frame>

This reference architecture consists of four main components:

* **Extraction**: An Airflow Dag calls the eCommerce store's API and writes the response data (customers, orders, products) to an object storage bucket. Each record type is extracted as a separate file, and only new or updated records are fetched on each run.
* **Loading**: A second Dag picks up the files from object storage and loads them into Snowflake raw tables. Each record type is loaded into its own table.
* **Transformation**: Once the raw tables are populated, SQL queries transform the data through multiple layers (base tables, intermediate joins, reporting views). Data quality checks run at the base table level to catch issues before they propagate downstream.
* **Dashboard**: The transformed data is displayed in a Streamlit dashboard that visualizes sales metrics and trends.

In addition to the main pipeline, two housekeeping Dags manage the object storage lifecycle by moving raw files from ingest to stage to archive as they are processed. This keeps the ingest location clean and provides an audit trail of all ingested data.

Data flows through the system in stages: API to object storage to Snowflake raw tables to transformed views to dashboard. Each Dag handles one phase and triggers the next through data-aware scheduling, so downstream work only starts when upstream data is ready.

### Airflow features

* [Object Storage](/docs/learn/airflow-object-storage-tutorial): The Airflow Object Storage API simplifies moving files between object storage locations (ingest, stage, archive) without writing provider-specific code. Files are streamed between paths, which keeps memory usage low even for large extracts.
* [Data-aware scheduling](/docs/learn/airflow-datasets): The extraction Dag runs on a time-based schedule, but the loading, transformation, and housekeeping Dags use assets to trigger only when the data they depend on has been updated. This chains the Dags together without hard-coded dependencies and ensures each phase runs exactly when its input data is ready.
* [Data quality checks](/docs/learn/airflow-sql-data-quality): Data quality checks run on the base Snowflake tables using the `SQLColumnCheckOperator` (validating column-level constraints like non-null and value ranges) and the `SQLTableCheckOperator` (validating table-level conditions like row counts). Some checks are blocking and stop the pipeline on failure, while others are non-blocking and only send a notification.
* [Notifications](/docs/learn/error-notifications-in-airflow): Non-blocking data quality check failures trigger an automatic Slack notification to the data quality team using an `on_failure_callback` at the task group level, so the team is informed without halting the entire pipeline.
* [Airflow retries](/docs/learn/rerunning-dags#automatically-retry-tasks): All tasks that interact with external services (the eCommerce API, object storage, Snowflake) are configured to automatically retry after an adjustable delay to handle transient failures.
* [Dynamic task mapping](/docs/learn/dynamic-tasks): During extraction, one mapped task is created per record type. The number of files is determined at runtime, so the Dag adapts automatically when new record types are added.
* [Custom XCom backend](/docs/learn/custom-xcom-backends-tutorial): Extracted records are passed between the extraction and loading Dags through XCom. Because the payloads can be large, XComs are stored in S3 using an Object Storage custom XCom backend instead of the Airflow metadata database.
* Modularization: SQL queries are stored in the `include` folder and executed by `SQLExecuteQueryOperator` tasks in the Dags. Python helper functions and data quality check definitions are modularized as well, separating orchestration logic from business logic and making individual components reusable across Dags.

## Next steps

To build your own ELT pipeline with Snowflake and Apache Airflow, explore the individual Learn guides linked in the Airflow features section for detailed implementation guidance on each pattern. Astronomer recommends deploying Airflow pipelines using a [free trial of Astro](https://www.astronomer.io/lp/signup/).


# ETL with DuckDB and Apache Airflow® for travel analytics
Source: https://astronomer.io/docs/learn/reference-architecture-etl-duckdb

Learn how to build an ETL pipeline with Apache Airflow®, DuckDB, and built-in data quality gates.

## Overview

This reference architecture shows how to build an ETL pipeline for a fictional interplanetary travel company. It ingests booking data, transforms it into a daily revenue report per destination in [DuckDB](https://duckdb.org/), and validates the output with data quality checks before publishing it as an asset for downstream consumers. A single [Apache Airflow®](https://airflow.apache.org/) Dag orchestrates the entire flow: ingest, transform, validate, publish.

Using a local DuckDB file makes this architecture a lightweight way to get started with an ETL pipeline without provisioning any external infrastructure. Because the pipeline uses Airflow's common SQL operators (`SQLExecuteQueryOperator`, `SQLColumnCheckOperator`), it is warehouse-agnostic by design. These operators rely on DB-API 2.0, Python's standard database interface, so the same Dag works with Snowflake, BigQuery, Postgres, or any other compliant database by changing only the Airflow connection. For production workloads, replace the local DuckDB file with a cloud-hosted option like [MotherDuck](https://motherduck.com/) or swap DuckDB for a traditional data warehouse entirely, without changing the Dag logic.

## Architecture

<Frame>
  <img alt="DuckDB reference architecture diagram." />
</Frame>

A single Dag runs on a `@daily` schedule and executes four tasks in sequence:

* **Ingest**: The first task calls `SQLExecuteQueryOperator` with a Jinja-templated SQL file that generates booking and payment records. Each run produces a configurable number of bookings (`params.n_bookings`), each with a random customer, route, passenger count, and optional promo code discount. The fare calculation happens inline in SQL: `passengers x base_fare x planet_multiplier x (1 - discount_pct)`.
* **Transform**: A second `SQLExecuteQueryOperator` runs an aggregation query that joins bookings with routes, destinations, payments, and promo codes. It groups the data by report date and destination, classifying each trip as active or completed and summing passengers, gross fares, discounts, net fares, and paid amounts. The result is upserted into a `daily_planet_report` table using ON CONFLICT DO UPDATE, so re-runs for the same date overwrite rather than duplicate data.
* **Validate**: A `SQLColumnCheckOperator` runs data quality checks on the report table before any downstream work proceeds. It validates that `planet_name` has no nulls and at least three distinct values, and that `total_passengers` has no nulls and a minimum value of one. If any check fails, the task fails and the asset isn't published.
* **Publish**: After validation passes, the Dag publishes a `daily_report` asset. Any downstream Dag that schedules on this asset (for example, a Dag that formats the report for a dashboard or sends a Slack summary) only triggers when fresh, validated data is available.

Data flows in a clear sequence within the single Dag: raw bookings and payments are generated, aggregated into a daily report per destination, validated against quality constraints, and then published as an asset that signals readiness to downstream consumers.

### Airflow features

* [Data-aware scheduling](/docs/learn/airflow-datasets): The Dag publishes a `daily_report` asset after successful validation. Downstream Dags schedule themselves on this asset, so they only trigger when fresh, quality-checked data is available. This decouples producers from consumers without hard-coded cross-Dag dependencies.
* [Data quality checks](/docs/learn/data-quality): `SQLColumnCheckOperator` validates the report table before the asset is published. Checks include non-null constraints on destination names, distinct value thresholds (at least three destinations), and minimum passenger counts. Failed checks block the asset update, which prevents downstream Dags from triggering on bad data. See [Run data quality checks using SQL check operators](/docs/learn/airflow-sql-data-quality) for detailed usage of all available SQL check operators.
* [Jinja templating in SQL](/docs/learn/templating): The Dag uses two distinct approaches to dynamic SQL. The ingestion step uses `params` with Jinja loops and conditionals to control the SQL structure itself (how many bookings to generate per run). The transformation step uses `parameters` to pass the execution date as a database-level bound parameter (`$reportDate`), which preserves type safety and protects against SQL injection. This separation, Jinja for structure, parameters for values, is a best practice for any SQL-heavy pipeline.
* [Airflow retries](/docs/learn/rerunning-dags#automatically-retry-tasks): All tasks are configured to automatically retry after a set delay to handle transient failures when interacting with the database.
* Modularization: SQL queries are stored in the `include/sql` folder and referenced by task using `template_searchpath`. The Dag file contains only orchestration logic (task definitions, dependencies, parameters), while the business logic lives entirely in SQL files. This separation makes the SQL independently testable and reusable across Dags.
* Idempotent design: The transformation query uses ON CONFLICT DO UPDATE (upsert) keyed on report date and destination. Re-running the Dag for the same date produces the same result without duplicating rows, making the pipeline safe to retry or backfill at any time.

<Info>
  **params vs parameters in SQLExecuteQueryOperator**

  The `SQLExecuteQueryOperator` supports two ways to pass values into SQL, and this architecture uses both:

  * **`params`**: Used for Jinja template rendering. Values are accessed in SQL files using `{{ params.param_name }}` and rendered as strings by default. Use `params` when the SQL structure itself needs to change, such as looping to generate a variable number of INSERT statements or conditionally including SQL clauses. Note that `params` aren't templated themselves.
  * **`parameters`**: Used for database-level parameterized queries. Values are passed directly to the database driver using syntax like `$param_name` or `%(param_name)s`, preserving native Python types (integers, floats, booleans). Use `parameters` when passing values into a fixed SQL structure, especially user-provided values, since the database driver handles escaping and type safety. Note that `parameters` are templated, so you can use Airflow macros like `{{ ds }}` in their values.

  Both approaches enable dynamic SQL, but they operate at different stages of query processing. Use `params` for structural flexibility (Jinja), `parameters` for safe value binding (database driver).
</Info>

## Considerations

* **DuckDB concurrency**: DuckDB supports concurrent reads but only a single writer at a time. Tasks that write to DuckDB should be sequenced with explicit dependencies or use `max_active_tis_per_dagrun=1` to avoid write conflicts. For production workloads with multiple concurrent writers, consider [MotherDuck](https://motherduck.com/) (managed DuckDB) or a traditional cloud warehouse.
* **Portability**: Because DuckDB is file-based and runs embedded, the entire pipeline is portable and runs identically on a laptop, in CI, or on Astro without provisioning external infrastructure. This makes it well-suited for development, testing, and small-scale analytics. For larger datasets, swap DuckDB for a cloud warehouse like Snowflake or BigQuery. The Dag code, SQL files, and data quality checks stay the same since Airflow's common SQL operators work with any DB-API 2.0 compliant database. Only the Airflow connection changes.
* **Data quality as a gate**: Placing the validation step before the asset publication means downstream consumers never see invalid data. This is a deliberate tradeoff: a validation failure blocks the entire downstream chain. For pipelines where partial results are acceptable, consider separating blocking checks (which stop the pipeline) from non-blocking checks (which send a notification but allow the pipeline to continue).

## Next steps

To build your own ETL pipeline with DuckDB, Snowflake, or any other SQL database supported, explore the individual Learn guides linked in the Airflow features section for detailed implementation guidance on each pattern. Astronomer recommends deploying Airflow pipelines using a [free trial of Astro](https://www.astronomer.io/lp/signup/).


# Processing User Feedback: an LLM-fine-tuning reference architecture with Ray on Anyscale
Source: https://astronomer.io/docs/learn/reference-architecture-fine-tuning-anyscale

Learn how to fine-tune an LLM to process and categorize user feedback with Airflow and Ray on Anyscale.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

The Processing User Feedback [GitHub repository](https://github.com/astronomer/astronomer-processing-user-feedback) is a free and open-source reference architecture showing how to use [Apache Airflow®](https://airflow.apache.org/) with [Anyscale](http://anyscale.com), a distributed compute platform built on [Ray](https://www.ray.io/), to build an automated system that processes and categorizes user feedback relating to video games using a fine-tuned Large Language Model (LLM). The repository includes full source code, documentation, and deployment instructions for you to adapt and implement this architecture in your own projects.

<Frame>
  <img alt="Screenshot of the Airflow UI showing the graph view of the Finetune_llmm_and_deploy_challenger DAG from the reference architecture." />
</Frame>

This reference architecture serves as a practical learning tool, illustrating how to use Apache Airflow to orchestrate fine-tuning of LLMs on the Anyscale platform. The Processing User Feedback application is designed to be adaptable, allowing you to tailor it to your specific use case. You can customize the workflow by:

* Changing the data that is ingested for fine-tuning and inference.
* Modifying the Anyscale jobs and services to align with your requirements.
* Adjusting the data processing steps and model fine-tuning parameters.

By providing a flexible framework, this architecture enables developers and data scientists to implement and scale their own LLM-based feedback processing systems using distributed compute.

<Note>
  This tutorial uses [Anyscale](http://anyscale.com) with the [Anyscale provider](https://github.com/astronomer/astro-provider-anyscale) to run Ray jobs. If you want to run Ray jobs on other platforms, you can use the [Ray provider](https://github.com/astronomer/astro-provider-ray/) instead. See also [Orchestrate Ray jobs on Anyscale with Apache Airflow®](/docs/learn/airflow-anyscale).
</Note>

## Architecture

<Frame>
  <img alt="Processing User Feedback reference architecture diagram." />
</Frame>

The Processing User Feedback use case consists of 2 main components:

* **Data ingestion**: new user feedback about video games is collected from several APIs, preprocessed, and stored in an [S3 bucket](https://aws.amazon.com/s3/).
* **Fine-tuning and deploying of Mistral-7B**: once a threshold of 200 new feedback entries is reached, the data is used to fine-tune a pre-trained LLM model, [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), on Anyscale using distributed compute. The fine-tuned model is deployed using [Anyscale Services](https://docs.anyscale.com/1.0.0/services/get-started/).

Additionally, the architecture includes an advanced [champion-challenger version](https://github.com/astronomer/astronomer-processing-user-feedback/blob/main/dags/automated_retraining/retrain.py) of the fine-tuning process.

### Airflow features

The DAGs in this reference architecture highlight several key Airflow features and best practices:

* [Branching](/docs/learn/airflow-branch-operator): Using Airflow Branching, DAGs can execute different paths based on runtime conditions or results from previous tasks. This allows for dynamic workflow adjustments depending on the data or processing requirements. In this reference architecture branching is used to determine whether the fine-tuning process should be executed.
* [Airflow retries](/docs/learn/rerunning-dags#automatically-retry-tasks): To protect against transient API failures and rate limits, all tasks are configured to automatically retry after an adjustable delay.
* [Dynamic task mapping](/docs/learn/dynamic-tasks): Transforming data from multiple data sources is split into multiple parallelized tasks using dynamic task mapping. The number of parallelized tasks is determined at runtime based on the number of data sources that need to be processed.
* [Data-aware scheduling](/docs/learn/airflow-datasets): The DAGs run on a data-driven schedule to regularly and automatically update the LLM model when new data has been ingested. Aside from data-driven scheduling, Airflow offers options such as [time-based scheduling](/docs/learn/scheduling-in-airflow) or scheduling based on external events detected using [sensors](/docs/learn/what-is-a-sensor).
* [Task groups](/docs/learn/task-groups): In the champion-challenger DAG, related tasks are organized into logical groups within the DAG with Airflow task groups. This improves the overall structure of complex workflows and makes them easier to understand and maintain.

## Next steps

Get the [Astronomer GenAI cookbook](https://www.astronomer.io/ebooks/gen-ai-airflow-cookbook/) to view more examples of how to use Airflow to build generative AI applications.

If you'd like to build your own pipeline using Anyscale with Airflow, feel free to fork the [repository](https://github.com/astronomer/astronomer-processing-user-feedback) and adapt it to your use case. We recommend deploying the Airflow pipelines using a [free trial of Astro](https://www.astronomer.io/lp/signup/).


# Hybrid Search for eCommerce reference architecture
Source: https://astronomer.io/docs/learn/reference-architecture-hybrid-search

Learn how to build a hybrid search application with Apache Airflow® and Weaviate.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

The Hybrid Search for eCommerce [GitHub repository](https://github.com/astronomer/astronomer-weaviate-hybrid-search) is a free and open-source reference architecture showing how to use [Apache Airflow®](https://airflow.apache.org/) with [Weaviate](https://weaviate.io/) to build an automated hybrid search application. A demo of the architecture was shown in the [Modern Infrastructure for World Class AI Applications](https://www.astronomer.io/events/webinars/modern-infrastructure-for-world-class-ai-applications-video/) webinar.

<Frame>
  <img alt="Screenshot of the Hybrid Search application frontend." />
</Frame>

This reference architecture demonstrates how to use Apache Airflow to orchestrate RAG data ingestion that powers a search application as well as a batch inference pipeline analyzing search queries. It also shows how to use Weaviate's advanced search capabilities. You can adapt the Hybrid Search application to your use case by ingesting your own data and adjusting the search queries in the website backend to fit your needs.

## Architecture

<Frame>
  <img alt="Hybrid search reference architecture diagram." />
</Frame>

The hybrid search reference architecture consists of 3 main components:

* **Data ingestion and embedding**: Sample data containing product descriptions and images is ingested from [Amazon S3](https://aws.amazon.com/s3/) and [Snowflake](https://www.snowflake.com/) into [Weaviate](https://weaviate.io/), a vector database. Embedding of the product descriptions uses [OpenAI](https://platform.openai.com/docs/api-reference/introduction) models.
* **Hybrid search**: The demo website with a [Flask](https://flask.palletsprojects.com/en/2.0.x/) backend and [React](https://reactjs.org/) frontend allows users to experiment with advanced Weaviate search by querying the product descriptions using hybrid search. An OpenAI embedding model is used to embed the user query.
* **Batch inference**: All user search queries are stored back in Weaviate so they can be used by a downstream Airflow DAG that runs an OpenAI batch inference pipeline to classify user queries and derive product insights. The results of this analysis are loaded into Snowflake to be displayed in a [Streamlit](https://streamlit.io/) dashboard.

### Airflow features

The DAGs that power this hybrid search application highlight several key Airflow best practices and features:

* [Airflow retries](/docs/learn/rerunning-dags#automatically-retry-tasks): To protect against transient API failures and rate limits, all tasks are configured to automatically retry after an adjustable delay.
* [Advanced data-driven scheduling](/docs/learn/airflow-datasets): The DAGs in this reference architecture run on data-driven schedules, including combined [asset and time scheduling](/docs/learn/airflow-advanced-asset-scheduling#combined-asset-and-time-based-scheduling) and [conditional asset scheduling](/docs/learn/airflow-advanced-asset-scheduling#conditional-asset-scheduling).
* [Dynamic task mapping](/docs/learn/dynamic-tasks): Product information extraction and ingestion into Weaviate are split into multiple parallelized tasks, the number of which is determined at runtime based on the number of ingestion folders with product information that needs to be processed.
* [Object Storage](/docs/learn/airflow-object-storage-tutorial): Interaction with files in object storage is simplified using the experimental Airflow Object Storage API.
* Modularization: Functions defining how information is extracted and checksums are calculated are modularized in the [`include`](https://github.com/astronomer/astronomer-weaviate-hybrid-search/blob/main/include/functions/utils.py) folder and imported into the DAGs. This makes the DAG code more readable and offers the ability to reuse functions across multiple DAGs.

## Next steps

Get the [Astronomer GenAI cookbook](https://www.astronomer.io/ebooks/gen-ai-airflow-cookbook/) to view more examples of how to use Airflow to build generative AI applications.

If you'd like to build your own hybrid search application, feel free to fork the [repository](https://github.com/astronomer/astronomer-weaviate-hybrid-search) and adapt it to your use case. We recommend to deploy the Airflow pipelines using a [free trial of Astro](https://www.astronomer.io/lp/signup/).


# Integration patterns for Apache Kafka® and Apache Airflow®
Source: https://astronomer.io/docs/learn/reference-architecture-kafka

Learn how to integrate Apache Kafka® with Apache Airflow® for real-time event processing using event-driven scheduling and the Airflow Kafka provider.

## Overview

[Apache Kafka®](https://kafka.apache.org/) is one of the most widely adopted platforms for real-time event streaming. Many data teams need to process the events flowing through Kafka in batches to use them in analytics, machine learning, and other pipelines that Airflow orchestrates, or run Airflow Dags in reaction to specific events in a Kafka topic.

This reference architecture shows two patterns for integrating Kafka with Airflow:

1. Event-driven pattern: Trigger a Dag as soon as a specific event message arrives in a Kafka topic.
2. Standard pattern: Land streaming data in object storage, where Airflow picks it up for batch processing at scale.

## Event-driven pattern

<Frame>
  <img alt="Event-driven Kafka architecture diagram showing Kafka topics triggering individual Airflow Dags." />
</Frame>

In this pattern, when a message arrives in a topic, Airflow's event-driven scheduling detects it, processes it, and triggers a Dag. The Dag then processes the message content for a use case such as running an inference call using an AI agent, updating a dashboard, or kicking off an operational workflow.

This architecture consists of three main components:

* **Kafka topics**: Multiple topics ingest real-time event data from different sources, such as application events, IoT sensors, or third-party APIs.
* **Airflow Dags**: Each Dag is associated with one or more Kafka topics through an AssetWatcher. When the watcher detects a new message, it creates an AssetEvent that triggers the Dag. The message payload is available to tasks through the Airflow context.
* **Downstream data products**: The triggered Dags produce business-critical outputs such as agentic AI inference results, fine-grained dashboard updates, or operational workflow executions.

This pattern is best suited for use cases where each message requires its own Dag run and low-latency orchestration of complex workflows is required.

## Standard pattern

<Frame>
  <img alt="Standard Kafka architecture diagram showing Kafka topics flowing through Kafka Sinks to Airflow Dags with many tasks." />
</Frame>

In this pattern, Kafka Sinks connect to topics and land streaming data in object storage such as Amazon S3 or Google Cloud Storage. Airflow Dags detect new data in storage using deferrable operators and then process the events in batches.

This architecture consists of four main components:

* **Kafka topics**: Multiple topics ingest real-time event data from different sources, identical to the event-driven pattern.
* **Kafka Sinks**: Kafka Connect sink connectors consume messages from topics and write them to object storage in batches. Each sink can pull from one or more topics.
* **Airflow Dags**: Sets of Dags with tens to thousands of tasks process data from any number of Kafka sinks. Deferrable operators detect new files in storage without occupying worker slots, and dynamic task mapping parallelizes processing across files. Tasks can also produce messages directly back to Kafka topics or consume directly from them using the [Airflow Kafka provider](https://airflow.apache.org/docs/apache-airflow-providers-apache-kafka/stable/index.html).
* **Downstream data products**: Dags produce outputs such as executive dashboards, customer-facing analytics, or trained ML models.

This pattern is best suited for use cases where event data needs to be processed in batches.

## Airflow features

* [**Event-driven scheduling**](/docs/learn/airflow-event-driven-scheduling): In the event-driven pattern, an AssetWatcher with a `MessageQueueTrigger` polls Kafka topics for new messages. When a message arrives, the watcher creates an AssetEvent that triggers the associated Dag, passing the message payload through the Airflow context.
* [**Data-aware scheduling**](/docs/learn/airflow-datasets): Both patterns use assets to connect producers and consumers. In the event-driven pattern, Kafka messages update assets that trigger Dags. In the standard pattern, Dags can publish assets after processing completes, starting downstream Dags when the data is ready.
* [**Dynamic task mapping**](/docs/learn/dynamic-tasks): In the standard pattern, dynamic task mapping parallelizes file processing from object storage. The number of mapped tasks is determined at runtime based on the amount of files that have been landed in object storage, with options to create one dynamically mapped task instance per file or per subdirectory.
* [**Airflow Kafka provider**](https://airflow.apache.org/registry/providers/apache-kafka/): In the standard pattern, the `ProduceToTopicOperator` and `ConsumeFromTopicOperator` enable direct bidirectional communication between Airflow tasks and Kafka topics. Tasks can consume messages from Kafka as part of a processing pipeline or produce results back to a topic for consumption by other systems.
* [**Deferrable operators**](/docs/learn/deferrable-operators): In the standard pattern, deferrable operators such as the `S3KeySensor` in `deferrable=True` mode detect new files written by Kafka Sinks. The deferrable operator hands off polling to the triggerer, freeing the worker slot until new data appears.

## Considerations

* **Choosing between patterns**: The event-driven pattern triggers a Dag run per message with lower latency, which suits use cases that require immediate processing of individual events. The standard pattern batches messages through Kafka Sinks and processes them in bulk, which is more efficient and allows you to compute aggregates over the data. Many production architectures use both patterns for different topics based on the downstream requirements.
* **Kafka Sink configuration and Dag schedules**: In the standard pattern, two factors affect how often Dags are triggered. The Dag schedule determines how often a new Dag run starts, with its first task waiting in `deferrable` mode for new data in the object storage location. The batch size and flush interval of your Kafka Sinks determine how often new data arrives there. For highly irregular patterns, consider a `@continuous` schedule with `max_active_runs=1` so there is always exactly one Dag run ready to process new data as soon as it arrives.
* **Scaling the event-driven pattern**: Each message in the event-driven pattern triggers a separate Dag run. At high message rates, this can create many concurrent Dag runs. Configure concurrency limits and pool slots to match your infrastructure capacity, and consider switching to the standard pattern for topics with sustained high throughput.

## Next steps

To learn more about the individual Airflow features used in this architecture, explore the Learn guides linked in the Airflow features section. For more information on the Airflow Kafka provider, see the [Airflow Kafka provider documentation](https://airflow.apache.org/docs/apache-airflow-providers-apache-kafka/stable/index.html).


# Batch inference for product insights with Apache Airflow®
Source: https://astronomer.io/docs/learn/reference-architecture-product-insights

Learn how to build a batch inference pipeline with Apache Airflow® and OpenAI.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

The [Batch inference for product insights](https://github.com/astronomer/batch-inference-product-insights) repository is a free and open-source reference architecture showing how to use [Apache Airflow®](https://airflow.apache.org/) and [OpenAI](https://platform.openai.com/docs/api-reference/introduction) to summarize product feedback and generate insights from it. The full source code is available on [GitHub](https://github.com/astronomer/batch-inference-product-insights).

<Frame>
  <img alt="Screenshot of a Slack message showing a product summary generated by the pipeline." />
</Frame>

This reference architecture was created as a learning tool to demonstrate how to use Apache Airflow to orchestrate data ingestion, tagging of feedback with relevant products, and per-product feedback summarization in a batch inference pipeline. You can adapt the pipeline for your use case by ingesting data from other sources and adjust the LLM model prompts to fit your needs.

## Architecture

<Frame>
  <img alt="Batch inference reference architecture diagram." />
</Frame>

This batch inference pipeline consists of 4 main components:

* **Data ingestion and embedding**: Product feedback is ingested from a variety of sources. The [`ingest_zendesk_tickets`](https://github.com/astronomer/batch-inference-product-insights/blob/main/dags/ingest_zendesk_tickets.py) DAG extracts feedback from Zendesk tickets stored in [Snowflake](https://www.snowflake.com/), the [`ingest_data_apis`](https://github.com/astronomer/batch-inference-product-insights/blob/main/dags/ingest_data_apis.py) DAG extracts feedback from the [GitHub](https://docs.github.com/en/rest) and [StackOverflow](https://api.stackexchange.com/) APIs, as well as from local files containing [G2](https://www.g2.com/) reviews.
* **Product/feature tagging**: Using [OpenAI](https://platform.openai.com/docs/api-reference/introduction), the feedback is tagged with the relevant product or feature.
* **Create feedback summaries and insights**: All feedback relating to one product/feature is aggregated and summarized using GPT-4o. The summaries are posted to a Slack channel.
* **Executive summary**: A final DAG aggregates all product summaries and insights into an executive summary that is posted to a Slack channel.

### Airflow features

The DAGs that power the batch inference pipelines highlight several key Airflow best practices and features:

* [Dynamic task mapping](/docs/learn/dynamic-tasks): Dynamic task mapping is used extensively to parallelize tasks throughout the pipeline. For example, feedback summarization and insight generation is parallelized to create one dynamically mapped task instance per product tag that is analyzed. Custom map indexing is used to make it easier to find specific summaries in the task logs.
* [Object Storage](/docs/learn/airflow-object-storage-tutorial): Interaction with files in object storage is simplified using the experimental Airflow Object Storage API.
* [Airflow retries](/docs/learn/rerunning-dags#automatically-retry-tasks): To protect against transient API failures and rate limits, all tasks are configured to automatically retry after an adjustable delay.
* [Advanced data-driven scheduling](/docs/learn/airflow-datasets): The DAGs in this reference architecture run on data-driven schedules, including [conditional asset scheduling](/docs/learn/airflow-advanced-asset-scheduling#conditional-asset-scheduling).
* Modularization: The [`ingest_data_apis`](https://github.com/astronomer/batch-inference-product-insights/blob/main/dags/ingest_data_apis.py) DAG serves as an example of a high level of modularization. Task functions are stored in the `include` folder and imported into the DAG file to be used in [`@task`](/docs/learn/airflow-decorators) decorators. Ingestion sources are defined in a list of configurations with a loop generating one parallel ingestion track per source.

## Next steps

Get the [Astronomer GenAI cookbook](https://www.astronomer.io/ebooks/gen-ai-airflow-cookbook/) to view more examples of how to use Airflow to build generative AI applications.

If you'd like to build your own batch inference pipeline, feel free to fork the [repository](https://github.com/astronomer/batch-inference-product-insights) and adapt it to your use case. We recommend deploying the Airflow pipelines using a [free trial of Astro](https://www.astronomer.io/lp/signup/).


# SnowPatrol: Snowflake Usage Anomaly Detection & Alerting System
Source: https://astronomer.io/docs/learn/reference-architecture-snowpatrol

Learn how to build an anomaly detection system for Snowflake with Apache Airflow®.

<Info>
  This is a reference architecture meant to serve as inspiration for how to use Airflow. The SnowPatrol repository may not be actively maintained. If you're looking for production-ready Snowflake cost management, consider [Astro Observe](https://www.astronomer.io/product/observe/).
</Info>

[SnowPatrol](https://github.com/astronomer/snowpatrol) is an anomaly detection and alerting system for [Snowflake](https://www.snowflake.com/) built with [Apache Airflow®](https://airflow.apache.org/). It uses an [Isolation Forest](https://en.wikipedia.org/wiki/Isolation_forest) model to detect anomalies in your Snowflake compute driving cost. Airflow Dags orchestrate feature engineering, model training, and prediction. Anomaly records are stored in a Snowflake table and alerts are sent to a [Slack](https://slack.com/) channel. The full source code is open-source and available on [GitHub](https://github.com/astronomer/snowpatrol).

<Frame>
  <img alt="Screenshot a chart depicting anomalies detected for different Warehouses in Snowflake." />
</Frame>

SnowPatrol serves a dual purpose:

* **Resource for Airflow + Snowflake users**: Many organizations use Snowflake and have experienced challenges in managing associated costs, especially incurred from virtual warehouse compute. SnowPatrol is a tool organizations can use to be alerted of anomalies in their Snowflake compute usage, and take action to reduce costs.
* **Learning Tool**: SnowPatrol is an MLOps reference implementation, showing how you can use Airflow to train, test, deploy, and monitor predictive models. The structure of the Dags can be adapted to other use cases, such as fraud detection.

## Architecture

<Frame>
  <img alt="SnowPatrol reference architecture diagram." />
</Frame>

SnowPatrol performs the following steps:

* **Data Ingestion and Feature Engineering**: Snowflake usage data is persisted in a Snowflake table to create a cost time series. [STL decomposition](https://www.statsmodels.org/dev/examples/notebooks/generated/stl_decomposition.html) is used to extract trends, seasonality, and residuals.
* **Model Training**: An Isolation Forest model is trained on the features to detect anomalies. Model training and versions are tracked using [Weights & Biases](https://wandb.ai/site).
* **Predictions**: The trained model is used to predict anomalies in Snowflake usage data.
* **Reporting and Alerting**: Anomalies are stored in a Snowflake table and alerts are sent to a Slack channel.

### Airflow features

The Dags that power SnowPatrol highlight several key Airflow best practices and features:

* [Dynamic task mapping](/docs/learn/dynamic-tasks): Every time the Dags run, the list of warehouses in Snowflake is determined at runtime and Airflow automatically creates a dynamically mapped task instance per warehouse for model training and prediction to run in parallel.
* [Data-aware scheduling](/docs/learn/airflow-datasets): While the first Dag runs on a time-based `@daily` schedule, all other Dags are triggered based on updates to the datasets they depend upon.
* [Notifications](/docs/learn/error-notifications-in-airflow): If any task fails, a Slack notification is automatically sent using an `on_failure_callback` in the `default_args` of the Dags.
* [Airflow retries](/docs/learn/rerunning-dags#automatically-retry-tasks): All tasks are configured to automatically retry after a set delay.
* Modularization: SQL statements are stored in the [`include`](https://github.com/astronomer/snowpatrol/tree/main/include/sql) folder and executed by [`SQLExecuteQueryOperator`](https://airflow.apache.org/registry/providers/common-sql#common-sql-sql-SQLExecuteQueryOperator) in the Dag. This makes the Dag code more readable and offers the ability to reuse SQL queries across multiple Dags.


# Run Soda Core checks with Airflow
Source: https://astronomer.io/docs/learn/soda-data-quality

Learn how to orchestrate Soda Core data quality checks with your Airflow DAGs.

<Info>
  This page hasn't yet been updated for Airflow 3. The concepts shown are relevant, but some code may need to be updated. If you run any examples, take care to update import statements and watch for any other breaking changes.
</Info>

[Soda Core](https://www.soda.io/core) is an open source framework for checking data quality. It uses the Soda Checks Language (SodaCL) to run checks defined in a YAML file.

Soda Core lets you:

* Define checks as YAML configuration, including many preset checks.
* Provide a SQL query within the YAML file and check against a returned value if no preset checks fit your use case.
* Integrate data quality checks with commonly used data engineering tools such as Airflow, Apache Spark, PostgreSQL, Snowflake [and more](https://www.soda.io/integrations).

In this tutorial, you'll learn about the key features of Soda Core and how to use Airflow to run data quality checks on a database.

## Time to complete

This tutorial takes approximately 30 minutes to complete.

## Assumed knowledge

To get the most out of this tutorial, make sure you have an understanding of:

* How to design a data quality process. See [Data quality and Airflow](/docs/learn/data-quality).
* The basics of Soda Core. See [How Soda Core works](https://docs.soda.io/soda-core/how-core-works.html).
* How to use the `BashOperator`. See [Using the `BashOperator`](/docs/learn/bashoperator).
* Relational Databases. See [IBM's "Relational Databases Explained."](https://www.ibm.com/cloud/learn/relational-databases)
* Familiarity with writing YAML configurations. See [yaml.org](https://yaml.org/).

## Prerequisites

To complete this tutorial, you need:

* The Soda Core package for your database backend. The [Soda documentation](https://docs.soda.io/) provides a list of supported databases and how to configure them. This tutorial uses Snowflake.
* The [Astro CLI](/docs/cli/v1.43/get-started-cli).

## Step 1: Configure your Astro project

Configure a new Astro project to run Airflow locally.

1. Create a new Astro project:

   ```sh wrap theme={null}
   $ mkdir astro-soda-tutorial && cd astro-soda-tutorial
   $ astro dev init
   ```

2. Add the following line to the `requirements.txt` file of your Astro project:

   ```text wrap theme={null}
   soda-core-snowflake
   ```

   This installs the relevant Soda Core Python package. If you are using a different database backend, replace `snowflake` with your backend. See the [Prerequisites](#prerequisites) section for more details.

3. Run the following command to start your project in a local environment:

   ```sh wrap theme={null}
   astro dev start
   ```

## Step 2: Create the configuration file

Create a configuration file to connect to your database backend. The following example uses the template from the Soda documentation to create the configuration file for Snowflake.

```yaml wrap theme={null}
# the first line names the datasource "MY_DATASOURCE"
data_source MY_DATASOURCE:
  type: snowflake
  connection:
    # provide your snowflake username and password in double quotes
    username: "MY_USERNAME"
    password: "MY_PASSWORD"
    # provide the account in the format xy12345.eu-central-1
    account: my_account
    database: MY_DATABASE
    warehouse: MY_WAREHOUSE
    # if your connection times out you may need to adjust the timeout value
    connection_timeout: 300
    role: MY_ROLE
    client_session_keep_alive:
    session_parameters:
      QUERY_TAG: soda-queries
      QUOTED_IDENTIFIERS_IGNORE_CASE: false
  schema: MY_SCHEMA
```

Save the YAML instructions in a file named `configuration.yml` and place the file into the `/include` directory of your Astro project.

## Step 3: Create the checks file

Define your data quality checks using the [many preset checks available for SodaCL](https://docs.soda.io/soda-cl/soda-cl-overview.html). For more details on creating checks, see [How it works](#how-it-works). If you can't find a preset check that works for your use case, you can create a custom one using SQL as shown in the following example.

```yaml wrap theme={null}
checks for example_table:
  # check that MY_EMAIL_COL contains only email addresses according to the
  # format name@domain.extension
  - invalid_count(MY_EMAIL_COL) = 0:
      valid format: email
  # check that all entries in MY_DATE_COL are unique
  - duplicate_count(MY_DATE_COL) = 0
  # check that MY_TEXT_COL has no missing values
  - missing_count(MY_TEXT_COL) = 0
  # check that MY_TEXT_COL has at least 10 distinct values using SQL
  - distinct_vals >= 10:
      distinct_vals query: |
        SELECT DISTINCT(MY_TEXT_COL) FROM example_table
  # check that MY_NUM_COL has a minimum between 90 and 110
  - min(MY_NUM_COL) between 90 and 110
  # check that example table has at least 1000 rows
  - row_count >= 1000
  # check that the sum of MY_COL_2 is bigger than the sum of MY_COL_1
  - sum_difference > 0:
      sum_difference query: |
        SELECT SUM(MY_COL_2) - SUM(MY_COL_1) FROM example_table
  # checks that all entries in MY_COL_3 are part of a set of possible values
  - invalid_count(MY_COL_3) = 0:
      valid values: [val1, val2, val3, val4]
```

Save the YAML instructions in a file named `checks.yml` and place the file in the `/include` directory of your Astro project.

## Step 4: Create your DAG

In your Astro project `dags/` folder, create a new file called `soda-pipeline.py`. Paste the following code into the file:

```python wrap theme={null}
from airflow.models.dag import DAG
from datetime import datetime

from airflow.operators.bash import BashOperator

SODA_PATH="<filepath>" # can be specified as an env variable

with DAG(
    dag_id="soda_example_dag",
    schedule='@daily',
    start_date=datetime(2022,8,1),
    catchup=False
) as dag:

    soda_test = BashOperator(
        task_id="soda_test",
        bash_command=f"soda scan -d MY_DATASOURCE -c \
            {SODA_PATH}/configuration.yml {SODA_PATH}/checks.yml"
    )
```

In this DAG, Soda Core checks are executed by using the `BashOperator` to run the `soda scan` command referencing the configuration and check YAML files.

## Step 5: Run your DAG and review check results

Go to the Airflow UI, unpause your `soda_example_dag` DAG, and trigger it to run the Soda Core data quality checks. Go to the task log to see a list of all checks that ran and their results.

This is an example of what your logs might look like when 3 out of 3 checks pass:

```text wrap theme={null}
[2022-08-04, 13:07:22 UTC] {subprocess.py:92} INFO - Scan summary:
[2022-08-04, 13:07:22 UTC] {subprocess.py:92} INFO - 3/3 checks PASSED:
[2022-08-04, 13:07:22 UTC] {subprocess.py:92} INFO -     MY_TABLE in MY_DATASOURCE
[2022-08-04, 13:07:22 UTC] {subprocess.py:92} INFO -       duplicate_count(MY_ID_COLUMN) = 0 [PASSED]
[2022-08-04, 13:07:22 UTC] {subprocess.py:92} INFO -       missing_count(MY_ID_COLUMN) = 0 [PASSED]
[2022-08-04, 13:07:22 UTC] {subprocess.py:92} INFO -       min(MY_NUM_COL) between 0 and 10 [PASSED]
[2022-08-04, 13:07:22 UTC] {subprocess.py:92} INFO - All is good. No failures. No warnings. No errors.
```

In the case of a check failure, the logs show which check failed and the `check_value` that caused the failure:

```text wrap theme={null}
[2022-08-04, 13:23:59 UTC] {subprocess.py:92} INFO - Scan summary:
[2022-08-04, 13:23:59 UTC] {subprocess.py:92} INFO - 2/3 checks PASSED:
[2022-08-04, 13:23:59 UTC] {subprocess.py:92} INFO -     MY_TABLE in MY_DATASOURCE
[2022-08-04, 13:23:59 UTC] {subprocess.py:92} INFO -       duplicate_count(MY_ID_COLUMN) = 0 [PASSED]
[2022-08-04, 13:23:59 UTC] {subprocess.py:92} INFO -       missing_count(MY_ID_COLUMN) = 0 [PASSED]
[2022-08-04, 13:23:59 UTC] {subprocess.py:92} INFO - 1/3 checks FAILED:
[2022-08-04, 13:23:59 UTC] {subprocess.py:92} INFO -     MY_TABLE in MY_DATASOURCE
[2022-08-04, 13:23:59 UTC] {subprocess.py:92} INFO -       max(MY_NUM_COL) between 10 and 20 [FAILED]
[2022-08-04, 13:23:59 UTC] {subprocess.py:92} INFO -         check_value: 3
[2022-08-04, 13:23:59 UTC] {subprocess.py:92} INFO - Oops! 1 failures. 0 warnings. 0 errors. 2 pass.
[2022-08-04, 13:24:00 UTC] {subprocess.py:96} INFO - Command exited with return code 2
[2022-08-04, 13:24:00 UTC] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/airflow/operators/bash.py", line 194, in execute
    raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 2.
```

## How it works

Soda Core uses the Soda Checks Language (SodaCL) to run data quality checks defined in a YAML file. Integrating Soda Core into your Airflow data pipelines lets you use the results of data quality checks to influence downstream tasks. For an overview of SodaCL, see the [SodaCL documentation](https://docs.soda.io/soda-cl/soda-cl-overview.html).

As shown in the following example, you can use Soda Core to run checks on different properties of your dataset against a numerically defined threshold:

```yaml wrap theme={null}
checks for MY_TABLE_1:
  # MY_NUM_COL_1 has a minimum of above or equal 0
  - min(MY_NUM_COL_1) >= 0
  # MY_TEXT_COL has less than 10% missing values
  - missing_percent(MY_TEXT_COL) < 10
checks for MY_TABLE_2:
  # MY_NUM_COL_2 has an average between 100 and 1000
  - avg(MY_NUM_COL_2) is between 100 and 1000
  # MY_ID_COL has no duplicates
  - duplicate_count(MY_ID_COL) = 0
```

You can add optional configurations, such as custom names for checks and error levels:

```yaml wrap theme={null}
checks for MY_TABLE_1:
  # fail the check when MY_TABLE_1 has less than 10 or more than a million rows
  # warn if there are less than 100 (but 10 or more) rows
  - row_count:
      warn: when < 100
      fail:
        when < 10
        when > 1000000
      name: Wrong number of rows!
```

You can use the following methods to check the validity of data:

* List of valid values
* Predefined valid format
* Regex
* SQL query

```yaml wrap theme={null}
checks for MY_TABLE_1:
  # MY_CATEGORICAL_COL has no other values than val1, val2 and val3
  - invalid_count(MY_CATEGORICAL_COL) = 0:
      valid values: [val1, val2, val3]
  # MY_NUMERIC_COL has no other values than 0, 1 and 2.
  # Single quotes are necessary for valid values checks involving numeric
  # characters.
  - invalid_count(MY_NUMERIC_COL) = 0:
      valid values: ['0', '1', '2']
  # less than 10 missing valid IP addresses
  - missing_count(IP_ADDRESS_COL) < 10:
      valid format: ip address
  # WEBSITE_COL has less than 5% entries that don't contain "astronomer.io"
  - invalid_percent(WEBSITE_COL) < 5:
      valid regex: astronomer\.io
  # The average of 3 columns for values of category_1 is between 10 and 100
  - my_average_total_for_category_1 between 10 and 100:
      my_average_total_for_category_1 query: |
        SELECT AVG(MY_COL_1 + MY_COL_2 + MY_COL_3)
        FROM MY_TABLE_1
        WHERE MY_CATEGORY = 'category_1'
```

Three more unique features of Soda Core are:

* [Freshness checks](https://docs.soda.io/soda-cl/freshness.html): Set limits to the age of the youngest row in the table.
* [Schema checks](https://docs.soda.io/soda-cl/schema.html): Run checks on the existence of columns and validate data types.
* [Reference checks](https://docs.soda.io/soda-cl/reference.html): Ensure parity in between columns in different datasets in the same data source.

```yaml wrap theme={null}
checks for MY_TABLE_1:
  # MY_DATE's youngest row is younger 10 days
  - freshness(MY_DATE) < 10d
  # The schema has to have the MY_KEY column
  - schema:
      fail:
        when required column missing: [MY_KEY]
  # all names listed in MY_TABLE_1's MY_NAMES column have to also exist
  # in the MY_CUSTOMER_NAMES column in MY_TABLE_2
  - values in (MY_NAMES) must exist in MY_TABLE_2 (MY_CUSTOMER_NAMES)
```


# ELT with Apache Airflow® and Databricks
Source: https://astronomer.io/docs/learn/use-case-airflow-databricks

Use Airflow to orchestrate data loading and transformation in Databricks.

## Overview

This reference architecture shows how to use [Apache Airflow®](https://airflow.apache.org/) to copy synthetic data about a green energy initiative from an S3 bucket into a [Databricks table](https://docs.databricks.com/en/tables/index.html) and run several [Databricks notebooks](https://docs.databricks.com/en/notebooks/index.html) as a Databricks job to analyze the data. A demo of the architecture is shown in the [How to Orchestrate Databricks Jobs Using Airflow](https://www.astronomer.io/events/webinars/orchestrate-databricks-jobs-using-airflow-video/) webinar.

[Databricks](https://databricks.com/) is a unified data and analytics platform built around fully managed Apache Spark clusters. Using the [Airflow Databricks provider package](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/index.html), you can create a Databricks job from Databricks notebooks running as a task group in your Airflow Dag. This lets you use Airflow's orchestration features in combination with Databricks Workflows, Databricks' most cost-effective compute option. For detailed instructions on using the Airflow Databricks provider, see [Orchestrate Databricks jobs with Airflow](/docs/learn/airflow-databricks).

<Frame>
  <img alt="Dag graph screenshot." />
</Frame>

You can adapt this architecture for your use case by changing the data source, adjusting the notebook logic, or adding transformation steps.

## Architecture

<Frame>
  <img alt="Databricks reference architecture diagram." />
</Frame>

This reference architecture consists of three main components:

* **Extraction**: An Airflow Dag moves CSV files containing green energy data from the local filesystem to an S3 bucket using the Airflow Object Storage API.
* **Loading**: A second set of tasks loads the files from S3 into a Databricks table using the [`DatabricksCopyIntoOperator`](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/copy_into.html). Each file is loaded in parallel through dynamic task mapping.
* **Transformation**: Databricks notebooks run as a Databricks job orchestrated by Airflow using the [`DatabricksWorkflowTaskGroup`](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/workflow.html) and [`DatabricksNotebookOperator`](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/notebook.html). The notebooks extract data from the table, transform it, and load the results back into Databricks tables.

Data flows in a clear sequence: local CSV files to S3 to a Databricks raw table to transformed tables through notebooks. The first Dag handles extraction and loading, then publishes an asset that triggers the second Dag for transformation.

### Airflow features

* [Airflow Databricks provider](/docs/learn/airflow-databricks): The Databricks provider package creates Databricks jobs directly from Airflow. The [`DatabricksWorkflowTaskGroup`](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/workflow.html) wraps multiple notebooks into a single Databricks Workflow job, while operators like [`DatabricksSqlOperator`](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/sql.html) and [`DatabricksCopyIntoOperator`](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/copy_into.html) handle SQL execution and data loading.
* [Task groups](/docs/learn/task-groups): The Databricks notebook execution is wrapped in a task group that maps to a single Databricks Workflow job. This keeps the Dag graph readable and allows the group to be collapsed in the Airflow UI.
* [Dynamic task mapping](/docs/learn/dynamic-tasks): Loading data from S3 into Databricks is parallelized per file using dynamic task mapping. The number of files is determined at runtime, so the Dag adapts automatically when new files are added to the S3 bucket.
* [Object Storage](/docs/learn/airflow-object-storage-tutorial): The Airflow Object Storage API simplifies moving files to S3 without writing provider-specific code. Files are streamed between paths, which keeps memory usage low even for large datasets.
* [Data-aware scheduling](/docs/learn/airflow-datasets): The extraction and loading Dag runs on a time-based schedule and publishes an asset when loading completes. The transformation Dag schedules itself on this asset, so it only triggers when fresh data is available in Databricks.

## Next steps

To build your own ELT pipeline with Databricks and Apache Airflow, explore the individual Learn guides linked in the Airflow features section for detailed implementation guidance on each pattern. Astronomer recommends deploying Airflow pipelines using a [free trial of Astro](https://www.astronomer.io/lp/signup/).


# Leverage SLAs for enhanced data quality monitoring
Source: https://astronomer.io/docs/learn/using-slas

Learn about use cases and best practices for SLAs.

With Service Level Agreements (SLAs) in Astro Observe, you can use monitoring and proactive alerting to help ensure the *timeliness* and *freshness* of the data your Apache Airflow pipelines deliver.

This guide covers:

* Options for implementing SLAs on Astro Observe.
* Use cases for setting up [proactive alerting](/docs/learn/proactive-alerting), timeliness SLAs, and freshness SLAs using Astro Observe.

## Assumed knowledge

To get the most out of this guide, you should have an understanding of:

* What Airflow is and when to use it. See: [Introduction to Apache Airflow](/docs/learn/intro-to-airflow).
* How to start an Astro trial and deploy Airflow dags to the platform. See: [Start your Astro trial](/docs/astro/trial).
* Basics of the Astro Observe UI. See: [Astro Observe overview](/docs/astro/astro-observe).

## Timeliness versus freshness

Service Level Agreements (SLA) are a set of criteria data has to meet in order to meet a business goal related to a [data product](/docs/learn/data-products). In the context of data pipelines, SLAs are often used to monitor the timeliness and freshness of data. For example, a timeliness SLA might be defined to require that a data product has to be delivered everyday by 9am EST, while a freshness SLA might require that the data in the product is never older than 2 hours.

Depending on your use case, you might want to use Astro Observe to define SLAs to monitor your data product's timeliness, freshness, or both.

## Implement SLAs on Astro Observe

On Astro Observe you can define both timeliness and freshness SLAs on your [data products](/docs/astro/create-data-products). After creating an SLA you can set up [proactive alerts](/docs/learn/proactive-alerting) to get notified when an SLA is at risk of being breached.

To set up an SLA on Astro Observe:

1. Click the **Overview** tab of the data product you want to monitor and then click **+ SLA**.

   <Frame>
     <img alt="SLA creation" />
   </Frame>

2. In the drawer that opens, fill out your SLA details:

   * **Name**: a descriptive name for the SLA.
   * **Description**: a brief description of the SLA, for example the business impact of missing it.
   * **SLA Type**: choose between timeliness and freshness.

   For a timeliness SLA, define:

   * **Days of the Week (UTC)**: the days on which the SLA should be evaluated.
   * **Verification Time (UTC)**: the time at which the SLA should be evaluated.
   * **Lookback Period**: how recent the data needs to have been updated before the SLA evaluation time in order to be considered fresh. For example a lookback period of 1 hour means that the data needs to have been updated within the last hour at time of SLA evaluation to meet the SLA.

   For a freshness SLA, define:

   * **Freshness Policy**: how often the data needs to be updated. For example a freshness policy of 2 hours means that this SLA is breached if the data has ever not been updated for more than 2 hours.

   <Frame>
     <img alt="SLA details" />
   </Frame>

3. Click **Create SLA** to save your SLA.

You can see how your data product performed with regards to its SLAs over time in the **Overview** tab of the data product.

<Frame>
  <img alt="SLA evaluations" />
</Frame>

For a more detailed tutorial that includes an example project, see [Get started with Astro Observe](/docs/learn/astro-observe-quickstart).