Airflow 3

Migrate to Airflow 3 from Airflow 2 in Astro Private Cloud

This guide describes the steps required to migrate your Airflow deployments from Airflow 2 to Airflow 3 in Astro Private Cloud.

1. Enable Airflow 3 on the cluster

  1. Navigate to the Cluster Configuration Override page in the Astro Private Cloud UI.
  2. Enable the Airflow 3 flag:
1astronomer:
2    houston:
3        deployments:
4            airflowV3:
5                enabled: true
  1. Enable the mandatory Dag processor component for Airflow 3:
1astronomer:
2    houston:
3        deployments:
4            dagProcessorEnabled: true
Airflow 3 requires the Dag processor to be enabled. Deployments without this component will fail.
  1. Enable Sidecar Logging for Airflow tasks:
1global:
2    loggingSidecar:
3        enabled: true
Airflow 3 requires sidecar logging. Deployments without this component won’t be able to view task logs.

2. Configure registry access

  1. Allowlist the Azure Container Registry (azurecr) repository and URL for pulling Airflow 3 images.
  2. If using third-party registries, configure mirroring to ensure reliable image pulls.
Airflow 3 images are hosted at astrocrpublic.azurecr.io.

3. Check Dag compatibility before migration

Before upgrading, follow the Airflow 2 → 3 upgrade guide to verify that your Dags are compatible with Airflow 3.

4. Update Deployment Dockerfile

  1. Modify your Dockerfile to use the Airflow 3 image and updated registry URL:
1FROM astrocrpublic.azurecr.io/runtime:3.1-2
  1. Build and push your updated Docker image.
    Airflow 2 images are available on Quay (quay.io/astronomer/airflow), while Airflow 3 images are hosted on astrocr (astrocrpublic.azurecr.io).

5. Update Deployment

Perform the Deployment update using one of the following methods:

  1. UI: Navigate to your Deployment and trigger an update with the new Airflow 3 image.
  2. API: Use the UpsertDeployment API to update the Deployment image.
  3. CLI: Use the Astro CLI to deploy the updated image.

Ensure the updated Deployment reflects the Airflow 3 runtime version.

6. Validate after migration

  • Confirm that Dags are running as expected in Airflow 3.
  • Monitor logs and metrics to ensure stability.
  • Verify that the Dag processor component is active and functioning.

This guide provides a safe and supported path to migrate from Airflow 2 to Airflow 3 in Astro Private Cloud.