Migrate to Airflow 3 from Airflow 2 in Astro Private Cloud

This guide describes the steps required to migrate your Airflow deployments from Airflow 2 (AF2) to Airflow 3 (AF3) in Astro Private Cloud 1.0.

1. Enable Airflow 3 on the cluster

1. Navigate to the Cluster Configuration Override page in the Astro Private Cloud UI.
2. Enable the Airflow 3 flag:

1
astronomer:
2
    houston:
3
        deployments:
4
            airflowV3:
5
                enabled: true

3. Enable the mandatory dag Processor component for AF3:

1
astronomer:
2
    houston:
3
        deployments:
4
            dagProcessorEnabled: true

1
global:
2
    loggingSidecar:
3
        enabled: true

2. Configure Registry access

1. Allowlist the Azure Container Registry (azurecr) repository and URL for pulling Airflow 3 images.
2. If using third-party registries, configure mirroring to ensure reliable image pulls.

3. Pre-migration dag compatibility check

Before upgrading, follow Astronomer’s Airflow 2 → 3 upgrade guide to verify that your dags are compatible with Airflow 3.

4. Update Deployment Dockerfile

1. Modify your Dockerfile to use the Airflow 3 image and updated registry URL:

1
FROM astrocrpublic.azurecr.io/runtime:3.1-2

2. Build and push your updated Docker image.
   AF2 images are available on Quay (quay.io/astronomer/airflow), while AF3 images are hosted on astrocr (astrocrpublic.azurecr.io).

5. Update Deployment

Perform the Deployment update using one of the following methods:

1. UI: Navigate to your Deployment and trigger an update with the new AF3 image.
2. API: Use the UpsertDeployment API to update the Deployment image.
3. CLI: Use the Astro CLI to deploy the updated image.

Ensure the updated Deployment reflects the Airflow 3 runtime version.

6. Post-migration validation

• Confirm that dags are running as expected in Airflow 3.
• Monitor logs and metrics to ensure stability.
• Verify that the Dag Processor component is active and functioning.