Upgrade an Astro project to Airflow 3

This guide provides steps to upgrade your Astro project from Apache Airflow 2 to Apache Airflow 3.

Prerequisites

• An Astro account with user permissions to create a new Deployment, such as Workspace Operator or Deployment Admin
• An existing Astro Deployment running Airflow 2.x
• Astro CLI >=1.34.0 installed on your local computer
• Your Astro project folder

Step 1: Prepare your local project for Airflow 3

Before upgrading to Airflow 3, you must test your Dags and dependencies locally to identify and resolve any compatibility issues.

Follow these steps to prepare your local project:

Upgrade test your local project

Test your project locally to identify any compatibility issues:

$
astro dev upgrade-test

Use the results of this test to identify compatibility issues. Address any compatibility issues by updating your Dags, custom operators, and dependencies. Common changes include:

• Updating import paths that have changed in Airflow 3
• Replacing deprecated features with their recommended alternatives
• Updating Airflow providers to versions compatible with Airflow 3 — see minimum provider versions required for Airflow 3

Run unit tests

In your local Astro project, update the Dockerfile to use an Airflow 3 image:

1
FROM astrocrpublic.azurecr.io/runtime:<astro-runtime-version>

Next, run the following to execute any unit tests:

$
astro dev pytest

For more information, see testing your Astro project locally.

Run your project locally

For a final check and to test Dags manually, start Airflow locally:

$
astro dev start

Refer to the docs on local Airflow development to make and test any changes.

Step 2: Choose your upgrade strategy

There are two supported upgrade strategies for Airflow 3. Choose the approach that fits your operational needs and workflow requirements.

Strategy A

1. Create a new Airflow 3 Deployment in Astro.
2. Update your local project to use an Airflow 3 image.
3. Deploy your updated project to the new Deployment.

Pros:

• Lower affected area; you can test your Dags without affecting production.
• Gradual, controlled migration of workflows.
• Easy rollback; you can switch back to your Airflow 2 Deployment until decommissioned.

Cons:

• No migration of historical task data.
• Slightly higher resource usage and cost while running both Deployments during migration.

To continue with this strategy, proceed to Step 3A.

Strategy B

1. Prepare and test your Dags on Airflow 3 locally.
2. Make an Airflow 3 deploy against your Airflow 2 Deployment, initiating an upgrade.

Pros:

• Preserves historical task data and metadata.
• No need for extra or temporary Deployments.

Cons:

• Higher affected area; all Dags must be compatible at once.
• Immediate production impact if issues arise.
• Any in-flight dag runs and task instances will be failed when the upgrade begins.
• Rollbacks to Airflow 2 require specific Runtime versions. See Airflow 3 to Airflow 2 rollback support and requirements

To continue with this strategy, move to Step 3B.

Step 3A: Upgrade to Airflow 3 with a new Deployment

1. Create a new Airflow 3 Deployment

1. In the Astro UI, go to the Deployments tab and click New Deployment.
2. Ensure the latest Airflow 3 Runtime is selected.
3. Configure your Deployment settings and complete the creation process.
4. Note your new Deployment ID for later use.

For more information, see Create a new Deployment.

2. Deploy to your new Airflow 3 Deployment

Deploy your updated project:

$
astro deploy <your-new-deployment-id>

For additional customization, refer to docs on deploying Astro project images.

Verify that your Dags are running correctly in the new environment.

3. Migrate your workflows

1. Update any CI/CD pipelines to deploy to your new Airflow 3 Deployment.
2. Redirect any integrations or external systems to use your new Deployment.
3. After you’ve confirmed everything is working correctly, you can decommission your Airflow 2 Deployment.

Step 3B: Upgrade an existing Deployment to Airflow 3

Upgrade your Airflow 2 Deployment to Airflow 3

Deploy your updated project:

$
astro deploy

Step 4: Update your workload identity policy bindings

• Update the Workload Identity configuration on the Deployment Details page. Re-enter the same IAM role (ARN/service account email/service principal) you were previously using. This step ensures your IAM role bindings are updated from the Webserver service account to the newly added API Server service account in Airflow 3, which is referenced in the upcoming setup script.
• When prompted, provide your IAM role ARN/service account email/service principal details. Copy and run the CLI command that is generated. After running it, click Save Configuration to store the IAM role so it can be reused as a selectable configuration.

Troubleshoot common issues

Pip package resolution

With Runtime 3.0, Python package resolution has switched by default from pip to uv, which is intended as a fast drop in replacement for pip. However, uv has a different behavior for packages that exist on multiple indexes, because pip’s behavior is unsafe against dependency confusion attacks. If you rely on the pip behavior, you can add # ASTRO_RUNTIME_USE_PIP to your requirements.txt file to use pip instead of uv. You can also opt to install your dependencies in a separate RUN line on your Dockerfile instead of using Runtime’s built-in pip installation.

Docker Compose override changes

When using astro dev start, you can specify a docker-compose.override.yml file. If you specified any overrides for the webserver container in your override file for Airflow 2, these break your ability to use astro dev start because there is no longer a webserver container in Airflow 3. Replace all references to webserver with api-server.

Dags folder location change

On Airflow 2, the dags/ folder is located at /usr/local/airflow/dags, but with the addition of Dag versioning in Airflow 3, you can no longer be sure of the exact path of the relevant Dags folder. Instead of using this hardcoded path or the AIRFLOW_HOME environment variable, use paths relative to the current file using the Python keyword file. Note also that because Dag bundling is not enabled when using astro dev start, you don’t find errors from this folder location change until your code is deployed to Astro.

For example, if you use the AIRFLOW_HOME variable to open a file in your Dags folder like the following code example, it breaks in Airflow 3:

1
open(f"{AIRFLOW_HOME}/dags/path/to/file.txt", "r")

PYTHONPATH doesn’t include the Dags folder

On Astro with Airflow 3, the dags/ folder isn’t on the PYTHONPATH, and you can’t customize the PYTHONPATH to include the dags/ folder or any other folders. This is a consequence of Dag versioning and Dag bundles in Airflow 3, where the exact path of the Dags folder isn’t fixed at runtime. Setting PYTHONPATH in your Dockerfile or as an environment variable has no effect on Astro.

If your Airflow 2 Dags imported modules directly from the dags/ folder using an import like from my_utils import helper, where my_utils.py lives in dags/, these imports break on Airflow 3. Imports that include the dags. prefix, such as from dags.my_utils import helper, continue to work.

Astronomer recommends that you move shared modules out of the dags/ folder and into the include/ folder, which remains on the PYTHONPATH, then update your imports. For example:

1
from include.my_utils import helper

If you want to keep helper files in the dags/ folder, for example, to avoid an image deploy when you change helper code, update your imports to use the dags. prefix:

1
from dags.my_utils import helper

See Manage Airflow code for the recommended project structure.

Airflow 3 to Airflow 2 rollback support and requirements

Astro supports rolling back from Airflow 3 to Airflow 2 but not all Airflow 3.x Deployments can be reverted to all Airflow 2.x versions.

• You can only roll back to Airflow 2.11, which runs on Astro Runtime 13.7.0 or above.
• Rollback to Airflow 2.11 is only possible if your Deployment is running an Airflow 3 version that includes rollback support: Runtime versions 3.0-11, or later.