Upgrade to Airflow 3

This guide provides steps to upgrade your Astro project from Apache Airflow 2 to Apache Airflow 3.

Prerequisites

• An Astro account with user permissions to create a new Deployment, such as Workspace Operator or Deployment Admin
• An existing Astro Deployment running Airflow 2.x
• Astro CLI >=1.34.0 installed on your local machine
• Your Astro project directory

Step 1: Prepare Your Local Project for Airflow 3

Before upgrading to Airflow 3, you must test your dags and dependencies locally to identify and resolve any compatibility issues.

Follow these steps to prepare your local project:

Upgrade Test Your Local Project

Test your project locally to identify any compatibility issues:

$
astro dev upgrade-test

Use the results of this test to identify compatibility issues. Address any compatibility issues by updating your dags, custom operators, and dependencies. Common changes include:

• Updating import paths that have changed in Airflow 3
• Replacing deprecated features with their recommended alternatives
• Updating Airflow providers to versions compatible with Airflow 3 see list of minimum versions in Airflow here

Run Unit Tests

In your local Astro project, update the Dockerfile to use an Airflow 3 image:

1
FROM astrocrpublic.azurecr.io/runtime:<astro-runtime-version>

Next, run the following to execute any unit tests:

astro dev pytest

To learn more, refer to full docs on testing.

Run Your Project Locally

For a final check and to test dags manually, start Airflow locally:

astro dev start

Refer to the docs on local Airflow development to make and test any changes.

Step 2: Choose Your Upgrade Strategy

There are two supported upgrade strategies for Airflow 3. Choose the approach that fits your operational needs and workflow requirements.

Strategy A

1. Create a new Airflow 3 Deployment in Astro.
2. Update your local project to use an Airflow 3 image.
3. Deploy your updated project to the new Deployment.

Pros:

• Lower blast radius; you can test your DAGs without affecting production.
• Gradual, controlled migration of workflows.
• Easy rollback; you can switch back to your Airflow 2 Deployment until decommissioned.

Cons:

• No migration of historical task data.
• Slightly higher resource usage and cost while running both Deployments during migration.

To continue with this strategy, proceed to Step 3A.

Strategy B

1. Prepare and test your dags on Airflow 3 locally.
2. Make an Airflow 3 deploy against your Airflow 2 Deployment, initiating an upgrade.

Pros:

• Preserves historical task data and metadata.
• No need for extra or temporary Deployments.

Cons:

• Higher blast radius; all DAGs must be compatible at once.
• Immediate production impact if issues arise.
• Rollback to Airflow 2 is only possible if you meet the requirements.

To continue with this strategy, move to Step 3B.

Step 3A: Upgrade to Airflow 3 with a New Deployment

1. Create a New Airflow 3 Deployment

1. In the Astro UI, go to the Deployments tab and click New Deployment
2. Make sure the latest Airflow 3 Runtime is selected
3. Configure your Deployment settings and complete the creation process
4. Note your new Deployment ID for later use

For more details, check out the docs to Create a new Deployment.

2. Deploy to Your New Airflow 3 Deployment

Deploy your updated project:

$
astro deploy <your-new-deployment-id>

For additional customization, refer to docs on deploying Astro project images.

Finally, verify that your dags are running correctly in the new environment.

3. Migrate Your Workflows

1. Update any CI/CD pipelines to deploy to your new Airflow 3 Deployment
2. Redirect any integrations or external systems to use your new Deployment
3. Once you’ve confirmed everything is working correctly, you can decommission your Airflow 2 Deployment

Step 3B: Upgrade an existing Deployment to Airflow 3

Upgrade your Airflow 2 Deployment to Airflow 3

Deploy your updated project:

$
astro deploy

Troubleshooting common issues

Pip package resolution

With Runtime 3.0, Python package resolution has switched by default from pip to uv, which is intended as a fast drop in replacement for pip. However, uv has a different behavior for packages that exist on multiple indexes, because pip’s behavior is unsafe against dependency confusion attacks. If you rely on the pip behavior, you can add # ASTRO_RUNTIME_USE_PIP to your requirements.txt file to use pip instead of uv. You can also opt to install your dependencies in a separate RUN line on your Dockerfile instead of using Runtime’s built-in pip installation.

Docker Compose override changes

When using astro dev start, you can specify a docker-compose.override.yml file. If you specified any overrides for the webserver container in your override file for Airflow 2, these break your ability to use astro dev start because there is no longer a webserver container in Airflow 3. Replace all references to webserver with api-server.

Dags folder location change

On Airflow 2, the dags/ folder is located at /usr/local/airflow/dags but with the addition of dag versioning in Airflow 3, you can no longer be sure of the exact path of the relevant dags folder. Instead of using this hardcoded path or the AIRFLOW_HOME environment variable, use paths relative to the current file using the python keyword file. Note also that because dag bundling is not enabled when using astro dev start, you don’t find errors from this folder location change until your code is deployed to Astro.

For example, if you use the AIRFLOW_HOME variable to open a file in your dags folder like the following code example, it breaks in Airflow 3:

1
open(f"{AIRFLOW_HOME}/dags/path/to/file.txt", "r")

Airflow 3 to Airflow 2 rollback support and requirements

Astro supports rolling back from Airflow 3 to Airflow 2 but not all Airflow 3.x Deployments can be reverted to all Airflow 2.x versions.

• You can only roll back to Airflow 2.10 or 2.11, which run on Astro Runtime 12.x or 13.x.
• Rollback to Airflow 2.10 or 2.11 is only possible if your Deployment is running an Airflow 3 version that includes rollback support: Runtime versions 3.0-11, or later.