Skip to main content

Upgrade to Airflow 3

This guide provides steps to upgrade your Astro project from Apache Airflow 2 to Apache Airflow 3.

tip

For more information on breaking changes between Airflow 2 and Airflow 3, as well as configuration linting see Upgrade from Apache Airflow® 2 to 3 Learn guide and the Airflow 3 Release Notes.

Prerequisites​

  • An Astro account with user permissions to create a new Deployment, such as Workspace Operator or Deployment Admin
  • An existing Astro Deployment running Airflow 2.x
  • Astro CLI >=1.34.0 installed on your local machine
  • Your Astro project directory

Step 1: Prepare Your Local Project for Airflow 3​

Before upgrading to Airflow 3, you must test your DAGs and dependencies locally to identify and resolve any compatibility issues.

Breaking change

Astro Runtime for 3.0 and higher includes only the provider packages required to run on Astro. Previous versions contained additional providers, and if you are using those providers you must explicitly add them to your requirements.txt.

Follow these steps to prepare your local project:

Upgrade Test Your Local Project​

Test your project locally to identify any compatibility issues:

astro dev upgrade-test

Use the results of this test to identify compatibility issues. Address any compatibility issues by updating your DAGs, custom operators, and dependencies. Common changes include:

  • Updating import paths that have changed in Airflow 3
  • Replacing deprecated features with their recommended alternatives
  • Updating Airflow providers to versions compatible with Airflow 3 see list of minimum versions in Airflow here

Run Unit Tests​

In your local Astro project, update the Dockerfile to use an Airflow 3 image:

FROM astrocrpublic.azurecr.io/runtime:<astro-runtime-version>

Next, run the following to execute any unit tests:

astro dev pytest

To learn more, refer to full docs on testing.

info

Replace <astro-runtime-version> with the specific Astro Runtime version that includes Airflow 3.x that you want to use.

Run Your Project Locally​

For a final check and to test DAGs manually, start Airflow locally:

astro dev start

Refer to the docs on local Airflow development to make and test any changes.

Step 2: Choose Your Upgrade Strategy​

There are two main strategies to upgrade to Airflow 3. The recommended strategy (Strategy A) for upgrading to Airflow 3 is to:

  1. Create a new Airflow 3 Deployment in Astro
  2. Update your local project to use an Airflow 3 image
  3. Deploy your updated project to the new Deployment

This approach allows you to:

  • Test your DAGs with Airflow 3 without affecting your production environment
  • Easily roll back if you encounter compatibility issues
  • Gradually migrate your workflows to the new environment

To continue with this strategy, proceed to Step 3A.

Alternatively, you can try a more advanced strategy (Strategy B) of doing an in-place upgrade to Airflow 3. In this strategy:

  1. Prepare and test your DAGs on Airflow 3 locally
  2. Make an Airflow 3 deploy against your Airflow 2 Deployment, initiating an upgrade
warning

Rollbacks are not yet supported from Airflow 3 -> Airflow 2. If you decide to upgrade using this strategy, you can only fix your Deployment by continuing to use Airflow 3 and making changes to your image and/or DAGs.

This approach allows you to:

  • Make changes on Deployments you're already running, without needing new infrastructure
  • One-shot moving all workflows to Airflow 3

To continue with this strategy, move to Step 3B.

Step 3A: Upgrade to Airflow 3 with a New Deployment​

1. Create a New Airflow 3 Deployment​

  1. In the Astro UI, go to the Deployments tab and click New Deployment
  2. Make sure the latest Airflow 3 Runtime is selected
  3. Configure your Deployment settings and complete the creation process
  4. Note your new Deployment ID for later use

For more details, check out the docs to Create a new Deployment.

2. Deploy to Your New Airflow 3 Deployment​

Deploy your updated project:

astro deploy <your-new-deployment-id>

For additional customization, refer to docs on deploying Astro project images.

Finally, verify that your DAGs are running correctly in the new environment.

3. Migrate Your Workflows​

  1. Update any CI/CD pipelines to deploy to your new Airflow 3 Deployment
  2. Redirect any integrations or external systems to use your new Deployment
  3. Once you've confirmed everything is working correctly, you can decommission your Airflow 2 Deployment

Step 3B: Upgrade an existing Deployment to Airflow 3​

Upgrade your Airflow 2 Deployment to Airflow 3​

warning

Rollbacks are not yet supported from Airflow 3 back to Airflow 2. If you decide to upgrade using this strategy, you can only fix your Deployment by continuing to use Airflow 3 and making changes to your image and/or DAGs.

Deploy your updated project:

astro deploy --force-upgrade-to-af3

Troubleshooting common issues​

Pip package resolution​

With Runtime 3.0, Python package resolution has switched by default from pip to uv, which is intended as a fast drop in replacement for pip. However, uv has a different behavior for packages that exist on multiple indexes, because pip's behavior is unsafe against dependency confusion attacks. If you rely on the pip behavior, you can add # ASTRO_RUNTIME_USE_PIP to your requirements.txt file to use pip instead of uv. You can also opt to install your dependencies in a separate RUN line on your Dockerfile instead of using Runtime's built-in pip installation.

Docker Compose override changes​

When using astro dev start, you can specify a docker-compose.override.yml. If you specified any overrides for the webserver container, these break your ability to use astro dev start because there is no longer a webserver container. Replace all references to webserver with api-server.

Was this page helpful?