Upgrade Airflow on Astronomer Software
Overview
Regularly upgrading your Software Deployments ensures that your Deployments continue to be supported and that your Airflow instance has the latest features and functionality.
To upgrade your Airflow Deployment to a later version of Airflow:
- Select a new Airflow version with the Software UI or CLI to start the upgrade.
- Change the FROM statement in your project's
Dockerfile
to reference an Astronomer Certified (AC) or Astro Runtime image that corresponds to your current Airflow version. See Customize Your Image. - Deploy to Astronomer.
Available Astronomer image versions
A cron job automatically pulls new Astronomer image versions from the Astronomer update service and adds them to the Software UI and CLI within 24 hours of their publication. You don't have to upgrade Astronomer to upgrade Airflow.
If you don't want to wait for new Astronomer image versions, you can manually trigger the cron job with the following Kubernetes command:
kubectl create job --namespace astronomer --from=cronjob/astronomer-houston-update-airflow-check airflow-update-check-first-run
If you get a message indicating that a job already exists, delete the job and rerun the command.
Step 1. Start the upgrade process
Starting the upgrade process doesn't interrupt or otherwise impact your Airflow Deployment. It only signals to Astronomer your intent to upgrade at a later time.
The Software UI and CLI only provide Airflow versions that are later than the version currently running in your Dockerfile
. For example, Airflow 2.1.0
is not available for an Airflow Deployment running 2.2.0
.
With the Software UI
- Go to Deployment > Settings > Basics > Airflow Version.
- Select an Airflow version.
- Click Upgrade.
With the Astro CLI
-
Run
$ astro auth login <base-domain>
to confirm you're authenticated. -
Run the following command to view your Airflow Deployment
Deployment ID
:astro deployment list
-
Copy the
DEPLOYMENT ID
value and run the following command to list the available Airflow versions:astro deployment airflow upgrade --deployment-id=<deployment-id>
-
Enter the Airflow version you want to upgrade to and press
Enter
.
Step 2: Update your Astro project
-
In your Astro project, open your
Dockerfile
. -
Update the
FROM
line of your project'sDockerfile
to reference a new Astronomer image. For example, to upgrade to the latest version of Astro Runtime, you would change theFROM
line to:FROM quay.io/astronomer/astro-runtime:12.0.0
For a list of supported Astro Runtime versions, see Astro Runtime maintenance and lifecycle policy.
After you upgrade your Airflow version, you can't revert to an earlier version.
-
Optional. Test your upgrade on your local machine by running:
astro dev restart
All 4 Airflow Docker containers restart and begin running your new image.
To confirm that your migration was successful, open the Airflow UI at
localhost:8080
and scroll to the bottom of any page. You should see your new Runtime version in the footer.
Step 3: Deploy to Astronomer
-
To push your upgrade to a Deployment on Astronomer Software, run:
astro deploy
Due to a schema change in the Airflow metadata database, upgrading a Software Deployment to AC 2.3.0 can take significant time. Depending on the size of your metadata database, upgrades can take 10 minutes to an hour or longer depending on the number of task instances that have been recorded in the Airflow metadata database. During this time, scheduled tasks continue to execute but new tasks are not scheduled.
To minimize the upgrade time for a Deployment, contact Astronomer support. Minimizing your upgrade time requires removing records from your metadata database.
- In the Software UI, open your Deployment and click Open Airflow.
- In the Airflow UI, scroll to the bottom of any page. You should see your new Runtime version in the footer.
Cancel Airflow upgrade
You can cancel an Airflow Deployment upgrade at any time if you haven't yet changed the Astronomer image in your Dockerfile
and deployed it.
In the Software UI, select Cancel next to Airflow Version.
Using the Astro CLI, run:
astro deployment airflow upgrade --cancel --deployment-id=<deployment-id>
For example, if you cancel an upgrade from Airflow 2.1.0 to Airflow 2.2.0 in the CLI, the following message appears:
astro deployment airflow upgrade --cancel --deployment-id=ckguogf6x0685ewxtebr4v04x
Airflow upgrade process has been successfully canceled. Your Deployment was not interrupted and you are still running Airflow 2.1.0.
Canceling the Airflow upgrade process does not interrupt or otherwise impact your Airflow Deployment or code that's running.