> ## Documentation Index > Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt > Use this file to discover all available pages before exploring further. # Upgrade Astro Runtime (Airflow version) on Astronomer Software ## Overview Regularly upgrading your Software Deployments ensures that your Deployments continue to be supported and that your Airflow instance has the latest features and functionality. To upgrade your Airflow Deployment to a later version of Airflow: * Select a new Airflow version with the Software UI or CLI to start the upgrade. * Change the `FROM` statement in your project's `Dockerfile` to reference an Astro Runtime image that corresponds to your current Airflow version. * Deploy your upgrade to Astronomer. ## Available Astronomer image versions A cron job automatically pulls new Astronomer image versions from the Astronomer [update service](http://updates.astronomer.io/) and adds them to the Software UI and CLI within 24 hours of their publication. You don't have to upgrade Astronomer to upgrade Airflow. If you don't want to wait for new Astronomer image versions, you can manually trigger the cron job with the following Kubernetes command: ```bash wrap theme={null} kubectl create job --namespace astronomer --from=cronjob/astronomer-houston-update-airflow-check airflow-update-check-first-run ``` If you get a message indicating that a job already exists, delete the job and rerun the command. ## Step 1: Review upgrade considerations Astro Runtime upgrades can include breaking changes, especially when you're upgrading to a new major version. Check the [upgrade considerations](#upgrade-considerations) for your upgrade version to anticipate any breaking changes or upgrade-specific instructions before you proceed. ## Step 2: (Optional) Start the upgrade process Now, by default, Astronomer Software uses the Runtime version that you configure in your Dockerfile in **Step 5: Update your Astro Project**. Prior to version 0.36, by default, you must first specify the Airflow and Runtime version you wanted to upgrade to, before starting your upgrade process. If you want to use the prior workflow, you must change the feature flag, `disableDesiredRuntimeVersion`, in your Houston API `values.yaml` configuration to `false`: ```yaml wrap theme={null} astronomer: houston: config: deployments: disableDesiredRuntimeVersion: false ``` After you apply the configuration, you can start the upgrade process. Starting the upgrade process doesn't interrupt or otherwise impact your Airflow Deployment. It only signals to Astronomer Software that you intend to upgrade at a later time. ### With the Software UI 1. Go to **Deployment** > **Settings** > **Basics** > **Airflow Version**. 2. Select an Airflow version. 3. Click **Upgrade**. ### With the Astro CLI 1. Run `astro login ` to confirm you're authenticated. 2. Run the following command to list your current Deployments. ```bash wrap theme={null} astro deployment list ``` Copy the ID of the Deployment you want to upgrade. 3. Run the following command to list the available Airflow versions: ```bash wrap theme={null} astro deployment airflow upgrade --deployment-id= ``` 4. Enter the Airflow version you want to upgrade to and press `Enter`. ## Step 3: (Optional) Pin provider package versions Major Astro Runtime upgrades can include significant upgrades to built-in provider packages. These package upgrades can sometimes include breaking changes for your DAGs. See the [Apache Airflow documentation](https://airflow.apache.org/docs/apache-airflow-providers/packages-ref.html) for a list of all available provider packages and their release notes. For the most stable upgrade path, Astronomer recommends pinning all provider package versions from your current Runtime version before upgrading. 1. Run the following command to check the version of all provider packages installed in your Astro Runtime version: ```bash wrap theme={null} docker run --rm quay.io/astronomer/astro-runtime: pip freeze | grep apache-airflow-providers ``` 2. After reviewing this list, pin the version for each provider package in your Astro project `requirements.txt` file. For example, Runtime 7.4.1 uses version 4.0.0 of `apache-airflow-providers-databricks`. To pin this version of the Databricks provider package when you upgrade to a later version of Runtime, you add the following line to your `requirements.txt` file: ```text wrap theme={null} apache-airflow-providers-databricks==4.0.0 ``` ## Step 4: (Optional) Run upgrade tests with the Astro CLI You can use the Astro CLI to anticipate and address problems before upgrading to a newer version of Astro Runtime. Before you upgrade, run the following command to run tests against the version of Astro Runtime you're upgrading to: ```bash wrap theme={null} astro dev upgrade-test --runtime-version ``` The Astro CLI then generates test results in your Astro project that identify dependency conflicts and import errors that you would experience using the new Astro Runtime version. Review these results and make the recommended changes to reduce the risk of your project generating errors after you upgrade. For more information about using this command and the test results, see [Test before upgrade your Astro project](/docs/cli/v1.43/test-your-astro-project-locally#test-before-an-astro-runtime-upgrade). ## Step 5: Update your Astro project 1. In your Astro project, open your `Dockerfile`. 2. Change the Docker image in the `FROM` statement of your `Dockerfile` to a new version of Astro Runtime. For example, to upgrade to the latest version of Runtime, you would change the `FROM` statement in your Dockerfile to: ```dockerfile wrap theme={null} FROM quay.io/astronomer/astro-runtime:13.2.0 ``` For a list of supported Astro Runtime versions, see [Astro Runtime maintenance and lifecycle policy](/docs/runtime/runtime-version-lifecycle-policy#astro-runtime-lifecycle-schedule). After you upgrade your Airflow version, you can't revert to an earlier version. 1. Save the changes to your `Dockerfile`. ## Step 6: Test Astro Runtime locally Astronomer recommends testing new versions of Astro Runtime locally to ensure that Airflow starts as expected before upgrading your Deployment on Astro. 1. Open your project directory in your terminal and run `astro dev restart`. This restarts the Docker containers for the Airflow webserver, scheduler, triggerer, and Postgres metadata database. 2. Access the Airflow UI of your local environment by navigating to `http://localhost:8080` in your browser. 3. Confirm that your local upgrade was successful by scrolling to the bottom of any page. You should see your new Astro Runtime version in the footer as well as the version of Airflow it is based on. Runtime Version banner - Local 4. (Optional) Run DAGs locally to ensure that all of your code works as expected. If you encounter errors after your upgrade, it's possible that your new Astro Runtime version includes a breaking provider package change. If you experience one of these breaking changes, follow the steps in [Upgrade or pin provider package versions](#step-3-optional-pin-provider-package-versions) to check your provider package versions and, if required, pin the provider package version from your previous Runtime version in your `requirements.txt` file. ## Step 7: Deploy to Astronomer 1. Run the following command to push your upgraded Astro project to your Deployment: ```bash wrap theme={null} astro deploy ``` 2. In the Software UI, open your Deployment and click **Open Airflow**. 3. In the Airflow UI, scroll to the bottom of any page. You should see your new Runtime version in the footer. ## Cancel Airflow upgrade As a System Admin, you can cancel an Airflow Deployment upgrade at any time if you haven't yet changed the Astronomer Runtime image in your `Dockerfile` and deployed it. In the Software UI, select **Cancel** next to **Airflow Version**. Using the Astro CLI, run: ```bash wrap theme={null} astro deployment airflow upgrade --cancel --deployment-id= ``` For example, if you cancel an upgrade from Airflow 2.1.0 to Airflow 2.2.0 in the CLI, the following message appears: ```bash wrap theme={null} Airflow upgrade process has been successfully canceled. Your Deployment was not interrupted and you are still running Airflow 2.1.0. ``` Canceling the Airflow upgrade process does not interrupt or otherwise impact your Airflow Deployment or code that's running. If you can't cancel your upgrade and receive an error message about using an unsupported Airflow version, set the following value in your `values.yaml` file and [apply the change](/docs/astro-private-cloud/v-0-36/apply-platform-config) to successfully cancel your upgrade. This configuration allows you to roll back to your current version of Airflow, even if it's not supported. ```yaml wrap theme={null} astronomer: houston: config: deployments: enableSystemAdminCanCreateDeprecatedAirflows: true ``` ## Upgrade considerations Consider the following when you upgrade Astro Runtime: * Astronomer does not support downgrading a Deployment on Astronomer Software to a lower version of Astro Runtime. * All versions of the Astro CLI support all versions of Astro Runtime. There are no dependencies between the two products. * Upgrading to certain versions of Runtime might result in extended upgrade times or otherwise disruptive changes to your environment. To learn more, see [Version-specific upgrade considerations](#version-upgrade-considerations). To stay up to date on the latest versions of Astro Runtime, see [Astro Runtime release notes](/docs/runtime/runtime-release-notes). For more information on Astro Runtime versioning and support, see [Astro Runtime versioning and lifecycle policy](/docs/astro-private-cloud/v-0-36/runtime-version-lifecycle-policy). For a full collection of Astro Runtime Docker images, go to the [Astro Runtime repository on Quay.io](https://quay.io/repository/astronomer/astro-runtime?tab=tags). ### Version upgrade considerations Each specific Astro Runtime versions has upgrade considerations that you need to be aware of and account for when you upgrade. These include breaking changes, database migrations, and other considerations. If an Astro Runtime version isn't included in this section, then there are no specific upgrade considerations for that version. #### Runtime 12 (Airflow 2.10) ##### Upgrade to Python 3.12 Starting with Python version 3.12, Python has removed the module [`imp`](https://docs.python.org/3.11/library/imp.html). This can cause errors for your DAGs if you use `imp` to access materials using the `import` statement. Python recommends migrating to [`importlib`](https://docs.python.org/3.11/library/importlib.html#module-importlib) instead. See [How do I migrate from `imp`](https://discuss.python.org/t/how-do-i-migrate-from-imp/27885) guidance from Python for remediation steps. ##### Incompatibility between Git-Sync and Runtime 12.1.0 Astro Runtime 12.1.0 is incompatible with [Git-Sync](https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html#using-git-sync). Upgrade to Astro Runtime 12.1.1 to use Git-Sync with Astronomer Software. #### Runtime 11 (Airflow 2.9) ##### Yanked versions of Runtime 11 Use caution if you create new Deployments or upgrade to the following versions of Runtime 11, which are `yanked`. These [Runtime versions](/docs/astro-private-cloud/v-0-36/runtime-version-lifecycle-policy#restricted-runtime-versions) warn you about upgrading to or creating a Deployment with a version that contains a known limitation or bug. * 11.0.0 * 11.1.0 ##### Bug affecting users running a local Astro Runtime environment In Airflow 2.9, a bug affecting custom actions in Airflow plugins ([#39421](https://github.com/apache/airflow/pull/39421)) prevented users from running the Astro Runtime environment locally for Astro Runtime versions 11.0.0, 11.1.0, and 11.2.0. Deployments running these versions on Astro are not affected. #### Runtime 9 (Airflow 2.7) ##### Yanked versions of Runtime 9 Use caution if you create new Deployments or upgrade to the following versions of Runtime 9, which are `yanked`. These [Runtime versions](/docs/astro-private-cloud/v-0-36/runtime-version-lifecycle-policy#restricted-runtime-versions) warn you about upgrading to or creating a Deployment with a version that contains a known limitation or bug. * 9.0.0 * 9.1.0 * 9.2.0 * 9.3.0 * 9.4.0 * 9.5.0 * 9.6.0 ##### Connection testing in the Airflow UI disabled by default In Airflow 2.7, connection testing in the Airflow UI is disabled by default. Astronomer does not recommend reenabling the feature unless you fully trust all users with edit/ delete permissions for Airflow connections. To reenable the feature, set the following environment variable in your Astro project Dockerfile: ```dockerfile wrap theme={null} ENV AIRFLOW__CORE__TEST_CONNECTION=Enabled ``` ##### Upgrade to Python 3.11 The base distribution of Astro Runtime 9 uses Python 3.11 by default. Some provider packages, such as `apache-airflow-providers-apache-hive`, aren't compatible with Python 3.11. To continue using these packages with a compatible version of Python, upgrade to the [Astro Runtime Python distribution](/docs/runtime/runtime-image-architecture#python-version-images) for your desired Python version. #### Runtime 8 (Airflow 2.6) Astro Runtime version 8.0.0 is a `yanked` version of the Astro Runtime, which means you are advised not to create Deployments with this runtime version. These [Runtime versions](/docs/astro-private-cloud/v-0-36/runtime-version-lifecycle-policy#restricted-runtime-versions) warn you about upgrading to or creating a Deployment with a version that contains a known limitation or bug. ##### Breaking change to `apache-airflow-providers-cncf-kubernetes` in version 8.4.0 Astro Runtime 8.4.0 upgrades `apache-airflow-providers-cncf-kubernetes` to 7.0.0, which includes a breaking change by [removing some deprecated features from `KubernetesHook`](https://github.com/apache/airflow/commit/a1f5a5425e65c40e9baaf5eb4faeaed01cee3569). If you are using any of these features, either pin your current version of `apache-airflow-providers-cncf-kubernetes` to your `requirements.txt` file or ensure that you don't use any of the deprecated features before upgrading. ##### Package dependency conflicts Astro Runtime 8 includes fewer default dependencies than previous versions. Specifically, the following provider packages are no longer installed by default: * `apache-airflow-providers-apache-hive` * `apache-airflow-providers-apache-livy` * `apache-airflow-providers-databricks` * `apache-airflow-providers-dbt-cloud` * `apache-airflow-providers-microsoft-mssql` * `apache-airflow-providers-sftp` * `apache-airflow-providers-snowflake` If your DAGs depend on any of these provider packages, add the provider packages to your Astro project `requirements.txt` file before upgrading. You can also [pin specific provider package versions](#step-3-optional-pin-provider-package-versions) to ensure that none of your provider packages change after upgrading. ##### Upgrade to Python 3.10 Astro Runtime 8 uses Python 3.10. If you use provider packages that don't yet support Python 3.10, use one of the following options to stay on Python 3.9: * Run your tasks using the `KubernetesPodOperator` or `PythonVirtualenvOperator`. You can configure the environment that these tasks run in to use Python 3.9. * Use the [`astronomer-provider-venv`](https://github.com/astronomer/astro-provider-venv) to configure a custom virtual environment that you can apply to individual tasks. ##### Provider incompatibilities There is an incompatibility between Astro Runtime 8 and the following provider packages installed together: * `apache-airflow-providers-cncf-kubernetes==6.1.0` * `apache-airflow-providers-google==10.0.0` That can be resolved by pinning `apache-airflow-providers-google==10.9.0` or greater in your `requirements.txt` file. This incompatibility results in breaking the `GKEStartPodOperator`. This operator inherits from the `KubernetesPodOperator`, but then overrides the hook attribute with the `GKEPodHook`. In the included version of the `cncf-kubernetes` providers package, the `KubernetesPodOperator` uses a new method, `get_xcom_sidecar_container_resources`. This method is present in the `KubernetesHook`, but not the `GKEPodHook`. Therefore, when it is called, it causes the task execution to break. #### Runtime 6 (Airflow 2.4) Smart Sensors were deprecated in Airflow 2.2.4 and removed in Airflow 2.4.0. If your organization still uses Smart Sensors, you need to start using deferrable operators. See [Deferrable operators](/docs/learn/deferrable-operators). #### Runtime 5 (Airflow 2.3) Astro Runtime 5.0.0, based on Airflow 2.3, includes changes to the schema of the Airflow metadata database. When you first upgrade to Runtime 5.0.0, consider the following: * Upgrading to Runtime 5.0.0 can take 10 to 30 minutes or more depending on the number of task instances that have been recorded in the metadata database throughout the lifetime of your Deployment on Astro. * After you upgrade successfully to Runtime 5, you might see errors in the Airflow UI that warn you of incompatible data in certain tables of the database. For example: ```txt wrap theme={null} Airflow found incompatible data in the `dangling_rendered_task_instance_fields` table in your metadata database, and moved... ``` These warnings have no impact on your tasks or DAGs and can be ignored. If you want to remove these warning messages from the Airflow UI, reach out to [Astronomer support](https://cloud.astronomer.io/open-support-request). If requested, Astronomer can drop incompatible tables from your metadata database. For more information on Airflow 2.3, see ["Apache Airflow 2.3.0 is here"](https://airflow.apache.org/blog/airflow-2.3.0/) or the [Airflow 2.3.0 changelog](https://airflow.apache.org/docs/apache-airflow/2.3.0/release_notes.html#airflow-2-3-0-2022-04-30).