Upgrade Runtime

Version upgrade considerations

Sometimes there are upgrade considerations for specific Astro Runtime versions. This includes breaking changes, database migrations, and other considerations.

If an Astro Runtime version isn’t included in this section, then there are no specific upgrade considerations for that version.

Runtime 3.0 (Airflow 3.0)

Runtime 3.0-9

This Runtime version is restricted and you can’t upgrade to it or create Astro Deployments using it. Airflow 3.0.5 was yanked due to a bug with conn.extra_dejson masking that causes tasks to fail (#54768), so the underlying Airflow version is not available for use.

Runtime 3.0-2

The Remote Execution Agent based on Airflow 3.0-1 has a known incompatibility with the Astro Runtime 3.0-2. If you use remote execution, do not upgrade to or create Remote Deployments that use the combination of Runtime Execution Agent 1.0.0 and Astro Runtime 3.0-2. A fix for the incompatibility will be released in Astro Runtime v3.0-3.

If you created a new remote Deployment with 3.0-2, you cannot rollback to 3.0-1. You must either create a new remote Deployment with 3.0-1 or wait until the 3.0-3 patch and then upgrade.

If you upgraded a Deployment from 3.0-1 to 3.0-2, Astronomer advises you to downgrade your Deployments or rollback to 3.0-1 by using a Dockerfile change or by rolling back your Deployment in the Astro UI.

Runtime 12 (Airflow 2.10)

Upgrade to Python 3.12

Starting with Python version 3.12, Python has removed the module imp. This can cause errors for your dags if you use imp to access materials using the import statement. Python recommends migrating to importlib instead.

See How do I migrate from imp guidance from Python for remediation steps.

Runtime 11 (Airflow 2.9)

Restricted versions of Runtime 11

You cannot create new Deployments or upgrade to the following versions of Runtime 11, which are yanked. These restricted runtime versions prevent you from upgrading to or creating a Deployment with a version that contains a known limitation or bug.

  • 11.0.0
  • 11.1.0
Bug affecting users running a local Astro Runtime environment

In Airflow 2.9, a bug affecting custom actions in Airflow plugins (#39421) prevented users from running the Astro Runtime environment locally for Astro Runtime versions 11.0.0, 11.1.0, and 11.2.0. Deployments running these versions on Astro are not affected.

To continue using these versions of the Astro runtime locally, set AIRFLOW__ASTRONOMER__UPDATE_CHECK_INTERVAL=0 in your Astro project .env file.

Runtime 9 (Airflow 2.7)

Restricted versions of Runtime 9

You cannot create new Deployments or upgrade to the following versions of Runtime 9, which are yanked. These restricted runtime versions prevent you from upgrading to or creating a Deployment with a version that contains a known limitation or bug.

  • 9.0.0
  • 9.1.0
  • 9.2.0
  • 9.3.0
  • 9.4.0
  • 9.5.0
  • 9.6.0
Connection testing in the Airflow UI disabled by default

In Airflow 2.7, connection testing in the Airflow UI is disabled by default. Astronomer does not recommend reenabling the feature unless you fully trust all users with edit/ delete permissions for Airflow connections.

To reenable the feature, set the following environment variable in your Astro project Dockerfile:

1ENV AIRFLOW__CORE__TEST_CONNECTION=Enabled
Upgrade to Python 3.11

The base distribution of Astro Runtime 9 uses Python 3.11 by default. Some provider packages, such as apache-airflow-providers-apache-hive, aren’t compatible with Python 3.11.

To continue using these packages with a compatible version of Python, upgrade to the Astro Runtime Python distribution for your desired Python version.

Runtime 8 (Airflow 2.6)

Astro Runtime version 8.0.0 is a restricted version of the Astro Runtime (yanked), which means you can’t create Deployments on Astro with this runtime version. These restricted runtime versions prevent you from upgrading to or creating a Deployment with a version that contains a known limitation or bug.

Breaking change to apache-airflow-providers-cncf-kubernetes in version 8.4.0

Astro Runtime 8.4.0 upgrades apache-airflow-providers-cncf-kubernetes to 7.0.0, which includes a breaking change by removing some deprecated features from KubernetesHook. If you are using any of these features, either pin your current version of apache-airflow-providers-cncf-kubernetes to your requirements.txt file or ensure that you don’t use any of the deprecated features before upgrading.

Upgrade directly to Astro Runtime 8.1

Astro Runtime 8.0 introduced a number of bugs and dependency conflicts which were subsequently fixed in Runtime 8.1. As a result, Astro Runtime 8.0 is not available in the Astro UI and no longer supported by Astronomer. To use Airflow 2.6, upgrade directly to Runtime 8.1.

Package dependency conflicts

Astro Runtime 8 includes fewer default dependencies than previous versions. Specifically, the following provider packages are no longer installed by default:

  • apache-airflow-providers-apache-hive
  • apache-airflow-providers-apache-livy
  • apache-airflow-providers-databricks
  • apache-airflow-providers-dbt-cloud
  • apache-airflow-providers-microsoft-mssql
  • apache-airflow-providers-sftp
  • apache-airflow-providers-snowflake

If your dags depend on any of these provider packages, add the provider packages to your Astro project requirements.txt file before upgrading. You can also pin specific provider package versions to ensure that none of your provider packages change after upgrading.

Upgrade to Python 3.10

Astro Runtime 8 uses Python 3.10. If you use provider packages that don’t yet support Python 3.10, use one of the following options to stay on Python 3.9:

  • Run your tasks using the KubernetesPodOperator or PythonVirtualenvOperator. You can configure the environment that these tasks run in to use Python 3.9.
  • Use the astronomer-provider-venv to configure a custom virtual environment that you can apply to individual tasks.
Provider incompatibilities

There is an incompatibility between Astro Runtime 8 and the following provider packages installed together:

  • apache-airflow-providers-cncf-kubernetes==6.1.0
  • apache-airflow-providers-google==10.0.0

That can be resolved by pinning apache-airflow-providers-google==10.9.0 or greater in your requirements.txt file.

This incompatibility results in breaking the GKEStartPodOperator. This operator inherits from the KubernetesPodOperator, but then overrides the hook attribute with the GKEPodHook. In the included version of the cncf-kubernetes providers package, the KubernetesPodOperator uses a new method, get_xcom_sidecar_container_resources. This method is present in the KubernetesHook, but not the GKEPodHook. Therefore, when it is called it causes the task execution to break.

Runtime 6 (Airflow 2.4)

Smart Sensors were deprecated in Airflow 2.2.4 and removed in Airflow 2.4.0. If your organization is still using Smart Sensors, you’ll need to start using deferrable operators. See Deferrable operators.

Runtime 5 (Airflow 2.3)

Astro Runtime 5.0.0, based on Airflow 2.3, includes changes to the schema of the Airflow metadata database. When you first upgrade to Runtime 5.0.0, consider the following:

  • Upgrading to Runtime 5.0.0 can take 10 to 30 minutes or more depending on the number of task instances that have been recorded in the metadata database throughout the lifetime of your Deployment on Astro.

  • Once you upgrade successfully to Runtime 5, you might see errors in the Airflow UI that warn you of incompatible data in certain tables of the database. For example:

    Airflow found incompatible data in the `dangling_rendered_task_instance_fields` table in your metadata database, and moved...

    These warnings have no impact on your tasks or dags and can be ignored. If you want to remove these warning messages from the Airflow UI, reach out to Astronomer support. If requested, Astronomer can drop incompatible tables from your metadata database.

For more information on Airflow 2.3, see “Apache Airflow 2.3.0 is here” or the Airflow 2.3.0 changelog.