August 10, 2023

Test Airflow Upgrades with the Astro CLI

David Koenitzer Product Manager Astronomer

Current Problems with Upgrading Airflow

Upgrading major and minor versions of Airflow can lead to several challenges for users, potentially causing their environments and DAGs to break. The most common issues we see include:

Incompatible Dependencies: New Airflow versions or providers may require dependency versions that are incompatible with a user’s existing DAGs, resulting in import errors during upgrades.
API Changes: Updates to the Airflow API in newer versions may break compatibility with existing DAGs, leading to DAG import errors during major Airflow updates.

Currently, users resort to trial and error to understand which DAGs need to be fixed. That makes the Airflow upgrade process time-consuming and error-prone. Some teams develop scripts to identify changes in Airflow dependencies between their current version and the version they are upgrading to, which can help identify issues. However, the process is still far from optimal and can be improved.

Why We Created a Local Upgrade Test Command

We developed a Local Upgrade Test command in the Astro CLI to eliminate these upgrade pains and give Astro users the confidence (and passed tests) they need to upgrade safely. This feature provides a more straightforward and efficient way for users to prepare for an upgrade by simplifying the upgrade process and offering local testing to avoid deployment corruptions. We realize that users need tools to simplify the upgrade process and help them identify and debug issues related to upgrades effectively. With the Local Upgrade Test command, users can test upgrades locally, mitigating the risk of deploying potentially problematic changes to their production environments running on Astro.

Overview of the Upgrade Test Command

The Upgrade Test command in the Astro CLI offers two main functionalities to assist users in testing and preparing for Airflow upgrades:

Version Test

The Version Test feature assists users in identifying the Airflow Providers and python packages that have changed with the new Runtime version. The CLI will categorize these changes by which packages have been added, removed, or had a major, minor, or patch update. Users can use this information to determine which packages may have had an update that broke their DAGs.

Apache Airflow Update:
apache-airflow 2.5.3+astro.4 >> 2.6.3+astro.2

Airflow Providers Major Updates:
apache-airflow-providers-google 8.12.0 >> 10.0.0
apache-airflow-providers-microsoft-azure 5.3.1 >> 6.2.1

Added Airflow Providers:
apache-airflow-providers-datadog==3.3.1

Major Updates:
aiofiles 22.1.0 >> 23.1.0
apispec 3.3.2 >> 5.2.2
azure-batch 13.0.0 >> 14.0.0

Removed Packages:
PyHive==0.6.5
PyNaCl==1.5.0

Example Dependency Comparison file for an upgrade

DAG Test

The DAG Test feature assists users in identifying DAGs that are not importing due to incompatibility with the new Airflow or Airflow Provider version. By providing a list of DAGs with import errors, users can easily determine which DAGs need adjustments to work with the new Airflow version.

alt_text

DAG test showing that three broken DAGs were detected after an upgrade

How to use the Local Upgrade Test Command

Install the Astro CLI.
Create an Astro Project and add your requirements and DAGs
Run the astro dev upgrade-test command. Read our documentation for detailed instruction.

Upgrade Example

Let’s walk through an example of using the Local Upgrade Test command to fix conflicts and broken DAGs during an upgrade:

Fixing Broken DAGs: After resolving conflicts, the user may still face issues with DAGs that are not importing correctly due to compatibility problems with the new Airflow or Provider version. The Version Test and DAG Test features come to the rescue by providing a list of DAGs with import errors and a list of Provider package updates. With this list the user can check for breaking changes in the updated packages’ changelogs. With this information they can make the necessary adjustments to their DAGs to ensure compatibility with the new Airflow version.

What to Do Once Tests Show No Errors

Once the Local Upgrade Test command shows no errors, users should proceed to push their code and test DAGs to their development environment, either locally or in the cloud. For Astro customers, the astro deploy command can be utilized to set up continuous integration and continuous deployment (CI/CD) pipelines, enabling them to seamlessly push changes to their Astro deployments.

Other Considerations with Upgrades

Upgrading Airflow can be a time-consuming process, depending on the complexity of the changes and the number of DAGs in the environment. Users should allocate sufficient time for thorough testing and debugging before pushing upgrades to production. Additionally, it is essential to keep an eye on potential compatibility issues when introducing new dependencies or making significant changes to DAGs. With our managed solution and tooling upgrading Airflow can become much easier for your team.

Conclusion

The new Local Upgrade Test command in the Astro CLI offers an efficient and user-friendly solution to tackle the challenges of upgrading Airflow. By helping users identify and resolve compatibility issues, and DAG import errors, we aim to streamline the upgrade process and empower users to confidently deploy Airflow updates. With the ability to test upgrades locally, users can mitigate the risk of breaking their production environments and enjoy a smoother upgrade experience overall. So, upgrade with confidence and make the most of the Astro CLI’s new testing capabilities! You can even try out a free trial of Astro today, deploy a few of your DAGs, and then run the upgrade process.