Develop with the CLIAdd Airflow providers and packages

Add Airflow providers, Python packages, and operating system packages

Most dags need additional Python or OS-level packages to run. You need to add Python packages, including Airflow Providers, to your Astro project’s requirements.txt file, and OS-level packages to the project’s packages.txt file.

There are two primary kinds of Python packages that you might need to add to your Astro project:

  • Non-provider Python libraries. If you’re using Airflow for a data science project, for example, you might use a data science library such as pandas or NumPy (numpy).
  • Airflow Providers. Airflow Providers are Python packages that contain relevant Airflow modules for a third-party service. For example, apache-airflow-providers-amazon includes the hooks, operators, and integrations you need to access services on Amazon Web Services (AWS) with Airflow. See Provider packages.

Adding the name of a package to the packages.txt or requirements.txt files of your Astro project installs the package to your Airflow environment.

  1. Add the package name to your Astro project. If it’s a Python package, add it to requirements.txt. If it’s an OS-level package, add it to packages.txt. The latest version of the package that’s publicly available is installed by default.

    To pin a version of a package, use the following syntax:

    <package-name>==<version>

    For example, to install NumPy version 1.23.0, add the following to your requirements.txt file:

    numpy==1.23.0

  2. Restart your local environment.

  3. Confirm that your package was installed:

    1astro dev bash --scheduler "pip freeze | grep <package-name>"

To learn more about the format of the requirements.txt file, see Requirements File Format in pip documentation. To browse Python libraries, see PyPi. To browse Airflow providers, see the Astronomer Registry.