Airflow Provider Framework
Providers are Python packages containing all relevant modules for a specific service provider (ie. AWS, Snowflake, Google etc.)
Unlike Apache Airflow 1.10, now Airflow 2.0 is delivered in multiple, separate but connected packages. The core of the Airflow scheduling system is delivered as apache-airflow package and there are around 60 Airflow Provider Packageswhich can be installed separately.
In Airflow 2.0, Providers are decoupled from core Airflow releases:
- Providers can now be independently versioned, maintained, and released
- Users just `pip install` the provider package and then import modules into their DAG files
- Providers can now be built from any public repository
The Astronomer Registry
Recently we announced The Astronomer Registry: a discovery hub for Airflow providers designed to bridge the gap between the Airflow community and the broader data ecosystem.
The Airflow providers framework has long been a valuable asset for the community. DAG authors benefit from access to a deep library of standard patterns while maintaining code-driven customizability; it enables a tight integration between Airflow and almost any tool in the modern data stack.
The easiest way to get started with providers and Apache Airflow 2.0 is by using the Astronomer CLI. To make it easy you can get up and running with Airflow by following our Quickstart Guide.
