Apache Airflow
The open source standard for workflow orchestration.
Apache Airflow is a way to programmatically author, schedule and monitor your data pipelines using Python and SQL.
Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top-Level Apache Project in 2019. Now, it’s widely recognized as the industry’s leading data orchestration solution.
With over 140 integrations and more added each month, the power of the community ensures that Airflow has the most comprehensive coverage of data sources and other providers, whatever your use case.
Built on a strong and growing community.
Apache Airflow Core Principles
Airflow is built on a set of core ideals that allow you to leverage the most popular open source workflow orchestrator on the market while maintaining enterprise-ready flexibility and reliability.
Flexible
Fully programmatic workflow authoring allows you to maintain full control of the logic you wish to execute.
Extensible
Leverage a robust ecosystem of open source integrations to connect natively to any third party datastore or API.
Open Source
Get the optionality of an open-source codebase while tapping into a buzzing and action-packed community.
Scalable
Scale your Airflow environment to infinity with a modular and highly-available architecture across a variety of execution frameworks.
Secure
Integrate with your internal authentication systems and secrets managers for an platform ops experience that your security team will love.
Modular
Plug into your internal logging and monitoring systems to keep all of the metrics you care about in one place.
Discover new features of Airflow 2.0
A New, Highly Available SchedulerNew
Expect faster performance with near-zero task latency. Launch Scheduler replicas to increase task throughputand ensure high-availability. Read more
Full REST APINew
Build programmatic services around your Airflow environment with Airflow's new API, now featuring a robust permissions framework. Airflow Docs
Smart SensorsNew
Fit Airflow to event-driven paradigms with resource-savingsensors that operate as single, long-running tasks and don't take up a worker slot. Airflow Docs
Task Flow APINew
Pass information between tasks with clean, efficient code that's abstracted from the task dependency layer. Includes support for custom XCom backends. Airflow Docs
Task GroupsNew
Replace SubDAGs with a new way to group tasks in the Airflow UI. Task Groups don't affect task execution behavior and do not limit parallelism. Airflow Docs
Simplified KubernetesExecutorNew
Make the most of the
pod_overrideparameter for easy 1:1 overrides and the new yamlpod_template_file, which replaces configs set in airflow.cfg. Airflow DocsIndependent ProvidersNew
Pull the latest version of any Airflow Provider at anytime, or follow an easy contribution process to build your own and install it as a Python package. Airflow Docs
UI/UX ImprovementsNew
Embrace the a new look for the Airflow UI, now boasting a clean, modern color palette and an auto-refresh toggle in the Graph view. Airflow Docs
Read our Introducing Airflow 2.0 blog postand the official Apache Airflow 2.0 release announcement.
Do Airflow the easy way.
Run production-grade Airflow out-of-the-box with Astronomer.
What's in Airflow 2.0?
Check out our Airflow 2.0 video series for a walkthrough of all of the great new features.
Find the information you’re looking for.
ResourcesAirflow, made better by Astronomer