Apache Airflow continues to win vs. other workflow orchestration tools due to two main factors:
- An active and expanding community
- Very deep, proven functionality
As a result, thousands of companies — including Fortune 500s, tech giants, and early-stage startups — are adopting Airflow as their “master” data pipeline orchestration tool.
The Airflow community is strong, healthy, and vibrant, with over 1,000 code contributors — and growing at a healthy pace.
The velocity of Github stars for Airflow continues to accelerate.
And as Airflow approaches a Version 2 release, the project has entered a second golden age of productivity in the past three quarters, eclipsing the number of commits from even the early days when the first generation of project developers Maxime, Bolke, Jeremiah, Chris, Sid and Arthur were building it out.
The Airflow project was already gaining momentum in 2018 thanks to open-source activities in companies like Airbnb, Lyft, Twitter, Glassdoor, and many, many others.
In 2019, an elevated level of activity has been driven by companies like Astronomer, Google, Polidea, and GoDataDriven, who are dedicating significant resources to the project, including an expanded group of people working on the project full-time.
Along the way, Airflow has accumulated an impressive amount of functionality that solves many data engineering challenges both on-prem and on any cloud provider. Being built on the backs of numerous data engineers working to solve countless edge cases, it’s a complete solution.
While all Airflow functionality isn’t evenly polished, work is now in motion to ensure that that isn’t true for much longer. The community is working on a multitude of projects including:
- Speeding up the scheduler
- Speeding up the web server
- Integrating with Knative
- An official production Docker image
- Building a world-class API
- Building a new UI
- Putting in place end-to-end integration and performance tests
So while Airflow isn’t perfect, its deep functionality and strong community pose a big challenge to “fresh start” alternatives. And with an accelerating number of people and organizations pouring energy into and committing to Airflow, the future is bright.
Ready to run production-grade Airflow?
Astronomer is the easiest way to run Apache Airflow. Choose from a fully hosted Cloud option or an in-house Enterprise option and run a production-grade Airflow stack, including monitoring, logging, and first-class support.