The New Astro Runtime
2 min read |
Apache Airflow has long been the open-source standard for defining and running data workflows. And Astro has been Astronomer’s enterprise-grade managed platform for teams running those workflows in production.
Airflow is no longer used only for scheduled batch pipelines. Teams use it to coordinate thousands of data and AI workflows: loading warehouse tables, building dbt models, preparing training data, running model evaluations, and reacting to external events. These workloads can release tens of thousands of tasks at once, keep hundreds of thousands running, and still require a newly ready task to start in milliseconds.
Our customers have pushed the limits of what orchestration can do, so we reimagined what those limits could be: Airflow re-engineered for speed and scale.
Over the past several years, we've rebuilt Airflow's critical scheduling, execution, scaling, and recovery systems on Astro to perform beyond any current demands, in a way that still works seamlessly with Airflow. Nothing needs to change on your end.
Customers can now run 500,000 concurrent Airflow tasks. In our load tests, Astro reached 230ms p95 task-start latency at 100,000 concurrent tasks, more than 100 times lower than Celery's 23.582 seconds at half the load. At 300,000 concurrent tasks, p95 latency held at 300ms.
It doesn't get fragile as it gets fast. Astro sees 85% fewer failures than open-source Airflow under load: fewer silent task failures, one-click failover if a region goes down, upgrades you roll back instead of migrate through.
Coordination
Executionone executor per deployment
Platform
Astro Scheduler
Event-driven scheduling that stays fast
The Astro Scheduler reacts to new work as it arrives instead of waiting for a polling cycle. Task starts stay fast as trigger volume and concurrency grow.
Run event-driven pipelines and AI workflows alongside ETL, ML, and batch workloads—all on one Airflow platform.
p95 task start latency: Celery has 23.582 seconds of task lag at 50,000 concurrent tasks; Astro at 100,000 takes 228 milliseconds; and Astro at 300,000 takes 294 milliseconds.
Today we are extremely proud to announce this to the world. Soon, the engineering team behind the work will share a deep technical breakdown of how they did it.
Most importantly, we can't wait to see what you build on it.
