---
title: 'The Astro Runtime: Airflow re-engineered for speed and scale'
description: >-
Open-source Airflow finds ready work on a timer; Astro gets a hint. 100x lower
p95 task-start latency at twice the load, on the same database and interfaces.
date: 2026-08-31T00:00:00.000Z
authors:
- author: src/content/people/ian-buss.md
- author: src/content/people/Michael-Claassen.md
- author: src/content/people/jed-cunningham.md
- author: src/content/people/neel-dalsania.md
- author: src/content/people/julian-laneve.md
- author: src/content/people/carter-page.md
tags:
- Apache Airflow
- Astro
- infrastructure
categories:
- engineering-blog
canonical_url: >-
https://www.astronomer.io/blog/astro-airflow-re-engineered-for-speed-and-scale/
---
Apache Airflow is the open-source standard for defining and running data
workflows. Astronomer builds Astro, a managed Airflow platform for teams running
those workflows in production.
Airflow is no longer used only for scheduled batch pipelines. Teams now use it
to coordinate thousands of data and AI workflows across an enterprise: loading
warehouse tables, building dbt models, preparing training data, running model
evaluations, and reacting to external events. These workflows can release tens of thousands of tasks at once, keep hundreds of thousands running, and still
require a newly ready task to start in hundreds of milliseconds.
Our customers hit ceilings in how fast Airflow starts a task and how many it
keeps running, so we rebuilt the path behind both.
Over the past several years, we have rebuilt Airflow's scheduling, execution, scaling, and recovery systems on Astro to perform beyond any current
demands. And we did it in a way where Astro still works seamlessly with Airflow
3+ releases, meaning nothing needs to change with how pipelines are defined.
In our load tests, Astro sustained **500,000 concurrent Airflow tasks** in a
single deployment and reached **228 milliseconds p95 task-start latency at 100,000 concurrent tasks**—less than one-hundredth of the 23.582 seconds we measured when we pushed
open-source Airflow to just half that load. At 300,000 concurrent tasks, p95 remained 294 milliseconds.
We made these changes without replacing Airflow's workflow model.
User Dags, task dependencies, task execution status, and operator code remain
compatible with open-source Airflow. Astro just changes components under the
hood to make task execution faster and more scalable while improving
reliability:
_Heavy borders mark the systems Astronomer built; the rest is Apache
Airflow. This is the control path around a task, not every service in a
deployment._
## How Airflow got here
Airflow began 10 years ago with a clear job: read workflows defined as Python
code, decide which tasks were ready, and send them somewhere to run. A scheduler
made those decisions, an executor handled the delivery, and a SQL database
recorded Dag runs and task states.
That design gave Airflow two useful properties. Dag authors could choose how
tasks ran without changing their workflow code, and a failed scheduler could
rebuild its view from the database. The database, rather than any scheduler
process, held the lasting account of the work.
As deployments added more Dags and tasks, the scheduler had to parse more
Python files, create more Dag runs, check more dependencies, and queue more
task instances. One scheduler also left the whole deployment dependent on one
process. Airflow 2.0 and beyond addressed those limits with a few large changes, without replacing the database model.
[AIP-15](https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/103092651/AIP-15%2BSupport%2BMultiple-Schedulers%2Bfor%2BHA%2BBetter%2BScheduling%2BPerformance)
let several scheduler replicas run at once. Each could examine Dag runs in
parallel, then use database row locks when moving task instances from
`scheduled` into an executor. Airflow also changed to store
[serialized Dags](https://airflow.apache.org/docs/apache-airflow/2.3.3/dag-serialization.html)
in the database, so a scheduler could make decisions from a stored graph
instead of importing every Python Dag file itself.
Airflow 2.2 introduced ways to offload tasks that were waiting for some
external system via the triggerer. Consider a daily ingestion Dag that cannot
start until a partner uploads `complete.json` to an object-storage bucket. A
sensor may spend hours checking for that file while doing almost no work, yet it
can still occupy a worker slot. A
[deferrable operator](https://airflow.apache.org/docs/apache-airflow/2.2.2/authoring-and-scheduling/deferring.html)
stores what it needs to resume, releases the worker, and hands the wait to a
triggerer. When the file arrives, the trigger fires and the scheduler checks
the task again before it can run.
Airflow 2.3 then introduced the separate Dag parser component, which separated
out user code from the core scheduler component and loop. This was much better
from a security/isolation perspective and also meant that Dag processing could
scale separately from scheduling.
Airflow 3 gave task execution a clear API contract. Under
[AIP-72](https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/311626182/AIP-72%2BTask%2BExecution%2BInterface%2Baka%2BTask%2BSDK),
task processes no longer need direct access to the metadata database like they
did in Airflow 2.x. The Task SDK and Execution API define what a task receives,
including its identity, connections, and variables, and what it reports,
including heartbeats, state changes, and XComs (used for cross-task
communication). Workers and task code use that interface instead of reading and
updating Airflow's internal tables.
Modern Airflow divides the original system's work among Dag processing,
active-active schedulers, triggerers, executors, API servers, and workers. This
is the architecture we helped build in Airflow: durable database state,
active-active scheduling, deferred execution, and a clear task API. At Astronomer, we led the HA scheduler, the triggerer and deferred tasks, and
the Task SDK and Execution API, and we contributed to moving Dag processing out
of the scheduler loop. As deployments grew and workflows carried more weight,
we also saw where the task-start path needed a different design.
## The lifecycle of an Airflow task
Here is what happens after one task finishes, first in open-source Airflow and
then in Astro. The Dag is a basic three-task ETL.
In our example Dag, when `extract` succeeds, Airflow records that result in the
metadata database. The scheduler must notice the change, confirm that
`transform` may run, check shared limits, queue it through the configured
executor, and wait for a worker to start its process. Only then does `transform`
begin running.
These steps run in order. `transform` cannot skip any of them, so each adds its
own wait to the total.
### 1. The upstream task finishes
The `dag_run` state for `example_etl` is running, the `task_instance` state for
`extract` is `success`, and the rows for `transform` and `publish` have a `NULL`
state. Airflow reads the dependency from the serialized Dag and evaluates it
against the task states for this run on a loop.
The lasting record of `transform` is its `task_instance` row, not an in-memory
object passed from one process to another. Different parts of Airflow update
that row as they make decisions about it.
### 2. A scheduling pass must discover the change
The standard scheduler works in batches. It queries for active Dag runs, calls
Airflow's dependency logic, and returns later for another pass, on a loop. Until
one of those passes examines our Dag run, `transform` remains `NULL` even though
`extract` has already recorded `success`.
Every tuning knob here trades one cost for another. Larger batches raise
throughput but let one scheduler hoard runs; running the loop more often cuts
the wait but issues more queries when nothing has changed, and at high load
those passes compete with heartbeats, state updates, and API calls for the same
database. Adding schedulers helps throughput and availability, but the replicas
still coordinate through database locks. When a scheduler finds `transform` and
its dependencies pass, it changes the row to `scheduled`.
_By this point, `transform` has waited for a scheduling pass and dependency
checks._
### 3. Shared limits must clear before the task queues
When a task is `scheduled`, that does not necessarily mean it's ready to run.
Airflow has a robust queueing and pooling system, which lets a Dag author
declare cross-Dag behavior for how tasks run. For example, an external service
may impose API rate limits, so the author declares a pool that limits how many
tasks interacting with that API can run.
Before changing a task to `queued`, the scheduler must account for limits shared
by many Dags: pool slots, global system concurrency, Dag concurrency, task
concurrency, and executor capacity. Its critical section protects those shared
limits while it selects a batch of tasks. More scheduler replicas can therefore
increase both scheduling capacity and contention for the same database state.
_By this point, `transform` has waited for a scheduling pass, dependency checks,
shared-limit checks, and a database update._
### 4. The executor must deliver the workload
Airflow separates the decision to run a task from the system that starts it.
Once the scheduler marks `transform` as `queued`, the configured executor takes
over. LocalExecutor starts a process on the same machine, CeleryExecutor sends a workload to a pool of workers through a broker, usually
Redis or RabbitMQ, or KubernetesExecutor asks
Kubernetes to create a pod.
We focus on
[CeleryExecutor](https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/celery_executor.html)
because it has long been a common choice for distributed Airflow deployments
and thus is the open-source execution path used in our benchmarks. Celery adds a
delivery chain on top of the scheduling work: the scheduler serializes a
workload, the broker stores it on the right queue, and a worker consumes it
subject to its free concurrency. During a burst, every link in that chain
handles more connections, deeper queues, and workers already holding prefetched
work—and adding workers does not remove the broker round trip.
_By this point, `transform` has waited for a scheduling pass, dependency checks,
shared-limit checks, a database update, and a broker round trip._
### 5. A worker must have a free slot and start the process
Receiving the workload does not start the task. `transform` remains `queued`
until the Celery worker has a concurrency slot and starts a child process. Under
Airflow 3, that process uses the Task SDK and Execution API to report that it is
`running`, send heartbeats, fetch the values it needs, and report its final
state.
If every matching worker is full, the workload waits in the broker. If no worker
exists, the deployment must start a worker VM or pod before the process can run.
Broker wait, worker capacity, and infrastructure startup therefore all count
toward the delay after the scheduler queues the task.
_By this point, `transform` has waited for a scheduling pass, dependency checks,
shared-limit checks, a database update, a broker round trip, and a worker slot.
Only now does it become `running`._
### Capacity must already exist
For there to be no infrastructure holdups, every step above assumes the
schedulers, database, broker, and workers it needs are already running. That
assumption has its own timing problem.
**Infrastructure metrics describe what a deployment is doing now, while
Airflow's tables often show what it will need next.** A trigger can wait for an
external event while using little CPU, then make thousands of task instances
runnable when the event arrives. A timetable can show that a large Dag run is
due before its first task consumes any worker resource. A growing queue can
appear in the database before a queue depth metric crosses a scaling threshold.
If queue depth is the only scaling input, new workers begin to start only after
the existing workers are busy and tasks are waiting. Schedulers, API servers,
triggerers, and database proxies have the same timing problem: each sees one
part of demand, but a task needs capacity across the whole path.
### How the waits add up
At modest load, each stage can finish quickly. As task counts grow, the wait at every gate below grows with it.
Two of those waits happen after the row says `queued`. The row does not
change again, so nothing you can query tells you which wait the task is in.
The usual fixes address one step at a time:
- Polling faster just makes the database do more work.
- Add schedulers, and shared locking and database work remain.
- Tune Celery, and the broker remains a second delivery system.
- Scale on CPU, and the metric moves after Airflow has already created the
demand.
The same choices that give Airflow durable state, recovery, and broad executor
support also define this task-start path. We kept those properties and changed what happens between the database write
and the running process.
## Making it go bigger and faster
Our job at Astronomer is to make Airflow work at the largest scales our
customers run, and the waits in the lifecycle above all have the same shape:
loops. The scheduler loops over Dag runs looking for ready tasks. Workers sit
waiting for tasks. The autoscaler waits for task demand deciding when to add
capacity. Each loop wakes on its own timer, checks for changes, and goes back to
sleep. A task starts only after every loop on its path has come around. Yet the
process that wrote `success` already had the information every one of those
loops is polling for. Airflow gives it no way to pass that on.
So we rebuilt the systems that start a task: whoever changes state now tells the
next service to look, right away, by passing along a "hint" instead of leaving
each loop to find the change on its own timer. The Astro Scheduler replaces
polling discovery and reacts the moment work changes. The Astro Executor
replaces the Celery delivery path and assigns tasks to workers directly. The
Astro Hypervisor scales deployments according to schedules operators define and
prepares capacity from Airflow's own state before tasks need it. Around them,
the Astro Runtime image packages the software the path runs, and cross-region recovery
protects the database everything else depends on.
We also studied each component to understand its scaling properties in an effort
to increase scalability. As customers demand more of the system, we want to make
sure that it's not only faster, but also can be run with more (and more
frequently running) Dags and tasks. For example, we undertook a huge effort with
the Airflow scheduler to reduce the database critical section so that we can run
across more replicas in parallel. For reference, with the standard scheduler we can generally run up to 10 replicas but with the Astro Scheduler
we've run hundreds of replicas concurrently.
An effort like this only counts if the result is still Airflow: the same Dags,
the same task states, the same operators, and the ecosystem built on them.
Rebuilding the core of a large open-source project risks producing a fork that
shares only the name. So we fixed two design principles before we wrote code:
1. **The Airflow metadata database remains the only source of truth for Dag-run
and task execution status**
2. **Existing Airflow Dags and tasks do not need to be rewritten to use the new
path**
The first principle keeps the hints we pass around deliberately small. On the
new path, services do not generally poll to find work. When a service changes
state, it publishes a hint—a few bytes over a message broker—that wakes the next
service. The hint can say which `dag_id` to look at, and nothing more. It does not carry the workload, and it does not replace the
`dag_run` and `task_instance` rows.
Why not put the full details in the hint? Because the hint would then become a
second record of the work and recovery would have to keep that record and the
database in agreement after every crash. Small, disposable hints avoid the
problem. The database stays the only record of the work, and the fallback checks still run if a hint is lost.
The second principle holds because the new path is built on Airflow's own
interfaces. `extract`, `transform`, and `publish` still use Airflow's task
states and dependency rules. The new scheduler uses Airflow's internal
scheduling and task state functions as much as possible to avoid creating a fork
situation, where we're racing to keep up with the open-source project. (Note:
We've actually seen customers and prospects fork Airflow with their own
customizations, and it inevitably leads to lots of maintenance work and a very
painful upgrade.) The Astro Executor implements Airflow's existing executor
contract and Airflow 3's Task SDK carries the workload back through the Execution API.
Together, the two rules split the path so each part has one job:
- The metadata database stores Dag runs, task states, ownership, and other facts
that must survive a process restart.
- The message broker carries small hints that something changed.
- The Astro Scheduler applies Airflow's Dag-run and dependency logic when a hint
arrives.
- The Astro Executor assigns queued tasks to workers without a Celery broker, as
soon as a hint reaches a worker with room.
- Workers ask for work and run assigned task processes.
- The Astro Hypervisor reads Airflow and Kubernetes state to scale services and
repair known faults.
Database discovery remains as the recovery path for a hint lost after a
successful API request, a hint removed by a process that then stops, or a
scheduler that restarts with no local record of its Dag runs. The fallback scans
still cost database work, but they no longer set the usual task-start delay.
## The Astro Scheduler follows hints
Not every run needs a
[sub-second start](https://www.astronomer.io/docs/astro/sub-second-pipelines). A
daily run that starts at midnight every night can afford the standard
scheduler's next pass. The runs that cannot wait—an external system reacting to
an event, a service fanning out work—typically arrive through the API. So that
is where the new path begins: eligible Dags take it, and other Dags continue through the standard scheduler (for now, although we're continuing work on
our scheduler to let it support a wider variety of Dags).
We've designed every hop on this path around the same three things: 1) commit
the change to the database 2) drop a hint, and 3) the next service wakes, reads
the database, and acts. To see how this works in action, let's return to our
three-task Dag. Step through it once before reading the hops below.
### Hop 1: the API creates a run
An external system creates a run with `POST /api/v2/dags/example_etl/dagRuns`.
The API transaction inserts one `dag_run` row and three `task_instance` rows,
telling Airflow that it has work to do. Only after that transaction commits does
an API-server plugin push a `schedule_dagruns` hint to a message broker. Its
payload identifies `example_etl`; it does not repeat the Dag run, the task rows, or the workload. The message broker tells a scheduler where to look, and the
scheduler can then look at the database for the state it must act on, so the hint is a pointer, not the record.
The Astro Scheduler consumes the hint from our message broker and then runs one
database statement that does two jobs:
1. It finds eligible queued runs for that Dag that do not already have an owner.
2. It inserts ownership rows for as many runs as the replica has room to manage.
We make use of PostgreSQL's `ON CONFLICT DO NOTHING` so that two replicas can
race on the same run, but only one insert returns that run to its caller.
On its first pass it loads the Dag run record, changes it from `queued` to `running`, and commits. It then reuses as much of Airflow's internal scheduling
logic to maintain behavior with the open-source project. (As Ash Berlin-Taylor,
a prominent Airflow PMC member, likes to say, for a project as large and
well-adopted as Airflow, we need to keep even the "bugs" consistent.) For a new
run, that may make `extract` runnable.
### Hop 2: a task finishes
The same pattern moves the run forward. When `extract` finishes, its worker
reports `success` through the Execution API, which turns into a database write
to the `task_instance` table. A task-state hint wakes the run's scheduler, which
calls the same dependency functions and finds `transform` ready. And again, we
re-use Airflow's actual dependency engine here so that as we add more
functionality to the open-source Airflow project, we're not maintaining our own
fork.
For each runnable task, the scheduler performs a guarded database update. The
production statement selects rows by their Airflow task-instance identity; in
plain SQL, the important part is:
```sql
UPDATE task_instance
SET state = 'queued', queued_dttm = now(), external_executor_id = NULL
WHERE id IN (...)
AND state = 'scheduled';
```
The `state = 'scheduled'` guard matters. If another scheduler moved the row
after this replica read it, PostgreSQL updates zero rows. If the update
succeeds, the row now carries `state=queued`, the time it entered the queue, and
no worker assignment. The hint only told the scheduler when to look.
### Hop 3: a hint wakes the executor
After the commit, the Astro Scheduler publishes a wake-up hint via the API
server. We'll talk about this more in the next section, but the Astro Executor
workers will long poll the API server so that they consume work immediately,
giving the API server a mechanism to forward the hint to the workers directly.
And at this point, queued tasks already exist in the database, so the Astro
Executor can find them through its fallback check if the hint disappears.
_On this path, `transform` reached `queued` after hints and a few guarded
database statements—no polling pass and no batch tradeoff._
### The system recovers gracefully if hints break
We've designed this to be purely additive, so nothing actually _depends_ on
hints arriving. Each failure below ends the same way: a database check finds the
work, so we are, in effect, gracefully degrading to Airflow's normal behavior.
**The API succeeds, but hint publication fails.** Because the database
transaction committed first, the Dag run and its task instances still exist. The
plugin logs the broker error and returns the successful API result. On its next pass, the fallback check finds a `queued` or `running` run with no ownership row, takes ownership, and continues down our scheduler's path, usually in less than a
second.
**Two hints arrive for the same Dag.** Both can make replicas query the
database, but the guarded ownership insert returns the run to only one of them.
The database objects supply the identity, so the hint needs no ID of its own.
**A task-state hint disappears.** The state reported through Airflow's Execution
API remains in `task_instance` even if our message broker loses the hint that
would wake this run at once. The per-run scheduler later wakes on its fallback
timer and calls `update_state()` again; this usually happens within a few
seconds.
**An Astro Scheduler replica stops.** Each replica updates the heartbeat on its
ownership rows and if there hasn't been an active heartbeat, another replica can
remove the stale row, find the active Dag run, and take ownership. The replacement
reads the latest database state rather than replaying a workload held by the
failed process. A stale replica that resumes cannot double-queue a task: the
guarded `scheduled`-to-`queued` update is a no-op once another scheduler has
moved it, and worker assignment adds one more check in the next section.
**The message broker remains unavailable.** Every hint stops, so the fallback checks must find the runs and the task changes, and the Astro
Executor must poll for queued work—slower and heavier on the database, but
acting on the same stored rows. This degrades to
Airflow's natural behavior, running slower but still correctly.
At the end of this section, the `transform` task has status `queued`: the
database records that scheduling chose it, but no user process is running. A
worker must now ask the API server for work so the Astro Executor can assign the
task without putting a workload in a broker.
## The Astro Executor assigns `transform`
The Astro Executor removes the Celery broker between workers and the
executor in favor of workers long polling the Airflow API server. As described
above, this lets our hint system propagate events from the Airflow system
components (i.e. scheduler and API server) to the workers directly.
Each worker tells the API server how much room it has. The API server assigns a
queued task by writing the worker's ID onto the same database row the scheduler
queued, and the worker learns about the task in the response to a request it
already had open. On the worker, task processes fork from a pre-warmed parent,
so the process start is cheap too.
Each worker heartbeat includes its identity, the queues it serves, its total and
free task slots, and whether it is shutting down. A worker with no free slots
receives no work, while a cordoned worker—one marked to stop taking new work—can
finish its current tasks without taking another one.
When a worker has room, the API server holds its heartbeat request open until a
matching task appears. This long poll removes the need for the worker to open a
new request every few milliseconds.
Suppose `worker-17` reports four free slots on the `default` queue. The API
server's allocator—the code that assigns queued tasks to waiting workers—
registers that worker as a waiter, then selects a batch of queued, unassigned
tasks for the queues its waiting workers serve. The selection rests on two plain
PostgreSQL primitives. `FOR UPDATE` locks the selected rows for this
transaction. `SKIP LOCKED` makes this statement pass over rows another replica already holds
instead of waiting for them. Every replica runs the same statement, so they
fill worker capacity at once without two of them taking the same task. And if the
executor has a hint that a task is ready, it matches that workload against free worker slots and sends it at once.
For each selected row, the Astro Executor finds a waiting worker that serves the
queue and still has a free slot. When it chooses `worker-17`, it writes that
worker's ID to `external_executor_id`, decrements the capacity held for the
heartbeat, and builds the workload. The workload carries
`dag_id`, `run_id`, `task_id`, `map_index`, the try number, details of the Dag
bundle (the versioned set of Dag files the worker runs from), queue and trace
data, plus a signed token for the Airflow Execution API.
The task assignment and worker identity live in the same metadata row that the
scheduler already queued. There is no broker message to reconcile with that row.
The worker receives the task in its heartbeat response once the assignment is
written.
### The worker starts the process
The worker receives its assignments in the heartbeat response. A local
coordinator reserves a slot for each one, then writes the workload to the worker process.
The worker validates the task data and forks an Airflow supervisor process. The
parent records the child process ID and reports that the process started back to
the local coordinator. The child calls Airflow's Task SDK supervisor with the Dag
bundle, task instance, log path, signed token, and Execution API address. The
Task SDK then starts the task runtime that imports the Dag and enters user code.
Forking from a warm parent, while not unique to Astro, is part of what makes
that start cheap. Before any task arrives, the worker parent imports the
expensive parts of Airflow once (the Task SDK, the task runner, the standard
operators). Each forked task process inherits those imports instead of paying
for them again. To make this as fast as possible, we have measured and optimized
every single step and action between the workload arriving at the worker and the
user's task code actually running.
Before the task transitions to `running`, there's a final check with the API
server to mark it as such, which also serves as a final check against the same
task running on two workers at the same time.
_By this point, `transform` is `running`. It waited for a hint, a guarded
assignment, one heartbeat response, and a fork from a warm parent—no broker._
Airflow 3 makes this split possible. The task process does not need a database
password or a route to the metadata database. Its short-lived token lets the supervisor use the Execution API for heartbeats, state changes, connections,
variables, and XComs. The same contract means a worker can run _anywhere_ that
can reach the API server. Astro's remote execution is built on this: workers run
inside the customer's own network—a cloud account, a data center, an on-prem GPU
cluster—and pull work from an orchestration plane that never sees their code,
logs, or data.
When something fails, the database acts as the source of truth for how far the
task has progressed and what needs to be done to recover. A worker that
disappears while `transform` is still `queued` has lost only an assignment:
recovery clears `external_executor_id` from its rows, and another healthy worker
claims them. A partitioned worker that returns and starts its cleared assignment
anyway cannot cause a double start: every task process must report `running`
through the Execution API before it reaches user code, that transition is
guarded like every other state change, and only one process wins it. A worker
that disappears after the process reports `running` is harder, because user code
may already have changed an external system; Airflow's heartbeat timeout marks
the task failed and its retry rules decide whether another try runs. Airflow
does not guarantee only-once execution so Dag authors make side effects safe to
retry (i.e., tasks are idempotent). If every worker is full, the task stays
`queued` in the database until a worker has room. There is no in-memory list for
an API server restart to lose.
The Astro Scheduler normally publishes a wake-up hint after it queues work. That wakes the assignment loop without making the message broker the source of
the task. The loop also checks for work on a short fallback interval, so a missed
hint delays the assignment rather than losing it.
The difference is how many hops sit between a queued row and a running process.
On the same hardware, an Astro Executor worker can run 70% more concurrent tasks
than a Celery worker—more task processes per worker, not faster user code.
Combined with the Astro Scheduler, this execution path has sustained 500,000
concurrent tasks. The load tests at the end of this post give the test and the method behind that number.
## The Astro Hypervisor scales the system around `transform`
The task path reaches user code only when each service has capacity at the same
time: a worker needs a free slot, an API server must hold the heartbeat, and the
Astro Scheduler must have room to own the Dag run.
We originally built the Astro Hypervisor to help us manage the operations of
many Airflow deployments in a single cluster. We run one Hypervisor service in
every cluster to interact with potentially hundreds of Airflow deployments,
without needing to go to every deployment individually. In doing so, it also
acts as a translation layer—every Airflow version might have a slightly
different metadata database structure and API format, so our Hypervisor acts as
a central place to put logic to help decide what to do according to the Airflow
version. It also helps with things like database connection pooling for
efficiency.
Kubernetes can scale a service from CPU or memory, but those measures describe
load after it reaches a process. Airflow has earlier signals. It knows how many
tasks are queued, how many Dag runs remain active, which triggers are waiting,
which runs are due soon, and whether schedulers and workers still send
heartbeats.
The Astro Hypervisor turns that Airflow state into deployment-level decisions.
Currently this works very well for the Airflow triggerer, which needs to scale
rapidly because a single mapped task can result in thousands of deferred tasks
to be placed on the triggerer. We have prototypes of scaling some of the other
Airflow components, like the scheduler and API server, according to Airflow's
state. Having an Airflow-specific hypervisor allows us to more intelligently
make scaling decisions than if we were using a more general-purpose scaling
system.
For each deployment, the Hypervisor can read three groups of input:
- The Airflow metadata database: `task_instance` rows in `queued` or `running`
grouped by queue, active and upcoming Dag runs, late runs, `trigger` rows,
worker slots, scheduler heartbeats, and table sizes.
- Kubernetes: pods, Deployments, custom resources, and current replica counts.
- Service health and metrics: worker health, Astro Scheduler latency, and
component-level metrics.
Those facts can, in theory, lead to different actions:
| Stored or observed fact | Potential Hypervisor action |
| -------------------------------------------------------------------------------------- | ------------------------------------------------------- |
| An unfinished `dag_run` exists | Keep scheduling services available |
| A Dag's next run enters the look-ahead window | Start needed services before the run is due |
| Active rows remain in `trigger` | Keep a triggerer available |
| A Kubernetes task pod has no matching active task instance, or belongs to an older try | Check whether the pod is safe to remove |
| A required scheduler metric is missing | Block scheduler scale-down rather than treating it as 0 |
The controller set depends on the deployment's Airflow version, executor, and
features, so the Hypervisor does not apply one replica formula to every pod.
Some of Airflow's features make autoscaling things like the scheduler very
tricky; for example, a user can declare a "dynamic Dag" that changes every time
it's parsed. Scheduler autoscaling is the piece still in prototype. We are working through
these edge cases internally before we release it to customers.
### Scaling from planned work
Before `extract` finishes, worker CPU metrics have no way to show that
`transform` will soon become runnable, but the unfinished Dag run and dependency
state in the metadata database already place it in planned work. This is where
having a domain-specific hypervisor becomes interesting.
The Hypervisor can keep schedulers active while a Dag run remains unfinished or
the Dag model says another run is due soon, and can keep the triggerer active
while `trigger` rows exist. We'll still use traditional autoscalers like KEDA,
the Kubernetes autoscaler, but the Hypervisor adds Airflow context, can pause
worker scaling during hibernation, and sequences scale-to-zero so one service
does not vanish while another still needs it.
Airflow has already written down the work that is coming, so the
Hypervisor reads that instead of guessing from CPU metrics.
### What the Hypervisor repairs
The Hypervisor can also detect late Dag runs, stale heartbeats, stuck tasks,
slow parsing, large database tables, unhealthy workers, and Astro Scheduler
latency. And because we operate thousands of production Airflow deployments, we
know which failures repeat and exactly what state marks them, so the Hypervisor
repairs those itself:
1. A Celery worker has free slots but has stopped accepting queued work.
2. A worker pod remains stuck in `Terminating` after its tasks are gone.
3. A Kubernetes task pod no longer matches an active task instance or try.
Each repair first checks Airflow and Kubernetes state. If that check fails, the
repair stops instead of acting on partial evidence. Other conditions become
incidents for an operator or another system.
### Hibernation is an ordered stop
The Hypervisor also turns hibernation schedules and manual overrides into one
decision: it scales services to zero in dependency order and later restores
their normal replica sources. Hibernation is not a lossless pause. Shutdown
marks remaining Dag runs and tasks failed or skipped, removes task pods, and
writes heartbeats for services stopped on purpose so health checks do not report
the shutdown as a fault. That trade fits idle non-production deployments; a
deployment with running production tasks keeps its dependencies active.
Across the measured deployment group, triggerer scaling and the repair controls
above cut Dag failures by 85%. The figure counts the failure classes the Hypervisor can
repair; faults in user code, external services, or data remain the Dag author's.
## The software beneath the path
The Astro Runtime image does its work before the Dag run arrives. It pins the
Airflow and Task SDK versions and sets bounds for the providers, Python
packages, database clients, and system code used by each service. Customer
projects can add packages, but they start from a known set.
We test that set across supported Python versions, processor architectures, base
operating systems, databases, and executors. Release candidates pass package,
Docker, CLI, Kubernetes, migration, and security tests. Supported releases also
receive daily security scans and selected backports for key bugs and CVEs for
two years.
## When a region fails
So far, every failure has assumed the metadata database remains available. A
message broker hint can disappear, a scheduler can stop, and a worker can lose
its connection while another process reads the same stored state.
A regional outage can remove the database, Kubernetes cluster, object storage,
workers, and network paths at once.
Astro cross-region recovery prepares a second dedicated cluster in another
region of the same cloud. The primary cluster runs deployments during normal
operation. The secondary remains ready for promotion. Three kinds of state move
between them:
- The Multi-Region DB copies deployment metadata, including Dag-run history,
task-instance state, XComs, connections, variables, and configuration.
- Multi-region object storage copies task logs.
- Image replication makes user-deployed images available in the second region.
### Promoting the secondary cluster
An Organization Owner begins failover through the Astro UI or API. Astro
promotes the secondary cluster and starts the deployments there. Deployment IDs,
settings, and service hostnames remain the same. Airflow UI and API traffic
begins routing to the active cluster in the second region.
Promotion must therefore restore more than a database replica.
Tasks can fail during the switch. After promotion, an operator checks deployment
health and retries work whose state requires it.
For our example, the outcome depends on the last state committed before the
outage:
- If `transform` completed and reported success, the secondary region can use
that state to consider `publish`.
- If `transform` was queued but had not started, the active recovery deployment
can assign it to a worker.
- If `transform` was running when the primary failed, its process is gone. Its
final database state and Airflow retry policy decide what happens next.
- If user code changed an external system before failing, its retry still needs
safe external writes.
### What recovery promises
For deployments using the task-log replication SLA, Astro's recovery point
objective is under 15 minutes and measures how far copied state may lag. The
metadata database is usually seconds old; this matters most, because it holds the real state of the Dags, including what
has and hasn't been run.
Task logs stored in object storage are generally minutes as opposed to seconds.
The recovery time objective is under one hour and measures how long the
secondary cluster and its deployments take to become available. We constantly
run our own internal benchmarks and verified both objectives under load in a
cross-region test with more than 80 deployments and more than 1,250 concurrent
task runs.
Failback is the same kind of switch. Astro reverses replication so the former
primary can catch up; once it has, an Organization Owner moves services and
hostnames back, and tasks can again fail during the change.
Astro manages the paired cluster, replication, promotion, stable service
hostnames, and failback. Customers prepare regional network routes, private
endpoints, identities, image credentials, and Dag settings, then check
deployment health and retry failed work after a switch.
We have now followed one task across the main failure boundaries: a lost hint, a
lost worker, a failed pod, and a lost region. Each layer handles a larger
failure. None changes the basic rule: durable task state belongs in the metadata
database, while user code must remain safe to retry.
## What the full system changes
In our benchmarks, we've pushed Astro to 500,000 sustained concurrent tasks, 228
milliseconds p95 at 100,000, and an 85% cut in Dag failures. No single test
measures the whole architecture, so we tested its layers apart and together.
### Isolating the execution path
We wanted to measure the Astro Scheduler and Astro Executor, not how fast
Kubernetes could add a worker or how soon an autoscaler noticed load. Before
each run, we started more worker slots than the target required. A 50,000-task
run, for example, had more than 60,000 slots ready before the test began. We also provisioned enough API server capacity and raised the replica and
resource limits that Astro exposes in the product, past the ceilings a user can
set today. These runs measure the Astro Scheduler and Astro Executor, not the
current product caps.
That setup does not make the test free of infrastructure. Reaching these counts
found limits elsewhere. Some database queries needed missing indexes before the
control path could progress to the next load level. At high pod counts, the
Kubernetes scheduler used for bin packing ran out of memory. We hit a vCPU
ceiling on our Kubernetes cluster twice. We had to separate those failures from
Astro Scheduler and Astro Executor behavior rather than counting every failed
run as the same limit.
The test holds a large group of long-sleeping tasks at a steady level, then
triggers a small probe Dag. The first measure asks how long the first task in
that new run takes to start. The second asks how long each following task waits
after its upstream task finishes. This shows whether a deployment can begin and
advance new work while hundreds of thousands of other tasks remain active.
### Task-start latency under load
At 50,000 concurrent tasks, the standard scheduler with Celery recorded 23.582
seconds of p95 task-start latency. We stopped the Celery baseline there: task
lag at that level already makes a deployment hard to operate, and pushing
further mainly measures the queue growing. With the Astro Scheduler and Astro
Executor, p95 was 228 milliseconds at 100,000 concurrent tasks—at twice the
load, less than one-hundredth of the latency. At 300,000 concurrent tasks, p95
remained 294 milliseconds.
### Maximum concurrent tasks
We also tested how much resident task state the execution path could manage.
These tests use long-running tasks so concurrency rises and stays high instead
of draining before the next group starts. The result measures how much running
state the path can hold, not how many short tasks it can start each second.
With the standard scheduler, the Astro Executor sustained 200,000
concurrent tasks. Combining it with the Astro Scheduler reached 500,000
sustained concurrent tasks.
| Scheduler | Executor | Sustained concurrent tasks |
| ---------------- | -------------- | -------------------------: |
| Standard Airflow | Astro Executor | 200,000 |
| Astro Scheduler | Astro Executor | 500,000 |
The first row separates Astro Executor capacity from Astro Scheduler capacity. At 200,000
tasks, the standard scheduler becomes the next limit. Replacing both halves of
the path raises the result to 500,000, so this is not an Astro Executor result alone.
Each result depends on the system we tested it on. Worker size, database
capacity, service replicas, Dag shape, task duration, queue count, and settings
all affect where the next limit appears.
## Back to `transform`
Airflow still records its dependencies and state. The Astro Scheduler reacts
when `extract` succeeds, takes ownership of the Dag run, and moves
`transform` to `queued` through a guarded database update. The Astro Executor assigns it to a worker that has room. That worker starts an Airflow Task SDK
process and reports state through the Execution API. The Astro Hypervisor reads
the same deployment state to keep the needed services active. The Astro Runtime
image supplies the tested software beneath them. If the whole region fails, the
Multi-Region DB and secondary cluster preserve the stored record from which
Airflow can recover.
Throughout that path, the Dag and Airflow task states stay the same. Astro
changes how quickly the system sees those states, how it assigns the work, and
how the services around the path scale.
We're really proud of the work we've done and don't plan to slow down with
either the open-source Apache Airflow project nor our commercial products. The
market is incredibly dynamic and we welcome this new class and scale of
workloads that have emerged as LLMs have unlocked new workflow use cases and
made it easier to produce code for the use cases we know and love.
_P.S. We're hiring. If you want to work on the systems in this post,
[we'd love to speak with you](/careers/)._