Configure API server autoscaling

The Airflow API server serves the Airflow UI, the Airflow REST API, and the task execution API that Astro and Celery workers use to fetch and report on tasks. By default, every Airflow 3 Deployment runs with two API server replicas. For workloads with high task concurrency, large Dags, or spikes in UI and REST API traffic, you can enable horizontal autoscaling so that Astro adds API server replicas when load is high and removes them when load decreases.

Use API server autoscaling to:

• Sustain workloads with thousands of concurrent tasks without saturating a fixed pair of API server replicas.
• Support smooth Airflow 2 to Airflow 3 upgrades for environments that pushed the previous webserver to its limits.
• Cap your spend by setting an explicit maximum replica count.

How autoscaling works

When you enable autoscaling, Astro provisions a Kubernetes Horizontal Pod Autoscaler (HPA) that tracks CPU utilization across the running API server replicas. The HPA adds or removes replicas to keep the average CPU utilization at 80 percent of the per-replica CPU request.

• When average CPU utilization is consistently above the 80 percent target, Astro adds replicas, up to your configured API Server Max Replicas value.
• When average CPU utilization is below the target, Astro removes replicas, down to the minimum of two.
• The minimum replica count is fixed at two. This preserves API availability during a Pod restart and is the same as the default replica count when autoscaling is disabled.
• The maximum replica count is configurable from 2 to 10 in the Astro UI. Setting API Server Max Replicas to 2 runs a fixed two-replica configuration and effectively prevents scaling.

The HPA evaluates utilization continuously, and replica changes typically take effect within one to two minutes. The same high-availability and anti-affinity rules that apply to other Deployment components apply to API server replicas. See Enable high availability.

Expected replica counts

The following table shows the approximate API server replica counts you can expect at different levels of task concurrency. Actual counts can vary based on Airflow UI and REST API traffic, Dag complexity, and other workload characteristics.

Prerequisites

API server autoscaling is supported on Airflow 3 Deployments running on Astro:

• Standard or dedicated clusters.
• Remote Execution Deployments.

API server autoscaling works with all executors: Astro, Celery, and Kubernetes.

API server autoscaling is not available for Airflow 2 Deployments. Airflow 2 uses the Airflow webserver, which doesn’t support horizontal autoscaling on Astro.

Enable or disable API server autoscaling

After you enable autoscaling, the Deployment Details page shows the configured range under API Server Autoscaling, for example Enabled (2-10 replicas).

Billing

API server replicas are sized the same as an A5 worker. Each Deployment includes two API server replicas at no additional cost. When autoscaling adds replicas above the included two, Astro charges for the additional replicas based on their uptime duration at the A5 unit rate, similar to KE/KPO chargeback. The charges appear on your invoice under Runtime Compute as API Server.

If autoscaling is enabled but the Deployment never exceeds two replicas, no additional API server charges apply.

Limitations

• The minimum replica count is fixed at two. You can’t scale the API server below two replicas.
• The maximum replica count is capped at ten in the Astro UI. If your workload requires more than ten replicas, Astronomer recommends moving some workloads to another Deployment.