Overprovision Airflow components

Overprovisioning in Astro Private Cloud (APC) sets resource requests for selected Airflow components to a fraction of their limits. You set it per cluster and can override it per Workspace or Deployment. This allows the scheduler to pack more pods on nodes (bin-packing) and each pod can still burst up to its limit.

How overprovisioning works

You configure three values in your cluster’s deployment config:

overProvisioningFactorCPU: Fraction of the CPU limit used as the CPU request (0 < value ≤ 1).
overProvisioningFactorMem: Fraction of the memory limit used as the memory request (0 < value ≤ 1).
overProvisioningComponents: List of component names that receive the overprovisioning factor.

For each component in the list, requests are set as:

requests.cpu = limits.cpu × overProvisioningFactorCPU
requests.memory = limits.memory × overProvisioningFactorMem

For example, if a component has limits: { cpu: "2000m", memory: "4Gi" } and both factors are 0.5, then requests become { cpu: "1000m", memory: "2Gi" }. Default factors of 1 mean no change (requests equal limits).

Configure overprovisioning

Set these keys in your cluster config (for example, in the cluster’s deployment config or the values used when registering the cluster). Values must be greater than 0 and less than or equal to 1.

overProvisioningFactorCPU: 0.5
overProvisioningFactorMem: 0.5
overProvisioningComponents:
  - scheduler
  - webserver
  - apiServer
  - workers
  - triggerer
  - flower
  - pgbouncer
  - statsd
  - dagProcessor

Only components that have resources defined in the deployment config are affected. Components not listed in overProvisioningComponents keep their existing request/limit values.

Supported components

You can include any of these in overProvisioningComponents:

Component	Description
`scheduler`	Airflow scheduler
`apiServer`	Airflow API server
`webserver`	Airflow webserver
`workers`	Celery workers
`triggerer`	Airflow triggerer
`flower`	Flower (Celery UI)
`pgbouncer`	PgBouncer
`statsd`	StatsD exporter
`dagProcessor`	Dag processor

Worker queues

Workers in a worker queue always request their full limits, even when workers is in overProvisioningComponents. When you turn on worker queues, the first update that gives an existing Deployment a queue raises its workers’ requests to their limits. See Upgrade to Astro Private Cloud 2.2.

Choose factor values

1 (default): No overprovisioning; requests equal limits. Use when you want predictable capacity and no bin-packing.
0.5: Requests are half of limits. Common choice for better bin-packing while keeping headroom.
Lower values (for example, 0.25): More aggressive bin-packing; ensure your workloads can tolerate less guaranteed CPU/memory.

Start with 0.5 for CPU and memory and adjust based on utilization and scheduling behavior.

Adopted Deployments

Adopting an Astro Runtime Operator Deployment keeps the resource requests from its custom resource. The factors take effect the first time you update the Deployment’s settings in APC, not when you push code. They apply only to the webserver or API server, because that’s the only component whose resources APC manages on an adopted Deployment. For example, with overProvisioningFactorMem: 0.5, a webserver with a 1920Mi memory limit keeps its 1920Mi request after adoption. After the first settings update, it requests 960Mi and restarts. To keep the requests from your custom resource, set resourceManagement.overProvisioningFactorCPU and resourceManagement.overProvisioningFactorMem to 1 in a Deployment-level configuration override before the first settings update. That update stores the reduced requests, so setting the factors to 1 afterward doesn’t restore them.

Best practices

Set factors per cluster and override where needed: The factors in the cluster’s deployment config apply to every Deployment on the cluster. A Workspace or Deployment configuration override can set different factors.
Include only components that have resources: Only components with resources.limits (and optionally resources.requests) in the deployment config receive modifications; listing others has no effect.
Monitor utilization: After enabling, watch pod scheduling and resource usage to confirm the factors match your workload and node capacity.

⌘I