> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure worker autoscaling on Astro Private Cloud Deployments

> Let each worker queue scale its Celery workers between a minimum and a maximum count with KEDA, based on how many tasks the queue is running and waiting to run.

*Worker autoscaling* lets a [worker queue](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-queues.md) run a range of workers instead of a fixed count. The queue adds workers when tasks back up and removes them when the work drains.

For example, a `heavy-etl` queue set to 1 to 8 workers with a worker concurrency of 5 runs one worker while idle.

A queue's worker count comes from:

```text theme={null}
wanted workers = ceil((running tasks + queued tasks) / worker concurrency)
worker count   = wanted workers, held between Min workers and Max workers
```

For `heavy-etl`, `ceil(40 / 5)` is 8, so the queue runs 8 workers. With 60 tasks, `ceil(60 / 5)` is 12, so it runs its maximum of 8. With 3 tasks, `ceil(3 / 5)` is 1, its minimum. [KEDA](https://keda.sh) asks the platform for this number every 30 seconds by default and sets the queue's worker count to it.

> [!NOTE]
>
> **Astro Private Cloud 2.2**
>
> This feature was introduced in Astro Private Cloud 2.2. To access this feature, upgrade your Astro Private Cloud installation to 2.2 or later.

You install and run KEDA yourself. Astro Private Cloud doesn't install it or check its version or whether it's running. You tell the platform where KEDA runs, and it creates the identity KEDA uses to ask for each queue's worker count.

```mermaid theme={null}
flowchart LR
  keda["KEDA, in your KEDA namespace"]
  laminar["Laminar API server, on the data plane"]
  db["Airflow metadata database"]
  workers["Worker Deployment for each queue"]

  keda -->|asks how many workers each queue needs| laminar
  laminar -->|counts running and queued tasks per queue| db
  keda -->|sets the worker count, between min and max| workers
```

> [!NOTE]
>
> This page covers worker queue autoscaling for Deployments in [Airflow operator
> mode](https://astronomer.io/docs/astro-private-cloud/v-2-x/airflow-operator-mode.md). Astro Private Cloud
> doesn't support KEDA autoscaling for Helm-mode Deployments.

## Prerequisites

* Astro Private Cloud 2.2 or later.
* A Deployment in [Airflow operator mode](https://astronomer.io/docs/astro-private-cloud/v-2-x/airflow-operator-mode.md) that uses the Celery executor, with [worker queues](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-queues.md) turned on. See [Enable worker queues](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-queues.md#enable-worker-queues).
* The Airflow operator that ships with Astro Private Cloud 2.2 (1.7.0) on each data plane. Earlier operators silently drop the scaling configuration. See [Verify the data plane](#verify-the-data-plane).
* Kubernetes permissions to install KEDA on each data plane cluster, and to let the platform's Helm release write into the namespace KEDA runs in.

A queue scales only when all four of the following are true. Each ships off or absent:

| Requirement | Who sets it | Default |
| - | - | - |
| KEDA 2.20.0 or later is installed on the data plane cluster. | You | Not installed |
| `global.keda.enabled` is `true` in the data plane's `values.yaml`. Without it, the platform reports an installed KEDA as missing. | Platform administrator | `false` |
| `global.laminar.enabled` is `true` in the data plane's `values.yaml`. Laminar serves the worker count KEDA scales on. | Platform administrator | `false` |
| `deployments.workerQueues.autoscaling.enabled` is `true` in the APC API configuration for the Deployment. | Platform administrator | `false` |

The Astro Private Cloud UI and the APC API combine the four into one answer for each Deployment, `canWorkerQueuesAutoscale`. See [Confirm autoscaling is available](#confirm-autoscaling-is-available).

## Step 1: Install KEDA

Install KEDA 2.20.0 or later on each data plane cluster that runs Deployments you want to autoscale. Follow the [KEDA deployment guide](https://keda.sh/docs/latest/deploy/). Note the namespace and the operator service account name. Step 2 needs both.

Install KEDA before you set `global.keda.enabled` on the data plane. The APC Helm chart creates a KEDA `ClusterTriggerAuthentication`, and an install or upgrade fails if KEDA's resource definitions aren't on the cluster yet.

### Why 2.20.0 is the minimum

| Requirement | Available from | What it's for |
| - | - | - |
| Time-bound service account tokens as a trigger authentication source | 2.17.0 | How KEDA proves its identity when it asks for a queue's worker count |
| Reliable pause and resume of a scaled object | 2.20.0 | How [hibernation](https://astronomer.io/docs/astro-private-cloud/v-2-x/hibernate-deployments.md) stops a queue's workers and starts them again. Earlier versions can leave a hibernated queue's workers running, or run one worker where the queue should have none. |

Check the version you run:

```bash theme={null}
kubectl get deployment keda-operator -n <keda-namespace> -o jsonpath='{.spec.template.spec.containers[0].image}'
```

> [!WARNING]
>
> Nothing in the platform checks the KEDA version. With an older KEDA, the
> platform install can still succeed, so a clean `helm upgrade` isn't evidence
> that your KEDA is new enough. Below 2.17, KEDA can't present its identity, the
> platform refuses its requests, and every queue stays at its minimum worker
> count. A queue with a minimum of zero runs nothing.

The platform detects only whether KEDA's resource definitions are on the cluster, not whether KEDA is running, so monitor KEDA yourself. See [When KEDA or Laminar is missing or stops](https://astronomer.io/docs/astro-private-cloud/v-2-x/troubleshoot-worker-autoscaling.md#when-keda-or-laminar-is-missing-or-stops).

## Step 2: Configure the data plane

Set the following in the `values.yaml` of each data plane, then upgrade the release. See [Apply a config change](https://astronomer.io/docs/astro-private-cloud/v-2-x/apply-platform-config.md).

```yaml theme={null}
global:
  keda:
    enabled: true
    namespace: keda
  laminar:
    enabled: true
```

| Value | Default | Description |
| - | - | - |
| `global.keda.enabled` | `false` | Creates the identity KEDA uses, lets the platform detect KEDA, and makes the Airflow operator restore a scaled object that someone changes or deletes. With it off, the platform reports KEDA as missing even when it's installed. |
| `global.keda.namespace` | `keda` | The namespace KEDA runs in. Required when `enabled` is `true`. The platform allows traffic from this namespace to the worker count endpoint, so it must match where the KEDA Pods run. |
| `global.keda.operatorServiceAccountName` | `keda-operator` | The service account KEDA runs as. It's the only account allowed to create a token for the scaling identity. |
| `global.keda.clusterObjectNamespace` | Same as `namespace` | Set this only if KEDA resolves cluster-scoped objects in a namespace other than its own. See [If KEDA resolves cluster-scoped objects elsewhere](#if-keda-resolves-cluster-scoped-objects-elsewhere). |
| `global.laminar.enabled` | `false` | Installs Laminar on the data plane. Laminar serves each queue's worker count, and [hibernation](https://astronomer.io/docs/astro-private-cloud/v-2-x/hibernate-deployments.md) also needs it. |

In a unified installation, set them in the single `values.yaml`. On a control plane, they create nothing.

If `global.keda.enabled` is `true` and `global.keda.namespace` is empty, the install fails with `global.keda.namespace is required when global.keda.enabled is true`.

> [!NOTE]
>
> If you upgrade from 2.1 and set `airflow-operator.kedaEnabled`, remove it.
> Astro Private Cloud 2.2 doesn't read it. The Airflow operator now follows
> `global.keda.enabled`.

### What the platform creates in your KEDA namespace

With `global.keda.enabled` set to `true`, a platform install or upgrade creates four objects: three in your KEDA namespace and one cluster-scoped.

| Object | Name | Purpose |
| - | - | - |
| `ServiceAccount` | `metrics-api-worker-trigger` | The identity KEDA presents when it asks for a queue's worker count. It holds no permissions of its own, and nothing mounts it. |
| `Role` | `metrics-api-worker-trigger-token-minter` | Allows `create` on `serviceaccounts/token` for that one account, and for no other account. |
| `RoleBinding` | `metrics-api-worker-trigger-token-minter` | Grants that role to the service account KEDA runs as. A stock KEDA install has no permission to create tokens, so without this binding KEDA's requests carry no token. |
| `ClusterTriggerAuthentication` | `metrics-api-worker-trigger` | Tells KEDA which account to create a token for. It's cluster-scoped, so one object serves every Deployment on the cluster. |

Each object carries the label `astronomer.io/platform-release`, set to your platform release name.

The platform creates these objects only on install or upgrade, not when you create a Deployment. Helm owns all four: upgrading updates them in place, and uninstalling removes them.

Setting `global.keda.enabled` back to `false` and upgrading also removes them. Do that only when no Deployment uses worker autoscaling, because without the identity KEDA can't get a worker count and every autoscaling queue drops to its minimum.

### Network policies

When `global.networkPolicy.enabled` is `true`, the platform allows traffic from the namespace named in `global.keda.namespace` to the Laminar API server on port 8000. If you restrict egress from your KEDA namespace yourself, also allow traffic from it to the platform namespace on port 8000. The platform doesn't write network policies into a namespace it doesn't manage.

### If KEDA resolves cluster-scoped objects elsewhere

KEDA looks for the service account named on a `ClusterTriggerAuthentication` in the namespace set by its `KEDA_CLUSTER_OBJECT_NAMESPACE` environment variable, which defaults to the namespace KEDA runs in. If you set that variable, set `clusterObjectNamespace` to the same namespace:

```yaml theme={null}
global:
  keda:
    enabled: true
    namespace: keda-system
    clusterObjectNamespace: keda-cluster
```

The service account, role, and role binding are then created in `keda-cluster`, and the role binding still names the KEDA service account in `keda-system`. Laminar trusts the account in `keda-cluster` as part of the same change.

### Verify the data plane

1. Confirm the three namespaced objects exist:

   ```bash theme={null}
   kubectl get serviceaccount,role,rolebinding -n <keda-namespace> -l astronomer.io/platform-release=<your-platform-release-name>
   ```

   If you set `clusterObjectNamespace`, look in that namespace instead.

2. Confirm the cluster-scoped object exists:

   ```bash theme={null}
   kubectl get clustertriggerauthentication metrics-api-worker-trigger
   ```

3. Confirm KEDA's service account can create a token for the identity:

   ```bash theme={null}
   kubectl create token metrics-api-worker-trigger -n <keda-namespace> --as=system:serviceaccount:<keda-namespace>:keda-operator
   ```

   If you set `clusterObjectNamespace`, pass it to `-n` and keep the namespace KEDA runs in for `--as`. If you set `operatorServiceAccountName`, use that name in place of `keda-operator`.

   > [!WARNING]
   >
   > `kubectl auth can-i create serviceaccounts/token` reports `no` even when
   > this works, because the role is scoped to one named account. Use `kubectl create token` instead.

4. Confirm both Airflow resource definitions carry the scaling trigger field. Each command must print `array`:

   ```bash theme={null}
   kubectl get crd airflows.airflow.apache.org -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.workers.items.properties.keda.properties.triggers.type}'
   ```

   ```bash theme={null}
   kubectl get crd workers.airflow.apache.org -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.keda.properties.triggers.type}'
   ```

   > [!WARNING]
   >
   > If either command prints nothing, the data plane runs an Airflow operator
   > older than the one Astro Private Cloud 2.2 ships. Kubernetes then drops the
   > scaling trigger without an error: saves succeed, but KEDA never asks the
   > platform for a worker count, and nothing in the platform reports it.
   > Upgrade the data plane's platform release, which upgrades the operator and
   > its resource definitions together. Don't patch the resource definitions by
   > hand: the next `helm upgrade` reverts the patch.

## Step 3: Turn on worker autoscaling

`deployments.workerQueues.autoscaling.enabled` controls worker autoscaling and ships `false`. Like other `deployments` settings, you can set it for the platform, a cluster, a Workspace, or a single Deployment. See [Config governance](https://astronomer.io/docs/astro-private-cloud/v-2-x/config-governance.md).

To set it for a cluster, go to your **Clusters** page, select your cluster, click **Edit** in the **Deployment Configuration** section, and add the following to the **Configuration Override** field:

```yaml wrap theme={null}
workerQueues:
  enabled: true
  autoscaling:
    enabled: true
```

For details on applying the override, see [Override base configuration](https://astronomer.io/docs/astro-private-cloud/v-2-x/override-data-plane-cluster.md#override-base-configuration). To set it for the whole platform instead, add the same `deployments` block under `astronomer.houston.config` in the control plane's `values.yaml`. See [Apply a config change](https://astronomer.io/docs/astro-private-cloud/v-2-x/apply-platform-config.md).

To set it for one Deployment, use a Deployment-level override:

```graphql wrap theme={null}
mutation enableWorkerAutoscaling {
  updateDeploymentConfig(
    deploymentUuid: "<deployment-uuid>"
    deploymentsConfigOverride: {
      workerQueues: { enabled: true, autoscaling: { enabled: true } }
    }
    reason: "Enable worker autoscaling"
  ) {
    config
    updatedAt
  }
}
```

> [!WARNING]
>
> `deploymentsConfigOverride` replaces the Deployment's entire stored override.
> A key you leave out stops overriding and inherits again, so read the current
> override first and send everything you want to keep. See [Update Deployment
> configuration](https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-update-deployment-config.md).

Turning the setting on doesn't change any running Deployment. The UI shows worker ranges on the next page load, and a queue scales once you give it a range and update the Deployment.

### Change how often KEDA asks for a worker count

`deployments.workerQueues.autoscaling.pollingIntervalSeconds` sets how often KEDA asks for each queue's worker count. It defaults to `30` and takes a whole number of seconds, 1 or more. You can set it at any of the four tiers. A shorter interval reacts faster and sends more requests to the data plane.

The new interval reaches each Deployment's scaling objects the next time the Deployment is updated.

### Confirm autoscaling is available

> [tab] Astro Private Cloud UI
>
> Open the Deployment, click **Settings**, and expand **Worker Queues**. When autoscaling is available, the queue editor shows **Min workers** and **Max workers**. See [Set a worker range on a queue](#set-a-worker-range-on-a-queue).
>
> When autoscaling isn't available, the editor shows a single **Num workers** field instead:
>
> ![The Edit worker queue form for the default queue on a Deployment where autoscaling isn't available, with a single Num workers field set to 1 and the hint that a min and a max replace this field where autoscaling is available.](https://mintcdn.com/astronomer/NVMeGUwVk6PcG6HU/images/astro-private-cloud/worker_autoscaling_disabled_num_worker.png?fit=max\&auto=format\&n=NVMeGUwVk6PcG6HU\&q=85\&s=156775cb5e69b5add295d2a65498027b)
>
> The warnings at the top of the **Settings** page explain why:
>
> ![The Settings tab of a Deployment with two information alerts, Autoscaling is unavailable and Hibernation is unavailable, each saying that the Deployment's data plane doesn't have both KEDA and Laminar.](https://mintcdn.com/astronomer/NVMeGUwVk6PcG6HU/images/astro-private-cloud/worker_autoscaling_autoscaling_unavailable.png?fit=max\&auto=format\&n=NVMeGUwVk6PcG6HU\&q=85\&s=636e4dd66608cb8d1093516f5f65e461)
>
> See [When KEDA or Laminar is missing or stops](https://astronomer.io/docs/astro-private-cloud/v-2-x/troubleshoot-worker-autoscaling.md#when-keda-or-laminar-is-missing-or-stops).

> [tab] APC API
>
> `canWorkerQueuesAutoscale` combines all four requirements for a Deployment:
>
> ```graphql wrap theme={null}
> query deploymentConfig($deploymentUuid: Uuid) {
>   deploymentConfig(deploymentUuid: $deploymentUuid) {
>     workerQueues {
>       canWorkerQueuesAutoscale
>       replicas {
>         min
>         max
>       }
>     }
>   }
> }
> ```
>
> To check only whether an administrator turned the feature on, read the flag on its own, and always pass the cluster ID: without it, `appConfig` returns the platform default, which can read `false` while the cluster has autoscaling on:
>
> ```graphql wrap theme={null}
> query appConfig($clusterId: Uuid) {
>   appConfig(clusterId: $clusterId) {
>     featureFlags {
>       workerAutoscalingEnabled
>     }
>   }
> }
> ```
>
> If `workerAutoscalingEnabled` is `true` and `canWorkerQueuesAutoscale` is `false`, the data plane is missing KEDA, `global.keda.enabled`, or Laminar.

> [tab] Astro CLI
>
> The Astro CLI doesn't report worker autoscaling for Astro Private Cloud. Use the Astro Private Cloud UI or the APC API.

## Set a worker range on a queue

> [tab] Astro Private Cloud UI
>
> 1. In the Astro Private Cloud UI, open your Deployment and click **Settings**.
> 2. Expand **Worker Queues**.
> 3. Click **Edit** on the queue, or click **Add worker queue**. For the queue's other fields, see [Add a worker queue](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-queues.md#add-a-worker-queue).
> 4. In **Min workers**, enter the fewest workers the queue runs. Enter `0` to let an idle queue scale to zero. See [Scale a queue to zero](#scale-a-queue-to-zero).
> 5. In **Max workers**, enter the most workers the queue runs. It must be at least the minimum. Enter the same value as the minimum to keep the queue at one size.
> 6. (Optional) In **Worker concurrency**, enter how many tasks each worker runs at once. See [Worker concurrency sets the scaling step](#worker-concurrency-sets-the-scaling-step).
> 7. Click **Save**.
> 8. Click **Deploy Changes**.
>
> ![The New worker queue form for a heavy-etl queue, with Min workers set to 1, Max workers set to 8, and worker concurrency left at the inherited default.](https://mintcdn.com/astronomer/NVMeGUwVk6PcG6HU/images/astro-private-cloud/new_worker_queue_autoscaling.png?fit=max\&auto=format\&n=NVMeGUwVk6PcG6HU\&q=85\&s=b7413eebb916de93c66b3f0961e3f8a7)

> [tab] APC API
>
> Set `minWorkerCount` and `maxWorkerCount` on the queue in the `workerQueues` argument of `upsertDeployment`. This example keeps a fixed-size `default` queue and lets `heavy-etl` scale from 1 to 8 workers:
>
> ```json expandable wrap theme={null}
> {
>   "deploymentUuid": "<deployment-uuid>",
>   "workerQueues": [
>     {
>       "name": "default",
>       "cpu": 1000,
>       "memory": 3840,
>       "ephemeralStorage": 51200,
>       "replicas": 1
>     },
>     {
>       "name": "heavy-etl",
>       "cpu": 2000,
>       "memory": 7680,
>       "ephemeralStorage": 51200,
>       "minWorkerCount": 1,
>       "maxWorkerCount": 8,
>       "workerConcurrency": 5
>     }
>   ]
> }
> ```
>
> For the mutation, see [Manage worker queues with the APC API](https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-manage-worker-queues.md).
>
> > [!WARNING]
> >
> > `workerQueues` is a full-list replace: a field you leave out goes back to its
> > default. If you leave out `minWorkerCount`
> > and `maxWorkerCount` on a queue that had a range, both fall back to
> > `replicas`, and the queue stops scaling. Send both bounds on every write that
> > should keep the range.
>
> How the three counts interact:
>
> * Send `minWorkerCount` and `maxWorkerCount` together. A bound you leave out falls back to `replicas`, not to the other bound, so sending only a minimum of 5 on a queue with `replicas` of 1 asks for a maximum of 1 and is rejected.
> * `replicas` is the size the queue runs at when it isn't autoscaling. Sent with a range, it must be inside the range. Left out with both bounds, it follows the minimum, with a floor of 1: a range of 0 to 8 stores a `replicas` of 1.
> * A queue that isn't autoscaling runs the larger of `replicas` and `minWorkerCount`. For example, a queue with a `replicas` of 2 and a minimum of 4 runs 4 workers.
> * A queue whose minimum equals its maximum doesn't autoscale. It runs that fixed count.

> [tab] Astro CLI
>
> The Astro CLI has no worker queue or worker autoscaling commands for Astro Private Cloud. `astro deployment create` and `astro deployment update` don't send worker queues, so they leave a Deployment's queues and ranges unchanged. The `astro deployment worker-queue` commands apply only to Astro.

### Limits on a range

| Bound | Lowest | Highest | Configured by |
| - | - | - | - |
| `minWorkerCount` | `0`, only where the cluster can autoscale | The per-queue worker limit | `deployments.resourceManagement.components[workers].extra.replicas.limit`, default `10` |
| `maxWorkerCount` | `minWorkerCount` | The per-queue worker limit | Same |

The per-queue worker limit applies to a fixed count and to both bounds of a range, so raising it for autoscaling also raises the cap on fixed-size queues. You can override it per cluster, Workspace, and Deployment. The UI doesn't check this limit on **Max workers**, so a maximum above it is refused when you save.

## Scale a queue to zero

A queue with a minimum of `0` stops all of its workers when it has no running or queued tasks. When a task arrives, KEDA starts a worker, so the first task waits for it to start.

Scale to zero needs the following:

* KEDA and Laminar on the data plane. Without them, nothing could start the first worker again, so a minimum of `0` is refused with `WorkerQueueZeroMinimumUnsupportedError`.
* To save a minimum of `0` from the Astro Private Cloud UI, a platform administrator must lower `deployments.resourceManagement.components[workers].extra.replicas.minimum` to `0`. The UI sends a worker count of `0` with a zero minimum, and the default minimum of `1` refuses it. Through the APC API, send both bounds and leave out `replicas`, which then defaults to `1`.

> [!WARNING]
>
> A queue at zero workers depends on KEDA and Laminar to start its first worker.
> If Laminar stops responding, the queue falls to its minimum of zero. If KEDA stops while the queue is at zero, no worker starts
> until KEDA returns. Use a minimum of 1 or more where a delay
> matters.

## How far a queue scales

### Worker concurrency sets the scaling step

A queue asks for its running and queued tasks divided by its worker concurrency. The same 48 tasks ask for 3 workers at a concurrency of 16 and 12 workers at a concurrency of 4. A queue with no concurrency set uses 16. A queue's concurrency can be at most 15 for each CPU core of that queue's workers, rounded down and never below 1. For example, workers with 1000 millicores (one core) allow up to 15, 500 millicores allow up to 7, and 2000 millicores allow up to 30.

#### If you set the worker concurrency environment variable

On a Deployment in Airflow operator mode, the `AIRFLOW__CELERY__WORKER_CONCURRENCY` environment variable doesn't change worker concurrency. The Deployment still shows the variable, but every worker has run 16 tasks at once, whatever value it shows. Set concurrency on the worker queue instead.

On a Helm-mode Deployment, the variable takes effect. Leave it in place and skip these steps.

Setting the queue to the value in your variable changes how your Deployment runs. If the variable says 4 and your workers have run 16, a queue of 3 workers goes from 48 tasks at once (3 × 16) to 12 (3 × 4), and tasks can start to queue where they didn't before. If you're unsure which value you want, 16 is what your Deployment has run successfully.

1. Decide on a value. Compare the variable with 16, which is what your workers run now.
2. In the Astro Private Cloud UI, open your Deployment, click **Settings**, and expand **Worker Queues**.
3. Click **Edit** on each queue that should run a different number of tasks per worker. If you have one queue, it's named `default`.
4. Set **Worker concurrency** to the value you chose, click **Save**, and click **Deploy Changes**.
5. After the Deployment finishes updating, confirm that each worker runs the number of tasks you set.
6. Remove `AIRFLOW__CELERY__WORKER_CONCURRENCY` from the Deployment's environment variables.

Remove the variable last. On its own, removing it changes nothing on an operator-mode Deployment, and removing it first loses the record of the value you meant to set.

### Parallelism

`AIRFLOW__CORE__PARALLELISM` limits how many tasks each scheduler runs and queues at once. Astro Private Cloud derives it from your queues:

```text theme={null}
parallelism = ceil(sum of (worker concurrency × worker ceiling) for each queue / schedulers)
```

A queue with no concurrency set counts as 16. The worker ceiling is the queue's maximum when it autoscales, and its fixed worker count when it doesn't.

For example, a Deployment with one scheduler, a fixed `default` queue of 2 workers with no concurrency set, and `heavy-etl` at 1 to 8 workers with a concurrency of 5:

```text theme={null}
default      2 workers × 16 = 32
heavy-etl    8 workers × 5  = 40
parallelism  ceil(72 / 1)   = 72
```

With two schedulers, parallelism is `ceil(72 / 2)`, which is 36 for each scheduler.

If you set `AIRFLOW__CORE__PARALLELISM` yourself, your value replaces the derived one. A value lower than the derived one can hold a queue below its maximum, because tasks past the limit wait as scheduled tasks and the queue doesn't count them. For example, with parallelism set to 32 and one scheduler, `heavy-etl` has at most 32 running or queued tasks, so it asks for at most `ceil(32 / 5)`, which is 7 workers, one short of its maximum. Nothing in the platform reports this. If a queue tops out below its maximum, check this variable first.

### Cluster capacity

A worker that doesn't fit on any node stays `Pending`. KEDA still asks for the full count, and the queue runs the workers that fit. If the queue uses a [node pool](https://astronomer.io/docs/astro-private-cloud/v-2-x/node-pools.md), the pool's total counts each queue at its maximum worker count, and the editor warns when a change takes a pool over its total. For example, `heavy-etl` at 1 to 8 workers with 2000 millicores and 7680 MiB each counts as 16 CPU (8 × 2) and 60 GiB (8 × 7.5 GiB) against its pool, even while it runs one worker.

The Deployment's namespace resource quota counts each autoscaling queue at its maximum, so `heavy-etl` counts as 8 workers even while it runs one.

## Scale-down and restarts

### Scale-down

When a queue needs fewer workers, the Kubernetes Horizontal Pod Autoscaler waits for its scale-down stabilization window, 300 seconds by default, before it removes workers. The last step from one worker to zero waits for KEDA's cooldown period, also 300 seconds, counted from the last time the queue had running or queued tasks.

For example, if the last of `heavy-etl`'s tasks finishes at 10:00 while it runs 8 workers, the queue drops to its minimum of 1 at about 10:05. A queue with a minimum of `0` stops its last worker at about 10:05 too, when the cooldown ends.

A removed worker has the worker termination grace period, 600 seconds by default, to finish its running tasks, and a task still running at the end stops. For example, a worker removed while it runs a task with 15 minutes left stops that task after 10 minutes. A platform administrator sets the grace period with `deployments.resourceManagement.components[workers].extra.terminationGracePeriodSeconds`.

### What restarts when you change a queue

| Change | What restarts |
| - | - |
| A queue's maximum, or its worker concurrency | Every Airflow component and every queue's workers, because parallelism changes |
| Only a queue's minimum | Nothing restarts. Parallelism counts an autoscaling queue at its maximum |
| A queue's CPU, memory, ephemeral storage, or node pool | That queue's workers |
| Turning worker autoscaling on or off, on a Deployment with a ranged queue | Every Airflow component, at that Deployment's next update, because parallelism changes |
| `pollingIntervalSeconds` | Nothing restarts |

Lowering a queue's maximum below the number of workers it runs stops the extra workers. The Astro Private Cloud UI doesn't warn you before it does.

## Monitor autoscaling

### The queue card

Each queue card in the **Worker Queues** section shows the queue's range, for example `1 to 8 workers`, and a **Current workers** bar. The bar shows running workers against the queue's maximum, with terminating workers marked separately. It doesn't show workers that are waiting for room on the cluster. For the CPU and memory readings on the same card, see [Monitor worker queues](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-queues.md#monitor-worker-queues).

![The Worker Queues section with two autoscaling queues, default at 1 to 5 workers and analytics at 1 to 3 workers, each with a Scaling history link. The default queue's Current workers bar shows one running worker and a hatched segment for one terminating worker, with the tooltip 1 running, 1 terminating, 5 max.](https://mintcdn.com/astronomer/NVMeGUwVk6PcG6HU/images/astro-private-cloud/worker_queue_terminating.png?fit=max\&auto=format\&n=NVMeGUwVk6PcG6HU\&q=85\&s=0f9c313970b210f6a7a6428c6a9dd206)

### The scaling history dialog

Click **Scaling history** on a queue card to see how the queue scaled. The link appears on a saved queue with a range, when the Deployment's data plane can autoscale. Anyone who can see the Deployment can open it.

![The Scaling history dialog for a default queue with a range of 1 to 5 workers over the last 15 minutes, showing the Peak wanted, At maximum, and Waiting for capacity tiles, a chart of the Workers, Running, and Wanted lines rising from 1 to 5 against the min and max lines, and the finding Ran at its maximum of 5 workers for 2 min.](https://mintcdn.com/astronomer/NVMeGUwVk6PcG6HU/images/astro-private-cloud/worker_queue_scaling_history.png?fit=max\&auto=format\&n=NVMeGUwVk6PcG6HU\&q=85\&s=3c15488c27bc2466562a9743c48dbefc)

Select a time range from **Last 15 Minutes** to **Last 14 days**. The default is **Last 1 day**. The chart shows three lines against the queue's minimum and maximum:

* **Wanted**: the workers the queue's running and queued tasks call for. Tasks held back by parallelism aren't counted.
* **Workers**: the count the autoscaler set.
* **Running**: the workers actually running.

Three tiles summarize the period: **Peak wanted**, **At maximum**, and **Waiting for capacity**. Under the chart, the dialog states what held the queue back, for example `Ran at its maximum of 8 workers for 2 h.` or `Workers waited for room on the cluster for 40 min.` When nothing applies, it shows `Stayed within its range over this period.` Time the Deployment spent hibernated isn't shown.

A state becomes a finding after it adds up to 15 minutes in a day, scaled to the selected range (15 minutes × range ÷ 1 day):

| Range | A finding appears after |
| - | - |
| **Last 1 hour** | About 40 seconds (15 × 60 ÷ 1,440 minutes) |
| **Last 1 day** | 15 minutes |
| **Last 7 days** | 1 hour 45 minutes (15 × 7) |
| **Last 14 days** | 3 hours 30 minutes (15 × 14) |

The dialog also reports a queue that wanted fewer workers than its minimum for at least 90% of the time it was running, for example `Needed fewer than its minimum of 2 workers for 95% of the time it was running.` That finding needs at least 2 hours of running time on a one-day range, scaled the same way.

### From the cluster

A platform administrator can read a queue's scaling objects directly:

```bash theme={null}
kubectl get scaledobject,hpa -n <deployment-namespace>
```

## Reference

### Configuration

| Key | Default | Description |
| - | - | - |
| `deployments.workerQueues.autoscaling.enabled` | `false` | Turns worker autoscaling on. Settable per platform, cluster, Workspace, and Deployment. |
| `deployments.workerQueues.autoscaling.pollingIntervalSeconds` | `30` | How often KEDA asks for each queue's worker count, in whole seconds. |
| `deployments.resourceManagement.components[workers].extra.replicas.limit` | `10` | The most workers one queue can run, for a fixed count and for both bounds of a range. |
| `deployments.resourceManagement.components[workers].extra.replicas.minimum` | `1` | The smallest fixed worker count. Lower it to `0` to save a zero minimum from the Astro Private Cloud UI. |
| `deployments.resourceManagement.components[workers].extra.terminationGracePeriodSeconds` | `600` | How long a removed worker has to finish its running tasks. |

For the data plane Helm values, see [Step 2: Configure the data plane](#step-2-configure-the-data-plane).

## Related documentation

* [Troubleshoot, upgrade, and roll back worker autoscaling](https://astronomer.io/docs/astro-private-cloud/v-2-x/troubleshoot-worker-autoscaling.md)
* [Configure worker queues](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-queues.md)
* [Manage worker queues with the APC API](https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-manage-worker-queues.md)
* [Configure node pools](https://astronomer.io/docs/astro-private-cloud/v-2-x/node-pools.md)
* [Hibernate Deployments](https://astronomer.io/docs/astro-private-cloud/v-2-x/hibernate-deployments.md)
* [Airflow operator mode](https://astronomer.io/docs/astro-private-cloud/v-2-x/airflow-operator-mode.md)
* [Config governance](https://astronomer.io/docs/astro-private-cloud/v-2-x/config-governance.md)
* [Override data plane cluster configurations](https://astronomer.io/docs/astro-private-cloud/v-2-x/override-data-plane-cluster.md)
* [Upgrade to a minor version](https://astronomer.io/docs/astro-private-cloud/v-2-x/upgrade-apc-minor-version.md)