> ## Documentation Index > Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt > Use this file to discover all available pages before exploring further. # Configure worker queues on Astro Private Cloud Deployments > Split a Deployment's Celery workers into named queues with independent sizing and placement, then route tasks to them with the Airflow queue argument. A *worker queue* is a named pool of Celery workers with its own CPU, memory, ephemeral storage, worker count, and task concurrency. A Deployment can run several queues at once, and Dag authors route individual tasks to a queue with the Airflow `queue` argument. Without worker queues, every task runs on one pool of identically sized workers, sized for the heaviest task. With queues, you can keep a small default pool and add a larger queue for memory-hungry work, a GPU-backed queue, or a queue pinned to a node group. Each queue becomes its own Kubernetes Deployment that consumes one named Celery queue. Queues survive code deploys, environment variable changes, and platform upgrades. A queue runs a fixed number of workers. With [worker autoscaling](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-autoscaling.md), a queue can instead run between a minimum and a maximum number of workers, based on how many of its tasks are running and queued. > [!NOTE] > > **Astro Private Cloud 2.2** > > This feature was introduced in Astro Private Cloud 2.2. To access this feature, upgrade your Astro Private Cloud installation to 2.2 or later. ## Prerequisites A Deployment can use worker queues only when all of the following are true: * Astro Private Cloud is on version 2.2.0 or later. Worker queues aren't available on earlier versions. See [Upgrade to a minor version](https://astronomer.io/docs/astro-private-cloud/v-2-x/upgrade-apc-minor-version.md). * The Deployment runs in [Airflow operator mode](https://astronomer.io/docs/astro-private-cloud/v-2-x/airflow-operator-mode.md). * The Deployment uses the Celery executor. To add a queue, a platform administrator must also have set `deployments.workerQueues.enabled` to `true` for the Deployment. See [Enable worker queues](#enable-worker-queues). The Astro CLI has no commands for managing worker queues or node pools. Use the Astro UI or the APC API. If the Deployment isn't in operator mode or doesn't use the Celery executor, the Worker Queues section doesn't appear in Deployment settings, and the `workerQueues` argument on `upsertDeployment` returns a typed error. ## Enable worker queues Worker queues ship turned off, so upgrading changes nothing until a platform administrator turns them on. Set `deployments.workerQueues.enabled` to `true` at the platform, cluster, Workspace, or Deployment level. Setting it on a single Deployment is the safest way to pilot the feature. To set it for a cluster, go to your **Clusters** page, select your cluster, click **Edit** in the **Deployment Configuration** section, and add the following override to the **Configuration Override** field: ```yaml wrap theme={null} workerQueues: enabled: true ``` For details on applying this override, see [Override base configuration](https://astronomer.io/docs/astro-private-cloud/v-2-x/override-data-plane-cluster.md#override-base-configuration). To set it for the whole platform instead, see [Apply a config change](https://astronomer.io/docs/astro-private-cloud/v-2-x/apply-platform-config.md). To set it for a single Deployment with the API, see [Enable worker queues for a Deployment](https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-manage-worker-queues.md#enable-worker-queues-for-a-deployment). Turning the flag on doesn't rebuild any Deployment. The Worker Queues section appears on the next page load, and the Airflow resource changes at the next Deployment update. ### Turn worker queues off The flag gates creating queues, not keeping them. Turning it off on a Deployment that already has queues leaves every queue running, visible, and editable. You can still resize and delete queues, but you can't add a queue or rename one, because a rename creates a queue under a new name. Adding a queue while the flag is off returns `WorkerQueueCreationDisabledError`, naming the queues it refused. ## The default queue An operator-mode Celery Deployment that uses worker queues has exactly one queue named `default`. Tasks that don't name a queue run on it. The default queue: * Can't be deleted or renamed. * Can be resized, given a different concurrency, and assigned to a node pool like any other queue. * Is created for you. A Deployment you create with the flag on gets one sized from its resolved worker configuration. ![The Worker Queues section on a Deployment that has never had queues, showing only the default queue badged as default with no delete control.](https://mintcdn.com/astronomer/NVMeGUwVk6PcG6HU/images/astro-private-cloud/worker_queues_default_only.png?fit=max\&auto=format\&n=NVMeGUwVk6PcG6HU\&q=85\&s=bdefd890023747fc6d187536f5e85ccf) On a Deployment that existed before worker queues were available, a platform administrator runs a one-time backfill after turning the flag on. The upgrade hook doesn't run it, so run it yourself from a Houston Pod: ```sh wrap theme={null} kubectl -n exec -it deploy/-houston -- yarn backfill-default-worker-queues ``` Add `--dry-run` to log which deployments it would touch without writing anything, or scope it with `--clusterId=` or `--deploymentUuid=`. The backfill writes a `default` queue that reproduces the Deployment's existing worker configuration, so worker sizing doesn't change. If you skip the backfill, the first update that sends `workers` or `workerQueues` creates the missing `default` queue from the same resolved configuration. Until then, the Worker Queues section can appear empty. ## Add a worker queue > [tab] Astro UI > > 1. In the Astro Private Cloud UI, open your Deployment and click **Settings**. > 2. Expand **Worker Queues**. > 3. Click **Add worker queue**. > 4. In **Queue name**, enter a name. Names must start with a lowercase letter and contain only lowercase letters, numbers, and hyphens, and they can't end in a hyphen. The name is the value your Dag code passes to `queue`. > 5. (Optional) In **Node pool**, select the node group these workers should run on. Leave it as **No pool (schedule anywhere)** to let Kubernetes place the workers on any eligible node. See [Configure node pools](https://astronomer.io/docs/astro-private-cloud/v-2-x/node-pools.md). > 6. (Optional) In **Worker concurrency**, enter the number of tasks each worker runs at once. Leave it empty to inherit the Deployment default. > 7. In **Num workers**, enter how many workers this queue runs. When worker autoscaling is available, the form shows **Min workers** and **Max workers** instead. See [Set a worker range on a queue](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-autoscaling.md#set-a-worker-range-on-a-queue). > 8. Set **CPU per worker**, **Memory per worker**, and **Ephemeral storage per worker**. > 9. Click **Save**. > 10. Click **Deploy Changes** to apply the change. > > Until you click **Deploy Changes**, a new queue exists only in the form. > > ![The Add worker queue form filled in for a new heavy-etl queue, with node pool, worker concurrency, num workers, and resource fields set, shown above the existing default queue.](https://mintcdn.com/astronomer/NVMeGUwVk6PcG6HU/images/astro-private-cloud/worker_queues_add_form.png?fit=max\&auto=format\&n=NVMeGUwVk6PcG6HU\&q=85\&s=1731ea9e8d6a06cf0ac4ccb7b30f5b6f) > [tab] APC API > > Send the complete list on `upsertDeployment`'s `workerQueues` argument, including the existing `default` queue: > > ```graphql expandable wrap theme={null} > mutation upsertDeployment( > $deploymentUuid: Uuid, > $workerQueues: [InputWorkerQueue!] > ) { > upsertDeployment( > deploymentUuid: $deploymentUuid, > workerQueues: $workerQueues > ) { > id > releaseName > workerQueues { > name > isDefault > cpu > memory > ephemeralStorage > replicas > workerConcurrency > nodePoolId > } > } > } > ``` > > With these variables: > > ```json expandable wrap theme={null} > { > "deploymentUuid": "", > "workerQueues": [ > { > "name": "default", > "cpu": 1000, > "memory": 3840, > "replicas": 1, > "workerConcurrency": 5 > }, > { > "name": "heavy-etl", > "cpu": 2000, > "memory": 7680, > "replicas": 3, > "workerConcurrency": 10, > "nodePoolId": "" > } > ] > } > ``` > > See [Manage worker queues with the APC API](https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-manage-worker-queues.md) for the full field reference and every error. After the update completes, a Kubernetes Deployment named `-worker-` runs the new queue's workers. Queue names aren't length-limited, but Kubernetes object names are, so a combined name longer than 63 characters is truncated and given a hash suffix. The queue still works, but is harder to identify. ### Units CPU is stored in millicores, and memory and ephemeral storage in mebibytes (MiB). Each field opens in its base unit, so you enter `1000` for one CPU and `3840` for 3.75 GiB. Use the unit selector next to each field to switch **mCPU** to **CPU** or **Mi** to **Gi**; values are stored in the base unit either way. Worker requests and limits are always set to the same value, so a worker gets exactly the resources you configure. ### Queue settings reference | Setting | UI label | Unit | Default | Platform default range | | - | - | - | - | - | | `name` | Queue name | — | — | Must match `^[a-z]([-a-z0-9]*[a-z0-9])?$` | | `cpu` | CPU per worker | Millicores | — | `100` to `3000` | | `memory` | Memory per worker | MiB | — | `384` to `11520` | | `ephemeralStorage` | Ephemeral storage per worker | MiB | `2048` | `1024` to `102400` | | `replicas` | Num workers | Workers | `1` | `1` to `10` | | `minWorkerCount` | Min workers | Workers | Same as `replicas` | `0` to `10`. `0` needs KEDA and Laminar on the data plane | | `maxWorkerCount` | Max workers | Workers | Same as `replicas` | `minWorkerCount` to `10` | | `workerConcurrency` | Worker concurrency | Tasks | Inherits the Deployment default | `1` to 15 per CPU core | | `nodePoolId` | Node pool | — | No pool | Must be a node pool on the Deployment's own cluster | The ranges come from `deployments.resourceManagement.components[workers]` and are overridable per cluster, Workspace, and Deployment, so your limits might differ from the platform defaults. `deployments.resourceManagement.maxPodCapacity` also caps the CPU and memory maximums, but binds only if you raise the component limit above it. The form shows the effective range under each field. `minWorkerCount` and `maxWorkerCount` take effect only when [worker autoscaling](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-autoscaling.md) is available. Otherwise, `replicas` sets a fixed worker count, and the platform default floor is `1`. A minimum of zero needs KEDA and Laminar on the data plane. See [Scale a queue to zero](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-autoscaling.md#scale-a-queue-to-zero). A platform administrator can also change the number of queues a Deployment can have with `deployments.workerQueues.maxPerDeployment`, which defaults to 10. > [!WARNING] > > The queue limit exists because of Kubernetes object size, not capacity. Every queue adds a full Pod template to the Deployment's Airflow resource, and a Kubernetes object has a hard ceiling of roughly 1.5 MiB. Past it, the apply is rejected and the Deployment stops reconciling. Raise `maxPerDeployment` only after measuring the resulting object size. ## Route tasks to a queue Set the Airflow `queue` argument on a task or on a Dag's default arguments: ```python wrap theme={null} from airflow.decorators import task @task(queue="heavy-etl") def transform(): ... ``` Tasks with no `queue` argument run on the `default` queue. > [!WARNING] > > There's no fallback for a queue that doesn't exist. If a task names a queue this Deployment doesn't have (typo, rename, or deletion), the task stays in the `queued` state indefinitely, with no explanation in the Airflow UI, task logs, or Deployment status. Create the queue before you ship the Dag code that uses it. ## Monitor worker queues Each queue card in the Worker Queues section shows a metrics strip: * **Current workers** shows how many workers are running, with terminating workers marked separately. * **CPU** and **Memory** show average usage across running workers as a share of each worker's configured size. A queue with a worker range also shows the range, for example `1 to 8 workers`, and, while the data plane can autoscale, a **Scaling history** link. See [Monitor autoscaling](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-autoscaling.md#monitor-autoscaling). ![A worker queue's metrics strip with cAdvisor and nodeExporter enabled, showing live CPU and memory percentages for the queue's workers.](https://mintcdn.com/astronomer/NVMeGUwVk6PcG6HU/images/astro-private-cloud/worker_queues_metrics_cadvisor_on.png?fit=max\&auto=format\&n=NVMeGUwVk6PcG6HU\&q=85\&s=28f5ba46b2bd3a1b196df5273b6ed679) CPU and memory need cAdvisor and nodeExporter metrics, which are off by default. When cAdvisor isn't enabled, the strip shows only the worker count and the line `CPU and memory usage are not available on this installation.` To turn cAdvisor and nodeExporter on, see [Deployment metrics](https://astronomer.io/docs/astro-private-cloud/v-2-x/deployment-metrics.md). With cAdvisor and nodeExporter enabled on both planes, the Deployment's **Metrics** page also breaks out CPU and memory per component once queues exist, listing each queue's worker and worker-gc sidecar as their own named series rather than folding them into a single `worker` line: ![The Deployment Metrics page's CPU Usage tooltip, listing each platform component and worker queue — including worker (default) and worker-gc (default) — as separate series.](https://mintcdn.com/astronomer/NVMeGUwVk6PcG6HU/images/astro-private-cloud/worker_queues_deployment_metrics_page.png?fit=max\&auto=format\&n=NVMeGUwVk6PcG6HU\&q=85\&s=0367b1ca318727e00da8392a885279cc) Queue depth isn't available. Logs are collected for every queue's workers, not only the default queue. See [Logs configuration](https://astronomer.io/docs/astro-private-cloud/v-2-x/logs-configuration.md). ## Edit a worker queue > [tab] Astro UI > > Open the Deployment's **Settings** tab, expand **Worker Queues**, click **Edit** on the queue, change the settings you want, click **Save**, then click **Deploy Changes**. > [tab] APC API > > Send the full list again with the changed values, through the same `upsertDeployment` mutation shown in [Add a worker queue](#add-a-worker-queue). This example doubles `heavy-etl`'s worker count and leaves `default` as it was: > > ```json expandable wrap theme={null} > { > "deploymentUuid": "", > "workerQueues": [ > { > "name": "default", > "cpu": 1000, > "memory": 3840, > "replicas": 1, > "workerConcurrency": 5 > }, > { > "name": "heavy-etl", > "cpu": 2000, > "memory": 7680, > "replicas": 6, > "workerConcurrency": 10, > "nodePoolId": "" > } > ] > } > ``` > > See [Manage worker queues with the APC API](https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-manage-worker-queues.md) for the full field reference. Resizing a queue rolls its workers with the new resources. Tasks already running get the termination grace period to finish. Changing a queue's worker count or concurrency also changes `AIRFLOW__CORE__PARALLELISM`, which is derived from the total across all queues. For example, raising `heavy-etl` from 3 to 6 workers at a concurrency of 10 adds 30 task slots. See [Parallelism](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-autoscaling.md#parallelism). Because the setting is Deployment-wide, the Airflow components that read it restart alongside the workers. A queue with a worker range counts at its maximum, so changing only its minimum restarts nothing. See [What restarts when you change a queue](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-autoscaling.md#what-restarts-when-you-change-a-queue). ### Rename a worker queue Renaming a queue deletes the old queue and creates a new one. The form shows a **Renaming replaces this queue** warning when you change the name of a saved queue. ![The Edit worker queue form showing the Renaming replaces this queue warning after changing a saved queue's name.](https://mintcdn.com/astronomer/NVMeGUwVk6PcG6HU/images/astro-private-cloud/worker_queues_rename_warning.png?fit=max\&auto=format\&n=NVMeGUwVk6PcG6HU\&q=85\&s=36820485ed1cea98f14c7f1c009581c7) Any task still routed to the old name waits indefinitely, as if the queue never existed. Update your Dag code first, or create the new queue, migrate the Dag code, and then delete the old queue. ## Delete a worker queue > [tab] Astro UI > > 1. Open the Deployment's **Settings** tab and expand **Worker Queues**. > 2. Click **Delete** on the queue. > 3. Enter the queue name to confirm, then click **Delete queue**. > 4. Click **Deploy Changes** to apply the change. > > The queue leaves the list immediately but isn't deleted until you click **Deploy Changes**. Cancel the form to keep it. > > ![The Delete queue confirmation dialog, requiring the queue name to be typed before the Delete queue button is enabled.](https://mintcdn.com/astronomer/NVMeGUwVk6PcG6HU/images/astro-private-cloud/worker_queues_delete_confirm.png?fit=max\&auto=format\&n=NVMeGUwVk6PcG6HU\&q=85\&s=ba9bb3922e518cb26f7d7410e1df228d) > [tab] APC API > > Send the full `workerQueues` list without the queue you want to remove. Dropping `heavy-etl` from the [Edit a worker queue](#edit-a-worker-queue) example deletes that queue and its workers. > > The `default` queue can't be dropped: a list that doesn't contain it is rejected. When the deletion is applied, the queue's workers stop immediately. Their pods are deleted without the termination grace period, so running tasks stop, and any task later routed to the queue waits indefinitely. Delete a queue only when it's idle. ## Permissions Deployment Admins and Deployment Editors can add, edit, and delete worker queues on their own Deployments. Deployment Viewers can see queues but not change them. Selecting a node pool needs read access to the cluster's node pools, which Deployment Editors have. Creating and editing node pools themselves is a cluster or system administrator task. See [Configure node pools](https://astronomer.io/docs/astro-private-cloud/v-2-x/node-pools.md). ## Troubleshoot worker queues ### A task stays in the queued state and nothing reports an error The task names a queue this Deployment doesn't have, and Airflow has no fallback. Check for a typo in the `queue` argument, a renamed queue, and a deleted queue. Compare the queue names in Deployment settings against the `queue` values in your Dag code. ### A queue's workers stay in the Pending state The workers can't be scheduled. Usual causes: a node pool whose node selector matches no nodes in the cluster, a node pool whose nodes carry taints the pool's tolerations don't cover, or a cluster with no spare capacity. Run `kubectl describe pod` against one of the Pending workers to see the scheduler's reason. ### A queue reports that its node pool isn't declared on this cluster The node pool the queue references no longer exists. Select another pool, or ask a cluster administrator to recreate it. The queue's workers aren't placed until you do. ### An out-of-range resource value returns an INTERNAL\_SERVER\_ERROR Out-of-range CPU, memory, and ephemeral storage values return `INTERNAL_SERVER_ERROR` rather than `BAD_USER_INPUT`. The message names the field, the value, and the allowed range. Component resource validation has the same known issue. ## See also * [Configure node pools](https://astronomer.io/docs/astro-private-cloud/v-2-x/node-pools.md) * [Manage worker queues with the APC API](https://astronomer.io/docs/astro-private-cloud/v-2-x/houston-manage-worker-queues.md) * [Scale Deployment resources](https://astronomer.io/docs/astro-private-cloud/v-2-x/scale-deployment-resources.md) * [Configure worker autoscaling](https://astronomer.io/docs/astro-private-cloud/v-2-x/configure-worker-autoscaling.md) * [Airflow operator mode](https://astronomer.io/docs/astro-private-cloud/v-2-x/airflow-operator-mode.md)