Configure node pools for Astro Private Cloud worker queues

A node pool is a named description of a node group that already exists in one of your data plane clusters. It records the node selector and tolerations that place a Pod on those nodes, and the largest worker those nodes can hold. Astro Private Cloud doesn’t create or discover node groups. A cluster administrator declares each group once as a node pool, and Deployment users select a pool by name when they configure a worker queue. You provision and maintain the underlying node group, including keeping enough nodes of the declared size and labels available. Node pools place worker queues only, not other Airflow components.

Astro Private Cloud 2.2This feature was introduced in Astro Private Cloud 2.2. To access this feature, upgrade your Astro Private Cloud installation to 2.2 or later.

Prerequisites

Astro Private Cloud 2.2.0 or later. See Upgrade to a minor version.
A registered data plane cluster. See Register a data plane.
A node group in that cluster, with a label you can match on. Taints are optional.
A role with node pool write permissions. Cluster Admin and System Admin can create, update, and delete pools; System Editor can update them. See Permissions.

The worker queues feature flag doesn’t gate node pools. The catalogue is available whether or not worker queues are turned on. The Astro CLI has no commands for managing node pools. Use the Astro UI or the APC API.

Create a node pool

Astro UI
APC API

In the Astro Private Cloud UI, go to Clusters and select your cluster.
Click the Node Pools tab.
Click Node Pool.
In Name, enter a name. Worker queues reference this name, which must be unique within the cluster.
(Optional) In Description, enter a note for other administrators. It isn’t used for scheduling.
In Node selector, enter the label key and value that identify these nodes. Click Add label to match on more than one label. Leave it empty for a pool that can use any node.
(Optional) Under Tolerations, click Add toleration for each taint on these nodes. See Tolerations.
(Optional) Set Max CPU (millicores), Max memory (MiB), and Max ephemeral storage (MiB). See Ceilings.
(Optional) In Nodes for worker queues, enter how many of these nodes worker queues can use.
Click Add Node Pool.

The Add Node Pool form, showing name, description, node selector, tolerations, ceilings, and node count fields.

mutation createNodePool($clusterId: Uuid!, $input: InputNodePool!) {
  createNodePool(clusterId: $clusterId, input: $input) {
    id
    name
    nodeSelector
    maxCpu
    maxMemory
  }
}

With these variables:

{
  "clusterId": "<cluster-id>",
  "input": {
    "name": "compute-optimized",
    "description": "Nodes reserved for ETL workloads",
    "nodeSelector": {
      "node-role": "etl"
    },
    "tolerations": [
      {
        "key": "workload",
        "operator": "Equal",
        "value": "airflow",
        "effect": "NoSchedule"
      }
    ],
    "maxCpu": 15000,
    "maxMemory": 30720,
    "maxEphemeralStorage": 51200,
    "maxNodeCount": 12
  }
}

See Manage node pools with the APC API for the full InputNodePool field reference and every possible error.

Tolerations

Add a toleration only if the nodes carry taints. Each toleration takes a key, an operator, and optionally a value, an effect, and a number of seconds:

Operator is Exists or Equal. With Exists, the value field is disabled because Exists matches the key regardless of value.
Leave the key empty only with the Exists operator. An empty key matches every taint.
Effect is NoSchedule, PreferNoSchedule, NoExecute, or Any effect to match all three.
The seconds field applies only to the NoExecute effect.

Ceilings

The three maximums describe what a single node in this group can give a worker, after the kubelet and daemon sets take their share, so they’re smaller than the instance type’s nominal size. Each ceiling caps a single worker. Saving a worker queue that asks for more CPU, memory, or ephemeral storage than its pool allows is rejected, naming the Deployment, queue, field, requested value, and ceiling. The block is hard because a Pod larger than its node stays Pending permanently with no actionable error. Leave a ceiling empty to impose no pool-level limit on that dimension. The platform’s own resource limits still apply. The form shows the equivalent beneath each field. For example, 15000 for Max CPU (millicores) shows 15 cores, and 30720 for Max memory (MiB) shows 30 GiB.

Node count and totals

Nodes for worker queues multiplied by the three ceilings gives the pool’s total budget. Two node pools that describe the same physical nodes should split the node count between them rather than each claiming all of them. Going over the total warns but doesn’t block. A queue counts against the total at its maximum worker count, so a pool can be committed beyond its budget with nothing broken yet. Workers past the total wait for a node that never arrives. For a queue with a worker range, the maximum is its Max workers, even while it runs fewer. The over-commitment warning appears in:

An Over total badge in the Committed column of the node pool list.
A panel in the node pool form showing which queues are committed against the pool, largest first.
A warning in the Worker Queues section of any Deployment whose queues would push the pool over.

Review node pools

The Node Pools tab lists every pool on the cluster:

The Node Pools tab on a cluster, listing node pools with their node selector, max pod size, node count, committed capacity, and worker queue count.

Column	Description
Node Pool	The pool name, with its description beneath
Node selector	The label keys and values, or `Any node without a selector`
Max pod size	The CPU and memory ceilings, in cores and GiB
Nodes	The declared node count
Committed	What the worker queues on this pool add up to, against the pool’s total
Worker queues	How many worker queues reference this pool
Created	When the pool was created
Created by	Who created it

Edit a node pool

Astro UI
APC API

Click Edit on the pool, change the fields you want, and click Save Node Pool.An update is a partial patch on the optional fields: a ceiling or toleration you leave alone keeps its stored value. To clear a ceiling, set it explicitly to empty.

mutation updateNodePool($nodePoolId: Uuid!, $input: InputNodePool!) {
  updateNodePool(nodePoolId: $nodePoolId, input: $input) {
    id
    name
    maxCpu
    pendingConvergenceDeploymentIds
  }
}

name and nodeSelector are non-null on InputNodePool, so every update must resend both. A call that sends only { "maxCpu": null } fails schema validation.The optional fields behave as a partial patch, unlike InputWorkerQueue on upsertDeployment:

An optional field you omit keeps its stored value.
An optional field you set to null is cleared.

See Manage node pools with the APC API for the full field reference and every possible error.

To fix a mis-set ceiling, clear it explicitly, because a pool in use can’t be deleted and recreated. Renaming a pool is safe, because worker queues reference a pool by its identifier, not its name. A name that collides with another pool on the same cluster is refused.

Lower a ceiling

Lowering a ceiling below what an existing worker queue requests is refused. The error names every conflicting Deployment, queue, field, requested value, and new ceiling. Resize those queues or move them to another pool first.

Change placement

Changing a pool’s node selector or tolerations doesn’t affect Deployments that already use the pool until each one is rebuilt. Until then those workers keep selecting the old node group, while the Node Pools page shows the new label. After you save a placement change, the UI asks Upgrade deployments? and names how many Deployments haven’t picked it up. Click Upgrade to rebuild exactly those Deployments.

The Upgrade deployments? dialog shown after a node pool's placement changes, naming how many Deployments haven't picked it up yet.

If you’re changing a pool’s placement because the old node group is being decommissioned, upgrade the affected Deployments instead of waiting for them to converge. Workers still pointed at nodes that disappear can’t be scheduled when they next restart.

Delete a node pool

Astro UI
APC API

Click Delete on the pool, then click Delete Node Pool.A pool that any worker queue references can’t be deleted. The dialog shows how many queues use it, and the Delete Node Pool button is disabled. Move those queues to another pool, or remove them, first.

The Cannot delete node pool dialog, shown when a node pool is still referenced by a worker queue, with the Delete Node Pool button disabled.

mutation deleteNodePool($nodePoolId: Uuid!) {
  deleteNodePool(nodePoolId: $nodePoolId) {
    id
    name
  }
}

A pool that any worker queue references can’t be deleted. The error names the Deployments and queues that still use it. Move or remove those queues first.See Manage node pools with the APC API for the full error reference.

Deleting an unused pool is immediate, and new worker queues can no longer target it. Node pools are hard-deleted, with no recovery.

Permissions

Role	Read	Create	Update	Delete
System Admin	Yes	Yes	Yes	Yes
System Editor	Yes	No	Yes	No
System Viewer	Yes	No	No	No
Cluster Admin	Yes	Yes	Yes	Yes
Workspace Admin	No	No	No	No
Deployment Admin or Editor	Yes, to populate the node pool selector	No	No	No

The underlying permissions are system.nodePools.get, system.nodePools.create, system.nodePools.update, and system.nodePools.delete at the system level, and the matching cluster.nodePools.* permissions scoped to a single cluster. A Cluster Admin holds the cluster.nodePools.* set on their own cluster and none on any other. See Role and permission reference. A user without node pool permissions who queries node pools directly gets a permission error, not an empty list.

​Prerequisites

​Create a node pool

​Tolerations

​Ceilings

​Node count and totals

​Review node pools

​Edit a node pool

​Lower a ceiling

​Change placement

​Delete a node pool

​Permissions

​See also

Prerequisites

Create a node pool

Tolerations

Ceilings

Node count and totals

Review node pools

Edit a node pool

Lower a ceiling

Change placement

Delete a node pool

Permissions

See also