> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Autoscale Remote Execution Agent workers on queue depth

> Scale Remote Execution Agent worker Pods on the number of queued tasks, so that I/O-bound tasks do not exhaust the task slots on each worker.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Each Remote Execution Agent worker Pod runs a fixed number of concurrent tasks. The `syncSlots` value sets that number, and each concurrent task uses one slot. A worker that has no free slot does not accept more tasks, and the extra tasks stay in the queue.

Tasks that wait on an external system, such as a warehouse query or an API call, use very little CPU and memory. These tasks fill every slot on a worker while CPU utilization stays low. A Horizontal Pod Autoscaler (HPA) that scales on CPU or memory does not add replicas in this state, so the queue grows and task latency increases.

This document shows how to scale worker Pods on queue depth instead. Queue depth is the number of tasks in the `queued` and `running` states, which each agent worker reports in the `astro_agent_client_queue_stats` metric. It covers two setups: [prometheus-adapter](#scale-with-prometheus-adapter) with the HPA that the Helm chart creates, and [KEDA](#scale-with-keda) for clusters that do not run prometheus-adapter.

## Slot capacity and queue depth

The task capacity of a worker Deployment is `syncSlots` multiplied by the number of replicas. A Deployment with `syncSlots: 20` and two replicas runs 40 tasks at the same time.

Use this capacity to size the autoscaler:

* Set `maxReplicaCount` to your peak number of concurrent tasks divided by `syncSlots`, rounded up. For a peak of 200 concurrent tasks and `syncSlots: 20`, set `maxReplicaCount: 10`.
* Set the metric target to `syncSlots`. The autoscaler then adds one replica for each `syncSlots` tasks of work in the queue.
* Set `minReplicaCount` to the capacity you want available before the autoscaler reacts. Each scaling decision takes at least one metric interval, and a new worker Pod takes time to start.

Raising `syncSlots` is the other way to add capacity for I/O-bound tasks. A slot holds a task process, so more slots need more memory on the worker Pod. Test a higher value against your own tasks before you use it in production.

## Prerequisites

* A Remote Execution Agent that runs Agent Client version `1.7.0` or later, which exposes the `/metrics` endpoint. See [Register and configure agents](/docs/astro/remote-execution-configure-agents).
* A Prometheus instance that scrapes the agent worker Pods. See [Scrape metrics from Remote Execution Agents](/docs/astro/remote-agents-metrics).
* Permission to install cluster components and to run `helm upgrade` against the agent release.
* One of the following:
  * prometheus-adapter, which serves the metric through the Kubernetes custom metrics API.
  * KEDA, which reads Prometheus directly and creates its own HPA.

## Choose a scaling method

| Method             | How it works                                                                                                                     | Choose it when                                                                                                                                      |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| prometheus-adapter | An adapter publishes the queue metric on the Kubernetes custom metrics API. The Helm chart creates the HPA from `workers[].hpa`. | Your platform runs prometheus-adapter, or you can install it. All autoscaling configuration stays in `values.yaml`.                                 |
| KEDA               | A `ScaledObject` queries Prometheus directly. KEDA creates and owns the HPA.                                                     | Your platform has no custom metrics API, or another component already occupies it. Only one adapter can serve `custom.metrics.k8s.io` in a cluster. |

Do not use both methods for the same worker Deployment. Two autoscalers that target one Deployment overwrite each other's decisions.

## Scale with prometheus-adapter

<Steps>
  <Step title="Scrape the workers with the Deployment name in the job label">
    The adapter rule in the next step maps the `job` label to a Kubernetes Deployment, so `job` must hold the name of the worker Deployment. The Helm chart creates one Service for each worker, labeled `deploymentName: <resourceNamePrefix>-worker-<name>`, which a `ServiceMonitor` can copy into `job`:

    ```yaml title="worker-servicemonitor.yaml" wrap theme={null}
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: astro-agent-workers
      namespace: re
    spec:
      jobLabel: deploymentName
      selector:
        matchLabels:
          app: astro-agent
          component: worker
      endpoints:
        - port: http
          path: /metrics
    ```

    If you use a standalone Prometheus with static scrape configs, relabel the target so that `job` holds the same value.
  </Step>

  <Step title="Publish the queue metric through prometheus-adapter">
    Add the following rule to your prometheus-adapter configuration:

    ```yaml title="prometheus-adapter-values.yaml" wrap theme={null}
    rules:
      - seriesQuery: '{__name__="astro_agent_client_queue_stats",container!="POD",namespace!="",pod!=""}'
        resources:
          overrides:
            job:
              resource: deployment
            namespace:
              resource: namespace
        metricsQuery: sum by (job) (max by (job, queue, state) (<<.Series>>{state=~"queued|running", <<.LabelMatchers>>}))
        name:
          matches: ".*astro_agent_client_queue_stats.*"
          as: "astro_agent_client_queued_or_running_tasks"
    ```

    The counts come from the Astro orchestration plane in the worker heartbeat response, so every Pod of a worker Deployment reports the same values. `max by (job, queue, state)` removes the duplicate series, and `sum by (job)` adds the `queued` and `running` series together into one queue-depth value for each worker Deployment.
  </Step>

  <Step title="Confirm that the custom metric is available">
    Query the custom metrics API for the worker Deployment. Replace `re` with your namespace and `astro-worker-default-worker` with your Deployment name:

    ```bash wrap theme={null}
    kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/re/deployments.apps/astro-worker-default-worker/astro_agent_client_queued_or_running_tasks"
    ```

    The response contains a `value` field with the current queue depth. An error means that the adapter rule or the `job` label does not match. Fix this before you enable the HPA.
  </Step>

  <Step title="Enable the HPA on the worker">
    Configure the worker in `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    workers:
      - name: default-worker
        queues: "default"
        syncSlots: 20
        hpa:
          enabled: true
          minReplicaCount: 2
          maxReplicaCount: 10
          metric:
            enabled: true
            name: astro_agent_client_queued_or_running_tasks
            target:
              type: AverageValue
              averageValue: "20"
          behavior:
            scaleUp:
              stabilizationWindowSeconds: 0
              policies:
                - type: Pods
                  value: 1
                  periodSeconds: 60
            scaleDown:
              stabilizationWindowSeconds: 300
        terminationGracePeriodSeconds: 600
    ```

    The `averageValue` target of `20` matches `syncSlots: 20`, so the HPA runs one replica for each 20 tasks in the queue. See [Metric target types](#metric-target-types) for why queue depth needs an `AverageValue` target.

    The chart ignores `replicas` when `hpa.enabled` is `true`.
  </Step>

  <Step title="Apply the configuration and check the HPA">
    ```bash wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    kubectl get hpa -n re
    ```

    The `TARGETS` column shows the current queue depth against the target. If it shows `<unknown>`, run `kubectl describe hpa -n re` to see which metric the HPA cannot read.
  </Step>
</Steps>

## Scale with KEDA

KEDA queries Prometheus directly, so the cluster needs no custom metrics adapter. KEDA creates the HPA for the worker Deployment and owns it.

<Steps>
  <Step title="Turn off the HPA in the Helm chart">
    Set `hpa.enabled: false` for the worker, so that the chart and KEDA do not create two autoscalers for one Deployment:

    ```yaml title="values.yaml" wrap theme={null}
    workers:
      - name: default-worker
        queues: "default"
        syncSlots: 20
        replicas: 2
        hpa:
          enabled: false
    ```

    Keep `replicas` equal to the `minReplicaCount` you set in the next step. Each `helm upgrade` writes `replicas` back to the Deployment, and KEDA restores the scaled count at its next poll.
  </Step>

  <Step title="Create a ScaledObject for the worker Deployment">
    ```yaml title="worker-scaledobject.yaml" wrap theme={null}
    apiVersion: keda.sh/v1alpha1
    kind: ScaledObject
    metadata:
      name: astro-worker-default-worker
      namespace: re
    spec:
      scaleTargetRef:
        name: astro-worker-default-worker
      minReplicaCount: 2
      maxReplicaCount: 10
      pollingInterval: 30
      advanced:
        horizontalPodAutoscalerConfig:
          behavior:
            scaleDown:
              stabilizationWindowSeconds: 300
      triggers:
        - type: prometheus
          metricType: AverageValue
          metadata:
            serverAddress: http://prometheus-operated.monitoring.svc.cluster.local:9090
            threshold: "20"
            query: |-
              sum(max by (queue, state) (astro_agent_client_queue_stats{namespace="re",job="astro-worker-default-worker",state=~"queued|running"}))
    ```

    `scaleTargetRef.name` is the name of the worker Deployment, which the chart builds as `<resourceNamePrefix>-worker-<worker name>`.

    `threshold` matches `syncSlots`, and the default `AverageValue` metric type divides the query result by the threshold. KEDA therefore runs one replica for each 20 tasks in the queue.

    The query must return a single value. Adjust the label matchers to your own scrape configuration, and add a matcher for the worker Deployment when one Prometheus instance scrapes several of them. Read the labels from a worker's `/metrics` endpoint before you write the query.
  </Step>

  <Step title="Apply the ScaledObject and check the scaling">
    ```bash wrap theme={null}
    kubectl apply -f worker-scaledobject.yaml
    kubectl get scaledobject -n re
    kubectl get hpa -n re
    ```

    The `READY` and `ACTIVE` columns of the `ScaledObject` report whether KEDA can read the metric. KEDA names the HPA that it creates `keda-hpa-<scaled-object-name>`.
  </Step>
</Steps>

<Warning>
  Do not set `minReplicaCount: 0` for an agent worker. The queue metric comes from the worker Pods, so a Deployment that scales to zero exposes no metric and never scales back up.
</Warning>

### Other KEDA scalers

The `prometheus` trigger fits any platform that keeps the agent metrics in Prometheus or in a Prometheus-compatible service, such as Amazon Managed Service for Prometheus, Azure Monitor managed service for Prometheus, or Google Cloud Managed Service for Prometheus.

If your queue depth lives in another system, KEDA can read it with an `external` trigger, which calls a gRPC service that you run. The arithmetic stays the same: report the number of queued and running tasks, and set the threshold to `syncSlots`. For the list of triggers, see the [KEDA scalers reference](https://keda.sh/docs/latest/scalers/).

## Metric target types

Kubernetes computes the replica count differently for each target type. Use `AverageValue` for queue depth.

| Target type    | Replica count                                    | Use it for                                                              |
| -------------- | ------------------------------------------------ | ----------------------------------------------------------------------- |
| `AverageValue` | `metric / target`, rounded up                    | A metric that counts work for the whole Deployment, such as queue depth |
| `Value`        | `metric / target × current replicas`, rounded up | A metric that already describes a single replica                        |

A `Value` target multiplies the ratio by the current replica count. With a queue-depth metric, the replica count therefore multiplies again in each scaling cycle for as long as the queue stays above the target, until the HPA reaches `maxReplicaCount`. For the full algorithm, see the [Kubernetes HPA documentation](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details).

## Combine queue depth with resource metrics

Queue depth measures outstanding work, not the load on a Pod. Tasks that use a lot of memory can still exhaust a worker before its slots are full. Add resource metrics with `hpa.extraMetrics`, which takes Kubernetes HPA metric objects:

```yaml title="values.yaml" wrap theme={null}
    hpa:
      enabled: true
      minReplicaCount: 2
      maxReplicaCount: 10
      metric:
        enabled: true
        name: astro_agent_client_queued_or_running_tasks
        target:
          type: AverageValue
          averageValue: "20"
      extraMetrics:
        - type: Resource
          resource:
            name: memory
            target:
              type: Utilization
              averageUtilization: 80
```

The HPA evaluates every metric and uses the highest replica count that any of them recommends. Resource metrics need `resources.requests` on the worker, because the HPA calculates utilization against the request.

With KEDA, add a second trigger of type `memory` or `cpu` to the same `ScaledObject`.

## Scale down without task failures

A worker that stops during a task fails that task. Two settings protect running work:

* `terminationGracePeriodSeconds` on the worker gives a Pod time to finish its tasks before Kubernetes stops it. The default is `600`. Set it higher than your longest task if your tasks run for more than 10 minutes.
* `behavior.scaleDown.stabilizationWindowSeconds` makes the autoscaler wait before it removes replicas. The examples on this page use `300`.

A queue of short tasks therefore makes the value dip and recover between scaling cycles. `scaleDown.stabilizationWindowSeconds` makes the HPA use the highest replica count it recommended over the trailing window, so a brief dip does not remove a worker.

## Related documentation

* [Helm chart configuration reference](/docs/astro/remote-agents-helm-reference)
* [Scrape metrics from Remote Execution Agents](/docs/astro/remote-agents-metrics)
* [Register and configure agents](/docs/astro/remote-execution-configure-agents)
* [Remote Execution Agent failure scenarios](/docs/astro/remote-agents-failure-scenarios)