> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Autoscale Remote Execution Agent workers on queue depth

> Scale Remote Execution Agent worker Pods on the number of queued tasks, so that I/O-bound tasks do not exhaust the task slots on each worker.

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Each Remote Execution Agent worker Pod runs a fixed number of concurrent tasks. The `syncSlots` value sets that number, and each concurrent task uses one slot. A worker that has no free slot does not accept more tasks, and the extra tasks stay in the queue.

Tasks that wait on an external system, such as a warehouse query or an API call, use very little CPU and memory. These tasks fill every slot on a worker while CPU utilization stays low. A Horizontal Pod Autoscaler (HPA) that scales on CPU or memory does not add replicas in this state, so the queue grows and task latency increases.

This document shows how to scale worker Pods on queue depth instead. Queue depth is the number of tasks in the `queued` and `running` states.

## Choose a scaling method

The Remote Execution Agent Helm chart includes support for two approaches:

| Method               | How it works                                                                                                                             | Choose it when...                                                                                                                                                                           |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [KEDA]               | The chart creates a KEDA `ScaledObject` that reads each worker's queue backlog from the Sentinel service. KEDA creates and owns the HPA. | <ul><li>...KEDA can be installed in the cluster.</li></ul>                                                                                                                                  |
| [prometheus-adapter] | An adapter republishes the queue metric on the Kubernetes custom metrics API. The chart creates an HPA.                                  | <ul><li>...you cannot install KEDA or do not wish to.</li><li>...your platform already runs Prometheus and prometheus-adapter.</li><li>...you cannot enable the Sentinel service.</li></ul> |

Astronomer recommends you use KEDA autoscaling whenever possible. This method needs no Prometheus and no metrics adapter, all configuration stays in `values.yaml`, and **it is the only method that supports scale-to-zero**.

You can only use one method (HPA or KEDA) for each worker deployment.

Learn more about:

* [Autoscale with KEDA](#autoscale-with-keda-recommended).
* [Autoscale with HPA and prometheus-adapter](#autoscale-with-hpa-and-prometheus-adapter).

[KEDA]: https://keda.sh/

[prometheus-adapter]: https://github.com/kubernetes-sigs/prometheus-adapter

## Slot capacity and queue depth

The task capacity of a worker deployment is `syncSlots` multiplied by the number of replicas. A deployment with `syncSlots: 20` and two replicas can run 40 tasks at the same time.

Use desired capacity to configure autoscaling:

* Set `maxReplicaCount` to your peak number of concurrent tasks divided by `syncSlots`, rounded up. For a peak of 200 concurrent tasks and `syncSlots: 20`, set `maxReplicaCount: 10`.
* If you are using HPA, set the metric target explicitly to the same value as
  the worker's `syncSlots`. (You do not need to configure this explicitly with KEDA: the chart configures it for you.)
* Set `minReplicaCount` to the capacity you want available at all times. The
  minimum value of `minReplicaCount` is 1 with HPA and 0 with KEDA.

Raising `syncSlots` is the other way to add capacity for I/O-bound tasks. A slot holds a task process, so more slots need more memory on the worker Pod. Test a higher value against your own tasks before you use it in production.

## Prerequisites

KEDA-based autoscaling requires:

* Helm chart version `2.4.0` or later.
* Remote Execution Agent version `1.8.4` or later.
* Sentinel enabled with `sentinel.enabled: true`. See [Sentinel for Remote Execution Agents](/docs/astro/remote-execution/remote-agents-sentinel).
* KEDA must be installed in the cluster. See [Deploying KEDA](https://keda.sh/docs/latest/deploy/).

Autoscaling with HPA and prometheus-adapter requires:

* Remote Execution Agent version `1.7.0` or later.
* A Prometheus instance that scrapes the agent worker Pods. See [Scrape metrics from Remote Execution Agents](/docs/astro/remote-execution/remote-agents-metrics).
* [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter),
  which serves the metric through the Kubernetes custom metrics API.

## Autoscale with KEDA (recommended)

Simply set `keda.enabled: true` on a worker configuration, and the chart will create a KEDA `ScaledObject` for that worker's Deployment, pre-configured to scale the worker based on queue depth.

The backlog metric is provided by the Sentinel, so even when workers are scaled to zero KEDA can still query the backlog and will scale back up when work arrives.

<Warning>
  The chart does not install KEDA or its CRDs, and it does not check that they are present. Install KEDA before you enable this feature. Without the CRDs, `helm upgrade` will fail because the cluster does not recognize the `ScaledObject` resource. With the CRDs but no running KEDA operator, the chart applies the `ScaledObject` and nothing acts on it.
</Warning>

<Steps>
  <Step title="Enable KEDA on the worker">
    Configure the worker in `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    sentinel:
      enabled: true

    workers:
      - name: default-worker
        queues: "default"
        syncSlots: 20
        keda:
          enabled: true
          minReplicaCount: 0
          maxReplicaCount: 10
        terminationGracePeriodSeconds: 600
    ```

    `minReplicaCount` defaults to `0`, which lets the worker scale to zero when its queues are empty. Set it higher to keep idle capacity ready for the first tasks of a run.

    `maxReplicaCount` defaults to `10`. This default caps the Deployment at `10 × syncSlots` concurrent tasks, so set it from your own peak concurrency if needed.

    <Warning>
      To avoid task interruptions, you must set `terminationGracePeriodSeconds` to a value longer than the longest-running task in your workflow. If you have tasks that can run for 8h, set `terminationGracePeriodSeconds: 28800`.
    </Warning>

    Consult the chart's default `values.yaml` (e.g. via `helm show values astronomer/astro-remote-execution-agent`) for additional configuration options you can use to tweak KEDA's behavior.
  </Step>

  <Step title="Apply the configuration and check the scaling">
    ```bash wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    kubectl get scaledobject -n re
    ```

    The `READY` and `ACTIVE` columns of the `ScaledObject` report whether KEDA can read the backlog metric. `ACTIVE` is `False` for an idle worker, which is the state in which KEDA holds the Deployment at `minReplicaCount`.
  </Step>
</Steps>

## Autoscale with HPA and prometheus-adapter

This method uses the `astro_agent_client_queue_stats` metric, which each agent worker exposes on its `/metrics` endpoint, and it needs Prometheus and prometheus-adapter in the cluster. Only one adapter can serve `custom.metrics.k8s.io` in a cluster, so this method is unavailable if another component already occupies that API.

<Steps>
  <Step title="Scrape the workers with the Deployment name in the job label">
    The adapter rule in the next step maps the `job` label to a Kubernetes Deployment, so `job` must hold the name of the worker Deployment. The Helm chart creates one Service for each worker, labeled `deploymentName: <resourceNamePrefix>-worker-<name>`, which a `ServiceMonitor` can copy into `job`:

    ```yaml title="worker-servicemonitor.yaml" wrap theme={null}
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: astro-agent-workers
      namespace: re
    spec:
      jobLabel: deploymentName
      selector:
        matchLabels:
          app: astro-agent
          component: worker
      endpoints:
        - port: http
          path: /metrics
    ```

    If you use a standalone Prometheus with static scrape configs, relabel the target so that `job` holds the same value.
  </Step>

  <Step title="Publish the queue metric through prometheus-adapter">
    Add the following rule to your prometheus-adapter configuration:

    ```yaml title="prometheus-adapter-values.yaml" wrap theme={null}
    rules:
      - seriesQuery: '{__name__="astro_agent_client_queue_stats",container!="POD",namespace!="",pod!=""}'
        resources:
          overrides:
            job:
              resource: deployment
            namespace:
              resource: namespace
        metricsQuery: sum by (job) (max by (job, queue, state) (<<.Series>>{state=~"queued|running", <<.LabelMatchers>>}))
        name:
          matches: ".*astro_agent_client_queue_stats.*"
          as: "astro_agent_client_queued_or_running_tasks"
    ```

    The counts come from the Astro orchestration plane in the worker heartbeat response, so every Pod of a worker Deployment reports the same values. `max by (job, queue, state)` removes the duplicate series, and `sum by (job)` adds the `queued` and `running` series together into one queue-depth value for each worker Deployment.
  </Step>

  <Step title="Confirm that the custom metric is available">
    Query the custom metrics API for the worker Deployment. Replace `re` with your namespace and `astro-worker-default-worker` with your Deployment name:

    ```bash wrap theme={null}
    kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/re/deployments.apps/astro-worker-default-worker/astro_agent_client_queued_or_running_tasks"
    ```

    The response contains a `value` field with the current queue depth. An error means that the adapter rule or the `job` label does not match. Fix this before you enable the HPA.
  </Step>

  <Step title="Enable the HPA on the worker">
    Configure the worker in `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    workers:
      - name: default-worker
        queues: "default"
        syncSlots: 20
        hpa:
          enabled: true
          minReplicaCount: 2
          maxReplicaCount: 10
          metric:
            enabled: true
            name: astro_agent_client_queued_or_running_tasks
            target:
              type: AverageValue
              averageValue: "20"
          behavior:
            scaleUp:
              stabilizationWindowSeconds: 0
              policies:
                - type: Pods
                  value: 1
                  periodSeconds: 60
            scaleDown:
              stabilizationWindowSeconds: 300
        terminationGracePeriodSeconds: 600
    ```

    The `averageValue` target of `20` matches `syncSlots: 20`, so the HPA runs one replica for each 20 tasks in the queue. See [Metric target types](#metric-target-types) for why queue depth needs an `AverageValue` target.

    The chart ignores `replicas` when `hpa.enabled` is `true`.

    <Warning>
      To avoid task interruptions, you must set `terminationGracePeriodSeconds` to a value longer than the longest-running task in your workflow. If you have tasks that can run for 8h, set `terminationGracePeriodSeconds: 28800`.
    </Warning>
  </Step>

  <Step title="Apply the configuration and check the HPA">
    ```bash wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    kubectl get hpa -n re
    ```

    The `TARGETS` column shows the current queue depth against the target. If it shows `<unknown>`, run `kubectl describe hpa -n re` to see which metric the HPA cannot read.
  </Step>
</Steps>

### Metric target types

Kubernetes computes the replica count differently for each target type. Use `AverageValue` for queue depth.

| Target type    | Replica count                                    | Use it for                                                              |
| -------------- | ------------------------------------------------ | ----------------------------------------------------------------------- |
| `AverageValue` | `metric / target`, rounded up                    | A metric that counts work for the whole Deployment, such as queue depth |
| `Value`        | `metric / target × current replicas`, rounded up | A metric that already describes a single replica                        |

A `Value` target multiplies the ratio by the current replica count. With a queue-depth metric, the replica count therefore multiplies again in each scaling cycle for as long as the queue stays above the target, until the HPA reaches `maxReplicaCount`. For the full algorithm, see the [Kubernetes HPA documentation](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details).

## Combine queue depth with resource metrics

Queue depth measures outstanding work, not the load on a Pod. Tasks that use a lot of memory can still exhaust a worker before its slots are full.

With KEDA, add a `cpu` or `memory` trigger to the generated `ScaledObject` with `keda.extraTriggers`, which takes a list of KEDA trigger objects:

```yaml title="values.yaml" wrap theme={null}
    keda:
      enabled: true
      maxReplicaCount: 10
      extraTriggers:
        - type: memory
          metricType: Utilization
          metadata:
            value: "80"
```

For the full list of triggers, see the [KEDA scalers reference](https://keda.sh/docs/latest/scalers/).

With an HPA, add resource metrics with `hpa.extraMetrics`, which takes Kubernetes HPA metric objects:

```yaml title="values.yaml" wrap theme={null}
    hpa:
      enabled: true
      minReplicaCount: 2
      maxReplicaCount: 10
      metric:
        enabled: true
        name: astro_agent_client_queued_or_running_tasks
        target:
          type: AverageValue
          averageValue: "20"
      extraMetrics:
        - type: Resource
          resource:
            name: memory
            target:
              type: Utilization
              averageUtilization: 80
```

In both cases the HPA evaluates every metric and uses the highest replica count that any of them recommends. Resource metrics need `resources.requests` on the worker, because the HPA calculates utilization against the request.

## Scale down without task failures

A worker that stops during a task fails that task. These settings protect running work:

* `terminationGracePeriodSeconds` on the worker gives a Pod time to finish its tasks before Kubernetes stops it. The default is `600`. Set it higher than your longest task if your tasks run for more than 10 minutes.
* `keda.cooldownPeriod` makes KEDA wait after the backlog last justified more than `minReplicaCount` replicas before it scales back down. The default is `300`.
* `hpa.behavior.scaleDown.stabilizationWindowSeconds` makes the HPA use the highest replica count it recommended over the trailing window, so a brief dip in queue depth does not remove a worker. The HPA examples on this page use `300`.

A queue of short tasks makes queue depth dip and recover between scaling cycles, which is what these windows absorb.

## Related documentation

* [Helm chart configuration reference](/docs/astro/remote-execution/remote-agents-helm-reference)
* [Sentinel for Remote Execution Agents](/docs/astro/remote-execution/remote-agents-sentinel)
* [Scrape metrics from Remote Execution Agents](/docs/astro/remote-execution/remote-agents-metrics)
* [Register and configure agents](/docs/astro/remote-execution/remote-execution-configure-agents)
* [Remote Execution Agent failure scenarios](/docs/astro/remote-execution/remote-agents-failure-scenarios)