> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure component size limits

Astro Private Cloud allows you to customize the minimum and maximum sizes of most Astronomer platform and Airflow components.

You can configure the CPU and memory resources of Airflow components through the Astro UI or with the APC API.

<a id="configure-max-pod-size" />

## Configure Deployment-level limits for individual Pod sizes

You can use `astronomer.houston.config.deployments.maxPodCapacity` to configure the maximum size any individual pod can be.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        maxPodCapacity:
          cpu: 3500
          memory: 13440
```

<a id="configure-cumulative-resource-limits" />

## Configure Deployment-level limits for resource usage

Astro Private Cloud limits the amount of resources that can be used by all pods in a Deployment by creating and managing a `LimitRange` and `ResourceQuota` for the namespace associated with each Deployment.

These values are automatically adjusted to account for the resource requirements of various components.

You can add additional resources, beyond the standard amount allocated based on the resource-requirements of standing components, to the `LimitRange` and `ResourceQuota`. Add resources by configuring `astronomer.houston.config.deployments.maxExtraCapacity` to account for the requirements of KubernetesExecutor and `KubernetesPodOperator` tasks.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        maxExtraCapacity:
          cpu: 40000
          memory: 153600
```

<a id="configure-deployment-components" />

## Configure the sizes of individual Deployment-level components

Components represent different parts of the Astro Private Cloud Deployment. You can customize the default configuration for a component by defining it in `astronomer.houston.config.deployments.components`.

A list of configurable components and options is provided in [Configurable Components](#configurable-components).

<Tip>KubernetesExecutor Task pod sizes are created on an as-needed basis, and don't have persisting resource requirements. Their resource requirements are [configured at the task level](#configure-kubernetes-task-pod-size).</Tip>

When defining components, you must include the full definition of the component in the list entry after the components key, instead of only the components you want to define.

For example, to increase the maximum size a Celery worker task from 3 Vcpu/11.5Gi to 3 Vcpu/192Gi, add the equivalent (in milli vCPU and Mi) full Celery worker component definition to `astronomer.houston.config.deployments.components` in your `values.yaml` with a higher limit:

<Tip>When increasing CPU or memory limits, ensure the [maximum pod size](#configure-max-pod-size) is large enough to avoid errors during pod creation.</Tip>

```yaml expandable wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        components:
        - name: workers
          resources:
            default:
              cpu: 1000
              memory: 3840
            minimum:
              cpu: 100
              memory: 384
            limit:
              cpu: 3000
              memory: 11520
            KubernetesExecutor:
              default:
                cpu: 100
                memory: 384
              minimum:
                cpu: 100
                memory: 384
              limit:
                cpu: 3000
                memory: 11520
          extra:
            - name: terminationGracePeriodSeconds
              default: 600
              minimum: 0
              limit: 36000
            - name: replicas
              default: 1
              minimum: 1
              limit: 10
        # any additional component configurations go here
        # - name: another-component
        #   resources:
        #     default: 10
        #     ...
```

### Configurable components

<Info>When defining components, you must include the full definition of the component in the list entry after the components key, instead of only the components you want to define.</Info>

<Tip>KubernetesExecutor task pod sizes are created on an as-needed basis and don't have persisting resource requirements. Their resource requirements are [configured at the task level](#configure-kubernetes-task-pod-size).</Tip>

Configurable components include:

#### Airflow scheduler

```yaml wrap theme={null}
- name: scheduler
  resources:
    default:
     cpu: 500
     memory: 1920
    minimum:
     cpu: 500
     memory: 1920
    limit:
     cpu: 3000
     memory: 11520
  extra:
    - name: replicas
      default: 1
      minimum: 1
      limit: 4
```

#### Airflow webserver

```yaml wrap theme={null}
- name: webserver
  resources:
    default:
     cpu: 500
     memory: 1920
    minimum:
     cpu: 500
     memory: 1920
    limit:
     cpu: 3000
     memory: 11520
```

#### Airflow `apiServer` (Airflow 3.0 and above)

```yaml wrap theme={null}
- name: apiServer
  resources:
    default:
     cpu: 1000
     memory: 3840
    minimum:
     cpu: 1000
     memory: 3840
    limit:
     cpu: 3000
     memory: 11520
   extra:
    - name: replicas
      default: 1
      minimum: 1
      limit: 4
```

#### StatsD

```yaml wrap theme={null}
- name: statsd
  resources:
    default:
     cpu: 200
     memory: 768
    minimum:
     cpu: 200
     memory: 768
    limit:
     cpu: 3000
     memory: 11520
```

#### Database connection pooler (PgBouncer)

```yaml wrap theme={null}
- name: pgbouncer
  resources:
    default:
     cpu: 200
     memory: 768
    minimum:
     cpu: 200
     memory: 768
    limit:
     cpu: 200
     memory: 768
```

#### Celery diagnostic web interface (Flower)

```yaml wrap theme={null}
- name: flower
  resources:
    default:
     cpu: 200
     memory: 768
    minimum:
     cpu: 200
     memory: 768
    limit:
     cpu: 200
     memory: 768
```

#### Redis

```yaml wrap theme={null}
- name: redis
  resources:
    default:
     cpu: 200
     memory: 768
    minimum:
     cpu: 200
     memory: 768
    limit:
     cpu: 200
     memory: 768
```

#### Celery workers

```yaml wrap theme={null}
- name: workers
  resources:
    default:
     cpu: 1000
     memory: 3840
    minimum:
     cpu: 100
     memory: 384
    limit:
     cpu: 3000
     memory: 11520
  extra:
    - name: terminationGracePeriodSeconds
      default: 600
      minimum: 0
      limit: 36000
    - name: replicas
      default: 1
      minimum: 1
      limit: 10
```

#### Triggerer

```yaml expandable wrap theme={null}
- name: triggerer
  resources:
    default:
     cpu: 500
     memory: 1920
    minimum:
     cpu: 500
     memory: 1920
    limit:
     cpu: 3000
     memory: 11520
  extra:
    - name: replicas
      default: 1
      minimum: 0
      limit: 2
```

#### Dag processor

```yaml wrap theme={null}
- name: dagProcessor
  resources:
    default:
     cpu: 500
     memory: 1920
    minimum:
     cpu: 500
     memory: 1920
    limit:
     cpu: 3000
     memory: 11520
  extra:
    - name: replicas
      default: 0
      minimum: 0
      limit: 3
```

<a id="disable-kubernetes-resources-ui-api" />

##  Manage Kubernetes worker CPU and memory with global platform config

You can use `workers.resources.enabled` for the `KubernetesExecutor` to manage worker Pod CPU and memory allocations with a platform configuration. When set to `false`, Astro Private Cloud ignores worker Pod CPU or memory allocations set in the API or UI. This allows you to control Pod resources with your own Kubernetes policies or admission controllers.

### API/UI manages Pod resources (Default KubernetesExecutor configuration)

To configure the default behavior where the API and UI manage worker Pod CPU and memory resources for the KubernetesExecutor, use the following configuration:

```yaml wrap theme={null}
deployments:
  executors:
    - name: KubernetesExecutor
      enabled: true
      components:
        - scheduler
        - webserver
        - apiServer
        - statsd
        - pgbouncer
        - triggerer
        - dagProcessor
      defaultExtraCapacity:
        cpu: 1000
        memory: 3840
      workers:
        ephemeralStorage:
          disabled: true
        resources:
          enabled: true
```

### Disable API/UI resource configuration

If you configure your KubernetesExecutor so that CPU and memory requests and limits are set outside of Astro Private Cloud, you must disable resource configuration set with the Astro Private Cloud API/UI. To disable the Astro Private Cloud UI/API configuration, add the `resources.enabled: false` flag to your `values.yaml` file. This ensures that Astro Private Cloud applies the worker resource settings exclusively from your preferred source, such as Pod mutation hooks or Pod configuration files. The following configuration sets them as an empty `dict`.

```yaml wrap theme={null}
deployments:
  executors:
    - name: KubernetesExecutor
      enabled: true
      components:
        - scheduler
        - webserver
        - apiServer
        - statsd
        - pgbouncer
        - triggerer
        - dagProcessor
      defaultExtraCapacity:
        cpu: 1000
        memory: 3840
      workers:
        ephemeralStorage:
          disabled: true
        resources:
          enabled: false
```

Existing deployments are unchanged unless you update this setting.

<a id="configure-kubernetes-task-pod-size" />

### Configure the size of KubernetesExecutor task pods

Kubernetes Executor task pods are defined at the task level when the Dag passes resource requests as part of `executor_config` into the Operator. When not defined, these tasks default to using 0.1 Vcpu/384 Mi of memory. This means that when you define resource requests or limits for CPU and memory, ensure the [maximum pod size](#configure-max-pod-size) is large enough to avoid errors during pod creation.

<Danger>Astro Private Cloud doesn't automatically raise the namespace-level cumulative resource limits for pods created by the KubernetesExecutor. To avoid pod creation failures, increase the `maxExtraCapacity` to support your desired level of resourcing and concurrency.</Danger>

The following example demonstrates how to configure resource limits and requests:

```text expandable wrap theme={null}
# import kubernetes.client.models as k8s
from kubernetes.client import models as k8s
# define an executor_config with the desired resources
my_executor_config={
  "pod_override": k8s.V1Pod(
      spec=k8s.V1PodSpec(
          containers=[
              k8s.V1Container(
                  name="base",
                  resources=k8s.V1ResourceRequirements(
                      requests={
                          "cpu": "50m",
                          "memory": "384Mi"
                      },
                      limits={
                          "cpu": "1000m",
                          "memory": "1024Mi"
                      }
                  )
              )
          ]
      )
  )
}
# pass in executor_config=my_executor_config to any Operator
#@task(executor_config=my_executor_config)
#def some_task():
 #   ...
#task = PythonOperator(
#    task_id="another_task",
#    python_callable=my_fun,
#    executor_config=my_executor_config
#)
```

Note that KubernetesExecutor task Pods are limited to the `LimitRanges` and `quotas` defined within the pod namespace.