> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure component size limits

Astronomer Software allows you to customize the minimum and maximum sizes of most Astronomer platform and Airflow components.

<a id="what-is-an-au" />

### Astro Units (AU's)

You can increase or decrease resources on Astro by units of size know as AU's (Astro Units). Adding or removing an AstroUnit changes a Deployment's size by changing the following:

* the size of a component by one-tenth of a v-core
* the size of the component by 384Mi of memory
* the limit of connections to the Postgres server by half of a connection
* the limit of connections used internally by Airflow to the database connection pooler by 5
* the size of the deployment-namespace's overall cumulative limits (ResourceQuota and LimitRange) by one-tenth of a v-core and 834Mi of memory.
* the limit of pods in the namespace by one pod (ResourceQuota)

<a id="configure-max-pod-size" />

### Configure Deployment-level limits for individual pod sizes

You can use `astronomer.houston.config.deployments.maxPodAu` to configure the maximum size any individual pod can be.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        maxPodAu: 35 # default is 35
```

<a id="configure-cumulative-resource-limits" />

### Configure Deployment-level limits for resource usage

Astronomer Software limits the amount of resources that can be used by all pods in a Deployment by creating and managing a `LimitRange` and `ResourceQuota` for the namespace associated with each Deployment.

These values are automatically adjust to account for the resource requirements of various components.

You can add additional resources, beyond the standard amount allocated based on the resource-requirements of standing components, to the `LimitRange` and `ResourceQuota`. Add resources by configuring `astronomer.houston.config.deployments.maxExtraAu` to account for the requirements of KubernetesExecutor and `KubernetesPodOperator` tasks.

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        maxExtraAu: 400 # default is 400
```

<a id="configure-deployment-components" />

### Configure the sizes of individual Deployment-level components

Components represent different parts of the Astronomer Software Deployment. You can customize the default configuration for a component by defining it in `astronomer.houston.config.deployments.components`.

A list of configurable components and options is provided in [Configurable Components](#configurable-components).

<Tip>KubernetesExecutor Task pod sizes are created on an as-needed basis, and don't have persisting resource requirements. Their resource requirements are [configured at the task level](#configure-kubernetes-task-pod-size).</Tip>

When defining components, you must include the full definition of the component in the list entry after the components key, instead of only the components you want to define.

For example, to increase the maximum size a Celery worker task from 30 AU (3 Vcpu/11.5Gi) to 50AU (3 Vcpu/192Gi), add the full Celery worker component definition to `astronomer.houston.config.deployments.components` in your `values.yaml` with a higher limit (defined in AUs):

<Tip>When increasing CPU or memory limits, ensure the [maximum pod size](#configure-max-pod-size) is large enough to avoid errors during pod creation.</Tip>

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        components:
        - name: workers
          au:
            default: 10
            minimum: 1
            limit: 50 # default was 30
          extra:
            - name: terminationGracePeriodSeconds
              default: 600
              minimum: 0
              limit: 36000
            - name: replicas
              default: 1
              minimum: 1
              limit: 10
        # any additional component configurations go here
        # - name: another-component
        #   au:
        #     default: 10
        #     ...
```

### Configurable Components

<Info>When [defining components](#configure-deployment-components), you must include the full definition of the component in the list entry after the components key, instead of only the components you want to define.</Info>

<Tip>KubernetesExecutor task pod sizes are created on an as-needed basis and don't have persisting resource requirements. Their resource requirements are [configured at the task level](#configure-kubernetes-task-pod-size).</Tip>

Configurable components include:

#### Airflow Scheduler

```yaml wrap theme={null}
- name: scheduler
  au:
    default: 5
    minimum: 5
    limit: 30
  extra:
    - name: replicas
      default: 1
      minimum: 1
      limit: 4
      minAirflowVersion: 2.0.0
```

#### Airflow Webserver

```yaml wrap theme={null}
- name: webserver
  au:
    default: 5
    minimum: 5
    limit: 30
```

#### StatsD

```yaml wrap theme={null}
- name: statsd
  au:
    default: 2
    minimum: 2
    limit: 30
```

#### Database Connection Pooler (PgBouncer)

```yaml wrap theme={null}
- name: pgbouncer
  au:
    default: 2
    minimum: 2
    limit: 2
```

#### Celery Diagnostic Web Interface (Flower)

```yaml wrap theme={null}
- name: flower
  au:
    default: 2
    minimum: 2
    limit: 2
```

#### Redis

```yaml wrap theme={null}
- name: redis
  au:
    default: 2
    minimum: 2
    limit: 2
```

#### Celery Workers

```yaml wrap theme={null}
- name: workers
  au:
    default: 10
    minimum: 1
    limit: 30
  extra:
    - name: terminationGracePeriodSeconds
      default: 600
      minimum: 0
      limit: 36000
    - name: replicas
      default: 1
      minimum: 1
      limit: 10
```

#### Triggerer

```yaml wrap theme={null}
- name: triggerer
  au:
    default: 5
    minimum: 5
    limit: 30
  extra:
    - name: replicas
      default: 1
      minimum: 0
      limit: 2
      minAirflowVersion: 2.2.0
```

<a id="configure-kubernetes-task-pod-size" />

### Configure the size of KubernetesExecutor task pods

Kubernetes Executor task pods are defined at the task level when the DAG passes resource requests as part of `executor_config` into the Operator. When not defined, these tasks default to using 1 AU, which is one-tenth of a Vcpu/384 Mi of memory. This means that when you define resource requests or limits for CPU and memory, ensure the [maximum pod size](#configure-max-pod-size) is large enough to avoid errors during pod creation.

<Danger>Astronomer Software does not automatically raise the namespace-level cumulative resource limits for pods created by the KubernetesExecutor. To avoid pod creation failures, increase the `maxExtraAu` to support your desired level of resourcing and concurrency.</Danger>

The following example demonstrates how to configure resource limits and requests:

```expandable wrap theme={null}
# import kubernetes.client.models as k8s
from kubernetes.client import models as k8s

# define an executor_config with the desired resources
my_executor_config={
  "pod_override": k8s.V1Pod(
      spec=k8s.V1PodSpec(
          containers=[
              k8s.V1Container(
                  name="base",
                  resources=k8s.V1ResourceRequirements(
                      requests={
                          "cpu": "50m",
                          "memory": "384Mi"
                      },
                      limits={
                          "cpu": "1000m",
                          "memory": "1024Mi"
                      }
                  )
              )
          ]
      )
  )
}

# pass in executor_config=my_executor_config to any Operator

#@task(executor_config=my_executor_config)
#def some_task():
 #   ...

#task = PythonOperator(
#    task_id="another_task",
#    python_callable=my_fun,
#    executor_config=my_executor_config
#)

```

Note that KubernetesExecutor task pods are limited to the `LimitRanges` and `quotas` defined within the pod namespace.

<a id="customize-au" />

### Customize Astro Units (AU's)

<Danger>Consult with your Astronomer Resident Architect before changing the amount of CPU or memory granted by an AU.</Danger>

Configurable options include:

* `astronomer.houston.config.deployments.astroUnit.cpu` - the amount an AU contributes to the size of a component, in thousandths of a vCPU, and the size of the `LimitRange` or `ResourceQuota`, defaults to `100`.
* `astronomer.houston.config.deployments.astroUnit.memory` - the amount an AU contributes to the size of a component, in Mi, and the size of the `LimitRange` or `ResourceQuota`, set to `384` by default.
* `astronomer.houston.config.deployments.astroUnit.pods` - the amount an AU contributes to the maximum amount of pods permitted by `ResourceQuota`, set to `1` by default.
* `astronomer.houston.config.deployments.astroUnit.actualConns` - the amount an AU contributes to the limit of connections to the Postgres server, set to `0.5` by default.
* `astronomer.houston.config.deployments.astroUnit.airflowConns` - The amount an AU contributes to the limit of connections used internally by Airflow to the database connection pooler, set to `5` by default.

The following code example shows how to set the resources per individual AU:

```yaml wrap theme={null}
astronomer:
  houston:
    config:
      deployments:
        astroUnit:
          # values are listed with their default values
          cpu: 100
          memory: 384
          pods: 1
          actualConns: 0.5 # upstream connections to postgres
          airflowConns: 5  # downstream connections to airflow
```