Manage Platform resources

Configure component size limits

Astro Private Cloud allows you to customize the minimum and maximum sizes of most Astronomer platform and Airflow components.

You can configure the CPU and memory resources of Airflow components through the Astro UI or with the Houston API.

Configure Deployment-level limits for individual Pod sizes

You can use astronomer.houston.config.deployments.maxPodCapacity to configure the maximum size any individual pod can be.

1astronomer:
2  houston:
3    config:
4      deployments:
5        maxPodCapacity:
6          cpu: 3500
7          memory: 13440

Configure Deployment-level limits for resource usage

Astro Private Cloud limits the amount of resources that can be used by all pods in a Deployment by creating and managing a LimitRange and ResourceQuota for the namespace associated with each Deployment.

These values are automatically adjusted to account for the resource requirements of various components.

You can add additional resources, beyond the standard amount allocated based on the resource-requirements of standing components, to the LimitRange and ResourceQuota. Add resources by configuring astronomer.houston.config.deployments.maxExtraCapacity to account for the requirements of KubernetesExecutor and KubernetesPodOperator tasks.

1astronomer:
2  houston:
3    config:
4      deployments:
5        maxExtraCapacity:
6          cpu: 40000
7          memory: 153600

Configure the sizes of individual Deployment-level components

Components represent different parts of the Astro Private Cloud Deployment. You can customize the default configuration for a component by defining it in astronomer.houston.config.deployments.components.

A list of configurable components and options is provided in Configurable Components.

KubernetesExecutor Task pod sizes are created on an as-needed basis, and don’t have persisting resource requirements. Their resource requirements are configured at the task level.

When defining components, you must include the full definition of the component in the list entry after the components key, instead of only the components you want to define.

For example, to increase the maximum size a Celery worker task from 3 Vcpu/11.5Gi to 3 Vcpu/192Gi, add the equivalent (in milli vCPU and Mi) full Celery worker component definition to astronomer.houston.config.deployments.components in your values.yaml with a higher limit:

When increasing CPU or memory limits, ensure the maximum pod size is large enough to avoid errors during pod creation.
1astronomer:
2  houston:
3    config:
4      deployments:
5        components:
6        - name: workers
7          resources:
8            default:
9              cpu: 1000
10              memory: 3840
11            minimum:
12              cpu: 100
13              memory: 384
14            limit:
15              cpu: 3000
16              memory: 11520
17            KubernetesExecutor:
18              default:
19                cpu: 100
20                memory: 384
21              minimum:
22                cpu: 100
23                memory: 384
24              limit:
25                cpu: 3000
26                memory: 11520
27          extra:
28            - name: terminationGracePeriodSeconds
29              default: 600
30              minimum: 0
31              limit: 36000
32            - name: replicas
33              default: 1
34              minimum: 1
35              limit: 10
36        # any additional component configurations go here
37        # - name: another-component
38        #   resources:
39        #     default: 10
40        #     ...

Configurable Components

When defining components, you must include the full definition of the component in the list entry after the components key, instead of only the components you want to define.
KubernetesExecutor task pod sizes are created on an as-needed basis and don’t have persisting resource requirements. Their resource requirements are configured at the task level.

Configurable components include:

Airflow Scheduler

1- name: scheduler
2  resources:
3    default:
4     cpu: 500
5     memory: 1920
6    minimum:
7     cpu: 500
8     memory: 1920
9    limit:
10     cpu: 3000
11     memory: 11520
12  extra:
13    - name: replicas
14      default: 1
15      minimum: 1
16      limit: 4

Airflow Webserver

1- name: webserver
2  resources:
3    default:
4     cpu: 500
5     memory: 1920
6    minimum:
7     cpu: 500
8     memory: 1920
9    limit:
10     cpu: 3000
11     memory: 11520

Airflow Apiserver (Airflow 3.0 and above)

1- name: apiServer
2  resources:
3    default:
4     cpu: 1000
5     memory: 3840
6    minimum:
7     cpu: 1000
8     memory: 3840
9    limit:
10     cpu: 3000
11     memory: 11520
12   extra:
13    - name: replicas
14      default: 1
15      minimum: 1
16      limit: 4
17
18#### StatsD
19
20```yaml
21- name: statsd
22  resources:
23    default:
24     cpu: 200
25     memory: 768
26    minimum:
27     cpu: 200
28     memory: 768
29    limit:
30     cpu: 3000
31     memory: 11520

Database Connection Pooler (PgBouncer)

1- name: pgbouncer
2  resources:
3    default:
4     cpu: 200
5     memory: 768
6    minimum:
7     cpu: 200
8     memory: 768
9    limit:
10     cpu: 200
11     memory: 768

Celery Diagnostic Web Interface (Flower)

1- name: flower
2  resources:
3    default:
4     cpu: 200
5     memory: 768
6    minimum:
7     cpu: 200
8     memory: 768
9    limit:
10     cpu: 200
11     memory: 768

Redis

1- name: redis
2  resources:
3    default:
4     cpu: 200
5     memory: 768
6    minimum:
7     cpu: 200
8     memory: 768
9    limit:
10     cpu: 200
11     memory: 768

Celery Workers

1- name: workers
2  resources:
3    default:
4     cpu: 1000
5     memory: 3840
6    minimum:
7     cpu: 100
8     memory: 384
9    limit:
10     cpu: 3000
11     memory: 11520
12  extra:
13    - name: terminationGracePeriodSeconds
14      default: 600
15      minimum: 0
16      limit: 36000
17    - name: replicas
18      default: 1
19      minimum: 1
20      limit: 10

Triggerer

1- name: triggerer
2  resources:
3    default:
4     cpu: 500
5     memory: 1920
6    minimum:
7     cpu: 500
8     memory: 1920
9    limit:
10     cpu: 3000
11     memory: 11520
12  extra:
13    - name: replicas
14      default: 1
15      minimum: 0
16      limit: 2
17
18#### Dag Processor
19
20```yaml
21- name: dagProcessor
22  resources:
23    default:
24     cpu: 500
25     memory: 1920
26    minimum:
27     cpu: 500
28     memory: 1920
29    limit:
30     cpu: 3000
31     memory: 11520
32  extra:
33    - name: replicas
34      default: 0
35      minimum: 0
36      limit: 3
37##  Manage Kubernetes worker CPU and memory with global platform config [#disable-kubernetes-resources-ui-api]
38
39You can use `workers.resources.enabled` for the `KubernetesExecutor` to manage worker Pod CPU and memory allocations with a platform configuration. When set to `false`, Astro Private Cloud ignores worker Pod CPU or memory allocations set in the API or UI. This allows you to control Pod resources with your own Kubernetes policies or admission controllers.
40
41### API/UI manages Pod resources (Default KubernetesExecutor configuration)
42
43To configure the default behavior where the API and UI manage worker Pod CPU and memory resources for the KubernetesExecutor, use the following configuration:
44
45```yaml
46deployments:
47executors:
48 - name: KubernetesExecutor
49   enabled: true
50   components:
51     - scheduler
52     - webserver
53     - apiServer
54     - statsd
55     - pgbouncer
56     - triggerer
57     - dagProcessor
58   defaultExtraCapacity:
59     cpu: 1000
60     memory: 3840
61   workers:
62     ephemeralStorage:
63       disabled: true
64     resources:
65       enabled: true

Disable API/UI resource configuration

If you configure your KubernetesExecutor so that CPU and memory requests and limits are set outside of Astro Private Cloud, you must disable resource configuration set with the Astro Private Cloud API/UI. To disable the Astro Private Cloud UI/API configuration, add the resources.enabled: false flag to your values.yaml file. This ensures that Astro Private Cloud applies the worker resource settings exclusively from your preferred source, such as Pod mutation hooks or Pod configuration files. The following configuration sets them as an empty dict.

1deployments:
2  executors:
3    - name: KubernetesExecutor
4      enabled: true
5      components:
6        - scheduler
7        - webserver
8        - apiServer
9        - statsd
10        - pgbouncer
11        - triggerer
12        - dagProcessor
13      defaultExtraCapacity:
14        cpu: 1000
15        memory: 3840
16      workers:
17        ephemeralStorage:
18          disabled: true
19        resources:
20          enabled: false

Existing deployments are unchanged unless you update this setting.

Configure the size of KubernetesExecutor task pods

Kubernetes Executor task pods are defined at the task level when the DAG passes resource requests as part of executor_config into the Operator. When not defined, these tasks default to using 0.1 Vcpu/384 Mi of memory. This means that when you define resource requests or limits for CPU and memory, ensure the maximum pod size is large enough to avoid errors during pod creation.

Astro Private Cloud does not automatically raise the namespace-level cumulative resource limits for pods created by the KubernetesExecutor. To avoid pod creation failures, increase the maxExtraCapacity to support your desired level of resourcing and concurrency.

The following example demonstrates how to configure resource limits and requests:

# import kubernetes.client.models as k8s
from kubernetes.client import models as k8s
# define an executor_config with the desired resources
my_executor_config={
  "pod_override": k8s.V1Pod(
      spec=k8s.V1PodSpec(
          containers=[
              k8s.V1Container(
                  name="base",
                  resources=k8s.V1ResourceRequirements(
                      requests={
                          "cpu": "50m",
                          "memory": "384Mi"
                      },
                      limits={
                          "cpu": "1000m",
                          "memory": "1024Mi"
                      }
                  )
              )
          ]
      )
  )
}
# pass in executor_config=my_executor_config to any Operator
#@task(executor_config=my_executor_config)
#def some_task():
 #   ...
#task = PythonOperator(
#    task_id="another_task",
#    python_callable=my_fun,
#    executor_config=my_executor_config
#)

Note that KubernetesExecutor task Pods are limited to the LimitRanges and quotas defined within the pod namespace.