Customize Deployment CPU and management resources per component

When you create a new Astronomer Deployment, you can specify the exact amount of CPU and memory that power its core components using the Custom Resources resource strategy.

Compared to using Astronomer Units (AU), which represent a fixed amount of CPU and memory, specifying custom resources gives you greater flexibility to define how your Deployments run. For example, you might need to allocate significantly more memory than CPU to your worker Pods if you need to run memory-intensive tasks, but at the same time you need more memory than CPU in your scheduler. In this scenario, using AUs isn’t sufficient because each component needs a different CPU to memory ratio.

When a Deployment uses the Custom Resources resource strategy, Deployment Admins can specify the exact amount of CPU and memory they want each component to have without any scaling limitations. The resources you specify are used both as the limit and request values for the Pods running your components.

Set custom resource usage in the Software UI

To switch from using AU to using custom resource specifications:

In the Software UI, open your Deployment.
In the Settings tab, change your Resource Strategy to Custom Resources.
For each Software component, adjust the CPU and memory sliders to match your resource requirements. You can change the unit of the resource by clicking the dropdown menu that shows your current unit.

When setting CPU resources for the Scheduler or Workers using milliCPU (mCPU) values, the values must be multiples of 100 (such as 100, 200, 300, etc). Non-multiples of 100 may cause Deployment failures.

To create a Deployment through the Houston API that uses the custom resource strategy be default, set astroUnitsEnabled: false in your Deployment creation mutation.

Disable Astronomer Units (AUs) from Deployment resource configurations

To limit users to only configure Deployments using custom resources, set the following configuration in your values.yaml file:

1 astronomer:
2   houston:
3     config:
4       deployments:
5         resourceProvisioningStrategy:
6           astroUnitsEnabled: false

Then, save this configuration and push it to your platform. See Apply a Platform Config Change.

Set custom resources with Houston API

You can also switch from using AUs to custom resources specifications with the Houston API.

Use Custom Resource Defaults

You can set custom defaults, limits and minimums for Airflow components and enable the use of the custom defaults with the feature flag componentsCustomDefaultsEnabled.

To enforce using custom resources only, you must enable custom defaults and disable Astro Units.

1 astronomer:
2   houston:
3     config:
4       componentsCustomDefaultsEnabled: true
5       resourceProvisioningStrategy:
6         astroUnitsEnabled: false

When componentsCustomDefaultsEnabled is set to false and Astro Units are enabled, users must use the values defined in the config file for the AU defaults. This means when you use the following code example to define a component, Deployments are created respecting the AU resource definitions and the UI uses these defaults.

1 astronomer:
2   houston:
3     config:
4       componentsCustomDefaultsEnabled: false
5       resourceProvisioningStrategy:
6         astroUnitsEnabled: true
7       components:
8         - name: scheduler
9           au:
10             default: 5
11             limit: 50
12             minimum: 1
13           extra:
14             - name: replicas
15               default: 1
16               minimum: 1
17               limit: 4
18               minAirflowVersion: "2.0.0"
19 
20 When `componentsCustomDefaultsEnabled` is set to `true` and Astro Units is enabled or disabled, users must use the values defined in the config file for the custom defaults. This means when you use the following code example to define a component, Deployments are created respecting the custom definitions and the UI uses these defaults.
21 
22 ```yaml
23 astronomer:
24   houston:
25     config:
26       componentsCustomDefaultsEnabled: true
27       resourceProvisioningStrategy:
28         astroUnitsEnabled: true
29       components:
30         - name: scheduler
31           custom:
32             default:
33               cpu: 1000
34               memory: 2000
35             minimum:
36               cpu: 500
37               memory: 1000
38             limit:
39               cpu: 6000
40               memory: 12000
41           extra:
42             - name: replicas
43               default: 1
44               minimum: 1
45               limit: 4
46               minAirflowVersion: "2.0.0"
47 
48 ## Set resource quotas per deployment
49 Astronomer Software lets you override default quota calculations by specifying resource quotas directly in the Houston API’s Deployment payload. Use the `quotas` object to set custom CPU and memory requests/limits for your deployment. If you do not provide `quotas`, Astronomer uses the platform’s default quota logic.
50 
51 ### Default quotas config
52 
53 Include the requests and limits in your `quotas` parameter of the `deployment create` or `update` API payload. If you set quotas, make sure your values are not less than required platform minimums and do not exceed the allowed platform maximums.
54 
55 The following example shows the `quotas` object and the platform defaults for CPU and memory requests and limits.
56 
57 ```json
58 "quotas": {
59   "requests": {
60     "cpu": 1,
61     "memory": "1920Mi"
62   },
63   "limits": {
64     "cpu": 2,
65     "memory": "1920Mi"
66   }
67 }

Configuration options

Use custom quotas if you need to guarantee or constrain Deployment resources beyond the system-determined logic. If you do not set quotas, the deployment will use platform default resource constraints.

Deployments will fail due to insufficient quotas if you set resource quotas to less than or greater than the Astronomer platform-provided minimum or maximum limits.

Typical platform defaults:

CPU: 10 vCPU
Memory: 28272Mi (~28Gi)

Configuration Name	Component	Description	Default Value (if not set)	Accepted Values
`quotas`	Deployment	Optional JSON object with custom resource quotas	Not set–platform logic is used	JSON object (`requests` and `limits`)
`quotas.requests.cpu`	Deployment	CPU quota guaranteed (requested)	Platform default	Number (integer or float)
`quotas.requests.memory`	Deployment	Memory quota guaranteed (requested)	Platform default	String, (`"1920Mi"`, `"2Gi"`)
`quotas.limits.cpu`	Deployment	CPU quota maximum (limit)	Platform default	Number (integer or float)
`quotas.limits.memory`	Deployment	Memory quota maximum (limit)	Platform default	String, ( `"1920Mi"`, `"28272Mi"`)

Configure Deployment-level limits for resource usage

Astronomer Software limits the amount of resources that can be used by all Pods in a Deployment by creating and managing a LimitRange and ResourceQuota for the namespace associated with each Deployment.

These values are automatically adjust to account for the resource requirements of various components.

You can add additional resources, beyond the standard amount allocated based on the resource-requirements of standing components, to the LimitRange and ResourceQuota. Add resources by configuring astronomer.houston.config.deployments.maxExtraCapacity and astronomer.houston.config.deployments.maxExtraPodCapacity to account for the requirements of KubernetesExecutor and KubernetesPodOperator tasks.

1 astronomer:
2   houston:
3     config:
4       deployments:
5         maxExtraCapacity:
6           cpu: 40000 # in milliCPUs (m)
7           memory: 153600 # in MiB (Mi)
8         maxPodCapacity:
9           cpu: 3500 # in milliCPUs (m)
10           memory: 13440 # in MiB (Mi)

Configurable Components

KubernetesExecutor task Pod sizes are created on an as-needed basis and don’t have persisting resource requirements. Their resource requirements are configured at the task level.

Configurable components include:

Airflow Scheduler

1 - name: scheduler
2   au:
3     default: 5
4     limit: 50
5     minimum: 1
6   custom:
7     default:
8       cpu: 1000
9       memory: 2000
10     minimum:
11       cpu: 500
12       memory: 1000
13     limit:
14       cpu: 6000
15       memory: 12000
16   extra:
17     - name: replicas
18       default: 1
19       minimum: 1
20       limit: 4
21       minAirflowVersion: "2.0.0"

Airflow DAG processor

To enable a standalone DAG processor, set the dagProcessorEnabled feature flag to true in your Houston API configuration in the config.yaml file:

1 astronomer:
2   houston:
3     config:
4       deployments:
5         dagProcessorEnabled: true

You can configure extra containers for the DAG processor in the values.yaml file. For example:

1 houston:
2   config:
3     deployments:
4       helm:
5         airflow:
6           dagProcessor:
7             extraContainers:
8               - name: <extra-container>
9                 env:
10                   - name: DAGS_LOCATION
11                     value: /dags
12                   - name: SECRET_LOCATION
13                     value: /secrets
14                   - name: SYNC_INTERVAL
15                     value: "15"
16                 image: ubuntu:latest
17                 command: ["/bin/bash", "-c", "--"]
18                 args: ["<arg-1>; <arg-2>; <arg-3>;"]

1 - name: dagProcessor
2   custom:
3     default:
4       cpu: 1000
5       memory: 3840
6     minimum:
7       cpu: 1000
8       memory: 3840
9     limit:
10       cpu: 3000
11       memory: 11520
12   extra:
13     - name: replicas
14       default: 0
15       minimum: 0
16       limit: 3

Airflow Webserver

1 - name: webserver
2  au:
3    default: 5
4    limit: 50
5    minimum: 1
6  custom:
7    default:
8      cpu: 1000
9      memory: 2000
10    minimum:
11      cpu: 500
12      memory: 1000
13    limit:
14      cpu: 6000
15      memory: 12000

StatsD

1 - name: statsd
2   au:
3     default: 2
4     minimum: 2
5     limit: 30
6   custom:
7     default:
8       cpu: 200
9       memory: 768
10     minimum:
11       cpu: 200
12       memory: 768
13     limit:
14       cpu: 3000
15       memory: 11520

Database Connection Pooler (PgBouncer)

1 - name: pgbouncer
2   au:
3     default: 2
4     minimum: 2
5     limit: 2
6   custom:
7     default:
8       cpu: 200
9       memory: 768
10     minimum:
11       cpu: 200
12       memory: 768
13     limit:
14       cpu: 200
15       memory: 768

Celery Diagnostic Web Interface (Flower)

1 - name: flower
2   au:
3     default: 2
4     minimum: 2
5     limit: 2
6   custom:
7     default:
8       cpu: 200
9       memory: 768
10     minimum:
11       cpu: 200
12       memory: 768
13     limit:
14       cpu: 200
15       memory: 768

Redis

1 - name: redis
2   au:
3     default: 2
4     minimum: 2
5     limit: 2
6  custom:
7    default:
8      cpu: 200
9      memory: 768
10    minimum:
11      cpu: 200
12      memory: 768
13    limit:
14      cpu: 200
15      memory: 768

Celery Workers

1 - name: workers
2  au:
3    default: 10
4    minimum: 1
5    limit: 30
6  custom:
7    default:
8      cpu: 1000
9      memory: 3840
10    minimum:
11      cpu: 100
12      memory: 384
13    limit:
14      cpu: 3000
15      memory: 11520
16  extra:
17    - name: terminationGracePeriodSeconds
18      default: 600
19      minimum: 0
20      limit: 36000
21    - name: replicas
22      default: 1
23      minimum: 1
24      limit: 20

Triggerer

1 - name: triggerer
2   au:
3     default: 5
4     limit: 50
5     minimum: 1
6   custom:
7     default:
8       cpu: 1000
9       memory: 2000
10     minimum:
11       cpu: 500
12       memory: 1000
13     limit:
14       cpu: 6000
15       memory: 12000
16   extra:
17     - name: replicas
18       default: 1
19       minimum: 0
20       limit: 4
21       minAirflowVersion: "2.2.0"