Skip to main content
Version: 0.37

Customize Deployment CPU and management resources per component

When you create a new Astronomer Deployment, you can specify the exact amount of CPU and memory that power its core components using the Custom Resources resource strategy.

Compared to using Astronomer Units (AU), which represent a fixed amount of CPU and memory, specifying custom resources gives you greater flexibility to define how your Deployments run. For example, you might need to allocate significantly more memory than CPU to your worker Pods if you need to run memory-intensive tasks, but at the same time you need more memory than CPU in your scheduler. In this scenario, using AUs isn't sufficient because each component needs a different CPU to memory ratio.

When a Deployment uses the Custom Resources resource strategy, Deployment Admins can specify the exact amount of CPU and memory they want each component to have without any scaling limitations. The resources you specify are used both as the limit and request values for the Pods running your components.

Set custom resource usage in the Software UI​

To switch from using AU to using custom resource specifications:

  1. In the Software UI, open your Deployment.
  2. In the Settings tab, change your Resource Strategy to Custom Resources.
  3. For each Software component, adjust the CPU and memory sliders to match your resource requirements. You can change the unit of the resource by clicking the dropdown menu that shows your current unit.
info

To create a Deployment through the Houston API that uses the custom resource strategy be default, set astroUnitsEnabled: false in your Deployment creation mutation.

Disable Astronomer Units (AUs) from Deployment resource configurations​

To limit users to only configure Deployments using custom resources, set the following configuration in your values.yaml file:

astronomer:
  houston:
    config:
      deployments:
        resourceProvisioningStrategy:
          astroUnitsEnabled: false

Then, save this configuration and push it to your platform. See Apply a Platform Config Change.

Set custom resources with Houston API​

You can also switch from using AUs to custom resources specifications with the Houston API.

Use Custom Resource Defaults​

You can set custom defaults, limits and minimums for Airflow components and enable the use of the custom defaults with the feature flag componentsCustomDefaultsEnabled.

To enforce using custom resources only, you must enable custom defaults and disable Astro Units.

astronomer:
  houston:
    config:
      componentsCustomDefaultsEnabled: true
      resourceProvisioningStrategy:
        astroUnitsEnabled: false

When componentsCustomDefaultsEnabled is set to false and Astro Units are enabled, users must use the values defined in the config file for the AU defaults. This means when you use the following code example to define a component, Deployments are created respecting the AU resource definitions and the UI uses these defaults.

astronomer:
  houston:
    config:
      componentsCustomDefaultsEnabled: false
      resourceProvisioningStrategy:
        astroUnitsEnabled: true
      components:
        - name: scheduler
          au:
            default: 5
            limit: 50
            minimum: 1
          extra:
            - name: replicas
              default: 1
              minimum: 1
              limit: 4
              minAirflowVersion: "2.0.0"

When `componentsCustomDefaultsEnabled` is set to `true` and Astro Units is enabled or disabled, users must use the values defined in the config file for the custom defaults. This means when you use the following code example to define a component, Deployments are created respecting the custom definitions and the UI uses these defaults.

```yaml
astronomer:
  houston:
    config:
      componentsCustomDefaultsEnabled: true
      resourceProvisioningStrategy:
        astroUnitsEnabled: true
      components:
        - name: scheduler
          custom:
            default:
              cpu: 1000
              memory: 2000
            minimum:
              cpu: 500
              memory: 1000
            limit:
              cpu: 6000
              memory: 12000
          extra:
            - name: replicas
              default: 1
              minimum: 1
              limit: 4
              minAirflowVersion: "2.0.0"

## Set resource quotas per deployment

Astronomer Software lets you override default quota calculations by specifying resource quotas directly in the Houston API’s Deployment payload. Use the `quotas` object to set custom CPU and memory requests/limits for your deployment. If you do not provide `quotas`, Astronomer uses the platform’s default quota logic.

### Default quotas config

Include the requests and limits in your `quotas` parameter of the `deployment create` or `update` API payload. If you set quotas, make sure your values are not less than required platform minimums and do not exceed the allowed platform maximums.

The following example shows the `quotas` object and the platform defaults for CPU and memory requests and limits.

```json
"quotas": {
  "requests": {
    "cpu": 1,
    "memory": "1920Mi"
  },
  "limits": {
    "cpu": 2,
    "memory": "1920Mi"
  }
}

Configuration options​

Use custom quotas if you need to guarantee or constrain Deployment resources beyond the system-determined logic. If you do not set quotas, the deployment will use platform default resource constraints.

warning

Deployments will fail due to insufficient quotas if you set resource quotas to less than or greater than the Astronomer platform-provided minimum or maximum limits.

Typical platform defaults:

  • CPU: 10 vCPU
  • Memory: 28272Mi (~28Gi)
Configuration NameComponentDescriptionDefault Value (if not set)Accepted Values
quotasDeploymentOptional JSON object with custom resource quotasNot set–platform logic is usedJSON object (requests and limits)
quotas.requests.cpuDeploymentCPU quota guaranteed (requested)Platform defaultNumber (integer or float)
quotas.requests.memoryDeploymentMemory quota guaranteed (requested)Platform defaultString, ("1920Mi", "2Gi")
quotas.limits.cpuDeploymentCPU quota maximum (limit)Platform defaultNumber (integer or float)
quotas.limits.memoryDeploymentMemory quota maximum (limit)Platform defaultString, ( "1920Mi", "28272Mi")

Configure Deployment-level limits for resource usage​

Astronomer Software limits the amount of resources that can be used by all Pods in a Deployment by creating and managing a LimitRange and ResourceQuota for the namespace associated with each Deployment.

These values are automatically adjust to account for the resource requirements of various components.

You can add additional resources, beyond the standard amount allocated based on the resource-requirements of standing components, to the LimitRange and ResourceQuota. Add resources by configuring astronomer.houston.config.deployments.maxExtraCapacity and astronomer.houston.config.deployments.maxExtraPodCapacity to account for the requirements of KubernetesExecutor and KubernetesPodOperator tasks.

astronomer:
  houston:
    config:
      deployments:
        maxExtraCapacity:
          cpu: 40000 # in milliCPUs (m)
          memory: 153600 # in MiB (Mi)
        maxPodCapacity:
          cpu: 3500 # in milliCPUs (m)
          memory: 13440 # in MiB (Mi)

Configurable Components​

tip

KubernetesExecutor task Pod sizes are created on an as-needed basis and don't have persisting resource requirements. Their resource requirements are configured at the task level.

Configurable components include:

Airflow Scheduler​

- name: scheduler
  au:
    default: 5
    limit: 50
    minimum: 1
  custom:
    default:
      cpu: 1000
      memory: 2000
    minimum:
      cpu: 500
      memory: 1000
    limit:
      cpu: 6000
      memory: 12000
  extra:
    - name: replicas
      default: 1
      minimum: 1
      limit: 4
      minAirflowVersion: "2.0.0"

Airflow DAG processor​

note

To enable a standalone DAG processor, set the dagProcessorEnabled feature flag to true in your Houston API configuration in the config.yaml file:

astronomer:
  houston:
    config:
      deployments:
        dagProcessorEnabled: true
info

You can configure extra containers for the DAG processor in the values.yaml file. For example:

houston:
  config:
    deployments:
      helm:
        airflow:
          dagProcessor:
            extraContainers:
              - name: <extra-container>
                env:
                  - name: DAGS_LOCATION
                    value: /dags
                  - name: SECRET_LOCATION
                    value: /secrets
                  - name: SYNC_INTERVAL
                    value: "15"
                image: ubuntu:latest
                command: ["/bin/bash", "-c", "--"]
                args: ["<arg-1>; <arg-2>; <arg-3>;"]
- name: dagProcessor
  custom:
    default:
      cpu: 1000
      memory: 3840
    minimum:
      cpu: 1000
      memory: 3840
    limit:
      cpu: 3000
      memory: 11520
  extra:
    - name: replicas
      default: 0
      minimum: 0
      limit: 3

Airflow Webserver​

- name: webserver
 au:
   default: 5
   limit: 50
   minimum: 1
 custom:
   default:
     cpu: 1000
     memory: 2000
   minimum:
     cpu: 500
     memory: 1000
   limit:
     cpu: 6000
     memory: 12000

StatsD​

- name: statsd
  au:
    default: 2
    minimum: 2
    limit: 30
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 3000
      memory: 11520

Database Connection Pooler (PgBouncer)​

- name: pgbouncer
  au:
    default: 2
    minimum: 2
    limit: 2
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 200
      memory: 768

Celery Diagnostic Web Interface (Flower)​

- name: flower
  au:
    default: 2
    minimum: 2
    limit: 2
  custom:
    default:
      cpu: 200
      memory: 768
    minimum:
      cpu: 200
      memory: 768
    limit:
      cpu: 200
      memory: 768

Redis​

- name: redis
  au:
    default: 2
    minimum: 2
    limit: 2
 custom:
   default:
     cpu: 200
     memory: 768
   minimum:
     cpu: 200
     memory: 768
   limit:
     cpu: 200
     memory: 768

Celery Workers​

- name: workers
 au:
   default: 10
   minimum: 1
   limit: 30
 custom:
   default:
     cpu: 1000
     memory: 3840
   minimum:
     cpu: 100
     memory: 384
   limit:
     cpu: 3000
     memory: 11520
 extra:
   - name: terminationGracePeriodSeconds
     default: 600
     minimum: 0
     limit: 36000
   - name: replicas
     default: 1
     minimum: 1
     limit: 20

Triggerer​

- name: triggerer
  au:
    default: 5
    limit: 50
    minimum: 1
  custom:
    default:
      cpu: 1000
      memory: 2000
    minimum:
      cpu: 500
      memory: 1000
    limit:
      cpu: 6000
      memory: 12000
  extra:
    - name: replicas
      default: 1
      minimum: 0
      limit: 4
      minAirflowVersion: "2.2.0"

Was this page helpful?