> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Configure resources for Airflow components on Astro Private Cloud Deployments
Use this document to configure resource usage for a Deployment's executor, webserver/apiserver, scheduler and triggerer components.
## Select an executor
The Airflow [executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/index.html) works closely with the Airflow scheduler to determine what resources complete tasks as they queue. The main difference between executors is their available resources and how they utilize those resources to distribute work.
Astro Private Cloud supports three executors:
* [Local executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/local.html)
* [Celery executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/celery.html)
* [Kubernetes executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html)
Though it largely depends on your use case, Astronomer recommends the Local executor for development environments and the Celery or Kubernetes executors for production environments operating at scale.
For a detailed description of each executor, see [Airflow executors explained](https://docs.astronomer.io/learn/airflow-executors-explained).
## Scale core resources
Apache Airflow requires four primary components:
* The Webserver or API server (Airflow 3)
* The Scheduler
* The Executor (and the workers it runs)
* The Triggerer
To scale these resources, adjust the corresponding slider in the Astro Private Cloud UI to increase its available resources.
Read the following sections to help you determine which core resources to scale and when.
### Airflow Webserver and API server
In Airflow 2, the **webserver** component renders the [Airflow UI](https://airflow.apache.org/docs/apache-airflow/stable/ui.html), providing access to Dag monitoring, task logs, and configuration.
Starting with Airflow 3, the **apiserver** replaces the webserver for serving the Airflow UI.
If loading pages or functions in the Airflow UI is slow or unresponsive, increase the resources allocated to the webserver (Airflow 2) or apiserver (Airflow 3).
### Scheduler
The [Airflow scheduler](https://airflow.apache.org/docs/apache-airflow/stable/scheduler.html) is responsible for monitoring task execution and triggering downstream tasks once dependencies have been met.
If you experience delays in task execution, which you can track via the [Gantt Chart](https://airflow.apache.org/docs/apache-airflow/stable/ui.html#gantt-chart) view of the Airflow UI, Astronomer recommends increasing the resources allocated towards the scheduler.
To set alerts that notify you via email when your Airflow scheduler is underprovisioned, configure an [Airflow alert](/docs/astro-private-cloud/v-0-37/airflow-alerts).
#### Scheduler count
Airflow 2.0 comes with the ability for users to run multiple schedulers concurrently to ensure high-availability, zero recovery time, and faster performance. You can provision up to 4 schedulers on any Deployment.
Each individual scheduler will be provisioned with the resources specified in **Scheduler Resources**. For example, if you set the CPU figure in **Scheduler Resources** to 5 CPUs and set **Scheduler Count** to 2, your Airflow Deployment will run with 2 Airflow schedulers using 5 CPUs each for a total of 10 CPUs.
To increase the speed at which tasks are scheduled and ensure high-availability, Astronomer recommends provisioning 2 or more Airflow schedulers for production environments.
#### Dag Processor
Complex, dynamically-generated Dags, sub-optimal Dag parsing practices, or a growing business that requires a larger data pipeline can strain Dag processing and threaten your Airflow scheduler's availability. Deployments can support high-scale environments more reliably by separating the Dag processor from the scheduler. You can now configure the number of Dag processors for the Deployment from the UI and the Houston API.
If you want to enable and provision resources for standalone Dag processors, you can set the `dagProcessorEnabled` feature flag to `true` in your Houston API configuration in the `config.yaml` file:
```yaml wrap theme={null}
astronomer:
houston:
config:
deployments:
dagProcessorEnabled: true
```
By adjusting the **Dag Processor** slider in the Astro Private Cloud UI, you can provision up to 3 Dag Processors on any Deployment. You can also configure CPU and Memory with the **Dag Processor Resources** sliders.
Airflow 3 requires the Dag Processor component to be enabled. Set `dagProcessorEnabled: true` in your Deployment configuration. If this flag is disabled, Airflow 3 Deployments will fail to start or process Dags correctly.
### Triggerer
Airflow 2.2 introduces the triggerer, which is a component for running tasks with [deferrable operators](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html). Like the scheduler, the triggerer is highly-available: If a triggerer shuts down unexpectedly, the tasks it was deferring can be recovered and moved to another triggerer.
By adjusting the **Triggerer** slider in the Astro Private Cloud UI, you can provision up to 2 triggerers on any Deployment running Airflow 2.2+. To take advantage of the Triggerer's high availability, we recommend provisioning 2 triggerers for production Deployments.
### (Kubernetes executor only) Set extra capacity
On Astronomer, resources required for the [`KubernetesPodOperator`](/docs/astro-private-cloud/v-1-x/kube-pod-operator) or the [Kubernetes Executor](/docs/astro-private-cloud/v-1-x/kubernetes-executor) are set as **Extra Capacity**.
The Kubernetes executor and `KubernetesPodOperator` each spin up an individual Kubernetes pod for each task that needs to be executed, then spin down the pod after that task is completed.
The amount of CPU and Memory allocated to **Extra Capacity** maps to [resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) on the [Kubernetes Namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) in which your Airflow Deployment lives on Astro Private Cloud. More specifically, **Extra Capacity** represents the maximum possible resources that could be provisioned to a pod at any given time.
Resources allocated to **Extra Capacity** do not affect scheduler or webserver/apiserver performance and do not represent actual usage.
### (Celery executor only) Configure workers
To optimize for flexibility and availability, the Celery executor works with a set of independent Celery workers across that it can delegate tasks. On Astro Private Cloud, you can configure your Celery workers to fit your use case.
#### Worker count
By adjusting the **Worker Count** slider, users can provision up to 20 Celery workers on any Airflow Deployment. If you would like users to have the ability to set more than 20 workers on any Deployment, you can change this limit by applying a [config change](/docs/astro-private-cloud/v-1-x/apply-platform-config).
Each individual worker will be provisioned with the resources specified in **Worker Resources**. If you set the CPU figure in **Worker Resources** to 5 CPUs and set **Worker Count** to 3, for example, your Airflow Deployment will run with 3 Celery workers using 5 CPUs each for a total of 15 CPUs.
#### Worker termination grace period
On Astro Private Cloud, Celery workers restart after every code deploy to your Airflow Deployment. This makes sure that workers execute with the most up-to-date code. To minimize disruption during task execution, however, APC supports the ability to set a **Worker Termination Grace Period**.
If a deploy is triggered while a Celery worker is executing a task and **Worker Termination Grace Period** is set, the worker will continue to process that task up to a certain number of minutes before restarting itself. By default, the grace period is ten minutes.