> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Airflow system components

A Deployment in Astro Private Cloud (APC) consists of multiple components that work together to orchestrate and execute your data pipelines. Each component has a specific role and configuration options.

## Core components

### Scheduler

The scheduler is the heart of Airflow. It monitors all Dags and tasks, triggers task instances when dependencies are complete, and submits tasks to the executor for running.

Default configuration:

```yaml wrap theme={null}
scheduler:
  enabled: true
  replicas: 1
  terminationGracePeriodSeconds: 10
  livenessProbe:
    initialDelaySeconds: 10
    timeoutSeconds: 20
    failureThreshold: 5
    periodSeconds: 60
```

Key responsibilities:

* Schedule tasks based on dependencies and triggers.
* Monitor task states and handle retries.
* Manage pools and task queues.
* Parse Dag files and create Dag runs. When the Dag processor is enabled, Dag parsing moves to the Dag processor and the scheduler handles only scheduling.

### Webserver

The webserver provides the Airflow UI for monitoring Dags, viewing logs, triggering runs, and managing configurations.

Default configuration:

```yaml wrap theme={null}
webserver:
  enabled: true
  replicas: 1
  terminationGracePeriodSeconds: 30
  allowPodLogReading: true
  livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
    failureThreshold: 5
    periodSeconds: 10
```

Key features:

* Dag visualization and monitoring.
* Task log viewing.
* Variable and connection management.
* User authentication and authorization.

### Workers

Workers execute tasks. Workers run as a persistent deployment only when using Celery Executor. When using Kubernetes executor, Airflow launches ephemeral task pods instead.

Default configuration (Celery Executor):

```yaml wrap theme={null}
workers:
  enabled: true
  replicas: 1
  terminationGracePeriodSeconds: 600
```

### Triggerer

The triggerer handles deferrable operators, allowing tasks to release worker slots while waiting for external events.

Default configuration:

```yaml wrap theme={null}
triggerer:
  enabled: true
  replicas: 1
  terminationGracePeriodSeconds: 60
  livenessProbe:
    initialDelaySeconds: 10
    timeoutSeconds: 20
    failureThreshold: 5
    periodSeconds: 60
```

### Dag processor

The Dag processor parses Dag files and updates the metadata database with Dag definitions. Available in Airflow 2.3+ and mandatory in Airflow 3+.

Default configuration:

```yaml wrap theme={null}
dagProcessor:
  enabled: ~  # Auto-enabled for Airflow 3+
  replicas: 1
  terminationGracePeriodSeconds: 60
  waitForMigrations:
    enabled: true
```

<Note>
  In Airflow 3, the Dag processor is automatically enabled and required for Dag discovery.
</Note>

### API server (Airflow 3+)

The API server is a new component in Airflow 3 that provides the REST API, separated from the webserver for better scalability.

Default configuration:

```yaml wrap theme={null}
apiServer:
  enabled: true
  allowPodLogReading: true
```

<Note>
  The platform manages the number of API server replicas.
</Note>

## Supporting components

### Redis

Message broker for Celery Executor. Handles task queue communication between scheduler and workers.

```yaml wrap theme={null}
redis:
  enabled: true
  persistence:
    enabled: true
    size: 1Gi
```

### StatsD exporter

Collects and exports Airflow metrics for monitoring systems like Prometheus.

```yaml wrap theme={null}
statsd:
  enabled: true
  terminationGracePeriodSeconds: 30
```

### PgBouncer (optional)

Connection pooler that sits between Airflow components and the metadata database. Reduces the number of direct database connections opened by the scheduler, webserver, and workers.

PgBouncer is only enabled when the cluster uses PostgreSQL and `pgbouncer.enabled` is set to `true` in your platform configuration. It is disabled when the cluster uses MySQL.

```yaml wrap theme={null}
pgbouncer:
  enabled: true  # depends on cluster database type and platform config
```

### Flower

Web UI for monitoring Celery workers. Only active when using Celery Executor.

```yaml wrap theme={null}
flower:
  enabled: true
```

## Airflow 2 vs Airflow 3 components

| Component       | Airflow 2                  | Airflow 3                  |
| --------------- | -------------------------- | -------------------------- |
| Scheduler       | Required                   | Required                   |
| Webserver       | Required (includes API)    | Required (UI only)         |
| API server      | N/A                        | Required                   |
| Dag processor   | Optional (2.3+)            | Required                   |
| Workers         | Celery Executor only       | Celery Executor only       |
| Triggerer       | Optional (2.2+)            | Optional                   |
| Redis           | Celery Executor only       | Celery Executor only       |
| Flower          | Celery Executor only       | Celery Executor only       |
| StatsD exporter | Required                   | Required                   |
| PgBouncer       | Optional (PostgreSQL only) | Optional (PostgreSQL only) |

## Resource recommendations

### Small workloads (\< 50 Dags)

```yaml wrap theme={null}
scheduler:
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "500m"
      memory: "1Gi"

webserver:
  resources:
    requests:
      cpu: "500m"
      memory: "1920Mi"
    limits:
      cpu: "500m"
      memory: "1920Mi"
```

### Medium workloads (50-200 Dags)

```yaml wrap theme={null}
scheduler:
  resources:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "1000m"
      memory: "2Gi"

dagProcessor:
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "500m"
      memory: "1Gi"
```

### Large workloads (200+ Dags)

```yaml wrap theme={null}
scheduler:
  replicas: 2
  resources:
    requests:
      cpu: "2000m"
      memory: "4Gi"
    limits:
      cpu: "2000m"
      memory: "4Gi"

dagProcessor:
  replicas: 2
  resources:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "1000m"
      memory: "2Gi"
```

## Scaling components

### Horizontal scaling

Components that support multiple replicas. Default limits apply unless your platform administrator overrides them in the platform configuration.

* **Scheduler**: Up to 4 replicas by default.
* **API server**: Up to 4 replicas by default.
* **Dag processor**: Up to 3 replicas by default.
* **Workers**: Up to 10 replicas by default.
* **Triggerer**: Up to 2 replicas by default.

### Vertical scaling

Increase resources for:

* **Scheduler**: Complex dependencies or high task volume. When the Dag processor is enabled, the scheduler focuses on scheduling only.
* **Dag processor**: Large number of Dag files or complex parsing requirements.
* **Workers**: Memory-intensive tasks.