> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Ephemeral storage configuration

Ephemeral storage in Astro Private Cloud (APC) uses Kubernetes emptyDir volumes for temporary data that doesn't need to persist across Pod restarts. This guide covers configuring ephemeral storage for Apache Airflow components including Dags, logs, and Redis.

## Ephemeral storage overview

Ephemeral storage (emptyDir volumes) provides:

* Temporary storage that exists for the lifetime of a Pod.
* Faster I/O if using memory-backed storage.
* No persistence across Pod restarts.
* Shared access between containers in the same Pod.

<Note>
  APC runs every Airflow container with `readOnlyRootFilesystem: true`. To keep Airflow writable where it needs to be, the chart mounts `emptyDir` volumes at specific subpaths under `/usr/local/airflow`, such as `/usr/local/airflow/logs`, and at `/usr/local/airflow/dags` when using git-sync or the emptyDir Dag mode. Depending on log volume and Dag size, these volumes can consume significant ephemeral storage and exceed namespace limits. Factor this into your ephemeral storage sizing. For details on adding more writable directories, see [Read-only root filesystem](/docs/astro-private-cloud/v-2-x/read-only-root-filesystem).
</Note>

## Configurable volumes

### Dags volume (gitSync)

When using git-sync for Dag deployment, Dags are stored in an emptyDir volume.

```yaml wrap theme={null}
dags:
  gitSync:
    enabled: true
    emptyDirConfig:
      sizeLimit: 2Gi
      medium: Memory  # Optional: use RAM instead of disk
```

Parameters:

* `sizeLimit`: Maximum storage size (for example, `1Gi`, `2Gi`).
* `medium`: Storage medium.
  * `""` (empty): Use node's default storage (disk).
  * `"Memory"`: Use RAM (tmpfs) for faster access.

### Logs volume

Task logs can be stored in ephemeral storage when persistence is disabled.

```yaml wrap theme={null}
logs:
  persistence:
    enabled: false
  emptyDirConfig:
    sizeLimit: 10Gi
    medium: ""  # Use disk for logs (recommended)
```

Recommendations:

* Use larger `sizeLimit` for high-volume task execution.
* Avoid `medium: Memory` for logs unless you have aggressive cleanup.
* Enable log groomer sidecar to prevent storage exhaustion.

### Redis volume

When Redis persistence is disabled, it uses ephemeral storage.

```yaml wrap theme={null}
redis:
  persistence:
    enabled: false
  emptyDirConfig:
    sizeLimit: 1Gi
    medium: Memory  # RAM-backed for performance
```

## Configuration examples

### Development environment

```yaml wrap theme={null}
dags:
  gitSync:
    enabled: true
    emptyDirConfig:
      sizeLimit: 1Gi

logs:
  emptyDirConfig:
    sizeLimit: 5Gi

redis:
  emptyDirConfig:
    sizeLimit: 512Mi
    medium: Memory
```

### Production environment

```yaml wrap theme={null}
dags:
  gitSync:
    enabled: true
    emptyDirConfig:
      sizeLimit: 5Gi
      medium: ""  # Use disk storage for durability

logs:
  persistence:
    enabled: true  # Use persistent storage in production
    size: 100Gi

redis:
  persistence:
    enabled: true  # Use persistent storage in production
    size: 8Gi
```

### High-performance configuration

For latency-sensitive workloads:

```yaml wrap theme={null}
dags:
  gitSync:
    enabled: true
    emptyDirConfig:
      sizeLimit: 2Gi
      medium: Memory  # Faster Dag parsing

redis:
  emptyDirConfig:
    sizeLimit: 2Gi
    medium: Memory  # Faster task queue operations
```

## Storage medium comparison

| Medium      | Speed    | Persistence  | Memory Impact | Use Case          |
| ----------- | -------- | ------------ | ------------- | ----------------- |
| `""` (disk) | Moderate | Pod lifetime | None          | Logs, large Dags  |
| `"Memory"`  | Fast     | Pod lifetime | Consumes RAM  | Redis, small Dags |

<Warning>
  Memory-backed volumes (`medium: Memory`) count against container memory limits. If the volume grows too large, Pods may be OOMKilled. Size memory limits accordingly or use disk-backed storage.
</Warning>

## Sizing guidelines

### Dags volume

| Dag count | Recommended size |
| --------- | ---------------- |
| \< 50     | 1Gi              |
| 50-200    | 2Gi              |
| 200-500   | 5Gi              |
| 500+      | 10Gi             |

### Logs volume

| Task Volume          | Recommended Size       |
| -------------------- | ---------------------- |
| \< 100 tasks/day     | 5Gi                    |
| 100-1000 tasks/day   | 10Gi                   |
| 1000-10000 tasks/day | 50Gi                   |
| 10000+ tasks/day     | Use persistent storage |

### Redis volume

| Worker Count  | Recommended Size |
| ------------- | ---------------- |
| 1-5 workers   | 512Mi            |
| 5-20 workers  | 1Gi              |
| 20-50 workers | 2Gi              |
| 50+ workers   | 4Gi              |

## Monitor storage usage

### Check volume usage

```bash wrap theme={null}
kubectl exec -n <namespace> <pod-name> -- df -h
```

### Check memory-backed volume

```bash wrap theme={null}
kubectl exec -n <namespace> <pod-name> -- mount | grep tmpfs
```

## Troubleshooting

### Pod eviction due to storage

**Symptom**: Pods evicted with `DiskPressure` or ephemeral storage exceeded.

**Cause**: emptyDir volume exceeded node's ephemeral storage limits.

**Solution**:

1. Increase `sizeLimit` in `emptyDirConfig`.
2. Enable log groomer with shorter retention.
3. Switch to persistent storage.

### Out of memory with memory-backed volumes

**Symptom**: Pods OOMKilled when using `medium: Memory`.

**Cause**: Memory-backed emptyDir counts against container memory limits.

**Solution**:

1. Increase container memory limits.
2. Reduce `sizeLimit` on memory-backed volumes.
3. Switch to disk-backed storage.

### Slow Dag parsing

**Symptom**: Dag processing takes too long.

**Cause**: Disk I/O latency on Dag volume.

**Solution**:

1. Use `medium: Memory` for Dag volume.
2. Ensure sufficient `sizeLimit`.
3. Consider SSD-backed nodes.

## Best practices

* Set explicit `sizeLimit` to prevent unbounded storage growth.
* Use memory sparingly for performance-critical, small volumes only.
* Monitor usage and set alerts for storage utilization.
* Use persistent storage for production logs since ephemeral storage loses logs on restart.
* Size for peak usage to account for burst workloads.
* Enable log groomer to prevent log accumulation.