Ephemeral storage configuration

Ephemeral storage in Astro Private Cloud (APC) uses Kubernetes emptyDir volumes for temporary data that doesn’t need to persist across Pod restarts. This guide covers configuring ephemeral storage for Apache Airflow components including Dags, logs, and Redis.

Ephemeral storage overview

Ephemeral storage (emptyDir volumes) provides:

• Temporary storage that exists for the lifetime of a Pod.
• Faster I/O if using memory-backed storage.
• No persistence across Pod restarts.
• Shared access between containers in the same Pod.

Configurable volumes

Dags volume (gitSync)

When using git-sync for Dag deployment, Dags are stored in an emptyDir volume.

1
dags:
2
  gitSync:
3
    enabled: true
4
    emptyDirConfig:
5
      sizeLimit: 2Gi
6
      medium: Memory  # Optional: use RAM instead of disk

Parameters:

• sizeLimit: Maximum storage size (for example, 1Gi, 2Gi).
• medium: Storage medium.
  • "" (empty): Use node’s default storage (disk).
  • "Memory": Use RAM (tmpfs) for faster access.

Logs volume

Task logs can be stored in ephemeral storage when persistence is disabled.

1
logs:
2
  persistence:
3
    enabled: false
4
  emptyDirConfig:
5
    sizeLimit: 10Gi
6
    medium: ""  # Use disk for logs (recommended)

Recommendations:

• Use larger sizeLimit for high-volume task execution.
• Avoid medium: Memory for logs unless you have aggressive cleanup.
• Enable log groomer sidecar to prevent storage exhaustion.

Redis volume

When Redis persistence is disabled, it uses ephemeral storage.

1
redis:
2
  persistence:
3
    enabled: false
4
  emptyDirConfig:
5
    sizeLimit: 1Gi
6
    medium: Memory  # RAM-backed for performance

Configuration examples

Development environment

1
dags:
2
  gitSync:
3
    enabled: true
4
    emptyDirConfig:
5
      sizeLimit: 1Gi
6
7
logs:
8
  emptyDirConfig:
9
    sizeLimit: 5Gi
10
11
redis:
12
  emptyDirConfig:
13
    sizeLimit: 512Mi
14
    medium: Memory

Production environment

1
dags:
2
  gitSync:
3
    enabled: true
4
    emptyDirConfig:
5
      sizeLimit: 5Gi
6
      medium: ""  # Use disk storage for durability
7
8
logs:
9
  persistence:
10
    enabled: true  # Use persistent storage in production
11
    size: 100Gi
12
13
redis:
14
  persistence:
15
    enabled: true  # Use persistent storage in production
16
    size: 8Gi

High-performance configuration

For latency-sensitive workloads:

1
dags:
2
  gitSync:
3
    enabled: true
4
    emptyDirConfig:
5
      sizeLimit: 2Gi
6
      medium: Memory  # Faster Dag parsing
7
8
redis:
9
  emptyDirConfig:
10
    sizeLimit: 2Gi
11
    medium: Memory  # Faster task queue operations

Storage medium comparison

Sizing guidelines

Dags volume

Logs volume

Redis volume

Monitor storage usage

Check volume usage

$
kubectl exec -n <namespace> <pod-name> -- df -h

Check memory-backed volume

$
kubectl exec -n <namespace> <pod-name> -- mount | grep tmpfs

Troubleshooting

Pod eviction due to storage

Symptom: Pods evicted with DiskPressure or ephemeral storage exceeded.

Cause: emptyDir volume exceeded node’s ephemeral storage limits.

Solution:

1. Increase sizeLimit in emptyDirConfig.
2. Enable log groomer with shorter retention.
3. Switch to persistent storage.

Out of memory with memory-backed volumes

Symptom: Pods OOMKilled when using medium: Memory.

Cause: Memory-backed emptyDir counts against container memory limits.

Solution:

1. Increase container memory limits.
2. Reduce sizeLimit on memory-backed volumes.
3. Switch to disk-backed storage.

Slow Dag parsing

Symptom: Dag processing takes too long.

Cause: Disk I/O latency on Dag volume.

Solution:

1. Use medium: Memory for Dag volume.
2. Ensure sufficient sizeLimit.
3. Consider SSD-backed nodes.

Best practices

• Set explicit sizeLimit to prevent unbounded storage growth.
• Use memory sparingly for performance-critical, small volumes only.
• Monitor usage and set alerts for storage utilization.
• Use persistent storage for production logs since ephemeral storage loses logs on restart.
• Size for peak usage to account for burst workloads.
• Enable log groomer to prevent log accumulation.