Install Remote Execution Agents in a restricted kubernetes namespace

Airflow 3

This feature is only available for Airflow 3.x Deployments.

You can install the Remote Execution Agent in a Kubernetes namespace with restricted pod security standards. Your organization might have different security standards for infrastructure supporting internal-only sandboxes compared to production environments.

Kubernetes Pod Security Standards define different security levels for Pods:

Privileged: No restrictions (least secure)
Baseline: Prevents known privilege escalations
Restricted: Highly-constrained settings following security best practices (most secure)

The Restricted profile enforces the following limitations:

Runs containers as non-root users
Prevents privilege escalation
Drops all Linux capabilities
Uses read-only root filesystems when possible
Requires a runtime default seccomp profile

However, because of these limitations, you need to complete the following additional Remote Execution Agent configuration set up.

Step 1: Create a restricted namespace

Create a Namespace in your Kubernetes manifest with the following restricted Pod security standards:

1 apiVersion: v1
2 kind: Namespace
3 metadata:
4   name: astro-agent-restricted
5   labels:
6     pod-security.kubernetes.io/enforce: restricted
7     pod-security.kubernetes.io/audit: restricted
8     pod-security.kubernetes.io/warn: restricted

Step 2: Configure Global Security Settings

Modify your Agent’s values.yaml file to set global security context settings that apply to all Agent components’ Pods and containers:

1 # Global pod security context for all components
2 podSecurityContext:
3   seccompProfile:
4     type: RuntimeDefault
5   runAsUser: 50000
6   fsGroup: 50000
7   runAsNonRoot: true
8 
9 # Global container security context for all components
10 containerSecurityContext:
11   allowPrivilegeEscalation: false
12   readOnlyRootFilesystem: true
13   capabilities:
14     drop:
15       - ALL

Step 3: Configure component-specific settings

When using the Agent in a restricted namespace, you must configure volume mounts because:

The container security context sets readOnlyRootFilesystem: true
These directories need write access during runtime
Using emptyDir volumes provides isolated, writable storage that meets security requirements

Worker configuration

1 workers:
2   - name: default-worker
3     # Other worker settings...
4 
5     # Required volumes for filesystem access
6     volumes:
7       - name: tmp
8         emptyDir: {}
9       - name: task-logs
10         emptyDir: {}
11 
12     volumeMounts:
13       - name: tmp
14         mountPath: /tmp # This folder is used by the Astro Agent to maintain the socket file
15       - name: task-logs
16         mountPath: /usr/local/airflow/logs # Or the configured folder that holds the task logs

Dag processor configuration

1 dagProcessor:
2   # Other Dag processor settings...
3 
4   # Required volumes for filesystem access
5   volumes:
6     - name: tmp
7       emptyDir: {}
8 
9   volumeMounts:
10     - name: tmp
11       mountPath: /tmp # This folder is used by the Astro Agent to maintain the socket file

Triggerer configuration

1 triggerer:
2   # Other triggerer settings...
3 
4   # Required volumes for filesystem access
5   volumes:
6     - name: tmp
7       emptyDir: {}
8     - name: task-logs
9       emptyDir: {}
10 
11   volumeMounts:
12     - name: tmp
13       mountPath: /tmp # This folder is used by the Astro Agent to maintain the socket file
14     - name: task-logs
15       mountPath: /usr/local/airflow/logs # Or the configured folder that holds the task logs

Step 4: (Optional) Add logging sidecar configuration

1 loggingSidecar:
2   # Other logging sidecar settings...
3 
4   securityContext:
5     allowPrivilegeEscalation: false
6     readOnlyRootFilesystem: true
7     capabilities:
8       drop:
9         - ALL
10   volumeMounts:
11     - name: task-logs
12       mountPath: /etc/airflow/logs # Or the configured folder that holds the task logs