Develop DAGsKubernetesPodOperator

Use secret environment variables

Astro environment variables marked as secrets are stored in a Kubernetes secret called env-secrets. To use a secret value in a task running on the Kubernetes executor, you pull the value from env-secrets and mount it to the Pod running your task as a new Kubernetes Secret.

Setup

1

Add import to your dag file

Add the Secret import to your dag file:

1from airflow.kubernetes.secret import Secret
2

Define a Kubernetes secret

Define a Kubernetes Secret in your dag instantiation using the following format:

1secret_env = Secret(deploy_type="env", deploy_target="<VARIABLE_KEY>", secret="env-secrets", key="<VARIABLE_KEY>")
2namespace = conf.get("kubernetes", "NAMESPACE")
3

Reference the environment variable key

Reference the key for the environment variable, formatted as $VARIABLE_KEY in the task using the KubernetesPodOperator.

Example

In the following example, a secret named MY_SECRET is pulled from env-secrets and printed to logs.

1import pendulum
2from airflow.kubernetes.secret import Secret
3
4from airflow.models import DAG
5from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
6from airflow.configuration import conf
7
8with DAG(
9        dag_id="test-kube-pod-secret",
10        start_date=pendulum.datetime(2022, 1, 1, tz="UTC"),
11        end_date=pendulum.datetime(2022, 1, 5, tz="UTC"),
12        schedule="@once",
13        catchup=False,
14) as dag:
15
16    secret_env = Secret(deploy_type="env", deploy_target="MY_SECRET", secret="env-secrets", key="MY_SECRET")
17
18    namespace = conf.get("kubernetes", "NAMESPACE")
19
20    k = KubernetesPodOperator(
21        namespace=namespace,
22        image="ubuntu:16.04",
23        cmds=["bash", "-cx"],
24        arguments=["echo $MY_SECRET && sleep 150"],
25        name="test-name",
26        task_id="test-task",
27        get_logs=True,
28        in_cluster=True,
29        secrets=[secret_env],
30    )