Documentation

Using the Airflow CLI in Astronomer Enterprise


Although Airflow's UI supports the majority of the behavior you would need when setting up your environment and building DAGs, there are cases when you want to use the Airflow CLI.

Configuring kubectl

Before interacting directly with the pods that are created with each Airflow deployment, it is necessary to first configure the Kubernetes command line kubectl. Instructions on how to install kubectl can be found: here. Connecting to your Kubernetes cluster will be different depending on where it is installed.

Switching to the correct context

Once connected to your cluster, you will need to set the appropriate context.

kubectx
gke_astronomer-demo_us-west2-a_astronomer-demo-ltgzkozwnr

kubectx gke_astronomer-demo_us-west2-a_astronomer-demo-ltgzkozwnr
Switched to context "gke_astronomer-demo_us-west2-a_astronomer-demo-ltgzkozwnr".

Switching to the correct namespace

After setting the context, you will want to switch into the appropriate namespace. Using the kubens command, you will see a separate namespace for each airflow instance you have created in Astronomer.

kubens
astronomer-demo-true-transit-3925
astronomer-demo-meteoric-meteorite-9699

kubens astronomer-demo-meteoric-meteorite-9699
Context "gke_astronomer-demo_us-west2-a_astronomer-demo-ltgzkozwnr" modified.
Active namespace is "astronomer-demo-meteoric-meteorite-9699".

Retrieving pod names

Once in the desired namespace, you can list the pod names. For our purposes, we need to make sure that we take a pod that has Airflow installed on it. This means the scheduler, webserver, or workers.

kubectl get pods
NAME                                                 READY     STATUS    RESTARTS   AGE
meteoric-meteorite-9699-flower-67b5c956bb-d2k27      1/1       Running   0          7m
meteoric-meteorite-9699-pgbouncer-874d47799-2j74x    2/2       Running   0          7m
meteoric-meteorite-9699-redis-0                      1/1       Running   0          7m
meteoric-meteorite-9699-scheduler-7f5ccc7fdb-pfdq5   1/1       Running   0          7m
meteoric-meteorite-9699-statsd-69548ddcbc-n8cgv      1/1       Running   0          7m
meteoric-meteorite-9699-webserver-684b7b56d4-zr7ff   1/1       Running   0          7m
meteoric-meteorite-9699-worker-0                     2/2       Running   0          7m

Examples

Once you have retrieved a pod name (remember, it needs to be one that has airflow installed!) your can exec into it and then execute the relevant airflow commands as below.

Adding Connections

kubectl exec -it meteoric-meteorite-9699-scheduler-7f5ccc7fdb-pfdq5 -- airflow connections --add --conn_id a_new_connection  --conn_type ' ' --conn_login etl --conn_password my_password
	Successfully added `conn_id`=a_new_connection :  ://etl:my_password@:

Adding Variables

kubectl exec -it meteoric-meteorite-9699-scheduler-7f5ccc7fdb-pfdq5 -- airflow variables --set my_key my_value