Using the Airflow CLI in Astronomer Enterprise
Although Airflow's UI supports the majority of the behavior you would need when setting up your environment and building DAGs, there are cases when you want to use the Airflow CLI.
Before interacting directly with the pods that are created with each Airflow deployment, it is necessary to first configure the Kubernetes command line
kubectl. Instructions on how to install
kubectl can be found: here. Connecting to your Kubernetes cluster will be different depending on where it is installed.
- Configuring GKE (Google) Access for kubectl
- Configuring EKS (Amazon) Access for kubectl
- Configuring AKS (Microsoft) Access for kubectl
Switching to the correct context
Once connected to your cluster, you will need to set the appropriate context. We highly recommend using kubectx to easily switch between Kubernetes clusters and namespaces.
$ kubectx gke_astronomer-demo_us-west2-a_astronomer-demo-ltgzkozwnr $ kubectx gke_astronomer-demo_us-west2-a_astronomer-demo-ltgzkozwnr Switched to context "gke_astronomer-demo_us-west2-a_astronomer-demo-ltgzkozwnr".
Switching to the correct namespace
After setting the context, you will want to switch into the appropriate namespace. Using the
kubens command, you will see a separate namespace for each airflow instance you have created in Astronomer.
$ kubens astronomer-demo-true-transit-3925 astronomer-demo-meteoric-meteorite-9699 $ kubens astronomer-demo-meteoric-meteorite-9699 Context "gke_astronomer-demo_us-west2-a_astronomer-demo-ltgzkozwnr" modified. Active namespace is "astronomer-demo-meteoric-meteorite-9699".
Retrieving pod names
Once in the desired namespace, you can list the pod names. For our purposes, we need to make sure that we take a pod that has Airflow installed on it. This means the scheduler, webserver, or workers.
$ kubectl get pods NAME READY STATUS RESTARTS AGE meteoric-meteorite-9699-flower-67b5c956bb-d2k27 1/1 Running 0 7m meteoric-meteorite-9699-pgbouncer-874d47799-2j74x 2/2 Running 0 7m meteoric-meteorite-9699-redis-0 1/1 Running 0 7m meteoric-meteorite-9699-scheduler-7f5ccc7fdb-pfdq5 1/1 Running 0 7m meteoric-meteorite-9699-statsd-69548ddcbc-n8cgv 1/1 Running 0 7m meteoric-meteorite-9699-webserver-684b7b56d4-zr7ff 1/1 Running 0 7m meteoric-meteorite-9699-worker-0 2/2 Running 0 7m
Once you have retrieved a pod name (remember, it needs to be one that has airflow installed!) your can
exec into it and then execute the relevant airflow commands as below.
$ kubectl exec -it meteoric-meteorite-9699-scheduler-7f5ccc7fdb-pfdq5 -- airflow connections --add --conn_id a_new_connection --conn_type ' ' --conn_login etl --conn_password my_password Successfully added `conn_id`=a_new_connection : ://etl:my_password@:
$ kubectl exec -it meteoric-meteorite-9699-scheduler-7f5ccc7fdb-pfdq5 -- airflow variables --set my_key my_value
Ready to build your data workflows with Airflow?
Astronomer is the data engineering platform built by developers for developers. Send data anywhere with automated Apache Airflow workflows, built in minutes...