Recap Preview
Kubernetes Executor
- Each Airflow task is launched as a pod
- Workers scale to zero
- Expose Kubernetes API to the data engineer so they can have more control over the resources of each task
Old Architechture
- Attempted to abstract Kubernetes API for “simplicity”
- Result: Lots of PRs to expose Kubernetes, lots of code to maintain, lost of steps before a pod is launched
- Goal: Offer flexibility of the Kubernetes API and reduce Airflow’s code footprint
