Airflow 2.0 + Kubernetes

WATCH ON DEMAND

In This Webinar We Covered:

  • Kubernetes Executor
  • Kubernetes Pod Operator
  • KEDA Autoscaler

Missed the Webinar? Sign up for the Recap

Recap Preview

Kubernetes Executor

  • Each Airflow task is launched as a pod
  • Workers scale to zero
  • Expose Kubernetes API to the data engineer so they can have more control over the resources of each task

Old Architechture

airflowKubernetesOldArc

  • Attempted to abstract Kubernetes API for “simplicity”
  • Result: Lots of PRs to expose Kubernetes, lots of code to maintain, lost of steps before a pod is launched
  • Goal: Offer flexibility of the Kubernetes API and reduce Airflow’s code footprint

Hosted By

Daniel Imberman

Daniel Imberman

Software Engineer

Viraj Parekh

Viraj Parekh

Field CTO