WEBINARS

Scaling Out Airflow

Watch On Demand


Hosted By

  • Alex Kennedy
  • Kenten Danas

1. Key Points About Scaling Airflow

2. High-Level Steps to Scale Airflow

scaling-out-airflow-image3

2. Why Scale Apache Airflow?

4. Symptoms That Mean You’re Ready to Scale

Principles of Scaling Systems

5. Basics of Scaling Systems

scaling-out-airflow-image6

Scaling Airflow as a Distributed Platform

6. Scaling with CeleryExecutor

https://airflow.apache.org/docs/apache-airflow/stable/executor/celery.html

7. Scaling with KubernetesExecutor

https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html TaskInstances run on K8s pods

scaling-out-airflow-image8

Parameter Tuning when Scaling Airflow

https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html

scaling-out-airflow-image4

scaling-out-airflow-image9

scaling-out-airflow-image10

scaling-out-airflow-image2

These parameters control the number of tasks that can be run at a time.

8. Sizing Pools

High Availability Airflow Components

Other Airflow Components can Scale!

Scheduler

Web Server

Logging on Distributed Airflow

9. Traits of a Great Logging System

scaling-out-airflow-image7

10. Importance of Good Logging Practices

11. Debugging Distributed Airflow

How to Scale your Deployment - Demo