We introduce Airflow 3.2 and its updates for teams that build and operate data pipelines.

Astronomer's Head of Customer Education, Marc Lamberti, and Senior Manager of Developer Relations, Kenten Danas, break down what's new, from asset partitioning to Async Python tasks and DAG versioning. They explore how these updates improve scheduling, performance and observability in production workflows.

Key Takeaways:

(00:00) Introduction.

(02:10) Airflow 3 architecture separates workers from the metadata database.

(03:05) Plugin versioning and UI-based backfills simplify operations.

(06:20) Asset partitioning enables granular, partition-level scheduling.

(07:15) Triggering DAGs on partitions instead of full datasets.

(11:05) Deferrable operators reduce worker slot usage.

(12:00) Async operators reduce database pressure and overhead.

(14:10) Async improves throughput, not single task speed.

(22:20) Inlets and outlets improve asset lineage visibility.

(23:00) DAG version markers show changes directly in the UI.

Resources Mentioned:

Apache Airflow

Astronomer

3.2 Webinar

Asset Partitioning Guide

Asynchronous Processes Guide

Release Notes

Provider Registry

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning