What's new in Apache Airflow 3.3

Airflow 3.3 is here, with a set of features to help with the messy realities of production pipelines: persisting state across retries, reacting intelligently to different failure types, and partitioning assets by more than just time. In this episode, Marc Lamberti, Education Content Lead at Astronomer, joins Kenten Danas to walk through what's new in the release and where each feature actually pays off.

Key Takeaways:

• 00:00 Introduction.
• 01:46 The new task state store (AIP-103) lets tasks persist state across retries, so a long-running Spark job can be reattached after a worker failure instead of being duplicated on retry.
• 03:46 The asset state store enables watermarking patterns: persist the last processed date or offset to an asset and resume from there on the next run.
• 05:33 Why this matters for agentic workflows: resume an agent from where it left off rather than replaying every action.
• 06:58 Why XComs don't solve this problem: they get reinitialized on every retry.
• 09:27 Pluggable retries let you attach a retry policy to a task that branches on the exception type. Retry on transient errors, stop immediately on a 403.
• 11:42 Subclassing the retry rule for more complex logic, including dynamic retry counts that used to require hacking the metadatabase.
• 15:52 Updates to asset partitions in 3.3: segment-based partitioning with fan-out and roll-up mappers for downstream DAGs.
• 21:42 Running tasks in Java and Go, moving Airflow toward a multi-language orchestrator.
• 23:33 DAG versioning improvement: choose whether a manual rerun uses the most recent DAG version or the original version from that run.
• 25:03 Advice for teams still on Airflow 2: use the upgrade ebook and Astro's AI migration tooling to handle the undifferentiated heavy lifting.

Resources Mentioned:

• Astronomer
• Airflow 3.3 Release Notes
• The Task State Store
• Updates to the asset partitions feature
• Retry policies
• Multi-language support
• 3.3 Webinar
• Upgrading from Airflow 2 to 3 ebook

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.