Remote Execution in Airflow 3 allows tasks to run securely in customer-managed environments without opening inbound connections. Only essential scheduling and health data leave the execution plane.
Remote Execution Agents on Astro enable tasks to run in user-managed hardware or private clouds with only outbound connections to Astro’s Orchestration Plane. Sensitive data stays local, ideal for regulated or multi-regional deployments.
Learn more about Execution modes on Astro.
Airflow 3’s Dag Versioning ensures that each pipeline run references its exact code snapshot, enabling complete historical traceability. Teams can rapidly audit, compare with and debug by viewing the historic Dag code/structure of each run, eliminating confusion and accelerating compliance checks.
Backfills solve one of the most common and time-consuming challenges in data orchestration: reliably reprocessing historical or newly available data. Previously, backfills in Airflow had to be triggered from a command-line process that could easily terminate if the session was lost, leaving longer reruns vulnerable to interruption and without robust monitoring. In Airflow 3, backfills become first-class citizens managed by the scheduler itself, enabling asynchronous API triggers, real-time monitoring through the UI, and the ability to pause or cancel jobs mid-run. This unified approach not only saves teams from manual scripting and fragile workarounds, but it also gives them confidence that large-scale historical recalculations—often critical for machine learning retraining and data integrity checks—will run consistently, even if they take hours or days to complete. To learn more about using backfills, see Rerun Dags and tasks guide.
Airflow 3 introduces a modern, React-based UI that unifies logs, task details, and dynamic Dag updates in a clean, intuitive interface. To learn more, see Airflow UI guide.
Event-driven scheduling in Airflow 3 lets pipelines react to near real-time data changes or external triggers, rather than relying solely on fixed time-based schedules. This means a Dag can automatically start running as soon as a message arrives in the message queue of a supported service. By removing the need for constant polling or hard-coded cron schedules, event-driven pipelines can process data the instant it arrives. This not only saves resources and shortens end-to-end processing time, but also enables more dynamic, near–real-time workflows that are crucial for modern data science, streaming analytics, and AI/ML applications. To learn how to implement it, see the event-driven scheduling guide.
Airflow 3.0 introduces several enhancements to support AI Inference Execution:
Collectively, these enhancements make Airflow 3 more capable of handling diverse inference scenarios, from batch processing to on-demand execution, while offering improved flexibility and performance for AI and ML workflows.