Airflow 3

Airflow 3 New Features

Astro Private Cloud can be used with all supported Airflow versions, including Airflow 3. Apache Airflow 3 introduces a suite of new features such as event-driven scheduling, advanced inference execution, a redesigned UI, and high-performance backfills. Complete documentation for Airflow can be found here.

Airflow 3 New Features Overview

Backfills: Backfills solve one of the most common and time-consuming challenges in data orchestration: reliably reprocessing historical or newly available data. Previously, backfills in Airflow had to be triggered from a command-line process that could easily terminate if the session was lost, leaving longer reruns vulnerable to interruption and without robust monitoring. In Airflow 3, backfills become first-class citizens managed by the scheduler itself, enabling asynchronous API triggers, real-time monitoring through the UI, and the ability to pause or cancel jobs mid-run. This unified approach not only saves teams from manual scripting and fragile workarounds, but it also gives them confidence that large-scale historical recalculations—often critical for machine learning retraining and data integrity checks—will run consistently, even if they take hours or days to complete.

UI Modernization: Airflow 3 introduces a modern, React-based UI that unifies logs, task details, and dynamic dag updates in a clean, intuitive interface.

Event-driven Scheduling: Event-driven scheduling in Airflow 3 lets pipelines react to near real-time data changes or external triggers, rather than relying solely on fixed time-based schedules. This means a dag can automatically start running as soon as a message arrives in the message queue of a supported service. By removing the need for constant polling or hard-coded cron schedules, event-driven pipelines can process data the instant it arrives. This not only saves resources and shortens end-to-end processing time, but also enables more dynamic, near–real-time workflows that are crucial for modern data science, streaming analytics, and AI/ML applications.

Inference Execution: Airflow 3.0 introduces several enhancements to support AI Inference Execution:

  • Ad-hoc scheduling: Airflow 3.0 allows dags to be run independently of any data interval, which is crucial for supporting inference execution. This feature enables on-demand execution of inference tasks without being constrained by predefined schedules.

  • Synchronous dag execution: The new version supports simultaneous execution of the same dag, allowing for synchronous inference runs. This is particularly useful for scenarios where multiple inference requests need to be processed concurrently.

  • API-triggered execution: Airflow 3.0 introduces the ability to trigger dags via API calls, enabling multiple instances to be initiated simultaneously for inference tasks. This feature facilitates experimentation and allows for dynamic, near real-time inference processing.

  • Event-driven scheduling: The new version supports automatic triggering of dags based on external events or data availability. This can be particularly useful for inference pipelines that need to react to new data or model updates in near real-time.

  • Language-agnostic Task Execution Interface: Airflow 3.x lays the groundwork to run tasks in any language. This enables users to implement inference tasks in the most suitable language for their models, without expensive code refactoring such as using C++, Golang, Java, etc. for more efficient execution.

    Collectively, these enhancements make Airflow 3 more capable of handling diverse inference scenarios, from batch processing to on-demand execution, while offering improved flexibility and performance for AI and ML workflows.

Supported Airflow 3 Features:

The following Airflow 3 features have been tested and are supported with Astro Private Cloud 1.0:

Core Platform

  • Deployment CRUD: Create, update, and delete Airflow deployments through Astro Private Cloud.
  • All Deployment Types: Airflow 3 is supported in both Unified and split Control Plane - Data Plane modes.
  • All Executors: Airflow 3 is supported with Celery and Kubernetes executors.

Observability and Operations

  • HITL (Human-in-the-Loop): Supported for manual approvals and task-level interventions.
  • Backfills: Reliably reprocess historical or newly available data with improved performance and visibility.
  • Deadline Alerts / SLA Enhancements: Improved SLA monitoring and alerting within Airflow 3.
  • Remote Logging: Integrated support for remote log streaming and storage via the platform.

User Experience and Extensibility

  • New UI and Plugins: New Airflow 3 UI and compatible Astronomer plugins supported.
  • Assets and Asset Decorators: Support for Airflow 3’s asset-based DAG authoring and tracking model.
  • Event-driven scheduling: React to near real-time data changes or external triggers.
  • Language-agnostic Task Execution Interface: Implement tasks in Golang.

Security and Access

  • Airflow 3 RBAC: Full support for Airflow’s built-in role-based access control model.

Migration and Compatibility

  • Connections, XComs, and Variables Migration: Migration tooling available to transition from Airflow 2.x to Airflow 3.

Unsupported Airflow 3 Features

The following Airflow 3 features are not supported with Astro Private Cloud 1.0:

  • DAG versioning: Tracking and managing versions of DAGs across deployments is not supported.
  • Remote workers: Triggering or running tasks remotely is not supported through Astro Private Cloud.