Airflow 3

Airflow 3 new features

Astro Private Cloud can be used with all supported Airflow versions, including Airflow 3. Apache Airflow 3 introduces a suite of new features such as event-driven scheduling, advanced inference execution, a redesigned UI, and high-performance backfills. See the complete documentation for Airflow.

Airflow 3 new features overview

Backfills: Backfills solve one of the most common and time-consuming challenges in data orchestration: reliably reprocessing historical or newly available data. Previously, backfills in Airflow had to be triggered from a command-line process that could easily terminate if the session was lost, leaving longer reruns vulnerable to interruption and without robust monitoring. In Airflow 3, backfills become first-class citizens managed by the scheduler itself, enabling asynchronous API triggers, real-time monitoring through the UI, and the ability to pause or cancel jobs mid-run.

UI Modernization: Airflow 3 introduces a modern, React-based UI that unifies logs, task details, and dynamic Dag updates.

Event-driven Scheduling: Event-driven scheduling in Airflow 3 lets pipelines react to near real-time data changes or external triggers, rather than relying solely on fixed time-based schedules. This means a Dag can automatically start running as soon as a message arrives in the message queue of a supported service.

Inference Execution: Airflow 3.0 introduces several enhancements to support AI Inference Execution:

  • Ad-hoc scheduling: Airflow 3.0 allows Dags to be run independently of any data interval, which is crucial for supporting inference execution. This feature enables on-demand execution of inference tasks without being constrained by predefined schedules.
  • Synchronous Dag execution: The new version supports simultaneous execution of the same Dag, allowing for synchronous inference runs. This is particularly useful for scenarios where multiple inference requests need to be processed concurrently.
  • API-triggered execution: Airflow 3.0 introduces the ability to trigger Dags via API calls, enabling multiple instances to be initiated simultaneously for inference tasks. This feature facilitates experimentation and allows for dynamic, near real-time inference processing.
  • Event-driven scheduling: The new version supports automatic triggering of Dags based on external events or data availability. This can be particularly useful for inference pipelines that need to react to new data or model updates in near real-time.
  • Language-agnostic Task Execution Interface: Airflow 3.x lays the groundwork to run tasks in any language. This enables users to implement inference tasks in the most suitable language for their models, without expensive code refactoring such as using C++, Golang, Java, etc. for more efficient execution.

Supported Airflow 3 features

The following Airflow 3 features have been tested and are supported with Astro Private Cloud 2.0:

Core platform

  • Deployment CRUD: Create, update, and delete Airflow deployments through Astro Private Cloud.
  • All Deployment Types: Airflow 3 is supported in both unified and split control plane - data plane modes.
  • All executors: Airflow 3 is supported with Celery Executor and Kubernetes executor.

Observability and operations

  • Human-in-the-loop (HITL): Supported for manual approvals and task-level interventions.
  • Backfills: Reliably reprocess historical or newly available data with improved performance and visibility.
  • Deadline Alerts / SLA Enhancements: Improved SLA monitoring and alerting within Airflow 3.
  • Remote Logging: Integrated support for remote log streaming and storage via the platform.

User experience and extensibility

  • New UI and Plugins: New Airflow 3 UI and compatible Astronomer plugins supported.
  • Assets and Asset Decorators: Support for Airflow 3’s asset-based Dag authoring and tracking model.
  • Event-driven scheduling: React to near real-time data changes or external triggers.
  • Language-agnostic Task Execution Interface: Implement tasks in Golang.

Security and access

  • Airflow 3 RBAC: Full support for Airflow’s built-in role-based access control model.

Migration and compatibility

  • Connections, XComs, and Variables Migration: Migration tooling available to transition from Airflow 2.x to Airflow 3.

Unsupported Airflow 3 features

The following Airflow 3 features aren’t supported with Astro Private Cloud 2.0:

  • Dag versioning: Tracking and managing versions of Dags across deployments isn’t supported.
  • Remote workers: Triggering or running tasks remotely isn’t supported through Astro Private Cloud.