Skip to main content
This is feature is only available if you are on the Enterprise tier or above. See Astro Plans and Pricing.
Airflow 3This feature is only available for Airflow 3.x Deployments.
Remote Execution is an execution mode on Astro that separates task execution from orchestration. With Remote Execution, you run Airflow tasks in your own Kubernetes infrastructure while Astro manages the orchestration components in the cloud. Use Remote Execution to keep your data and code within your own infrastructure for security or compliance requirements.
Astro Remote architecture overview

​
How Remote Execution works

Remote Execution uses a decoupled architecture with two planes: Orchestration plane (Astro-managed) The orchestration plane runs in Astro’s cloud infrastructure and includes:
  • Scheduler: Determines when dags and tasks should run
  • Web Server/API Server: Provides the Airflow UI and REST API
  • Metadata Database: Stores Dag and task metadata
  • Remote Execution API: Manages agent communication and task distribution
The orchestration plane assigns tasks to agents, monitors their health via heartbeats, and provides visibility through logging and observability features. Execution plane (Customer-managed) The execution plane runs in your Kubernetes cluster and includes:
  • Remote Execution Agents: Deployed via Helm charts, each agent includes:
    • Dag Processor: Parses and serializes Dag code
    • Triggerer: Manages deferrable tasks
    • Worker: Executes Airflow tasks
    • Sentinel: Monitors agent health and reports status to the orchestration plane
Agents pull tasks from the orchestration plane via secure HTTPS connections, execute them locally, and report status back. Agents maintain frequent heartbeats with the API Server. If an agent loses its heartbeat, the orchestration plane automatically reroutes tasks to healthy agents. See Remote Execution Agent failure scenarios for more information.

​
Key concepts

​
Remote Execution Agents

Agents are the core component of Remote Execution. Each agent installation is a set of Airflow components (worker, Dag processor, and triggerer) that one Helm release deploys as a single unit in your Kubernetes cluster. You can deploy multiple agent installations with unique configurations across different clusters, regions, or node types to meet your workload requirements. Each Pod in the agent installation runs its own agent process, and each process registers with the orchestration plane under its own identity and sends its own heartbeat. To learn more about how Astro allocates tasks to worker agents, see How Astro allocates tasks to workers. Agents communicate with Astro using:
  • Agent tokens: Authenticate agents to the orchestration plane
  • Outbound-only connections: Enable communication from your infrastructure to Astro without requiring inbound traffic
  • Heartbeat mechanism: Monitor agent health with regular status checks

​
Tenants

A tenant is one worker queue in a Deployment. Each queue is served by one or more autoscaling worker deployments, and can have its own image, hardware profile, cloud identity, and secrets scope. See Remote Execution isolation and multi-tenancy for how to separate teams and workloads across tenants, agents, namespaces, and clusters. A tenant counts as active while workers serving its queue are heartbeating and available for task execution. Each Deployment includes one tenant and supports up to 10 active tenants. At the limit, a worker that tries to register a new queue is refused until a slot frees. Contact Astronomer Support if your Deployment requires more than 10 active tenants. See Tenant limits. Remote Execution Deployments bill at a fixed hourly rate instead of by metered task minutes. The rate depends on whether the Deployment has high availability enabled, and both rates include unlimited task minutes with no metering and no overages. The included tenant is part of that rate, and each additional active tenant adds an hourly charge. Organizations on the previous model keep task-minute billing instead, with no tenant limit until they move. See Astro pricing for the rates.

​
Dag bundles

Dag bundles are collections of Dag files and supporting code. Remote Execution supports two types:
  • GitDagBundle: Dags stored in a Git repository (recommended for production)
  • LocalDagBundle: Dags stored in the container image or persistent volume
GitDagBundle provides automatic Dag versioning, allowing you to track changes and view different versions in the Airflow UI.

​
XCom backend

XCom (cross-communication) allows Airflow tasks to share data. With Remote Execution, you must configure an object storage backend (AWS S3, Azure Blob Storage, or GCP Cloud Storage) to pass XCom data between tasks running on different agents.

​
Secrets backend

Remote Execution Agents must be configured with a secrets backend to securely access Airflow connections and variables. This can be AWS Secrets Manager, Azure Key Vault, Google Cloud Secret Manager, or HashiCorp Vault.

​
State store backend

Starting with Airflow 3.3 (Astro Runtime 3.3), tasks and asset watchers can persist state between runs through the Airflow worker-side state store. With Remote Execution, agents store this state in your execution plane instead of the Astro-hosted metadata database. The Remote Execution Agent Helm chart (2.3.0 and later) sets a working default, and you can point it at object storage for production.

​
What you need to configure

Configure the following required and optional components for Remote Execution.

​
Required components

  • Remote Execution Agents: Deployed via Helm in your Kubernetes cluster
  • Secrets Backend: To securely store and access Airflow connections and variables
  • XCom Backend: Object storage for passing data between tasks
  • State Store Backend: Storage for task and asset state on Astro Runtime 3.3 and later. The Helm chart (2.3.0 and later) sets a working default
  • Dag Sources: Configure how agents access your dag code (Git or local)
  • Sentinel: Monitor agent health and report status to the orchestration plane. Astronomer recommends enabling Sentinel for all production deployments.

​
Optional components

  • Logging: Export task logs to external logging platforms or object storage
  • OpenLineage: Enable data lineage and observability features

​
Next steps