Astro Private Cloud data plane architecture

The Astro Private Cloud (APC) data plane hosts the execution layer for your Airflow deployments. When you set the global.plane.mode: data in your values.yaml file, the Helm chart deploys only the runtime-facing components while relying on a separate control plane for user management, configuration, registry orchestration, and token orchestration. This document summarizes the data plane’s responsibilities, the services that run in this mode, and how the services integrate with the control plane. For the management plane, see Control Plane Architecture, or review Unified Architecture if you want to run both in a single cluster.

Responsibilities

A data plane cluster focuses on:

• Running customer Airflow Deployments: The Deployment orchestrator (Commander) installs and upgrades each Deployment’s runtime chart using configuration synced from Houston.
• Serving Airflow ingress: The ingress controller (Data plane NGINX) exposes deployments.<domain-prefix>.<base-domain> (and any per-Deployment vanity hostnames) to route user traffic into the correct Airflow namespace.
• Collecting telemetry: The metrics collector (Prometheus) scrapes Deployment namespaces and either exposes a federate endpoint or remote-writes metrics to the control plane. Optional logging stacks (log forwarder Vector and log store Elasticsearch) gather task logs.
• Handling image distribution: The platform registry (Registry) stores runtime and dag images local to the data plane and syncs credentials via the secret distribution job (Config Syncer).
• Maintaining secure connectivity: Config Syncer distributes Houston-issued tokens and certificates so Deployments can authenticate back to Houston, the registry, and other platform services.

Core Components in Data Plane Mode

When global.plane.mode is set to data or unified, APC enables the following charts:

• Commander (charts/astronomer/templates/commander/*): Polls Houston for desired state and applies/rolls back Helm releases for each Deployment.
• Config Syncer (charts/astronomer/templates/config-syncer/*): Periodically mirrors platform secrets (registry credentials, Houston tokens, runtime settings) into each Airflow namespace.
• Data plane NGINX (charts/nginx/templates/dataplane/*): Provides ingress for the per-Deployment Airflow UI and API as well as for platform components like the Registry, Prometheus, and Elasticsearch. This component is not installed when OpenShift is enabled.
• Platform registry (Registry) (charts/astronomer/templates/registry/*): Optional container registry for runtime/dag images when the bundled registry is enabled.
• Vector (charts/vector/templates/*): Daemonset, or a sidecar container if global.loggingSidecar is enabled, that tails task logs and ships them to Elasticsearch or an external destination.
• Elasticsearch (charts/elasticsearch/templates/*): Optional log storage for Deployments. Only deployed in data plane or unified modes when enabled.
• Cluster state exporter (kube-state-metrics) (charts/kube-state/templates/*): Scrapes namespace-level object metadata for Prometheus.
• Metrics gateway (Prometheus federation/auth) (charts/prometheus/templates/prometheus-federation-*): Adds the auth proxy, Service, and federation jobs necessary for the control plane to scrape metrics securely.
• Auxiliary services: External-es proxy, namespace pool RBAC, and other helper charts that only make sense near the workloads.

Prometheus, Postgres, and other shared charts still exist. In data mode, Prometheus pushes or exposes metrics back to the control plane rather than aggregating globally.

Network Endpoints

Data plane ingress typically includes:

• deployments.<domain-prefix>.<base-domain>: Airflow web UIs and APIs for each Deployment. Commander configures path-based routing for these Deployments.
• registry.<domain-prefix>.<base-domain>: When hosting the registry in the data plane.
• Commander endpoints for registering clusters and creating and updating Deployments.
• Optional vanity hostnames per Deployment managed by Commander’s Helm releases.
• Data plane metrics collector (Prometheus) serves a federate endpoint scraped by the control plane.

Outbound connections include:

• To Houston: Commander, Config Syncer, and cronjobs call the control plane API over TLS.
• To external registries/log stores: Depending on how runtime images and logs are hosted.

Control Plane Integration

Data planes authenticate against Houston using service accounts and tokens that the control plane provisions. The typical workflow is:

1. Registration: A platform admin registers a data plane entry in Houston, which generates unique Commander and Config Syncer tokens.
2. Secret distribution: During install you provide Houston tokens via astronomer.houston.config.dataplane.* values. The secret distribution job (Config Syncer) keeps runtime secrets fresh.
3. Deployment lifecycle: Houston API pushes requests to Commander, which reconciles Deployments. If the control plane issues upgrades or scale instructions, Commander applies them locally.
4. Telemetry forwarding: The metrics collector (Prometheus) and log forwarder (Vector) transport metrics and logs to the control plane (or third-party sinks) so administrators have a single-pane view.

Monitoring and Alerting

Data plane Prometheus scrapes:

• Airflow Deployments, including scheduler, webserver/API server, workers.
• Platform Pods, like Commander, Config Syncer, NGINX, and Vector.
• Kube-state-metrics for namespace-wide object counts.

Use Alertmanager rules provided by the chart or integrate with the control plane Alertmanager to drive notifications.

Comparison to Other Modes

Split deployments keep data plane blast radius confined to workload execution, letting you scale your data planes independently and enforce network boundaries to sensitive data resources.

• For the management plane overview see Control Plane Architecture.
• For a single-cluster footprint see Unified Architecture.

Next Steps

• Deploy a data plane using the Install data plane guide.
• Register the data plane with your control plane and verify Commander heartbeat.
• Configure DNS, TLS certificates, and networking policies based on the ingress endpoints.
• Integrate telemetry (metrics, logs) with your central observability tools.