Skip to main content
This guide walks you through enabling data plane failover on an existing Astro Private Cloud (APC) installation. You configure the control plane and each participating data plane cluster separately. For a conceptual overview of the feature and its components, see Data plane failover.

​
Prerequisites

  • A working APC installation with at least one control plane cluster (global.plane.mode: control) and at least two data plane clusters (global.plane.mode: data).
  • A database server hostname that is network-accessible from both the source and destination data plane clusters. APC provisions the logical databases automatically, but the server itself must be reachable from both clusters. For supported topologies, see Database requirements.
  • An external secrets store supported for APC data plane failover. APC currently supports AWS Secrets Manager and Google Cloud Secret Manager through the External Secrets Operator (ESO). Hashicorp Vault support is introduced in Astro Private Cloud 2.1.
  • The External Secrets Operator (ESO) secret store configured on each data plane cluster — the ClusterSecretStore, or a synced SecretStore when you use namespace pools. See Configure External Secrets Operator security. The name you give the secret store is the value you provide for global.dataPlaneFailover.externalSecretManagerName.
  • A single externally managed container registry endpoint, configured on the control plane, that serves the Apache Airflow Deployment images used by your Deployments to every region where a data plane may run, with the same repository paths and tags in each region. Every data plane cluster must be able to pull from this endpoint. APC currently supports only one registry endpoint per APC installation. For details, see Container registry requirements. To configure the registry backend, see Use a registry backend.
  • An external sink for Airflow logs — an external Elasticsearch instance — that is reachable from every data plane cluster, so that task logs remain accessible after a Deployment moves between data planes. For supported topologies, see Airflow log sink requirements. For details on how Astro Private Cloud collects and exports task logs to Elasticsearch, see Configure task log collection and exporting to ElasticSearch.
  • Helm 3.6 or later.
  • A Kubernetes ClusterRole for the identity running helm install or helm upgrade on each data plane cluster. ESO installs cluster-scoped CRDs, which require cluster-level permissions.
  • Access to your APC Helm values files.

​
Configure the External Secrets Operator secret store

Data plane failover uses the External Secrets Operator (ESO) to replicate each Deployment’s Airflow secrets between data plane clusters. Before you apply the failover Helm values, set up ESO and its secret store on each data plane cluster: the backend credentials or workload identity, the ClusterSecretStore (or a synced SecretStore when you use namespace pools), and — for Hashicorp Vault — the auth mount, policy, and role. See Configure External Secrets Operator security for the full setup and the available modes, and Configure Hashicorp Vault for data plane failover if your backend is Vault. The name you give the secret store is the value you set for global.dataPlaneFailover.externalSecretManagerName in the following steps.

​
Step 1: Configure the control plane

Add the following values to your control plane values.yaml. Setting global.dataPlaneFailover.enabled: true activates Navigator, DP-Link, and the APC API dispatcher when global.plane.mode is control.
Set externalSecretManagerName to the name of the secret store you created in the ESO setup (astronomer-secret-store in these examples). It must be identical on the control plane and every data plane cluster.
ESO isn’t required on the control plane. Don’t set external-secrets.enabled: true in your control plane values.

​
Step 2: Configure each data plane

Add the following values to each data plane values.yaml. Setting global.dataPlaneFailover.enabled: true activates Pilot and the Flightdeck database bootstrap when global.plane.mode is data.
Use the same value for externalSecretManagerName as on the control plane. Both clusters must reference the same ClusterSecretStore.
The external-secrets key enables the bundled ESO subchart, which installs cluster-scoped CRDs. The identity running helm upgrade must have a ClusterRole on the data plane cluster. If you already run ESO separately, set external-secrets.enabled: false and ensure your existing ESO installation recognizes the ClusterSecretStore that APC expects.
The deployment orchestrator bootstraps the Flightdeck database as an init container during startup. If the bootstrap fails, the deployment orchestrator Pod doesn’t start. Check the flightdeck-bootstrapper and flightdeck-db-migrations init container logs if the deployment orchestrator fails to come up after enabling this feature.

​
Step 3: Apply the changes

Apply the updated values to each cluster using helm upgrade. Upgrade the control plane first.
Run the same command for each data plane cluster, substituting the appropriate release name, namespace, and values file.

​
Step 4: Verify the deployment

After the upgrade completes, confirm that the new components are running on each cluster. On the control plane, verify that the following Pods are running:
On each data plane, verify that the deployment orchestrator started successfully and Pilot is running:
Check deployment orchestrator logs to confirm Flightdeck initialized correctly:

​
Advanced configuration

Changing any of the values in this section can meaningfully affect resource usage on your Kubernetes clusters and may adversely affect failover functionality. Change and test these values in a non-production environment before applying them to production.

​
Tune Pilot behavior

Pilot’s claim, retry, and circuit breaker behavior is configurable via environment variables. Set these under astronomer.pilot.env in your data plane values.yaml. For data planes with a larger number of Airflow Deployments (roughly 50 or more), or for cross-region failovers where each Deployment takes longer to come up because the deployment orchestrator has to pull container images from a remote-region registry endpoint or fetch secrets from a remote-region secrets backend, consider raising PILOT_MAX_INFLIGHT_PER_WORKER above the default of 5. A higher value lets Pilot bring more Deployments up on the destination cluster in parallel, which reduces overall failover time and helps amortize cross-region latency. Each in-flight flight runs additional work on the data plane cluster (secret syncs, Helm installs, and database operations) and consumes additional bandwidth to the registry and secrets store, so only raise this value if your data plane cluster has spare CPU, memory, and API server headroom and your registry/secrets backends can handle the extra concurrent traffic. Validate the new value in a non-production environment first.

​
Tune Navigator behavior

Navigator’s reconcile loop timing is configurable via environment variables. Set these under astronomer.navigator.env in your control plane values.yaml. DP-Link determines cluster health based on heartbeat age. Adjust these thresholds under astronomer.dpLink.env in your control plane values.yaml.

​
Tune the APC API dispatcher behavior

The APC API dispatcher dispatches flights from the control plane to the deployment orchestrator on each data plane. Its loop timing, concurrency, retry, and circuit breaker behavior are configurable through environment variables. Set these under astronomer.houston.env in your control plane values.yaml.

​
Dispatcher loop

​
Circuit breaker

​
Verify secret replication before a failover

A ClusterSecretStore (or SecretStore, if your cluster uses namespace pools) reporting READY=True confirms that ESO can reach and authenticate to the secrets store. It doesn’t confirm that ESO can write to it. If the secrets store policy is missing write permissions, the PushSecret write performed by the deployment orchestrator fails, but this failure is logged as a warning and doesn’t block Deployment provisioning. Nothing surfaces the problem until a failover is triggered and the destination cluster can’t find the secrets it needs. Before triggering a failover, confirm that PushSecret resources are succeeding on the source cluster:
Each PushSecret should report a Synced status. If any report a failure, review the secrets store’s write permissions before triggering a failover. Also confirm that the following connection secrets resolve successfully. Data plane failover fencing reads these directly, and a failover for a Deployment fails if any of them are missing or hold an invalid database connection string:
  • <release-name>-active-metadata
  • <release-name>-active-result-backend
  • <release-name>-inactive-metadata
  • <release-name>-inactive-result-backend