> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure Hashicorp Vault for data plane failover

This document explains how to configure [Hashicorp Vault](https://www.vaultproject.io/) as the external secrets store for Astro Private Cloud (APC) data plane failover, which is supported starting in Astro Private Cloud 2.1. Use it after you've read [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover) and before you complete [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover).

APC authenticates to Vault with Vault's Kubernetes auth method: ESO presents its Kubernetes service account token, and Vault validates it and issues a Vault token. AppRole and token auth aren't supported in Astro Private Cloud 2.1.

<Note>
  This document covers Vault as the `ClusterSecretStore` or `SecretStore` backend that the External Secrets Operator (ESO) uses to replicate Airflow secrets between data planes during failover. It's a different integration from using Vault as an [Airflow secrets backend](/docs/astro-private-cloud/v-2-x/secrets-backend-hashicorp) for Airflow variables and connections within a single Deployment. You can use either integration independently, or both together.
</Note>

## Prerequisites

* A Vault server reachable from every data plane cluster that will use it.
* Permission to create policies, auth mounts, and roles in Vault.
* The Vault CLI, or equivalent API access, authenticated with an admin or operator token.
* `kubectl` access to each data plane cluster. Depending on your ESO mode, you need permission to create either a cluster-scoped `ClusterSecretStore` or a namespaced `SecretStore`. See [Which secret store to create](#which-secret-store-to-create).

<Note>
  The manifests and commands here assume the platform is installed in the `astronomer` namespace. If you installed it into a different namespace, replace `astronomer` throughout, including `bound_service_account_namespaces` and the `serviceAccountRef.namespace`.
</Note>

## Which secret store to create

Vault works behind either an `external-secrets.io` `ClusterSecretStore` or a namespaced `SecretStore`. Which kind you create — a single shared store, a platform-synced store, or one per pool namespace — is determined by your External Secrets Operator mode, not by Vault. See [External Secrets Operator security](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security) to choose a mode, and the [manifests reference](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#secret-store-manifests-modes-1-and-2) for the store manifests. This page covers only the Vault-specific configuration you add to that store.

## Grant Vault access for data plane failover

APC currently offers ESO integration with Vault only for data plane failover, not as a standalone mechanism for syncing other secrets. The policy in this document grants read and write access unconditionally:

* ESO creates `ExternalSecret` resources on every failover-enabled cluster, which read the replicated secrets from Vault.
* The deployment orchestrator on the source cluster also creates `PushSecret` resources, which write the fernet key, environment variables, and database credentials to Vault so the destination cluster can pull them.

<Note>
  Enabling data plane failover (`global.dataPlaneFailover.enabled: true`) doesn't enable ESO by itself. ESO installs from its own Helm subchart, gated separately by `external-secrets.enabled`. See [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover).
</Note>

## Step 1: Create a Vault policy

[Create a Vault policy](https://developer.hashicorp.com/vault/docs/concepts/policies) that grants read and write access:

```hcl wrap theme={null}
path "secret/data/*" {
  capabilities = ["create", "read", "update", "list"]
}

path "secret/metadata/*" {
  capabilities = ["create", "read", "update", "list"]
}
```

This requires capabilities on both the `secret/data/*` and `secret/metadata/*` paths. `PushSecret` uses the metadata path to manage secret versions, so a policy that grants access only to `secret/data/*` still causes `PushSecret` writes to fail.

Write the policy to Vault:

```bash wrap theme={null}
vault policy write <policy-name> <path-to-policy-file>.hcl
```

## Step 2: Enable a Kubernetes auth mount for each data plane cluster

Vault's `auth.kubernetes` method validates a service account token against one specific Kubernetes API server per auth mount. If more than one data plane cluster authenticates to the same Vault instance, each cluster needs its own auth mount — a single mount can't validate tokens from more than one cluster.

1. Enable a dedicated mount for the cluster:

   ```bash wrap theme={null}
   vault auth enable -path=<data-plane-name>-kubernetes kubernetes
   ```

2. Configure the mount with that cluster's API server details:

   ```bash wrap theme={null}
   vault write auth/<data-plane-name>-kubernetes/config \
     kubernetes_host="<cluster-api-server-url>" \
     kubernetes_ca_cert=@<path-to-ca-cert> \
     token_reviewer_jwt=@<path-to-reviewer-jwt>
   ```

   Retrieve `kubernetes_host` from the cluster's API server endpoint, and retrieve `kubernetes_ca_cert` and `token_reviewer_jwt` from a service account token that Vault uses to validate other tokens. See [Vault's Kubernetes auth method documentation](https://developer.hashicorp.com/vault/docs/auth/kubernetes) for how to generate the reviewer JWT for your Kubernetes version.

Repeat this step for every data plane cluster that authenticates to this Vault instance, substituting a unique `-path` value each time.

## Step 3: Create a Vault role

Create a role that binds the policy from Step 1 to the auth mount from Step 2, scoped to the service account ESO runs as:

```bash wrap theme={null}
vault write auth/<data-plane-name>-kubernetes/role/<role-name> \
  bound_service_account_names=<eso-service-account> \
  bound_service_account_namespaces=astronomer \
  policies=<policy-name> \
  ttl=1h
```

`<eso-service-account>` is the Kubernetes service account that the ESO Pod runs as on this data plane cluster. If you installed ESO through the APC Helm chart's bundled subchart (`external-secrets.enabled: true`), this is the release's ESO service account in the `astronomer` namespace.

<Note>
  If the cluster uses [External Secrets Operator security](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security) Mode 3 (customer-managed isolated identity), map the per-namespace service account here instead of the shared ESO service account, so each Deployment authenticates to Vault as its own identity.
</Note>

## Step 4: Add the Vault provider to your secret store

Reference the auth mount and role you created from the `spec.provider.vault` block of your ESO secret store. The example below is a `ClusterSecretStore` (the shared-identity form). For a namespaced `SecretStore` — the platform-synced store in Mode 2 or a per-pool-namespace store in Mode 3 — use the identical `spec.provider.vault` block in the `SecretStore` shape shown in the [manifests reference](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security-manifests#secret-store-manifests-modes-1-and-2).

```yaml wrap theme={null}
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: astronomer-secret-store
spec:
  provider:
    vault:
      server: <vault-server-url>
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: <data-plane-name>-kubernetes
          role: <role-name>
          serviceAccountRef:
            name: <eso-service-account>
            namespace: astronomer
```

Use the same value for `metadata.name` on every data plane cluster that shares this Vault instance — this is the value you provide for `global.dataPlaneFailover.externalSecretManagerName` in your Helm values. Use the same `path` on every cluster as well, since it identifies the Vault secrets engine that holds the replicated secrets, and the destination cluster reads from the same location the source cluster writes to. The `auth.kubernetes.mountPath` and `auth.kubernetes.role` values differ per cluster.

## Network requirements

If your data plane clusters run in separate networks, Vault must be reachable from each of them over the network, not just from its own cluster's in-cluster DNS.

* Expose Vault through a network load balancer reachable from every data plane cluster that needs it.
* Open firewall or security group rules that allow traffic from each data plane cluster's CIDR range to the Vault endpoint.
* Confirm connectivity from each cluster before you rely on the configuration — see the verification steps in the following section.

## Verify the configuration

Before applying Helm values that depend on this `ClusterSecretStore` or `SecretStore`, confirm the setup works end to end.

1. Write a test secret to Vault:

   ```bash wrap theme={null}
   vault kv put secret/variables/test-variable value=test-value
   ```

2. Create a test `ExternalSecret` in the data plane cluster's `astronomer` namespace that references the store and reads the test secret. This example uses a `ClusterSecretStore`; on a namespace-pools cluster, set `secretStoreRef.kind` to `SecretStore` instead.

   ```yaml wrap theme={null}
   apiVersion: external-secrets.io/v1
   kind: ExternalSecret
   metadata:
     name: <test-secret-name>
     namespace: astronomer
   spec:
     refreshInterval: 10s
     secretStoreRef:
       name: astronomer-secret-store
       kind: ClusterSecretStore
     target:
       name: <test-secret-name>
       creationPolicy: Owner
     data:
       - secretKey: value
         remoteRef:
           key: variables/test-variable
           property: value
   ```

3. Confirm it syncs:

   ```bash wrap theme={null}
   kubectl get externalsecret <test-secret-name> -n astronomer
   ```

   The `STATUS` column should show `SecretSynced`. If it doesn't, check the ESO Pod logs for authentication or permission errors before continuing.

Repeat this test from each data plane cluster that shares the Vault instance, since each cluster authenticates through its own auth mount.

After you enable data plane failover (`dataPlaneFailover.enabled: true`), also complete the `PushSecret` and connection-secret checks in [Verify secret replication before a failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover#verify-secret-replication-before-a-failover). Those checks confirm write access, which the preceding `ExternalSecret` test doesn't cover.

## Troubleshooting

* **`PushSecret` reports a failure, or Airflow secrets don't appear on the destination cluster after a failover**: The Vault policy likely grants read-only access on a cluster where `dataPlaneFailover.enabled` is `true`. Update the policy to include `create` and `update` on both `secret/data/*` and `secret/metadata/*`, then retry.
* **`ExternalSecret` or `PushSecret` fails to authenticate from one data plane cluster but succeeds from another**: Confirm the failing cluster has its own Kubernetes auth mount and that the mount's `kubernetes_host` and reviewer JWT match that cluster, not another cluster's.
* **A cluster in a different network can't reach Vault**: Confirm Vault is exposed beyond in-cluster DNS for that network, and that firewall or security group rules allow traffic from the cluster's CIDR range. See [Network requirements](#network-requirements).

## Related documentation

* [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover)
* [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover)
* [External Secrets Operator security](/docs/astro-private-cloud/v-2-x/external-secrets-operator-security)
* [Run a failover upgrade](/docs/astro-private-cloud/v-2-x/run-failover-upgrade)
* [Trigger a data plane failover](/docs/astro-private-cloud/v-2-x/trigger-data-plane-failover)
* [Hashicorp Vault as an Airflow secrets backend](/docs/astro-private-cloud/v-2-x/secrets-backend-hashicorp)
* [External Secrets Operator Vault provider documentation](https://external-secrets.io/latest/provider/hashicorp-vault/#kubernetes-authentication)