> ## Documentation Index > Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt > Use this file to discover all available pages before exploring further. # Run a failover upgrade Deployments created before you [enabled data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover) on a cluster don't have the inactive database connection that failover needs — only Deployments created after you enable it get one automatically. Running a failover upgrade retrofits existing Deployments with that missing connection so they become eligible for failover, without you needing to recreate them. For a conceptual overview of failover, see [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover). The examples on this page show common ways to use the APC API — they're not a complete API reference. For the full, interactive API documentation for your installation, including every available query, mutation, and type, go to `https://houston./v1`, and click on the **Docs** tab. See [Develop and test APC API queries](/docs/astro-private-cloud/v-2-x/houston-api-develop-test) for more on using the built-in GraphQL explorer. ## Prerequisites * You've enabled data plane failover on the cluster. See [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover). * The Deployments you want to upgrade use image-based deploys. Deployments that use git-sync or Dag-only deploy mechanisms aren't eligible. * The Deployments aren't cordoned, and the cluster doesn't have a failover in progress. If you hard-delete a Deployment that used a manually specified (custom) release name on a failover-enabled cluster, the secrets pushed to your external secrets store aren't removed immediately — they enter that provider's soft-delete window first (for AWS Secrets Manager, 30 days by default, configurable down to a minimum of 7 days). If you recreate a Deployment with the **same** release name during that window, every other component comes up normally, but the `PushSecret` for that Deployment fails, because the secret name is still reserved by the pending deletion. Either wait out the provider's recovery window, force-delete the secrets from the store to skip it, or use a different release name when recreating the Deployment. ## Upgrade one Deployment ### From the APC UI 1. Open the Deployment you want to upgrade. 2. Click **Failover Upgrade**. ### Using the APC API Use the `upgradeDeploymentsForFailover` mutation with `deploymentIds` set to the Deployment's ID: ```graphql wrap theme={null} mutation upgradeDeploymentsForFailover( $deploymentIds: [Uuid] ) { upgradeDeploymentsForFailover( deploymentIds: $deploymentIds ) { missions { deploymentId missionId __typename } skipped { deploymentId reason __typename } __typename } } { "deploymentIds": ["deployment-id-1"] } ``` ## Upgrade multiple Deployments ### From the APC UI 1. Go to the cluster's Deployments list. 2. Select the Deployments you want to upgrade, or select all of them to upgrade every eligible Deployment on the cluster at once. 3. Click **Failover Upgrade**. ### Using the APC API Provide exactly one of `clusterId` or `deploymentIds`. Providing both, or neither, returns an error. To upgrade every eligible Deployment on a cluster at once, pass `clusterId`: ```graphql wrap theme={null} mutation upgradeDeploymentsForFailover( $clusterId: Uuid ) { upgradeDeploymentsForFailover( clusterId: $clusterId ) { missions { deploymentId missionId __typename } skipped { deploymentId reason __typename } __typename } } { "clusterId": "cluster-id" } ``` To upgrade a specific set of Deployments instead, pass `deploymentIds` and omit `clusterId`: ```graphql wrap theme={null} mutation upgradeDeploymentsForFailover( $deploymentIds: [Uuid] ) { upgradeDeploymentsForFailover( deploymentIds: $deploymentIds ) { missions { deploymentId missionId __typename } skipped { deploymentId reason __typename } __typename } } { "deploymentIds": ["deployment-id-1", "deployment-id-2"] } ``` ## What happens APC upgrades each selected Deployment individually. A Deployment that's already upgraded, cordoned, or using an unsupported deploy type is skipped automatically — it doesn't block the other Deployments in the same batch. If you used the API, the mutation response lists each Deployment APC started an upgrade for (`missions`, with a `missionId` you can use to track that upgrade) and each Deployment APC didn't upgrade (`skipped`, with a `reason`): | Reason | Meaning | | ------------------------------ | --------------------------------------------------------------------- | | `ALREADY_UPGRADED` | The Deployment already has the inactive database connection it needs. | | `UNSUPPORTED_DEPLOY_TYPE` | The Deployment doesn't use an image-based deploy. | | `CLUSTER_NOT_FAILOVER_CAPABLE` | The Deployment's cluster doesn't have data plane failover enabled. | | `CLUSTER_FAILOVER_IN_PROGRESS` | A cluster-level failover is currently in progress. | | `DEPLOYMENT_CORDONED` | The Deployment is cordoned. | | `CLUSTER_CORDONED` | The Deployment's cluster is cordoned. | | `CURRENT_MISSION_CONFLICT` | The Deployment already has another migration in progress. | If you used the UI, the cluster shows an overall failover upgrade status so you can confirm when every eligible Deployment has finished: * **Not started**: You haven't upgraded any eligible Deployments yet. * **In progress**: An upgrade is running for at least one Deployment. * **Complete**: Every eligible Deployment on the cluster has the inactive database connection it needs for failover. ## Check failover readiness Each Deployment on a failover-enabled cluster shows a failover-readiness shield in the Deployments list. The shield tells you, at a glance, whether that Deployment can fail over right now, still needs a failover upgrade, or needs your attention first: * **No shield**: Failover isn't enabled on the Deployment's cluster, so readiness doesn't apply. * **Gray — Needs upgrade**: The Deployment is on a failover-enabled cluster but hasn't been upgraded yet. Run a failover upgrade to make it eligible. * **Yellow — Upgrading**: A failover upgrade is currently running for the Deployment. * **Green — Ready**: The Deployment has everything it needs for failover — the inactive database connection, an image-based deploy, and its secrets synced to the external secrets store. * **Red — Needs attention**: Either the Deployment's last failover upgrade failed, it's no longer eligible after being upgraded, or its secrets failed to sync to the external secrets store (a `PushSecret` failure). A `PushSecret` failure can also happen if you recently recreated the Deployment with the same manually specified release name as a hard-deleted one — see the note under [Prerequisites](#prerequisites). See [Verify secret replication before a failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover#verify-secret-replication-before-a-failover). ### Debug a red shield Start by checking the `PushSecret` resources in the Deployment's namespace: ```bash wrap theme={null} kubectl -n get pushsecrets ``` A healthy Deployment shows every `PushSecret` as `Synced`: ```text wrap theme={null} NAME AGE STATUS LAST SYNC -active-metadata 41s Synced 40s -active-result-backend 41s Synced 40s -elasticsearch 41s Synced 41s -env 41s Synced 41s -fernet-key 41s Synced 41s -inactive-metadata 41s Synced 40s -inactive-result-backend 40s Synced 39s ``` If one instead shows an error, or no status at all, describe it to see why: ```bash wrap theme={null} kubectl -n describe pushsecret ``` Check the `Events` section at the bottom of the output for the sync failure reason — typically a permission error from the external secrets store rejecting the write, or (per the note under [Prerequisites](#prerequisites)) a secret name still reserved by a pending soft-deletion. ## Related documentation * [Data plane failover](/docs/astro-private-cloud/v-2-x/data-plane-failover) * [Enable data plane failover](/docs/astro-private-cloud/v-2-x/enable-data-plane-failover) * [Trigger a data plane failover](/docs/astro-private-cloud/v-2-x/trigger-data-plane-failover)