> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Manage cluster status

Astro Private Cloud (APC) tracks the operational status of every data plane cluster so that workloads only run on healthy infrastructure. This page describes the cluster status values, how Houston determines status, the GraphQL operations for querying and updating status, and how to troubleshoot unhealthy clusters.

## Authenticate to Houston

Every operation on this page requires a Houston API token sent as a bearer credential. Send your token in the `Authorization` header on each request to the Houston GraphQL endpoint:

```bash wrap theme={null}
curl -X POST https://houston.<your-base-domain>/v1 \
  -H "Authorization: <your-token>" \
  -H "Content-Type: application/json" \
  -d '{"query": "query { self { user { username } } }"}'
```

For step-by-step instructions on obtaining a user token or creating a system service account token, see [Authenticate to the Houston API](/docs/astro-private-cloud/v-1-x/houston-api-authenticate).

## Required roles and permissions

Cluster operations are gated by RBAC permissions. The following table maps each operation to the permission Houston checks and the default role that grants it.

| Operation                     | Required permission      | Default role that grants access |
| ----------------------------- | ------------------------ | ------------------------------- |
| `paginatedClusters`           | Authenticated user       | Any signed-in user              |
| `cluster`                     | `system.clusters.get`    | System Admin                    |
| `updateCluster`               | `system.clusters.update` | System Admin                    |
| `reconcileClusterMetadataJob` | `system.clusters.update` | System Admin                    |

The System Admin role inherits every `system.clusters.*` permission.

## Cluster status values

| Status     | Description                                             | Allows new deployments | Allows configuration updates |
| ---------- | ------------------------------------------------------- | ---------------------- | ---------------------------- |
| `ACTIVE`   | Cluster is healthy and reachable                        | Yes                    | Yes                          |
| `INACTIVE` | Cluster is unreachable or reporting an unhealthy status | No                     | No                           |

## Status determination

Houston derives cluster status from the `healthStatus` field in Commander's `/metadata` response. The mapping is binary:

| Commander `healthStatus` | Houston cluster status |
| ------------------------ | ---------------------- |
| `HEALTHY`                | `ACTIVE`               |
| Any other value          | `INACTIVE`             |
| Fetch error or timeout   | `INACTIVE`             |

A CronJob in the control plane reconciles cluster metadata by calling Commander's `/metadata` endpoint. The default schedule is `0 * * * *` (every hour at minute 0), and is configurable through the `houston.syncDataplaneClusters.schedule` value on the Astronomer Helm chart.

```mermaid actions={true} wrap theme={null}
sequenceDiagram
    participant H as Houston (control plane)
    participant C as Commander (data plane)
    H->>C: GET /metadata
    C-->>H: { healthStatus: "HEALTHY", ... }
    Note over H: Map healthStatus → cluster status<br/>HEALTHY → ACTIVE, else → INACTIVE
```

You can list the reconcile CronJob and recent runs with the following command:

```bash wrap theme={null}
kubectl get cronjob,jobs -n astronomer | grep sync-dataplane-clusters
```

## Query cluster status

### List clusters

The `paginatedClusters` query returns clusters the caller has access to. Pagination uses the `take` argument, plus either `cursor` (a cluster UUID) or `pageNumber`. The response object contains a `clusters` list and a total `count`.

```graphql wrap theme={null}
query {
  paginatedClusters(
    take: 50
    status: ACTIVE
  ) {
    clusters {
      id
      name
      status
      statusReason
      healthStatus
      k8sVersion
      cloudProvider
      region
      createdAt
      updatedAt
    }
    count
  }
}
```

### Get a single cluster

```graphql wrap theme={null}
query {
  cluster(id: "<cluster-id>") {
    id
    name
    status
    statusReason
    healthStatus
    k8sVersion
    cloudProvider
    region
    dpChartVersion
    commanderVersion
    config
    configOverride
  }
}
```

<Note>
  The `healthStatus` field returns a JSON object containing the full health payload Houston received from Commander, not a single string. The `statusReason` field is also a JSON object. See [Update cluster status](#update-cluster-status) for the shape Houston writes.
</Note>

### Filter by cloud provider and region

```graphql wrap theme={null}
query {
  paginatedClusters(
    status: INACTIVE
    cloudProvider: "aws"
    region: "us-east-1"
    take: 25
  ) {
    clusters {
      id
      name
      statusReason
    }
    count
  }
}
```

Other supported filter arguments include `searchPhrase`, `k8sVersion`, `id`, `sortBy`, and `sortDirection`.

## Update cluster status

A user with permission to update clusters can change a cluster's status manually. The `statusReason` argument accepts a JSON object whose shape isn't enforced by the schema, but Houston itself writes the value Commander returns in its `/metadata` response when reconciling. To stay consistent, use the same shape Houston uses or include a descriptive `message` field.

```graphql wrap theme={null}
mutation {
  updateCluster(
    id: "<cluster-id>"
    status: INACTIVE
    statusReason: { message: "Maintenance window — cluster offline for upgrades" }
  ) {
    id
    status
    statusReason
  }
}
```

For status changes, supply `id` (required), `status`, and `statusReason`. The `updateCluster` mutation also accepts `name` and `deploymentsConfigOverride` for non-status changes; see [Update data plane cluster configurations](/docs/astro-private-cloud/v-1-x/override-data-plane-cluster) for those workflows.

<Note>
  Houston blocks configuration updates (`deploymentsConfigOverride`, `name`) while the cluster status is `INACTIVE` and returns the error `This operation is not allowed as the cluster is not active.` Status itself can still be updated in any state.
</Note>

## Force a metadata reconciliation

Use the `reconcileClusterMetadataJob` query to make Houston refetch metadata from Commander immediately, instead of waiting for the next CronJob run. The query accepts a list of cluster UUIDs; if you pass `null` or omit the argument, Houston reconciles every cluster the caller is authorized to update.

```graphql wrap theme={null}
query {
  reconcileClusterMetadataJob(
    clusterIds: ["<cluster-id-1>", "<cluster-id-2>"]
  ) {
    successfulClusterIds
    failedClusterIds
    skippedClusterIds
  }
}
```

A cluster appears in `skippedClusterIds` when it lacks a dataplane URL or when the caller isn't authorized to reconcile it.

Use this query in the following situations:

* After resolving a network or DNS issue between the control plane and a data plane.
* After restarting Commander.
* To verify cluster health after a maintenance window.
* When debugging connectivity from the control plane.

## Troubleshoot unhealthy clusters

<Steps>
  <Step title="Check the cluster's current status">
    ```graphql wrap theme={null}
    query {
      cluster(id: "<cluster-id>") {
        status
        statusReason
        healthStatus
        updatedAt
      }
    }
    ```
  </Step>

  <Step title="Verify Commander connectivity">
    From a Pod in the control plane namespace with network access to Commander, call the metadata endpoint:

    ```bash wrap theme={null}
    curl -s https://<commander-url>/metadata | jq .
    ```

    A healthy response includes (among other fields) the following:

    ```json wrap theme={null}
    {
      "kubernetesVersion": "<k8s-version>",
      "baseDomain": "<cluster-base-domain>",
      "healthStatus": "HEALTHY",
      "cloudProvider": "<provider>",
      "region": "<region>",
      "dataplaneChartVersion": "<chart-version>",
      "commander": {
        "version": "<commander-version>",
        "url": "<commander-grpc-url>",
        "status": "HEALTHY",
        "airflowChartVersion": "<airflow-chart-version>"
      }
    }
    ```

    The full response also includes `mode`, `dataplaneUrl`, `dataplaneId`, `releaseName`, `releaseNamespace`, `dbType`, `namespacePools`, and `registry`.
  </Step>

  <Step title="Check Commander health and pods">
    ```bash wrap theme={null}
    curl -s https://<commander-url>/healthz
    ```

    ```bash wrap theme={null}
    kubectl get pods -n astronomer -l app=commander
    ```
  </Step>

  <Step title="Force a metadata refresh">
    ```graphql wrap theme={null}
    query {
      reconcileClusterMetadataJob(clusterIds: ["<cluster-id>"]) {
        successfulClusterIds
        failedClusterIds
        skippedClusterIds
      }
    }
    ```
  </Step>

  <Step title="Review Commander logs">
    Replace `<release-name>` with your Helm release name, which is `astronomer` by default:

    ```bash wrap theme={null}
    kubectl logs -n astronomer deployment/<release-name>-commander --tail=100
    ```
  </Step>
</Steps>

## Common issues and resolutions

### Cluster stuck in `INACTIVE`

Possible causes:

1. The Commander Pod isn't running.
2. Network connectivity between Houston and Commander is broken (firewall, DNS, service mesh).
3. TLS certificate problems on the metadata endpoint.
4. Commander's `/metadata` endpoint returns a non-2xx response or a payload without `healthStatus: "HEALTHY"`.

Resolution steps:

```bash wrap theme={null}
kubectl get pods -n astronomer -l app=commander
```

```bash wrap theme={null}
kubectl describe pod <commander-pod> -n astronomer
```

```bash wrap theme={null}
kubectl logs -n astronomer deployment/<release-name>-commander
```

Test connectivity from Houston (replace `<release-name>` with your Helm release, default `astronomer`):

```bash wrap theme={null}
kubectl exec -it deployment/<release-name>-houston -n astronomer -- \
  curl -v https://<commander-url>/metadata
```

After the underlying issue is resolved, force a reconciliation through the `reconcileClusterMetadataJob` query.

### Configuration updates rejected

Houston returns this error when a configuration update is attempted on a non-`ACTIVE` cluster:

```text wrap theme={null}
This operation is not allowed as the cluster is not active.
```

Resolution:

1. Confirm the cluster is reachable and run `reconcileClusterMetadataJob` to refresh status.
2. If the cluster reports healthy but Houston hasn't yet reconciled, wait for the next reconcile cycle or trigger one manually.
3. As a last resort, a System Admin can manually set the cluster status back to `ACTIVE`:

```graphql wrap theme={null}
mutation {
  updateCluster(
    id: "<cluster-id>"
    status: ACTIVE
    statusReason: { message: "Manually verified healthy" }
  ) {
    id
    status
  }
}
```

## Best practices

* Monitor cluster status proactively. Configure alerts for clusters transitioning to `INACTIVE` and surface status on operations dashboards.
* Always provide a meaningful `statusReason` when manually changing status. The reason is preserved in the cluster record and is useful when diagnosing later incidents.
* Distribute Deployments across multiple clusters so that a single `INACTIVE` cluster doesn't affect every workload.
* Validate connectivity from the control plane Pod after firewall, DNS, or certificate changes; don't rely on the next scheduled reconciliation to surface the problem.

## Related documentation

* [Authenticate to the Houston API](/docs/astro-private-cloud/v-1-x/houston-api-authenticate)
* [Use the Houston API on Astro Private Cloud](/docs/astro-private-cloud/v-1-x/houston-api)
* [Register a data plane](/docs/astro-private-cloud/v-1-x/register-data-plane)
* [Deregister a data plane cluster](/docs/astro-private-cloud/v-1-x/deregister-data-plane)
* [Update data plane cluster configurations](/docs/astro-private-cloud/v-1-x/override-data-plane-cluster)
* [Data plane architecture](/docs/astro-private-cloud/v-1-x/data-plane-architecture)