> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Sentinel for Remote Execution Agents

> Learn what the Sentinel service monitors, what it doesn't detect, and how to enable it for your Remote Execution Agents.

## Overview

The **Sentinel** service is a lightweight Go service that runs as its own single-replica Deployment in the same namespace as your Remote Execution Agent components (worker, Dag processor, triggerer). It provides real-time monitoring and operational visibility for agents running in your Kubernetes cluster.

Sentinel provides:

* **Health aggregation**: Every 30 seconds, Sentinel evaluates the health of all agent Pods in its namespace and sends a single aggregate heartbeat to the Astro control plane. The evaluation interval is set by `reconcile_interval`. The heartbeat cadence is set by the Astro control plane, which returns the interval to use in each heartbeat response.
* **Independent reporting path**: Sentinel observes agents from outside the agent process, so it keeps reporting infrastructure-level status even when an agent component is unhealthy or restarting.

<Info>
  Sentinel is included in the Remote Execution Agent Helm chart starting in version 1.2.0. Astronomer recommends enabling Sentinel for all Remote Execution deployments.
</Info>

Sentinel is namespace-scoped: each Sentinel instance only watches the Pods in its own namespace. If you run two agent deployments in separate namespaces, each has its own Sentinel instance.

<Note>
  The health data Sentinel reports is used internally by Astronomer Customer Support to monitor and triage agent issues. It isn't exposed to you in the Astro UI or any customer-facing interface.
</Note>

## What Sentinel monitors

Sentinel evaluates agent health using these sources:

| Source                     | What it checks                                                                                                                                                 |
| -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Kubernetes Pod status      | Whether each agent container is running and ready, and whether any Pod condition reports a failure                                                             |
| Kubernetes container state | Last termination reason, such as `OOMKilled` or `CrashLoopBackOff`, and exit codes                                                                             |
| Kubernetes Pod events      | Recent warning events for unhealthy Pods, which carry detail that Pod status doesn't, such as `Readiness probe failed: HTTP probe failed with statuscode: 503` |
| Agent `/status` endpoint   | An authenticated HTTP request to each Pod's `http` port that reads `checks.xcom_backend`, `checks.secrets_backend`, and `checks.state_store_backend`           |

If any Pod shows a non-Running or non-Ready state, or reports a backend connectivity failure through its `/status` endpoint, Sentinel reflects that in its aggregate heartbeat to the control plane. Pods that are still starting report an `initializing` state instead of an unhealthy one, so a normal rollout doesn't register as a failure.

The following table summarizes the failure scenarios that Sentinel detects:

| Scenario                             | Detected by Sentinel             |
| ------------------------------------ | -------------------------------- |
| Pod terminated by `OOMKilled`        | Yes — Kubernetes container state |
| `CrashLoopBackOff`                   | Yes — Kubernetes container state |
| Pod evicted or not scheduled         | Yes — Kubernetes Pod conditions  |
| Readiness or liveness probe failures | Yes — Kubernetes Pod events      |
| XCom backend unreachable             | Yes — `/status` endpoint check   |
| Secrets backend unreachable          | Yes — `/status` endpoint check   |
| State store backend unreachable      | Yes — `/status` endpoint check   |

## Understand monitoring limitations

Sentinel monitors infrastructure-level health. It doesn't have visibility into application-level connectivity failures that leave a Pod in a Running and Ready state. The following scenarios aren't detected by Sentinel:

| Scenario                                              | Why Sentinel doesn't detect it                                          |
| ----------------------------------------------------- | ----------------------------------------------------------------------- |
| Agent outbound heartbeat to the API server fails      | Application-internal state not exposed to Kubernetes                    |
| HTTP connection pool exhaustion                       | Pod appears healthy to Kubernetes while outbound connections are broken |
| Retry storms that exhaust the agent's connection pool | Application-internal state                                              |
| Agent deregistered by the control plane               | Control plane state not surfaced in the `/status` endpoint              |

In these cases, the affected Pod remains Running and Ready from a Kubernetes perspective, and the agent's `/status` endpoint continues to report healthy XCom, secrets, and state store backends. Sentinel's aggregate heartbeat shows a healthy status even though outbound connectivity to the control plane is broken.

<Note>
  If you observe agent connectivity issues where Pods appear healthy but tasks aren't executing, check the agent Pod logs for repeated `PoolTimeout` or HTTP error entries. These failures require a Pod restart to resolve. See [Remote Execution Agent failure and recovery scenarios](/docs/astro/remote-execution/remote-agents-failure-scenarios) for more details about heartbeat behavior during connectivity disruptions.
</Note>

## Set up Sentinel

Sentinel is enabled by default in Remote Execution Agent Helm chart version 2.0.0 and later. If you use an earlier chart version, enable it explicitly in your Helm chart configuration:

```yaml title="values.yaml" wrap theme={null}
sentinel:
  enabled: true
```

Astronomer recommends that you host the Sentinel image in your organization's registry and update the image reference in your Helm chart configuration.

To customize which agent Pods Sentinel observes, set `agent_component_app_label` in Sentinel's configuration. By default, Sentinel only monitors Pods labeled `app=astro-agent`, but you can restrict or broaden this scope as needed.

<Warning>
  The `sentinel.config` value replaces Sentinel's entire generated configuration instead of merging with it. If you set it, you must also supply every value the chart would otherwise generate, including `api_server.base_url`, the API token, `agent_auth`, and `health.port`. Omitting them prevents Sentinel from starting or from authenticating to the control plane.
</Warning>

## Verify Sentinel is running

After you enable Sentinel, confirm it's running in your cluster:

<Steps>
  <Step title="Check the Sentinel Pod status">
    Run the following command to verify the Sentinel Pod is running in your agent namespace:

    ```bash theme={null}
    kubectl get pods -n <your-agent-namespace> -l app=astro-agent,component=sentinel
    ```

    The output shows the Sentinel Pod in `Running` status.
  </Step>

  <Step title="Review Sentinel logs">
    Check the Sentinel logs to confirm it's sending heartbeats and evaluating agent health:

    ```bash theme={null}
    kubectl logs -n <your-agent-namespace> -l app=astro-agent,component=sentinel --tail=50
    ```

    Look for `Component health evaluation completed`, which Sentinel logs after each health evaluation, and `Heartbeat sent successfully`, which it logs after each heartbeat to the control plane. Both appear at the default `info` log level.

    If every backend check reports `unknown`, Sentinel can't authenticate to the agent `/status` endpoint. Confirm that the agent and Sentinel share the same value for `sentinelAuthSecret`.
  </Step>
</Steps>

## Security and scope

* Sentinel only observes Pods with a specific label in its namespace.
* All status data flows outbound to Astro. Sentinel doesn't require inbound connectivity from outside your cluster, but it does need in-namespace network access to each agent Pod's `http` port, and Kubernetes permissions to list and watch Pods and events in its namespace.
* No Dag code, task logs, or business data leaves your environment.