> ## Documentation Index > Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt > Use this file to discover all available pages before exploring further. # Scrape metrics from Remote Execution Agents **Airflow 3** This feature is only available for Airflow 3.x Deployments. This guide lists available metrics when running Astro's Remote Execution Agents and explains how to scrape metrics using OpenTelemetry and Prometheus. ## What you can monitor A self-managed Prometheus instance can collect the following classes of metrics from agent components: * **Agent client metrics** that each agent component exposes on its own `/metrics` endpoint, including a `component_health` gauge and heartbeat, agent proxy, and Python runtime metrics. The Dag processor also exposes parsing-pipeline metrics. See [Agent client metrics](#agent-client-metrics) for the full list. * **Airflow application metrics** that Airflow emits in StatsD format from the worker, Dag processor, and triggerer. Examples include scheduler heartbeats, task instance state counts, and Dag parse times. * **Kubernetes infrastructure metrics** for the agent Pods, such as CPU, memory, and Pod status. These come from `kube-state-metrics`, `cAdvisor`, and `node-exporter` on your cluster. * **Sentinel runtime metrics** Sentinel reports agent and integration health back to Astro, and you can scrape its Pod for local observability. Requires [Sentinel to be enabled](/docs/astro/remote-agents-sentinel). For metrics that Astro generates outside your cluster, such as orchestration plane scheduler activity, see [Export metrics from Astro](/docs/astro/export-metrics). ## Prerequisites * A Remote Execution Agent (running Agent Client version `1.7.0` and above) installed in your Kubernetes cluster. See [Register and configure agents](/docs/astro/remote-execution-configure-agents). * A running Prometheus deployment in the same cluster, or one that can reach the agent namespace over the network. The Prometheus Operator with `PodMonitor` or `ServiceMonitor` resources is supported, and so is a standalone Prometheus that uses static scrape configs. * Cluster-level access to deploy Helm chart updates and, if you use the Prometheus Operator, to create custom resources. * `kube-state-metrics` and `cAdvisor` available in your cluster, if you want to collect Kubernetes infrastructure metrics. ## Monitor Airflow application metrics Airflow can expose metrics to [OpenTelemetry](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#setup-opentelemetry) or [StatsD](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#setup-statsd). This guide covers a minimal monitoring example using OpenTelemetry. Features such as Kubernetes namespace isolation are out of scope for this guide. Code examples assume the components are deployed in a Kubernetes namespace named `re`. See additional configuration here: * [Airflow StatsD metrics example](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/metrics.html#setup-statsd) * [OpenTelemetry Collector configuration](https://opentelemetry.io/docs/collector/configuration) ### Step 1: Install OpenTelemetry Dependency Add `apache-airflow[otel]` to `requirements-client.txt` and deploy the Remote Execution agents (using `astro remote deploy`). ### Step 2: Install OpenTelemetry This step installs an OpenTelemetry Collector on your Kubernetes cluster using a Helm chart. If you already have OpenTelemetry running, you can skip this step. From your terminal, add the `open-telemetry` Helm repository and download the latest metadata: ```sh wrap theme={null} helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm repo update ``` Create a YAML file named for example `otel-values.yaml` for OpenTelemetry configuration: ```yaml title="otel-values.yaml" expandable wrap theme={null} mode: deployment image: repository: otel/opentelemetry-collector-contrib command: name: otelcol-contrib config: receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 exporters: debug: verbosity: detailed prometheus: endpoint: 0.0.0.0:8889 service: pipelines: metrics: receivers: [otlp] exporters: [debug, prometheus] ports: otlp-http: enabled: true containerPort: 4318 servicePort: 4318 protocol: TCP prometheus: enabled: true containerPort: 8889 servicePort: 8889 protocol: TCP ``` Install the OpenTelemetry Collector: ```sh wrap theme={null} helm install otel-collector open-telemetry/opentelemetry-collector -n re -f otel-values.yaml ``` ### Step 3: Configure Airflow to ship metrics to OpenTelemetry Configure these environment variables in your Remote Execution `values.yaml` under `commonEnv`: ```yaml title="values.yaml" wrap theme={null} commonEnv: - name: AIRFLOW__METRICS__OTEL_ON value: "True" - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://otel-collector-opentelemetry-collector.re.svc.cluster.local:4318 - name: OTEL_EXPORTER_OTLP_PROTOCOL value: http/protobuf ``` This ensures Airflow services push metrics to the OTLP endpoint. Upgrade your Remote Execution Helm chart: ```sh wrap theme={null} helm upgrade astro-agent astronomer/astro-remote-execution-agent --values values.yaml ``` ### Step 4: Verify metrics At this stage, Airflow application metrics should arrive in the OTLP collector. You can verify this by port forwarding the otel-collector: ```sh wrap theme={null} kubectl port-forward -n re deploy/otel-collector-opentelemetry-collector 8889:8889 ``` Browse to [http://localhost:8889/metrics](http://localhost:8889/metrics) and check if you observe any metrics. Note that you might need to run some Airflow tasks for more metrics to show. Here’s an example of metrics that you might observe: ```text wrap expandable theme={null} # HELP airflow_airflow_io_load_filesystems # TYPE airflow_airflow_io_load_filesystems gauge airflow_airflow_io_load_filesystems{job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version=""} 209.85527300035756 # HELP airflow_dag_example_astronauts_get_astronauts_duration # TYPE airflow_dag_example_astronauts_get_astronauts_duration gauge airflow_dag_example_astronauts_get_astronauts_duration{job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version=""} 3903.361 # HELP airflow_operator_successes_pythondecoratedoperator_total # TYPE airflow_operator_successes_pythondecoratedoperator_total counter airflow_operator_successes_pythondecoratedoperator_total{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 1 # HELP airflow_operator_successes_total # TYPE airflow_operator_successes_total counter airflow_operator_successes_total{dag_id="example_astronauts",job="airflow",operator="_PythonDecoratedOperator",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 1 # HELP airflow_serde_load_serializers # TYPE airflow_serde_load_serializers gauge airflow_serde_load_serializers{job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version=""} 0.8434970004600473 # HELP airflow_task_duration # TYPE airflow_task_duration gauge airflow_task_duration{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 3903.361 # HELP airflow_ti_finish_example_astronauts_get_astronauts_success_total # TYPE airflow_ti_finish_example_astronauts_get_astronauts_success_total counter airflow_ti_finish_example_astronauts_get_astronauts_success_total{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 1 # HELP airflow_ti_finish_total # TYPE airflow_ti_finish_total counter airflow_ti_finish_total{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",state="success",task_id="get_astronauts"} 1 # HELP airflow_ti_start_example_astronauts_get_astronauts_total # TYPE airflow_ti_start_example_astronauts_get_astronauts_total counter airflow_ti_start_example_astronauts_get_astronauts_total{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 1 # HELP airflow_ti_start_total # TYPE airflow_ti_start_total counter airflow_ti_start_total{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 1 # HELP airflow_ti_successes_total # TYPE airflow_ti_successes_total counter airflow_ti_successes_total{dag_id="example_astronauts",job="airflow",otel_scope_name="airflow.sdk._shared.observability.metrics.otel_logger",otel_scope_schema_url="",otel_scope_version="",task_id="get_astronauts"} 1 ``` If you see metrics similar to the above, your setup is successful. From here, you can configure your metrics backend such as Prometheus to scrape the metrics from `otel-collector:8889`. A common metrics/monitoring setup is Prometheus as the metrics backend, plus Grafana for visualization. ## Agent client metrics Each agent component (worker, Dag processor, and triggerer) runs an internal HTTP server that exposes Prometheus-format metrics directly on a `/metrics` endpoint, on port `39091` by default (configurable through the setting `http_server.port`, or using the environment variable `ASTRO_AGENT_CLIENT_HTTP_SERVER__PORT`). These metrics describe the agent client's own runtime, including its health, its heartbeat traffic with the Astro orchestration plane, and the Python process it runs in. Agent client metrics are independent of the Airflow application metrics described in [Monitor Airflow application metrics](#monitor-airflow-application-metrics), and you can scrape both endpoints from the same Prometheus instance. The most important health signal is the `component_health` gauge. Each internal subsystem reports `1` when healthy and `0` when unhealthy, for example: ```text wrap theme={null} component_health{component="TriggererHeartbeater"} 1.0 component_health{component="TriggererProc"} 1.0 component_health{component="Server"} 1.0 ``` The `component` label values vary by agent client. For example, the worker reports its own set of subsystems, and the Dag processor reports subsystems for the parsing pipeline. Alert when any `component_health` series drops to `0`. If you are using Prometheus to scrape the metrics of your Agent clients, you can configure the Helm chart's `annotations` with the following annotations to ensure that your metrics get collected: ```yaml title="values.yaml" wrap theme={null} # inside your values.yaml file # make sure to update the port annotation if you changed the default value annotations: prometheus.io/scrape: "true" prometheus.io/path: "/metrics" prometheus.io/scheme: "http" prometheus.io/port: "39091" ``` ### Generic metrics All three agent clients ship a common set of metrics that cover Python runtime, process resources, heartbeat traffic with the API server, queue state, and the agent proxy. | Metric | Type | Description | | ------------------------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------- | | `component_health` | Gauge | Health status of agent subsystems. `1` is healthy and `0` is unhealthy. | | `heartbeat_requests_total` | Counter | Total heartbeat attempts, with `component` and `outcome` labels (`success`, `timeout`, `error`). | | `heartbeat_duration_seconds` | Histogram | Time from the start of a heartbeat request to the receipt of the provider response. | | `heartbeat_payload_bytes` | Histogram | Size in bytes of the serialized heartbeat request body before it is sent. | | `heartbeat_requests_received_total` | Counter | Total heartbeat requests received from tasks. | | `heartbeat_requests_sent_total` | Counter | Total heartbeat requests sent to the API server. | | `heartbeat_requests_error_total` | Counter | Total heartbeat requests that failed to be sent to the API server. | | `matched_proxy_errors_total` | Counter | Total proxy failures for matched routes, with `route` and `error_type` labels. | | `astro_agent_client_queue_stats` | Gauge | Number of tasks in each state in each queue. | | `astro_agent_proxy_http_requests_total` | Counter | Total agent proxy requests, with `method`, `status`, and `handler` labels. | | `astro_agent_proxy_http_request_duration_seconds` | Histogram | Agent proxy request latency, with `handler` and `method` labels. Use this when aggregation by handler matters. | | `astro_agent_proxy_http_request_duration_highr_seconds` | Histogram | High-resolution agent proxy request latency for accurate percentile calculations. | | `astro_agent_proxy_http_request_size_bytes` | Summary | Content length of incoming agent proxy requests, by `handler`. | | `astro_agent_proxy_http_response_size_bytes` | Summary | Content length of outgoing agent proxy responses, by `handler`. | | `python_info` | Gauge | Python platform information, including the interpreter implementation and version labels. | | `python_gc_objects_collected_total` | Counter | Objects collected during garbage collection, by `generation`. | | `python_gc_objects_uncollectable_total` | Counter | Uncollectable objects found during garbage collection, by `generation`. | | `python_gc_collections_total` | Counter | Number of times each generation was collected. | | `process_cpu_seconds_total` | Counter | Total user and system CPU time, in seconds. | | `process_virtual_memory_bytes` | Gauge | Virtual memory size, in bytes. | | `process_resident_memory_bytes` | Gauge | Resident memory size, in bytes. | | `process_start_time_seconds` | Gauge | Start time of the process since the Unix epoch, in seconds. | | `process_open_fds` | Gauge | Number of open file descriptors. | | `process_max_fds` | Gauge | Maximum number of open file descriptors. | ### Dag processor metrics In addition to the generic metrics, the Dag processor exposes parsing-pipeline metrics. | Metric | Type | Description | | ----------------------------------------- | ------- | --------------------------------------------------------------------------------------------------- | | `dag_processor_heartbeat_dags_sent_total` | Counter | Total Dags included in heartbeat requests. | | `dag_processor_results_queue_depth` | Gauge | Number of parsed Dag results waiting in the coordinator queue. Use it as a back-pressure indicator. | | `dag_processor_cache_hits_total` | Counter | Total Dags skipped because their `dag_hash` matched the cached value. | | `dag_processor_cache_misses_total` | Counter | Total Dags processed because they were new, changed, or cold in the cache. | | `dag_processor_cache_size` | Gauge | Number of entries in the Dag processor `dag_hashes` cache. | **Note** that the `heartbeat_*` metrics described earlier in the `Generic Metrics` section are also available for the Dag Processor component, and can be filtered by `component=dag_processor`. ## Related documentation * [Sentinel for Remote Execution Agents](/docs/astro/remote-agents-sentinel) * [Helm chart configuration reference](/docs/astro/remote-agents-helm-reference) * [Export metrics from Astro](/docs/astro/export-metrics) * [Register and configure agents](/docs/astro/remote-execution-configure-agents)