View task-level utilization metrics in the Airflow UI
View task-level utilization metrics in the Airflow UI
Astro can show CPU and memory utilization for individual Dags, tasks, and task instances directly in the Airflow UI. This is different from the Deployment-wide worker, scheduler, and triggerer metrics available on the Astro UI’s Analytics page. See View metrics for Astro Deployments.
Prerequisites
- An Astro Deployment running the Astro Executor on:
- Hosted execution mode with Astro Runtime 3.1+, or
- Remote Execution mode on Astro Runtime 3.1+, using a Remote Execution Agent 1.5.0+ and Helm chart 2.0.0+. See Helm chart versioning for the Agent-to-chart compatibility matrix.
What you’ll see
A Resource Metrics tab appears in the Airflow UI with three views:
- Dag: Aggregated CPU and memory usage across all tasks in a Dag.
- Task: CPU and memory usage for a specific task across its recent runs.
- Task Instance: CPU and memory usage for a single task run, mapped to the worker that ran it.
The Resource Metrics tab supports both light and dark mode. Use it to answer questions like:
- Which tasks are consuming the most CPU or memory?
- Which tasks are likely causing out-of-memory (OOM) errors?
- Which Dags are driving infrastructure cost?
- Which tasks are good candidates for right-sizing or optimization?
How Astro collects task-level metrics
Astro samples task-level metrics on their own interval, separate from the interval used for Deployment-wide worker metrics. For long-running tasks, samples can lag up to five seconds behind the task.
This sampling shapes what each metric represents:
- CPU values account for all work the task performs, but averaging across the sampling interval smooths very short-lived peaks.
- Memory values reflect usage at each sample, so a spike shorter than the sampling interval on an otherwise idle task doesn’t appear.
Astro collects metrics for work that runs in the task slot of an Astro Executor worker, and the Resource Metrics tab shows the most recent seven days of data.