Show Remote Execution Agent task logs in Airflow UI

You can display task logs in the Airflow UI by exporting logs to object storage and configuring the Astro API Server to retrieve them. Start by enabling log display after task completion, then optionally extend the setup to stream logs in real time as tasks run.

This guide explains configuring post-task log display and expanding that configuration to support real-time log streaming.

Displaying task logs after task completion

Set up log uploading so logs are visible in the Airflow UI after task completion. This requires:

• Remote Execution Agent configuration (values.yaml)
• Astro UI Deployment configuration
• Workload identities: write access for the Remote Execution Agent, read access for the Astro API Server

1
commonEnv:
2
  - name: AIRFLOW__LOGGING__REMOTE_LOGGING
3
    value: "True"
4
  - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
5
    value: "astro_aws_logging"
6
  - name: AIRFLOW_CONN_ASTRO_AWS_LOGGING
7
    value: "s3://"
8
  - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
9
    value: "s3://<bucket>/<deployment-id>"
10
  - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
11
    value: "astronomer.runtime.logging.logging_config"
12
  - name: ASTRONOMER_ENVIRONMENT
13
    value: "cloud"

1
  - name: ASTRO_LOGGING_AWS_WEB_IDENTITY_TOKEN_FILE
2
    value: "/tmp/logging-token"

Displaying task logs during task execution

Once you have post-completion log visibility, you can enable real-time log display. Remote Execution prevents the Airflow API server from reading logs directly from workers until they reach object storage. Use Vector, included in the RE agent Helm chart, to upload partial logs while tasks are running.

Prerequisites

Before you configure Vector, ensure that your Remote Execution Deployment is already set up to upload task logs to object storage after task completion.

Enable Vector sidecar

Use Vector to watch for log file changes and upload updates to object storage during task execution.

In your Helm values.yaml:

• Set loggingSidecar.enabled to true:

1
loggingSidecar:
2
  enabled: true

• Configure loggingSidecar.volumeMounts:

1
loggingSidecar:
2
  volumeMounts:
3
    - name: task-logs
4
      mountPath: /var/log/airflow/task_logs
5
      readOnly: true
6
    - name: vector-data
7
      mountPath: /var/lib/vector

Configure AWS S3 log upload

• Configure loggingSidecar.config:

1
loggingSidecar:
2
  config: |
3
    # Vector configuration for Astronomer Remote Execution agents for uploading Airflow task logs to AWS S3
4
5
    sources:
6
      airflow_task_logs:
7
        type: file
8
        include:
9
          - /var/log/airflow/task_logs/**/*.log
10
        read_from: beginning
11
12
    transforms:
13
      strip_path_prefix:
14
        type: remap
15
        inputs: [airflow_task_logs]
16
        source: |
17
          .log_path, err = replace(.file, "/var/log/airflow/task_logs/", "")
18
          if err != null {
19
            abort
20
          }
21
22
    sinks:
23
      s3:
24
        # For more Vector AWS S3 configuration options, see https://vector.dev/docs/reference/configuration/sinks/aws_s3
25
        type: aws_s3
26
        inputs: [strip_path_prefix]
27
        bucket: # set bucket name e.g. airflow_logs
28
        region: # set AWS region e.g. us-east-2
29
        compression: none
30
        encoding:
31
          codec: text
32
        key_prefix: '{{ "{{" }} log_path {{ "}}" }}.'
33
        filename_time_format: "%Y-%m-%dT%H-%M-%S"
34
        filename_append_uuid: false
35
        batch:
36
          max_bytes: 1000000  # configure based on your storage costs and log frequency requirements - see Caveats section below
37
          timeout_secs: 10  # configure based on your storage costs and log frequency requirements - see Caveats section below

1
debug:
2
  type: console
3
  inputs: [strip_path_prefix]
4
  encoding:
5
    codec: json

• Configure workers[*].volumes:

1
volumes:
2
  - name: task-logs
3
    emptyDir: {}
4
  - name: vector-data
5
    emptyDir: {}

• Configure workers[*].volumeMounts:

1
volumeMounts:
2
  - name: task-logs
3
    mountPath: /usr/local/airflow/logs

• Configure triggerer.volumes:

1
volumes:
2
  - name: task-logs
3
    emptyDir: {}
4
  - name: vector-data
5
    emptyDir: {}

• Configure triggerer.volumeMounts:

1
volumeMounts:
2
  - name: task-logs
3
    mountPath: /usr/local/airflow/logs

• Set AIRFLOW__LOGGING__DELETE_LOCAL_LOGS in commonEnv:

1
commonEnv:
2
  - name: AIRFLOW__LOGGING__DELETE_LOCAL_LOGS
3
    value: "True"

Log upload process

Partial logs are uploaded and displayed as follows:

1. Airflow worker or triggerer writes local task log files, as set by AIRFLOW__LOGGING__LOG_FILENAME_TEMPLATE.
2. Vector watches /var/log/airflow/task_logs/**/*.log and uploads log changes in chunks while the task runs.
3. Vector appends a timestamp to the file name before uploading each chunk.
4. Airflow scans object storage for log chunks when displaying the UI log view.
5. The UI displays all log content to the user.

Caveats

Duplicate log storage

After task completion, Airflow uploads the complete log to object storage and deletes the local copy. This causes duplication:

1. Partial logs from Vector
2. Complete log from Airflow

The Airflow API server deduplicates log lines by timestamp and message. Only storage usage is affected; logs are displayed once.

Small file problem

High-frequency, small log file uploads can create many small objects. This may increase storage costs, load on object storage, or trigger rate limits. Adjust file size and upload frequency in your Vector config to balance performance and cost.

• AWS bills object retrieval at a 128KB minimum on certain storage classes (source).
• A large number of small objects means more object requests (PUTs, GETs, LISTs) and more load on metadata/indexing; this can result in rate limits or latency issues.

Ensure a proper balance between filesize/timeout and log upload frequency in your Vector config.