Show Remote Execution Agent task logs in Airflow UI

You can make task logs visible directly in the Airflow UI by exporting task logs to your object storage, and configuring the Astro API Server to securely retrieve them.

This requires:

Configuration in the Remote Execution Agent's values.yaml.
Deployment configuration in the Astro UI.
Proper workload identities for the Remote Execution Agent, write access, and the Astro API Server, read access.

AWS
GCP
Azure

Configure the following environment variables in the Helm chart's values.yaml:

commonEnv:
  - name: AIRFLOW__LOGGING__REMOTE_LOGGING
    value: "True"
  - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
    value: "astro_aws_logging"
  - name: AIRFLOW_CONN_ASTRO_AWS_LOGGING
    value: "s3://"
  - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
    value: "s3://<bucket>/<deployment-id>"
  - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
    value: "astronomer.runtime.logging.logging_config"
  - name: ASTRONOMER_ENVIRONMENT
    value: "cloud"

info

The path for the AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER value is configurable. This is only an example format.

Run helm upgrade to apply the change to your Agents.
In the Astro UI, navigate to your Deployment and click the Details tab. Click Edit in the Advanced section to access your logging configurations.
Select Bucket Storage in the Task Logs field and fill in the Bucket URL as s3://<bucket>/<deployment-id>. Or, use the path that you configured for AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER in your Remote Agent's Helm chart's values.yaml.
In the Workload Identity for Bucket Storage section, select Customer Managed Identity and follow the instructions to set up your Customer Managed Identity so that the identity you create has read access to the specified bucket and path.

Configure the following environment variables in the Helm chart's values.yaml:

commonEnv:
  - name: AIRFLOW__LOGGING__REMOTE_LOGGING
    value: "True"
  - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
    value: "astro_gcs_logging"
  - name: AIRFLOW_CONN_ASTRO_GCS_LOGGING
    value: "gcs://"
  - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
    value: "gs://<bucket>/<deployment-id>"
  - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
    value: "astronomer.runtime.logging.logging_config"
  - name: ASTRONOMER_ENVIRONMENT
    value: "cloud"

info

The path for the AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER value is configurable. This is only an example format.

Do a helm upgrade to apply the change to your Agents.
In the Astro UI, navigate to your Deployment and click the Details tab. Click Edit in the Advanced section.
In the Task Logs field, select Bucket Storage and fill in the Bucket URL as gs://<bucket>/<deployment-id>. Or, use the path that you configured for AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER in your Remote Agent's Helm chart's values.yaml.
In the Workload Identity for Bucket Storage section, select Customer Managed Identity and follow the instructions to set up your Customer Managed Identity so that the identity you create has read access to the specified bucket and path.

tip

On GCP, you can authorize your Deployment to cloud resources using workload identity with the alternative methods described here.

Configure the following environment variables in the Helm chart's values.yaml:

 commonEnv:
  - name: AIRFLOW__LOGGING__REMOTE_LOGGING
    value: "True"
  - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
    value: "astro_azure_logs_override"
  - name: AIRFLOW_CONN_ASTRO_AZURE_LOGS_OVERRIDE
    value: "wasb://<storage-account>@<storage-account>.blob.core.windows.net"
  - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
    value: "wasb-<storage-account>"
  - name: AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER
    value: "<storage-account-container-name>"
  - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
    value: "astronomer.runtime.logging.logging_config"
  - name: ASTRONOMER_ENVIRONMENT
    value: "cloud"

info

The path for the AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER value is configurable. This is only an example format.

Do a helm upgrade to apply the change to your Agents.
In the Astro UI, navigate to your Deployment and click the Details tab. Click Edit in the Advanced section.
Select Bucket Storage in the Task Logs field and fill in the Bucket URL as wasb-<storage-account>. Or, use the path that you configured for AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER in your Remote Agent's Helm chart's values.yaml.
In the Workload Identity for Bucket Storage section, select Customer Managed Identity and follow the instructions to set up your Customer Managed Identity so that the identity you create has read access to the specified bucket and path.
In the Astro UI, navigate to your Deployment and click the Environment tab. Click Edit Variables, then click Add Variable to add the following environment variables to your Deployment:

ASTRO_LOGGING_AZURE_CLIENT_ID: <Client ID for the managed identity that you want the API server to use to fetch task logs>
AIRFLOW_CONN_ASTRO_AZURE_LOGS_OVERRIDE: wasb://<storage-account>@<storage-account>.blob.core.windows.net
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: astro_azure_logs_override

Was this page helpful?