Manage DeploymentsRemote ExecutionConfigure logging

Show Remote Execution Agent task logs in Airflow UI

You can make task logs visible directly in the Airflow UI by exporting task logs to your object storage, and configuring the Astro API Server to securely retrieve them.

This requires:

  • Configuration in the Remote Execution Agent’s values.yaml.
  • Deployment configuration in the Astro UI.
  • Proper workload identities for the Remote Execution Agent, write access, and the Astro API Server, read access.

The Astro Orchestration Plane provides secure private connectivity with a pre-configured S3 Gateway Endpoint.

  1. Configure the following environment variables in the Helm chart’s values.yaml, and replace the path for the AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER value with your information:
1commonEnv:
2  - name: AIRFLOW__LOGGING__REMOTE_LOGGING
3    value: "True"
4  - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
5    value: "astro_aws_logging"
6  - name: AIRFLOW_CONN_ASTRO_AWS_LOGGING
7    value: "s3://"
8  - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
9    value: "s3://<bucket>/<deployment-id>"
10  - name: AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS
11    value: "astronomer.runtime.logging.logging_config"
12  - name: ASTRONOMER_ENVIRONMENT
13    value: "cloud"
Mounting credentials manually

If you do not use workload identity and instead want to manually mount a credential, you must also add the following environment variable defining the location of a token file to your Remote Agent’s values.yaml file. You can customize the file path, /tmp/logging-token, to the name of your logging token file.

1  - name: ASTRO_LOGGING_AWS_WEB_IDENTITY_TOKEN_FILE
2    value: "/tmp/logging-token"

  1. Run helm upgrade to apply the change to your Agents.

  2. In the Astro UI, navigate to your Deployment and click the Details tab. Click Edit in the Advanced section to access your logging configurations.

  3. Select Bucket Storage in the Task Logs field and fill in the Bucket URL as s3://<bucket>/<deployment-id>. Or, use the path that you configured for AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER in your Remote Agent’s Helm chart’s values.yaml.

  4. In the Workload Identity for Bucket Storage section, select Customer Managed Identity and follow the instructions to set up your Customer Managed Identity so that the identity you create has read access to the specified bucket and path.

  5. (Optional) If your log bucket is in a different region from your Astro Deployment, you need to define the AWS region in the AIRFLOW__ASTRONOMER_PROVIDERS_LOGGING__AWS_REGION environment variable for Astronomer-managed components. In the Astro UI, navigate to your Deployment and click the Environment tab. Click Environment Variables, then click (+) Environment Variable to add the following environment variables to your Deployment:

  • AIRFLOW__ASTRONOMER_PROVIDERS_LOGGING__AWS_REGION <The region in which the S3 bucket is configured>