You can make task logs visible directly in the Airflow UI by exporting task logs to your object storage, and configuring the Astro API Server to securely retrieve them.
This requires:
values.yaml
.The Astro Orchestration Plane provides secure private connectivity with a pre-configured S3 Gateway Endpoint.
values.yaml
, and replace the path for the AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
value with your information:If you do not use workload identity and instead want to manually mount a credential, you must also add the following environment variable defining the location of a token file to your Remote Agent’s values.yaml
file. You can customize the file path, /tmp/logging-token
, to the name of your logging token file.
Run helm upgrade
to apply the change to your Agents.
In the Astro UI, navigate to your Deployment and click the Details tab. Click Edit in the Advanced section to access your logging configurations.
Select Bucket Storage in the Task Logs field and fill in the Bucket URL as s3://<bucket>/<deployment-id>
. Or, use the path that you configured for AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
in your Remote Agent’s Helm chart’s values.yaml
.
In the Workload Identity for Bucket Storage section, select Customer Managed Identity and follow the instructions to set up your Customer Managed Identity so that the identity you create has read access to the specified bucket and path.
(Optional) If your log bucket is in a different region from your Astro Deployment, you need to define the AWS region in the AIRFLOW__ASTRONOMER_PROVIDERS_LOGGING__AWS_REGION
environment variable for Astronomer-managed components. In the Astro UI, navigate to your Deployment and click the Environment tab. Click Environment Variables, then click (+) Environment Variable to add the following environment variables to your Deployment:
AIRFLOW__ASTRONOMER_PROVIDERS_LOGGING__AWS_REGION <The region in which the S3 bucket is configured>