Export logs to a Secondary GCS Bucket

Export logs to a Secondary GCS Bucket

By forwarding Airflow task logs from your Astro Deployment to an additional, customer‑managed GCS bucket, you can keep redundant copies of your execution history, integrate with existing log‑processing pipelines, and satisfy compliance or retention requirements that extend beyond the built‑in Astro logs.

Prerequisites

• Your Deployment must run Astro Runtime 11.7.0 or later. See Upgrade Astro Runtime.
• Your image must include astronomer-providers-logging==1.6.4 or later. The secondary GCS logging feature was introduced in this version.
• You need a GCP account where you can create a GCP bucket and IAM resources.
• A role with the minimum required permissions

Minimum Required Permissions

For all authentication methods, the service account or identity needs the following minimum permissions on the GCS bucket:

• storage.objects.list: List matching log files
• storage.objects.create: To create new log files
• storage.objects.get: To read existing log files (for append operations)
• storage.objects.update: To update existing log files

These permissions are included in the predefined roles/storage.objectUser role, or you can create a custom role with just these specific permissions.

Overview of configuration options

There are three methods to configure your Astro deployment to write logs to a secondary GCS bucket:

1. Customer‑provided Workload Identity: Recommended if you already use this mechanism for other GCP services.
2. Service Account Impersonation: Use this approach if you want to use an existing service account which is different than your Deployment’s workload identity.
3. Astro-provided Workload Identity: Use this approach if you do not have an existing service account that your Deployment can impersonate.

The following environment variables are supported by this feature:

Option 1: Customer‑Provided Workload Identity

This approach uses Google Cloud Workload Identity to allow your Astro deployment to authenticate with Google Cloud using its Kubernetes service account identity.

1. Attach a service account to your Astro Deployment
2. Set the following environment variables in the Deployment:

1
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_BUCKET_ENABLED=true
2
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_BASE_LOG_FOLDER="gs://your-bucket-name/logs"

Option 2: Service Account Impersonation

Use this approach if you want to use an existing service account which is different than your Deployment’s workload identity.

1. Follow the steps to setup service account impersonation, which allows your Astro deployment to impersonate a service account with the necessary permissions.
2. Set the following environment variables in the Deployment:

1
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_BUCKET_ENABLED=true
2
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_BASE_LOG_FOLDER="gs://your-bucket-name/logs"
3
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_IMPERSONATION_CHAIN="target-service-account@YOUR_PROJECT_ID.iam.gserviceaccount.com"

Option 3: Astro-provided Workload Identity

Use this approach if you do not have an existing service account that your deployment can impersonate.

1. Follow instructions to grant an IAM role to your Deployment Workload Identity.
2. Set the following environment variables in the Deployment:

1
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_BUCKET_ENABLED=true
2
AIRFLOW__ASTRO_SECONDARY_LOGS__GCS_BASE_LOG_FOLDER="gs://your-bucket-name/logs"

Troubleshooting

If you encounter issues with logging to the secondary GCS bucket:

1. Check that the environment variables are correctly set in your Astro deployment
2. Verify that your service account or Workload Identity has the necessary permissions
3. For Workload Identity Federation, ensure the Kubernetes service account annotation is correctly configured
4. For impersonation, check that the impersonation chain is correctly configured and the source identity has token creator permissions
5. Check for any errors in your Airflow logs related to GCS authentication or permissions
6. Test your authentication setup using the gcloud CLI or Google Cloud Console

If issues persist, contact Astronomer Support with the Deployment ID and any relevant error output.