Export logs to a Secondary WASB Container
Export logs to a Secondary WASB Container
By forwarding Airflow task logs from your Astro Deployment to an additional, customer‑managed Azure Storage (WASB) container, you can keep redundant copies of your execution history, integrate with existing log‑processing pipelines, and satisfy compliance or retention requirements that extend beyond the built‑in Astro logs.
Prerequisites
- Your Deployment must run Astro Runtime 11.7.0 or later. See Upgrade Astro Runtime.
- Your image must include
astronomer-providers-logging==1.5.4
or later. The secondary WASB logging feature was introduced in this version. - An Azure Storage Account with a container created to hold the logs.
- An Azure Managed Identity (or Service Principal) that has read/write access to the container.
Required environment variables
Variable | Description | Example |
---|---|---|
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_ENABLED | Turn on secondary WASB logging | true |
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_TENANT_ID | Azure AD tenant where the Managed Identity lives | random‑86f1‑41af‑91ab‑2d7cd011db47 |
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_CLIENT_ID | Client ID of the Managed Identity | a1b2c3d4‑e5f6‑1234‑5678‑random |
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_STORAGE_ACCOUNT | Name of the Storage Account (without protocol) | mystorageacct |
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_CONTAINER | Name of the container where logs are stored | airflow‑task‑logs |
Add these environment variables in the Deployment.
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_ENABLED=true
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_TENANT_ID="<YOUR_TENANT_ID>"
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_CLIENT_ID="<YOUR_CLIENT_ID>"
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_STORAGE_ACCOUNT="<YOUR_STORAGE_ACCOUNT_NAME>"
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_CONTAINER="<YOUR_CONTAINER_NAME>"
Step 1: Create Azure resources
- Create a Storage Account.
- Inside the account, create a blob container to hold the logs.
Step 2: Configure Astro Workload Identity
Follow Authorize Deployments to your Azure cloud to assign a Managed Identity to the Deployment.
Step 3: Grant storage permissions
In Azure, assign the Managed Identity the Storage Blob Data Contributor or equivalent scoped role to the Storage Account or container.
Step 4: Add environment variables and redeploy
Set the environment variables listed in required environment variables and redeploy.
Troubleshooting
If task logs do not appear in the container:
- Confirm that all required environment variables are set and have no extra quotes or spaces.
- Verify that the Managed Identity has read/write access to the Storage Account and container.
- Check Deployment logs for authentication or authorization errors related to Azure.
- Ensure the container exists, Astro will not create it automatically.
If issues persist, contact Astronomer Support with the Deployment ID and any relevant error output.