Skip to main content

Export logs to a Secondary WASB Container

Export logs to a Secondary WASB Container​

By forwarding Airflow task logs from your Astro Deployment to an additional, customer‑managed Azure Storage (WASB) container, you can keep redundant copies of your execution history, integrate with existing log‑processing pipelines, and satisfy compliance or retention requirements that extend beyond the built‑in Astro logs.

Prerequisites​

  • Your Deployment must run Astro Runtime 11.7.0 or later. See Upgrade Astro Runtime.
  • Your image must include astronomer-providers-logging==1.5.4 or later. The secondary WASB logging feature was introduced in this version.
  • An Azure Storage Account with a container created to hold the logs.
  • An Azure Managed Identity (or Service Principal) that has read/write access to the container.

Required environment variables​

VariableDescriptionExample
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_ENABLEDTurn on secondary WASB loggingtrue
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_TENANT_IDAzure AD tenant where the Managed Identity livesrandom‑86f1‑41af‑91ab‑2d7cd011db47
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_CLIENT_IDClient ID of the Managed Identitya1b2c3d4‑e5f6‑1234‑5678‑random
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_STORAGE_ACCOUNTName of the Storage Account (without protocol)mystorageacct
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_CONTAINERName of the container where logs are storedairflow‑task‑logs

Add these environment variables in the Deployment.

AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_ENABLED=true
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_TENANT_ID="<YOUR_TENANT_ID>"
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_CLIENT_ID="<YOUR_CLIENT_ID>"
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_STORAGE_ACCOUNT="<YOUR_STORAGE_ACCOUNT_NAME>"
AIRFLOW__ASTRO_SECONDARY_LOGS__WASB_CONTAINER="<YOUR_CONTAINER_NAME>"

Step 1: Create Azure resources​

  1. Create a Storage Account.
  2. Inside the account, create a blob container to hold the logs.

Step 2: Configure Astro Workload Identity​

Follow Authorize Deployments to your Azure cloud to assign a Managed Identity to the Deployment.

Step 3: Grant storage permissions​

In Azure, assign the Managed Identity the Storage Blob Data Contributor or equivalent scoped role to the Storage Account or container.

Step 4: Add environment variables and redeploy​

Set the environment variables listed in required environment variables and redeploy.

Troubleshooting​

If task logs do not appear in the container:

  1. Confirm that all required environment variables are set and have no extra quotes or spaces.
  2. Verify that the Managed Identity has read/write access to the Storage Account and container.
  3. Check Deployment logs for authentication or authorization errors related to Azure.
  4. Ensure the container exists, Astro will not create it automatically.

If issues persist, contact Astronomer Support with the Deployment ID and any relevant error output.

Was this page helpful?