Deploy Dags with an object storage mount

You can deploy Dags on Astro Private Cloud (APC) by mounting cloud object storage directly into your Airflow Deployment with a Container Storage Interface (CSI) driver. APC supports the native object-storage CSI driver for each cloud:

Cloud	Storage	CSI driver
GCP	Google Cloud Storage (GCS)	`gcsfuse.csi.storage.gke.io`
AWS	Amazon S3	`s3.csi.aws.com` (Mountpoint for Amazon S3)
Azure	Azure Blob Storage	`blob.csi.azure.com`

Unlike Astro CLI deploys, an object storage mount doesn’t require rebuilding a Docker image or restarting your Airflow service. A Dag you add to the bucket or container appears in the Airflow UI without downtime.

Astro Private Cloud 2.2This feature was introduced in Astro Private Cloud 2.2. To access this feature, upgrade your Astro Private Cloud installation to 2.2 or later.

How object storage mount deploys work

When you configure an object storage mount for a Deployment:

APC selects the CSI driver from the cluster’s cloudProvider (gcp, aws, or azure).
APC mounts your bucket or container read-only at /usr/local/airflow/dags (or a custom mountPath) on the Airflow components that read Dags.
APC sets AIRFLOW__CORE__DAGS_FOLDER to the mount path on the Dag-reading components.
The mounted storage authenticates as the cloud IAM identity you provide, bound to the Deployment’s Kubernetes service accounts (KSAs).
You deploy Dags by writing files directly to the bucket or container.

Cloud storage (bucket / container)
       │  (CSI driver, read-only)
       ▼
┌─────────────────────────────┐
│ Airflow components          │
│  • dag-processor            │
│  • scheduler                │
│  • triggerer                │
│  • workers  (CeleryExecutor)│
└─────────────────────────────┘
       /usr/local/airflow/dags

Prerequisites

APC 2.2 installed with the standalone dag-processor enabled
An existing storage bucket or container in your cloud (GCS bucket, S3 bucket, or Azure Blob container)
A cloud IAM identity (GCP service account, AWS IAM role, or Azure managed identity) with read access to the storage, bound to the service accounts that the Deployment’s Pods use: dag-processor, scheduler, triggerer, and worker
The native object-storage CSI driver enabled on your cluster (see Provision cloud prerequisites)
The cluster’s cloudProvider set correctly to aws, azure, or gcp

APC normally seeds the cluster’s cloud provider from the region’s cloud when you create the cluster. To verify or change it, go to the Clusters page in the APC UI, select your cluster, and confirm the cloud provider is aws, azure, or gcp. If it’s unset or wrong, the mount can’t select a driver and fails with Object Storage mount is not supported for cloud provider: ....

Implementation considerations

When you configure an object storage mount for a Deployment, you can’t use the Astro CLI or service accounts to deploy Dags to that Deployment. The mount becomes the exclusive Dag deployment mechanism.

Object storage mount requires the standalone dag-processor component.
The mount deploys only Dags. To add Python dependencies or system packages, update your requirements.txt and packages.txt files and deploy using the CLI or CI/CD.
The cluster’s cloudProvider must be aws, azure, or gcp (case-insensitive) so APC can derive the CSI driver. See Prerequisites.

Cloud-specific notes

AWS (S3)
GCP (GCS)
Azure (Blob)

If the S3 bucket is in a different region than the EKS cluster, you must pass the region of the bucket in volumeAttributes.
The Mountpoint driver doesn’t support inline ephemeral volumes. APC provisions a cluster-scoped PersistentVolume and PersistentVolumeClaim for the mount. This requires permission to create cluster-scoped resources, the same constraint as NFS.
Unlike GKE, the S3 driver doesn’t inject a sidecar.

The GCS Fuse CSI driver injects a sidecar container into every Pod that mounts the volume. APC pins the sidecar’s resource limits so a namespace LimitRange default can’t push the Pod’s summed limits over the per-Pod maximum and fail Pod creation. The limits are:

gke-gcsfuse/cpu-limit: "250m"
gke-gcsfuse/memory-limit: "256Mi"
gke-gcsfuse/ephemeral-storage-limit: "5Gi"

Account for this sidecar overhead when sizing the dag-processor, scheduler, triggerer, and worker components.

Enable object storage mount

A System Admin must enable object storage mount deploys at the cluster level.

Open the cluster

In the APC UI, go to your Clusters page and select your cluster.

Add the override

On the cluster details page, click Edit. In the Cluster Deployments Configuration YAML editor, click Find, search for the feature you want to override, then add the following override:

deployMechanisms:
  objectStorageMountDagDeployment:
    enabled: true
  configureDagDeployment:
    enabled: true

For details on using the UI for configuration, see Override base configuration.

Save your changes

Save and apply your changes in the UI.

Provision cloud prerequisites

Each tab enables the CSI driver on the cluster, creates a read-only IAM identity for your storage, and binds it to the Deployment’s KSAs. You end with the two values you enter in APC: a Storage URI and an IAM Identity.

AWS (S3)
GCP (GCS)
Azure (Blob)

Associate the OIDC provider

Ensure an OpenID Connect (OIDC) provider is associated with the EKS cluster:

eksctl utils associate-iam-oidc-provider --cluster <cluster> --region <region> --approve

Create an IAM policy

Create an IAM policy that grants read on the bucket. Mountpoint needs ListBucket and GetObject:

aws iam create-policy --policy-name <cluster>-s3-read \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      { "Sid": "ListBucket", "Effect": "Allow", "Action": ["s3:ListBucket"],
        "Resource": ["arn:aws:s3:::<bucket>"] },
      { "Sid": "GetObjects", "Effect": "Allow", "Action": ["s3:GetObject"],
        "Resource": ["arn:aws:s3:::<bucket>/*"] }
    ]
  }'

Install the Mountpoint for Amazon S3 CSI driver add-on

Install the Mountpoint for Amazon S3 CSI driver add-on with an IAM role for service accounts (IRSA) for its service account (kube-system:s3-csi-driver-sa):

aws iam create-role --role-name <cluster>-s3-driver \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": { "Federated": "arn:aws:iam::<account_id>:oidc-provider/<oidc_host>" },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "<oidc_host>:aud": "sts.amazonaws.com",
          "<oidc_host>:sub": ["system:serviceaccount:kube-system:s3-csi-driver-sa"]
        }
      }
    }]
  }'

aws iam attach-role-policy --role-name <cluster>-s3-driver \
  --policy-arn arn:aws:iam::<account_id>:policy/<cluster>-s3-read

eksctl create addon --cluster <cluster> --region <region> \
  --name aws-mountpoint-s3-csi-driver \
  --service-account-role-arn arn:aws:iam::<account_id>:role/<cluster>-s3-driver --force

Create the workload IAM role

Create the workload IAM role that the Dag Pods assume, and attach the read policy. Because APC’s per-Deployment KSA names aren’t known ahead of time, trust a wildcard subject with StringLike:

aws iam create-role --role-name <cluster>-dags-reader \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": { "Federated": "arn:aws:iam::<account_id>:oidc-provider/<oidc_host>" },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": { "<oidc_host>:aud": "sts.amazonaws.com" },
        "StringLike": { "<oidc_host>:sub": "system:serviceaccount:astronomer-*:*" }
      }
    }]
  }'

aws iam attach-role-policy --role-name <cluster>-dags-reader \
  --policy-arn arn:aws:iam::<account_id>:policy/<cluster>-s3-read

If you use namespace pools or manual release names, you can use an exact match for the namespaces that need an object storage mount, or attach the policy after you create the Deployment and know its namespace name.

Enter the following values in APC:

Storage URI: s3://<bucket>/<prefix>
IAM Identity: The workload role ARN, arn:aws:iam::<account_id>:role/<cluster>-dags-reader. APC stamps it as eks.amazonaws.com/role-arn on the KSAs, and the static PV uses authenticationSource: pod.

Enable the GCS Fuse CSI driver

Enable the GCS Fuse CSI driver on the GKE cluster. GKE Standard doesn’t enable it by default, but Autopilot does:

gcloud container clusters update <cluster> \
  --project=<project> --zone=<zone> \
  --update-addons=GcsFuseCsiDriver=ENABLED

Create a Dag-reader service account

Create a Dag-reader Google service account (GSA) and grant it read on the bucket:

gcloud iam service-accounts create <gsa-name> \
  --project=<project> --display-name="Dag reader"

gcloud storage buckets add-iam-policy-binding gs://<bucket> \
  --project=<project> \
  --member="serviceAccount:<gsa-name>@<project>.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

Bind Workload Identity

Bind Workload Identity so the Deployment’s KSAs can impersonate the GSA. A pool-wide binding covers every per-Deployment KSA:

gcloud iam service-accounts add-iam-policy-binding \
  <gsa-name>@<project>.iam.gserviceaccount.com \
  --project=<project> \
  --role="roles/iam.workloadIdentityUser" \
  --member="principalSet://iam.googleapis.com/projects/<project-number>/locations/global/workloadIdentityPools/<project>.svc.id.goog/*"

To bind a single KSA instead, use --member="serviceAccount:<project>.svc.id.goog[astronomer-<release>/<release>-airflow-dag-processor]". Repeat for -scheduler, -triggerer, and -worker.

Enter the following values in APC:

Storage URI: gs://<bucket>/<prefix>
IAM Identity: <gsa-name>@<project>.iam.gserviceaccount.com. APC stamps it as iam.gke.io/gcp-service-account on the KSAs.

Enable the Blob CSI driver

Enable the Blob CSI driver on the AKS cluster. It isn’t enabled by default:

az aks update --resource-group <rg> --name <cluster> --enable-blob-driver --yes

Create a managed identity

Create a user-assigned managed identity and grant it Storage Blob Data Reader on the storage account:

az identity create --name <identity> --resource-group <rg> --location <location>

az role assignment create \
  --assignee-object-id <identity-principal-id> --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Reader" \
  --scope <storage-account-resource-id>

Create federated identity credentials

Create one federated identity credential for each of the Deployment’s KSAs. Azure has no pool-wide binding, so repeat this step for every new Deployment:

az identity federated-credential create \
  --name fic-<release>-airflow-dag-processor \
  --identity-name <identity> --resource-group <rg> \
  --issuer <aks-oidc-issuer-url> \
  --subject system:serviceaccount:astronomer-<release>:<release>-airflow-dag-processor \
  --audience api://AzureADTokenExchange

Repeat for -airflow-scheduler, -airflow-triggerer, and -airflow-worker.

Enter the following values in APC:

Storage URI: https://<account>.blob.core.windows.net/<container>/<prefix>
IAM Identity: The managed identity client ID. APC stamps it as azure.workload.identity/client-id on the KSAs.

Configure object storage mount for a Deployment

Open the Deployment

In the APC UI, create a new Deployment or open an existing one.

Select the mechanism

Go to DAG Deployment in the Deployment settings and select Object Storage Mount as the mechanism.

Enter the storage details

Enter the Storage URI and IAM Identity from your cloud setup. Optionally set a custom Mount Path. The default is /usr/local/airflow/dags.

Apply your changes

Click Create Deployment or Deploy Changes.

Use upsertDeployment to configure a Deployment’s Dag deployment mechanism as an object storage mount:

mutation {
  upsertDeployment(
    workspaceUuid: "<workspace-uuid>"
    label: "<deployment-name>"
    dagDeployment: {
      type: object_storage_mount
      storageUri: "gs://<bucket>/<prefix>"
      iamIdentity: "<gsa-name>@<project>.iam.gserviceaccount.com"
    }
  ) {
    id
    releaseName
  }
}

storageUri is the cloud storage URI (gs://…, s3://…, or an Azure blob endpoint or container). iamIdentity is the cloud IAM identity: a GCP service account email, AWS IAM role ARN, or Azure managed identity client ID. Set an optional mountPath to override the default /usr/local/airflow/dags.

Deploy Dags to storage

Upload Dag files to your bucket or container under the prefix in your Storage URI.

AWS (S3)
GCP (GCS)
Azure (Blob)

aws s3 cp your_dag.py s3://<bucket>/<prefix>/your_dag.py --region <region>

gcloud storage cp your_dag.py gs://<bucket>/<prefix>/your_dag.py

az storage blob upload \
  --account-name <account> --container-name <container> \
  --name <prefix>/your_dag.py --file your_dag.py \
  --auth-mode login --overwrite

Verify the object storage mount

Verify the volume is mounted in the Dag-processing Pods:

kubectl exec -n <deployment-namespace> <dag-processor-pod> -- ls -l /usr/local/airflow/dags

On AWS, check that the static PV and PVC were created:

kubectl get pv | grep dags
kubectl get pvc -n <deployment-namespace> | grep dags

Troubleshooting

Dags aren’t appearing

Confirm the file landed in the bucket or container under the prefix in your Storage URI.

Confirm the mount is present:

kubectl exec -n <deployment-namespace> <dag-processor-pod> -- ls -l /usr/local/airflow/dags

Confirm the IAM identity is bound to the Pod’s service account and has read access to the storage.

Dags take a long time to appear

Change the dag-processor defaults:

refresh_interval (AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL): How often the Dag processor scans the Dag source for new or removed files. The default is 300 seconds.
min_file_process_interval (AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL): The minimum interval before the Dag processor re-parses each file. This governs how quickly edits to existing Dags are picked up. The default is 30 seconds. Lower values raise Dag processor CPU usage.

`failed to find the sidecar container in Pod spec` (GKE)

APC automatically sets the gke-gcsfuse/volumes: "true" annotation, which injects the gcsfuse sidecar. If you see this error, confirm the GCS Fuse CSI driver add-on is enabled on the cluster. See the GCP tab in Provision cloud prerequisites.

Region mismatch (AWS)

If the mount is empty or fails, check whether the S3 bucket and the EKS cluster are in the same AWS region. If they’re in different regions, pass the bucket’s region in volumeAttributes. See the AWS tab in Cloud-specific notes.

Authentication failures (Azure)

Confirm a federated identity credential exists for each of the Deployment’s KSAs (dag-processor, scheduler, triggerer, and worker).

`Object Storage mount is not supported for cloud provider`

The cluster’s cloudProvider is unset or not one of aws, azure, or gcp. See Prerequisites. Set it with the updateCluster mutation:

mutation {
  updateCluster(
    id: "<cluster-id>"
    cloudProvider: "<cloud-provider>"
  ) {
    id
    name
    cloudProvider
  }
}

​How object storage mount deploys work

​Prerequisites

​Implementation considerations

​Cloud-specific notes

​Enable object storage mount

​Provision cloud prerequisites

​Configure object storage mount for a Deployment

​Deploy Dags to storage

​Verify the object storage mount

​Troubleshooting

​Dags aren’t appearing

​Dags take a long time to appear

​failed to find the sidecar container in Pod spec (GKE)

​Region mismatch (AWS)

​Authentication failures (Azure)

​Object Storage mount is not supported for cloud provider

​Alternative Dag deploy methods

How object storage mount deploys work

Prerequisites

Implementation considerations

Cloud-specific notes

Enable object storage mount

Provision cloud prerequisites

Configure object storage mount for a Deployment

Deploy Dags to storage

Verify the object storage mount

Troubleshooting

Dags aren’t appearing

Dags take a long time to appear

`failed to find the sidecar container in Pod spec` (GKE)

Region mismatch (AWS)

Authentication failures (Azure)

`Object Storage mount is not supported for cloud provider`

Alternative Dag deploy methods