> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy Dags with an object storage mount

> Mount cloud object storage (GCS, Amazon S3, or Azure Blob) read-only into Airflow with a CSI driver to deploy Dags to Airflow Deployments on APC.

You can deploy Dags on Astro Private Cloud (APC) by mounting cloud object storage directly into your Airflow Deployment with a [Container Storage Interface (CSI)](https://kubernetes-csi.github.io/docs/) driver. APC supports the native object-storage CSI driver for each cloud:

| Cloud | Storage | CSI driver |
| - | - | - |
| GCP | Google Cloud Storage (GCS) | `gcsfuse.csi.storage.gke.io` |
| AWS | Amazon S3 | `s3.csi.aws.com` (Mountpoint for Amazon S3) |
| Azure | Azure Blob Storage | `blob.csi.azure.com` |

Unlike Astro CLI deploys, an object storage mount doesn't require rebuilding a Docker image or restarting your Airflow service. A Dag you add to the bucket or container appears in the Airflow UI without downtime.

> [!NOTE]
>
> **Astro Private Cloud 2.2**
>
> This feature was introduced in Astro Private Cloud 2.2. To access this feature, upgrade your Astro Private Cloud installation to 2.2 or later.

## How object storage mount deploys work

When you configure an object storage mount for a Deployment:

1. APC selects the CSI driver from the cluster's `cloudProvider` (`gcp`, `aws`, or `azure`).
2. APC mounts your bucket or container read-only at `/usr/local/airflow/dags` (or a custom `mountPath`) on the Airflow components that read Dags.
3. APC sets `AIRFLOW__CORE__DAGS_FOLDER` to the mount path on the Dag-reading components.
4. The mounted storage authenticates as the cloud IAM identity you provide, bound to the Deployment's Kubernetes service accounts (KSAs).
5. You deploy Dags by writing files directly to the bucket or container.

```text wrap theme={null}
Cloud storage (bucket / container)
       │  (CSI driver, read-only)
       ▼
┌─────────────────────────────┐
│ Airflow components          │
│  • dag-processor            │
│  • scheduler                │
│  • triggerer                │
│  • workers  (CeleryExecutor)│
└─────────────────────────────┘
       /usr/local/airflow/dags
```

## Prerequisites

* APC 2.2 installed with the standalone `dag-processor` enabled
* An existing storage bucket or container in your cloud (GCS bucket, S3 bucket, or Azure Blob container)
* A cloud IAM identity (GCP service account, AWS IAM role, or Azure managed identity) with read access to the storage, bound to the service accounts that the Deployment's Pods use: `dag-processor`, `scheduler`, `triggerer`, and `worker`
* The native object-storage CSI driver enabled on your cluster (see [Provision cloud prerequisites](#provision-cloud-prerequisites))
* The cluster's `cloudProvider` set correctly to `aws`, `azure`, or `gcp`

> [!NOTE]
>
> APC normally seeds the cluster's cloud provider from the region's cloud when you create the cluster. To verify or change it, go to the **Clusters** page in the APC UI, select your cluster, and confirm the cloud provider is `aws`, `azure`, or `gcp`. If it's unset or wrong, the mount can't select a driver and fails with `Object Storage mount is not supported for cloud provider: ...`.

## Implementation considerations

> [!WARNING]
>
> When you configure an object storage mount for a Deployment, you can't use the Astro CLI or service accounts to deploy Dags to that Deployment. The mount becomes the exclusive Dag deployment mechanism.

* Object storage mount requires the standalone `dag-processor` component.
* The mount deploys only Dags. To add Python dependencies or system packages, update your `requirements.txt` and `packages.txt` files and deploy using the CLI or CI/CD.
* The cluster's `cloudProvider` must be `aws`, `azure`, or `gcp` (case-insensitive) so APC can derive the CSI driver. See [Prerequisites](#prerequisites).

### Cloud-specific notes

> [tab] AWS (S3)
>
> * If the S3 bucket is in a different region than the EKS cluster, you must pass the region of the bucket in `volumeAttributes`.
> * The Mountpoint driver doesn't support inline ephemeral volumes. APC provisions a cluster-scoped `PersistentVolume` and `PersistentVolumeClaim` for the mount. This requires permission to create cluster-scoped resources, the same constraint as NFS.
> * Unlike GKE, the S3 driver doesn't inject a sidecar.

> [tab] GCP (GCS)
>
> The GCS Fuse CSI driver injects a sidecar container into every Pod that mounts the volume. APC pins the sidecar's resource limits so a namespace `LimitRange` default can't push the Pod's summed limits over the per-Pod maximum and fail Pod creation. The limits are:
>
> ```yaml wrap theme={null}
> gke-gcsfuse/cpu-limit: "250m"
> gke-gcsfuse/memory-limit: "256Mi"
> gke-gcsfuse/ephemeral-storage-limit: "5Gi"
> ```
>
> Account for this sidecar overhead when sizing the dag-processor, scheduler, triggerer, and worker components.

> [tab] Azure (Blob)
>
> * Azure has no equivalent of a pool-wide Workload Identity binding. Each Deployment's KSAs need their own federated identity credential on the managed identity, so repeat the federated credential setup for each new Deployment. See the Azure tab in [Provision cloud prerequisites](#provision-cloud-prerequisites).
> * Unlike GKE, the Blob driver doesn't inject a sidecar.

## Enable object storage mount

A System Admin must enable object storage mount deploys at the cluster level.

**1. Open the cluster**

In the APC UI, go to your **Clusters** page and select your cluster.

**2. Add the override**

On the cluster details page, click **Edit**. In the **Cluster Deployments Configuration** YAML editor, click **Find**, search for the feature you want to override, then add the following override:

```yaml wrap theme={null}
deployMechanisms:
  objectStorageMountDagDeployment:
    enabled: true
  configureDagDeployment:
    enabled: true
```

For details on using the UI for configuration, see [Override base configuration](https://astronomer.io/docs/astro-private-cloud/v-2-x/override-data-plane-cluster.md#override-base-configuration).

**3. Save your changes**

Save and apply your changes in the UI.

## Provision cloud prerequisites

Each tab enables the CSI driver on the cluster, creates a read-only IAM identity for your storage, and binds it to the Deployment's KSAs. You end with the two values you enter in APC: a **Storage URI** and an **IAM Identity**.

> [tab] AWS (S3)
>
> **1. Associate the OIDC provider**
>
> Ensure an OpenID Connect (OIDC) provider is associated with the EKS cluster:
>
> ```bash wrap theme={null}
> eksctl utils associate-iam-oidc-provider --cluster <cluster> --region <region> --approve
> ```
>
> **2. Create an IAM policy**
>
> Create an IAM policy that grants read on the bucket. Mountpoint needs `ListBucket` and `GetObject`:
>
> ```bash wrap theme={null}
> aws iam create-policy --policy-name <cluster>-s3-read \
>   --policy-document '{
>     "Version": "2012-10-17",
>     "Statement": [
>       { "Sid": "ListBucket", "Effect": "Allow", "Action": ["s3:ListBucket"],
>         "Resource": ["arn:aws:s3:::<bucket>"] },
>       { "Sid": "GetObjects", "Effect": "Allow", "Action": ["s3:GetObject"],
>         "Resource": ["arn:aws:s3:::<bucket>/*"] }
>     ]
>   }'
> ```
>
> **3. Install the Mountpoint for Amazon S3 CSI driver add-on**
>
> Install the Mountpoint for Amazon S3 CSI driver add-on with an IAM role for service accounts (IRSA) for its service account (`kube-system:s3-csi-driver-sa`):
>
> ```bash wrap theme={null}
> aws iam create-role --role-name <cluster>-s3-driver \
>   --assume-role-policy-document '{
>     "Version": "2012-10-17",
>     "Statement": [{
>       "Effect": "Allow",
>       "Principal": { "Federated": "arn:aws:iam::<account_id>:oidc-provider/<oidc_host>" },
>       "Action": "sts:AssumeRoleWithWebIdentity",
>       "Condition": {
>         "StringEquals": {
>           "<oidc_host>:aud": "sts.amazonaws.com",
>           "<oidc_host>:sub": ["system:serviceaccount:kube-system:s3-csi-driver-sa"]
>         }
>       }
>     }]
>   }'
> ```
>
> ```bash wrap theme={null}
> aws iam attach-role-policy --role-name <cluster>-s3-driver \
>   --policy-arn arn:aws:iam::<account_id>:policy/<cluster>-s3-read
> ```
>
> ```bash wrap theme={null}
> eksctl create addon --cluster <cluster> --region <region> \
>   --name aws-mountpoint-s3-csi-driver \
>   --service-account-role-arn arn:aws:iam::<account_id>:role/<cluster>-s3-driver --force
> ```
>
> **4. Create the workload IAM role**
>
> Create the workload IAM role that the Dag Pods assume, and attach the read policy. Because APC's per-Deployment KSA names aren't known ahead of time, trust a wildcard subject with `StringLike`:
>
> ```bash wrap theme={null}
> aws iam create-role --role-name <cluster>-dags-reader \
>   --assume-role-policy-document '{
>     "Version": "2012-10-17",
>     "Statement": [{
>       "Effect": "Allow",
>       "Principal": { "Federated": "arn:aws:iam::<account_id>:oidc-provider/<oidc_host>" },
>       "Action": "sts:AssumeRoleWithWebIdentity",
>       "Condition": {
>         "StringEquals": { "<oidc_host>:aud": "sts.amazonaws.com" },
>         "StringLike": { "<oidc_host>:sub": "system:serviceaccount:astronomer-*:*" }
>       }
>     }]
>   }'
> ```
>
> ```bash wrap theme={null}
> aws iam attach-role-policy --role-name <cluster>-dags-reader \
>   --policy-arn arn:aws:iam::<account_id>:policy/<cluster>-s3-read
> ```
>
> If you use namespace pools or manual release names, you can use an exact match for the namespaces that need an object storage mount, or attach the policy after you create the Deployment and know its namespace name.
>
> Enter the following values in APC:
>
> * **Storage URI**: `s3://<bucket>/<prefix>`
> * **IAM Identity**: The workload role ARN, `arn:aws:iam::<account_id>:role/<cluster>-dags-reader`. APC stamps it as `eks.amazonaws.com/role-arn` on the KSAs, and the static PV uses `authenticationSource: pod`.

> [tab] GCP (GCS)
>
> **1. Enable the GCS Fuse CSI driver**
>
> Enable the GCS Fuse CSI driver on the GKE cluster. GKE Standard doesn't enable it by default, but Autopilot does:
>
> ```bash wrap theme={null}
> gcloud container clusters update <cluster> \
>   --project=<project> --zone=<zone> \
>   --update-addons=GcsFuseCsiDriver=ENABLED
> ```
>
> **2. Create a Dag-reader service account**
>
> Create a Dag-reader Google service account (GSA) and grant it read on the bucket:
>
> ```bash wrap theme={null}
> gcloud iam service-accounts create <gsa-name> \
>   --project=<project> --display-name="Dag reader"
>
> gcloud storage buckets add-iam-policy-binding gs://<bucket> \
>   --project=<project> \
>   --member="serviceAccount:<gsa-name>@<project>.iam.gserviceaccount.com" \
>   --role="roles/storage.objectViewer"
> ```
>
> **3. Bind Workload Identity**
>
> Bind Workload Identity so the Deployment's KSAs can impersonate the GSA. A pool-wide binding covers every per-Deployment KSA:
>
> ```bash wrap theme={null}
> gcloud iam service-accounts add-iam-policy-binding \
>   <gsa-name>@<project>.iam.gserviceaccount.com \
>   --project=<project> \
>   --role="roles/iam.workloadIdentityUser" \
>   --member="principalSet://iam.googleapis.com/projects/<project-number>/locations/global/workloadIdentityPools/<project>.svc.id.goog/*"
> ```
>
> To bind a single KSA instead, use `--member="serviceAccount:<project>.svc.id.goog[astronomer-<release>/<release>-airflow-dag-processor]"`. Repeat for `-scheduler`, `-triggerer`, and `-worker`.
>
> Enter the following values in APC:
>
> * **Storage URI**: `gs://<bucket>/<prefix>`
> * **IAM Identity**: `<gsa-name>@<project>.iam.gserviceaccount.com`. APC stamps it as `iam.gke.io/gcp-service-account` on the KSAs.

> [tab] Azure (Blob)
>
> **1. Enable the Blob CSI driver**
>
> Enable the Blob CSI driver on the AKS cluster. It isn't enabled by default:
>
> ```bash wrap theme={null}
> az aks update --resource-group <rg> --name <cluster> --enable-blob-driver --yes
> ```
>
> **2. Create a managed identity**
>
> Create a user-assigned managed identity and grant it `Storage Blob Data Reader` on the storage account:
>
> ```bash wrap theme={null}
> az identity create --name <identity> --resource-group <rg> --location <location>
>
> az role assignment create \
>   --assignee-object-id <identity-principal-id> --assignee-principal-type ServicePrincipal \
>   --role "Storage Blob Data Reader" \
>   --scope <storage-account-resource-id>
> ```
>
> **3. Create federated identity credentials**
>
> Create one federated identity credential for each of the Deployment's KSAs. Azure has no pool-wide binding, so repeat this step for every new Deployment:
>
> ```bash wrap theme={null}
> az identity federated-credential create \
>   --name fic-<release>-airflow-dag-processor \
>   --identity-name <identity> --resource-group <rg> \
>   --issuer <aks-oidc-issuer-url> \
>   --subject system:serviceaccount:astronomer-<release>:<release>-airflow-dag-processor \
>   --audience api://AzureADTokenExchange
> ```
>
> Repeat for `-airflow-scheduler`, `-airflow-triggerer`, and `-airflow-worker`.
>
> Enter the following values in APC:
>
> * **Storage URI**: `https://<account>.blob.core.windows.net/<container>/<prefix>`
> * **IAM Identity**: The managed identity client ID. APC stamps it as `azure.workload.identity/client-id` on the KSAs.

## Configure object storage mount for a Deployment

> [tab] UI
>
> **1. Open the Deployment**
>
> In the APC UI, create a new Deployment or open an existing one.
>
> **2. Select the mechanism**
>
> Go to **DAG Deployment** in the Deployment settings and select **Object Storage Mount** as the mechanism.
>
> **3. Enter the storage details**
>
> Enter the **Storage URI** and **IAM Identity** from your cloud setup. Optionally set a custom **Mount Path**. The default is `/usr/local/airflow/dags`.
>
> **4. Apply your changes**
>
> Click **Create Deployment** or **Deploy Changes**.

> [tab] API
>
> Use `upsertDeployment` to configure a Deployment's Dag deployment mechanism as an object storage mount:
>
> ```graphql wrap theme={null}
> mutation {
>   upsertDeployment(
>     workspaceUuid: "<workspace-uuid>"
>     label: "<deployment-name>"
>     dagDeployment: {
>       type: object_storage_mount
>       storageUri: "gs://<bucket>/<prefix>"
>       iamIdentity: "<gsa-name>@<project>.iam.gserviceaccount.com"
>     }
>   ) {
>     id
>     releaseName
>   }
> }
> ```
>
> `storageUri` is the cloud storage URI (`gs://…`, `s3://…`, or an Azure blob endpoint or container). `iamIdentity` is the cloud IAM identity: a GCP service account email, AWS IAM role ARN, or Azure managed identity client ID. Set an optional `mountPath` to override the default `/usr/local/airflow/dags`.

## Deploy Dags to storage

Upload Dag files to your bucket or container under the prefix in your Storage URI.

> [tab] AWS (S3)
>
> ```bash wrap theme={null}
> aws s3 cp your_dag.py s3://<bucket>/<prefix>/your_dag.py --region <region>
> ```

> [tab] GCP (GCS)
>
> ```bash wrap theme={null}
> gcloud storage cp your_dag.py gs://<bucket>/<prefix>/your_dag.py
> ```

> [tab] Azure (Blob)
>
> ```bash wrap theme={null}
> az storage blob upload \
>   --account-name <account> --container-name <container> \
>   --name <prefix>/your_dag.py --file your_dag.py \
>   --auth-mode login --overwrite
> ```

## Verify the object storage mount

Verify the volume is mounted in the Dag-processing Pods:

```bash wrap theme={null}
kubectl exec -n <deployment-namespace> <dag-processor-pod> -- ls -l /usr/local/airflow/dags
```

On AWS, check that the static PV and PVC were created:

```bash wrap theme={null}
kubectl get pv | grep dags
kubectl get pvc -n <deployment-namespace> | grep dags
```

## Troubleshooting

### Dags aren't appearing

1. Confirm the file landed in the bucket or container under the prefix in your Storage URI.

2. Confirm the mount is present:

   ```bash wrap theme={null}
   kubectl exec -n <deployment-namespace> <dag-processor-pod> -- ls -l /usr/local/airflow/dags
   ```

3. Confirm the IAM identity is bound to the Pod's service account and has read access to the storage.

### Dags take a long time to appear

Change the `dag-processor` defaults:

* `refresh_interval` (`AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL`): How often the Dag processor scans the Dag source for new or removed files. The default is 300 seconds.
* `min_file_process_interval` (`AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL`): The minimum interval before the Dag processor re-parses each file. This governs how quickly edits to existing Dags are picked up. The default is 30 seconds. Lower values raise Dag processor CPU usage.

### `failed to find the sidecar container in Pod spec` (GKE)

APC automatically sets the `gke-gcsfuse/volumes: "true"` annotation, which injects the `gcsfuse` sidecar. If you see this error, confirm the GCS Fuse CSI driver add-on is enabled on the cluster. See the GCP tab in [Provision cloud prerequisites](#provision-cloud-prerequisites).

### Region mismatch (AWS)

If the mount is empty or fails, check whether the S3 bucket and the EKS cluster are in the same AWS region. If they're in different regions, pass the bucket's region in `volumeAttributes`. See the AWS tab in [Cloud-specific notes](#cloud-specific-notes).

### Authentication failures (Azure)

Confirm a federated identity credential exists for each of the Deployment's KSAs (`dag-processor`, `scheduler`, `triggerer`, and `worker`).

### `Object Storage mount is not supported for cloud provider`

The cluster's `cloudProvider` is unset or not one of `aws`, `azure`, or `gcp`. See [Prerequisites](#prerequisites). Set it with the `updateCluster` mutation:

```graphql wrap theme={null}
mutation {
  updateCluster(
    id: "<cluster-id>"
    cloudProvider: "<cloud-provider>"
  ) {
    id
    name
    cloudProvider
  }
}
```

## Alternative Dag deploy methods

* [Deploy Dags with NFS](https://astronomer.io/docs/astro-private-cloud/v-2-x/deploy-nfs.md)
* [Configure git-sync code deploys](https://astronomer.io/docs/astro-private-cloud/v-2-x/deploy-git-sync.md)
* [Deploy code overview](https://astronomer.io/docs/astro-private-cloud/v-2-x/deploy-code-overview.md)