> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Configure DAG sources

Remote Execution Agents require configuration to access your DAG code. This guide covers configuring DAG bundles, which are collections of DAG files and supporting code introduced in Airflow 3.

<Note>
  This feature requires Airflow 3.x Deployments. Configuring multiple DAG bundles in a single Deployment is only supported in Remote Execution mode.
</Note>

## DAG bundle types

Choose between two types of DAG bundles:

* **GitDagBundle**: Dags stored in a Git repository (recommended for production)
* **LocalDagBundle**: Dags stored in the container image or persistent volume (default)

### When to use each bundle type

**Use GitDagBundle when**:

* Running production deployments
* Tracking DAG versions with full rerun capabilities
* Storing dags in version control systems
* Managing multiple teams or DAG repositories

**Use LocalDagBundle when**:

* Running development or testing environments
* Building dags into container images
* Using existing PVC-based DAG management
* Preferring simpler configuration

See [GitDagBundle compared to LocalDagBundle](#gitdagbundle-compared-to-localdagbundle) for functional differences.

## Dag hashing

The Dag processor computes a hash of each Dag and caches it. On each processing cycle, the Dag processor compares the current hash of a Dag against the cached value:

* If the hash matches, the Dag is unchanged, so the Dag processor skips it instead of re-sending it to the Astro orchestration plane.
* If the Dag is new, changed, or not yet in the cache, the Dag processor processes it and updates the cache.

Because the Dag processor skips unchanged Dags, Dag hashing lowers memory utilization and network bandwidth, and speeds up Dag updates in Deployments with many Dags.

To monitor cache behavior, use the `dag_processor_cache_hits_total`, `dag_processor_cache_misses_total`, and `dag_processor_cache_size` metrics. See [Dag processor metrics](/docs/astro/remote-agents-metrics#dag-processor-metrics).

<Note>
  Dag hashing functionality is enabled by default, starting on Astro Agent client release `1.8.0` and later. You can disable that feature at any time with the following environment variable: `ASTRO_AGENT_CLIENT_DAG_PROCESSOR__ENABLE_DAG_CACHING=False`.
</Note>

## Configure GitDagBundle

GitDagBundle fetches dags from Git repositories and provides automatic versioning capabilities.

<Tip>
  GitDagBundle is recommended for production Remote Execution deployments.
</Tip>

### Supported authentication methods

GitDagBundle supports the following authentication methods:

* Access tokens (personal access tokens, OAuth tokens)
* SSH keys
* SSH agent

Choose the method that aligns with your security requirements and infrastructure.

### Required token permissions by provider

When you create an access token, grant the minimum permissions required to read repository contents:

| Provider  | Token type       | Required permissions                                                                            |
| --------- | ---------------- | ----------------------------------------------------------------------------------------------- |
| GitHub    | Fine-grained PAT | `Contents: Read-only` on the target repository. `Metadata: Read-only` is included automatically |
| GitHub    | Classic PAT      | `repo` scope for private repositories. Public repositories require no scope                     |
| GitLab    | PAT              | `read_repository`                                                                               |
| Bitbucket | App password     | `Repositories: Read`                                                                            |

### Configure public repository

For public repositories, no authentication configuration is required. Configure only the repository URL and tracking reference:

```yaml title="values.yaml" wrap theme={null}
dagBundleConfigList: '[{"name": "public-dags", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"repo_url": "https://github.com/your-org/public-dags", "tracking_ref": "main", "subdir": "dags"}}]'
```

### Configure private repository

For private repositories, configure both the DAG bundle and an Airflow connection for authentication.

<Steps>
  <Step title="Create Git connection">
    Add an Airflow connection environment variable in `values.yaml`. The connection name suffix must match the `git_conn_id` value in your DAG bundle configuration.

    <Tabs>
      <Tab title="Access token (HTTPS)">
        Use this method with a Personal Access Token (PAT) or OAuth token. Set `login` to your Git username and `password` to the token value.

        ```yaml title="values.yaml" wrap theme={null}
        commonEnv:
          - name: AIRFLOW_CONN_GIT_REPO
            value: >-
              {
                "conn_type": "git",
                "login": "<git-username>",
                "password": "<personal-access-token>",
                "host": "github.com",
                "schema": "https",
                "extra": {
                  "repo": "<your-org>/<private-repo>",
                  "branch": "main"
                }
              }
        ```

        See [Required token permissions by provider](#required-token-permissions-by-provider) for the minimum permissions each provider requires.
      </Tab>

      <Tab title="SSH key (deploy key)">
        Use this method with an SSH deploy key. Set `login` to `git` and provide the private key in the `extra` field.

        ```yaml title="values.yaml" wrap theme={null}
        commonEnv:
          - name: AIRFLOW_CONN_GIT_REPO
            value: >-
              {
                "conn_type": "git",
                "login": "git",
                "host": "<your-git-host>",
                "schema": "ssh",
                "extra": {
                  "private_key": "<private-ssh-key>"
                }
              }
        ```
      </Tab>
    </Tabs>

    <Note>
      The connection name `AIRFLOW_CONN_GIT_REPO` creates a connection with ID `git_repo`. This ID must match the `git_conn_id` value in your DAG bundle configuration.
    </Note>

    For production environments, store connection credentials in a [secrets backend](/docs/astro/remote-execution-configure-secrets-backend) instead of `values.yaml`. See [Use a secrets backend for Git credentials](#use-a-secrets-backend-for-git-credentials) for an example using Azure Key Vault.
  </Step>

  <Step title="Configure DAG bundle">
    Configure the DAG bundle with matching `git_conn_id`:

    ```yaml title="values.yaml" wrap theme={null}
    dagBundleConfigList: '[{"name": "private-dags", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"tracking_ref": "main", "subdir": "dags", "repo_url": "https://github.com/<your-org>/<private-repo>.git", "git_conn_id": "git_repo"}}]'
    ```

    Note that `git_conn_id: "git_repo"` matches the connection ID from the `AIRFLOW_CONN_GIT_REPO` environment variable.
  </Step>

  <Step title="Update Helm release">
    Apply the configuration:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Step>
</Steps>

### Configure refresh interval

Control how frequently agents check for repository updates using the `refresh_interval` parameter:

```yaml title="values.yaml" wrap theme={null}
dagBundleConfigList: '[{"name": "private-dags", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": {"repo_url": "https://github.com/your-org/private-dags", "tracking_ref": "main", "subdir": "dags", "git_conn_id": "git_repo", "refresh_interval": 300}}]'
```

The default refresh interval is 300 seconds. Reducing this value across many bundles may increase the risk of hitting Git provider rate limits.

### Use a secrets backend for Git credentials

For production environments, use a secrets backend to store Git connection credentials instead of hardcoding them in `values.yaml`. The following example shows how to configure Azure Key Vault with workload identity authentication on Azure AKS.

<Steps>
  <Step title="Configure Azure Key Vault as secrets backend">
    Add the secrets backend configuration to your `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    secretBackend: airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend

    commonEnv:
      - name: AIRFLOW__SECRETS__BACKEND_KWARGS
        value: '{"connections_prefix": "airflow-connection", "variables_prefix": "airflow-variable", "vault_url": "<your-vault-url>", "workload_identity_tenant_id": "<your-tenant-id>", "managed_identity_client_id": "<your-managed-identity-client-id>"}'
    ```

    This configuration uses Azure workload identity for authentication, which is the recommended approach for Azure AKS environments. For other authentication methods, see [Azure Key Vault secrets backend](/docs/astro/secrets-backend/azure-key-vault).
  </Step>

  <Step title="Store Git connections in Azure Key Vault">
    Create secrets in Azure Key Vault for each Git connection. The secret name must follow the pattern `<connections_prefix>-<connection-id>`. For example, to create a connection with ID `git-repo1-conn`:

    1. In Azure Key Vault, create a secret named `airflow-connection-git-repo1-conn`.
    2. Set the secret value to a JSON connection string:

    ```json wrap theme={null}
    {
      "conn_type": "git",
      "login": "<git-username>",
      "password": "<personal-access-token>",
      "host": "github.com",
      "schema": "https",
      "extra": {
        "repo": "<your-org>/<your-repo>",
        "branch": "main"
      }
    }
    ```

    Repeat this process for each Git repository connection you need.
  </Step>

  <Step title="Configure DAG bundles with connection references">
    Configure your DAG bundles to reference the connections stored in Azure Key Vault:

    ```yaml title="values.yaml" wrap theme={null}
    dagBundleConfigList: '[
      {
        "name": "dags-folder",
        "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
        "kwargs": {
          "repo_url": "https://github.com/<your-org>/<repo-1>.git",
          "tracking_ref": "main",
          "subdir": "dags",
          "refresh_interval": 120,
          "git_conn_id": "git-repo1-conn"
        }
      },
      {
        "name": "dags-folder2",
        "classpath": "airflow.providers.git.bundles.git.GitDagBundle",
        "kwargs": {
          "repo_url": "https://github.com/<your-org>/<repo-2>.git",
          "tracking_ref": "main",
          "subdir": "dags",
          "refresh_interval": 120,
          "git_conn_id": "git-repo2-conn"
        }
      }
    ]'
    ```

    The `git_conn_id` values must match the connection IDs you created in Azure Key Vault (without the `airflow-connection-` prefix).
  </Step>

  <Step title="Update Helm release">
    Apply the configuration:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Step>
</Steps>

## Configure LocalDagBundle

LocalDagBundle reads dags from the local filesystem. This is the default dag bundle type.

### DAG storage options

Choose one of two methods to provide dags to agents:

**Option 1: Include dags in container image**

Build a custom agent image that includes your DAG files. Copy dags into the `/dags` folder during image build.

**Option 2: Mount Persistent Volume Claim**

Create a PVC containing your dags and mount it into all agent components (Dag Processor, Worker, and Triggerer) at the same path.

### Configure DAG path

LocalDagBundle looks for dags in `/dags` by default. Specify a different path using the `path` parameter:

```yaml title="values.yaml" wrap theme={null}
dagBundleConfigList: '[{"name": "local-dags", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/opt/airflow/dags"}}]'
```

### Configure with container image

<Steps>
  <Step title="Build custom image">
    Create a Dockerfile extending the base agent image:

    ```dockerfile title="Dockerfile" wrap theme={null}
    FROM images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.0

    # Copy dags into the image
    COPY dags/ /dags/

    # Install additional dependencies if needed
    COPY requirements.txt /tmp/requirements.txt
    RUN pip install -r /tmp/requirements.txt
    ```
  </Step>

  <Step title="Update values file">
    Reference your custom image in `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    workers:
      - name: default-worker
        image: your-registry.example.com/custom-agent:1.0.0

    dagProcessor:
      image: your-registry.example.com/custom-agent:1.0.0

    triggerer:
      image: your-registry.example.com/custom-agent:1.0.0

    dagBundleConfigList: '[{"name": "local-dags", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/dags"}}]'
    ```
  </Step>

  <Step title="Update Helm release">
    Apply the configuration:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Step>
</Steps>

### Configure with Persistent Volume Claim

<Steps>
  <Step title="Create PVC">
    Create a PersistentVolumeClaim in your Kubernetes namespace:

    ```yaml title="pvc.yaml" wrap theme={null}
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: dags-pvc
      namespace: <your-namespace>
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 20Gi
      storageClassName: <your-storage-class>
    ```

    Apply the PVC:

    ```sh wrap theme={null}
    kubectl apply -f pvc.yaml
    ```
  </Step>

  <Step title="Configure volume mounts">
    Update `values.yaml` to mount the PVC into all components:

    ```yaml title="values.yaml" expandable wrap theme={null}
    workers:
      - name: default-worker
        volumes:
          - name: dags-volume
            persistentVolumeClaim:
              claimName: dags-pvc
        volumeMounts:
          - name: dags-volume
            mountPath: /opt/airflow/dags
            readOnly: true

    dagProcessor:
      volumes:
        - name: dags-volume
          persistentVolumeClaim:
            claimName: dags-pvc
      volumeMounts:
        - name: dags-volume
          mountPath: /opt/airflow/dags
          readOnly: true

    triggerer:
      volumes:
        - name: dags-volume
          persistentVolumeClaim:
            claimName: dags-pvc
      volumeMounts:
        - name: dags-volume
          mountPath: /opt/airflow/dags
          readOnly: true

    dagBundleConfigList: '[{"name": "local-dags", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/opt/airflow/dags"}}]'
    ```
  </Step>

  <Step title="Update Helm release">
    Apply the configuration:

    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Step>
</Steps>

## GitDagBundle compared to LocalDagBundle

Both bundle types support DAG versioning in the Airflow UI, but GitDagBundle provides additional capabilities:

| Scenario                             | LocalDagBundle                                  | GitDagBundle                                          |
| ------------------------------------ | ----------------------------------------------- | ----------------------------------------------------- |
| **Viewing previous DAG runs**        | Displays DAG as it existed at run time          | Displays DAG as it existed at run time                |
| **Creating new DAG runs**            | Uses current DAG code                           | Uses current DAG code                                 |
| **Rerunning whole previous DAG run** | Uses current DAG code                           | Uses DAG version from original run time               |
| **Rerunning individual tasks**       | Uses latest version for rerun tasks             | Uses task code from original run time                 |
| **Code changes during DAG run**      | Uses current DAG code at task start time        | Completes using bundle version from run start         |
| **Running backfills**                | Uses current DAG code                           | Uses latest bundle version                            |
| **Version creation**                 | Every structural DAG change creates new version | Every committed structural change creates new version |

## DAG versioning

Airflow 3 automatically tracks DAG versions when you use DAG bundles. Each DAG run associates with a specific DAG version visible in the Airflow UI.

Key behaviors:

* New versions are created for structural changes (tasks, dependencies, schedules)
* The scheduler uses the latest DAG version to create new runs
* You can view code for any previous DAG version in the UI
* GitDagBundle allows rerunning tasks with their original code version

See [Airflow DAG versioning](/docs/learn/airflow-dag-versioning) for detailed information about versioning behavior.

## Next steps

After configuring DAG sources:

* [Deploy Remote Execution project](/docs/astro/deploy-project-remote-execution) - Build and deploy your Airflow project
* [Configure logging](/docs/astro/remote-execution-logging-overview) - Set up task log collection
* [Configure OpenLineage](/docs/astro/remote-execution-configure-openlineage) - Enable data lineage tracking

## Related documentation

* [Remote Execution overview](/docs/astro/remote-execution-overview)
* [DAG versioning documentation](/docs/astro/dag-versioning)
* [Secrets backend configuration](/docs/astro/remote-execution-configure-secrets-backend)