> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy a dbt project to Remote Execution Agents

<Note> This is feature is only available if you are on the **Enterprise** tier or above. See [Astro Plans and Pricing](https://www.astronomer.io/pricing/).</Note>

<Note>
  **Airflow 3**

  This feature is only available for Airflow 3.x Deployments.
</Note>

Remote Execution Agents run tasks in your own Kubernetes infrastructure. Because Remote Execution mode doesn't support `astro dbt deploy`, you must make your dbt project files available to agents through other methods.

This document covers two approaches for deploying dbt code to Remote Execution Agents, ordered by Astronomer's recommendation.

## Prerequisites

* A [Remote Execution Deployment](/docs/astro/remote-execution-overview) with at least one registered [agent](/docs/astro/remote-execution-configure-agents).
* A [secrets backend](/docs/astro/remote-execution-configure-secrets-backend) and [XCom backend](/docs/astro/remote-execution-configure-xcom-backend) configured for your agents.
* A dbt project in a Git repository.
* [Cosmos](https://astronomer.github.io/astronomer-cosmos/) installed in your agent client image for orchestrating dbt models.

<a id="option-1" />

## Option 1: Include the dbt project as a Git submodule (recommended)

Use Git submodules to include an external dbt project repository inside your Dag repository. This approach keeps your dbt and Airflow code in separate repositories while combining them at deploy time.

This is the recommended approach because it:

* Preserves separation of ownership between dbt and Airflow teams.
* Pins the dbt project to a specific commit, giving you control over which version of dbt code runs.
* Works with `GitDagBundle` for automatic Dag versioning.

### How it works

Your Dag repository includes the dbt project as a Git submodule. When agents fetch the Dag bundle from Git, the submodule contents are fetched alongside your Dags. Cosmos then reads the dbt project from the submodule folder to orchestrate dbt models as Airflow tasks.

### Configure the Dag repository

<Steps>
  <Step title="Add the dbt project as a submodule">
    From your Dag repository root, run:

    ```sh wrap theme={null}
    git submodule add <dbt-repository-url> dbt/<project-name>
    ```

    For example, to add a dbt project called `jaffle-shop`:

    ```sh wrap theme={null}
    git submodule add https://github.com/your-org/jaffle-shop dbt/jaffle-shop
    ```

    This creates a `.gitmodules` file and clones the dbt repository into the `dbt/jaffle-shop` folder.
  </Step>

  <Step title="Verify the repository structure">
    Your Dag repository should look similar to the following:

    ```text wrap theme={null}
    dag-repo/
    ├── .gitmodules
    ├── dags/
    │   └── dbt_dag.py
    └── dbt/
        └── jaffle-shop/       # Git submodule
            ├── models/
            ├── seeds/
            └── dbt_project.yml
    ```
  </Step>

  <Step title="Create a Cosmos Dag">
    Create a Dag that uses Cosmos to orchestrate the dbt project. Set the `dbt_project_path` to reference the submodule location within the Dag bundle:

    ```python title="dags/dbt_dag.py" wrap theme={null}
    from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig
    from pathlib import Path
    from datetime import datetime

    dbt_dag = DbtDag(
        project_config=ProjectConfig(
            dbt_project_path=Path("/usr/local/airflow/dags/dbt/jaffle-shop"),
        ),
        profile_config=ProfileConfig(
            profile_name="jaffle_shop",
            target_name="dev",
            profiles_yml_filepath=Path("/usr/local/airflow/dags/dbt/jaffle-shop/profiles.yml"),
        ),
        execution_config=ExecutionConfig(
            dbt_executable_path="/home/astro/dbt_venv/bin/dbt",
        ),
        schedule="@daily",
        start_date=datetime(2024, 1, 1),
        catchup=False,
        dag_id="dbt_jaffle_shop",
    )
    ```

    <Note>
      The exact path depends on your `GitDagBundle` configuration. If your bundle uses a `subdir` parameter, adjust the path accordingly.
    </Note>
  </Step>

  <Step title="Commit and push">
    ```sh wrap theme={null}
    git add .gitmodules dbt/jaffle-shop dags/dbt_dag.py
    git commit -m "Add dbt project as submodule with Cosmos DAG"
    git push
    ```
  </Step>
</Steps>

### Configure agents to fetch submodules

By default, `GitDagBundle` does not recursively fetch Git submodules. You must configure the `git_conn_id` connection to include submodule support and ensure the agent can authenticate to the submodule repository.

If the dbt submodule repository uses the same authentication as the Dag repository, configure the Dag bundle with the same `git_conn_id`. If the submodule is in a public repository, no additional authentication is required.

For private submodule repositories that require separate credentials, configure an additional Git connection in your secrets backend or `values.yaml`.

### Update the dbt submodule

When the dbt team pushes new changes, update the submodule reference in your Dag repository:

```sh wrap theme={null}
git submodule update --remote dbt/jaffle-shop
git add dbt/jaffle-shop
git commit -m "Update jaffle-shop submodule to latest"
git push
```

Agents pick up the change on the next `GitDagBundle` refresh cycle.

For more information about working with Git submodules in an Astro context, see [Use Git submodules with an Astro project](/docs/astro/best-practices/git-submodules).

<a id="option-2" />

## Option 2: Include dbt code in the agent client image

Build the dbt project directly into your custom agent client image. This approach works well when dbt code changes infrequently and you want a self-contained image.

<Steps>
  <Step title="Create a custom Dockerfile">
    Extend the base agent image to include your dbt project and dependencies:

    ```dockerfile title="Dockerfile.client" wrap theme={null}
    FROM images.astronomer.cloud/baseimages/astro-remote-execution-agent:3.1-3-python-3.12-astro-agent-1.2.0

    # Install dbt into a virtual environment
    RUN python -m venv /home/astro/dbt_venv && \
        source /home/astro/dbt_venv/bin/activate && \
        pip install --no-cache-dir dbt-postgres && \
        deactivate

    # Install Cosmos
    RUN pip install astronomer-cosmos

    # Copy dbt project into the image
    COPY dbt/jaffle-shop /usr/local/airflow/dbt/jaffle-shop
    ```
  </Step>

  <Step title="Build and push the image">
    ```sh wrap theme={null}
    docker build -f Dockerfile.client -t your-registry.example.com/custom-agent:1.0.0 .
    docker push your-registry.example.com/custom-agent:1.0.0
    ```
  </Step>

  <Step title="Update the Helm values">
    Reference the custom image in your `values.yaml`:

    ```yaml title="values.yaml" wrap theme={null}
    workers:
      - name: default-worker
        image: your-registry.example.com/custom-agent:1.0.0

    dagProcessor:
      image: your-registry.example.com/custom-agent:1.0.0

    triggerer:
      image: your-registry.example.com/custom-agent:1.0.0
    ```
  </Step>

  <Step title="Apply changes">
    ```sh wrap theme={null}
    helm upgrade astro-agent astronomer/astro-remote-execution-agent -f values.yaml
    ```
  </Step>
</Steps>

This approach requires rebuilding and redeploying the agent image every time dbt code changes. For teams that update dbt models frequently, consider Option 1 instead.

## Compare approaches

| Criteria                                      | Git submodule |       Agent image      |
| --------------------------------------------- | :-----------: | :--------------------: |
| dbt and Airflow code in separate repositories |      Yes      |           Yes          |
| Independent dbt deploy cycle                  |    Partial    |           No           |
| Works with `GitDagBundle` versioning          |      Yes      |           No           |
| Requires image rebuild for dbt changes        |       No      |           Yes          |
| Additional infrastructure                     |      None     |          None          |
| Best for                                      |   Most teams  | Infrequent dbt changes |

## Related documentation

* [Remote Execution overview](/docs/astro/remote-execution-overview)
* [Configure Dag sources](/docs/astro/remote-execution-configure-dag-sources)
* [Deploy Remote Execution project](/docs/astro/deploy-project-remote-execution)
* [Deploy dbt projects to Astro](/docs/astro/deploy-dbt-project)
* [Use Git submodules with an Astro project](/docs/astro/best-practices/git-submodules)
* [Orchestrate dbt Core jobs with Airflow and Cosmos](/docs/learn/airflow-dbt)