> ## Documentation Index
> Fetch the complete documentation index at: https://astronomer.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# How to use Otto to automatically investigate Dag failures and PR a fix
This feature is in [Labs](/docs/astro/feature-previews).
[Otto](/docs/astro/otto-overview) is Astronomer's AI agent, specialized in data engineering. Otto helps you write, test, upgrade, and investigate your Airflow Dags running on Astro, and you can interact with Otto from the Astro UI, Astro CLI, and the Astro API.
This diagram shows the architecture you'll implement in this tutorial.
* Setting up an Astro alert that gets triggered on the failure of a Dag and responds by starting a run of the auto fix Dag.
* The auto-fix Dag which uses Otto to investigate the failure having all necessary context about your environment, uses `@task.agent` to create a fix based on Otto's suggestions, and submits a PR to GitHub for your review.
## Assumed knowledge
To get the most out of this tutorial, you should have an understanding of:
* Basic [Airflow concepts](/docs/learn/intro-to-airflow).
* How to use the [GitHub REST API](https://docs.github.com/en/rest).
## Prerequisites
* An API key of an LLM provider that is [compatible with Pydantic AI](https://pydantic.dev/docs/ai/models/overview/). This tutorial uses an OpenAI API key. This API key is needed for the agentic tasks that create the GitHub PR. Otto interacts with models through Astronomer's LLM gateway to run the root-cause-analysis, and does not need an API key.
## Step 0: Sign up for Astro and install the Astro CLI
If you already have an Astro account and the latest version of the [Astro CLI](https://www.astronomer.io/docs/astro/cli) installed, continue with [Step 1](#step-1-create-a-new-astro-deployment).
1. If you don't have an Astro account yet, sign up for a [free trial of Astro](https://www.astronomer.io/lp/signup/), which gives access to Otto.
You do not need to select a template in the onboarding flow. Click **Or skip this and go to your workspace** when asked to select a template.
2. Use the following command to install the latest version of the [Astro CLI](https://www.astronomer.io/docs/astro/cli).
```sh theme={null}
curl -sSL install.astronomer.io | sudo bash -s
```
If you cannot install the Astro CLI locally, you can still complete this tutorial by deploying the tutorial repository directly from GitHub using the [GitHub integration](/docs/astro/deploy-github-integration).
## Step 1: Create a new Astro Deployment
[Create a new Deployment](/docs/astro/create-deployment). You can choose any settings. Astronomer recommends the `Development` template for testing.
## Step 2: Fork and clone the tutorial repository
[Fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) and [clone](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) this [GitHub repository](https://github.com/astronomer/otto-rca-auto-fix-tutorial).
It contains a fully functional Airflow project with two Dags:
* `create_tracking_labels`: This is a test Dag designed to fail with an error for Otto to investigate automatically.
* `otto_rca_to_gh_pr`: This is the Dag that runs the investigation.
The `otto_rca_to_gh_pr` Dag consists of six tasks:
* `parse_alert`: Extracts the source Dag ID and run ID from the triggering alert. If the alert message doesn't include a recognizable Dag ID or is missing a run ID, the task raises an error.
* `dedup_check`: Checks GitHub for an existing open pull request labeled with the source Dag ID. If one already exists, the task skips the rest of the Dag run to avoid opening a duplicate fix. This is a simple example of deduplication logic. You might need more complex logic in a production setup specifying the issue to be fixed by task ID and deployment ID in addition to the Dag ID.
* `get_diagnosis`: Requests a root cause diagnosis for the failed Dag run from Otto, using the Astro Organization ID, Deployment ID, and an investigation token.
* `fetch_source_file`: Retrieves the current content of the failing Dag's source file from the base branch of the GitHub repository.
* `propose_fix`: An agent task that sends the diagnosis and current file content to an LLM, which proposes the smallest set of exact search-and-replace edits that resolve the diagnosed root cause.
* `open_pr`: Applies the proposed edits to the source file, formats the result with `ruff`, then creates a branch, commits the change, and opens a labeled GitHub pull request.
You can review the [full Dag code](https://github.com/astronomer/otto-rca-auto-fix-tutorial/blob/main/dags/otto_rca_to_gh_pr.py). The code is modularized, and imports helper functions from the `include` folder. The `request_diagnosis` function in [`include/airflow_rca`](https://github.com/astronomer/otto-rca-auto-fix-tutorial/blob/main/include/airflow_rca.py) shows the interaction with Otto.
The failing Dag and the auto-fix Dag are running in the same Astro Deployment for convenience in the context of the tutorial. You can use the `otto_rca_to_gh_pr` Dag to investigate any Dag failures in any of your Astro Deployments.
## Step 3: Deploy the tutorial project
Run the following commands in your repository's root to sign in to Astro and push code to your Astro Deployment, created in [Step 1](#step-1-create-a-new-astro-deployment).
```bash theme={null}
astro login
astro deploy
```
If you cannot install the Astro CLI locally, you can still complete this tutorial by deploying the tutorial repository directly from GitHub using the [GitHub integration](/docs/astro/deploy-github-integration).
You can also interact with Otto from the Astro CLI. Just run `astro otto` and ask anything about Airflow, Astro, and your projects.
## Step 4: Create a Deployment token
In order for an Astro alert to start a Dag in one of your Deployments, you need to give it a Deployment token.
1. On your Deployment, click **Access**, then **API Tokens**, and **Add API Token**.
2. Create a new token with the name `AlertToken` of the `Standard` Kind with the `Deployment Admin` Role.
3. Click `Create API Token` and store the token value in a safe location.
## Step 5: Create an Astro alert
In order for any Dag failure to start the investigation Dag, you need to create an [Astro alert](/docs/astro/alerts) with a Dag trigger.
1. In the Astro UI, open the **Alerting** dropdown and click **Alerts** -> **+ New Alert**.
2. The first part of the alert form describes *when* the alert should fire. Select the `DAG Failure` type, your workspace, and Deployment name. For this tutorial we'll scope the alert to only trigger when the `create_tracking_labels` Dag fails.
Note that if you are running the `otto_rca_to_gh_pr` Dag in the same Deployment as Dags that need to be fixed, be careful about selecting `All DAGs` for the alert. You want to avoid a situation where any failure in the `otto_rca_to_gh_pr` triggers another run of the same, creating an infinite loop.
3. Next you need to add the `DAG trigger` notification channel. Click `+ New Notification Channel` and create a channel with the name `otto_rca_to_gh_pr Dag`, the type `DAG trigger`, and aim it at your Deployment and the `otto_rca_to_gh_pr` Dag. Copy your Deployment API token that you saved at the end of [Step 4](#step-4-create-a-deployment-token) into the `DEPLOYMENT API TOKEN` field. Lastly make the channel available to your entire organization, workspace, or just to this tutorial Deployment.
4. Click `Create Notification Channel` to create the channel and `Create Alerts` to save the alert.
## Step 6: Create a GitHub token
In order to be able to raise a PR on GitHub, the AI agent needs access to your GitHub repository. For a production use case, you'll likely want to setup a scoped [GitHub App](https://docs.github.com/en/apps/creating-github-apps/about-creating-github-apps/about-creating-github-apps). For this tutorial, you can use a scoped personal access token.
1. Go to your [GitHub account's Developer Settings](https://github.com/settings/personal-access-tokens) and click **Generate new token** to create a new fine-grained token.
2. Give your GitHub any name and scope it to your fork of the tutorial repository.
3. Add four permissions: Contents (Read and write), Metadata (Read-only), Pull requests (Read and write), and Issues (Read and write).
4. Create the token and copy it to a safe location.
## Step 7: Add environment variables and a connection
The last set up step is to add the necessary environment variables and the Airflow connection to your model provider to your Deployment.
1. Go to your Astro Deployment and click **Environment**, then **Environment Variables** and **Edit Deployment Variables**.
2. Add the following three environment variables, marking all tokens and keys as `SECRET` with the toggle.
* `ASTRO_API_TOKEN`: You can reuse the same Astro Deployment token you created in step [Step 4](#step-4-create-a-deployment-token). This is the credential the Airflow task uses to authenticate to Astro in order to be able to run the Otto investigation.
* `GITHUB_REPO`: Your fork of the tutorial repository in the format `/`.
* `GITHUB_TOKEN`: Your GitHub token, retrieved in [Step 6](#step-6-create-a-github-token).
3. Click **Update Environment Variables** to save your changes.
4. Still on your Deployment's environment tab, click **Connections** to add your Pydantic AI connection for the `@task.agent` task that drafts your PR.
5. Select the `Generic` connection form and create a connection to your LLM provider. If you are using OpenAI, add the following values.
* **Connection ID**: `pydanticai_default`
* **Connection Type**: `pydanticai`
* **Password**: Your OpenAI API Key.
* **Extra**: `{"model":"openai:gpt-5"}`. You can use any valid model for your model provider.
Alternatively, you can use another LLM provider, as long as it is [compatible with Pydantic AI](https://pydantic.dev/docs/ai/models/overview/) and you adjust the extra provided to the `pydantic-ai-slim[]` in the `requirements.txt` file for your Astro project. For more options on how to configure the Pydantic AI connection used with the Common AI provider's `@task.agent` decorator, see the [Common AI provider connection documentation](https://airflow.apache.org/docs/apache-airflow-providers-common-ai/stable/connections/pydantic_ai.html).
## Step 8: Test the Dag
Now it is time to test this setup!
1. Go to the Airflow UI and make sure both Dags are unpaused.
2. Run the `create_tracking_labels` Dag manually. It should fail its last task with a `KeyError`.
3. The failing Dag causes the Astro alert to run, and automatically triggers the `otto_rca_to_gh_pr`, which investigates the failure and proposes a fix.
4. Wait for the Dag to finish, then check your GitHub repository for a new PR.
## Conclusion
Congratulations! You've implemented a self-healing Dag using Otto to investigate a failure and AI orchestration with the Common AI provider to create a PR fixing the issue. This is just one of many possible ways you can include Otto in your pipelines. Another example is [to post Otto's diagnosis to Slack](/docs/astro/otto-investigate#example-post-the-diagnosis-to-slack).