How to use Otto to automatically investigate Dag failures and PR a fix

Otto is Astronomer’s AI agent, specialized in data engineering. Otto helps you write, test, upgrade, and investigate your Airflow Dags running on Astro, and you can interact with Otto from the Astro UI, Astro CLI, and the Astro API.

This diagram shows the architecture you’ll implement in this tutorial.

• Setting up an Astro alert that gets triggered on the failure of a Dag and responds by starting a run of the auto fix Dag.
• The auto-fix Dag which uses Otto to investigate the failure having all necessary context about your environment, uses @task.agent to create a fix based on Otto’s suggestions, and submits a PR to GitHub for your review.

Assumed knowledge

To get the most out of this tutorial, you should have an understanding of:

• Basic Airflow concepts.
• How to use the GitHub REST API.

Prerequisites

• An API key of an LLM provider that is compatible with Pydantic AI. This tutorial uses an OpenAI API key. This API key is needed for the agentic tasks that create the GitHub PR. Otto interacts with models through Astronomer’s LLM gateway to run the root-cause-analysis, and does not need an API key.

Step 0: Sign up for Astro and install the Astro CLI

If you already have an Astro account and the latest version of the Astro CLI installed, continue with Step 1.

1. If you don’t have an Astro account yet, sign up for a free trial of Astro, which gives access to Otto.

You do not need to select a template in the onboarding flow. Click Or skip this and go to your workspace when asked to select a template.

2. Use the following command to install the latest version of the Astro CLI.

1
curl -sSL install.astronomer.io | sudo bash -s

Step 1: Create a new Astro Deployment

Create a new Deployment. You can choose any settings. Astronomer recommends the Development template for testing.

Step 2: Fork and clone the tutorial repository

Fork and clone this GitHub repository.

It contains a fully functional Airflow project with two Dags:

• create_tracking_labels: This is a test Dag designed to fail with an error for Otto to investigate automatically.
• otto_rca_to_gh_pr: This is the Dag that runs the investigation.

The otto_rca_to_gh_pr Dag consists of six tasks:

• parse_alert: Extracts the source Dag ID and run ID from the triggering alert. If the alert message doesn’t include a recognizable Dag ID or is missing a run ID, the task raises an error.
• dedup_check: Checks GitHub for an existing open pull request labeled with the source Dag ID. If one already exists, the task skips the rest of the Dag run to avoid opening a duplicate fix. This is a simple example of deduplication logic. You might need more complex logic in a production setup specifying the issue to be fixed by task ID and deployment ID in addition to the Dag ID.
• get_diagnosis: Requests a root cause diagnosis for the failed Dag run from Otto, using the Astro Organization ID, Deployment ID, and an investigation token.
• fetch_source_file: Retrieves the current content of the failing Dag’s source file from the base branch of the GitHub repository.
• propose_fix: An agent task that sends the diagnosis and current file content to an LLM, which proposes the smallest set of exact search-and-replace edits that resolve the diagnosed root cause.
• open_pr: Applies the proposed edits to the source file, formats the result with ruff, then creates a branch, commits the change, and opens a labeled GitHub pull request.

You can review the full Dag code. The code is modularized, and imports helper functions from the include folder. The request_diagnosis function in include/airflow_rca shows the interaction with Otto.

Step 3: Deploy the tutorial project

Run the following commands in your repository’s root to sign in to Astro and push code to your Astro Deployment, created in Step 1.

$
astro login
$
astro deploy

Step 4: Create a Deployment token

In order for an Astro alert to start a Dag in one of your Deployments, you need to give it a Deployment token.

1. On your Deployment, click Access, then API Tokens, and Add API Token.

2. Create a new token with the name AlertToken of the Standard Kind with the Deployment Admin Role.

3. Click Create API Token and store the token value in a safe location.

Step 5: Create an Astro alert

In order for any Dag failure to start the investigation Dag, you need to create an Astro alert with a Dag trigger.

1. In the Astro UI, open the Alerting dropdown and click Alerts -> + New Alert.

2. The first part of the alert form describes when the alert should fire. Select the DAG Failure type, your workspace, and Deployment name. For this tutorial we’ll scope the alert to only trigger when the create_tracking_labels Dag fails.

3. Next you need to add the DAG trigger notification channel. Click + New Notification Channel and create a channel with the name otto_rca_to_gh_pr Dag, the type DAG trigger, and aim it at your Deployment and the otto_rca_to_gh_pr Dag. Copy your Deployment API token that you saved at the end of Step 4 into the DEPLOYMENT API TOKEN field. Lastly make the channel available to your entire organization, workspace, or just to this tutorial Deployment.

4. Click Create Notification Channel to create the channel and Create Alerts to save the alert.

Step 6: Create a GitHub token

In order to be able to raise a PR on GitHub, the AI agent needs access to your GitHub repository. For a production use case, you’ll likely want to setup a scoped GitHub App. For this tutorial, you can use a scoped personal access token.

1. Go to your GitHub account’s Developer Settings and click Generate new token to create a new fine-grained token.

2. Give your GitHub any name and scope it to your fork of the tutorial repository.

3. Add four permissions: Contents (Read and write), Metadata (Read-only), Pull requests (Read and write), and Issues (Read and write).

4. Create the token and copy it to a safe location.

Step 7: Add environment variables and a connection

The last set up step is to add the necessary environment variables and the Airflow connection to your model provider to your Deployment.

1. Go to your Astro Deployment and click Environment, then Environment Variables and Edit Deployment Variables.

2. Add the following three environment variables, marking all tokens and keys as SECRET with the toggle.

   • ASTRO_API_TOKEN: You can reuse the same Astro Deployment token you created in step Step 4. This is the credential the Airflow task uses to authenticate to Astro in order to be able to run the Otto investigation.
   • GITHUB_REPO: Your fork of the tutorial repository in the format <account>/<repo>.
   • GITHUB_TOKEN: Your GitHub token, retrieved in Step 6.

3. Click Update Environment Variables to save your changes.

4. Still on your Deployment’s environment tab, click Connections to add your Pydantic AI connection for the @task.agent task that drafts your PR.

5. Select the Generic connection form and create a connection to your LLM provider. If you are using OpenAI, add the following values.

   • Connection ID: pydanticai_default
   • Connection Type: pydanticai
   • Password: Your OpenAI API Key.
   • Extra: {"model":"openai:gpt-5"}. You can use any valid model for your model provider.

Step 8: Test the Dag

Now it is time to test this setup!

1. Go to the Airflow UI and make sure both Dags are unpaused.

2. Run the create_tracking_labels Dag manually. It should fail its last task with a KeyError.

3. The failing Dag causes the Astro alert to run, and automatically triggers the otto_rca_to_gh_pr, which investigates the failure and proposes a fix.

4. Wait for the Dag to finish, then check your GitHub repository for a new PR.

Conclusion

Congratulations! You’ve implemented a self-healing Dag using Otto to investigate a failure and AI orchestration with the Common AI provider to create a PR fixing the issue. This is just one of many possible ways you can include Otto in your pipelines. Another example is to post Otto’s diagnosis to Slack.