Skip to main content

​
Overview

A large and growing share of code at many organizations is written by AI. That work is moving off the individual developer’s machine, where one developer works with one or more AI agents running with a local harness, and into deployed pipelines that orchestrate multiple agents in cloud environments and sandboxes to autonomously create draft pull requests for engineering tickets. This architecture takes one ticket, such as a bug report or a feature request, and produces a pull request for a human to approve. A planning agent decides what the change requires, developer and reviewer agents do the work in parallel as mapped tasks, and deterministic tasks run the full test suite before the pull request is opened. Code suits this pattern particularly well because the success criteria are checkable. Test suites and linters can verify whether the generated code passes existing tests and matches repository conventions.
For general information on how to orchestrate agents with Airflow, see Agent orchestration with Apache Airflow®.
For information on how to use AI agents to write data pipelines, see Develop Apache Airflow® Dags with AI.

​
Architecture

Agentic software development reference architecture diagram. A ticket is the input to a planning agent, followed by parallel developer and reviewer agents, a consolidation agent, deterministic test and linter tasks, and a pull request that a human reviews before release.
This architecture consists of six main components:
  • Planning agent: Often also referred to as the orchestrator agent. It reads the ticket, devises a plan to fulfill it, explores the codebase to find out which files the change affects, and splits the work into individual pieces. For each piece it writes the prompt and the acceptance criteria that the reviewer agents check against.
  • Developer agents: One task instance per piece of work, each using the prompt the planner wrote for it. Developer agents have access to toolsets that let them run tests and the linter while they work. For feature development, it is advisable to instruct developer agents to follow a test-driven development pattern, writing tests first, then the product code.
  • Reviewer agents: Check the developer agents’ output against the acceptance criteria for that piece and run section-specific test suites and linters using toolsets.
  • Consolidation agent: Reconciles the individual pieces into one coherent change.
  • Deterministic checks and pull request: Regular Airflow tasks run the full test suite, format the result, and post the pull request to GitHub or another version control system.
  • Human review: A person approves or rejects the pull request. Note that in this architecture, human review happens outside the pipeline in the version control system, but it is also possible to use human-in-the-loop operators within the Airflow Dag to review AI output at any point in a pipeline.
The planning, developer, and reviewer agents all have access to the same agentic resources through toolsets and MCP servers: test suites, linters, the existing codebase, documented best practices and conventions, skills, and sometimes even a channel to ask a senior (human) engineer a question. Each agent task can use a different harness and model, where the model is often chosen by the planning agent to fit the scope and complexity of the piece of work assigned to an agent. One Dag run can use a large model for the planning and consolidation work while the individual developer agents use cheaper and faster models.

​
Airflow features

  • Dynamic task mapping: The planning agent returns a list of one dictionary per piece of work, containing that piece’s prompt, model_id, and system_prompt. expand_kwargs maps over the list, creating one developer agent task instance for each dictionary in it, so the planning agent’s assessment of the ticket determines how many developer agents run.
  • @task.agent: The agent decorator in the Common AI provider. Used to define the planning, developer, reviewer, and consolidation agents. Note that you can also run AI agents with any other harness orchestrated by an Airflow task. See Run an agent with any harness.
  • Toolsets: Give the agents access to the codebase, the test suites, and the linter through MCP servers, Airflow hooks, or custom toolsets.
  • Structured output: The planning agent’s output is restricted by a schema defined as a Pydantic model to ensure the output is complete and matches what the downstream tasks need as input.
  • Automatic retries: A developer agent that hits a rate limit or a transient provider error retries automatically. With retry policies, you can determine which errors should lead to a retry and which should fail the task.
  • Durable execution: Setting durable=True on the agent tasks caches each completed model call and tool call in the task state store. A developer agent that fails partway through its piece of work resumes from its last completed step on the retry, and the test and linter runs from the previous attempt are reused.

​
Considerations

  • Give the agents success criteria and tests they can use to evaluate their own work in a loop. Agents benefit from the same coding best practices as humans, writing tests first, then continuously checking generated code against test suites and linters. Centralized conventions help prevent large differences in code generation between developer agents, which means less work for the consolidation agent.
  • Run the full test suite in a deterministic task. Code tests are deterministic, so you can run them without using tokens.
  • Balance the size of the work pieces depending on the task. Bigger pieces leave less for the consolidation agent to reconcile but give each developer agent more to do, which might result in needing more advanced models and larger context windows. Smaller pieces are often easier to work on and can be completed by fast and cheap models but require more work in the consolidation step.
  • Scope tool access in the Airflow connection. An agent will eventually do everything it is able to do with its tools. Give the pipeline credentials that can’t perform destructive actions. See Restrict what an agent can do.
  • Limit spend per agent. The planner decides how many developer agents a run creates, so cost per run varies depending on the complexity of the ticket. usage_limits on each agent task limits requests, tokens, and tool calls; exceeding a limit raises UsageLimitExceeded and fails that one task. See Control.
  • A human is responsible. In the end, a human needs to review the generated pull request and take responsibility for its downstream effects.

​
Next steps