The Customer
Together AI is a significant player in the modern AI stack. Backed by NVIDIA and funded to the tune of $800M in its Series C, Together AI operates an AI acceleration cloud that helps teams build, fine-tune, and deploy generative AI using open-source models. The company's inference platform serves tens of millions of requests per week and is used by researchers, startups, and enterprise teams who want access to high-performance GPU compute without the lock-in of proprietary model providers.
Running that business generates an enormous amount of data, and making sense of it is the job of a lean, high-velocity data engineering team. Their data warehouse pipelines feed reporting across the company: customer billing and revenue, inference telemetry, board-level metrics delivered to BI dashboards, ML model pipeline data from the research team, and feeds from the billing, CRM, and finance SaaS systems the business runs on. When those pipelines are healthy, reporting runs smoothly. When they're not, everyone notices.
The scale of that responsibility is matched by the scale of the data itself. Together AI's inference business generates billions of rows of telemetry weekly, and more than a dozen source systems flow into over a thousand tables across Snowflake, Athena, and MotherDuck. The engineering organization grew by an order of magnitude in 18 months, with no intention of slowing down. Automation isn't an aspiration at Together AI. It's the operating model: every function is expected to scale through tooling rather than headcount, and the data warehouse is no exception.
Data at Together AI started the way it does at most hypergrowth startups: generalist engineers did a heroic zero-to-one job standing up pipelines while the company scaled. As the business grew, a dedicated data engineering function was formed to build on that foundation, bringing data engineering rigor and a modern warehouse stack to what the early builders had bootstrapped. The team saw what the warehouse needed to become: not just reliable infrastructure, but an environment where AI agents could do much of the engineering work. Their thesis was clear: Airflow's deterministic, inspectable execution model made it the right foundation for agent-driven pipelines.
The Challenge
Before the data engineering team could build the future, they had to fix the present. The orchestration environment had grown organically as the company moved fast, without a unified orchestration strategy. It worked well enough when the company was small. It was incompatible with what the team knew was coming.
The core problems were structural:
- Nine separate MWAA environments spread across multiple AWS accounts, one per account, leaving the team managing isolated deployments with no unified visibility, no consistent patterns, and no path to consolidation.
- Bespoke Argo Workflows: 17 ML pipelines the research team had built to meet their own needs before a shared orchestration platform existed, running outside the Airflow ecosystem.
- Slow, manual CI/CD. Deploying DAG changes required a manual, multi-step process. Build times that should take seconds stretched to ten minutes or more. In a world where agents need to iterate rapidly, that's a dealbreaker.
- No observability. When a critical pipeline failed because of a missing configuration, the team had no SLA alerting, no dependency tracking, and no way to surface root cause without manual log diving. Failures were discovered after the fact.
And the pressure was only increasing. Together AI was on a trajectory to 10x its DAG count within 12 months, and the team's agent-native roadmap would require far more than that. MWAA's rigid deployment model and overhead-heavy operations made it a ceiling, not a foundation. Agents can't iterate against infrastructure that takes ten minutes to deploy to.
"We had a very high reliance on Airflow. It was super valuable. But the setup had grown organically as the company scaled: nine environments, no shared patterns, and MWAA couldn't keep up with where we were heading. We needed something that could actually scale with us." Jordan Kail Data Engineer, Together AIWhy Airflow on Astro Works Well for Agents
The team's bet on Airflow was a considered argument about what agents actually need from an orchestration platform. Their argument is direct: the question isn't whether agents can use Airflow. It's that Airflow's properties — explicit dependencies, inspectable state, mature conventions — line up unusually well with how agents work.
"What makes Airflow work well for agents is that it was built to make complex workflows explicit and inspectable. Every dependency, every retry, every failure. It's all visible. Agents don't have to guess. They read the logs, patch the code, and re-trigger. That loop depends on transparent infrastructure, and Airflow gives us that." Jordan Kail Data Engineer, Together AI
The Solution
The POV
The team's goal was never just to replace MWAA. It was to build a warehouse stack where AI agents could author, test, and ship production data pipelines autonomously. To get there, they needed a stable, production-grade Airflow foundation first. Having used Astro before, the team championed it internally from day one: a single platform with proper CI/CD and observability, replacing the patchwork they'd been managing for months.
Within days of kicking off a trial, the team had a dynamic, YAML-configured pipeline moving data from Athena to Snowflake to MotherDuck running in production on Astro before the formal POV concluded. The result made the business case better than any presentation could.
The Migration
With the POV won, the plan was clear: consolidate nine MWAA environments onto one platform, migrate the 17 Argo Workflows the ML team had been running in isolation, and standardize the development workflow so any engineer could contribute DAGs without managing infrastructure. The whole thing was done in under 30 days, with core business metrics delivered to the board before the migration was complete.
Astro delivered the platform fundamentals that made all of it possible. GitHub integration automated CI/CD end-to-end, taking branch-to-deployment from ten minutes down to seconds. Cosmos (Astronomer's open-source library that runs dbt projects natively inside Airflow) gave agents a natural runtime for data transformations: 12 dbt projects, 700+ models, each rendered as an Airflow task with proper dependencies, schema tests paired automatically, hourly silver runs and asset-triggered gold runs handled natively in Airflow 3. Astro Observe added the SLA alerting and full dependency graph that had been entirely absent, including PagerDuty for on-call and open lineage tracking across Snowflake, Athena, and MotherDuck.
What Astro Orchestrates
In practice, Astro now orchestrates Together AI's data warehouse workloads: Bronze to Silver to Gold transformations over CDC-landed data across 700+ dbt models, telemetry aggregates moving into MotherDuck, and pre-aggregated analytics reports delivered to BI dashboards for board-level review. The warehouse's batch layers run through Airflow on Astro: ingestion, transformation, quality checks, and delivery. (Streaming ingestion and other parts of the broader data platform run on separate systems.)
The Agent Layer
Then the agent layer went on top. The team connected their AI agents to Astro via the Airflow MCP server. MCP, or Model Context Protocol, is an open standard that lets AI agents connect to external tools and services, giving them the ability to read data, trigger actions, and get live feedback from systems they don't natively control. Astro's Airflow MCP implementation turns a managed Airflow deployment into a programmable interface that agents can call directly. The workflow runs with humans at the two ends — the ticket and the code review: an engineer creates a ticket specifying source tables, target schema, and SLA. An agent picks it up, reads the repo conventions, authors the dbt models and Airflow DAG, triggers a test run on Astronomer via the Airflow MCP, diagnoses any failures, patches the code, and re-triggers until the pipeline runs green. It then opens a PR linked back to the ticket, and on merge Astro deploys to production automatically.
The Results
The platform fundamentals paid off immediately. CI/CD dropped from ~10 minutes on MWAA to seconds. Manual deployment operations were replaced entirely by automated branch-based promotion. Astro Observe's SLA alerting replaced the manual log-diving that incident response used to require.
But the more significant results came from what agents built on top of that foundation. With Astro as the runtime, Together AI's agent workflow has produced 12 dbt projects in production across billing, inference, and analytics, covering 700+ dbt models largely authored and maintained by agents, with humans reviewing at the PR gate. Pipeline development time dropped from weeks to hours.
One example makes the agent advantage concrete. Together AI's inference platform uses three different internal systems that each referred to the same GPU hardware with different names. The inconsistency had gone unnoticed across all three. An agent built a reconciliation mapping automatically by correlating naming patterns across the systems, producing what the team now calls the "GPU Naming Rosetta Stone." As the team put it: "The agent produced institutional knowledge that would normally live in one person's head, or not exist at all, as a byproduct of probing the warehouse's source tables."
"Agents used to be consumers: they'd query pipelines that humans built. Now they're building the pipelines themselves. Reading the requirements, writing the dbt models, authoring the DAGs, testing on Astronomer, opening the PR that ships to prod. That's not a demo. That's our day-to-day workflow." Jordan Kail Data Engineer, Together AI
What's Next
Together AI plans to expand Astro across more batch workloads and onboard additional engineering teams to the warehouse platform, with Astronomer continuing as the orchestration and observability layer.
But the bigger picture is an industry thesis the data engineering team is living out in real time. The traditional data warehouse was built for humans: ten queries per analyst session, overnight dashboard refreshes, schema changes through committees. Keeping pace with billions of rows a week across a thousand-plus tables by hand isn't realistic for a lean team. Their answer is a warehouse where agents do much of the engineering, on infrastructure that is reliable, inspectable, and programmable enough for agents to work against.
That's what Airflow on Astro provides. Deterministic execution that agents can reason about. Cosmos as a native dbt runtime agents already understand. The Airflow MCP turning Astro from a UI for humans into an API for agents. The team's bet is that this stack is a strong foundation for agentic data engineering. Together AI is an early proof point.
