Airflow in Action: How Cargill Feeds Data to 60+ Teams with Airflow on Astro
4 min read |
In the Enhancing Dags for Data Processing episode of the Data Flowcast, Cargill data engineer William Orgertrice III explains why Airflow is a core component of Minerva, Cargill's new data platform. William also shares the Dag design and monitoring practices he has refined over more than four years of building with Airflow.
Listeners will learn how America’s largest private company is modernizing its data pipelines, and which best practices keep those pipelines running reliably.
The data platform that runs the business
Cargill has operated for more than 160 years, employs over 160,000 people and serves more than 125 markets across 70 countries. As a food company, its work reaches billions of people every day through customers including Chick-fil-A, McDonald's, Delta and Starbucks.
William works on the Cargill Data Platform (CDP) team, which supports more than 60 internal teams. All internal and external data flows through CDP, and the data products his team delivers support finance teams, inventory for Cargill's plants and other functions across the organization. His leadership sums up the platform's role in one line: “CDP runs the business”. Pipeline reliability therefore shapes decisions across the company.
Legacy ETL has a shelf life
Cargill is now building Minerva, a new data platform designed to deliver new data products, stronger security and modern technology. Airflow sits at its core alongside AWS, Azure and other platforms, replacing pipelines built on legacy Hadoop-based systems. William's verdict on the old stack was pragmatic: it worked, but the technology was aging, and Cargill wanted supported tools that make developers more productive.
Three Airflow capabilities most valued by the team are:
- Sensors that alert engineers when pipelines miss their SLAs, which William says other platforms do not make easy.
- Built-in dependency management to simplify how pipelines handle their upstream requirements.
- A faster development process for data ingestion that shortens the path from pipeline design to production.
Cargill runs Airflow on Astro, Astronomer's managed service. CDP teams are migrating onto Minerva, and some already run in production. William singles out monitoring as the capability where Airflow and Astro stand apart from other data tools, because early alerts let his team deliver a root cause analysis that gets production data flowing faster.
Listen: The Data Flowcast with Cargill exploring Airflow best practices
Your pipeline did not fail. That is the problem.
William's best practices come from hard experience, including an early pipeline that packed every step into a single task. He calls debugging it a complete nightmare. His recommendations:
- Document every step so other developers can see what each task does and find where it breaks.
- Give each task one purpose, so a failure points to a single step instead of an entire job.
- Send notifications on failure and success, so the team learns the outcome of every run without checking the UI.
- Monitor run duration. William has seen Dags run 10 hours when they should take one. A Dag that never fails never alerts, yet it keeps spending money, and the first sign of trouble becomes an escalation from another team.
- Embed owner and contact details through tags and Dag docs. In shared environments, a team may own the job but not the source data, and a distribution list routes the problem to the right owner.
- Limit concurrent runs so a new ingestion cannot start until the previous one finishes.
William discussed some of these at his Airflow Summit session in 2025.
Getting started
Cargill's Minerva rollout shows what modernization looks like at enterprise scale: legacy ETL replaced by Airflow on Astro, more than 60 teams served by one platform, and Dag practices that surface problems before they reach the business. William's advice applies to any team running Airflow in a shared environment, where silent failures cost both money and trust.
Listen to the full episode of the Data Flowcast to hear William's practices in his own words. Ready to run Airflow without managing infrastructure? Get started free with Astro, the easiest way to run Airflow.
