Airflow in Action: Scaling 1,000+ Dags for ML-Based Driver Safety at Descartes

At Airflow Summit 2026, Matt Stavinga and Matt McCormack of Descartes showed how a team with three data engineers scaled to more than 1,000 active Dags on Airflow.

The session covers the config-driven framework, data quality layer and automated test deployments that let the team add customers without code changes. Teams running multi-tenant pipelines will learn how to grow Dag count without growing headcount.

Risk models are only as good as ingestion

Descartes Fleet Safety™ is an AI-powered driver safety and performance management solution that feeds trucking fleets' HR, telematics and accident data into machine learning models that predict risky drivers and behaviors. Descartes, a global logistics technology company, acquired the solution through the April 2026 acquisition of Idelic, which was founded in Pittsburgh in 2016. Stavinga (senior software engineer) and McCormack (senior data engineer) each had around six years at Idelic before joining Descartes.

Data arrives in multiple formats dictated by however each customer can send it: spreadsheets over SFTP, S3 and email, APIs, queues and screen-scraped websites. Each format is another way for model inputs to potentially break.

Cheap and easy stopped scaling

Before Apache Airflow®, data ingestion ran as Python scripts on Celery, RabbitMQ and Redis, with config, schedules and logs in PostgreSQL. The stack broke as complexity grew:

• Observability relied on manual database logging, so failures were hard to diagnose.
• Testing a pipeline meant hacking together Python environments to fake a message broker.
• Scaling up and down was manual, so every load change cost engineering time.
• Deployments had no graceful shutdown, so engineers had to time each release for a moment when no jobs were running or risk losing data and rerunning pipelines.

Every config change was a code deploy

To get ahead of the complexity their initial stack had created, the team evaluated a range of data orchestration options including Cadence, Conductor and Luigi. It chose Airflow for scalability, native Python, ease of deployment, community support and observability. Descartes runs it on Astro, Astronomer's managed Airflow platform.

In the first Airflow setup, every pipeline had its own Dag template, a custom generation script took 10 to 15 minutes to render all Dags in the UI, and job config lived in JSON files in the repo. Adding a Dag or changing a customer’s settings meant a code change and a release.

Slow down to scale up

The team paused new pipeline work to rebuild around one sequence: standardization, abstraction, scalability. Descartes runs about 200 pipelines, and each Dag pairs a pipeline with a client, so one pipeline serving 70 customers produces 70 Dags. Five changes let the team absorb that growth:

• Dags read from an internal config API, so support staff fix customer settings without an engineer or a deploy.
• Each pipeline is one file that generates a Dag per client from the API, so client Dags appear and disappear automatically.
• Every pipeline accepts files over SFTP, S3 or email, so a customer who switches delivery methods needs a config update instead of an engineer.
• A config spec defines expected columns and transformation logic, so the team no longer builds new spreadsheet pipelines.
• Standardized tags, config names and task dependencies shrink a pipeline definition to a few unique lines.

Figure 1: Descartes’ standard Dag in the Airflow UI. A few lines of pipeline code produce the full extract, validate, transform and load flow. Image source.

Schema, row-level and column-level checks enforce what downstream systems require, so an employee record missing a name never reaches the models. A parquet data lake on S3 replaced the team's former "data swamp," letting support trace any record and engineers reprocess corrupt data without new development.

For releases, Jenkins creates an isolated Airflow deployment per pull request, copy-server deployments use real production data, an AI agent compares test output to production as a first QA pass, and a smoke test Dag validates framework and environment changes. With no dedicated QA engineers, support staff get an AI summary of what changed before they review.

Matt Stavinga discusses Descartes’ framework in more detail on Astronomer’s podcast, The Data Flowcast.

Next: fleets build their own Dags

Project Quick Start will let customers create and trigger Dags for trusted integrations from the Safety Suite UI. Airflow 3's human-in-the-loop capability will let customers validate their own data and approve it before it loads, and an AI chatbot will guide onboarding, spreadsheets included. Each approval step puts customer-checked data in front of the risk models.

Getting started

Watch the replay to see the framework code, the Dag it renders and the automation that keeps 1,000 Dags deployable.

Descartes built its framework on task groups and dynamically generated Dags. Download Best Practices for Running and Managing Apache Airflow® at Scale to learn how to apply those patterns, tune key configurations and troubleshoot Airflow in production, with working code examples.