Airflow in Action: Expedia’s Multi-Tenant Platform at 200+ Clusters, 14,000+ Pipelines

In his Airflow Summit session From Centralization to Autonomy, Silver Pang walks through Expedia Group’s shift from fragmented, team-by-team Apache Airflow® deployments to a platform-engineered, multi-tenant Airflow model.

You’ll learn how Expedia used Kubernetes isolation, Backstage templates, and fully automated CI/CD to cut deployment friction while improving security, compliance, and operational consistency at scale. Today Expedia runs 200+ isolated Airflow clusters supporting 180+ engineering teams and orchestrating 14,000+ unique pipelines orchestrating ~1.5 million task executions every month.

2018: Autonomy Turned Into Fragmentation

Rewind to 2018 and every Expedia team owned its pipelines end to end, using whatever tooling and deployment style they preferred. In practice, that freedom created siloed innovation and a growing cognitive load. Teams invested time in infrastructure decisions, security patches, and orchestration upgrades instead of shipping data products.

Some teams ran highly manual workflows. Others built pipelines running on sophisticated infra stacks. Some struggled with basic execution constraints. The underlying issue was opportunity cost. Expedia’s teams formed around business domains, not orchestration mechanics, and the reinvention tax kept rising.

This is where platform engineering stepped in, standardizing on Airflow. Silver frames the platform team as the group that designs and maintains shared infrastructure and guardrails so domain teams can run workflows reliably and securely without becoming infrastructure specialists.

The Multi-Tenant Pivot: Isolation Without Chaos

Expedia’s core strategy landed in the middle ground between one monolithic Airflow instance and hundreds of disconnected ones. They built a multi-tenant Airflow platform that can provision isolated, self-contained environments for each team.

The implementation centers on Kubernetes isolation. Each tenant gets its own Kubernetes namespace, which allows teams to run different Python dependencies without conflict and keeps connections, workload boundaries, and operational blast radius separate. This namespaced model also simplifies compliance by design: each cluster assumes a least-privilege IAM role and integrates with a centralized vault to fetch only authorized credentials.

Teams keep flexibility where it matters. Each tenant can define its own Terraform and Docker configuration, including “flavored” dependencies or internal operators, while the platform team maintains a secure, scanned base image.

Figure 1: Multi-tenant Airflow architecture at Expedia supporting 200+ isolated clusters and 180+ engineering teams. Image source.

Templates That Make the Platform the Default

Silver makes the point that architecture only works if the developer experience drives adoption. To pull teams onto the platform, Expedia made the “right” path fast. They use Backstage as an internal developer portal to generate production-ready pipeline repositories in minutes. The generated repo comes pre-populated with boilerplate, utilities, Dag structure, and tests.

The goal is to compress time-to-deploy for a new pipeline and keep engineers focused on business logic, not Airflow scaffolding or deployment mechanics.

CI/CD as the Engine Room

CI/CD turns the multi-tenant model into an operating system. After engineers open a pull request, GitHub Actions automatically runs tests, vulnerability scanning, and builds artifacts including Docker images. Deployments publish to centralized artifact repositories, then downstream tooling syncs Dags, variables, and connections into the destination cluster based on a simple config.

Most of the process is hands-off. Silver describes it as fully-automated with status checks in the pull request confirming what shipped and where. The payoff is fewer manual errors, standardized delivery across teams, and less routine deployment work for DevOps and platform engineers.

Scale and Impact: Numbers That Prove It

As noted earlier, Expedia’s platform now runs 200+ isolated Airflow clusters supporting 180+ engineering teams and orchestrating 14,000+ pipelines. Adoption continues, with 20+ new pipelines created each month via templates.

The operational delta is material. Standing up and managing a cluster used to take about a full week of a full-time engineer. With the platform, teams spend about a day to configure what they need, plus roughly a day of platform support, which Silver calls about a 50% reduction in effort. Dag deployment speed also tightened: from 15 minutes to under 5 minutes across teams.

The multi-tenant model unlocked additional benefits beyond speed: cleaner DevOps operations, cost attribution through chargebacks, and the ability to run multiple Airflow versions side-by-side so teams can migrate on their own schedule. Silver flags the next milestone: an upcoming migration to Airflow 3.

The Hard Parts: Consistency and Onboarding

Decentralized repos create predictable obstacles. Expedia hit the consistency problem first: shared utilities, variables, and connections can devolve into copy-paste sprawl. Their solution was twofold: a centralized Python library for shared utilities and a configuration-as-code approach for variables and connections stored in a secure repo and synced through CI/CD.

They also faced the learning curve of an opinionated workflow. Their response combined documentation, a community support channel, and even an internal developer assistant that helps engineers troubleshoot by surfacing job and cluster context, access gaps, and operational health signals.

Takeaways: Platform Wins When It Balances Power and Guardrails

Expedia’s story provides a number of lessons: standardize the infrastructure, isolate tenants to reduce risk, automate delivery so teams stop thinking about deployment, and reduce adoption friction with templates. The deeper lesson is cultural and architectural: platform engineering succeeds when it pairs secure, reliable foundations with real autonomy for domain teams.

Watch the replay to see the full walkthrough of Expedia’s multi-tenant architecture, templates, and CI/CD flow.

If you want these benefits without building the platform yourself, a managed Airflow layer can get you there faster. With Astro, the fully-managed Airflow service from Astronomer, you spin up isolated Airflow environments with guardrails already in place, without building the Kubernetes, security, and upgrade machinery yourself. You standardize “golden path” deployment through CI/CD, images, secrets, and runtime configuration so each team can ship Dags quickly while the platform enforces reliability, cost controls, and consistent operations across tenants.