The Customer
Booking.com is one of the world’s largest online travel platforms, serving hundreds of millions of travelers across more than 200 countries. Behind every search result, recommendation, and price decision sits one of the largest data and AI platforms in Europe: the Booking Data Exchange (BDX).
BDX powers data movement across the entire business, supporting teams in engineering, analytics, product, marketing, finance, and customer operations. It processes petabytes of data and supports hundreds of AI and machine learning models that generate billions of predictions every day, used by thousands of data practitioners across the organization.
The Booking.com AI Trip Planner is powered by BDX. Source: Booking.com
BDX is the foundation of Booking.com’s long-term vision: the connected trip, a customer experience where flights, accommodations, transportation, and activities are seamlessly coordinated through AI and real-time data. At the center of it all is orchestration.
"Virtually every important workload we run, from analytics and business intelligence to machine learning and AI, depends on orchestration. Airflow is the single standard for making that work happen." Huy Dao Director of Data & ML Platform, Booking.com
The Challenge
Booking.com’s data platform didn’t start in the cloud. It started in a different era, built on-premises on Hadoop and Oozie for batch processing. It worked, but it was not built for real-time data, machine learning, and AI-driven experiences. As Booking.com moved to a cloud-first architecture, orchestration became the bottleneck. Early attempts to run Airflow, first on-premises, then self-managed in the cloud, exposed a hard truth:
"We were running at a fraction of today’s scale, and even then we weren’t seeing the scalability and reliability we required." Jeroen Schmidt Senior Data Engineer, Booking.com
A migration too big to fail
The transformation ran on several fronts at once:
- Moving off legacy Hadoop-based infrastructure
- Replatforming data and ML workloads to the cloud
- Scaling a shared platform across dozens of engineering teams
- Expanding AI and machine learning use cases across the business
The stakes were exceptionally high. A significant share of Booking.com’s direct costs, including bookings, marketing spend, and partner payouts, depend on pipelines orchestrated through BDX. This was one of the largest data migrations ever in Europe. It could not fail and it could not disrupt the business, which meant Booking.com needed a platform that could scale reliably and eliminate operational overhead entirely.
The Solution
After self-managing open source Airflow, Booking.com standardized on Astro, Astronomer’s managed Airflow platform, to serve as the orchestration backbone of its modern data and AI platform on AWS. The objective was clear: support the mission-critical migration and deliver reliability at massive scale, while keeping engineering resources focused on innovation rather than managing Airflow infrastructure.
Migrating one of Europe’s largest data platforms
"Our data migration, one of the largest in European history, would not have been possible without Astronomer. The amount of downtime we would have faced running Airflow ourselves would have been unimaginable." Jeroen Schmidt Senior Data Engineer, Booking.com
The migration was anything but trivial. Booking.com's self-managed Airflow environment was deeply woven into its internal systems, and moving it to externally-hosted cloud infrastructure meant untangling the open-source Airflow service from the company's internal network and service mesh.
Through Astronomer’s Center of Excellence (CoE) professional services engagement, the teams worked side by side to resolve complex network and infrastructure challenges, establish repeatable migration patterns, and provide hands-on support during critical phases. Momentum accelerated quickly. Within a matter of months, Booking.com migrated thousands of DAGs, completing one of the largest multi-petabyte data platform migrations in Europe without disrupting business-critical systems.
One platform team, building for everyone else
Behind Astro sits a deliberate operating model. A central platform team, Workflow Management, builds and runs the shared service, while dozens of consumer teams across BDX build on top of it. The platform team owns an in-house operator framework, with standardized operators for PySpark, dbt, and Booking.com’s internal data-asset and execution services, so every team builds pipelines the same way without touching the underlying infrastructure.
Teams across the company author and amend their own DAGs through a simple workflow definition, run Spark, dbt, and ML pipelines, debug with full lineage, and spin up isolated development environments, all on the same managed platform.
One platform team provides the service that dozens of consumer teams across Booking.com build on.
"It’s incredibly easy to spin up isolated environments. Every developer can test on a real deployment instead of their local machine, with the same AWS integrations and network access as production. Scaling horizontally with Astro has been straightforward, and we don’t have to manage the infrastructure ourselves. At our current scale, staying on self-managed Airflow would have been a tough place to be." Alexander Shmidt Senior Software Developer, Booking.com "Maintaining the infrastructure is far easier with Astro. Version upgrades and changes that we used to worry about just aren’t a concern anymore, and Astro brings deep Airflow expertise that gives us better insight into how things should be done." Omid Karami Software Engineer, Booking.com
Orchestrating LLMs and AI Agents at Booking.com Scale
With the platform in place, Booking.com built the data foundation for its growing suite of customer-facing AI tools on Astro. Hundreds of AI data pipelines run on the platform, feeding publicly launched experiences across the booking journey, including the AI Trip Planner, AI Trip Support, AI Voice Support, the AI Rental Helper, and the agentic Smart Messenger and Auto-Reply tools that Booking.com describes as its first in-house agentic AI. These experiences serve travelers and partners in real time downstream, and the data foundation they depend on is built and kept fresh on Astro.
"Bringing our first in-house agentic AI solutions to market is an exciting milestone and, together with our latest AI innovations, we’re making strides on delivering new levels of personalization and a faster, more seamless platform." James Waters, Chief Business Officer Chief Business Officer, Booking.com 1
How Booking.com’s travel AI gets its data, orchestrated end to end on Astro.
The pattern is consistent across each experience. Raw user journeys are collected and normalized with PySpark, then turned into embeddings through scheduled SageMaker batch jobs. Those embeddings power retrieval-augmented generation (RAG), giving each assistant relevant, up-to-date context to draw on, while other pipelines refresh agent memory and sanitize sensitive data before it reaches a model. Quality stays closed-loop: LLM evaluation pipelines score assistant responses, and OpenTelemetry traces from the agentic stack are ingested and modeled with dbt so teams can see how the systems behave in production.
Astro sits between Booking.com’s data and AI services and the assistants that travelers interact with.
The result is a single orchestration layer between Booking.com’s data and AI/ML services and the experiences travelers interact with. Astro keeps these pipelines running reliably against Snowflake, SageMaker, and the LLM tooling layered on top, so the data behind every assistant stays fresh.
"Astronomer and Airflow enable many of our ML & AI use cases." Jeroen Schmidt Senior Data Engineer, Booking.com
From Reactive to Proactive
As orchestration scaled, visibility became increasingly important, not just for the platform team, but for the many engineers across Booking.com who rely on data pipelines every day. Booking.com turned on Astro’s observability to bring a unified view across its pipelines, replacing fragmented tooling and reactive debugging. Despite a highly customized Airflow environment, initial implementation took only a matter of days.
The capabilities are designed to change how reliability is managed:
- Engineers can trace issues end-to-end across pipelines, tables, and dependencies without switching tools
- Teams can search across all deployments to quickly locate failing DAGs, datasets, or jobs
- AI-powered log summaries reduce the time to understand failures and identify root causes
- Proactive alerting surfaces issues before downstream systems and business processes are impacted
The goal is a shift from a central platform team investigating every issue toward teams across the organization debugging and resolving problems independently, reducing bottlenecks and improving time to resolution.
AI for the engineers themselves: Otto
Observability is not the only place AI shows up for the platform team. Booking.com is also among the heaviest users of Otto, Astronomer’s AI agent for data engineering. Teams use it to investigate failures by surfacing likely root causes from logs, and engineers leaned on it heavily during the Airflow 3 migration.
"We’ve been using Otto throughout our Airflow 3 migration, asking it to examine our codebase, flag potential issues, and summarize what we need to do. That’s been genuinely helpful." Alexander Schmidt Senior Software Developer, Booking.com
The Results
Astro enabled Booking.com to operate its data and AI platform with high reliability, at massive scale, and with growing self-service across the organization. For a platform where pipelines directly impact bookings, payments, and partner payouts, these improvements translate into reduced business risk and more reliable customer experiences. Beyond the headline outcomes above, that has meant:
- Thousands of DAGs migrated in a matter of months, supporting one of Europe’s largest data platform migrations
- Hundreds of ML and AI models orchestrated across fraud detection, ranking, personalization, and agents
- Hundreds of thousands of task runs per day powering data, ML, and AI workflows
"Astro is the front door to our data platform, and first impressions there really matter. It gives engineers a single place to see what’s happening across the system, including pipelines, dependencies, deployments, and data, so they can understand and debug problems quickly, without waiting on the platform team." Jeroen Schmidt Senior Data Engineer, Booking.com
What’s Next
Booking.com continues to build on this foundation with Astro at the core. The team is upgrading to Airflow 3 to unlock event-driven orchestration, more responsive machine learning workflows, and role-based access for teams, and it is exploring Astronomer’s Cosmos to unify dbt and Airflow in a single workflow.
"Astronomer is a key partner, not just for where we are today, but for where we’re going. We’re excited to be working with them on next-generation agentic capabilities to improve developer experience and productivity. The partnership helps us move faster and build with confidence at scale." Jeroen Schmidt Senior Data Engineer, Booking.com
