The Customer
idealo is one of Europe's largest price comparison platforms, helping millions of shoppers compare prices and products across thousands of merchants.
idealo's engineering unit keeps the comparison engine itself accurate, current, and fast. From there, the data platform retrieves and prepares the data for internal analysis and for B2B customers. That includes selling insights back to merchants, work that spans more than 900 million offer updates for the B2B data alone. The data platform provides the scalable infrastructure and platform capabilities that empower teams across the company to continuously process merchant offer data from partners like Amazon and eBay, marketing and advertising performance data from channels like TikTok and YouTube and ad-tech platforms like Adjust, and the data foundation for internal business intelligence and analytics.
VP of Data David Hellwig, who leads that Data Unit, describes their remit in business terms: “Our job is to make sure every part of the business, from merchant offer feeds to marketing performance to internal BI, can trust the data it's built on. That trust is what lets the rest of idealo move fast.”
The data unit is made up of three teams. One of them, Processing and Analytics Tooling, owns idealo's managed ETL platform, and provides the shared pipeline infrastructure that more than 20 teams across the company build on. A team of eight (six engineers, one product owner, and one technical lead) run this data backbone day to day, enabling fellow data engineers, data analysts, and engineering teams to process data before it moves downstream, into Athena for querying and Tableau for reporting.
The Challenge
Before Astro, idealo ran its pipelines on AWS Managed Workflows for Apache Airflow (MWAA). On paper, a managed service should have meant less operational overhead. In practice, idealo's team still carried most of it.
- Full operational ownership, managed label or not: idealo's team had to stand up the MWAA environment itself and continuously monitor whether it was healthy.
- Downtime for every environment change: whether the team was adding a new Python package a Dag depended on or upgrading the Airflow version itself, MWAA required rebuilding and restarting the whole environment. That caused roughly 30 minutes of downtime, felt by every one of idealo's 20+ teams on the platform regardless of whether their own pipelines were affected.
- Lagging Airflow releases: Airflow 3 shipped well before it was available on MWAA, leaving idealo on an older version longer than it wanted to be, in part because upgrading meant more of that downtime.
- A pipeline-authorship bottleneck: every pipeline still had to be hand-written as a Python Dag. That was no obstacle for the core data engineering team, but it shut out engineering teams elsewhere in the business that needed their own pipelines, engineers who wrote backend or systems code rather than Python. Every one of their requests had to route through Processing and Analytics Tooling. As Product Owner Ola Zajac put it, not knowing Python was “the biggest blocker of adoption” for the platform, for years.
The Solution
idealo chose Astro because it removed the operational burden MWAA left on their plate, gave them faster access to new Airflow releases, and replaced downtime-heavy environment rebuilds with deploys that take minutes. The switch took about a month: a two-week proof of concept to confirm the fit, followed by two to three weeks hardening the setup for production. The one real snag was metadata: idealo had no way to connect directly to MWAA's metadata store to carry over Dag run history, so the team exported backups of its existing Dags and restored them directly on Astro instead.
Astro's central role is orchestrating idealo's managed ETL platform built on AWS Glue. Raw data reaches S3 through two paths: Airbyte handles some source syncs directly and is itself triggered by Astro, while other teams run their own Kafka-based pipelines outside of Astro's orchestration. Either way, once raw data lands in S3, Astro triggers the Glue jobs that process it, writing the transformed tables back to S3 for Athena to query, giving that step a single control layer instead of being stitched together by hand.
From there, Astro sequences the Athena tables that data lands in, then pushes a Tableau extract refresh so dashboards update automatically rather than on a manual cycle. Astro also gives the Processing and Analytics Tooling team the governance layer they need to run a shared platform: guidelines and team-level configuration that keep all 20+ teams deploying through Astro within bounds.
The diagram below shows how it all comes together, from ingestion through to the dashboards it powers.
Merchant, marketing, and operational sources reaching idealo through Astro-triggered Airbyte syncs and separate team-owned Kafka pipelines outside Astro, both landing raw data in S3, then Astro's Glue job transforming it, sequencing Athena, and triggering a Tableau refresh
The migration also changed how idealo schedules its pipelines. Before Astro, jobs ran on fixed, cron-style schedules set directly through AWS Glue's native triggers. On Astro, idealo has moved to asset-based (data-aware) scheduling, so a downstream job now fires automatically the moment its upstream dependency finishes, instead of waiting on a clock.
"Once we moved to Astro, we could schedule based on a data asset instead of a fixed time. Once the upstream job completes, the downstream job triggers automatically, and the continuity is there." Shaleena Ann Thomas Senior Data Engineer, idealo
Blueprint: opening pipeline authorship beyond the data team
Blueprint is Astronomer's no-code, self-service Dag authoring capability, live in the Astro IDE. Platform teams write governed templates in Python once; everyone else composes pipelines from those templates using YAML or a visual, drag-and-drop canvas, with no Python or Airflow knowledge required. It's built for exactly the bottleneck idealo described: teams that need pipelines but shouldn't have to become Airflow experts to get one.
Before Blueprint, standing up a new pipeline at idealo meant hand-writing a Python Dag template from scratch, a 106-line file, and working through a 3-hour onboarding document just to understand the Airflow-specific pieces required to fill it in. idealo rolled out Blueprint to its non-Python engineering teams and then took it a step further, building an internal AI-assisted authoring layer on top of it: an agent that turns a short set of user inputs directly into a validated Blueprint YAML file. Together, the two turned that 106-line template into 6 lines of YAML, and cut the process of onboarding a new team from about 3 hours down to roughly 10 minutes of filling out a short form, an 18X improvement.
Four new teams have onboarded so far, several staffed by engineers who write backend or systems code rather than Python. Ola Zajac also saw the opposite case show up: less-technical data analysts who'd previously struggled to configure a pipeline at all are now able to get a job running on their own too. Both audiences are shipping their first pipelines the same day they get started.
The drag-and-drop Blueprint builder.
"Blueprint works for both ends of our spectrum. Our most technical engineers, the ones who never wrote in Python, use it to get running fast. And some of our least technical analysts, who used to struggle just to set up a job, find it easier too. Either way, new teams get started almost entirely on their own, and some ship their first job the same day they onboard." Ola Zajac Product Owner, idealo
The Results
The operational pain idealo came in with is gone, and the team can now point to a stable, self-service platform in its place.
- 18X faster engineer onboarding: authoring a first Dag with Blueprint now takes about 10 minutes, down from the roughly 3 hours engineers used to spend hand-writing a 106-line Python template.
- 6X faster deploys: installing a new Python package once meant up to 30 minutes of downtime on MWAA; on Astro, the same change deploys in about 5 minutes with no downtime.
- 3 to 5 nines of platform availability: Upgrading even minor Airflow versions on MWAA involved roughly 30 minutes of downtime every two months, working out to about three-nines availability. With Airflow upgrades now landing without any downtime at all, idealo is operating much closer to five nines.
- Zero infrastructure incidents since migrating: per David Hellwig, VP of Data, idealo hasn't logged a single infrastructure-related incident since the move.
- More frequent Airflow upgrades: with no downtime tradeoff left to manage around, idealo now upgrades Airflow, including minor versions, far more often than it did on MWAA.
- Self-service pipeline authorship: four new teams, including non-Python engineering groups, have onboarded to Blueprint and shipped their first pipelines independently.
Those results reach past the data unit itself. Merchant offer feeds, marketing performance numbers, and internal BI all run through Astro, so when the platform underneath them stops causing incidents, the business units relying on that data feel it directly, not just the engineers maintaining it.
"Since adopting Astro, we haven't had a single infrastructure incident, and the feedback we're getting from the business has been genuinely positive. That stability doesn't just help my team. It changes how the entire business operates." David Hellwig VP Data, idealo
What's Next
idealo is lining up two new initiatives behind Astro: a rebuild of its product recommendation system, which will use the platform to build new training and serving datasets, and a new B2B data-sharing capability that processes and delivers data directly to merchant partners.
The team is also awaiting Spark lineage support to unlock full value for downstream data-product visibility and compliance use cases, and is beginning to evaluate Astronomer's data engineering agent, Otto, alongside the custom AI-assisted authoring tooling it has already built in-house.
"The whole goal of moving to Astronomer was to build the data platform for our future. That's exactly what we're still building on today." David Hellwig VP Data, idealo
