Red Ventures Brings Reliability and Resilience to Its AI Stack with Astro

A Marketing Solutions Business Powered by AI

Companies that find themselves overwhelmed by the chaotic character of modern digital advertising often turn to Red Digital, the marketing services arm of the media company Red Ventures, to help them get the most out of their online advertising spend.

Red Ventures, the company behind brands like TV Guide, ZDNet, CNET, Metacritic, and GameSpot, created Red Digital to help clients make the most of their digital channels and engagement mechanisms, and to develop customized content that clients can use to target customers. To do this, Red Digital’s marketing analysts rely on the insights produced by Red Ventures’ custom-engineered AI stack, which enable them to distinguish authentic from manufactured engagement and to track customer engagement across multiple touchpoints.

The Limits of Ad Hoc Workflow Management

Things weren’t always so smooth, however. When director of data engineering Jason Rich first came to Red Digital, the company was using a mix of scripting and tool-specific schedulers to “orchestrate” its pipelines.

“Historically, we had separate schedulers or queues within each tool,” Rich says. “So Fivetran has a job queue, dbt has a job queue — but these queues don’t talk to one another. I don’t want my ‘T’ task in dbt to fire off until that ‘EL’ piece in Fivetran is done. So the problem becomes, how do you make them talk to one another?”

Data engineers were spending a significant amount of time on that problem, building, testing, and maintaining custom logic to orchestrate tasks and manage dependencies. And even then, there were frequent issues: It was not uncommon for a number of Red Digital’s tasks to fail to run — or successfully complete — on a given day.

Whenever tasks failed, the flow of data to the ML models that marketing analysts used to make media buys was disrupted, causing their work to come screeching to a halt.

The Power of Comprehensive Orchestration

Rich wanted to replace Red Digital’s smorgasbord of schedulers, scripts, and software dependencies with Apache Airflow, an open source framework that simplifies distributed scheduling and workflow management.

But Airflow alone wouldn’t give Red Digital a way to orchestrate data — i.e., to reliably manage interoperability among multiple concurrent workflows and dataflows.

Rich knew that Red Digital needed orchestration to knit together the disparate data pipelines that fed data to Red Digital’s marketing analysts, data scientists, and other experts. This would eliminate the common failures — and make it easier to diagnose and remediate the edge cases — that kept causing things to break in Red Digital’s production environment.

Rich turned to Astro, Astronomer’s fully managed Apache Airflow service, which he had used at a previous company. Astro is a turnkey Airflow service that provides essential capabilities unavailable with basic Apache Airflow, including built-in support for data lineage; an administrative single pane of glass Red Digital could use to create and manage its multiple Airflow deployments; and easy-to-configure integration with its version control and CI/CD platforms.

Because Astro is a fully managed service, Red Digital didn’t need to worry about standing up, managing, maintaining, or upgrading the infrastructure resources that make Airflow go. “I want our engineers to focus on data pipelines that benefit our customers, not managing Airflow or building an orchestration platform,” Rich says. “When I consider the alternatives, the productivity they get with Astro and the Airflow experts at Astronomer saves Red Ventures more than $500,000 each year.”

Supporting Team-Based Deployments

Rich wanted to be able to custom-tailor different types of Airflow environments to suit the needs of Red Digital’s diverse constituencies, including its data engineers, data scientists, and analytic engineers. Each of these teams has its own preferred tools, methods, and processes, and some teams also have their own software development repositories. By building support for these preferences and needs into custom Airflow environments, Red Digital hoped to give each team a reusable, reproducible Airflow experience that just worked. Rich knew from prior experience that Astro includes essential software that would eliminate most of the complexity involved in building, deploying, and maintaining customized Airflow deployments.

Astro Runtime gave Red Digital a secure, hardened Airflow environment it could use as a kind of template for customization: each team would start with the same base Airflow environment — Astro Runtime —and build their own team-specific customizations into it. “I was able to focus on customizing Airflow to give our data engineers and data scientists just what they needed,” Rich says.

The Astro CLI gave Red Digital’s teams a standard, reproducible Airflow development environment they could run on their desktops or laptops, making it easier for experts to build, debug, and push DAGs to production.

Finally, Astro’s central control plane gave Red Digital a single pane of glass from which it could observe and manage all of its decentralized Airflow deployments. With the basic Apache distribution of Airflow, organizations must manage each of their deployments individually, using a separate management console. This means an organization with multiple deployments must switch between multiple browser tabs to manage each of them.

Observability That Yields Actionable Insights

Beyond the visibility needed for effective administrative control, Red Digital would also need observability into the health, performance, and behavior of its Airflow deployments, including how individual teams used Airflow.

Since Apache Airflow doesn’t have a built-in way to extract lineage metadata, Rich counted on Astro’s built-in lineage capabilities to provide this observability, as well as to make it easier for support engineers to diagnose and resolve data outages. By automatically extracting lineage metadata and collecting it in the Astro control plane, Astro allows support engineers to drill down into problems and devise remediations.

Lineage metadata has also simplified root-cause analysis and impact analysis for Red Digital, as well as enabled it to design higher-level analytics, like custom metrics, alerts, and KPIs that it can use to monitor the performance of its data pipelines and the quality of its data.

With Astro, Rich says, “we know where to focus our energy, and it makes the trouble-shooting aspect a lot easier.”

Hassle-Free Upgrades

One of the things customers like best about Astro is that it enables a hassle-free upgrade experience and gives them day-one access to useful new features, bug fixes, and security patches. Red Digital recently put this to the test, upgrading to a new version of Airflow. The experience went much more smoothly than Rich anticipated. “The ease of use with Astro is just hard to describe,” he says. “To upgrade, literally all I have to do is go in and edit a configuration file and change it to whatever new version I want to run. In just a few minutes, I’m running on the latest version of Airflow, and I never have to worry about Airflow breaking when I upgrade.”

Rich says that Astro gives Red Digital peace of mind. It is installed in Red Digital’s own single-tenant cloud, enabling the company to retain control of its data and IP. And Astronomer takes care of managing, maintaining, and updating everything — not just Airflow’s core infrastructure components, but Kubernetes, too. “If I were going to run Apache Airflow myself, I would have to worry about all of the upgrades that come along with each change log release, along with upgrades to all the supporting software that Airflow depends on.”

A Partnership with a Future

According to Rich, Red Digital was looking for one more thing from Astronomer: a partnership. “I wanted the integrity of a relationship with a service provider like Astronomer,” he explains. “I didn’t want Astro to be an immediate solution to a simple-but-complex problem, I wanted a platform that could grow with us, and a company that would put the needs of customers like us first.”

Today, Red Digital depends on Astro to orchestrate the pipelines that feed the models its marketing analysts use to make media buys, and the keyword-search service it exposes to clients. The data scientists and data engineers responsible for operationalizing these models rely on Astro to acquire and transform data to support exploratory data analysis, to accelerate feature extraction and model training, and to give them a secure path from developing models to deploying them in production. So far as Rich and Red Digital are concerned, Astro-powered orchestration just works.

Going forward, Rich says, the goal is to extend the benefits of orchestration to more people across the company. “I think of how it can make a data scientist’s life a lot easier, for example, how it can give them back time in the day,” he says. “It’s a successful platform for us already, and it would be great to get it out to the masses, so a data scientist or business analyst doesn’t have to come to me and say, ‘I’d like to import this, how do I do that?’ So they can just dive in and do it themselves.”