A Marketing Solutions Business Powered by AI
Companies that find themselves overwhelmed by the chaotic character of modern digital advertising often turn to Red Digital, the marketing services arm of the media company Red Ventures, to help them get the most out of their online advertising spend.
Red Ventures, the company behind brands like TV Guide, ZDNet, CNET, Metacritic, and GameSpot, created Red Digital to help clients make the most of their digital channels and engagement mechanisms, and to develop customized content that clients can use to target customers. To do this, Red Digital’s marketing analysts rely on the insights produced by Red Ventures’ custom-engineered AI stack, which enable them to distinguish authentic from manufactured engagement and to track customer engagement across multiple touchpoints.
The Limits of Ad Hoc Workflow Management
Things weren’t always so smooth, however. When director of data engineering Jason Rich first came to Red Digital, the company was using a mix of scripting and tool-specific schedulers to “orchestrate” its pipelines.
“Historically, we had separate schedulers or queues within each tool,” Rich says. “So Fivetran has a job queue, dbt has a job queue — but these queues don’t talk to one another. I don’t want my ‘T’ task in dbt to fire off until that ‘EL’ piece in Fivetran is done. So the problem becomes, how do you make them talk to one another?”
Data engineers were spending a significant amount of time on that problem, building, testing, and maintaining custom logic to orchestrate tasks and manage dependencies. And even then, there were frequent issues: It was not uncommon for a number of Red Digital’s tasks to fail to run — or successfully complete — on a given day.
Whenever tasks failed, the flow of data to the ML models that marketing analysts used to make media buys was disrupted, causing their work to come screeching to a halt.
The Power of Comprehensive Orchestration
Rich wanted to replace Red Digital’s smorgasbord of schedulers, scripts, and software dependencies with Apache Airflow, an open source framework that simplifies distributed scheduling and workflow management.
But Airflow alone wouldn’t give Red Digital a way to orchestrate data — i.e., to reliably manage interoperability among multiple concurrent workflows and dataflows.
Red Ventures is a major media company with a marketing services arm, Red Digital, that helps clients maximize engagement with their customers and make more effective media buys.
Red Digital’s data engineers spent much of their time writing custom code to coordinate a confusing assortment of tools and job schedulers, and to recover from failures. Data pipelines failed several times a day, bringing operations to a screeching halt.
The Astro Solution
Astro gives Red Digital orchestration that just works. It makes it easy to custom-tailor different kinds of Airflow environments to suit the needs of internal teams. Push-button deployment capabilities give each team a secure, controlled path from building pipelines as code to running them in production. And built-in support for data lineage lets engineers observe and improve the performance of pipelines and the quality of data.
Astro-powered orchestration eliminates the failures that kept causing things to break in Red Digital’s production environment.
Faster Data Provisioning
Astro enables developers to iterate quickly as they build, deploy, and maintain data pipelines.
DevOps personnel have a single control plane for creating and managing Airflow deployments.
Support for data lineage gives Red Digital the insights it needs to quickly pinpoint and resolve data outages and to monitor and improve data quality.
Red Digital saves more than $500,000 a year with Astro.
I want our engineers to focus on data pipelines that benefit our customers, not managing Airflow or building an orchestration platform. The productivity they get with Astro and the Airflow experts at Astronomer saves Red Ventures more than $500k each year.
Rich knew that Red Digital needed orchestration to knit together the disparate data pipelines that fed data to Red Digital’s marketing analysts, data scientists, and other experts. This would eliminate the common failures — and make it easier to diagnose and remediate the edge cases — that kept causing things to break in Red Digital’s production environment.
Rich turned to Astro, Astronomer’s fully managed Apache Airflow service, which he had used at a previous company. Astro is a turnkey Airflow service that provides essential capabilities unavailable with basic Apache Airflow, including built-in support for data lineage; an administrative single pane of glass Red Digital could use to create and manage its multiple Airflow deployments; and easy-to-configure integration with its version control and CI/CD platforms.
Because Astro is a fully managed service, Red Digital didn’t need to worry about standing up, managing, maintaining, or upgrading the infrastructure resources that make Airflow go. “I want our engineers to focus on data pipelines that benefit our customers, not managing Airflow or building an orchestration platform,” Rich says. “When I consider the alternatives, the productivity they get with Astro and the Airflow experts at Astronomer saves Red Ventures more than $500,000 each year.”
Supporting Team-Based Deployments
Rich wanted to be able to custom-tailor different types of Airflow environments to suit the needs of Red Digital’s diverse constituencies, including its data engineers, data scientists, and analytic engineers. Each of these teams has its own preferred tools, methods, and processes, and some teams also have their own software development repositories. By building support for these preferences and needs into custom Airflow environments, Red Digital hoped to give each team a reusable, reproducible Airflow experience that just worked. Rich knew from prior experience that Astro includes essential software that would eliminate most of the complexity involved in building, deploying, and maintaining customized Airflow deployments.
Astro Runtime gave Red Digital a secure, hardened Airflow environment it could use as a kind of template for customization: each team would start with the same base Airflow environment — Astro Runtime —and build their own team-specific customizations into it. “I was able to focus on customizing Airflow to give our data engineers and data scientists just what they needed,” Rich says.
The Astro CLI gave Red Digital’s teams a standard, reproducible Airflow development environment they could run on their desktops or laptops, making it easier for experts to build, debug, and push DAGs to production.
Finally, Astro’s central control plane gave Red Digital a single pane of glass from which it could observe and manage all of its decentralized Airflow deployments. With the basic Apache distribution of Airflow, organizations must manage each of their deployments individually, using a separate management console. This means an organization with multiple deployments must switch between multiple browser tabs to manage each of them.
“I’ve got seven Airflow deployments, and Astro allows me to see everything in one place,” Rich says. “It’s a single pane of glass. I don’t have to go from console to console to console to manage my deployments. I can go to one place.”
I’ve got seven Airflow deployments, and Astro allows me to see everything in one place. It’s a single pane of glass. I don’t have to go from console to console to console to manage my deployments.