A Sports Betting Company That Lives on Data

FanDuel Group is a sports-tech entertainment company and the premier gaming destination in the United States. It consists of a portfolio of leading brands across gaming, sports betting, daily fantasy sports, advance-deposit wagering, and TV/media, and has a presence across all 50 states, with approximately 17 million customers and nearly 30 retail locations.

FanDuel’s internal and external stakeholders depend on timely data that feeds into reports, dashboards, and other analytics that support the organization’s day-to-day, strategic, and operational decision making.

Growing Business, Growing Data Needs

At first, open-source Apache Airflow^® handled the challenge of orchestrating FanDuel’s business intelligence (BI) and analytic workflows. But business growth led to higher daily data volumes, which fueled the demand for new sources and richer analytics.

FanDuel’s data platform team responded by increasing Airflow’s resources and reviewing ETL jobs, ensuring they were all optimized and running efficiently. But eventually, the number and complexity of the ETL pipelines that FanDuel’s business required started reaching the limits of Airflow’s built-in executor, which could not scale to handle the very large number of concurrent users and workloads. The data platform team needed an orchestration tool more powerful than what open source Airflow offered.

Astro Migration in Preparation for NFL Season

To support its robust data architecture in meeting the new demand, FanDuel partnered with Astronomer. In advance of the 2022 NFL season — which would be the company’s busiest to date in terms of daily volume of data — FanDuel worked with Astronomer’s Professional Services team to migrate its existing open source Airflow deployment to Astro, the cloud-based orchestration platform powered by Airflow. Astro gave FanDuel the elastic autoscaling capability it lacked, helping to ensure its orchestration capabilities could easily scale to meet the growing demand.

To ensure that things ran as reliably and performantly as possible in Astro, FanDuel’s data platform team and Astronomer’s Professional Services team conducted a thorough review of some of FanDuel’s most resource-intensive DAGs, looking for ways to reduce complexity and wring out extra performance at run time. They zeroed in on three custom-built Airflow Smart Sensors, replacing them with a new Airflow feature, Deferrable Operators, that allows for more efficient and reliable execution of long-running asynchronous operations.

Asynchronous operation can be cheaper and more efficient than standard, synchronous operation. A standard Airflow task schedules a job with an upstream resource and waits for it to complete. By contrast, an asynchronous task basically hands its work off to an upstream resource and “defers” itself, instantly freeing up the cloud resources used by its worker.

Apart from the positive impact on pipeline performance and stability, this change enabled FanDuel to reduce the amount of resources it uses to operate its DAGs and, as a result, lower its cloud spend.

The combination of deferrable operators and Astro’s autoscaling features enabled FanDuel to reduce the amount of cloud resources used by Airflow and its infrastructure by 35%.

Damian Grech, Director of Engineering, FanDuel

ROI: Improvements in Efficiency and Cost Reduction

In the week following the switchover of the first custom operator to a deferrable operator, FanDuel’s Airflow workers were able to handle an average of 305% more tasks each. Damian Grech, Director of Engineering for FanDuel’s data platform, says the results were even better than anticipated. “After just one of the three sensors was replaced, the combination of deferrable operators and Astro’s autoscaling features enabled the company to reduce the amount of cloud resources used by Airflow and its infrastructure by 35%,” Grech says.

“We’ve also seen a more stable usage of the infrastructure across all times,” he adds, “even during peak hours when we have heavy ELT pipelines getting executed. This helped us achieve better results, not just in meeting our internal SLAs, but also having more consistency around when stakeholders are expecting our pipelines to finish executing.”

The graph below shows that there have been no extreme peaks in Kubernetes (K8s) pod utilization following the switchover to a deferrable operator. Because FanDuel’s resource usage is now both lower and more stable, the company doesn’t need to add extra capacity in K8s — which is costly and time consuming — to accommodate demand. And now that Airflow doesn’t have to wait for extra K8s capacity to come online, tasks run faster and more reliably.

This improvement in efficiency is an early indicator of likely future infrastructure cost savings, especially once the two additional deferrable operators come online.

We’ve seen a more stable usage of the infrastructure across all times, even during peak hours when we have heavy ELT pipelines getting executed. This helped us achieve better results, not just in meeting our internal SLAs, but also having more consistency around stakeholders’ expectations.

Damian Grech, Director of Engineering, FanDuel

Conclusion

The move from open-source Airflow to Astro enabled FanDuel data teams to continue reliably delivering on their daily BI and analytical requirements even as they scale up rapidly to meet the business growth needs.

Astro’s auto-scaling has given FanDuel dependable, available orchestration. The Astro CLI gives FanDuel’s teams a local, reproducible development environment they can use to build, debug, and harden their DAGs, as well as a secure path from development to production.

Meanwhile, the work delivered by Astronomer’s Professional Services team has helped FanDuel ensure their pipelines continue running in the most optimized and efficient way possible — enabling performance, reliability, and cost improvements.