How Black Wealth Data Center Scaled Their Impact with Astro

The Black Wealth Data Center (BWDC), conceived and funded by Bloomberg Philanthropies’ Greenwood Initiative, is on a mission to advance racial wealth equity by providing actionable data to inform policy and investment decisions, especially at the city level. The BWDC aggregates data across five key topic areas – assets and debt, business ownership, education, employment, and homeownership – from both public and private sectors and presents it in easily digestible formats for leaders across every industry as well as non-data experts.

The Challenge

The BWDC faced challenges in its early stages of growth. They began without any data engineers, relying solely on data scientists to manually manage and process vast amounts of data. As the number of datasets grew, so did the complexity of managing them. It became evident that they needed a robust system to automate and streamline their data processes. 

The Airflow Journey to Astro

Recognizing the need for automation, the team turned to Apache Airflow®, a platform to programmatically author, schedule, and monitor workflows. One of their lead data scientists had prior experience with Airflow and knew its potential. 

However, as they scaled, they encountered challenges with version management, cluster updates, and package management. “We needed to quickly scale our data pipelines without proportionally scaling the team,” highlighted Amee Amin, Lead Data Scientist at Black Wealth Data Center. "As part of scaling our data pipelines, we wanted to upgrade our Airflow environment and use the latest versions of Airflow without needing the expertise and maintenance time to rebuild and upgrade the entire Airflow cluster every time.”

These issues led them to begin exploring solutions that would require less work to maintain, and free up their team’s valuable time to build new use cases. 

Enter Astronomer

To address these challenges, the BWDC turned to Astronomer after being referred by a friend who had already migrated to Astro. Astronomer not only provided an updated and managed Airflow environment, but also offered hands-on support throughout the migration process. With Astronomer's assistance, the BWDC smoothly transitioned their workflows, even addressing deprecated syntax and other issues that arose due to Airflow version differences. As Amin proclaims, “It's Airflow knowledge as a service as well.” 

Astronomer's integrated environment also allowed the BWDC to utilize tools like dbt in conjunction with Airflow. To simplify their workflow management and avoid the complexity of using dbt Cloud and managing their pipelines from two different user interfaces, they began leveraging Cosmos to manage their dbt workflows. “Cosmos and Astronomer gave us a single pane of glass to monitor our entire data pipeline at once, allowing us to reap the benefits of dbt without sacrificing visibility,” stated Amin. This integration streamlined their data processing, making it easier to manage and visualize their pipelines, especially as they scaled. 

With Astronomer, the BWDC has achieved remarkable progress. Their data scientists now have the autonomy to create and manage their own data pipelines, enabling them to scale efficiently and overcome the challenges of managing increasing datasets with a small team. 

Furthermore, they improved data quality by using tools like dbt and Cosmos to ensure the reliability and consistency of their data, crucial for their mission of informing policy decisions. This new found efficiency allows the team to shift their focus away from managing and troubleshooting their Airflow environment and concentrate on their primary goal of advancing racial wealth equity. 

The Future

The BWDC has ambitious plans in the year ahead, which includes increasing the amount of datasets available on the platform. With Astronomer by their side, they're well-equipped to continue making a significant impact in their domain.