Building Resilient Data Systems for Modern Enterprises at Astrafy with Andrea Bombino
NOV 7 2024
Efficient data orchestration is the backbone of modern analytics and AI-driven workflows. Without the right tools, even the best data can fall short of its potential. In this episode, Andrea Bombino, Co-Founder and Head of Analytics Engineering at Astrafy, shares insights into his team’s approach to optimizing data transformation and orchestration using tools like datasets and Pub/Sub to drive real-time processing. Andrea explains how they leverage Apache Airflow and Google Cloud to power dynamic data workflows.
Key Takeaways:
(01:55) Astrafy helps companies manage data using Google Cloud.
(04:36) Airflow is central to Astrafy’s data engineering efforts.
(07:17) Datasets and Pub/Sub are used for real-time workflows.
(09:59) Pub/Sub links multiple Airflow environments.
(12:40) Datasets eliminate the need for constant monitoring.
(15:22) Airflow updates have improved large-scale data operations.
(18:03) New Airflow API features make dataset updates easier.
(20:45) Real-time orchestration speeds up data processing for clients.
(23:26) Pub/Sub enhances flexibility across cloud environments.
(26:08) Future Airflow features will offer more control over data workflows.
Resources Mentioned:
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
