Lessons from the past decade of data engineering reveal how much the ecosystem has changed and what has stayed surprisingly consistent.

In this episode, Benjamin Rogojan, Owner and Data Consultant at Seattle Data Guy, joins us to reflect on how the data engineering landscape has evolved alongside Apache Airflow. We explore when Airflow makes sense as an orchestrator, why batch processing is still dominant and how AI is reshaping the workflows and responsibilities of modern data engineers.

Key Takeaways:

00:00 Introduction. 03:00 Airflow becomes valuable when workflows involve many pipelines, teams and dependencies. 05:00 Data engineers are still focused on making data accessible and aligning work with business needs. 05:30 Batch pipelines remain the most common approach even as real-time use cases grow. 07:45 Many “real-time” requests are actually event-driven batch workflows. 09:00 Airflow replaced many custom-built pipeline systems with built-in dependency management. 11:00 Modern orchestration tools often build on Airflow concepts or differentiate from them. 14:00 AI can assist with writing SQL and pipelines but still requires experienced engineers. 15:30 Organizations are collecting increasingly granular data creating more engineering demand. 19:00 The data stack has shifted rapidly from Hadoop-era systems to modern cloud platforms.

Resources Mentioned:

Apache Airflow Airflow Summit / Airflow Conference Snowflake HubSpot Data Sharing / APIs MLflow

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow^® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.