In this episode, we take a step back from implementation-specific topics to explore what it actually takes to build a career in data engineering — and how AI is reshaping that path.

Shrividya Hegde, a data and AI engineer and an Airflow champion in Astronomer’s Champions program, joins us to discuss getting into data engineering, contributing to open source and why good data engineering should make AI output trustworthy rather than confidently wrong.

Key Takeaways:

• 00:00 Introduction.
• 04:08 Build fundamentals before chasing trending tools — understanding what a tool does, why it exists and what problem it solves has to come first.
• 07:19 Data engineering fundamentals mean SQL query performance under joins and aggregations, how data moves between pipelines, DAG failure recovery and idempotency — not just writing queries.
• 08:10 The most common mistake newer data engineers make is skipping fundamentals to chase trends — it is a sequencing problem, not a talent problem.
• 13:15 AI creates more opportunity for data engineers because AI output quality is directly determined by the quality of the data pipeline feeding it — confidently wrong output is harder to catch than obviously wrong output.
• 15:06 Airflow's supporting operators make AI outputs production-ready — orchestration is what converts experimental AI into something reliable.
• 17:14 AI-generated DAGs help newer engineers understand underlying concepts rather than just producing working code.
• 23:12 The Airflow open source community is more welcoming than most people expect for a project of its size — raising issues and reviewing PRs are viable entry points for first contributions.

Resources Mentioned:

• Astronomer
• Women in Data website
• Apache Airflow Slack
• Shrividya's Medium writing
• Shrividya’ Substack writing

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.