Astronomer's the Dataflow Cast

Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich

Efficient orchestration and maintainability are crucial for data engineering at scale. Gil Reich, Data Developer for Data Science at Wix, shares how his team reduced code duplication, standardized pipelines, and improved Airflow task orchestration using a Python-based framework built within the data science team.

In this episode, Gil explains how this internal framework simplifies DAG creation, improves documentation accuracy, and enables consistent task generation for machine learning pipelines. He also shares lessons from complex DAG optimization and maintaining testable code.

Key Takeaways:

(03:23) Code duplication creates long-term problems.

(08:16) Frameworks bring order to complex pipelines.

(09:41) Shared functions cut down repetitive code.

(17:18) Auto-generated docs stay accurate by design.

(22:40) On-demand DAGs support real-time workflows.

(25:08) Task-level sensors improve run efficiency.

(27:40) Combine local runs with automated tests.

(30:09) Clean code helps teams scale faster.

 

Resources Mentioned:

Wix website

DS DAG Framework

Apache Airflow

https://www.astronomer.io/events/roadshow/london/

https://www.astronomer.io/events/roadshow/new-york/

https://www.astronomer.io/events/roadshow/sydney/

https://www.astronomer.io/events/roadshow/san-francisco/

https://www.astronomer.io/events/roadshow/chicago/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

Be Our Guest

Interested in being a guest on The Data Flowcast? Fill out the form and we will be in touch.

By proceeding you agree to our Privacy Policy, our Website Terms and to receive emails from Astronomer.

Build, run, & observe
your data workflows.
All in one place.

Try Astro today and get up to $500 in free credits during your 14-day trial.