Modern data orchestration at scale demands reliability, speed and thoughtful adoption of new tooling. As organizations grow, keeping pipelines efficient while supporting more teams becomes a critical challenge.

In this episode, we’re joined by Ethan Shalev, Data Engineer at Wix, to discuss how Wix operates Airflow at massive scale, migrates to Airflow 3 and uses AI to accelerate development.

Key Takeaways:

(00:00) Introduction.

(02:13) Wix structures data engineering across multiple product-focused organizations.

(03:40) Migrating nearly 8,000 DAGs to Airflow 3 requires careful planning.

(04:31) Migration creates an opportunity to remove long-standing legacy Airflow code.

(05:32) Internal playbooks and Cursor rules standardize and speed up DAG migrations.

(07:39) Airflow 3 introduces backfills, DAG versioning and asset-aware scheduling.

(09:16) Deferrable operators reduce scheduler congestion in large Airflow environments.

(12:54) AI-generated code still requires review and strong testing practices.

(14:52) Moving to managed Airflow reduces operational burden on internal platform teams.

(15:57) Improving multi-tenancy and UI personalization remains a key Airflow need.

Resources Mentioned:

Ethan Shalev: https://www.linkedin.com/in/eshalev/

Wix: https://www.linkedin.com/company/wix-com/ | https://www.wix.com/

Apache Airflow: https://airflow.apache.org/

Astronomer Astro Platform: https://www.astronomer.io/

Trino: https://trino.io/

Apache Iceberg: https://iceberg.apache.org/

Cursor: https://cursor.sh/

Airflow Summit: https://airflowsummit.org/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow^® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow