Consolidating legacy job schedulers with Airflow at Trading Technologies

Migrating two decades of legacy job orchestration is the kind of project most teams quietly avoid.

In this episode, Marc Lamberti is joined by Sanket Patel, Director of Engineering at Trading Technologies, to talk through how his team is consolidating 20 years of C#/.NET-based scheduling and a recently acquired company's stack onto Airflow, Astronomer, and Snowflake. The conversation covers orchestrator selection, observability, the operational economics of managed Airflow, the human side of migrations, and where AI fits into the new data platform.

Key Takeaways:

• 00:00 Introduction.
• 02:11 Sanket's background and role leading data platform initiatives at Trading Technologies.
• 04:32 The data landscape at TT: silos, OLTP-driven reporting, and an acquired company running orchestration on a 17-18 year old ASP.NET stack.
• 10:53 Choosing Airflow and Snowflake as the foundation of the new data platform.
• 11:36 Why Airflow won over legacy schedulers like Autosys plus Informatica, especially past the 2,000-3,000 job mark.
• 16:34 Collapsing thousands of legacy jobs into 15-20 Airflow DAGs.
• 17:38 Argo vs Airflow: community support and ecosystem as the deciding factors.
• 18:20 Why Astronomer over self-managing Airflow on MWAA or Cloud Composer: CI/CD, observability, SLAs, PagerDuty, and billing alerts.
• 24:07 The operational math: why running Airflow globally needs dedicated specialists, and why a generalist cannot handle upgrades.
• 27:00 Migration strategy and the human side: addressing the "why" so teams move off legacy tools willingly.
• 30:31 Where AI fits in: letting customers dialogue with their own data, and using cloud skills with co-pilots to generate boilerplate DAGs.
• 36:00 The honest take on AI coding tools: accelerators, not replacements for engineering judgment.

Resources Mentioned:

• Trading Technologies
• Apache Airflow
• Astronomer
• Snowflake

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.