Running Airflow 3 in a regulated environment at OTPP

Running Apache Airflow at a major pension fund means balancing strict compliance requirements with the need to move fast on new capabilities. On this episode, Kowsy Narayan, Cloud Data Platform Lead, Data Engineering at Ontario Teachers' Pension Plan, joins host Kenten Danas to walk through OTPP's cloud migration, their move to Airflow 3, and going fully live on remote execution.

Key Takeaways

• 00:00 Introduction.
• 01:18 Inside the OTPP data platform team and what they're responsible for across cloud migration, standards, and enablement.
• 02:33 What's driving OTPP's multi-year move off on-prem to a cloud architecture built around scalability and resilience.
• 02:57 The new stack: Snowflake as the enterprise data platform, dbt for transformation, and Airflow as the orchestrator in the middle.
• 04:15 Why OTPP chose Astronomer: active contributions to the Airflow OSS project, fast runtime releases, and built-in monitoring, observability, and RBAC.
• 05:50 Evolving from dbt core with Bash operators to dbt Cosmos for model-level granularity, lineage, and precise failure recovery, plus a performance boost from watcher mode.
• 08:00 Upgrading from Airflow 2.9 to Airflow 3, using the Astro CLI and linters to catch deprecations quickly.
• 09:32 The drivers behind adopting remote execution: keeping data inside the security perimeter and scaling workloads on their own Kubernetes cluster.
• 11:35 How remote execution replaced a complex network architecture of VPN tunnels and firewall rules, removing latency along the way.
• 12:53 The POV process, success criteria, and a six week timebox to validate remote execution before going to production.
• 14:14 Going fully live: OTPP's last hosted deployment was sunset just before recording.
• 15:06 What Kowsy wants next from Airflow: AI orchestration capabilities and continued maturation of remote execution.

Resources Mentioned

• Ontario Teachers' Pension Plan
• Apache Airflow
• Astronomer
• Cosmos

Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.