Modern Airflow isn’t just orchestration. It's a contribution.

In this episode, we explore how open source investment drives real performance gains and deeper observability. We’re joined by Christos Bisias, Open Source Software Engineer, Apache Airflow at G-Research, to discuss how his team uses Airflow for large-scale data transformations, contributes upstream and improves scheduler throughput and OpenTelemetry support. From trace-level observability to CI-enforced metrics governance and a major scheduler optimization, this conversation spans strategy, engineering and community impact.

Key Takeaways:

• 00:00 Introduction.
• 01:20 How G-Research applies machine learning and big data to predict financial market movements.
• 02:15 Contributing to open source is a business decision.
• 03:10 Maintaining a fork is costly.
• 04:30 OpenTelemetry collects metrics, logs and traces to provide deep system visibility.
• 06:10 Custom spans help identify bottlenecks inside tasks and enable performance optimization.
• 08:05 OpenTelemetry integration works properly in Airflow 3.0 and above.
• 10:00 A YAML-based metrics registry with CI enforcement ensures consistency between docs and exported metrics.
• 12:10 Scheduler throughput improved significantly by applying concurrency limits earlier in the database query.
• 15:20 Future Task SDK changes may enable language-agnostic DAG authoring beyond Python.

Resources Mentioned:

• Apache Airflow
• OpenTelemetry
• Prometheus
• Grafana
• Jaeger

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow^® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.