Semiconductor chip design is not where most people expect to find Apache Airflow, but Qualcomm's CPU team runs millions of pre-silicon simulations and post-silicon validation jobs through it every day. In this episode, Kenten Danas is joined by Dheeraj Turaga, Principal Engineer at Qualcomm (https://www.qualcomm.com/) and an Apache Airflow committer, to talk through two very different Airflow use cases: orchestrating semiconductor design workflows across on-prem compute farms, and driving robotic handlers in post-silicon validation labs.
Key Takeaways:
00:00 Introduction.
01:36 What Qualcomm does and where the CPU team fits in the Snapdragon product line.
04:07 Reframing Airflow as a general-purpose workflow orchestration platform, not just a data pipeline tool.
05:26 How semiconductor design workflows look as Dags: simulations, power analysis, gate-level conversion, and silicon placement.
09:28 Bringing Airflow in-house and bridging the 20% gap between data pipeline assumptions and chip design needs.
11:36 Using celery with small persistent workers that spawn ephemeral workers to dynamically consume on-prem compute.
13:25 Orchestrating silicon validation labs: limited chips, limited workstations, and multiple teams timesharing hardware.
16:40 Targeting specific lab machines using the Edge Executor with machine-specific queues and pools.
18:45 Managing failures in physical hardware workflows using the new task state store and pluggable retry policies in Airflow 3.3.
21:12 Contributing a new Dag drain state to Airflow 3.4 that lets existing runs finish before pausing a Dag.
Resources Mentioned:
Apache Airflow (https://airflow.apache.org/) Edge Executor (https://airflow.apache.org/docs/apache-airflow-providers-edge3/stable/index.html) Airflow Summit 2025 session: Semiconductor Chip Design Workflow Orchestration with Airflow (https://airflowsummit.org/sessions/2025/semiconductor-chip-design-workflow-orchestration-with-airflow/) Airflow Summit 2026 session: Dags Move Robots - Closed-Loop Orchestration for Silicon Validation Labs with Airflow (https://airflowsummit.org/sessions/2026/dags-move-robots-closed-loop-orchestration-for-silicon-validation-labs-with-airflow/) Qualcomm Snapdragon (https://www.qualcomm.com/snapdragon) Dheeraj Turaga on LinkedIn (https://www.linkedin.com/in/dheerajturaga/)
Thanks for listening to "The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI." If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
