How WesTrac's Data Platform Became as Tough as a Dozer in the Outback

The Customer

WesTrac is Australia’s largest authorised Caterpillar dealer and among the largest globally, supplying machinery and service solutions to the mining, construction, and transport industries across Western Australia, New South Wales, and the Australian Capital Territory. With thousands of dozers, diggers, and trucks working hard across some of the harshest conditions on earth — from the Pilbara to the Hunter Valley — the company relies on timely insights to keep operations humming.

Kumar Saurabh, Data Engineering Lead, spearheads a team ensuring data flows seamlessly to executives, analysts, and frontline crews. As Kumar explains, “Our mission is simple — keep the data pipes flowing clean and reliable so every part of WesTrac can make decisions with confidence.”

The Challenge

WesTrac originally relied on Azure SQL and Databricks, but when the decision was made to consolidate on Snowflake as the primary warehouse, new orchestration gaps emerged in their ETL/ELT pattern of ingestion (via Informatica), transformation (via dbt), data warehouse load, and dashboard refresh (Power BI):

• Azure Data Factory (ADF) lacked the flexibility to coordinate complex dependencies — it could chain jobs, but struggled when pipelines required branching logic, retries, or conditional paths across systems. As Kumar noted, “We kept running into walls with ADF. It could do the basics, but anything beyond straight-line jobs turned into a mess.”

• Databricks Notebooks bundled ingestion and transformation in ways that were brittle and hard to maintain — for example, single notebooks often combined API ingestion with heavy transformations, making failures hard to debug, version control clumsy, and reuse across teams nearly impossible. Kumar explained, “One failure in those notebooks and you’d be stuck sifting through hundreds of lines trying to figure out what went wrong. It just didn’t scale for us.”

• Snowflake Tasks couldn’t easily trigger dbt or external jobs — they required awkward workarounds through GitHub or manual triggers, lacked straightforward support for conditional runs, and made monitoring cumbersome — leaving orchestration fragmented and error-prone. As Kumar put it, “Snowflake Tasks looked good on paper, but once we tried to connect it to dbt or anything external, it all fell apart. We needed something that could actually tie everything together.”

Together these gaps led to inefficiencies, limited scalability, and fragile pipelines. As Kumar put it, “We had tools that worked fine in isolation, but once we tried to scale and connect everything, it fell apart. We were constantly fighting brittle jobs and wasting cycles on fixes — like a ute [utility vehicle] bogged in red dirt.”

Required Capabilities

To successfully consolidate onto Snowflake and support WesTrac’s enterprise analytics, the data engineering team identified several must-haves:

• Cross-platform orchestration: ability to trigger and coordinate Informatica ingestion jobs, dbt, and Power BI pipelines from a single control plane, with support for event-based triggers to keep pipelines responsive.
• Dependency management: support for complex DAGs where downstream tasks only run once upstream jobs succeed.
• Scalability: handle hundreds of daily and hourly jobs without manual babysitting.
• Observability & cost insight: dashboards to monitor runtime, resource usage, and highlight inefficient code.
• Ease of integration: seamless connectivity with GitHub for CI/CD and Snowflake for execution.
• Low operational overhead: managed infrastructure to avoid stretching a small team too thin.
• Alerting & reliability: sufficient monitoring and notifications so the team can respond quickly when things go wrong.
• Security: dedicated cluster and private networking for their Azure connections, including Azure VNet Peering, VHub Peering, and Private Link endpoints.

The Solution

After initially considering options such as dbt Cloud and OSS Airflow, WesTrac determined that “only Airflow could connect the whole chain end-to-end, from ingestion to reporting.” Having decided on Airflow, WesTrac chose Astro, Astronomer’s managed Airflow service. The deciding factors were speed to value, reliability, and not having to worry about infrastructure. Kumar adds, “When you’re pressed for time, it’s better to go with the market-leading solutions rather than hopping around and trying to figure it all out yourself.”

Chris Hidding, Head of Digital in WesTrac’s IT department, described the turning point this way:

“When we kicked off our data journey, we thought we were set. We had our ingestion tools lined up, Snowflake as the powerhouse platform, and everything looked great on paper. But pretty quickly it became obvious we were missing something vital, like hosting a concert with every musician ready to play but no one holding the baton. Enter Astronomer.

Astronomer instantly became the conductor of our data orchestra, pulling all the moving parts together and keeping them in sync. Without it, we would’ve had a lot of noise and very little music. With it, suddenly the whole show made sense. Honestly, it feels less like just another tool and more like the puppeteer making sure the entire act doesn’t collapse on stage.” Chris Hidding Head of Digital

The Results

Like a finely tuned orchestra under a steady baton, Astronomer keeps WesTrac’s data environment in sync, transforming potential noise into harmony across analytics and operations. Key outcomes include:

• 30%+ faster recovery from failures with pinpoint dbt re-runs via Astronomer’s Cosmos (an open-source library that makes it easy to orchestrate dbt projects as native Airflow tasks). Kumar noted, “Instead of re-running entire hourly pipelines, we can zero in on just the model that failed. That alone saves us heaps of time every day.”

• 36% annual savings from optimised job execution identified through Astronomer dashboards. As Kumar explained, “When costs went up, it was a signal our code was inefficient. The dashboards gave us the insight to fix it and cut waste — like the pit supervisor’s radio call warning us before small issues became breakdowns.”

• 25% infrastructure management time savings by offloading Airflow management to Astronomer. Kumar shared, “With only a handful of engineers, running our own Airflow stack would’ve been like throwing another shrimp on the barbie when the grill was already full. Astronomer takes that weight off our shoulders.”

• Seamless GitHub + Snowflake integration delivering faster deployment cycles. Kumar added, “Hooking Astronomer into GitHub and Snowflake was a cakewalk — just click, click, click. It made CI/CD feel effortless.”

What’s Next

Shifting away from arbitrary batch schedules, WesTrac plans to extend orchestration to automatically refreshing Power BI reports and dashboards when upstream data is complete, ensuring reports are always fresh and accurate. They’re also preparing to migrate machine learning models, such as one for oil sample analysis, onto Snowflake and Astronomer to power predictive maintenance for heavy machinery.

“With Astronomer, we’re building a data platform that’s as tough and reliable as our dozers in the Pilbara – no worries, mate.” Kumar Saurabh Data Engineering Lead