Fast Lane to Fleet Safety: How Idelic Orchestrates 800+ Data Pipelines on Astronomer

The Customer

Idelic is a Pittsburgh-based fleet safety and risk management SaaS company that helps trucking fleets improve driver safety and reduce accidents. The company’s VP of Engineering, Sean Graham, leads teams responsible for both customer-facing software and the orchestration of thousands of customer data integrations. These pipelines deliver timely, high-quality data into Idelic’s SaaS platform to power machine learning models, analytics, and driver training workflows.

The Challenge

Idelic’s customers each rely on multiple integrations with internal systems and third-party tools. On average, each customer fleet has 5-10 integrations, creating thousands of orchestrated jobs across the platform. Before adopting Astronomer, orchestration was managed through tools like Cron (a basic Unix-based job scheduler) and then Celery (a distributed task queue system). As pipeline volumes grew, Celery started to buckle under the load. “We needed more reliability and scalability than Celery could provide, and we needed to spend less time scaling our infrastructure and troubleshooting issues,” Sean explains.

The Solution

The team knew they wanted to migrate to Apache Airflow® because of its superior reliability and scalability compared to prior tools. They ran a bakeoff between AWS’s Managed Workflows for Apache Airflow (MWAA) and Astro. Ultimately, they selected Astro, as Sean explains:

“We found Astronomer’s release cadence and upgrade process far more reliable than MWAA’s, which lagged on Airflow v2 support. Astronomer also simplified deployment and user management through its web UI, API, and CLI. MWAA’s highest-tier scheduler couldn’t handle our DAG volume, while Astronomer’s scaled smoothly. Finally, Astronomer’s Docker-based deployment model was a major improvement over MWAA’s manual ZIP-based process.” Sean Graham VP of Engineering

Today, Idelic operates over 800 DAGs which process about 1 million records daily as data flows from customer systems into Idelic’s data lake, and then on to Idelic’s risk analytics tooling and machine learning models for driver risk scoring and safety analytics. Four different role profiles – engineers, product owners, architects, and implementation specialists – leverage Astro in their daily work. Astronomer integrates seamlessly with Idelic’s stack (Python, Java, Scala, React), and connects with monitoring through Datadog, which provides trend analysis, alerting, and anomaly detection.

The Results

Astronomer has elevated Idelic’s ability to orchestrate and govern complex, large-scale data pipelines with reliability and confidence:

100x faster infrastructure scaling: Previously, scaling workers would be a full day project; today it takes five minutes or less. “With Astro, we can rely on the infrastructure to just work – no manual tuning, no distractions, no delays,” says Sean.

30x faster troubleshooting with enhanced observability: With Astro’s UI, diagnosing issues that once took over an hour now takes as little as two minutes. Teams can search across DAGs, review historical runs, and access logs in one place, making troubleshooting faster and easier. Product, reliability, and support teams are now self-sufficient in resolving issues without relying on engineering.

Operational efficiency: Standardized DAGs and Astronomer’s orchestration and management features accelerate integration setup, enabling faster onboarding for new fleet customers. Sean notes, “Standardizing DAGs has made it much easier for us to onboard new customers quickly and efficiently.”

Improved developer experience: Astronomer’s UI and CLI tools improved developer experience (DevX) and streamlined platform management compared to both Celery and MWAA.

Cost optimization: By monitoring scheduler sizing and leveraging hibernation, Idelic optimizes resources without sacrificing reliability. Sean notes, “We resized our schedulers a few times and recently implemented hibernation to keep costs efficient without losing reliability.”

What’s Next

Looking forward, Idelic is investing more in DAG standardization and streamlined setup processes to enable faster onboarding at scale. The team also appreciates Astronomer’s consistent cadence of updates and enhancements, which continue to improve reliability and usability.

“We definitely love Astro and love using it. We’re always thinking ahead with the challenges we have, and it’s great to see Astronomer not only keeping pace but anticipating our needs with each new version. It’s become a core part of how we deliver value to our customers.” Sean Graham VP of Engineering, Idelic