Fast Lane to Fleet Safety: Idelic Orchestrates 800+ Pipelines on Astronomer

The Customer

Idelic is a fleet safety and risk management SaaS company that helps trucking fleets improve driver safety and reduce accidents. The company’s VP of Engineering, Sean Graham, leads teams responsible for both customer-facing software and the orchestration of thousands of customer data integrations. These pipelines deliver timely, high-quality data into Idelic’s SaaS platform to power machine learning models, analytics, and driver training workflows.

The Challenge

Idelic’s customers each rely on multiple integrations with internal systems and third-party tools. On average, each customer fleet has 5–10 integrations, creating thousands of orchestrated jobs across the platform. Before adopting Astronomer, orchestration was managed through tools like Cron (a basic Unix-based job scheduler) and then Celery (a distributed task queue system), which lacked observability and required custom workarounds. As pipeline volume grew, this approach became unsustainable. “Errors were getting lost and not logged correctly. We needed more observability and scalability than Celery could provide,” Sean explains.

The Solution

The team knew they wanted to migrate to Apache Airflow® because of its superior reliability, scalability, and observability compared to prior tools. They ran a bakeoff between AWS’s Managed Workflows for Apache Airflow (MWAA) and Astro. Ultimately, they selected Astro for its richer feature set, more robust UI, and deeper operational insights. As Sean put it, “MWAA had the basics, but Astro gave us the observability, UI, and insights we needed to manage hundreds of jobs reliably.”

Today, Idelic operates over 800 DAGs which process about 1 million records daily as data flows from customer systems into Idelic’s data lake, and then on to Idelic’s risk analytics tooling and machine learning models for driver risk scoring and safety analytics. Four different role profiles – engineers, product owners, architects, and implementation specialists – leverage Astro in their daily work. Astronomer integrates seamlessly with Idelic’s stack (Python, Java, Scala, React), and connects with monitoring through Datadog, giving the team real-time alerts and visibility into job performance.

The Results

Astronomer has elevated Idelic’s ability to orchestrate and govern complex, large-scale data pipelines with reliability and confidence:

Scalability at fleet scale: Thousands of integrations now run reliably, supporting SLAs for system availability and data freshness. Sean notes, “It wasn’t manageable going forward without Astronomer—scalability was a key driver for us.”

Enhanced observability: Teams can search across DAGs, review historical runs, and quickly troubleshoot via logs. “Astronomer gives us visibility into which jobs pass or fail, lets us search across hundreds of DAGs, and drill into history when something goes wrong,” says Sean.

Operational efficiency: Standardized DAGs and Astronomer’s orchestration and management features accelerate integration setup, enabling faster onboarding for new fleet customers. Sean notes, “Standardizing DAGs has made it much easier for us to onboard new customers quickly and efficiently.”

Cost optimization: By monitoring scheduler sizing and leveraging hibernation, Idelic optimizes resources without sacrificing reliability. Sean notes, “We resized our schedulers a few times and recently implemented hibernation to keep costs efficient without losing reliability.”

“We definitely love Astro and love using it. We’re always thinking ahead with the challenges we have, and it’s great to see Astronomer not only keeping pace but anticipating our needs with each new version. It’s become a core part of how we deliver value to our customers.” Sean Graham VP of Engineering

What’s Next

Looking forward, Idelic is extending observability with richer integration status dashboards and automated alerts, helping teams distinguish actionable failures and proactively serve customers. The company is also investing more in DAG standardization and streamlined setup processes to enable faster onboarding at scale.