Data Ready at Market Open: How Janus Henderson Investors Built Self-Healing Pipelines with Astro, Otto

The Customer

Janus Henderson Investors (JHI) is a global active asset manager with approximately $480 billion in assets under management, serving clients from offices across 26 cities worldwide. Investment teams in fixed income, equities, multi-asset, and alternatives all depend on accurate, timely data to drive portfolio decisions every trading day.

Dipesh Kataria leads the Quantitative Development group within JHI's Fixed Income division, where his team builds and maintains the data infrastructure that powers portfolio analysis: ingesting index feeds from Bloomberg, Bank of America Merrill Lynch, and iBoxx; transforming them through a medallion architecture (bronze raw feeds through silver-layer enrichment and on to gold business-ready outputs); and producing the quantitative signals and fair value models that investment teams rely on each morning.

Across JHI, data engineers in multi-asset strategies run additional pipelines covering Moody's capital structure data, GoldenSource entity data, global rates models, macro credit, and asset allocation signals.

Data pipelines, in other words, are not just operational infrastructure at JHI. They are the foundation of investment intelligence and, as the firm builds out AI model deployment across its data platforms, an increasingly critical part of JHI's AI strategy.

The Challenge

Managing 130-plus production pipelines with a small team creates a compounding set of risks. Most pipelines fire between 3 and 4 AM. The team arrives around 7 AM — less than one hour before markets open. For an active asset manager, that window is unforgiving. If the overnight pipelines haven't run cleanly, portfolio managers don't have the quantitative signals and fair value models they depend on for trading decisions. A data infrastructure failure isn't just a technical problem. At market open, it's a business problem.

When failures did surface, resolution time was a gamble:

• Tribal knowledge gaps: Pipeline expertise was unevenly distributed. An engineer who wrote a Dag could diagnose it in minutes; anyone else might spend an hour working through logs. Vacations and on-call rotations made the gap worse.
• Legacy orchestration overhead: JHI had inherited both Autosys and Azure Data Factory as orchestration layers, neither built for the complexity of modern Airflow-native workflows. Managing multiple systems created fragmentation and operational drag across teams.
• No centralized Airflow ownership: Central IT managed data warehouses but had no Kubernetes experience and could not absorb Airflow infrastructure. The burden fell on Dipesh's quant team, engineers hired to build quantitative models, not manage container orchestration.

The Solution

In mid-2025, JHI onboarded Astro, deploying on Azure in line with the firm's cloud infrastructure strategy. For a quant team hired to build investment models rather than manage container infrastructure, the decision was straightforward. "We needed someone who genuinely owned the Airflow platform, not just hosted it," said Dipesh. "With Astronomer as one of the main backers of Airflow, and a fully managed environment that meant no Kubernetes for our engineers, it was the clear choice." The migration from their self-hosted open source Airflow deployment was completed in full by January 2026.

Astro Observe gave JHI their first real view into Dag-level performance. For the first time, teams could understand the downstream blast radius when something went wrong.

Otto is Astronomer's AI data engineering agent, purpose-built for Airflow. Otto Investigations use proprietary Airflow, Astro, and observability context to diagnose Dag failures and return a structured output with root cause, severity, and suggested fix programmatically. At Janus, Otto Investigations analyse every production failure in the Fixed Income division, performing an initial diagnosis of root cause type that routes the failure to the relevant downstream workflow.

JHI's overnight triage challenge made them a natural early access partner when Astronomer began developing the Otto Investigations capability. Within three weeks JHI Quantitative Developer Kieran Rajasansir had a working end-to-end autonomous remediation prototype in a demo environment. Otto provides the Airflow-native diagnostic layer, while JHI's Lighthouse system turns those diagnostics into action by combining them with internal context, agent routing, GitHub automation, Astro API remediation, and Teams escalation.

Today, Lighthouse is the first line of defence for every prod failure in the Fixed Income division, kicking off JHI's agent orchestration framework to route failures, restart jobs, and open PRs for code issues. Now, rather than waking up and firefighting issues, the quant dev team can review changes proposed by Lighthouse. The flow in brief:

1. A Dag fails; Astro fires a trigger notification, calling Otto's Investigation API.
2. Otto runs its investigation, drawing on proprietary Airflow context including Astro Observe's lineage data to surface findings and assess blast radius. Otto returns an initial diagnosis that is programmatically consumed downstream
3. Lighthouse receives Otto's output, enriches it with additional context, and routes it to the relevant subagent
4. Automated sub-agents handle the response: code bugs are fixed and a pull request opened in GitHub; failed pipeline tasks are retriggered via the Astro API.
5. Unknown or complex failures escalate to the engineering team via Microsoft Teams.

The Results

Moving to Astro delivered immediate operational gains even before autonomous remediation was in the picture. The OSS Airflow migration removed infrastructure management from the quant team entirely.

The combination of Lighthouse and Otto changed the character of the team's work. Where failures once waited until 7 AM for a human to investigate, they are now diagnosed the moment they occur. The key outcomes:

• Zero-wait triage: Otto classifies every failure the instant it happens, regardless of the hour or who is on holiday.
• Consistent resolution time: Resolution went from a variable range of five minutes to an hour, entirely dependent on who was available and how well they knew the pipeline, to a predictable automated response that runs around the clock.
• Tribal knowledge reduced as a bottleneck: The same diagnostic logic applies to every failure, whether the Dag was written last week or three years ago.
• Human in the loop oversight & controls: For code changes, Lighthouse opens pull requests for engineer review rather than pushing directly to production, preserving JHI's existing software delivery controls
• Scale of operations protected: 230,000-plus task successes per month across 27 production deployments spanning fixed income quant and multi-asset strategies, all now covered by automated triage and remediation.

What's Next

The team is already partnering with Astronomer on the next iteration of the system: self-healing pipelines that improve with every failure, through team- and pipeline-level memory stores in Otto and institutional memory from Lighthouse.

From the fixed income quant team, autonomous remediation is expected to expand across JHI's broader enterprise data platform, bringing the same capability to every data engineering team in the firm.

The AI ambitions run deeper than pipeline operations. JHI is actively deploying machine learning models to production. As that footprint grows, the same Astro-powered orchestration and observability infrastructure that supports quant pipelines today will underpin the firm's AI development workflows tomorrow.