Why AI Workloads Break Legacy Job Schedulers: How Modern Orchestration Fixes It

Legacy job schedulers like Control-M, AutoSys, and Tidal were built for linear, time-based batch jobs, not the dynamic, GPU-aware, event-driven workflows that production AI requires. That architectural gap is a leading reason so few AI initiatives make it into production, and why data engineering teams are being asked to fix an orchestration problem their scheduler was never designed to solve.

If your team has been asked to "just get the AI pipeline running" on the same scheduler that runs your nightly ETL, you've probably already found the wall. This page explains where that wall comes from, why patching it doesn't work, what a modern orchestration layer needs to do instead, and how organizations are actually making the move.

The honest answer isn't a lack of good models. It's what's underneath them.

Gartner's April 2026 survey of infrastructure and operations leaders found that only 28% of AI use cases fully succeed and meet ROI expectations. Gartner has also projected that through the end of 2026, a majority of AI projects built without AI-ready data pipelines will be abandoned outright.

Look at why they fail and a pattern emerges. A March 2026 survey from Digital Applied quizzing 650 enterprise technology leaders found that the top two cited causes of AI pilots failing to reach production were integration complexity and insufficient monitoring infrastructure, ahead of model quality issues. In other words, the model usually works. What breaks is everything responsible for getting the right data to it, on the right trigger, with the right visibility and quality once it's live: the orchestration layer. Because of these issues, the survey found that only 14% of projects reach enterprise scale.

This lines up with what Astronomer sees directly. A third of all Airflow users have AI pipelines in production, with that number climbing to 62% of Astro customers, according to Astronomer's 2026 State of Airflow survey, meaning this is showing up on data engineering teams' desks right now, not in some future roadmap.

Control-M, AutoSys, and Tidal are well-engineered products — just not for the AI workloads enterprises are now asking them to run.

Control-M, from BMC, was originally built to schedule IBM mainframe batch jobs. AutoSys, now owned by Broadcom following its acquisition of CA Technologies, was designed for distributed Unix environments. Tidal Workload Automation follows the same pattern: built to coordinate fixed, enterprise batch schedules, not to trigger a GPU training run or hold an agent workflow at a human checkpoint. All three assume the same basic shape of work: a fixed chain of jobs, triggered on a schedule, running against infrastructure that looks the same today as it did yesterday.

That model works fine for a nightly file transfer or a batch reporting job. It breaks down the moment a workflow needs to:

• Coordinate an autonomous agent workflow that needs a human review checkpoint before it's allowed to proceed
• Trigger and monitor GPU-accelerated training runs and evals on platforms like SageMaker, Vertex AI, or Databricks
• Branch dynamically based on a model's output, where step three doesn't exist until step two returns a result
• Manage long-running jobs with retry logic and fault tolerance built in, not bolted on

None of this is a configuration problem you can solve with a clever workaround. Legacy schedulers have no native concept of GPU-aware scheduling, event-driven triggers, dynamic branching, or agent orchestration. The gap is architectural. It's baked into how the tool was designed to think about a "job" in the first place.

The instinct, understandably, is to patch it. Wrap the AI pipeline in a shell script the scheduler can call. Poll for a file to simulate an event trigger. Chain a series of fixed jobs and hope the branching logic never needs to change. This works for a demo. It doesn't hold up in production, for three reasons.

Time-based triggers can't represent event-driven reality.

An AI pipeline often needs to start the moment new data lands, a model finishes training, or an upstream system emits an event, not on a fixed schedule. Faking this with polling scripts adds latency, fragility, and a growing pile of workaround code that someone has to maintain.

Linear job chains can't represent dynamic workflows.

When the next step depends on what a model returns, you're not describing a fixed Dag of jobs anymore. You're describing logic. Legacy schedulers have no native way to express "run step three only if the model's confidence score is below X," so that logic ends up buried in scripts outside the scheduler's visibility, which is exactly where governance and observability break down.

Retry and fault-tolerance logic was designed for short jobs, not long ones.

GPU training runs can take hours. Agent workflows can pause for human review indefinitely. Schedulers built around short-lived batch jobs handle this poorly, which means engineers end up building custom monitoring just to know whether a "failed" job actually failed or is still legitimately running.

Every one of these workarounds is invisible to the platform team, undocumented outside the person who wrote it, and one departure away from becoming an outage. That's technical debt accumulating specifically around your highest-visibility initiative, and it's a large part of why the "integration complexity" and "monitoring infrastructure" failure modes above show up so consistently in the research.

This objection is worth taking seriously, because it's increasingly common. The legacy vendors have seen the same shift you have, and they're responding. BMC now markets Control-M as a "trusted AI orchestration" platform, with agentic AI across the workflow lifecycle and native integrations for frameworks like CrewAI, LangGraph, and Snowflake Cortex. Broadcom is repositioning AutoSys and Automic as an "Intelligent Control Plane" for AI, with AI job types, MCP orchestration, and bring-your-own-model support on its latest roadmap. Tidal markets orchestration for "agentic AI systems" too. If your incumbent tells you they have this covered, they aren't making it up—they are shipping AI features.

The question isn't whether a scheduler can call an AI model or trigger an agent. It's whether the platform underneath was rebuilt for this class of work or had it bolted on. Three things separate the two.

• Integrating with agents isn't the same as being architected for them. These vendors are adding AI integrations and assistants on top of the same agent-based, GUI-driven batch engines that have run their schedulers for decades. Airflow 3 was re-architected around event-driven scheduling, remote execution, and dynamic task generation, so the triggering, branching, and long-running logic lives natively in the orchestrator, expressed as code—not wrapped around a batch core.
• You're migrating either way—so migrate to the standard, not to more lock-in. Most of these AI capabilities only land on the vendor's newer SaaS edition or latest major release, which means adopting them is itself a migration off what you run today. If you're going to absorb a migration, the real choice isn't "stay put versus move." It's whether to spend that effort renewing a single-vendor, proprietary platform or moving to the open industry standard that ends the lock-in.
• The companies building frontier AI standardized on Airflow, not on a scheduler that integrates with AI tools. OpenAI runs roughly 7,000 Airflow pipelines; GitHub Copilot, Anthropic, and other leading AI teams run production workloads on it. When the organizations defining the category build on the open standard, that's the strongest signal a buyer has.

"Trusted and governed" is the differentiator every one of these vendors now leads with—and it's table stakes, not a moat. Astro matches it: fine-grained RBAC down to the individual pipeline, end-to-end lineage and observability, and remote execution that keeps data and IP inside your environment for regulated workloads. Governance is not a reason to stay on a proprietary scheduler.

Modern orchestration platforms, with Apache Airflow as the industry standard, are built around the same shifts that broke legacy schedulers in the first place:

• Event-driven scheduling, so a pipeline can start because something happened, not just because a clock struck a particular time
• Dynamic task generation, so the workflow can branch, expand, or reroute based on upstream outputs, including model outputs
• Remote and distributed execution, so GPU-accelerated training jobs, multi-cloud steps, and long-running agent tasks can all be coordinated from a single control plane instead of stitched together with scripts

Airflow 3 in particular was built with this gap in mind. Its remote execution model and event-driven scheduling exist specifically to support the trigger-based, multi-environment workflows that AI and agent workloads require. That's not a marginal improvement over legacy schedulers. It's a different architecture for a different category of work.

With 19+ million monthly downloads, Airflow is the orchestration technology relied upon by leading AI foundation labs like OpenAI and Anthropic, AI-native companies like Notion and together.ai, as well as organizations like Uber, Stripe, Apple, and Robinhood. Airflow's role has expanded well past traditional ETL. It's increasingly the layer that connects data pipelines to AI behavior in a single, governed, observable runtime, which is exactly the layer the 2026 research points to as the difference between a pilot and a production system.

This is usually where the conversation stalls. Even teams that fully accept the architectural argument hesitate, because "migrate off the scheduler that runs our production operations" sounds like a multi-year, high-risk undertaking. In practice, it doesn't have to be, and a growing number of enterprises have already done it.

Astronomer's Kickstart program runs migrations as a structured, phased engagement rather than a rip-and-replace: discovery and dependency mapping, architecture design, automated translation, custom development for complex workflows, validation, and staged cutover, with the legacy scheduler kept running in parallel until the team is confident enough to switch over. A typical Phase 1 migration completes in 8 to 16 weeks.

Much of the mechanical work is handled by Otto, Astronomer's data engineering agent, which converts Control-M, AutoSys, Tidal, Automic, and other legacy scheduler definitions directly into production-ready Airflow Dags and maps every job dependency as it goes, which removes the single biggest source of migration surprises. Otto and the Kickstart methodology together typically automate 70%+ of the mechanical translation, so engineering time goes toward the genuinely complex workflows instead of repetitive conversion work.

What this looks like in practice:

• Société Générale consolidated four legacy orchestrators into a bank-grade, Airflow-as-a-service platform on Astro Private Cloud, standing up 500+ production deployments for 1,000+ engineers writing Dags and taking environment provisioning from days to on-demand inside the bank.
• Northern Trust, which manages $160 billion in assets, migrated 200+ jobs off Control-M with zero business disruption, improving workflow execution time by 20% and eliminating its Control-M licensing costs entirely.
• Janus Henderson stood up quant and AI/ML pipelines ready at market open, including production-ready autonomous remediation, in eight weeks.
• Autodesk migrated 536 workflows in 12 weeks (versus an 18-month-plus estimate for a DIY approach), reaching Airflow productivity within the first month and improving data quality by 90% post-migration.

None of these were AI-only migrations. But in each case, the same orchestration layer that resolved the legacy scheduler's operational and licensing problems is what now supports the organization's AI and ML workloads going forward, which is the point: you don't need a separate initiative to become AI-ready. Modernizing the orchestration layer does both jobs at once.

For a data engineering team, legacy schedulers show up as a technical problem: a pipeline that can't express the logic you need. But it ladders up to a business risk that's bigger than any one pipeline.

Every AI initiative that gets stuck at the orchestration layer is an initiative that doesn't ship, regardless of how good the underlying model is.

Enterprises that keep patching legacy schedulers to squeeze AI workloads through them aren't avoiding the cost of modernization. They're deferring it, while quietly adding engineering hours, fragile custom code, and delayed time-to-value to every AI project that touches that scheduler.

Meanwhile, the organizations landing in Gartner's successful 28% are disproportionately the ones that fixed the orchestration layer first, and they're compounding that advantage with every pipeline they ship after the first one.

If your team is already fielding requests to "make AI work" on top of a scheduler that was never built for it, that's not a sign you need a better script. It's a sign the orchestration layer itself needs to change, and there's now a well-worn path for making that change without betting the business on it.

Frequently asked questions

For a deeper look at what's driving enterprises to move off Control-M, AutoSys, and Tidal, including the cost, governance, and talent pressures alongside the AI gap, read Moving Beyond Legacy Schedulers. And if you're ready to look at what a migration path could look like for your environment, see how Astronomer approaches legacy scheduler migration.