How We Built Cross-Region Disaster Recovery for Astro on Azure

Azure is the last stop on a rollout we started back in April: cross-region disaster recovery (DR) for Astro is now generally available on all three major clouds. We shipped DR on AWS first, brought it to Google Cloud in July, and now it's Azure's turn. This post is about that last build: what's the same as the other two clouds, what's different about Azure's primitives, and the one problem that turned out to be easier to solve here than anywhere else.

Why Cross-Region DR Matters for Airflow

Our customers run business-critical data pipelines. Whether that's financial regulatory reporting, patient data processing, or some other essential workflow, the moment one of these pipelines stops, the downstream impact is immediate and real.

Most teams running Airflow at enterprise scale eventually decide that cross-region resilience isn't optional. Historically, the only way to get there was to build it yourself: stand up parallel infrastructure, wire up your own replication, write and rehearse a failover runbook, and then keep all of it working as your platform evolves. That's a three-to-six month project for most teams, and it doesn't end at launch. Someone has to own it indefinitely.

We wanted to replace that project with a toggle. The setup experience is identical no matter which cloud you're on: turn DR on, pick a secondary region, and Astro does the rest. So is the bar we hold ourselves to, a recovery time objective (RTO) under one hour and a recovery point objective (RPO) under 15 minutes.

The State That Has to Survive a Region Outage

Losing a single Availability Zone is already something the Astro data plane is built to handle. Every dedicated Azure data plane runs across three AZs in a region, so a zone going down degrades things at worst, it doesn't take your workloads offline. Cross-region DR exists for the bigger failure mode: the entire region goes away.

Three things have to make it to the secondary region for a failover to actually work, and each one gets handled differently on Azure:

• Airflow metadata. The database backing Dag runs, task instance metadata, connections, variables, and XComs.
• Task logs. Stored in Azure Blob Storage, needed for debugging and audit trails.
• Container images. Customer-deployed images that must be available in the secondary region.

Keeping all three in sync is a mix of an Azure Database for PostgreSQL flexible server geo-replica for the metadata, Blob Storage object replication for logs and images, and a standby network and compute footprint in the secondary region that's ready but idle. The rest of this post walks through each piece.

One Cluster, Two Regions: The Control Plane Model

As with AWS and GCP, we continue to model DR as an extension of the existing cluster record, not as a second, first-class cluster. A drRegion field marks a cluster as DR-enabled, and an isFailedOver flag tracks which region is currently active.

We considered the alternative, giving the secondary its own ID and its own lifecycle, and it wasn't close. That path means duplicating schema and keeping it in sync, adding special-case guards across dozens of API endpoints, and untangling how Deployments, API tokens, and Workspace mappings, all of which assume a single cluster, would work across two. Keeping DR as a property of one cluster avoids all of that. From the control plane's point of view, there's still just one cluster. The actual complexity of running two regions' worth of infrastructure stays contained in our provisioning and manifest systems, which is where it belongs.

One thing that falls out of this naturally, a cluster's identity doesn't change when it fails over. Names, namespaces, IDs, and every hostname a customer would ever reference, the Airflow UI, the Airflow API, the Remote Execution API, keep working exactly as they did before. They just quietly point at a different region now.

Enabling DR: One API Call, No Migration

It doesn't matter whether you're standing up a brand-new dedicated cluster or turning DR on for one that's already running Deployments, the process on Azure looks the same either way, and it's lighter than you'd expect from a DR setup.

Adding DR to a database that's already live is usually the hard part: migrate to a different engine or edition, move the storage somewhere new, and block off a maintenance window to do it without losing anything. Azure let us skip almost all of that, for two reasons specific to how it works under the hood.

The first is that geo-replicas don't require the primary to stop. Azure Database for PostgreSQL flexible server builds a cross-region replica straight from the running primary's base backup, streamed over the network, and the primary never goes down for it. No engine swap, no edition change to plan for.

The second is that object replication can catch up on history it missed. Blob Storage object replication policies normally only pick up new writes going forward, but you can scope a policy with a creation-time filter that pulls in everything that already exists. That let us fold a cluster's existing task logs and registry images into the initial sync instead of running a separate backfill job after the fact.

Because neither of those needs a migration, setting the DR region and turning DR on can happen in the same API call, and existing clusters can enable DR straight from the Cloud UI, no support ticket, no maintenance window on the calendar.

Database: Geo-Replication

The metadata database is the piece with the least room for error, and it's the one the whole failover sequence really hinges on.

DR-enabled Azure clusters run their metadata database on Azure Database for PostgreSQL flexible server with geo-replication turned on. Astro provisions a geo-replica in the paired secondary region, and it stays in sync through asynchronous physical replication, which we monitor directly through Azure's physical_replication_delay_in_seconds metric.

Rather than routing through a shared endpoint that has to be repointed at failover time, each region's Airflow deployment connects directly to that region's own PostgreSQL hostname. This also ensures that after a failover workloads running in a non-active region are communicating directly with a now read-only database instance that prevents split-brain in the event we are not able to successfully scale down workloads in the non-active region. That's one less moving part in the failover path, and one less thing that has to work correctly under pressure.

Storage: Object Replication for Logs and Images

Task logs and the customer image registry both live in Azure Blob Storage, and both rely on object replication policies to stay in sync with the secondary region.

We actually looked at Azure's built-in multi-region storage accounts first, which are similar in spirit to what other clouds call dual-region buckets. The catch is that they only work with a small, fixed list of predefined region pairings, you don't get to choose your own. That wasn't going to work for customers who want to pick their own DR region, so instead we pair a normal, single-region storage account on each side with explicit object replication policies connecting them.

Each policy gets a policy ID once it's created, and that same ID has to be set on both the source and destination storage accounts for anything to actually replicate. We keep the logs and registry policy IDs on the cluster's metadata and feed them into our observability stack, more on that below. One limit worth knowing: a source account can only replicate to two destination accounts at most, each under its own policy.

Task logs get the same Task Logs Replication SLA option we introduced on GCP, targeting a 15-minute RPO, and it's on by default across every cloud we support.On Azure, the SLA is only available when your primary and secondary regions share the same continent, and we validate that automatically. If the region pair you choose doesn't support it, cluster provisioning fails validation rather than silently falling back to best-effort replication, so you always know upfront whether your chosen pair meets the guarantee.

Either way, the day-to-day mechanics stay simple: each region's task logs point at that region's own storage account. Nothing has to be looked up dynamically, and nothing changes when a failover happens.

Networking: One CIDR Range Covers the Secondary Cluster

Setting up networking for the secondary environment is a lot less involved than a typical DR setup would suggest. You give us one VNet CIDR range for the secondary cluster, the same input you'd provide for any new dedicated cluster, and Astro carves out the node, pod, and service subnets the secondary AKS cluster needs on its own, along with the range used to peer it back to the primary VNet.

If you're planning this out, budget an additional CIDR range of the same size for your secondary cluster, exactly like you would for a new one.

Compute: Warm Standby on AKS

The secondary AKS cluster sits in warm standby. It's healthy, observable, and ready to take on traffic, but it isn't running your Airflow workloads day to day.

That's deliberate. A Deployment can only be active in one region at a time, otherwise you'd risk two copies of the same workload stepping on each other with any external systems they talk to. Deployments only ever exist on the primary cluster. When a failover happens, Astro creates them fresh on the newly promoted cluster rather than scaling up something that was already sitting there. In our benchmark testing, every task was running successfully again within about 40 minutes, comfortably inside our 1-hour RTO target. That's a separate number from our 15-minute RPO target, which measures replication lag rather than how long a failover takes, so the two aren't in tension with each other.

Everything else, node autoscaling, the manifest reconciler, our observability collectors, the foundational cluster services, keeps running on the secondary the whole time, so it's already reconciled and monitored well before anyone needs it.

Workload identity needs one manual step

Every Azure customer configures their own managed identity through Microsoft Entra Workload ID to authorize Deployments against Azure services, and that identity needs to be set up for the secondary cluster ahead of time, not just the primary. Customer-managed identity through Microsoft Entra Workload ID is the one place you'll need to do something ahead of time.

Here's why: AKS workload identity ties a managed identity to a Kubernetes service account through the cluster's OIDC issuer, and the primary and secondary clusters each have their own issuer. A federated credential you set up for the primary doesn't carry over to the secondary automatically. You configure this from a Deployment's Advanced settings in the Cloud UI, where you also have the option to just reuse the primary's identity instead of creating a new one. The setup command Astro hands you is already targeted at the secondary's issuer, and you only need to run it once, well before you'd ever need to fail over.

The same logic applies to private networking. If a deployment depends on a private endpoint connection, that connection needs to be requested and accepted for the secondary environment too, and that has to happen ahead of time. Both of these live inside your Azure tenant, so they're the two things Astro genuinely can't do on your behalf.

Ready to Fail Over: What You Set Up, What Astro Handles

Before you ever trigger a failover, a short list of things needs to already be in place, and all of them live inside your Azure tenant, which means Astro can't set them up for you:

• Customer-managed workload identity for the secondary cluster's OIDC issuer
• Private endpoint connections into the secondary VNet
• imagePullSecrets for Kubernetes Pod Operators, if your Deployments pull from a private registry
• Any customer-managed network routing on the secondary cluster

Once those are handled, triggering a failover is the easy part. Astro takes care of everything else on its own:

• Deployments and data pipelines
• Dag run history, task instance metadata, and XComs
• Deployment configuration
• Environment variables, connections, Airflow variables, and metrics exports
• Task logs, with a 15-minute RPO when the Task Logs Replication SLA is enabled

Observability: Replication Health, Not a Guessing Game

We run the full observability stack on the Azure secondary cluster from the day DR is enabled, not just after a failover. Instead of inferring whether replication is healthy, we check it directly: the logs and registry replication policy IDs we capture at provisioning time let our metrics pipeline query Azure Storage for the actual replication status behind that specific cluster's DR pair. On the database side, we pull physical_replication_delay_in_seconds straight from the geo-replica alongside the rest of our Postgres metrics.

The result is the same thing we built for AWS and GCP: if a replication link degrades, it shows up as a monitored signal right away, instead of being something we, or you, discover the next time a failover is actually needed.

The Failover Sequence

Triggering a failover from the UI flips the cluster's isFailedOver field and kicks off a cluster update. From there, it's a fixed, repeatable sequence:

1. Promote the geo-replica. Astro calls the PostgreSQL flexible server promotion operation on the secondary geo-replica.
2. Inflate the secondary cluster. Manifests get regenerated for every plugin and Deployment in their failed-over form, creating those workloads on the newly promoted cluster for the first time rather than scaling up something already running there.
3. Point DNS at the new primary. We update the cluster's ingress record to the secondary's load balancer. Because every Deployment-level hostname is a CNAME back to that top-level record, this one change is all that's needed for traffic to land in the right place.
4. Resume replication in reverse. With the secondary now active, Astro sets up replication back toward the original primary so that region is ready to take back over once it's healthy again.

Failing back to the primary follows the same sequence in reverse.

Programmatic Control: API and Terraform

You get the same API and Terraform coverage on Azure that we built for the other two clouds, plus one thing that's specific to Azure. You can create DR-enabled clusters, trigger failover and failback, and flip the Task Logs Replication SLA on or off, all through the Astro API or the Terraform provider.

Because there's no migration involved in enabling DR on an existing Azure cluster, it's also the only one of the three clouds where you can turn DR on or off for a cluster that's already running, entirely through Terraform. On AWS and GCP, that still means a support request. On the SLA setting specifically, Terraform defaults it to enabled, and if the region pair you've picked doesn't actually support it, terraform apply will fail with a validation error instead of quietly turning it off, so you always know where you stand before you apply.

Lessons Learned

Getting the control plane model right the first time pays off more with each cloud you add. We built the one-cluster, two-region abstraction for AWS and haven't touched it since, not for GCP, and not now for Azure. Three clouds, three completely different replication primitives underneath, and the way the control plane represents a DR pair hasn't changed once. That's the payoff of spending the time on the abstraction before the second cloud, let alone the third

The best migration is the one you never have to write. Azure's geo-replicas attach to a database that's already running, and Blob Storage object replication can backfill what already exists instead of only catching new writes. Understanding a cloud's replication primitives well enough to route around a migration entirely is worth the extra research time before any provisioning code gets written.

Manual steps live in identity and private networking, not data. No matter which cloud we're on, the things that don't fail over on their own fall into those two buckets, both of which require someone to act inside their own cloud tenant. We can own data replication end to end. Identity and networking, by their nature, need a human on the other side to finish the job.

Get Started

Cross-region DR on Azure requires the Enterprise Business Critical tier and is now generally available alongside AWS and GCP. To enable it on a new or existing dedicated Azure cluster, reach out to your account team or contact Astronomer support.

For the full setup walkthrough, including how to prepare for DR and configure it on new or existing clusters, see the Disaster recovery documentation. And if you want the fuller picture of how Astro handles resilience before a failover ever enters the conversation, the Resilience overview is a good next read.