Multi-Region Failover: What Actually Breaks in Production

Every vendor's marketing page makes multi-region failover sound like flipping a switch: primary goes down, replica takes over, traffic reroutes, nobody notices. I've been through four real failovers now — two planned, two not — and none of them matched that description. The gap between "we have a replica in another region" and "we can actually fail over without losing data or serving stale reads" is where most of the real engineering work lives.

This post covers the failure modes that don't show up until you're already in an incident, and the design decisions that determine whether failover is a non-event or a multi-hour outage.

Replication Lag Is Not a Rounding Error

Cross-region replication lag gets treated as a metric to watch, not a design constraint. That's backwards. Lag isn't just "how far behind is the replica" — it's "how much data will we lose if we promote right now."

-- Check replication lag on a PostgreSQL streaming replica
SELECT
  client_addr,
  state,
  pg_wal_lsn_diff(sent_lsn, replay_lsn) AS bytes_behind,
  extract(epoch FROM (now() - reply_time)) AS seconds_behind
FROM pg_stat_replication;

A replica 200ms behind under normal load can be 30+ seconds behind during a traffic spike or a large batch write — exactly the conditions under which a primary is likely to fail. If your failover runbook says "promote the replica," it needs to also say "and here's how many seconds of writes we're willing to lose to do it."

// ❌ Blind promotion — no visibility into data loss
async function failover(replicaId: string): Promise<void> {
  await promoteReplica(replicaId);
  await updateDnsRecord(replicaId);
}
 
// ✅ Bounded promotion — refuses to fail over past a data-loss threshold
async function failover(
  replicaId: string,
  maxAcceptableLagSeconds: number,
): Promise<FailoverResult> {
  const lag = await getReplicationLag(replicaId);
 
  if (lag.seconds > maxAcceptableLagSeconds) {
    return {
      status: "blocked",
      reason: `Lag ${lag.seconds}s exceeds threshold ${maxAcceptableLagSeconds}s`,
      estimatedDataLoss: lag.approximateWriteCount,
    };
  }
 
  await promoteReplica(replicaId);
  await updateDnsRecord(replicaId);
  return { status: "completed", lagAtPromotion: lag.seconds };
}

The threshold itself is a business decision, not an engineering one. Get product and leadership to sign off on an acceptable data-loss window before the incident, not during it.

Split-Brain: Two Primaries Are Worse Than Zero

The scenario nobody wants to talk about: the "failed" primary isn't actually dead, it's just unreachable from your health checker. Meanwhile you've promoted the replica, and now both databases are accepting writes independently. This is the single most destructive failure mode in multi-region setups, and it's caused by network partitions, not hardware failure.

// ❌ Health check with no fencing — promotes on a single failed probe
async function checkPrimaryHealth(): Promise<boolean> {
  try {
    await pingDatabase(PRIMARY_HOST, { timeout: 2000 });
    return true;
  } catch {
    return false; // triggers immediate promotion elsewhere
  }
}
 
// ✅ Fencing before promotion — actively prevents the old primary from writing
async function promoteWithFencing(
  candidateReplica: string,
  oldPrimary: string,
): Promise<void> {
  // 1. Confirmed multiple independent probes agree primary is down
  const consensus = await getQuorumHealthCheck(oldPrimary, { probes: 3 });
  if (consensus.healthy) throw new Error("Primary reachable — abort promotion");
 
  // 2. Fence the old primary at the network or storage layer BEFORE promoting
  await revokeWriteAccess(oldPrimary); // e.g. STONITH, IAM policy, VPC rule
 
  // 3. Only now is it safe to promote
  await promoteReplica(candidateReplica);
}

Fencing — actively cutting off the old primary's ability to write, rather than just assuming it's dead — is the part teams skip because it's operationally annoying. It's also the part that prevents the worst outcome: two "sources of truth" that have diverged and can't be reconciled automatically.

DNS and Connection Pools Don't Failover as Fast as You Think

Even a clean, fenced promotion doesn't help if your application layer keeps talking to the old primary. Two culprits show up every time:

1. DNS TTL caching — clients and intermediate resolvers hold onto the old IP well past your configured TTL, especially on managed platforms with aggressive resolver caching.
2. Connection pools — a pool that already opened 50 connections to the old primary will happily keep using them until they error out, not until DNS changes.

// ❌ Pool holds stale connections indefinitely after failover
const pool = new Pool({
  host: "db-primary.internal",
  max: 50,
});
 
// ✅ Pool with bounded connection lifetime forces periodic re-resolution
const pool = new Pool({
  host: "db-primary.internal",
  max: 50,
  maxLifetimeSeconds: 300, // connections recycle every 5 minutes
  connectionTimeoutMillis: 3000,
});
 
pool.on("error", async (err) => {
  if (isConnectionRefused(err)) {
    await pool.end(); // force full reconnect, re-resolve DNS
  }
});

Read Replicas Serving Stale Data During the Transition

During the window between "primary is down" and "promotion is complete," read traffic often keeps flowing to replicas that are now the most current data source — but application code frequently assumes replicas are always slightly behind and routes accordingly. This produces a confusing failure: writes fail, but reads succeed and return data that's "wrong" only because the app doesn't know the topology just changed.

interface DatabaseTopology {
  primaryRegion: string;
  writableEndpoint: string;
  readEndpoints: string[];
  lastTopologyChange: Date;
}
 
// ✅ Centralize topology awareness instead of hardcoding endpoints
class TopologyAwareRouter {
  private topology: DatabaseTopology;
 
  async route(query: Query): Promise<string> {
    if (query.requiresWrite) {
      return this.topology.writableEndpoint;
    }
 
    // During transition windows, prefer the writable endpoint for reads too
    const secondsSinceChange =
      (Date.now() - this.topology.lastTopologyChange.getTime()) / 1000;
 
    if (secondsSinceChange < this.settlingPeriodSeconds) {
      return this.topology.writableEndpoint;
    }
 
    return this.pickReadEndpoint(this.topology.readEndpoints);
  }
 
  private settlingPeriodSeconds = 120;
}

The core idea: topology changes should be a first-class event your application reacts to, not something inferred from connection errors after the fact.

Testing Failover When Nothing Is Actually on Fire

The uncomfortable truth is that most failover procedures are only ever tested during real incidents, which is the worst possible time to discover a broken assumption. Game days — scheduled, deliberate failover drills — are the only reliable way to validate the whole chain: fencing, promotion, DNS propagation, pool recycling, and application behavior.

Running this on a schedule, with the whole on-call rotation involved, turns failover from a theoretical capability into a rehearsed procedure. The first time you promote a replica should not be during a customer-facing outage.

Key Takeaways

1. Lag is a data-loss budget, not just a metric — define an acceptable threshold before you need it, and make automated failover respect it.
2. Fence before you promote — assuming the old primary is dead without cutting off its write access is how you get split-brain.
3. DNS and connection pools lag behind topology changes — plan for stale connections explicitly, don't rely on TTLs alone.
4. Make topology changes a first-class application event — don't let read routing assumptions silently serve stale or wrong data during transitions.
5. Rehearse failover on a schedule — game days surface the broken assumptions that incidents otherwise reveal at the worst possible time.
6. Get the data-loss threshold signed off by the business, not just engineering — it's a product decision disguised as a technical one.