Database High Availability and Disaster Recovery: Architecture Patterns Explained

Every production incident post-mortem eventually arrives at the same question: why did the database stay down as long as it did? Usually the answer isn't a missing backup. It's an architecture that was never designed to fail gracefully — no tested failover path, no clear ownership of RTO and RPO targets, and a team that assumed "we take nightly backups" was a strategy rather than a single, fairly weak line of defense.

Backups protect you against data loss. They do almost nothing for availability, and on their own they make a poor disaster recovery plan — restoring a multi-terabyte backup can take hours, during which the business is offline. High availability (HA) and disaster recovery (DR) are engineering disciplines with their own architecture patterns, failure models, and cost tradeoffs. This article works through the patterns that matter for engineers who own uptime and recoverability, not just schema design.

HA vs DR: two different problems

High availability is about surviving the failure of a single component — a node, a disk, an availability zone — with minimal or zero downtime, while the overall system keeps serving traffic. It's measured in "nines" (99.95%, 99.99%) and in seconds of interruption. The failure domain is narrow and the response is typically automatic: a health check fails, a replica is promoted, traffic resumes.

Disaster recovery is about surviving the loss of an entire site, region, or the underlying infrastructure provider itself — floods, regional cloud outages, ransomware, a botched migration that corrupts data across all live replicas. DR is measured by two numbers: RTO (recovery time objective — how long you're down) and RPO (recovery point objective — how much data you can afford to lose). DR plans are invoked far less often, involve more manual judgment, and often trade off write latency and cost far more aggressively than HA does.

The architectures diverge because the failure domains diverge. An HA design that keeps replicas in the same region survives a node crash but not a regional outage. A DR design that only fails over on human decision is too slow to be an HA mechanism. You need both, deliberately layered, not one masquerading as the other.

High availability keeps the lights on when a bulb blows; disaster recovery keeps the building standing when the block burns down.

Replication topologies that enable HA

Almost every HA pattern is built on replication. The differences are in consistency guarantees, failover speed, and what happens when nodes disagree.

1. Asynchronous replication. The primary commits a write and returns success to the client before the replica has acknowledged it. This is the default in most self-managed PostgreSQL and MySQL deployments because it keeps write latency low regardless of replica distance. The cost is a non-zero RPO — if the primary dies before replicating the last few transactions, they're gone. Replication lag is the metric to watch; under load it can grow from milliseconds to minutes, silently expanding your real RPO far beyond what the architecture diagram implies.

2. Synchronous replication. The primary waits for acknowledgment from one or more replicas before confirming the write to the client. PostgreSQL's synchronous_commit parameter and synchronous_standby_names control this directly. This gets you an RPO of effectively zero for the covered replica set, at the cost of added write latency — every commit now pays the network round trip to the standby. Synchronous replication is practical within a metro area or across availability zones in the same region; it becomes a serious performance liability across geographic regions, which is why it's usually reserved for HA, not DR.

3. Primary-replica (single-writer) failover. The most common topology: one primary accepts writes, one or more replicas serve reads and stand ready for promotion. Simple to reason about, simple to debug, and the default choice unless you have a specific reason to need multi-writer. The failover event itself is where the complexity lives — covered below.

4. Multi-primary replication. Multiple nodes accept writes concurrently, replicating to each other. This removes the single-writer bottleneck and can reduce failover time to near zero since there's no promotion step, but it introduces write conflicts: two nodes accepting conflicting updates to the same row before the other has caught up. Conflict resolution strategies (last-write-wins, application-level merge logic, conflict-free replicated data types) add real engineering cost, and debugging a conflict-resolution bug in production is unpleasant. Multi-primary is the right call when you have a genuine requirement for multi-region active-active writes and the team to support it — not as a default HA pattern.

5. Quorum-based consensus systems. Systems like etcd, CockroachDB, and YugabyteDB use Raft or Paxos-family consensus: a write is only committed once a majority of nodes acknowledge it. This gives you strong consistency and automatic leader election without a separate orchestration layer, because the consensus protocol itself defines who the leader is at any moment. The tradeoff is write latency bounded by quorum round-trip time, and operational complexity in understanding quorum loss scenarios — lose the majority and the cluster stops accepting writes entirely, which is a deliberate safety choice, not a bug.

Failover mechanics: automatic vs manual, and split-brain risk

Automatic failover looks simple from the outside: primary dies, replica gets promoted, traffic resumes. The mechanics underneath determine whether that promise holds under real failure conditions.

Health checks need to distinguish "the primary is dead" from "the primary is briefly unreachable from this one observer" — a network partition that isolates the health-check node from the primary looks identical to an actual primary failure. Naive failover systems that promote on a single missed heartbeat are a classic source of unnecessary failovers and, worse, split-brain.

Leader election is how the system decides which replica gets promoted when a failure is confirmed. Tools like Patroni for PostgreSQL, or the built-in election logic in Raft-based systems, use a consensus mechanism among the surviving nodes rather than trusting any single observer's judgement — this avoids two different nodes each independently deciding they're the new primary.

Fencing is the step people skip and later regret. After a new primary is elected, the old primary must be prevented from accepting writes if it comes back online — via STONITH (shoot the other node in the head), revoking its network access, or a fencing token that downstream consumers reject once a newer one is issued. Without fencing, a "recovered" old primary can start accepting writes again while a new primary is also live.

That's split-brain: two nodes both believe they are the primary and both accept writes. It is arguably the worst failure mode in database HA, worse than downtime, because it silently produces divergent, conflicting data that has to be manually reconciled — and by the time anyone notices, the corruption has usually propagated to dependent systems. Quorum requirements (only a node that can see a majority of the cluster may become primary) and fencing are the two controls that prevent it. Any HA design that promotes based on a single node's opinion, without quorum, is one network blip away from split-brain.

Managed cloud database services change the calculus

Amazon RDS Multi-AZ and Aurora, Google Cloud SQL with regional high availability, and Azure SQL Managed Instance with its zone-redundant and auto-failover group options all implement synchronous replication and automated failover for you at the infrastructure layer. They handle the health checking, leader election, fencing, and DNS/endpoint cutover that you'd otherwise build with tools like Patroni or Orchestrator. They also typically bundle point-in-time recovery (PITR) — continuous transaction log backup that lets you restore to any second within a retention window, not just to the last nightly snapshot.

This is a genuine reduction in operational burden, and for most teams it's the right default. But it doesn't eliminate the HA/DR design problem — it moves the boundary of what you're responsible for.

What you still own: schema and query design that determines whether replication lag stays trivial or balloons under load; region strategy — Multi-AZ protects against a zone failure, not a regional one, so cross-region DR (Aurora Global Database, Cloud SQL cross-region replicas, Azure auto-failover groups) is a separate decision with separate cost; connection handling during failover, since a promoted endpoint doesn't help if your application connection pool holds onto dead connections for 30 seconds; and testing — cloud providers give you the mechanism, but nobody validates that your specific application tolerates a failover cleanly except you. Treating "it's on RDS Multi-AZ" as equivalent to "we have an HA/DR strategy" is a common and costly category error.

A real-world example

Consider a payments platform for a financial services company with a contractual requirement: sub-minute RTO for a zonal failure, and near-zero data loss for anything short of full regional loss. The design that meets this typically looks like a two-tier setup.

Tier one, HA within the region: synchronous replication across three availability zones, with automated failover managed by the cloud provider or a self-hosted consensus-based orchestrator. A zone failure triggers promotion in seconds, and because replication is synchronous, no committed transaction is lost. Write latency absorbs the cross-AZ round trip — typically single-digit milliseconds within a region, an acceptable cost for a payments workload where correctness dominates.

Tier two, DR across regions: an asynchronous replica in a geographically distant region, replicating continuously but not in the write path. This protects against the scenario tier one can't: the entire primary region becoming unavailable. Because it's asynchronous, there's a real RPO — typically seconds, sometimes more under lag — which the team accepts explicitly rather than by omission, and documents in the DR runbook. Promoting the cross-region replica to primary is a decision made deliberately, often with a manual confirmation step, because falsely triggering a full regional failover is expensive and disruptive in its own right.

The tradeoff made explicitly here: synchronous replication is confined to the region, where latency cost is tolerable, and the DR tier accepts a small, quantified data-loss window in exchange for keeping cross-region writes off the critical path. The alternative — synchronous replication across regions — was evaluated and rejected because it would have added tens to hundreds of milliseconds to every transaction, unacceptable for a transaction-processing system with strict throughput requirements.

Common mistakes to avoid

Never testing an actual failover. A failover mechanism that has only ever run in a design document is unproven. Teams that discover their orchestrator's fencing logic doesn't actually work usually discover it during a real incident. Failover drills, like restore drills, belong on a calendar.

Using synchronous replication across regions to "be extra safe." This is one of the most common over-engineering mistakes in DR design. It feels more protective, but the latency penalty on every write can be severe enough to make the primary application slower than the business tolerates, and in the worst case a slow or unreachable DR replica can stall the primary entirely if the replication mode isn't configured to degrade gracefully. Reserve synchronous replication for same-region or same-metro HA; use asynchronous replication for cross-region DR and own the resulting RPO deliberately.

Treating a read replica as a DR plan. A read replica in the same region protects against neither a regional outage nor, in many configurations, logical corruption — a bad DELETE or a schema migration gone wrong replicates to the replica just as faithfully as a good write does. A real DR plan requires geographic separation, a defined promotion procedure, and — critically — a way to recover from logical corruption that replication alone can't undo, such as PITR or delayed replicas.

Frequently asked questions

What RPO and RTO should we target? There's no universal number — it should come from the business cost of downtime and data loss, not from what's technically convenient. A team that hasn't had this conversation with stakeholders is usually building to an implicit RTO of "however long the postmortem takes," which is not a target anyone chose.

Do we need multi-region active-active for HA, or is Multi-AZ enough? For most workloads, Multi-AZ (or the equivalent zone-redundant configuration) covers HA sufficiently, since zone failures are far more common than full regional outages. Multi-region active-active solves a narrower, harder problem — low write latency for geographically distributed users, or true DR against regional loss — and carries real conflict-resolution and operational cost. Don't reach for it unless the requirement is specific.

How often should we run failover and restore drills? Quarterly is a reasonable floor for most production systems; anything handling regulated or financial data often warrants monthly. The exact cadence matters less than the discipline of tracking whether the last drill's measured RTO still meets the target — if it doesn't, that's the signal to fix the architecture before the next real incident does it for you.

HA and DR aren't a checkbox you get by enabling a managed service flag — they're an ongoing design discipline that combines replication topology, tested failover mechanics, and a clear-eyed accounting of what your cloud provider covers versus what you still own. Get the architecture right, then prove it works under drill conditions regularly, because the only RTO number that matters is the one you've actually measured.