Availability Patterns — Failover and Replication

Availability is the fraction of time a system is operational and serving requests correctly. It is typically expressed as a percentage: 99.9% ("three nines") means the service can be down for no more than 8 hours 46 minutes per year. 99.99% ("four nines") allows only ~52 minutes of downtime per year.

Every percentage point of availability is exponentially harder to achieve. Going from 99% to 99.9% requires eliminating cascading failures and adding redundancy. Going from 99.9% to 99.99% requires active-active topologies, data replication across zones, automated failover under 30 seconds, and rigorous chaos engineering.

High availability is achieved through two complementary patterns: failover (switching to a standby when the primary fails) and replication (keeping multiple copies of data so the standby can serve requests immediately).

In active-passive failover, one server (the active) handles all traffic. A second server (the passive/standby) sits idle, receiving a heartbeat from the active every few seconds. If the active fails to send a heartbeat, the passive promotes itself to active — typically by assuming the active's virtual IP address or DNS name — and begins serving traffic.

Downtime during failover depends on whether the passive is "hot" (fully booted, synchronized, and ready to take over in seconds) or "cold" (requires a boot sequence, taking minutes). Hot standby is more expensive but gives sub-minute recovery times.

Downsides: only the active handles writes, so if it fails before data is replicated to the passive, that data is lost. The passive is idle capacity you are paying for. Also called master-slave failover.

In active-active failover, both (or all) servers actively handle traffic simultaneously, sharing the load. A load balancer distributes requests across all active nodes. If one fails, the load balancer detects the failure (via health checks) and stops routing to it; the remaining nodes absorb the traffic.

Active-active improves both availability (no idle standby) and throughput (both nodes serve real requests). The tradeoff: coordinating writes across multiple writers is hard. Most active-active systems are either eventually consistent (writes propagate asynchronously, so nodes may briefly disagree) or suffer higher write latency due to synchronous coordination.

For stateless services (web/API servers), active-active is straightforward — any node can handle any request. For stateful services (databases), active-active requires careful conflict resolution strategies. Also called master-master failover.

Master-slave replication has one primary (master) that accepts all writes. Writes are replicated asynchronously to one or more secondaries (slaves), which serve read traffic. This is the most common database topology for read-heavy workloads.

Advantages: reads scale horizontally by adding more slaves; the master handles all write consistency; slaves provide read redundancy. Disadvantages: if the master fails, a slave must be promoted (manual or automated), causing a brief outage; if the master fails before a write is replicated, that write is lost; slaves can lag behind the master, so reads from slaves may be stale.

Used by: MySQL with replication, PostgreSQL streaming replication, MongoDB replica sets (primary-secondary model).

Master-master (multi-primary) replication allows any node to accept writes. Writes are replicated to all other nodes. This avoids the single-writer bottleneck and enables full active-active failover for write-heavy workloads.

The hard problem: conflict resolution. If two clients write to the same record on different masters simultaneously, both writes must be reconciled. Strategies include last-write-wins (timestamps), vector clocks (track causal order), or CRDTs (data structures that merge automatically). All introduce complexity and most sacrifice strict consistency.

Used by: Cassandra (multi-primary with eventual consistency), CockroachDB (distributed SQL with consensus), and multi-region DynamoDB global tables.

Calculating availability for systems with multiple components:

In sequence: if two services A and B are both required to serve a request, total availability = A × B. If A = 99.9% and B = 99.9%, total = 99.8%. Chaining unreliable dependencies multiplicatively degrades overall availability.

In parallel: if either A or B can serve the request (redundant), total availability = 1 − (1−A) × (1−B). If both are 99.9%, parallel availability = 99.9999%. Redundancy dramatically improves availability.

This is why critical services run multiple replicas across independent failure domains (separate racks, zones, or regions). If one zone goes down, traffic routes to the others without the user noticing.