Redundancy and Replication

Any single component — server, disk, network link, power supply — will eventually fail. Redundancy ensures that the failure of any one component does not bring down the entire system. This is the foundation of high availability.

Ship analogy: a ship has redundant engines, bilge pumps, and navigation systems. If one fails, the others keep the ship running. The cost is having two of everything — but the alternative (sinking) is far worse.

In distributed systems, redundancy applies at every layer: multiple app servers behind a load balancer, replicated databases, RAID disks, redundant network switches, multiple data centers, and even multiple cloud providers.

The most common DB replication pattern. One primary (master) accepts writes. One or more replicas (slaves) receive a continuous stream of changes from the primary and apply them to stay in sync. Reads can go to any replica; writes must go to the primary.

Benefits: (1) Scale reads — add replicas to handle more read traffic. (2) Failover — if the primary fails, promote a replica to primary. (3) Reporting — point analytics queries at a replica without slowing down the primary.

Replication lag: changes propagate to replicas asynchronously (typically within milliseconds). During lag, a read from a replica may return stale data. For most reads (feed, catalog browsing), this is acceptable (eventual consistency). For reads-after-writes (user sees their own update immediately), route reads to the primary or use synchronous replication.

Asynchronous: primary writes to disk, acknowledges the write to the application, and replicates to replicas in the background. Lowest write latency. Risk: if the primary fails before replication completes, data acknowledged to the application may be lost.

Synchronous: primary waits for at least one replica to confirm it has written the data before acknowledging the write. No data loss (zero RPO). Cost: write latency increases by the replica's network RTT (typically 1-5 ms within a data center, 50-100 ms cross-region).

Semi-synchronous (MySQL default): primary waits for one replica to confirm receipt but not for it to apply the change. Balances durability and latency.

Choice: financial systems use synchronous replication (no data loss tolerated). Web apps typically use asynchronous (a few milliseconds of potential lag is acceptable).

Multiple masters: any node can accept writes. Nodes exchange changes with each other. Needed for multi-region writes (users in different regions write to local masters). Problem: write conflicts — two masters accept conflicting writes to the same row simultaneously. Conflict resolution strategies: last-write-wins (timestamp), custom merge functions, or surface conflicts to the application.

Multi-master adds significant complexity. Avoid unless you have a specific need (multi-region write availability). CockroachDB and Google Spanner handle multi-master conflicts using atomic clock-based timestamps (TrueTime) to impose a global ordering.

Manual failover: operations team detects failure, promotes a replica to primary, updates connection strings. MTTR measured in minutes to hours. Not suitable for high-availability requirements.

Automatic failover: monitoring tools (Orchestrator, MHA, Patroni) detect primary failure via heartbeat, elect a new primary from among the replicas, and update DNS/config automatically. MTTR measured in seconds (20-60 sec typical).

Failover risks: (1) Split-brain — two nodes both think they are primary (solved by fencing/STONITH). (2) Replication lag — the promoted replica may be behind, causing data loss (RPO > 0). (3) Application reconnection — clients must detect the primary change and reconnect.