Database Replication

Replication keeps copies of the same data on multiple nodes. It serves three purposes: availability (the system continues if a node fails), read scalability (reads spread across copies), and latency (copies placed near users). Replication does not increase write capacity for a single dataset, since every copy must apply every write. Write scaling requires sharding.

The central difficulty is keeping replicas consistent while the data changes and nodes and networks fail.

Leader-follower replication

One node is the leader (primary, master). It accepts all writes. Followers (replicas, secondaries) receive the leader's changes and apply them in the same order. Reads can go to the leader or to followers.

flowchart LR
    C["Client writes"] --> L["Leader"]
    L -->|"replication log"| F1["Follower 1"]
    L -->|"replication log"| F2["Follower 2"]
    R["Client reads"] --> F1
    R --> F2

How changes are shipped:

New followers are set up from a consistent snapshot of the leader plus the log from the snapshot's position onward.

Synchronous vs asynchronous

• Synchronous: the leader waits for a follower to confirm before acknowledging the write to the client. The follower is guaranteed to have the data, but any slow or unavailable synchronous follower blocks writes.
• Asynchronous: the leader acknowledges after its own commit and ships changes afterward. Writes stay fast and are not blocked by followers, but committed writes can be lost if the leader fails before they replicate.
• Semi-synchronous: one follower is synchronous and the rest asynchronous. If the synchronous follower becomes slow, another is promoted to that role. This guarantees at least two copies of each acknowledged write. Some systems allow requiring acknowledgement from a quorum of followers.

Most deployments use asynchronous or semi-synchronous replication. Strict synchronous replication to all followers makes availability worse, since the write path depends on every node.

Replication lag and its anomalies

With asynchronous followers, a follower can be behind the leader. This is replication lag. Reads from a lagging follower are eventually consistent: given no new writes, followers eventually converge. Lag is normally milliseconds but can grow to seconds or minutes under load or network problems. Anomalies and typical remedies:

These guarantees are defined in consistency models.

sequenceDiagram
    participant U as User
    participant L as Leader
    participant F as Follower (lagging)
    U->>L: write comment
    L-->>U: ok
    L-->>F: replicate (delayed)
    U->>F: read comments
    F-->>U: comment missing
    L->>F: replication arrives

Failover

Failover replaces a failed leader with a follower. Manual or automatic, the steps are:

1. Detect the failure, usually by missing heartbeats for a timeout.
2. Choose a new leader, typically the follower with the most up-to-date log, by an election (see leader election and consensus).
3. Reconfigure clients and followers to use the new leader. The old leader, if it returns, must recognize it is demoted.

Failure modes:

• Lost writes. With asynchronous replication the new leader may lack the old leader's last acknowledged writes. If the old leader rejoins, its conflicting writes are usually discarded, which loses data and can break external systems that observed them (for example, reused auto-increment IDs).
• Split brain. Two nodes both believe they are leader and accept writes, causing divergence. Mitigations are quorum-based leader election, leases, and fencing that prevents the old leader from writing.
• Wrong timeout. Too short triggers unnecessary failovers under load spikes, which add more load. Too long prolongs the outage.

Multi-leader replication

Several nodes accept writes and replicate to each other. It is used mostly for multi-datacenter deployments (a leader per region, so local writes are fast and a region outage does not stop others), for offline-capable clients (each device is a leader), and for collaborative editing.

flowchart LR
    subgraph R1["Region A"]
        L1["Leader A"]
    end
    subgraph R2["Region B"]
        L2["Leader B"]
    end
    L1 <-->|"async replication"| L2

The main problem is write conflicts: the same record is modified concurrently in two regions. Approaches:

• Avoid conflicts: route all writes for a given record to one leader (for example, by user's home region).
• Last write wins (LWW): keep the write with the highest timestamp. Simple, but concurrent writes are silently lost and clock skew affects the outcome.
• Merge: combine values, using CRDTs or application logic.
• Record conflicts and resolve them on read or by the application.

Autoincrement keys, unique constraints, and triggers behave poorly across leaders. Multi-leader setups are avoided where single-leader suffices.

Leaderless replication

Any replica accepts writes and reads. There is no failover. Examples: Amazon Dynamo (the original paper), Cassandra, Riak.

• The client (or a coordinator node) sends each write to N replicas and considers it successful after W acknowledgements.
• A read queries R replicas and takes the most recent version by version metadata or timestamp.
• When R + W > N, read and write sets overlap, so a read sees at least one replica with the latest acknowledged write (absent edge cases such as sloppy quorums). Example: with N = 3, W = 2, R = 2, a write reaches at least 2 of 3 replicas and a read contacts 2 of 3, so at least one contacted replica holds the write. With W = 1 and R = 1, a read can hit a replica that has not received the write. See quorums and replication protocols.
• Replicas that missed writes catch up through read repair (a read that finds a stale replica writes the newer value to it) and background anti-entropy processes.
• Concurrent writes require conflict resolution by LWW or version vectors (see time, clocks, and ordering).

Comparison

Interview questions

• Why does adding read replicas not help a write-bound database? Every replica must apply every write, so write capacity is bounded by what a single node can process. Partitioning the data across leaders is needed to scale writes.
• What is lost when a leader with asynchronous replication fails? Writes acknowledged to clients but not yet received by any follower. Semi-synchronous replication limits this by guaranteeing at least one follower has each acknowledged write.
• How would you provide read-your-writes consistency with follower reads? Read data the user recently modified from the leader for a period after a write, or have the client carry the log position or timestamp of its last write and read only from a follower that has reached it.
• What is split brain and how is it prevented? Two nodes act as leader at once and accept divergent writes. It is prevented with quorum-based election so only one candidate can win a majority, and with fencing tokens or leases so a deposed leader cannot write.
• When is multi-leader replication appropriate? When writes must be accepted locally in several regions or offline, and conflicts are rare or can be merged deterministically.