Multi-Region Architecture and Disaster Recovery

A single data center or cloud region can fail as a whole: power or network loss, a control-plane failure, a natural disaster, or a faulty deployment. Disaster recovery (DR) is the set of plans and mechanisms for restoring service after such a failure. A multi-region architecture runs the system in more than one geographic region so that a regional failure does not stop it. Multi-region also reduces latency for distributed users and can satisfy data residency rules.

Within one region, availability zones (separate data centers with independent power and networking) protect against the loss of a single facility. Multi-region protects against loss of a whole region and is considerably more expensive and complex.

RTO and RPO

Two targets define DR requirements.

• Recovery time objective (RTO): the maximum acceptable time between the failure and restored service.
• Recovery point objective (RPO): the maximum acceptable amount of data loss, measured as time. An RPO of 5 minutes means the last 5 minutes of writes may be lost.

flowchart LR
    A[Last good data point] -->|RPO: data loss window| B[Disaster]
    B -->|RTO: downtime window| C[Service restored]

RPO is determined by how often data is copied to another location. Asynchronous replication gives an RPO of the replication lag (seconds). Periodic backups give an RPO of the backup interval (hours). Synchronous replication across regions gives an RPO of zero at a cost in write latency. RTO is determined by how much must be started, restored, and verified after the failure, and by whether it is automated.

Example: a database replicates asynchronously with a typical lag of 2 seconds and 5 seconds at peak. Its RPO is therefore about 5 seconds. If promoting the replica, scaling compute, and redirecting traffic takes 12 minutes, its RTO is about 12 minutes.

Targets are set per system from business impact. Lower RTO and RPO cost more, so a payment system and an internal reporting tool need different targets.

DR strategies

The ranges are typical, not guaranteed. They depend on the implementation and on how much of the failover is automated.

Active-passive and active-active

Active-passive

One region (the primary) takes all writes and serves users. A second region receives replicated data and is ready to take over.

• Single writer per dataset, so no write conflicts.
• Failover requires detecting the failure, promoting the secondary database to primary, and redirecting traffic.
• Unreplicated writes at the moment of failure are lost (asynchronous replication) unless replication is synchronous.
• The passive region's capacity is idle in normal operation.

Active-active

Every region serves users and accepts writes. Users are routed to the nearest healthy region.

• Lower latency and no failover event: if a region fails, the others absorb its traffic (they need spare capacity for this).
• Write conflicts must be handled if the same data can be written in two regions. Options:
  • Partition data by home region (for example each user or tenant belongs to one region). Writes go to the home region, so there are no conflicts, but reads and writes for that user from another region cross regions.
  • Multi-leader or leaderless replication with conflict resolution such as last-write-wins or CRDTs. See database replication.
  • Globally consistent database using consensus across regions (Spanner-style). Writes pay cross-region latency, in the tens to a hundred or more milliseconds depending on distance.
• Cost of complexity: every component (caches, queues, search, sessions) needs a multi-region strategy.

flowchart TD
    U[Users] --> DNS["Global routing: GeoDNS or anycast with health checks"]
    DNS --> A["Region A: services and database"]
    DNS --> B["Region B: services and database"]
    A <-->|Cross-region replication| B

Cross-region replication and consistency

Region-to-region round-trip latency is roughly tens to hundreds of milliseconds, far above intra-region latency (around a millisecond). This drives the trade-off described by the CAP theorem and PACELC:

• Synchronous replication waits for the remote region to confirm each write. RPO is zero, but every write pays the round trip, and if the remote region is unreachable, writes stall or the system must relax the guarantee.
• Asynchronous replication acknowledges after the local write. Writes are fast, but the replica lags, and a failover can lose the unreplicated tail and may expose stale reads in the meantime.
• Quorum-based systems let you choose per operation how many replicas must acknowledge. Cassandra, for example, offers consistency levels such as LOCAL_QUORUM (a quorum within the local data center) and EACH_QUORUM (a quorum in every data center). See quorums.

Non-database state also needs planning: object storage replication, cache warm-up (caches in a failover region start cold and can overload the database), message queues (messages in a failed region's broker may be inaccessible until it recovers), and secrets and configuration.

Failover

Failover is the switch of traffic and roles from a failed component to a standby.

1. Detection. Health checks from multiple vantage points decide that a region is down. A single failed check is not enough, to avoid false positives.
2. Decision. Automatic or manual. Automatic failover is faster but can trigger on partial or transient failures. Manual failover with a runbook is slower and safer for stateful systems.
3. Database promotion. A replica is promoted to leader. The old leader must be prevented from accepting writes if it returns, otherwise there is split brain, with two leaders diverging. Use fencing (see leader election).
4. Traffic redirection. Update DNS or global load balancer routing. DNS changes are limited by TTL and resolver caching, so use short TTLs on failover records or anycast/global load balancers that steer traffic without waiting for DNS caches.
5. Verification. Confirm the standby serves correct traffic.
6. Failback. Returning to the original region requires resynchronizing data first. Failback is a second migration and needs its own plan. Writes that the old primary accepted but never replicated must be reconciled or discarded deliberately, because the new primary has diverged from them.

A failover plan that has never been exercised is unreliable. Run regular DR drills, including actually failing over, and measure real RTO and RPO. See chaos engineering.

Data residency

Data residency rules require certain data, often personal data, to be stored or processed within a country or legal area (for example the GDPR restricts transfers of personal data outside the EU unless conditions are met, and some countries require local storage). Effects on design:

• Partition users by jurisdiction and store their data only in permitted regions. This conflicts with replicating everything to every region.
• DR copies must also comply, so the failover region must be in a permitted location.
• Analytics, logs, and backups can carry personal data. Include them in the residency scope.
• Routing must send each user to the region holding their data.

Specific legal requirements vary and need legal review. The architecture must make placement of data a controllable property.

Backups

Replication is not a backup. Replication copies corruption and accidental deletes to the replica immediately. Backups are point-in-time copies that allow restoring earlier states.

• Full, incremental, and differential backups. Full copies everything. Incremental copies changes since the last backup of any type. Differential copies changes since the last full backup.
• Point-in-time recovery (PITR). Combine a base backup with a continuous archive of the write-ahead log to restore to any moment within the retention window, for example right before a bad deployment or a dropped table.
• 3-2-1 rule. Three copies of data, on two different media types, with one off-site.
• Immutability and isolation. Keep copies that ransomware or a compromised account cannot delete or alter, such as object-lock storage or a separate account.
• Restore testing. A backup that has never been restored is unverified. Automate periodic restores and validation, and record the restore duration since it determines the RTO for the backup strategy.
• Encryption and access control. Backups contain all the data. Encrypt them and restrict access.

Interview questions

• What is the difference between RTO and RPO? RTO is how long the service may be down. RPO is how much recent data may be lost. RTO is governed by recovery procedures, RPO by replication or backup frequency.
• Why is replication not a substitute for backups? Replication propagates every change, including erroneous deletes and corruption, to replicas. Backups retain earlier states so the system can be restored to a point before the error.
• What is the main difficulty of active-active across regions? Concurrent writes to the same data in different regions conflict. The system must partition data by region, resolve conflicts with a rule or CRDTs, or pay cross-region latency for consensus on each write.
• How can DNS-based failover be too slow, and what helps? Resolvers and clients cache records until the TTL expires, so some users keep reaching the failed region. Short TTLs help partially. Global load balancing or anycast redirects traffic without waiting on caches.
• What is split brain during failover, and how is it prevented? Two nodes both act as leader and accept writes, producing divergent data. Prevent it with fencing tokens, quorum-based leader election, or by isolating the old leader before promoting another.