Collaborative Editing: Failure Modes, Interview Questions, and Runbooks

This lesson covers sections 6 to 10 of Chapter 18 and continues Collaborative Editing: Deep Dives, Peritext, and Data Models.

6. Failure Modes, Edge Cases & Resilience

FAILURE MODES & RECOVERY RUNBOOKS

Failure Mode	Detection Metric	Automated Recovery Action
1. Collab Node Crash (Room Eviction)	Engine pod heartbeat missing for > 3s	Gateway fails over to backup node; rebuilds from S3/Kafka
2. Split-Brain Room Partition	Dual master detected in hash ring	Redis Distributed Lock fences rogue node; forces reconnect
3. Tombstone Memory Bloat Leak	Collab engine RAM usage > 85%	Background worker triggers aggressive snapshot & compact
4. Client Stale Reconnection	Offline vector > 14-day cutoff window	Force full document re-sync via snapshot download

Failure Mode 1: Collab Room Engine Node Crash

An engine hosting 3,000 active document rooms crashes due to hardware failure or out-of-memory kernel panic.

The Failure: 15,000 connected WebSockets terminate abruptly.

The Automated Recovery:

1. The WebSocket Sync Gateway detects broken TCP sockets to the failed engine node.
2. Gateways re-calculate consistent hashing assignments for affected document_ids, routing them to healthy engine nodes.
3. The new engine node loads the latest base snapshot from Amazon S3 (taking < 80ms) and replays any uncommitted mutations from the Apache Kafka partition checkpoint.
4. Clients auto-reconnect via WebSocket within 500ms, transmit their local state vectors, and synchronize any un-replicated delta operations. Zero document data is lost.

Failure Mode 2: Network Partition & Split-Brain Room Formation

A temporary network partition causes two different Collab Engine nodes to believe they are the authoritative host for Document X.

The Staff Defense (Redis Fencing Token):

When an engine node boots a document room, it acquires a lease in Redis with an atomic fencing token: SET lock:room:{doc_id} {node_id}:{fence_token} NX PX 10000 Kafka consumers and persistence workers only accept snapshot writes matching the active fencing token. Furthermore, because CRDT operations are commutative, even if clients temporarily connect to two different nodes, when the network heals, the CRDT engines exchange state vectors and merge flawlessly into an identical state!

7. Ten Probing Follow-Up Questions (With Staff Answers)

Q1: "Why use WebSockets over HTTP/2 Server-Sent Events (SSE) or WebRTC?"

Senior Trap: "WebSockets are bidirectional and faster."

Staff Answer: "While SSE provides efficient server-to-client streaming, it requires separate HTTP POST requests for client-to-server keystrokes, which creates significant TCP/TLS connection overhead at 5 keystrokes per second. WebRTC DataChannels provide lower latency via UDP, but corporate enterprise firewalls routinely block non-standard UDP ports. Persistent WebSockets over TLS (port 443) provide full-duplex, low-latency, bidirectional framing with universal firewall and corporate proxy traversal."

Q2: "How do you handle Rich Text formatting (Bold, Italic, Hyperlinks) in a Sequence CRDT?"

Senior Trap: "Treat opening and closing HTML tags like and as individual CRDT characters."

Staff Answer: "Treating format tags as independent characters creates severe syntactic corruption under concurrent edits (e.g. overlapping tags like HelloWorld , breaking the DOM). Modern collaborative editors use Peritext formatting spans: Formatting is decoupled from character nodes and stored as an overlay range of formatting spans: {start_id: ItemA, end_id: ItemB, format: 'bold', clock: 14}. When characters are inserted inside an existing span, the span expands dynamically without creating conflicting boundary characters."

Q3: "What happens if a user goes offline for three weeks and reconnects with edits?"

Senior Trap: "Merge their edits into the document automatically."

Staff Answer: "Merging three-week-old offline edits into an actively evolving document presents severe semantic problems, even if the CRDT converges mathematically. If Chapter 4 was rewritten or deleted by the team two weeks ago, an old offline edit might resurrect deleted text or interleave nonsensically. We enforce an Offline Retention SLA (e.g. 14 days): If a client's state vector is older than our oldest preserved snapshot, automatic fast-forward merge is rejected. The client's changes are branched into a separate revision: 'Offline Conflict Draft (Saved August 14)', giving the user a side-by-side diff UI to review and manually merge their changes."

Q4: "Why not use a standard relational database with JSON columns to store document state?"

Senior Trap: "PostgreSQL JSONB is fast and allows querying document fields."

Staff Answer: "Under active collaborative typing, 20 users generate 100 mutations per second. Updating a single JSONB document row in PostgreSQL 100 times per second triggers massive write amplification, continuous write-ahead log (WAL) flushing, and table page lock contention. PostgreSQL is designed for transactional durability, not sub-25ms collaborative keystroke loops. We hold active document CRDTs in RAM on stateful collab engine nodes, write mutations asynchronously to Kafka, and commit compacted snapshots to PostgreSQL/S3 once every few minutes."

Q5: "How do you implement an 'Undo' stack in a multi-user CRDT document?"

Senior Trap: "Pop the last operation off the local stack and delete the character."

Staff Answer: "In a multi-user document, standard undo is ambiguous: You must undo your last edit without undoing someone else's edit that occurred in the meantime! In a sequence CRDT, Undo is implemented as a new forward operation: If my last operation was Insert('A', ID_1), my Undo command generates a new Delete(target: ID_1) operation. If my last operation was Delete(ID_2), my Undo command un-tombstones ID_2. This ensures that undo actions commute cleanly across peer clients without altering past history."

Q6: "How do you support 'View-Only' users without wasting room resources?"

Senior Trap: "Connect view-only users to the same Collab Room Engine."

Staff Answer: "During an executive town hall, 5,000 employees may open a document while only 2 executives are typing. Routing all 5,000 WebSockets to the stateful Collab Room Engine saturates CPU and memory. We implement a Two-Tier Fan-Out Architecture: View-only clients connect to stateless Read-Edge Proxy Nodes. The Collab Room Engine broadcasts a single update to the read-edge proxy pool, which fans out the text changes to thousands of view-only clients. The primary room engine only processes the 2 active writers."

Q7: "How do you prevent a malicious client from injecting invalid CRDT operations?"

Senior Trap: "Validate JSON schema at the WebSocket gateway."

Staff Answer: "Schema validation is insufficient; an attacker could forge vector clocks, claim someone else's ClientID, or inject circular origin references that hang the linked-list traversal. The Collab Room Engine enforces cryptographic invariants: (1) ClientID Identity Enforcement: Every client is assigned a cryptographic ClientID tied to their verified session token; (2) Monotonic Clock Verification: Incoming operations must strictly increment the client's known vector clock; (3) Acyclic Traversal Guard: Cycle detection algorithms abort any operation where left_origin == right_origin or origins form circular dependencies."

Q8: "How does Figma's collaboration engine differ from Google Docs?"

Senior Trap: "They are the same thing; both use WebSockets and CRDTs."

Staff Answer: "Google Docs is an ordered 1D sequence of characters where relative text position is everything. Figma is a 2D hierarchical scene graph of visual nodes (rectangles, frames, vectors). Figma does not use character-level sequence CRDTs for layout; they use Last-Write-Wins (LWW) with fractional indexing and Lamport timestamps per property: If User A changes fillColor and User B changes rotation , the edits do not conflict. If both edit width , the higher Lamport timestamp wins. Because graphical properties are independent, LWW provides superior 60fps rendering performance over heavy sequence CRDTs."

Q9: "Why not use WebRTC Peer-to-Peer mesh instead of a central WebSocket server?"

Senior Trap: "WebRTC P2P eliminates server costs and reduces latency."

Staff Answer: "A full WebRTC peer-to-peer mesh has O(N^2) network scaling: In a room with 20 editors, each client must maintain 19 independent peer connections, saturating upload bandwidth and client battery life on mobile devices. Furthermore, P2P lacks an authoritative persistence authority: If all users close their laptop lids simultaneously, unsaved document state evaporates. A client-server topology using WebSocket gateways provides O(N) network efficiency, consistent access control enforcement, and guaranteed persistence."

Q10: "How do you detect and heal document desynchronization in production?"

Senior Trap: "CRDTs mathematically guarantee convergence, so desync never happens."

Staff Answer: "While CRDT theory is mathematically proven, real-world software contains implementation bugs (integer overflows, floating-point rounding, race conditions in linked-list splices). We deploy an Asynchronous Checksum Verification Loop: Every 10 seconds, the Collab Room Engine computes a MurmurHash3 checksum of the document's plaintext state. It broadcasts {rev: 1204, hash: 0x8F3A2B}. If a client's local text hash diverges from the server hash, the client silently reports a telemetry alert and requests a fast-forward delta re-sync to repair the divergence before the user notices."

8. Production Hardening & Operational Runbooks

PRODUCTION SLOs & ALERT THRESHOLDS

Metric	Target Threshold	Severity & Action
Peer Keystroke Latency	< 30ms (P99)	P1 Alert: Audit Collab Room Engine CPU & GC pauses
Client Desync Rate	< 0.0001% of sessions	P1 Alert: Check for CRDT bug or client version mismatch
Engine Pod Memory	< 80% RAM utilization	P2 Alert: Trigger background tombstone compaction sweep
WebSocket Disconnects	< 0.05% drop / min	P2 Alert: Audit edge gateway network routes & NAT pools

Incident Runbook: Mitigating Collaborative Room Desync Alert

When an automated checksum mismatch triggers across connected clients:

1. Step 1 (Silence Local Writes): Issue an ephemeral control frame FreezeEditing to connected clients for 500ms to halt conflicting in-flight keystrokes.
2. Step 2 (Fetch Authoritative Server Snapshot): The Collab Engine generates a definitive base revision snapshot and computes its authoritative checksum.
3. Step 3 (Force Client Resynchronization): Transmit the snapshot delta to divergent clients, forcing their local linked list to reconcile against the authoritative server state.
4. Step 4 (Unfreeze & Telemetry Capture): Resume real-time typing and capture the raw operation history of the divergent client to a debugging bucket for root-cause analysis.

9. The Five Structural Fault Lines

Every design decision in real-time collaborative document editing balances algorithmic purity against pragmatic operational scale:

THE 5 STRUCTURAL FAULT LINES

Fault Line	Option A vs Option B	Who Pays?
1. Conflict Model	CRDT (Commutative) vs OT (Central Transform)	CRDT: RAM pays 50x metadata; OT: Engineering pays TP2 bugs
2. Granularity	Character-Level vs Block / LWW Hierarchy	Char: Network pays 5 ops/sec; Block: Users pay coarse typing
3. Network Topology	Client-Server WS vs WebRTC P2P Mesh	Server: Infra pays cluster RAM; P2P: Clients pay O(N^2) battery
4. Awareness Bus	Ephemeral In-Memory vs Persistent Kafka Logs	Ephemeral: Zero durability; Persistent: DB melts on cursor
5. Compaction SLA	Eager Tombstone Purge vs Lazy TTL Window	Eager: Breaks offline clients; Lazy: Cluster pays higher RAM

10. Chapter Summary & Staff Decision Matrix

STAFF DECISION MATRIX: REAL-TIME COLLABORATION

Decision Point	Default Staff Position
Conflict Resolution	Sequence CRDT (YATA / RGA) for local-first zero-lag
Character Identification	Immutable <ClientID, LamportClock> coordinate
Real-Time Transport	Persistent TLS WebSockets via sticky edge gateways
Room Routing Topology	Consistent Hash Ring on document_id to RAM room
Cursor Presence	Ephemeral Redis Pub/Sub throttled to 30Hz (No DB)
Mutation Persistence	Asynchronous Apache Kafka partitioned by doc_id
Long-Term Durability	Periodic base snapshots to S3 + PostgreSQL metadata
Compaction Strategy	Prune tombstones older than 14-day offline window

The Staff Invariant Checklist

Before completing a collaborative document editing system design interview, verify that you have covered these four core principles:

• I have committed to a conflict resolution model: I chose Sequence CRDTs over OT, explicitly explaining the local-first zero-latency benefit and commutativity.
• I have addressed the CRDT metadata bloat tradeoff: I designed background snapshotting and tombstone garbage collection tied to a minimum vector clock.
• I have decoupled document text from cursor awareness: Ephemeral cursor presence uses lightweight Redis Pub/Sub throttled at 30Hz with zero disk persistence.
• I have handled network partition and offline re-sync: Clients synchronize missing operations using a two-phase state vector handshake upon reconnecting.