Collaborative Editing: Framing, Scale Math, and Architecture

This lesson covers sections 1 to 3 of Chapter 18, Real-Time Collaborative Document Editing, and is part of Part II, Production System Blueprints. The chapter continues in Collaborative Editing: Deep Dives, Peritext, and Data Models.

1. Executive Summary & The Problem Space

The Scene: The Boardroom Redline Desync

At 9:15 AM on a Monday, the legal team and executive leadership of a Fortune 50 company gather in the main boardroom to finalize the terms of a $4.2 billion corporate acquisition.

Three attorneys and the Chief Legal Officer open the definitive merger agreement simultaneously in the company's collaborative document editor.

The document is 420 pages long, containing 180,000 words.

Attorney A is seated in the San Francisco boardroom on high-speed fiber. Attorney B is on a flight over Kansas, connected to spotty in-cabin satellite WiFi with 800ms latency and 15% packet loss. Attorney C is reviewing the document on an iPad from a taxi in New York.

At 9:22 AM, Attorney A navigates to Section 14.2 (Indemnification Cap) and edits the clause: "The maximum aggregate liability shall not exceed $50,000,000." Attorney A places their cursor after $ and types 1. The clause now reads: "$150,000,000."

Simultaneously, Attorney B, typing through intermittent satellite WiFi, selects the word "maximum" at the start of the sentence and replaces it with "sole and exclusive".

Because the engineering team implemented naive Operational Transformation (OT) on a centralized node without strict concurrency invariants, Attorney B's offline buffer flushes just as Attorney A's keystrokes arrive.

The transformation function fails to correctly account for the compound index offset caused by the deletion of "maximum" and the insertion of "1".

The characters interleave catastrophically.

On Attorney A's screen, the liability cap displays as "$50,000,000". On Attorney B's screen, the liability cap displays as "$150,000,000". On the CEO's display, the document text corrupts into: "The sole and ex1clusive aggregate liability shall not exceed $50,000,000."

When the contract is exported to PDF for signature, neither the attorneys nor the executives realize that the two sides are signing legally divergent versions. The acquisition is stalled for six weeks in litigation.

In the emergency post-mortem, a Senior engineer recommended adding an optimistic database locking mechanism: "When someone starts typing in a paragraph, we lock that paragraph in Redis with a 5-second lease so nobody else can edit it."

A Staff engineer rejected the proposal immediately: "Pessimistic or paragraph-level locking destroys the entire user experience of modern collaborative software. Figma, Google Docs, and Notion do not lock paragraphs. The failure occurred because we relied on stateful Operational Transformation whose transformation functions cannot handle multi-party offline network partitions. We must transition to Conflict-Free Replicated Data Types (CRDTs). With sequence CRDTs, operations are mathematically commutative: every client converges to the exact same character sequence regardless of network latency, packet arrival order, or offline disconnection."

Clarifying Intent: Three Incompatible Collaboration Engines

When an interviewer asks you to "Design Google Docs," "Design Figma," or "Design Notion," they are testing whether you understand the fundamental conflict resolution model required by the product intent:

THE THREE COLLABORATION INTENTS

1. Intent 1: Real-Time Character-Level Text (The Google Docs Standard): Keystroke-by-keystroke concurrent editing. Requires sequence CRDTs where every character has an immutable Lamport ID. Offline clients must merge seamlessly upon reconnection.
2. Intent 2: Block-Based Rich Text (The Notion Standard): The document is an ordered tree of blocks (headers, paragraphs, callouts, tables). Edits inside a block can be LWW (Last-Write-Wins) or CRDT-based, while tree structure mutations (reordering, indenting) use Tree CRDTs.
3. Intent 3: Multiplayer 2D Canvas (The Figma Standard): Edits mutate spatial properties (x , y , width , fillColor ) of independent shape nodes. Concurrency is resolved via Last-Write-Wins with Lamport timestamps per property, prioritized for 60fps visual smoothness.

STAFF MOVE

Senior: "We will use WebSockets and Operational Transformation (OT) like Google Docs originally did, using a central server to transform operations."

Staff: "Operational Transformation requires a single authoritative server to totally order operations and fails complex multi-party offline edge cases (TP2 condition). I will design a local-first architecture using Sequence CRDTs (such as YATA or RGA). Characters receive immutable <ClientID, LamportClock> coordinates. Operations are mathematically commutative ( A ∘ B = B ∘ A ), allowing local edits to render instantaneously at 0ms latency while guaranteeing mathematical convergence across online and offline clients without central lock contention."

Why: The Staff engineer recognizes that modern collaboration must be local-first and mathematically robust against network partitions.

Scale & Back-of-the-Envelope Economics

To understand the system constraints of a hyperscale collaborative suite:

Daily Active Documents: 50,000,000 active documents / day.

Peak Concurrent Active Documents (Rooms): 2,500,000 concurrently open documents.

Concurrent Editors per Active Document:

• 95% of documents: 1 to 2 concurrent editors.
• 4.9% of documents: 3 to 10 concurrent editors.
• 0.1% of documents (All-Hands meetings, company town halls): 100 concurrent typists and 1,000 view-only readers.

Keystroke Mutation Throughput:

• Average typing speed: 5 keystrokes / second per active typist.
• For a busy document with 20 active typists: 20 * 5 = 100 mutations / second.
• For a global fleet of 2.5 million active rooms (assuming 10% active typing at any given second): 250,000 rooms × 2 typists × 5 ops/sec = 2,500,000 keystroke mutations / second across the global edge tier!

Memory Footprint & CRDT Metadata Overhead:

• Plaintext: 1 character = 1 byte.
• CRDT Item Struct: <ClientID (8B), Clock (8B), LeftID (16B), RightID (16B), Content (2B), Flags (1B)> ≈ 51 bytes per character!
• A 100,000-word document (≈ 600,000 characters) requires: 600,000 * 51 bytes = ~30.6 Megabytes of RAM in active CRDT memory.
• 2,500,000 active documents in RAM: 2,500,000 * 30 MB = 75 Terabytes of cluster RAM across the collaboration engine tier.

The Staff Insight: This metadata overhead mandates Tombstone Compaction and Background Snapshotting to keep memory costs manageable.

COLLABORATION SYSTEM CAPACITY PROFILE

2. Requirements & High-Level Architecture

Functional Requirements

1. Real-Time Keystroke Collaboration: Multiple users can concurrently type, insert, and delete text in the same document with immediate local feedback (0ms) and sub-25ms peer propagation.
2. Deterministic Conflict Convergence: When multiple users edit the exact same character offset concurrently (or while offline), all clients must automatically converge to identical text without data loss.
3. Cursor & Selection Presence: Display real-time cursor positions, text selections, and user avatar badges with distinct colors for all active room participants.
4. Offline Support & Re-sync: Users can disconnect, edit for hours in airplane mode, reconnect, and have their edits merge seamlessly.
5. Version History & Revert: Capture immutable snapshots over time, allowing users to browse and restore past document revisions.

Non-Functional Requirements (Staff Invariants)

1. Local-First Zero Latency: The local typist must never experience keyboard latency or typing stutter waiting for server acknowledgments.
2. Mathematical Strong Eventual Consistency (SEC): Any two nodes that have received the same set of operations must be in the exact same state, regardless of the order in which operations arrived.
3. Fault-Isolated Collaboration Rooms: A failure or CPU spike in Document A must never impact users editing Document B.
4. Bandwidth Efficiency: Real-time cursor broadcasts and operation deltas must not saturate client mobile bandwidth (< 50 KB/sec per client).

3. End-to-End System Architecture

The architecture separates the ultra-low latency In-Memory Collaboration Tier from the Ephemeral Presence Awareness Bus and the asynchronous Mutation Persistence Pipeline:

Figure 18.1: End-to-End Real-Time Collaborative Document Editing Architecture

Architectural Components in Depth

COMPONENT 1: COLLABORATIVE CLIENT RUNTIME (WEB / DESKTOP / MOBILE)

The client-side execution environment embedded in the user's application (React/Wasm, Swift, or Kotlin):

• In-Memory Client CRDT (Yjs / Automerge Engine): Maintains an active, memory-mapped representation of the document's sequence CRDT in client memory. All character insertions and deletions are applied locally first.
• 0ms Local Feedback Loop: When the user types a character, the client inserts the character into its local CRDT linked list immediately and renders it to the screen DOM in sub-1ms. The user never experiences typing lag, cursor stutter, or network-bound keystroke buffering.
• Operation Generation & Encoding: Translates user edits into discrete, commutative mutation structs: Insert(ID: {client_id, clock}, content: "k", left_origin: ID_L, right_origin: ID_R). Operations are binary-encoded using compact variable-length integers (lib0 encoding) to keep keystroke payloads under 32 bytes.
• Offline Mutation Journal (IndexedDB / SQLite): If the client loses network connectivity (e.g. laptop lid closes or subway tunnel transit), the client continues accepting edits smoothly, appending all operations to an append-only transaction log in IndexedDB. Upon reconnection, the client transmits its local state vector and flushes pending mutations in order.
• WebSocket Transport Manager: Maintains a persistent TLS WebSocket connection (wss://collab.service.com/v1/connect ) with automated heartbeat pings (every 15s) and truncated binary exponential backoff reconnect logic.

COMPONENT 2: WEBSOCKET SYNC GATEWAY TIER

The horizontally scalable front-line connection termination fleet:

• High-Density Connection Termination: Manages millions of concurrent persistent TCP/TLS WebSocket connections across stateless Go or Rust worker pods utilizing Linux epoll or macOS kqueue. Each 16-core gateway node sustains 100,000 idle persistent connections consuming under 2.5 GB of RAM.
• Authentication & Token Decoupling: Validates client JWT bearer tokens and evaluates document-level access control lists (ACLs) against the authorization service cache before admitting a socket into a document room.
• Sticky Routing via Consistent Hash Ring: Because document collaboration is stateful in RAM, gateways route incoming connections to the specific Collab Room Engine instance hosting that document. The gateway maps document_id across a consistent hash ring (with 256 virtual nodes per physical engine): TargetNode = ConsistentHashRing.Get(document_id) If the target node fails, the hash ring deterministically shifts the room to the next replica node in the ring.
• Traffic Protocol Multiplexing: Decouples heavy, causal document text mutations (protocol: "doc-mutation" ) from high-frequency, ephemeral cursor presence messages (protocol: "awareness" ), routing them to independent backend tiers.

COMPONENT 3: IN-MEMORY COLLAB ROOM ENGINE TIER

The authoritative real-time state synchronization engine:

• Room Actor Model: Each active collaborative document is instantiated as an isolated, single-threaded actor process in memory (utilizing an Akka/Pekko or Go-channel per-room mailbox pattern). An engine node hosts 2,000 to 5,000 concurrent active document rooms without lock contention.
• Authoritative Sequence CRDT: Holds the master linked list of CRDT items for the document.
• Commutative Conflict Convergence (YATA / RGA): When concurrent operations arrive from multiple clients, the actor evaluates deterministic tie-breaking rules (comparing original reference anchors, Lamport clocks, and client IDs). Operations can arrive out of order from the network and still produce identical text states across all nodes.
• Vector Clock Tracking: Tracks the continuous Lamport vector clock for each active collaborator: StateVector = {ClientA: 142, ClientB: 88, ClientC: 12}.
• Sub-25ms Delta Broadcast: Immediately upon accepting a valid mutation, the engine broadcasts the operation delta to all connected peer gateways for that room, achieving sub-25ms peer-to-peer keystroke latency globally.
• Changelog Egress: Emits an append-only mutation event stream to Apache Kafka for durable persistence.

COMPONENT 4: EPHEMERAL PRESENCE & AWARENESS SERVICE (REDIS PUB/SUB)

The high-frequency visual awareness and remote cursor routing tier:

• State Tracking: Tracks real-time cursor offsets (line: 42, col: 18 ), highlighted selection ranges (anchor: 104, head: 112 ), and collaborator identity badges (user_id , display_name , avatar_url , color_hex ).
• Zero Database Writes: Cursor coordinates are strictly ephemeral. Writing cursor movements to a disk database at 30 updates/sec per active collaborator would destroy IOPS with zero business utility.
• Transport via Redis Pub/Sub Cluster: Gateways publish throttled awareness frames directly to Redis pub/sub channels keyed by room:presence:{doc_id}. Redis fans out packets to all gateway instances holding active sockets for that document.
• Client-Side Linear Interpolation (Lerp): Clients sample cursor broadcasts at 30Hz and apply client-side linear interpolation (ease-out smoothing) to animate remote cursors at a buttery 60 FPS, eliminating visual jumpiness.
• Heartbeat Expiration & Ghost Cleanup: Every presence packet carries a 5-second TTL. If a client crashes or disconnects abruptly, the awareness engine automatically evicts the user's cursor from peers' screens within 5 seconds.

COMPONENT 5: ASYNCHRONOUS MUTATION PIPELINE (APACHE KAFKA)

The non-blocking durability and backpressure buffering buffer:

• Key-Partitioned Topic: Document mutations are published to Kafka topic document.mutations.log partitioned strictly by hash(document_id) % num_partitions. This guarantees that all mutations for a specific document are processed sequentially in absolute causal order.
• Decoupled Persistence: Completely isolates the ultra-fast in-memory collaboration engine from database disk latency. Even if the persistence database experiences a 10-second checkpoint stall, active typists experience zero typing lag.
• Retention & Replay Log: Retains 7 days of raw mutation logs, providing an immutable audit trail and enabling catastrophic disaster recovery replay if an in-memory engine crashes before snapshotting.

COMPONENT 6: DOCUMENT PERSISTENCE & S3 VERSION STORAGE

The long-term durable system of record:

• Relational Metadata Store (PostgreSQL Primary/Replica): Stores document ownership, access permissions, document titles, folder hierarchy, and monotonically increasing revision counters.
• Binary CRDT Snapshot Storage (Amazon S3 / Cloudflare R2): Periodically writes compressed binary snapshots of the compiled document state (rev_12000.bin ) using Zstandard compression. Loading a document with 1,000,000 historical edits requires fetching only the latest 40 KB snapshot and replaying the few mutations that occurred since the snapshot.
• Graceful Room Eviction: When the last user disconnects from a document room, the Collab Room Engine flushes the final snapshot to S3, updates PostgreSQL, holds the room in memory for a 10-minute grace period (to handle quick page refreshes), and then unloads the room actor from RAM.

COMPONENT 7: TOMBSTONE COMPACTION WORKER FLEET

The background garbage collection and memory optimization engine:

• Tombstone Pruning Engine: Evaluates inactive documents and long-running sessions. Safely purges deleted character markers (tombstones ) whose vector clocks are strictly older than the established Maximum Offline Window (e.g., 14 days).
• Snapshot Flattening: Re-indexes the sprawling linked-list structure into contiguous plaintext arrays with sparse coordinate maps, slashing RAM consumption by over 80% and keeping memory predictable at hyperscale.

ByteByteGo-Style Step-by-Step Collaboration Walkthrough

To master the Staff interview, you must be able to trace a single keystroke from the user's finger to all peer screens and permanent storage:

STEP-BY-STEP REAL-TIME COLLABORATION MESSAGE FLOW

flowchart TD
    UA["User A: Client"] -->|"(1) Types 'K' -> Updates Local CRDT in 0ms"| GW["WebSocket Sync Gateway"]
    GW -->|"(2) Decodes binary mutation frame; hashes doc_id on ring"| ENG["In-Memory Collab Room Engine"]
    ENG -->|"(3) Actor applies YATA tie-break; updates state vector in RAM"| ENG
    ENG -->|"(4) Emits delta broadcast to Gateway peers (<25ms global latency)"| PEERS["Peer Clients (User B, C)"]
    PEERS -->|"(5) Receive delta; merge into local CRDT; render 'K' on screen"| KAFKA["Asynchronous Kafka Buffer"]
    KAFKA -->|"(6) Partitioned write: doc.mutations.log [Key=doc_id]"| PW["Persistence Worker Fleet"]
    PW -->|"(7) Batches 100 mutations -> Flushes compressed snapshot to S3 + PostgreSQL"| STORE["S3 + PostgreSQL"]

STEP 1: ZERO-LATENCY LOCAL CRDT EXECUTION

User A types 'K'. The local in-memory CRDT engine inserts 'K' into its local linked list and renders it to the screen DOM in under 1ms, generating the immutable mutation tuple <ID A_104, 'K', Left: ID A_103, Right: ID B_42>.

STEP 2: INGRESS PROTOCOL TERMINATION & CONSISTENT HASHING

The client transmits the compact 28-byte binary frame over WebSocket to the WebSocket Gateway. The gateway validates the JWT session token and hashes doc_id across a 256-virtual-node hash ring to locate the assigned Collab Engine node.

STEP 3: AUTHORITATIVE ROOM ACTOR & YATA CONFLICT TIE-BREAKING

The Collab Engine actor processes the mutation inside its single-threaded mailbox loop, verifies anchor positions, executes deterministic YATA conflict tie-breaking against any concurrent edits, updates its vector clock, and constructs a broadcast delta packet.

STEP 4: SUB-25MS PEER DELTA FAN-OUT

The engine pushes the delta packet back to the WebSocket gateways connected to active collaborators (User B and User C), achieving sub-25ms global peer-to-peer latency.

STEP 5: COMMUTATIVE PEER CRDT CONVERGENCE

User B's client receives the delta, merges it into its local CRDT linked list without locking, and renders 'K' at the identical visual document coordinate without screen jitter.

STEP 6: ASYNCHRONOUS WRITE-BEHIND DURABILITY (KAFKA)

Simultaneously with Step 4, the engine emits the mutation to Apache Kafka topic document.mutations.log partitioned strictly by hash(document_id).

STEP 7: COMPACTED SNAPSHOT CHECKPOINTING (S3 & POSTGRESQL)

Persistence consumers batch mutations every 10 seconds or 1,000 edits, compiling an immutable Zstandard-compressed binary snapshot to Amazon S3 while bumping the authoritative document revision in PostgreSQL.