Chat and Presence: Framing, Scale Math, and Architecture

This lesson covers sections 1 to 3 of Chapter 11, Real-Time Collaborative Chat & Presence (Production System Blueprint 8: Persistent Connection Gateways, Monotonic Conversation Sequencing, and Scoped Presence). It continues in Chat and Presence: Sequencing, Fan-Out, and Deep Dives.

Production System Blueprint 8: Persistent Connection Gateways, Monotonic Conversation Sequencing, and Scoped Presence

At a glance

1. The Scenario & Problem Framing

PROMPT:

"Design a real-time messaging and collaborative presence platform (like WhatsApp, Slack, or Discord) supporting 50,000,000 concurrent connected users and delivering 100,000,000,000 messages per day.

The platform must support 1-to-1 direct messaging, small group chats (up to 256 members), and large broadcast channels (up to 50,000 members) with sub-100ms real-time delivery. The system must guarantee strict message ordering within each conversation, provide reliable delivery across intermittent mobile connectivity via a three-tick acknowledgment protocol (Server ACK, Device Delivery ACK, Read Receipt), synchronize unread state across multiple devices, and manage real-time online presence without triggering multi-billion event broadcast storms."

Anyone can build a working chat prototype in thirty minutes using Node.js, Socket.io, and MongoDB. In a toy application with a few dozen users, the server broadcasts every message over open WebSockets, and clients render them as they arrive.

At enterprise scale—serving 50,000,000 concurrent persistent TCP connections and ingesting 100 Billion messages per day (~1,150,000 writes/second sustained, 5,000,000 writes/second peak)—the trivial assumptions of chat architecture violently break down:

1. The Epoll Connection Memory Tax: Holding 50,000,000 open TCP sockets requires massive operating system kernel memory. In a default Linux configuration, each TCP socket allocates 128 KB of read and write buffers. Multiplying 128 KB by 50M sockets requires 6.4 Terabytes of kernel RAM just to hold idle connections, bankrupting infrastructure before a single message is transmitted.

2. The Timestamp Ordering Fallacy: Junior engineers instinctively order messages by client timestamp (sent_at). In distributed mobile environments, device clocks drift by minutes, users manually alter system time, and network latency varies wildly. If User A sends a message at 12:00:01 and User B replies at 12:00:02, a transient cellular retry can cause User B's reply to arrive at the server first. Relying on timestamps causes replies to render above questions, destroying conversational context.

3. The Broadcast Presence Catastrophe: In naive designs, when a user logs in or disconnects, the server publishes a status update to all their contacts. If 50,000,000 active users each have an average of 200 contacts, a wave of morning logins triggers:

   Presence Fan-Out = 50,000,000 users * 200 contacts = 10,000,000,000 presence events!

   Broadcasting 10 Billion status packets across the network saturates inter-service bandwidth, crashes Redis pub/sub clusters, and drains smartphone batteries, even though 99.5% of those 200 contacts do not have an active chat open with the user!

4. The Group Fan-Out Write-Amplification Trap: In a direct 1-to-1 chat, one message produces one delivery. In a 50,000-member community channel (like Discord or Slack #general), writing a single message into 50,000 recipient inboxes generates 50,000 database writes per message. At 100 messages/second in a hot channel, the database must absorb 5,000,000 disk writes per second for a single room, triggering storage engine lockups.

THE CORE PARADOX

Chat feels like a real-time transport stream, but at scale it is fundamentally a store-and-forward persistence engine with an ephemeral latency-optimization layer on top.

Staff Goal: Decouple stateful Epoll WebSocket gateways from stateless message routing, enforce monotonic atomic sequence IDs per conversation in wide-column LSM stores, implement hybrid fan-out at the 256-member boundary, and replace broadcast presence with scoped view-based pub/sub.

The L5 vs L6 Contrast — What the Evaluator Actually Hears

The Senior Candidate (L5): Immediately focuses on the WebSocket protocol, suggesting an open-source gateway or Redis Pub/Sub to connect users. When asked how messages are ordered, they say: "We record the timestamp on the server." When asked what happens when a user is offline, they say: "We publish a push notification via APNs." When asked about group chat, they suggest publishing the message to all members using a message queue. They design exclusively for the online, happy path.

The Staff Candidate (L6): Reframes the problem around delivery reliability contracts, storage-first durability, and organizational ownership within the first 90 seconds: "Chat is a store-and-forward system; the WebSocket layer is merely a latency optimization that can fail completely without data loss. Every message must be atomically committed to a distributed wide-column store (ScyllaDB) and assigned a monotonic per-conversation sequence number before real-time dispatch is attempted. Furthermore, group messaging requires a hybrid fan-out model: write-time inbox duplication for groups under 256 members, and read-time shared timeline queries for large broadcast channels. Finally, we eliminate presence broadcast storms by decoupling presence into lazy-pulled contact sidebars and scoped ephemeral pub/sub channels active only when a conversation screen is open."

2. Step 1: Commit to Intent & Scale Math

Before choosing database engines or sizing connection gateways, an L6 Staff engineer establishes the core operational intent. Real-time messaging encompasses three conflicting system profiles:

The Three Incompatible Intents

1. Consumer Messaging (WhatsApp / Signal):

• The Constraint: Extreme privacy, mobile battery conservation, and offline store-and-forward reliability across low-bandwidth cellular networks.
• The Architectural Commit: End-to-End Encryption (E2E) via the Signal Protocol, per-device message queues, and zero server-side plaintext indexing.

2. Enterprise Collaboration (Slack / Microsoft Teams):

• The Constraint: Corporate compliance, e-discovery, real-time message editing, thread branching, and instant server-side full-text search.
• The Architectural Commit: Plaintext server storage, centralized Elasticsearch indexing, workspace-level channel access control, and audit logging.

3. Massive Community Broadcast (Discord / Telegram Channels):

• The Constraint: Extreme fan-out efficiency for public channels containing 10,000 to 100,000 simultaneous online participants.
• The Architectural Commit: Read-time shared timelines, epoll gateway local RAM fan-out, and eventual consistency for presence and typing indicators.

STAFF MOVE

Senior: "We will use Redis Pub/Sub to route messages between WebSocket gateway nodes."

Staff: "Redis Pub/Sub is fire-and-forget; it has zero persistence and zero buffering. If a recipient's WebSocket gateway pod restarts or suffers network jitter during a publish, the message is permanently lost. For chat routing, we must use a durable store-and-forward architecture: the message is committed to ScyllaDB first, the gateway registry in Redis tracks user-to-gateway mappings, and messages are routed over gRPC. If the recipient is offline, the message remains safely queued in the primary store awaiting their reconnect sync."

Why: Differentiating ephemeral pub/sub from durable store-and-forward message routing is a foundational Staff distinction.

Capacity Planning & Scale Math

At 50,000,000 concurrent connected users and 100,000,000,000 messages per day:

1. INGESTION THROUGHPUT & BANDWIDTH SIZING

Daily Message Volume: 100,000,000,000 messages/day.

Average Ingestion Rate:

Average Write Throughput = 100,000,000,000 / 86,400 seconds ≈ 1,157,400 messages/second

Peak Write Multiplier (4.0x during global evening peak):

Peak Write Throughput = 1,157,400 * 4.0 ≈ 4,630,000 messages/second (~4.6M IOPS!)

Average Message Payload Size: Text payload (160 bytes), metadata (Message ID: 16 bytes, Sender ID: 16 bytes, Conversation ID: 16 bytes, Sequence ID: 8 bytes, Timestamp: 8 bytes, Headers: 32 bytes) ≈ 256 bytes average.

Bandwidth Sizing:

Peak Ingestion Bandwidth = 4,630,000 msgs/sec * 256 bytes ≈ 1.18 Gigabytes/second (9.44 Gbps)

2. SIZING THE EPOLL WEBSOCKET GATEWAY FLEET

Concurrent Connected Sockets: 50,000,000 persistent TCP connections.

Kernel Socket Memory Tuning:

By default, Linux allocates 128 KB per socket (rmem/wmem).

We tune the Linux kernel via sysctl:

net.ipv4.tcp_rmem = 4096 4096 16384
net.ipv4.tcp_wmem = 4096 4096 16384

Tuning caps socket memory overhead at 8 Kilobytes per connection (4 KB read, 4 KB write).

RAM Required for 50,000,000 Sockets:

Total Socket Memory = 50,000,000 * 8 KB ≈ 400 Gigabytes of RAM fleet-wide

Gateway Node Hardware Profile: High-memory network instances (e.g., 16 vCPUs, 64 GB RAM, 25 Gbps NICs). Each gateway node safely holds 100,000 active concurrent WebSocket connections (~800 MB socket RAM + 2 GB application heap for Go/Netpoll runtime ≈ ~3 GB RAM per node, operating at <10% host memory).

Gateway Fleet Sizing:

Gateway Hosts Required = 50,000,000 sockets / 100,000 sockets/node = 500 Gateway Nodes

Deploying 500 gateway nodes across 3 Availability Zones provides massive redundancy, handling 100k conns/node with comfortable CPU headroom.

3. MESSAGE STORAGE SIZING (SCYLLADB / WIDE-COLUMN STORE)

Daily Storage Ingestion:

Daily Raw Data = 100 Billion messages * 256 bytes = 25.6 Terabytes / day

With Replication Factor (RF = 3): 25.6 TB * 3 ≈ 76.8 Terabytes / day!

30-Day Hot Storage Retention:

30-Day Active Message Volume = 76.8 TB/day * 30 days ≈ 2.3 Petabytes

Storage requires a distributed wide-column NoSQL database (ScyllaDB / Apache Cassandra) running on NVMe SSD storage nodes. Segments older than 30 days are compressed and tiered to cold cloud object storage (S3).

4. CONNECTION REGISTRY WORKING SET (REDIS)

The system must route incoming messages to the specific gateway node holding the recipient's open socket.

Redis Key-Value Mapping:

Key: conn:user:<user_id> -> Value: gateway_pod_id:socket_fd (e.g., "gw-node-142:99182").

50,000,000 active keys * 64 bytes per entry ≈ 3.2 Gigabytes of RAM. A 3-node Redis cluster with 16 GB RAM easily holds the entire global connection routing table in memory!

5. THE TWO GENERALS' PROBLEM & CELLULAR DELIVERY REALITIES

Why is "exactly-once" delivery impossible across mobile networks?

In cellular topologies, client acknowledgment packets (ACKs) are routinely lost when smartphones traverse tunnels or switch cell towers. If the server retries sending after an ACK loss, the recipient client receives a duplicate message. If the server does not retry, the message might have been lost before reaching the client screen.

The Staff Design Invariant:

We reject the fiction of network-level exactly-once delivery. Instead, we enforce At-Least-Once Transport + Client-Side Idempotency (UUIDv7 tokens) + Monotonic Sequence Deduplication. If a client receives a message with an already processed sequence_id, it drops the duplicate payload in RAM, writes nothing to disk, and immediately re-emits the Delivery ACK to silence server retries!

Key System Sizing Numbers Worth Memorizing

3. End-to-End System Architecture

The architecture cleanly decouples the stateful connection edge from the stateless business logic and the distributed wide-column storage tier:

Figure 11.1: End-to-End Real-Time Collaborative Chat & Presence Architecture

Architectural Components in Depth

COMPONENT 1: EPOLL WEBSOCKET GATEWAY FLEET

The high-density stateful network termination tier:

• Kernel-Tuned Persistent Connections: A horizontally scaled cluster of 500 nodes running non-blocking Linux epoll event loops (Go Netpoll or C++ epoll). Tuned kernel socket parameters (rmem_default = 4096, wmem_default = 4096) drop memory consumption from the Linux default 128 KB down to 8 KB per idle socket, allowing each 64 GB gateway host to easily maintain 100,000 live WebSocket connections.
• TLS Handshake & JWT Offloading: Terminates WSS (WebSocket Secure) connections, verifies OAuth2/JWT tokens, and extracts caller identity.
• Zero Business Logic Invariant: Contains zero application business logic or database queries. The gateway strictly translates binary WebSocket frames into internal gRPC calls and streams outbound message envelopes down the physical TCP file descriptor.
• Graceful Connection Draining: During Kubernetes cluster rolling updates, the gateway stops accepting new connections and sends a WebSocket 1001 Going Away control frame, causing client daemons to reconnect smoothly over a 60-second jittered window without storming the fleet.

COMPONENT 2: CONNECTION REGISTRY TIER (DISTRIBUTED REDIS CLUSTER)

The low-latency routing directory that maps active users to their specific physical gateway pod:

• In-Memory Routing Directory: Redis Cluster holding key-value records: conn:user:<user_id> -> {"gateway_pod": "gw-pod-142", "socket_fd": 9821}.
• Sub-Millisecond Routing SLA: Resolves recipient socket locations in under 1.2ms across the internal VPC network.
• Heartbeat Leasing with Grace Threshold: Clients emit periodic 30-second heartbeat pings over WebSocket. The gateway extends the Redis key with a 60-second TTL. If a client disconnects cleanly, the gateway immediately purges the key; if a device drops power abruptly, the lease expires naturally within 60 seconds.
• Surge-Protected Multi-Master Sharding: Registry keys are sharded across Redis nodes by hash(user_id) % num_nodes, preventing hot-node CPU bottlenecks when millions of users ping simultaneously.

COMPONENT 3: MESSAGE SEQUENCER & STATELESS ROUTER SERVICE

The transactional orchestration brain that guarantees total causal message order:

• Conversation-Scoped Monotonic Sequencing: When User A sends a message to User B, the router claims an atomic, strictly increasing 64-bit sequence ID scoped to that conversation: sequence_id = Redis.INCRBY("seq:conv:" + conv_id, 1) This guarantees that all conversation participants observe identical message ordering, even if packets traverse different mobile internet paths.
• Client Idempotency Deduplication: Checks incoming client_msg_id (UUIDv7 generated by client) against a 5-minute bloom filter/cache. If a mobile network retry resubmits the same message, the router returns the existing ACK without re-writing to database or re-routing to recipient.
• Dual-Path Routing Dispatch: Commits the message to ScyllaDB and immediately queries the Connection Registry:
  • Online Recipient: Emits a direct internal gRPC call to the recipient's active gateway pod for sub-20ms delivery.
  • Offline Recipient: Emits an event to the Push Notification Gateway (APNs / FCM) to trigger a native mobile banner alert.

COMPONENT 4: DISTRIBUTED MESSAGE STORE (SCYLLADB / CASSANDRA)

The mission-critical persistent system of record optimized for heavy write throughput:

• Append-Only Wide-Column LSM Engine: ScyllaDB (C++ reimplementation of Cassandra) utilizing non-blocking Seastar reactor thread-per-core architecture. Eliminates JVM garbage collection pauses completely.
• Composite Partition Key Layout: Messages are partitioned by (conversation_id, bucket_month) and clustered in descending or ascending chronological order by sequence_id: PRIMARY KEY ((conversation_id, bucket_month), sequence_id) WITH CLUSTERING ORDER BY (sequence_id ASC);
• Sequential Disk Writes: Because new messages append sequentially to commit logs and SSTables in memory without in-place row updating, the storage engine sustains over 4,000,000 write ops/second at sub-2ms latency.

COMPONENT 5: SCOPED EPHEMERAL PRESENCE ENGINE

The high-frequency status tracking tier that avoids network broadcast storms:

• Decoupled Heartbeat State: Stores user presence states (ONLINE, AWAY, OFFLINE, last_seen_ts) in Redis Hashes.
• Pull-on-Demand Contact Discovery: When a user opens their chat sidebar, the client requests status for the visible 20 contacts via a single batched HMGET query.
• Ephemeral 1-to-1 Subscriptions: Real-time presence updates are never broadcast globally. A live Redis Pub/Sub subscription channel (presence:user:B) is established strictly while User A has an active chat window open with User B. When User A closes the window, the subscription is torn down, eliminating billions of wasted packets.

COMPONENT 6: HYBRID GROUP FAN-OUT ENGINE

The routing engine governing group messaging dynamics:

• Small Groups (< 256 members): Implements Fan-Out on Write. The engine copies the message pointer into each member's private inbox timeline. When a member opens the app, loading their inbox is an ultra-fast (O(1)) single-key read.
• Large Channels / Broadcasts (> 256 members): Implements Fan-Out on Read. The message is appended once to the shared channel timeline. Gateway nodes fan out the message in memory directly to connected local WebSocket file descriptors for that channel.
• Mathematical Inflection Optimization: Eliminates write amplification: sending a message to a 100,000-member community channel executes 1 write instead of 100,000 writes!

COMPONENT 7: END-TO-END KEY INFRASTRUCTURE (SIGNAL PROTOCOL)

The zero-knowledge cryptographic security tier:

• Cryptographic Pre-Key Directory: Stores public Identity Keys, Signed Pre-Keys, and a pool of One-Time Pre-Keys for each registered user.
• Signal Double Ratchet Implementation: Clients exchange cryptographic keys during initial session setup via Extended Triple Diffie-Hellman (X3DH). For every message transmitted, the ratchet advances the cryptographic key, ensuring Forward Secrecy (past messages cannot be decrypted if the current key is stolen) and Break-in Recovery.
• Untrusted Server Invariant: The entire server infrastructure (gateways, routers, databases, caches) handles only opaque, ciphertext payloads. The server cannot inspect message bodies, attachments, or conversation titles.

ByteByteGo-Style Step-by-Step 1-to-1 Chat & Delivery ACK Flow

To ace the Staff interview, trace a 1-to-1 chat message through its complete bidirectional three-tick lifecycle:

sequenceDiagram
    participant A as "Sender (User A)"
    participant G14 as "WebSocket Gateway 14"
    participant S as "Message Sequencer & Router"
    participant DB as "ScyllaDB"
    participant R as "Connection Registry"
    participant G88 as "WebSocket Gateway 88"
    participant B as "Recipient (User B)"
    Note over A,B: STEP-BY-STEP 1-TO-1 CHAT & THREE-TICK ACK MESSAGE FLOW
    Note over A,DB: MESSAGE DISPATCH & SERVER PERSISTENCE
    A->>G14: (1) WSS Frame: {client_msg_id, text, conv_id}
    G14->>S: (2) gRPC
    Note over S: (3) Assigns atomic monotonic seq_id = 42
    S->>DB: (4) Commits to ScyllaDB (Quorum 2/3)
    S-->>A: (5) Server ACK emitted to User A (Sender gets Single Gray Tick)
    Note over S,B: RECIPIENT ROUTING & DELIVERY ACK
    S->>R: (6) Queries Connection Registry: "User B is on GW-88"
    S->>G88: dispatch
    G88->>B: (7) Pushes message down socket to Recipient (User B)
    B->>G88: (8) Writes to local SQLite DB#59; emits Delivery ACK
    G88->>G14: (9) Routes Delivery ACK to Gateway 14
    G14-->>A: Sender gets Double Gray Tick
    Note over A,B: READ RECEIPT NOTIFICATION
    B->>G88: (10) User B opens screen, emits Read ACK {last_read_seq: 42}
    G88-->>A: (Step 11) Sender gets Double Blue Tick

STEP 1: CLIENT INGRESS FRAME

User A composes a message and the client device transmits a compact binary Protocol Buffer frame over the active WebSocket (WSS) connection to its assigned edge node, Gateway 14: { client_msg_id: "m_9021", conv_id: "c_401", payload_ciphertext: "...", timestamp: 1718002000 }.

STEP 2: INTERNAL GATEWAY GRPC RELAY

Gateway 14 verifies the client session token, attaches the sender identity (sender_id: "u_101"), and forwards the payload via a non-blocking internal gRPC call to the Message Sequencer & Router Service.

STEP 3: MONOTONIC SEQUENCE ID CLAIM

The Message Sequencer atomically claims a strictly incrementing, conversation-scoped 64-bit sequence identifier via an in-memory Redis script: sequence_id = INCR seq:conv:c_401 (e.g., returning 42). This assigns an immutable causal order to the message.

STEP 4: DURABLE WIDE-COLUMN STORAGE COMMIT

The sequencer commits the message record into the ScyllaDB wide-column cluster using Quorum write consistency (LOCAL_QUORUM = 2 of 3 replicas). Because ScyllaDB uses an append-only LSM engine, this write finishes in < 2ms without table lock contention.

STEP 5: SERVER ACK & SINGLE GRAY TICK

With durability guaranteed in storage, the sequencer emits a lightweight ServerACK back to Gateway 14, which pushes it down User A's WebSocket. User A's mobile interface immediately renders the Single Gray Tick ("Message stored on server").

STEP 6: ACTIVE CONNECTION REGISTRY RESOLUTION

The router queries the distributed Redis Connection Registry for User B's current online location: GET conn:user:u_202. The registry returns { "gateway_pod": "gw-pod-88", "socket_fd": 4092 }.

STEP 7: EDGE SOCKET PUSH TO RECIPIENT

The sequencer issues a gRPC dispatch call to Gateway 88. Gateway 88 serializes the message frame and pushes it down the physical TCP socket directly to User B's mobile device.

STEP 8: CLIENT SQLITE COMMIT & DELIVERY ACK

User B's background client daemon receives the frame, checks its local SQLite database to confirm sequence_id: 42 has not been written yet, persists the message, and automatically transmits a DeliveryACK over its WebSocket back to Gateway 88.

STEP 9: DELIVERY RELAY & DOUBLE GRAY TICK

Gateway 88 routes the DeliveryACK back through the internal message bus to Gateway 14. Gateway 14 emits the event to User A, whose chat UI updates the status to a Double Gray Tick ("Delivered to recipient hardware").

STEP 10: USER B CHAT SCREEN OPEN & READ RECEIPT INGRESS

User B unlocks their phone, taps the push alert, and opens the conversation screen. The client records that messages up to sequence 42 have entered the viewport, transmitting a ReadReceiptACK: { conv_id: "c_401", last_read_seq: 42 }.

STEP 11: READ RECEIPT SYNCHRONIZATION & DOUBLE BLUE TICK

The receipt is committed to ScyllaDB's conversation read watermark and relayed to Gateway 14. User A's mobile screen transitions the ticks from gray to the Double Blue Tick ("Read by recipient"), completing the three-tick lifecycle.