Chat and Presence: Interview Calibration, Opinions, and Runbooks

This lesson covers sections 7 to 10 and the chapter checklist of Chapter 11 and continues Chat and Presence: Sequencing, Fan-Out, and Deep Dives.

7. Staff Interview Calibration: Five Probing Questions

Probe 1: "Why not use timestamps for message ordering?"

Senior Trap: "We use server-side timestamps to order messages because the server clock is synchronized via NTP."

Staff Answer: "Server timestamps fail because multiple concurrent messages sent to the same group arrive at different times across different gateway pods, and network jitter can invert their arrival sequence. Furthermore, two messages arriving within the same millisecond collide. We enforce strict ordering by assigning a monotonically increasing 64-bit sequence integer scoped to each conversation (sequence_id). The sequence ID is assigned atomically at write time before persisting to ScyllaDB. Clients use a reorder buffer to hold out-of-order sequence packets, ensuring deterministic transcript rendering regardless of network latency or NTP clock drift."

Probe 2: "User B is offline. How does the system deliver the message, and what happens when they reconnect with a new phone?"

Senior Trap: "We store the message in Redis until they reconnect, or send a push notification containing the message text."

Staff Answer: "Messages are never stored temporarily in volatile Redis; they are committed to persistent wide-column storage (ScyllaDB) immediately. In an E2E encrypted system, push notifications (APNs/FCM) contain zero message content—only a generic wake-up payload ('You have a new message'), because the server cannot read the ciphertext and Apple/Google push networks must be treated as untrusted relays. When User B reconnects with a new phone, their new device does not possess the historical ratchet keys. To restore history, the client must either execute a local peer-to-peer Wi-Fi transfer from their old device or restore from a user-held password-encrypted cloud backup."

Probe 3: "A group chat has 50,000 members. Do you fan out on write or fan out on read?"

Senior Trap: "We fan out on write because read speed is what matters to users."

Staff Answer: "Fan-out on write for 50,000 members creates a massive 50,000x write amplification that crashes storage engines. We enforce a hybrid fan-out model with a 256-member threshold. Below 256 members, we fan out on write into recipient inboxes for instant O(1) reads. Above 256 members (e.g., Discord or public broadcast channels), we switch to fan-out on read: the message is written once to a shared channel timeline. When dispatched, gateway nodes receive a single copy over gRPC and fan out the payload in memory to connected client sockets, eliminating millions of redundant database writes."

Probe 4: "How do you track online presence for 50 million concurrent users without triggering 10 billion broadcast events per minute?"

Senior Trap: "We publish presence status changes to a Kafka topic and let microservices fan it out."

Staff Answer: "Broadcasting presence to 200 friends at 50M users is an O(N M) catastrophe that generates 10 Billion packets per wave. We solve this by implementing Scoped Presence: presence is never broadcast globally. For contact lists and friend sidebars, status is pulled lazily via batch MGET in Redis only when the user opens the application. Live real-time status updates and typing indicators are published to an ephemeral Pub/Sub channel only while an active 1-to-1 chat window is open, and unsubscribed immediately upon closing. This slashes presence network traffic by over 99.8%."

Probe 5: "Product wants full-text message search, but security mandates End-to-End encryption. How do you resolve this conflict?"

Senior Trap: "We can decrypt the message on the server, index it in Elasticsearch, and delete the plaintext."

Staff Answer: "Decrypting on the server completely invalidates End-to-End Encryption; if the server can decrypt text for indexing, any rogue engineer, subpoena, or server compromise exposes all user communications. E2E encryption is a zero-knowledge commitment: the server must never see plaintext. We resolve this by moving full-text search entirely to the client device. The mobile client indexes decrypted messages locally using embedded SQLite FTS5. Search queries execute instantly in local device flash storage with zero server communication, preserving 100% cryptographic privacy."

8. Staff Insiders: Controversial Opinions

8.1 "Store-and-Forward Is the System; WebSockets Are Merely a Latency Optimization"

Junior engineers treat WebSockets as the core of chat architecture, designing message pipelines around open sockets. Staff engineers recognize that mobile connections are fundamentally unreliable. The true core of chat is the offline store-and-forward persistence engine. If every WebSocket server in the world crashes simultaneously, zero messages should be lost. The system degrades to delayed delivery; when gateways recover, clients sync their monotonic offsets and catch up cleanly.

8.2 "Global Message Ordering Is a Fallacy That Destroys Concurrency"

Attempting to establish a single global ordering across an entire messaging company is an architectural mistake that caps throughput and creates single-point-of-failure bottlenecks. Causality and ordering are only meaningful within the boundary of a single conversation. Scope your sequencers to conversation IDs; let independent conversations execute completely in parallel across thousands of database partitions.

8.3 "End-to-End Encryption Is a Product Commitment, Not a Security Checkbox"

Engineers frequently pitch E2E encryption as a simple security upgrade. In reality, E2E encryption permanently destroys server-side search, spam filtering, automated moderation, and cloud backups. If product executives do not explicitly sign off on losing these capabilities, do not build E2E encryption.

8.4 "Broadcasting Presence Status Is Architectural Suicide at Scale"

Pushing online status changes to friend lists is the most common reason chat backends melt down during peak hours. Users do not need millisecond-level precision on whether their 300 contacts are online. Pull presence lazily on app launch; reserve real-time pub/sub streams strictly for the active conversation window.

8.5 "Exactly-Once Message Delivery across Mobile Networks Does Not Exist"

The Two Generals' Problem and mobile cellular handoffs prove that guaranteed exactly-once delivery over the network is physically impossible. An ACK packet can always be dropped by a cell tower. Strive for at-least-once delivery with client-side deduplication: clients assign UUIDv7 idempotency tokens, the server deduplicates at write time, and recipient clients filter duplicate sequence numbers.

8.6 "Building a Custom Chat Protocol over UDP Is Premature Optimization"

Unless you are building voice/video streaming or ultra-low-latency financial trading, do not invent custom binary UDP protocols. Standard WebSocket over TLS with binary Protobuf encoding provides sub-50ms latency, traverses enterprise corporate firewalls without getting blocked, and benefits from decades of OS-level TCP congestion control tuning.

9. The Architectural Fault Lines & Default Staff Positions

Decision	Fault Line	Who Pays	The Real Operational Cost
Store-and-Forward vs Real-Time Pub/Sub	Durability vs Transport Simplicity	Database Disk I/O	Committing every message to ScyllaDB before dispatch adds ~2ms latency to writes.
Monotonic Sequencing vs Server Timestamps	Correct Ordering vs Implementation Speed	Sequencer	Maintaining atomic per-conversation counters in Redis requires distributed coordination.
Hybrid Fan-Out vs Pure Inbox Fan-Out	Code Complexity vs Write Amplification	Storage	System must maintain two code paths: inbox writes (<256) and timeline queries (>256).
Scoped Presence vs Global Broadcast	Instant Status Freshness vs Network Bandwidth	Presence Latency	Contact sidebar shows static presence cached up to 60 seconds old to save 99% bandwidth.
End-to-End Encryption vs Plaintext	User Cryptographic Privacy vs Server Features	Product Functionality	Eliminates server search, spam filtering, compliance audits, and web history sync.

DEFAULT STAFF POSITIONS

1. Storage & Delivery Contract: Default: Store-and-Forward to ScyllaDB first; WebSocket push is optimization. When to deviate: Ephemeral typing indicators and live audio streams only.

2. Message Sequencing: Default: Monotonic 64-bit integer per conversation; client reorder buffer. When to deviate: Never use client timestamps or un-sequenced UUIDs.

3. Group Fan-Out Architecture: Default: Hybrid fan-out with a hard 256-member threshold boundary. When to deviate: Pure read-time fan-out for public broadcast networks.

4. Presence Management: Default: Scoped presence (Lazy MGET for sidebars, ephemeral pub/sub for chat). When to deviate: Never broadcast presence to global contact graphs.

5. Encryption & Search: Default: Client-side SQLite FTS5 for E2E; Elasticsearch for enterprise chat. When to deviate: Strictly dictated by consumer privacy vs enterprise compliance

10. Failure Modes, 3 AM Incident Runbooks & Active Crisis Drills

Incident 1: WebSocket Gateway Crash & 500,000-Socket Thundering Herd

The Failure: Gateway Node 42 suffers hardware kernel panic, abruptly severing 100,000 active WebSocket connections. All 100,000 mobile clients immediately attempt to reconnect simultaneously.

The Threat: The sudden reconnection storm slams surviving gateway nodes with TLS handshakes, exhausting CPU and triggering cascading gateway crashes across the entire fleet.

Runbook Action:

1. Verify client reconnection backoff: ensure mobile clients enforce Exponential Backoff with Full Jitter (T = min(60s, 2^n * rand(0.5, 1.5))).
2. Engage edge connection rate limiters at AWS ALB / Envoy ingress proxies to rate-shape inbound TLS handshakes to 5,000 handshakes/second per gateway.
3. Purge orphaned connection keys: run background script to evict stale Redis entries pointing to gw-node-42: redis-cli --eval evict_gateway.lua , gw-node-42
4. Scale gateway deployment by 20% to absorb reconnecting sockets smoothly.

Incident 2: ScyllaDB Hot Partition Mutation Freeze

The Failure: A viral celebrity creates a massive group chat with 10,000 members. High-frequency messaging concentrates millions of mutations onto a single partition key (conv_viral_id, '2026-09'), exceeding ScyllaDB partition limits (100 MB per partition recommended).

The Threat: ScyllaDB node hosting that partition experiences memory compaction lockups and 5-second read timeouts.

Runbook Action:

1. Identify the hot partition: check nodetool tablestats and inspect large partition warnings.
2. Implement Dynamic Sub-Partition Bucketing: Transition the conversation partition key from monthly to hourly buckets: (conversation_id, bucket_day_hour).
3. Evict the group from Write-Time Fan-Out: force the group into Read-Time Shared Timeline Fan-Out mode immediately via feature flag: chat-admin --set-fanout-mode read-time --conv-id conv_viral_id

Incident 3: Redis Pub/Sub Message Loss During Cluster Failover

The Failure: Primary Redis node hosting real-time message routing fails. Sentinel promotes a replica. Because Redis replication is asynchronous, messages published during the 3-second failover window are dropped.

The Threat: Users in active chats stop receiving live messages, believing their friends have gone silent.

Runbook Action:

1. The client-side gap detection protocol catches the failure automatically!
2. When the next message arrives with a non-contiguous sequence ID, the client detects the missing sequence and issues a sync request to ScyllaDB.
3. ScyllaDB serves the missing messages from disk; zero data loss occurs.
4. SRE action: verify Redis Sentinel quorum health and monitor replica lag metrics (master_repl_offset).

Crisis Drill 1: Split-Brain Connection Registry Ghost Sockets

The Scenario: A mobile client abruptly switches from cellular 5G to home Wi-Fi. The new connection establishes on Gateway 99, but Gateway 12 still believes the old socket is open, creating two conflicting Redis entries.

The Mitigation Protocol:

1. Every connection handshake generates a monotonic Connection Epoch Token.
2. When Gateway 99 writes to Redis, it executes an atomic CAS operation: HSET conn:user:B gateway_id "gw-99" epoch 12
3. If Gateway 12 attempts to deliver a message using Epoch 11, the router rejects it, forcing Gateway 12 to close the orphaned zombie socket immediately.

Crisis Drill 2: E2E Pre-Key Bundle Exhaustion Attack

The Scenario: A malicious bot script initiates 100,000 empty chat sessions with target User B, consuming all 100 of User B's uploaded One-Time Pre-Keys in seconds. Subsequent legitimate users cannot establish an encrypted session.

The Mitigation Protocol:

1. The Pre-Key service enforces rate limits: at most 5 Pre-Key bundle fetches per sender per hour.
2. When User B's available One-Time Pre-Key count drops below 20, the server emits a Silent Background Push to User B's mobile app: {"action": "REFILL_PREKEYS"}.
3. User B's device wakes, generates 100 fresh Pre-Keys in the background, and uploads them to the server, neutralizing the exhaustion attack.

Crisis Drill 3: Viral Celebrity Channel Write-Amplification Meltdown

The Scenario: An administrator accidentally sets a 50,000-member community channel to "Write-Time Fan-Out" mode. A user posts a message, triggering 50,000 concurrent database write tasks that spike queue depth past 1,000,000.

The Mitigation Protocol:

1. The ingestion validator enforces a Hard Architectural Guardrail:

   if group.member_count > 256 and group.fanout_mode == "WRITE":
       logger.warn("Illegal fanout configuration! Forcing read-time timeline mode.")
       group.fanout_mode = "READ"

2. The code path automatically diverts the message to the single-write shared channel timeline, preventing catastrophic write amplification before any background tasks are spawned.

Chapter Summary Checklist

STAFF CHECKLIST: Real-Time Collaborative Chat & Presence

When designing or defending a real-time messaging system in an L6 loop:

• Store-and-Forward Foundation: Did you verify that every message is persisted to durable storage before any WebSocket push is attempted?
• Monotonic Sequencing: Are messages ordered by server-assigned per-conversation sequence IDs rather than unreliable client timestamps?
• Client Gap Detection: Can recipient clients detect missing sequence numbers and fetch missing message deltas automatically?
• Three-Tick Delivery State Machine: Did you model Server ACK (Single Tick), Delivery ACK (Double Gray), and Read Receipt (Double Blue)?
• Hybrid Group Fan-Out: Did you enforce write-time fan-out for small groups (<256) and read-time shared timelines for massive channels?
• Kernel Socket Tuning: Are TCP read/write buffers tuned to 8 KB per socket to support 100,000 connections per gateway host?
• Scoped Presence Model: Did you replace global broadcast presence with lazy MGET sidebars and ephemeral open-chat pub/sub channels?
• End-to-End Encryption Realism: Did you articulate the five server capabilities permanently lost when adopting E2E encryption?
• Media Pipeline Decoupling: Are large media files uploaded directly to S3 via presigned URLs and encrypted client-side?
• Multi-Device Synchronization: Are messages encrypted per target device and tracked via independent per-device sequence pointers?