Design a Chat System

A chat system delivers messages between users in near real time, for one-to-one conversations and groups. The design must keep long-lived connections to millions of clients, route each message to the servers holding the recipients' connections, persist history, and preserve message order within a conversation.

Requirements

Functional

One-to-one and group chat (up to a few hundred members).
Text messages, and attachments referenced by URL.
Delivery status: sent, delivered, read.
Online presence indicator.
Message history, synchronized across a user's devices.
Push notifications when the recipient is offline.

Non-functional

Low latency: delivery to an online recipient in well under a second.
Ordering: messages in a conversation appear in the same order for every participant.
Durability: an accepted message is never lost.
Availability: tolerate server failures; clients reconnect automatically.
Scale: hundreds of millions of users, tens of millions of concurrent connections.

Estimation

Assume 50 million DAU, each sending 40 messages per day.

Messages: 50e6 x 40 = 2e9/day, or about 23,000/s average; peak 3x is about 70,000/s.
Storage: assume 200 bytes per message including metadata: 2e9 x 200 B = 400 GB/day, about 146 TB/year before replication.
Concurrent connections: if 20% of DAU are connected at peak, 10 million connections. A tuned server can hold on the order of 100,000 idle WebSocket connections, so about 100 chat servers, more for headroom and CPU load.
Read amplification: a group message with 100 members generates 100 deliveries, so delivery rate is higher than message rate.

What the numbers imply

70,000 messages per second at peak with about 3 recipients on average is roughly 200,000 deliveries per second, so delivery routing, not storage writes, is the heavier path; the message store sees 70,000 appends per second, which needs a partitioned LSM store rather than a single relational primary.
400 GB per day at replication factor 3 is 1.2 TB per day, so tiering old data to cheaper storage matters after weeks.
10 million connections at about 10 to 20 KB each is 1 to 2 GB of memory per server at 100,000 connections. Heartbeats at 30 seconds are 330,000 per second; do not turn each one into a Redis write. Use TCP or WebSocket ping handled at the chat server, and refresh the registry TTL on connect and every few minutes.

API design

Clients use a persistent WebSocket for real-time traffic (see WebSockets, SSE and long polling) and HTTP for everything else.

WebSocket frames (JSON or Protobuf):

Code

client -> server: {"type":"send","conv_id":"c1","client_msg_id":"uuid","body":"hi"}
server -> client: {"type":"ack","client_msg_id":"uuid","msg_id":"...","seq":42}
server -> client: {"type":"message","conv_id":"c1","msg_id":"...","seq":42,"sender":"u1","body":"hi"}
client -> server: {"type":"receipt","conv_id":"c1","up_to_seq":42,"status":"read"}

HTTP endpoints:

Code

GET  /v1/conversations                          list conversations
GET  /v1/conversations/{id}/messages?before=<seq>&limit=50   history
POST /v1/conversations                          create group
POST /v1/attachments                            get upload URL

client_msg_id is generated by the client so that a retry after a lost ack does not create a duplicate message; the server deduplicates on it.

Data model

Access pattern: fetch recent messages of one conversation in order, and append. That matches a wide-column or key-value store partitioned by conversation (Cassandra, HBase, DynamoDB style; see wide-column stores).

Table	Partition key	Clustering / sort key	Fields
`messages`	`conv_id` (or `conv_id, time bucket` for very long conversations)	`seq` descending	msg_id, sender_id, body, type, created_at
`conversations`	`conv_id`	-	type, name, created_at
`members`	`conv_id`	`user_id`	role, last_read_seq
`user_conversations`	`user_id`	`last_activity` descending	conv_id, unread_count
`sessions`	`user_id`	`device_id`	chat_server_id, last_heartbeat (stored in Redis with TTL)

A relational database would need heavy sharding for this write volume, and the queries never join across conversations.

High-level design

flowchart LR
    C1["Client A"] <--> |WebSocket| CS1["Chat server 1"]
    C2["Client B"] <--> |WebSocket| CS2["Chat server 2"]
    CS1 --> MQ["Message bus"]
    CS2 --> MQ
    MQ --> CS1
    MQ --> CS2
    CS1 --> Seq["Sequencer / message service"]
    Seq --> Store[("Message store")]
    CS1 --> Sess[("Session registry (Redis)")]
    CS2 --> Sess
    Seq --> Push["Notification service"]
    Presence["Presence service"] --> Sess
    C1 --> API["HTTP API servers"]
    API --> Store
    API --> Groups[("Conversation and member DB")]

Component responsibilities:

Chat servers: terminate WebSocket connections, authenticate, forward incoming messages, and push outgoing messages to connected clients. They hold connection state but no durable data.
Session registry: maps each user_id and device to the chat server holding its connection.
Message service: assigns the message ID and sequence number, persists the message, and triggers delivery.
Message bus: carries messages from the service to the chat servers that hold recipients (per-server topics or channels).
HTTP API servers: stateless; history, conversation management, and attachment upload URLs.
Notification service: sends mobile push for offline recipients (see notification system).
Presence service: tracks online status.

One-to-one message flow:

sequenceDiagram
    participant A as Client A
    participant S1 as Chat server 1
    participant M as Message service
    participant DB as Message store
    participant R as Session registry
    participant S2 as Chat server 2
    participant B as Client B
    A->>S1: send (client_msg_id)
    S1->>M: forward message
    M->>M: dedup, assign seq
    M->>DB: persist
    M-->>S1: ack (msg_id, seq)
    S1-->>A: ack
    M->>R: where is B?
    R-->>M: chat server 2
    M->>S2: deliver via bus
    S2->>B: message
    B-->>S2: receipt delivered

If B has no live session, the message stays in the store, and a push notification is sent. When B reconnects it asks for messages after its last seen seq.

Deep dive

Message ordering and IDs

Ordering matters per conversation, not globally. Using timestamps from clients or servers is unsafe because of clock skew. Options:

Per-conversation sequence number: the message service assigns seq as the next integer for the conversation, so ordering is total and gaps are detectable (a client that receives seq 5 then 7 knows to fetch 6). It requires a single writer per conversation: route all messages of one conversation to the same shard or partition (partition a log by conv_id, see Kafka internals) so a single consumer sequences them.
Time-sortable global IDs (Snowflake-style, see unique ID generator): easy to generate anywhere, but ordering is only approximate under clock skew between generators.

Deduplication works by recording (conv_id, client_msg_id) with the assigned seq when the message is first persisted. A retry with the same pair finds the existing record and returns the original ack instead of assigning a new seq.

Per-conversation sequencing is the stronger choice for chat, because gap detection also drives sync. The cost is that a hot group serializes on one partition; group sizes are limited, so rate per conversation remains manageable. If the sequencer for a partition fails, another node takes over that partition and resumes from the highest persisted seq of each conversation, so the counter never moves backwards.

Routing messages between chat servers

The sender and recipient are usually on different chat servers. Options:

Session registry plus direct server-to-server: look up the recipient's server in Redis and send to that server. Efficient but needs an inter-server channel.
Pub/sub per user or per server: every chat server subscribes to a topic for itself (or for each connected user) on a message bus, and the message service publishes to the recipient's server topic (see message queues vs pub/sub).

Group messages: the message service fetches the member list, looks up each member's server, and publishes once per server that has at least one member (not once per member). For large groups, fan-out is done asynchronously by workers.

Registry entries carry a TTL refreshed by client heartbeats, so a crashed chat server's entries expire.

Delivery guarantees and receipts

Use at-least-once delivery with idempotent handling:

The client resends unacknowledged messages with the same client_msg_id; the server dedups.
The server keeps a per-device cursor (last_delivered_seq). On reconnect, the client sends its last seen seq per conversation and receives the missing messages.
Delivered receipt: client reports the message reached the device. Read receipt: client reports up_to_seq read, which is stored as members.last_read_seq, one small update per read event instead of one per message. Group read status is derived from all members' cursors.

Concrete settings and consistency. Store messages with quorum writes (for example LOCAL_QUORUM) so an acknowledged message survives a node loss; reads for history use the same or LOCAL_ONE for speed, since seq gap detection catches stale reads. Keep partitions bounded: about 200 bytes per message means a partition of 500,000 messages is 100 MB, so long-lived busy conversations get a time bucket (for example per month) in the partition key. Sequence assignment alternatives: a Redis INCR per conversation is fast but can regress after failover, so recover from the highest persisted seq; a partitioned log with one consumer per partition gives the same guarantee without a counter. Registry: TTL of 60 seconds refreshed roughly every 20 to 30 seconds; presence TTL about 30 seconds. Alternative routing: consistent-hash users to chat servers so no registry is needed, at the cost of moving connections when servers are added.

Presence

Each client sends a heartbeat (for example every 5 to 30 seconds) over its connection. The presence service stores last_seen with a TTL in Redis; a user is online while the key exists. A missed heartbeat window marks the user offline, which tolerates brief disconnects. Broadcasting every status change to all friends is expensive, so status is fetched on demand when a chat is opened, or pushed only to friends currently viewing the user's chat, not to all contacts.

Multiple devices

Each device holds its own connection and its own cursor. The message service delivers to all of a user's registered sessions; history sync uses seq per conversation, so a new device downloads history through the HTTP API.

Attachments

Files are uploaded directly to object storage with a pre-signed URL and served through a CDN. The message carries only the attachment reference.

Bottlenecks and scaling

Connections: chat servers are stateful in their connections but not their data. Scale by adding servers behind a layer-4 load balancer; clients reconnect with jittered backoff after failures, to avoid a reconnect storm (see load balancing).
Message store: partition by conversation, replicate across nodes and availability zones. The write path is append-only, so LSM-based stores perform well.
Session registry: Redis cluster sharded by user ID.
Hot groups: cap group size; for very large channels, switch to a broadcast model with fan-out on read.
Server failure: connected clients reconnect elsewhere and sync from their last seq; no message is lost since the store is the source of truth.
Cross-region: route users to the nearest region; a conversation spanning regions needs replicated storage and a consistent sequencer per conversation.

Failure modes

Chat server dies. With 100,000 connections per server, 100,000 clients reconnect. With jittered backoff spread over 30 to 60 seconds, that is about 2,000 to 3,000 reconnects per second across the fleet, which is manageable; without jitter it is a stampede on authentication and the registry. No message is lost; clients resync from their last seq.
Session registry down. Delivery cannot find recipient servers; fall back to storing and pushing notifications, and clients pull on next connect. Degraded, not lossy.
Message bus or sequencer partition down. Sends for affected conversations fail or queue at the client (with client_msg_id retries); other conversations continue. Failover resumes from the highest persisted seq.
Store node loss. Quorum writes tolerate one replica down; hinted handoff and repair restore it.
At 10x (500 million DAU, 100 million connections). About 1,000 chat servers, registry sharded across many nodes, cross-region routing, and hot large groups moved to fan-out on read. Presence traffic is the first thing to become unaffordable, so it moves to on-demand lookup.

Trade-offs and follow-up questions

WebSocket vs long polling: WebSocket gives lower latency and overhead, but needs sticky connection state and infrastructure that supports upgrades.
End-to-end encryption: the server stores ciphertext only, so server-side search, moderation, and history backfill become client responsibilities, and group key management is needed.
How to store history forever cheaply? Move old partitions to cheaper storage tiers; keep recent messages hot.
How to handle message edits and deletes? Store them as new events referencing the original msg_id and apply them at read time.
How to guarantee order when a sender is offline and sends later? Ordering is by server-assigned seq at arrival, not by client send time.
What if a user sends 100 messages while offline-flaky? The client queues them with client_msg_ids; the server dedups and assigns seq in arrival order.
What if two devices read at once? Read cursors are monotonic (max(up_to_seq)), so concurrent updates converge.
What if a group has 100,000 members? Do not fan out per member on write; store once and let members pull by seq, with push notifications as a hint.
Why not deliver directly between chat servers without a bus? Possible with the registry, but the bus gives buffering and retries when a server is slow.