Design a Chat System

Chat requires the server to push messages to clients in real time. Standard HTTP is client-initiated — the client must ask for messages; the server cannot push unprompted. Three approaches:

Short polling: client asks "any new messages?" every few seconds. Simple but wasteful — most responses are empty. Creates unnecessary server load.

Long polling: client sends a request; server holds it open until a message arrives (or a timeout occurs), then responds. Better than short polling but has drawbacks: stateless servers mean the server holding a user's long-poll connection may not be the server that receives an incoming message for that user, requiring cross-server communication.

WebSocket: client initiates a WebSocket handshake (an HTTP upgrade request). Once established, the connection is bidirectional and persistent — the server can push messages to the client at any time without the client requesting them. A single TCP connection handles all traffic in both directions. WebSocket is the standard for chat applications.

The system has three main service categories:

Stateless services (behind a load balancer): authentication, user profile API, friend management, search. Standard HTTP request-response. Can scale horizontally without concern for connection affinity.

Stateful services: chat servers. Each client maintains a persistent WebSocket connection to one chat server. A client does not switch chat servers unless the current one fails. Service discovery (Zookeeper or Consul) tracks which chat server each user is connected to.

Third-party integration: push notification servers deliver messages to users when they are offline and do not have an active WebSocket connection.

Storage: two types of data. Generic data (user profiles, friend lists, settings) goes in a relational database with replication and sharding. Chat history goes in a key-value store (Facebook uses HBase, Discord uses Cassandra) because chat generates enormous write volume, access patterns favor recent data, and key-value stores offer low latency and horizontal scalability.

1-on-1 chat message table: message_id (primary key), message_from, message_to, content, created_at.

Group chat message table: channel_id (partition key), message_id (clustering key), user_id, content, created_at.

Message ID requirements: IDs must be unique across the system and sortable by time (newer messages have larger IDs). Auto-increment in a single database does not scale. UUID is 128 bits and not time-sortable. The recommended solution: a local sequence number generator within each channel. Messages only need to be ordered within a conversation, not globally. Each channel maintains its own atomic counter (stored in Redis or a database). This is simpler and more efficient than a global Snowflake-style generator for most chat use cases.

User A sends a message to User B:

1. User A's client sends the message to Chat Server 1 (A's WebSocket server). 2. Chat Server 1 assigns a message ID (from the sequence generator). 3. Chat Server 1 puts the message in the message sync queue. 4. The message is stored in the key-value store for persistence. 5. Service discovery determines which chat server User B is connected to. If User B is online, the message is forwarded to Chat Server 2 (B's WebSocket server), which pushes it to B's client via WebSocket. If User B is offline, the message is forwarded to the push notification server, which delivers a push notification via APNS or FCM.

A user may be logged in on both a phone and a laptop simultaneously. Both devices must receive all messages.

Each device maintains a local variable cur_max_message_id — the highest message ID the device has received. When a device comes online (after being offline, or after reconnecting), it queries the key-value store for all messages with message_id > cur_max_message_id for the user. This pulls any messages missed while the device was offline.

For real-time delivery: when a message arrives for a user, all of that user's currently active chat server connections are notified. Each active device receives the message via its WebSocket connection.

When User A sends a message to a group chat with members A, B, and C:

1. The message is stored in the group's message channel in the key-value store. 2. A copy of the message is placed in the message sync queue for each group member (B and C). Each member has their own "inbox" — a message sync queue that aggregates messages from all conversations they participate in. 3. When B or C's chat server detects a new message in their inbox queue, it pushes the message to their active WebSocket connections.

This design simplifies message sync for clients: each client only needs to check its own inbox to discover new messages, regardless of how many group chats it participates in. For small groups (up to 100 members as in the design requirements), storing a message copy per recipient is acceptable.

Presence servers manage online/offline status. When a user logs in and establishes a WebSocket connection to a chat server, the chat server informs the presence server. The presence server stores the user's online status and last_active_at timestamp in the key-value store.

When a user logs out, the presence server updates their status to offline. When a user's connection drops unexpectedly (network interruption), the system must handle transient disconnections gracefully — a user who briefly loses signal should not appear offline to all friends.

Heartbeat mechanism: online clients send a heartbeat event to the presence server every 5 seconds. If the presence server does not receive a heartbeat within 30 seconds, it marks the user offline. This prevents brief network glitches from triggering an offline event.

Status propagation: when User A's status changes, their friends need to be notified. Presence servers use a publish-subscribe model. Each friendship pair has a dedicated channel. Status changes are published to the relevant channels and delivered to subscribed friends.